Skip to main content

​
Troubleshooting

​
Workflow Deployment Error

If there were any errors during the workflow deployment process, the Ingestion Pipeline Entity will still be created, but no workflow will be present in the Ingestion container.
  • You can then Edit the Ingestion Pipeline and Deploy it again.
  • From the Connection tab, you can also Edit the Service if needed.

​
Connector Debug Troubleshooting

This section provides instructions to help resolve common issues encountered during connector setup and metadata ingestion in Collate. Below are some of the most frequently observed troubleshooting scenarios.

​
How to Enable Debug Logging for Any Ingestion

To enable debug logging for any ingestion workflow in Collate:
  1. Navigate to Services Go to Settings > Services > Service Type (e.g., Database) in the Collate UI.
  2. Select a Service Choose the specific service for which you want to enable debug logging.
  3. Access Ingestion Tab Go to the Ingestion tab and click the three-dot menu on the right-hand side of the ingestion type, and select Edit.
  4. Enable Debug Logging In the configuration dialog, enable the Debug Log option and click Next.
  5. Schedule and Submit Configure the schedule if needed and click Submit to apply the changes.

​
Permission Issues

If you encounter permission-related errors during connector setup or metadata ingestion, ensure that all the prerequisites and access configurations specified for each connector are properly implemented. Refer to the connector-specific documentation to verify the required permissions.

​
File Not Found or Access Denied

If you encounter file access errors when connecting to local Access database files: 
Error: [Errno 2] No such file or directory: '/path/to/database.accdb'
or 
PermissionError: [Errno 13] Permission denied: '/path/to/database.accdb'
This indicates that the ingestion process cannot access the specified .accdb or .mdb file.

​
Resolution

  1. Verify the File Path: Ensure the path to the Access database file is correct and the file exists.
  2. Check File Permissions: Verify that the user running the ingestion process has read access to the file.
  3. For Directory Paths: If pointing to a directory, ensure the directory exists and contains .accdb or .mdb files.
  4. Path Expansion: If using ~ in the path, ensure it expands correctly to the user’s home directory.

​
S3 Access Denied

If you encounter S3 permission errors: 
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the GetObject operation
This indicates that the AWS credentials do not have the required permissions to access the S3 bucket.

​
Resolution

  1. Verify AWS Credentials: Ensure the AWS Access Key ID and Secret Access Key are correct.
  2. Check IAM Permissions: Verify that the IAM user or role has the following permissions:
    • s3:GetObject - to read files from the bucket
    • s3:ListBucket - to list files in the bucket
  3. Bucket Policy: Check if the S3 bucket policy allows access from your credentials.
  4. Correct Region: Ensure the AWS Region is set correctly for your S3 bucket.
Example IAM policy: 
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::<your-bucket-name>",
                "arn:aws:s3:::<your-bucket-name>/*"
            ]
        }
    ]
}

​
Corrupted or Invalid Database File

If you encounter errors indicating the file is not a valid Access database: 
Error: File is not a valid Microsoft Access database
This indicates the file is corrupted or is not an Access database file.

​
Resolution

  1. Verify File Type: Ensure the file has a .accdb or .mdb extension and is a valid Microsoft Access database.
  2. Check File Integrity: Try opening the file in Microsoft Access to verify it’s not corrupted.
  3. File Transfer Issues: If downloaded from S3 or copied over a network, ensure the file was transferred completely without corruption.
  4. Supported Formats: Verify that the file format is supported (Access 2007+ .accdb or older .mdb formats).

​
S3-Compatible Storage (MinIO) Connection Issues

If you encounter connection errors when using S3-compatible storage like MinIO: 
EndpointConnectionError: Could not connect to the endpoint URL
This indicates the endpoint URL is incorrect or unreachable.

​
Resolution

  1. Verify Endpoint URL: Ensure the endpoint URL is correct (e.g., http://localhost:9000 for local MinIO).
  2. Check Network Access: Verify that the system running the ingestion can reach the endpoint.
  3. SSL/TLS Issues: For HTTPS endpoints, ensure SSL certificates are valid.
  4. Port Configuration: Verify the port number is correct in the endpoint URL.

​
AssumeRole Authentication Failed

If you encounter errors when using IAM role assumption: 
Error: Not authorized to perform sts:AssumeRole
This indicates the AWS credentials do not have permission to assume the specified role.

​
Resolution

  1. Verify Role ARN: Ensure the Assume Role ARN is correct and exists.
  2. Trust Relationship: Check that the IAM role’s trust policy allows your AWS credentials to assume it.
  3. Role Permissions: Verify the assumed role has the necessary S3 permissions.
  4. AWS Region: Ensure the AWS Region is specified when using AssumeRole authentication.
Example trust policy for the IAM role: 
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:user/your-user"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

​
No Tables Found

If the ingestion completes but no tables are found: 
Warning: No tables found in database
This may occur if the Access database is empty or all tables are filtered out.

​
Resolution

  1. Verify Database Contents: Open the Access database and verify it contains tables.
  2. Check Table Filters: Review the Table Filter Pattern to ensure it’s not excluding all tables.
  3. System Tables: Microsoft Access system tables (starting with MSys) are automatically excluded.
  4. Directory Ingestion: If pointing to a directory, ensure .accdb or .mdb files in the directory contain tables.