Skip to main content

​
Metadata Ingestion - Incremental Extraction

The default Metadata Ingestion roughly follows these steps:
  1. Fetch all the information from the Source.
  2. Compare the information with Collate to update it properly.
  3. Compare the information with Collate to delete entities that were deleted.
While on one hand this is a great simple way of doing things that works for most use cases since at every ingestion pipeline run we get the whole Source state, on the other hand this is fetching and comparing a lot of data without need since if there were no structural changes we already know there is nothing to update in Collate. We implemented the Incremental Extraction feature to improve performance by reducing the extraction and comparison of unneeded data. How this is done depends a lot on the Source itself, but the general idea is to follow these steps:
  1. Fetch the last successful pipeline run.
  2. Add a small safety margin.
  3. Get all the structural changes since then.
  4. Flag deleted entities.
  5. Fetch/Compare only the entities with structural changes.
  6. Delete entities flagged for deletion.

​
Validate Incremental Runs

Validate incremental extraction after enabling it and after changing source permissions or filters:
  1. Make a known structural change in the source, such as creating a test table or adding a column.
  2. Run the incremental Metadata Agent.
  3. On the service detail page, open the Agents tab and confirm that the run completed successfully. If the run failed or completed with partial success, review its logs before validating the catalog.
  4. Open the affected asset in Collate and confirm that the structural change appears. For a deletion, confirm that the asset is marked as deleted when the agent has the corresponding Mark Deleted option enabled.
  5. Review the run logs for warnings about source change-history access, skipped objects, or filters. A successful run confirms that the workflow completed, but you should still verify representative source changes in Collate.
Incremental extraction uses the most recent successful run for that agent as its starting point. If no successful run is found within Successful Pipeline Run Lookback Days, the agent performs a full extraction instead.

​
Schedule a Periodic Full Run

Use two Metadata Agents for the same database service when you want frequent incremental updates and a periodic full reconciliation:
  1. Keep Incremental Extraction enabled on the frequently scheduled agent.
  2. Add a second Metadata Agent for the same service and leave Incremental Extraction disabled.
  3. Give the agents distinct names so their run histories are easy to identify.
  4. Schedule the full agent less frequently, such as weekly, and avoid overlapping the two schedules.
  5. Use the full run to reconcile the complete source state. Review its logs and spot-check any metadata that was missing after incremental runs.
Configure both agents with the same database, schema, and table filters. Different scopes can produce inconsistent results when any Mark Deleted option is enabled.

​
External Ingestion

When using the Incremental Extraction feature with External Ingestions (ingesting using YAML files instead of setting it up from the UI), you must pass the ingestion pipeline fully qualified name to the configuration. This should be {service_name}{pipeline_name} Example:

​
Feature available for

​
Databases

BigQuery

BETA | Collate

Redshift

BETA | Collate

Snowflake

BETA | Collate