Skip to main content

​
Ingest Lineage from dbt

Collate reads lineage information from your dbt manifest.json file and uses it to build upstream and downstream traceability for your data models. Collate extracts lineage from two sources in the manifest file: the depends_on key (node-level relationships) and the compiled_code or compiled_sql key (query-level relationships).
Note: To capture lineage, the compiled_code field must be present in your manifest.json.
  • If compiled_code is missing, Collate will not capture lineage for that node.
  • To make sure compiled_code is populated, run one of the following commands in your dbt project before ingestion:
    • dbt compile
    • dbt docs generate

​
How Collate Builds dbt Lineage

Collate uses two approaches to build lineage — node dependencies and query parsing.

​
Step 1: Lineage from depends_on

Collate reads node-level lineage from the depends_on.nodes field in manifest.json. Here’s an example — the customers model depends on three staging models:
Collate builds the lineage graph from these dependencies: dbt lineage graph for customers model

​
Step 2: Lineage from dbt Queries

Collate also parses the compiled SQL query for each model to extract finer-grained source and target relationships. The query sits under node_name → compiled_code (or compiled_sql in older dbt versions):
Collate runs this query through its lineage parser to identify the source and target tables and add them to the lineage graph.
Note: If the lineage parser can’t interpret the query, lineage for that node won’t be created. Check the ingestion logs for parsing errors.