Ingest Lineage from dbt
Ingest the lineage information from dbt manifest.json file into OpenMetadata.
OpenMetadata extracts the lineage information from the depends_on and compiled_query/compiled_code keys from the manifest file.
To capture lineage, the compiled_code field must be present in the manifest.json file.
- If
compiled_codeis missing, lineage will not be captured for that node. - To ensure
compiled_codeis populated in your dbt manifest, run the following commands in your dbt project:dbt compiledbt docs generate
1. Lineage information from dbt "depends_on" key
Openmetadata fetches the lineage information from the manifest.json file. Below is a sample manifest.json file node containing lineage information under node_name->depends_on->nodes.
For the above case the lineage will be created as shown in below:
dbt Lineage
2. Lineage information from dbt queries
Openmetadata fetches the dbt query information from the manifest.json file.
Below is a sample manifest.json file node containing dbt query information under node_name->compiled_code or node_name->compiled_sql.
The query from dbt will be parsed by the Lineage parser to extract source and target tables to create the lineage.
The lineage may not be created if the lineage parser is not able to parse the query. Please check the logs for any errors in this case.