dbt Workflow

Configure dbt workflow from OpenMetadata UI

Run dbt Workflow Externally

Configure the dbt Workflow from the CLI.

Auto Ingest dbt Artifacts (dbt-core)

Configure the auto dbt ingestion for dbt-core.

dbt Integration

Feature	Status
Stage	PROD
dbt Queries
dbt Lineage
dbt Tags
dbt Tiers
dbt Glossary
dbt Owner
dbt Descriptions
dbt Tests
dbt Exposures
Supported dbt Core Versions	`v1.2` `v1.3` `v1.5` `v1.5` `v1.6` `v1.7` `v1.8` `v1.9`

Requirements

AWS S3

If we have the artifacts on the bucket MyBucket, the user running the ingestion should have, at least, the permissions from the following policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::MyBucket",
                "arn:aws:s3:::MyBucket/*"
            ]
        }
    ]
}

Note that it’s not enough to point the resource to arn:aws:s3:::MyBucket. We need its contents as well!

OpenMetadata integrates the below metadata from dbt

1. dbt Queries

Queries used to create the dbt models can be viewed in the dbt tab dbt-query

2. dbt Lineage

Lineage from dbt models can be viewed in the Lineage tab. For more information on how lineage is extracted from dbt take a look here dbt-lineage

To capture lineage, the compiled_code field must be present in the manifest.json file.

If compiled_code is missing, lineage will not be captured for that node.
To ensure compiled_code is populated in your dbt manifest, run the following commands in your dbt project:
- dbt compile
- dbt docs generate

3. dbt Tags

Table and column level tags can be imported from dbt Please refer here for adding dbt tags dbt-tags

4. dbt Owner

Owner from dbt models can be imported and assigned to respective tables Please refer here for adding dbt owner dbt-owner

5. dbt Descriptions

Descriptions from dbt manifest.json and catalog.json can be imported and assigned to respective tables and columns. For more information and to control how the table and column descriptions are updated from dbt please take a look here dbt-descriptions

6. dbt Tests and Test Results

Tests from dbt will only be imported if the run_results.json file is passed. dbt-tests

7. dbt Tiers

Table and column level Tiers can be imported from dbt Please refer here for adding dbt tiers dbt-tiers

8. dbt Glossary

Table and column level Glossary can be imported from dbt Please refer here for adding dbt glossary dbt-glossary

Troubleshooting

For any issues please refer to the troubleshooting documentation here