Setup Multiple dbt Projects

Collate supports ingesting metadata from multiple dbt projects in a single workflow — each project can have its own manifest.json, catalog.json, and run_results.json files.

Tip: Multiple dbt project support is available for S3, GCS, and Azure configurations only.

Before you configure the workflow, organize your dbt files so Collate can find them:

Place each project’s files in its own directory and name them manifest.json, catalog.json, and run_results.json.
If your dbt tests produce multiple run_results.json files, keep them in the same directory as their corresponding manifest.json. Add a unique suffix to each file name — for example: run_results_one.json, run_results_two.json, run_results_three.json.

Collate scans the prefix path you specify and traverses each subfolder to locate the dbt files automatically.

Resolve Metadata From Overlapping Projects

More than one manifest can reference the same Collate table. Collate processes each manifest separately, and the result depends on the metadata field and the dbt agent configuration:

Metadata	Resolution Behavior
Lineage	With Override Lineage disabled, which is the default, Collate adds each manifest’s dbt lineage edges to the existing edges. The resulting lineage is additive across manifests. Enabling Override Lineage allows a later ingestion to replace existing dbt lineage for the affected table, so do not enable it when you need the combined graph.
dbt tab	A table stores one dbt data model. Each matching manifest updates that model, so the dbt tab shows the manifest processed most recently. Do not rely on folder names or artifact timestamps to control processing order.
Descriptions	With Update Descriptions disabled, which is the default, a dbt description fills an empty table or column description but does not replace an existing one. When enabled, a non-empty description from the manifest processed most recently overwrites the current value.
Owners	With Update Owners disabled, which is the default, a dbt owner can populate an empty table owner but does not replace an existing owner. When enabled, a non-empty owner from the manifest processed most recently overwrites the current owner.
Tags	With Include Tags enabled, Collate merges table and column tags from each manifest with the tags already on the asset. Tags therefore accumulate across runs. For a mutually exclusive classification, an incoming tag replaces an existing tag from the same classification.

If overlapping projects define different descriptions, owners, or dbt model details for the same table, choose one project as the authority and prevent the other projects from matching that table. You can use the database, schema, and table filter patterns to keep project scopes distinct.

How to Configure Multiple dbt Projects (S3 Example)

Follow these steps to set up the dbt workflow for multiple projects using an S3 bucket.

Step 1: Organize the dbt Files in Your S3 Bucket

Arrange each dbt project in a separate directory inside your bucket. For example, here’s a bucket named dbt_bucket with three dbt projects:

bucket_home/
├── dbt_files/
    ├── dbt_project_one/
    │   ├── manifest.json
    │   ├── catalog.json
    │   └── run_results.json
    └── dbt_new_projects/
        ├── dbt_project_two/
        │   ├── manifest.json
        │   ├── catalog.json
        │   ├── run_results_one.json
        │   ├── run_results_two.json
        │   └── run_results_three.json
        └── dbt_project_three/
            ├── manifest.json
            ├── catalog.json
            └── run_results.json

Step 2: Configure the dbt Workflow in Collate

Set the bucket and prefix path so Collate knows where to look.

In the dbt Bucket Name field, enter your bucket name — in this example, dbt_bucket.
In the dbt Object Prefix field, enter the path to the parent folder containing your projects — in this example, bucket_home/dbt_files/.

To scan the entire bucket, leave the dbt Object Prefix field empty.

Step 3: Run the dbt Workflow

Trigger the workflow and verify all three projects are ingested. After the workflow completes, Collate ingests the dbt metadata from all three projects in a single run.

Connectors

Connectors

Setup Multiple dbt Projects

Setup Multiple dbt Projects

Resolve Metadata From Overlapping Projects

How to Configure Multiple dbt Projects (S3 Example)

Step 1: Organize the dbt Files in Your S3 Bucket

Step 2: Configure the dbt Workflow in Collate

Step 3: Run the dbt Workflow

​Setup Multiple dbt Projects

​Resolve Metadata From Overlapping Projects

​How to Configure Multiple dbt Projects (S3 Example)

​Step 1: Organize the dbt Files in Your S3 Bucket

​Step 2: Configure the dbt Workflow in Collate

​Step 3: Run the dbt Workflow

Setup Multiple dbt Projects

Resolve Metadata From Overlapping Projects

How to Configure Multiple dbt Projects (S3 Example)

Step 1: Organize the dbt Files in Your S3 Bucket

Step 2: Configure the dbt Workflow in Collate

Step 3: Run the dbt Workflow