dbt Artifact Configuration

Using dbt Cloud? You don’t need storage configuration. Go directly to the dbt Cloud API guide for a simpler setup.

This guide is for dbt Core users. When using dbt Core, Collate needs access to dbt-generated artifacts (manifest.json, catalog.json, run_results.json) to extract metadata, lineage, and test results. Since dbt Core runs within your infrastructure, you must configure a storage method to make these artifacts accessible to Collate.

How It Works

If you’re using dbt Core, you need to set up artifact storage:

Generate artifacts: Run dbt run, dbt test, and dbt docs generate to create manifest.json, catalog.json, and run_results.json
Upload to storage: Configure your workflow to upload these files to S3, GCS, Azure, HTTP server, or shared filesystem
Configure Collate: Point Collate to the storage location to pull and process the artifacts

Storage Options for dbt Core

Choose the storage method that matches your infrastructure:

AWS S3

Complete S3 setup with Airflow DAG, IAM policies, and configuration

Google Cloud Storage

GCS bucket setup with service accounts and Cloud Composer integration

Azure Blob Storage

Azure storage account setup with managed identity and container config

HTTP Server

Nginx, Apache, or S3+CloudFront configuration for artifact hosting

Local Filesystem

Docker volumes, Kubernetes PVC, or NFS shared filesystem setup

Quick Setup Summary

AWS S3

# Create bucket
aws s3 mb s3://your-dbt-artifacts

# Upload artifacts (manual)
aws s3 sync target/ s3://your-dbt-artifacts/dbt/ --include "*.json"

# Or use automated Airflow DAG (see detailed guide)

View complete S3 guide →

Google Cloud Storage

# Create bucket
gsutil mb gs://your-dbt-artifacts

# Upload artifacts (manual)
gsutil -m cp target/*.json gs://your-dbt-artifacts/dbt/

# Or use Cloud Composer DAG (see detailed guide)

View complete GCS guide →

Azure Blob Storage

# Create storage account and container
az storage account create --name yourdbtartifacts
az storage container create --name dbt-artifacts

# Upload artifacts
az storage blob upload-batch \
  --destination dbt-artifacts \
  --source target/ \
  --pattern "*.json"

View complete Azure guide →

HTTP Server

# Upload via rsync to static file server
rsync -avz target/*.json user@server:/var/www/dbt/

# Or configure S3 + CloudFront for HTTPS access

View complete HTTP guide →

Local/Shared Filesystem

# Docker Compose volume mount example
volumes:
  - ./dbt/target:/dbt-artifacts

# Collate reads directly from /dbt-artifacts path

View complete Local guide →

Common Requirements Across All Methods

Regardless of which storage method you choose, you need:

1. Required dbt Artifacts

File	Generated By	Purpose
`manifest.json`	`dbt run`, `dbt compile`, `dbt build`	Required - Model definitions, sources, lineage, tests
`catalog.json`	`dbt docs generate`	Recommended - Column names, types, descriptions
`run_results.json`	`dbt test`, `dbt run`, `dbt build`	Optional - Test execution results and timing

2. dbt Command Sequence

Run these commands to generate all artifacts:

dbt run          # Execute models, generates manifest.json
dbt test         # Run tests, updates run_results.json
dbt docs generate # Generate catalog.json with column metadata

3. Collate Configuration

After artifacts are accessible, configure Collate ingestion:

UI Method: See Configure dbt Workflow
CLI Method: See Run dbt Workflow Externally
Auto-ingest: See Auto Ingest dbt Core

Troubleshooting Common Issues

Issue	Possible Cause	Solution
Artifacts not found	Upload failed or wrong path	Verify artifacts uploaded successfully to correct location
Access denied	Insufficient permissions	Check IAM policies, service account permissions, or access keys
Stale metadata	Old artifacts	Ensure upload happens after dbt completes; verify timestamps
Missing lineage	No compiled_code in manifest	Run `dbt compile` or `dbt docs generate` before upload
Missing column descriptions	No catalog.json	Ensure `dbt docs generate` runs and catalog.json is uploaded
Test results not showing	No run_results.json	Ensure `dbt test` runs and run_results.json is uploaded

For storage-specific troubleshooting, see the individual guides.

Next Steps

Choose your storage method using the decision matrix above
Follow the detailed guide for your chosen method
Configure Collate ingestion after artifacts are accessible
Set up scheduling to keep metadata synchronized

Questions? See the main dbt Overview or dbt Troubleshooting guide.

​dbt Artifact Configuration

​How It Works

​Storage Options for dbt Core

AWS S3

Google Cloud Storage

Azure Blob Storage

HTTP Server

Local Filesystem

​Quick Setup Summary

​AWS S3

​Google Cloud Storage

​Azure Blob Storage

​HTTP Server

​Local/Shared Filesystem

​Common Requirements Across All Methods

​1. Required dbt Artifacts

​2. dbt Command Sequence

​3. Collate Configuration

​Troubleshooting Common Issues

​Next Steps

dbt Artifact Configuration

How It Works

Storage Options for dbt Core

Quick Setup Summary

AWS S3

Google Cloud Storage

Azure Blob Storage

HTTP Server

Local/Shared Filesystem

Common Requirements Across All Methods

1. Required dbt Artifacts

2. dbt Command Sequence

3. Collate Configuration

Troubleshooting Common Issues

Next Steps