Deploy a Lineage Workflow

Collate gives you three ways to build lineage for your data assets — automatically through metadata ingestion, through a dedicated Lineage Agent that reads query logs, or manually through the UI. The lineage workflow is supported for all major database connectors, including BigQuery, Snowflake, Redshift, MSSQL, Clickhouse, PostgreSQL, Databricks, Athena, MySQL, Oracle, and many more.

View Lineage from Metadata Ingestion

When metadata ingestion runs, Collate automatically checks whether each table is a view. For sources where Collate can retrieve the view definition query, it pulls in view lineage alongside the regular metadata — no extra configuration needed. Here’s how Collate builds view lineage:

Ingest all tables: Collate completes metadata ingestion for all tables in the service.
Identify views: Collate flags any table whose type is View.
Parse the view query: Collate extracts the SQL query used to create the view.
Resolve source and target tables: Collate checks whether the referenced tables exist in Collate.
Create the lineage relationship: Collate links the source and target tables in the lineage graph.

Collate generates both table-level and column-level lineage from views automatically.

Lineage Agent from UI

The Lineage Agent reads query logs from your database and uses them to build lineage. Metadata ingestion captures view lineage only — the Lineage Agent captures all query-based lineage, giving you a more complete picture of how data flows through your systems.

Step 1: Add a Lineage Agent

Navigate to your service and launch the Lineage Agent wizard.

In the left navigation, click Connections and select your service.
Open the Agents tab and click + Add Agent > Lineage.
Give the agent a recognisable name and click Next.

Step 2: Configure the Agent

Configure the core parameters for this agent.

Agent Setup

Field	Default	Description
Query Log Duration	`1`	How many days back to look in the query log.
Query Log File Path	—	Path to a query log file. Use this instead of reading directly from the database.
Result Limit	`1000`	Maximum number of query log rows to process per run.
Parsing Timeout Limit	`300`	Timeout in seconds for parsing a single SQL query.
Query Parser Type	`Auto`	SQL parser for lineage extraction. Auto tries SqlGlot first, then SqlFluff, then SqlParse. SqlGlot offers high performance with excellent dialect support. SqlFluff provides comprehensive dialect support.
Filter Condition	—	SQL condition applied to the query history result set to exclude specific queries. See Usage Query Filtering.
Number of Threads	`1`	Number of threads to use for parallel lineage ingestion.
Cross Database Service Names	—	Names of other Collate services to search when resolving table references during lineage extraction. List specific service names. The underlying lookup returns a limited number of matches, so `*` can miss the correct service or resolve to the wrong table when many services or tables share names.
Override View Lineage	Off	When enabled, overwrites existing view lineage on each run.
Process View Lineage	On	Parse view definitions for lineage.
Process Query Lineage	On	Parse query logs for lineage.
Process Stored Procedure Lineage	On	Parse stored procedure definitions for lineage.
Process Cross Database Lineage	Off	Build lineage across tables in different databases or projects (for example, across BigQuery projects). Also set Cross Database Service Names when enabling this option. See Cross Database Lineage.
Enable Temp Table Lineage	Off	Handle lineage for Snowflake temporary and transient tables.
Incremental Lineage Processing	On	Process only new or changed lineage data on each run.

Filter Patterns Apply regex patterns to include or exclude specific databases, schemas, tables, or stored procedures. Scope & Behaviour

Field	Description
Enable Debug Log	Turn on verbose logging to help troubleshoot ingestion issues.

Step 3: Schedule and Deploy

Set your desired run schedule and click Add & Deploy.

Run Lineage Workflow Externally

Lineage

After running a Metadata Ingestion workflow, we can run Lineage workflow. While the serviceName will be the same to that was used in Metadata Ingestion, so the ingestion bot can get the serviceConnection details from the server.

1. Define the YAML Config

This is a sample config for Lineage:

You can learn more about how to configure and run the Lineage Workflow to extract Lineage data from here

2. Run with the CLI

After saving the YAML config, we will run the command the same way we did for the metadata ingestion:

metadata ingest -c <path-to-yaml>

Lineage from dbt

Collate builds lineage from dbt ingestion too. When you run a dbt workflow, Collate reads your manifest.json to extract query history and table relationships — including column-level lineage. To set this up, provide the path to your catalog.json and manifest.json files in the dbt workflow configuration. Learn more about dbt lineage and lineage ingestion.

Lineage from Query Logs (CSV)

If your connector doesn’t support the Lineage Agent, you can still bring in lineage by uploading a CSV file of query logs. See Lineage Workflows using Query Logs for setup instructions.

Manual Lineage

You can also add and edit lineage directly in the Collate UI — without running any ingestion. See adding lineage manually.

Explore the Lineage View

Explore the rich lineage view in Collate.

​Deploy a Lineage Workflow

​View Lineage from Metadata Ingestion

​Lineage Agent from UI

​Step 1: Add a Lineage Agent

​Step 2: Configure the Agent

​Step 3: Schedule and Deploy

​Run Lineage Workflow Externally

​Lineage

​1. Define the YAML Config

​2. Run with the CLI

​Lineage from dbt

​Lineage from Query Logs (CSV)

​Manual Lineage