Usage Workflow

Learn how to configure the Usage workflow from the UI to ingest Query history data from your data sources. This workflow is available ONLY for the following connectors:

If your database service is not yet supported, you can use this same workflow by providing a Query Log file! Learn how to do so 👇

Usage Workflow through Query Logs

Configure the usage workflow by providing a Query Log file.

UI Configuration

Once the metadata ingestion runs correctly and we are able to explore the service Entities, we can add Query Usage information. This will populate the Queries tab from the Table Entity Page.

We can create a workflow that will obtain the query log and table creation information from the underlying database and feed it to Collate. The Usage Ingestion will be in charge of obtaining this data.

Step 1: Add a Usage Ingestion

On the Connections page, select the service type.
Go to the Agents tab.
Click Add Agent and select Usage.

Step 2: Agent Setup

Configure the core parameters for the Usage agent:

Field	Description
Query Log Duration	How many days back to look in query logs to process usage data. Default is `1`.
Stage File Location	Absolute file path for the temporary file used to store query logs before processing. Default is `/tmp/query_log`.
Filter Condition	Optional condition to filter which queries are included from the query history.
Result Limit	Maximum number of query log results to process at a time. Default is `1000`.
Query Log File Path	File path to the query logs, if reading from a file rather than querying the source directly.
Process Query Cost	Toggle to enable processing of query cost data alongside usage.

Step 3: Schedule and Deploy

Choose how the usage agent should run:

Schedule — Runs repeatedly based on a defined interval. Set the Frequency (Hourly, Daily, Weekly, Monthly, or Custom) and the Time to run.
On Demand — Runs once when manually triggered.

Configure the remaining options:

Number of Retries — How many times to retry if the ingestion fails.
Raise on Error — Toggle on to surface errors immediately rather than silently skipping them.

Click Add & Deploy to save and activate the usage agent.

YAML Configuration

In the connectors section we showcase how to run the metadata ingestion from a JSON/YAML file using the Airflow SDK or the CLI via metadata ingest. Running a usage workflow is also possible using a JSON/YAML configuration file. This is a good option if you wish to execute your workflow via the Airflow SDK or using the CLI; if you use the CLI a usage workflow can be triggered with the command metadata usage -c FILENAME.yaml. The serviceConnection config will be specific to your connector (you can find more information in the connectors section), though the sourceConfig for the usage will be similar across all connectors.

Query Usage

The Query Usage workflow will be using the query-parser processor. After running a Metadata Ingestion workflow, we can run Query Usage workflow. While the serviceName will be the same to that was used in Metadata Ingestion, so the ingestion bot can get the serviceConnection details from the server.

1. Define the YAML Config

This is a sample config for Usage:

2. Run with the CLI

After saving the YAML config, we will run the command the same way we did for the metadata ingestion:

metadata usage -c <path-to-yaml>

Guide for Data Users

Usage Workflow Guide | Ingestion Workflows

Usage Workflow

Usage Workflow through Query Logs