> ## Documentation Index
> Fetch the complete documentation index at: https://docs.getcollate.io/llms.txt
> Use this file to discover all available pages before exploring further.

# OpenLineage Compatibility

> Send OpenLineage events to Collate and understand dataset resolution, supported facets, and ingestion limits.

# OpenLineage Compatibility

Collate accepts OpenLineage `RunEvent` payloads and converts their input and output datasets into lineage edges. You can send events directly over HTTP or consume events through the OpenLineage pipeline connector.

## Choose an Ingestion Method

Use the method that matches where your producer sends events.

| Method | Configuration | Use Case |
| - | - | - |
| HTTP API | Point the producer's HTTP transport to `/api/v1/openlineage/lineage`. | Send events directly from a client or integration. |
| Kafka or AWS Kinesis | Configure the [OpenLineage connector](https://docs.getcollate.io/ai-2-0/connectors/pipeline/openlineage.md) to read the broker or stream. | Consume events already published to Kafka or Kinesis. |
| Native Lineage API | Send entity references to `/api/v1/lineage`. | Create explicit relationships between assets already in Collate. |

The HTTP API runs in the Collate server. It does not require a Kafka broker, Kinesis stream, or scheduled connector workflow. The connector has its own connection and ingestion settings. The HTTP settings below apply to direct event ingestion.

## Prepare Collate

Before you send events, complete these steps:

1. Create or ingest the database services, databases, schemas, and tables referenced by the events. Include their columns if you need column-level lineage.
2. Obtain an Collate JSON Web Token (JWT) for a user or bot with `EditLineage` permission on the `table` resource.
3. Configure namespace mappings so each dataset resolves to the intended database service.
4. To associate events with pipelines, create or ingest the pipeline service named by `defaultPipelineService`. Its default name is `openlineage`.

For more information, see [API Authentication](https://docs.getcollate.io/ai-2-0/developer-guide/api-reference/authentication.md).

### Match Datasets to Catalog Assets

An OpenLineage dataset has a `namespace` and a `name`. The namespace identifies its source system. The name identifies the dataset within that system.

For example, this dataset can resolve to the Collate table `prod-postgres.analytics.public.orders`:

```json theme={null}
{
  "namespace": "postgresql://warehouse.example.com:5432",
  "name": "analytics.public.orders"
}
```

Map `postgresql://warehouse.example.com:5432` to the existing database service `prod-postgres` through `namespaceToServiceMapping`. Use the exact namespace emitted by your producer. Include the database and schema in the dataset name when available.

The resolver tries the namespace mapping, datasource facet information, and table-name suffix matches. A namespace is not automatically an Collate service name. Suffix matches can be ambiguous when several services contain the same database, schema, and table names.

For more information, see [Cross Database Lineage](https://docs.getcollate.io/ai-2-0/how-to-guides/data-lineage/cross-database-lineage.md).

### Configure HTTP Ingestion Settings

An administrator can read the current configuration with `GET /api/v1/system/settings/openLineageSettings`. To update it, send a settings object to `PUT /api/v1/system/settings`.

Preserve existing mappings and other settings when you update the configuration. The following example processes completed runs against assets already in the catalog:

```json theme={null}
{
  "configType": "openLineageSettings",
  "configValue": {
    "enabled": true,
    "autoCreateEntities": false,
    "eventTypeFilter": ["COMPLETE"],
    "defaultPipelineService": "openlineage",
    "namespaceToServiceMapping": {
      "postgresql://warehouse.example.com:5432": "prod-postgres"
    }
  }
}
```

The API is enabled by default. If `eventTypeFilter` is absent or empty, the server processes `COMPLETE` events. It also treats an omitted `eventType` as eligible when `COMPLETE` is allowed. Send an explicit event type to make the producer's intent clear.

`autoCreateEntities` defaults to `true`. In this release, the server can create missing tables only under an existing database schema. It can also create pipelines under an existing `defaultPipelineService`. This setting does not create database services, databases, or schemas. Ingest those assets before sending events.

## Configure an HTTP Producer

For the OpenLineage Python client, save this configuration as `openlineage.yml`:

```yaml theme={null}
transport:
  type: http
  url: https://your-company.getcollate.io
  endpoint: /api/v1/openlineage/lineage
  auth:
    type: api_key
    api_key: "<COLLATE_JWT>"
```

Replace the host and token with your deployment values. Set `OPENLINEAGE_CONFIG` to the file path if the file is outside the client's working directory. The `api_key` authentication provider sends the Collate JWT in the `Authorization: Bearer` header.

Set the endpoint explicitly. The client's default `/api/v1/lineage` path is the native Collate Lineage API, which expects a different request body.

For more information, see the [OpenLineage Python Client Configuration](https://openlineage.io/docs/client/python/configuration/#http-transport). Other clients and integrations have their own configuration syntax. Configure the same URL, endpoint, and bearer token using that client's HTTP transport options.

## Understand the Supported Metadata

The HTTP mapper uses the following event fields and facets:

| Event Metadata | Collate Behavior |
| - | - |
| `inputs` and `outputs` | Creates a lineage edge for each resolved input and output pair. Both lists must contain a dataset. |
| Dataset `schema` facet | Supplies column names and types when the server creates a table. It does not replace metadata ingestion for existing tables. |
| Job `sql` facet | Stores `job.facets.sql.query` on the lineage edge. SQL text alone does not supply input and output datasets. |
| Output `columnLineage` facet | Maps output fields to input fields. The mapper reads `outputFacets.columnLineage` first, then `facets.columnLineage`. |
| Run `parent` facet | Uses the parent job to resolve the pipeline reference. Without a usable parent job, it uses the event's job. |

For column-level lineage, each `inputFields` entry must match an input dataset's namespace and name. Its `field` must identify a catalog column. In this release, ingest the input tables and columns before sending column mappings.

The mapper can resolve storage datasets to existing container assets by path. It does not create missing containers. Compatibility depends on the dataset names and facets emitted by your integration. An accepted event does not mean that every custom facet becomes Collate metadata.

## Send an Event and Verify Lineage

Use the complete request examples in the [OpenLineage Events API](https://docs.getcollate.io/ai-2-0/developer-guide/api-reference/lineage/openlineage.md) to send a single event or a batch.

After sending an event, check `lineageEdgesCreated`. A `200` response with `status: success` can contain zero edges. Filtered event types, missing inputs or outputs, unresolved datasets, and failed edge writes can all produce that result.

If no edges appear, check the event type and dataset names first. Then check namespace mappings, existing catalog assets, and the Collate server logs. Use the [Get Lineage API](https://docs.getcollate.io/ai-2-0/developer-guide/api-reference/lineage/get.md) or the asset's **Lineage** tab to confirm the resulting graph.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.