Ingestion & Connectors

Five hard breaks here: the pipelineStatuses shape, the Databricks Pipeline connection, the log-stream payload, the progress payload and the move to Python 3.12, plus a Great Expectations major-version jump.

`pipelineStatuses` is now an array

Breaking. Affects every client reading ingestion pipelines: dashboards, health checks and SDK users.

  "pipelineStatuses": {
-   "description": "Last of executions and status for the Pipeline.",
-   "$ref": "#/definitions/pipelineStatus"
+   "description": "List of the most recent executions and status for the Pipeline.",
+   "type": "array",
+   "items": { "$ref": "#/definitions/pipelineStatus" }
  }

The list endpoint now returns the last five statuses per pipeline, which also removes an N+1 query.

1.13 response
2.0 response

{ "name": "my_pipeline", "pipelineStatuses": { "runId": "…", "pipelineState": "success" } }

{ "name": "my_pipeline",
  "pipelineStatuses": [ { "runId": "…", "pipelineState": "success" },
                        { "runId": "…", "pipelineState": "failed" } ] }

Strongly typed clients throw on the type change. Loosely typed scripts reading pipelineStatuses.pipelineState silently get undefined or None. The migration also strips any stale single-object pipelineStatuses value that a GET → PUT round-trip might have persisted into stored entity JSON.

Read pipelineStatuses[0] for the latest run, and regenerate SDK models.

Databricks Pipeline connection requires `authType`

Breaking. Affects Databricks Pipeline services created via API or infrastructure-as-code.

- "required": ["hostPort", "token"]
+ "required": ["hostPort", "authType"]

1.13
2.0

{ "hostPort": "…", "token": "dapi…" }

{ "hostPort": "…", "authType": { "token": "dapi…" } }

authType accepts three variants: Personal Access Token, DatabricksOAuth and Azure AD.

Stored service configurations are migrated automatically. External ingestion YAMLs are not. Any YAML you keep in source control, in a CI job, or on a hybrid runner must be updated by hand or the run fails validation.

Update Terraform, Ansible and scripted service creation, plus any ingestion YAML that sets token at the top level.

Log stream payload changed

Breaking. GET /v1/services/ingestionPipelines/logs/{fqn}/stream/{runId} now emits one JSON event per frame instead of one raw log line.

For example, a completion event looks like this:

{
  "eventType": "complete",
  "runId": "scheduled__2026-08-01T00:00:00+00:00",
  "after": "opaque-cursor",
  "replay": false,
  "reason": "runFinished"
}

after is an opaque resume cursor. Pass it back as the after query parameter to resume without re-reading delivered content.
reason is set only on a complete event. A stream that ends without one was cut short. Reconnect from the last cursor.
replay: true marks chunks replayed from the server buffer because the stream was already running when the client connected.
The path now accepts an id or fully qualified name, and runId is a free-form string, so Airflow’s scheduled__<ts> run ids no longer 404.

Clients parsing data: as log text must parse it as JSON and read event.logs.

Ingestion progress payload changed

Breaking. Affects consumers of GET /v1/services/ingestionPipelines/progress/{fqn}/stream/{runId}.

Progress changes from a flat per-entity-type map to a hierarchical tree. Each node counts its direct children: the root counts databases, a database counts schemas, a schema counts tables.

1.13
2.0

Progress by entity type. Keys are entity types, values contain total, processed and estimatedRemainingSeconds.

{
  "label": "",
  "entityType": "Database",
  "processed": 3,
  "expected": 12,
  "active": true,
  "overflow": 0,
  "children": [ /* active or relevant child nodes only */ ]
}

expected is null when the producer was iterated lazily.

A new service-level stream is added at GET /v1/services/ingestionPipelines/progress/service/{serviceType}/{serviceFqn}/stream, so the Services page can show live progress across all agents for a service. Coverage added in 2.0:

Database connectors generally, plus Databricks and Unity Catalog explicitly.
Airflow REST and dbt Cloud pipeline totals.
Looker and Tableau via manual progress mode.
The profiler and auto-classification workflows.

Ingestion images move to Python 3.12

Breaking. Affects anyone who builds a custom ingestion image or installs the ingestion package into their own Python environment.

	Released 1.13.x	2.0
Ingestion image	`apache/airflow:3.2.x-python3.10`	`apache/airflow:3.3.0-python3.12`
Operator image	`python:3.10-bookworm`	`python:3.12-slim-trixie`

cp310 wheels will not load. Any custom connector, driver or dependency pinned to a CPython 3.10 wheel must be rebuilt or repinned for CPython 3.12.

Rebuild every custom ingestion image, and re-resolve any private requirements file, before pointing agents at 2.0.

Great Expectations 1.3 or later is required

Breaking. Affects the great-expectations plugin.

- "great-expectations": "great-expectations~=0.18.0",
- "great-expectations-1xx": "great-expectations~=1.0",
+ "great-expectations": "great-expectations~=1.3",

Great Expectations 0.x is no longer supported, and the separate great-expectations-1xx extra is removed. Version 1.3 is the floor because Great Expectations only gained the validation-action registry there. On 1.0–1.2 Checkpoint.actions is a closed union that rejects the Collate validation action outright. If you adopted the interim 1.x module during 1.13, rename it in your checkpoint: metadata.great_expectations.action1xx and OpenMetadataValidationAction1xx have been removed:

# Before
from metadata.great_expectations.action1xx import OpenMetadataValidationAction1xx

# After
from metadata.great_expectations.action import OpenMetadataValidationAction

Checkpoints that already reference metadata.great_expectations.action and OpenMetadataValidationAction keep those names, but the surrounding checkpoint definition still has to move to the 1.x API: pass actions as objects to Checkpoint(actions=[...]) rather than as an action_list of dictionaries, and call checkpoint.run() instead of great_expectations checkpoint run.

Test case results now report row counts only: unexpected_count, missing_count, element_count and observed_value. The percentage values are no longer sent: unexpected_percent, unexpected_percent_total, missing_percent and success_rate. Collate charts every result value of a test case on a single axis, and percentages plotted next to row counts were unreadable. Percentages remain derivable as unexpected_count / element_count.

Replace pip install "openmetadata-ingestion[great-expectations-1xx]" with pip install "openmetadata-ingestion[great-expectations]".

Other dependency floors raised

Behavioural. Affects pinned and air-gapped installations.

Package	1.13	2.0	Why
`requests`	`>=2.23`	`>=2.32.4`	security
`sqlalchemy_exasol`	`>=6,<7`	`>=7.1.1,<8`	Exasol connector
`sqlalchemy-pytds`	`~=0.3`	`~=1.0`	0.3.x raises `AttributeError` on every server-side cursor fetch with python-tds 1.x
`gitpython`	`~=3.1.34`	`>=3.1.50`	security
`paramiko`	N/A	`>=3.5,<6`	new `sftp` plugin

New plugin extras: prefect and sftp.

New connectors

Additive.

Type	Connector	Service type value
Database	SAP BW/4HANA	`SapBw4Hana`
Dashboard	Omni	`Omni`
Pipeline	Prefect (Cloud and Server auth)	`Prefect`
Pipeline	SAP BW/4HANA	`SapBw4HanaPipeline`

Connector configuration changes

Default filter patterns now exclude system objects

Behavioural. Applies to existing services too.

Connector	Field	New default excludes
Redshift	`tableFilterPattern`	`^(?:.\.)?mv_tbl__.__\d+$`
Kafka	`topicFilterPattern`	`^__.`, `^_schemas$`, `^_confluent.`
Redpanda	`topicFilterPattern`	`^__.`, `^_schemas$`, `^_confluent.`

If you deliberately catalogued Redshift materialised-view backing tables or Kafka internal topics, set an explicit filter pattern. The next ingestion run will otherwise skip them and, with stale deletion enabled, soft-delete them.

SSIS `databaseConnection` is now optional

Relaxed requirement, supporting file-only SSIS mode. Existing configurations are unaffected.

Snowflake

Change	Detail
`includeSemanticViews`	New, default `false`. `tableType` gains `SemanticView`
Access-history lineage	New opt-in path
`policyAgentConfig`	Declared with defaults and backfilled onto existing services. See Data Governance

Other connector changes

Connector	Change
Unity Catalog	Incremental metadata extraction, `policyAgentConfig` added
Databricks	Partner user-agent for telemetry attribution, `policyAgentConfig` defaults
Postgres	`policyAgentConfig` removed from the schema and stripped from stored rows
MySQL	Custom `queryHistoryTable` for usage and lineage
ADLS	`containerName` field on the storage connection
Exasol	Comment access and query usage support
PowerBI	Datamart support, per-workspace cache scoping
dbt	`includeMetrics` toggle for semantic-layer metrics
Pipeline metadata	`ownershipUpdateMode` (`replace` or `append`) controls how source owners merge with existing pipeline owners
Storage	Compressed archive support, auto-classification for containers
Kafka Connect	Lineage for `EventRouter` and `RegexRouter` routed topics

Stale-entity cleanup API

New pattern for connectors.

DELETE /v1/tables/deleteStale

{
  "scopeFqn": "service.database.schema",
  "scopeEntityType": "databaseSchema",
  "seenFqns": ["…", "…"],
  "dryRun": false,
  "hardDelete": false,
  "recursive": true
}

Entities inside the scope that are not in seenFqns are considered stale and soft-deleted by default. Available on 18 entity types.

A seenFqns list truncated by a partial connector run will mark everything else stale. Always run with dryRun first when wiring this into custom connectors.

Ingestion pipeline management

Change	Class	Detail
`agentType` list filter	Additive	`GET /v1/services/ingestionPipelines?agentType=metadata\|application`
Queued-status polling	Behavioural	The server no longer polls orchestrators for queued status, reducing Airflow API load
`queuedStatusTimeoutSeconds`	Additive	Default `3600`. Controls how long a `queued` status stays visible before being treated as stale
Force delete	Additive	Admins can force-delete ingestion pipelines
`policyAgent` pipeline type	Additive	New pipeline type

Connector framework internals

Behavioural. Affects custom connector authors only.

mlmodel, metadata, messaging, storage, search and drive connectors migrated to BaseConnection.
get_connection_dict dropped from BaseConnection.
data-diff gated via a protocol.
Duplicated test-connection helpers consolidated.
Athena migrated to the declarative test-connection framework.
A shared GetPipelines step added to the pipeline test-connection vertical.
A ClassifiableEntityAdapter replaces scattered isinstance checks, and a TagRegistry domain layer is introduced.
ConnectionsRouterClassBase enables pluggable connection routing in the UI.

Rebuild and re-test custom connectors against the 2.0 ingestion package before upgrading production agents.

Collate 2.0

Breaking Changes by Component

Breaking Changes - Ingestion & Connectors | Official Documentation

Ingestion & Connectors

`pipelineStatuses` is now an array

Databricks Pipeline connection requires `authType`

Log stream payload changed

Ingestion progress payload changed

Ingestion images move to Python 3.12

Great Expectations 1.3 or later is required

Other dependency floors raised

New connectors

Connector configuration changes

Default filter patterns now exclude system objects

SSIS `databaseConnection` is now optional

Snowflake

Other connector changes

Stale-entity cleanup API

Ingestion pipeline management

Connector framework internals

​Ingestion & Connectors

​pipelineStatuses is now an array

​Databricks Pipeline connection requires authType

​Log stream payload changed

​Ingestion progress payload changed

​Ingestion images move to Python 3.12

​Great Expectations 1.3 or later is required

​Other dependency floors raised

​New connectors

​Connector configuration changes

​Default filter patterns now exclude system objects

​SSIS databaseConnection is now optional

​Snowflake

​Other connector changes

​Stale-entity cleanup API

​Ingestion pipeline management

​Connector framework internals

Ingestion & Connectors

`pipelineStatuses` is now an array

Databricks Pipeline connection requires `authType`

Log stream payload changed

Ingestion progress payload changed

Ingestion images move to Python 3.12

Great Expectations 1.3 or later is required

Other dependency floors raised

New connectors

Connector configuration changes

Default filter patterns now exclude system objects

SSIS `databaseConnection` is now optional

Snowflake

Other connector changes

Stale-entity cleanup API

Ingestion pipeline management

Connector framework internals