Ingestion & Connectors
Five hard breaks land here: thepipelineStatuses shape, the Databricks Pipeline connection, the
log-stream payload, the progress payload, and the move to Python 3.12 — plus a Great Expectations
major-version floor and a set of default filter-pattern changes that apply to existing services too.
pipelineStatuses is now an array
Breaking. Affects every client reading ingestion pipelines: dashboards, health checks and SDK users.pipelineStatuses.pipelineState silently get undefined or None. Read pipelineStatuses[0] for
the latest run, and regenerate SDK models. The migration also strips any stale single-object value a
GET → PUT round-trip might have persisted.
The log-stream and progress payloads both change shape
Breaking. Affects consumers ofGET .../logs/{fqn}/stream/{runId} and
GET .../progress/{fqn}/stream/{runId}.
The log stream now emits one JSON event per frame instead of one raw log line, per the
LogStreamEvent schema:
data: as raw log text must parse it as JSON and read logs instead. after is an
opaque cursor pointing just past the delivered logs — pass it back as the after query parameter
to resume without re-reading what was already sent. reason is set only on a complete event; a
stream that ends without one was cut short and should reconnect from the last cursor. truncated
signals that the server couldn’t replay the whole backlog, so the client must fetch earlier history
from the paginated log endpoint. The path also now accepts an id or fully qualified name, and
runId is free-form, so Airflow’s scheduled__<ts> run ids no longer 404.
Progress changes from a flat per-entity-type map to a hierarchical tree, where each node counts
its direct children (root counts databases, a database counts schemas, a schema counts tables), with
expected: null when the producer was iterated lazily. A new service-level stream
(GET .../progress/service/{serviceType}/{serviceFqn}/stream) shows live progress across all agents
for a service. Coverage is added for database connectors generally, Databricks, and Unity Catalog
explicitly, Airflow REST and dbt Cloud pipeline totals, and Looker/Tableau via manual progress mode.
Databricks Pipeline connection requires authType
Breaking. Affects Databricks Pipeline services created via API or infrastructure-as-code.authType accepts three variants — Personal Access Token, DatabricksOAuth, and Azure AD. Stored
service configurations migrate automatically; external ingestion YAMLs do not and fail
validation until updated by hand.
Ingestion images move to Python 3.12, Great Expectations 1.3+ required
Breaking. Affects anyone who builds a custom ingestion image, installs the ingestion package into their own Python environment, or uses thegreat-expectations plugin.
cp310 wheels will not load. Any custom connector, driver, or dependency pinned to a CPython 3.10
wheel must be rebuilt or repinned.
This move was also backported into the 1.13 line after 1.13.3 — verified directly against the
ingestion
Dockerfile on the 1.13.0 and 1.13.4 release branches. If you’re already running
1.13.4, you’ve made this jump; upgrading from 1.13.0–1.13.3 you haven’t.great-expectations-1xx extra is removed — 1.3 is the floor because Great Expectations only gained
the validation-action registry there. Checkpoints referencing
metadata.great_expectations.action1xx move to metadata.great_expectations.action, and test case
results now report row counts only (unexpected_count, missing_count, element_count,
observed_value) — the percentage fields are no longer sent, since Collate charts every result value
on a single axis and percentages next to row counts were unreadable.
Other dependency floors that move with this release: requests to >=2.32.4 (security),
sqlalchemy-pytds to ~=1.0 (the 0.3.x line raised AttributeError on every server-side cursor
fetch with python-tds 1.x), and gitpython to >=3.1.50 (security).
Default filter patterns now exclude system objects, on existing services too
Behavioural. Applies retroactively, not just to newly created services.policyAgentConfig also lands on Snowflake, Databricks, and Unity Catalog with backfilled defaults
(see Data Governance) and is removed entirely
from Postgres.
New connectors ship in 2.0, all additive:
SSIS’s
databaseConnection also becomes optional, supporting file-only mode (existing configurations
are unaffected). Other per-connector additions:
Connector setup moves to a step-by-step onboarding wizard
Collate-specific, additive. This has no OpenMetadata OSS equivalent — it’s part of the AI Mode experience. Every connector’s setup flow is redesigned around a guided, multi-step wizard: add the service, select the connector, name it, configure the connection and ingestion runner, test the connection, set filter patterns for what to ingest, then create and deploy. This replaces the single embedded connection-form widget used in 1.13. The new flow is documented per connector — see any connector’s setup guide under Connectors for the full walkthrough with screenshots.A generic stale-entity cleanup pattern replaces manual delete loops
Additive, changes how connectors should reconcile deletions.DELETE /v1/tables/deleteStale
(and 17 other entity types) accepts a seenFqns list scoped to a schema or other container; anything
inside the scope not in that list is soft-deleted by default.
Connector framework internals change for custom connector authors
Behavioural. Affects custom connector authors only. mlmodel, metadata, messaging, storage, search and drive connectors migrate toBaseConnection, and get_connection_dict is dropped from
it. Duplicated test-connection helpers are consolidated, Athena migrates to the declarative
test-connection framework, and a ClassifiableEntityAdapter replaces scattered isinstance checks.
Rebuild and re-test custom connectors against the 2.0 ingestion package before upgrading production
agents.