> ## Documentation Index
> Fetch the complete documentation index at: https://docs.getcollate.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Breaking Changes - Ingestion & Connectors | Official Documentation

> pipelineStatuses becomes an array, the log-stream and progress payloads change shape, Databricks Pipeline auth changes, and ingestion images move to Python 3.12 in Collate 2.0.

# Ingestion & Connectors

Five hard breaks land here: the `pipelineStatuses` shape, the Databricks Pipeline connection, the
log-stream payload, the progress payload, and the move to Python 3.12 — plus a Great Expectations
major-version floor and a set of default filter-pattern changes that apply to existing services too.

## pipelineStatuses is now an array

**Breaking.** Affects every client reading ingestion pipelines: dashboards, health checks and SDK
users.

```diff theme={null}
  "pipelineStatuses": {
-   "description": "Last of executions and status for the Pipeline.",
-   "$ref": "#/definitions/pipelineStatus"
+   "description": "List of the most recent executions and status for the Pipeline.",
+   "type": "array",
+   "items": { "$ref": "#/definitions/pipelineStatus" }
  }
```

The list endpoint now returns the **last five** statuses per pipeline (also removing an N+1 query).
Strongly typed clients throw on the type change; loosely typed scripts reading
`pipelineStatuses.pipelineState` silently get `undefined` or `None`. Read `pipelineStatuses[0]` for
the latest run, and regenerate SDK models. The migration also strips any stale single-object value a
`GET → PUT` round-trip might have persisted.

## The log-stream and progress payloads both change shape

**Breaking.** Affects consumers of `GET .../logs/{fqn}/stream/{runId}` and
`GET .../progress/{fqn}/stream/{runId}`.

The log stream now emits **one JSON event per frame** instead of one raw log line, per the
`LogStreamEvent` schema:

```json theme={null}
{
  "eventType": "logs | complete | error",
  "runId": "scheduled__2026-08-01T00:00:00+00:00",
  "logs": "…appended log content, empty on complete/error…",
  "after": "opaque-cursor",
  "replay": false,
  "truncated": false,
  "reason": "runFinished | idleTimeout | maxDuration | maxBytes",
  "message": "…set on error events, and on complete events that ended early…"
}
```

Clients parsing `data:` as raw log text must parse it as JSON and read `logs` instead. `after` is an
opaque cursor pointing just past the delivered `logs` — pass it back as the `after` query parameter
to resume without re-reading what was already sent. `reason` is set **only** on a `complete` event; a
stream that ends without one was cut short and should reconnect from the last cursor. `truncated`
signals that the server couldn't replay the whole backlog, so the client must fetch earlier history
from the paginated log endpoint. The path also now accepts an id **or** fully qualified name, and
`runId` is free-form, so Airflow's `scheduled__<ts>` run ids no longer 404.

Progress changes from a flat per-entity-type map to a **hierarchical tree**, where each node counts
its direct children (root counts databases, a database counts schemas, a schema counts tables), with
`expected: null` when the producer was iterated lazily. A new service-level stream
(`GET .../progress/service/{serviceType}/{serviceFqn}/stream`) shows live progress across all agents
for a service. Coverage is added for database connectors generally, Databricks, and Unity Catalog
explicitly, Airflow REST and dbt Cloud pipeline totals, and Looker/Tableau via manual progress mode.

## Databricks Pipeline connection requires authType

**Breaking.** Affects Databricks Pipeline services created via API or infrastructure-as-code.

```diff theme={null}
- "required": ["hostPort", "token"]
+ "required": ["hostPort", "authType"]
```

`authType` accepts three variants — Personal Access Token, DatabricksOAuth, and Azure AD. Stored
service configurations migrate automatically; **external ingestion YAMLs do not** and fail
validation until updated by hand.

## Ingestion images move to Python 3.12, Great Expectations 1.3+ required

**Breaking.** Affects anyone who builds a custom ingestion image, installs the ingestion package
into their own Python environment, or uses the `great-expectations` plugin.

|                 | Released 1.13.x                   | 2.0                               |
| --------------- | --------------------------------- | --------------------------------- |
| Ingestion image | `apache/airflow:3.2.x-python3.10` | `apache/airflow:3.3.0-python3.12` |
| Operator image  | `python:3.10-bookworm`            | `python:3.12-slim-trixie`         |

**`cp310` wheels will not load.** Any custom connector, driver, or dependency pinned to a CPython 3.10
wheel must be rebuilt or repinned.

<Note>
  This move was also backported into the 1.13 line after 1.13.3 — verified directly against the
  ingestion `Dockerfile` on the `1.13.0` and `1.13.4` release branches. If you're already running
  1.13.4, you've made this jump; upgrading from **1.13.0–1.13.3** you haven't.
</Note>

Separately, Great Expectations 0.x is no longer supported and the
`great-expectations-1xx` extra is removed — 1.3 is the floor because Great Expectations only gained
the validation-action registry there. Checkpoints referencing
`metadata.great_expectations.action1xx` move to `metadata.great_expectations.action`, and test case
results now report row counts only (`unexpected_count`, `missing_count`, `element_count`,
`observed_value`) — the percentage fields are no longer sent, since Collate charts every result value
on a single axis and percentages next to row counts were unreadable.

Other dependency floors that move with this release: `requests` to `>=2.32.4` (security),
`sqlalchemy-pytds` to `~=1.0` (the 0.3.x line raised `AttributeError` on every server-side cursor
fetch with python-tds 1.x), and `gitpython` to `>=3.1.50` (security).

## Default filter patterns now exclude system objects, on existing services too

**Behavioural.** Applies retroactively, not just to newly created services.

| Connector | Field                | New default excludes                   |
| --------- | -------------------- | -------------------------------------- |
| Redshift  | `tableFilterPattern` | `^(?:.*\.)?mv_tbl__.*__\d+$`           |
| Kafka     | `topicFilterPattern` | `^__.*`, `^_schemas$`, `^_confluent.*` |
| Redpanda  | `topicFilterPattern` | `^__.*`, `^_schemas$`, `^_confluent.*` |

<Warning>
  If you deliberately catalogued Redshift materialised-view backing tables or Kafka internal topics,
  set an explicit filter pattern. The next ingestion run will otherwise skip them and, with stale
  deletion enabled, soft-delete them.
</Warning>

`policyAgentConfig` also lands on Snowflake, Databricks, and Unity Catalog with backfilled defaults
(see [Data Governance](/ai-2-0/release-2.0/breaking-changes/data-governance)) and is **removed** entirely
from Postgres.

**New connectors** ship in 2.0, all additive:

| Type      | Connector                       | Service type value   |
| --------- | ------------------------------- | -------------------- |
| Database  | SAP BW/4HANA                    | `SapBw4Hana`         |
| Dashboard | Omni                            | `Omni`               |
| Pipeline  | Prefect (Cloud and Server auth) | `Prefect`            |
| Pipeline  | SAP BW/4HANA                    | `SapBw4HanaPipeline` |

SSIS's `databaseConnection` also becomes optional, supporting file-only mode (existing configurations
are unaffected). Other per-connector additions:

| Connector         | Change                                                                                                         |
| ----------------- | -------------------------------------------------------------------------------------------------------------- |
| Snowflake         | `includeSemanticViews` (new, default `false`, `tableType` gains `SemanticView`), opt-in access-history lineage |
| Unity Catalog     | Incremental metadata extraction, `policyAgentConfig` added                                                     |
| Databricks        | Partner user-agent for telemetry attribution, `policyAgentConfig` defaults                                     |
| MySQL             | Custom `queryHistoryTable` for usage and lineage                                                               |
| ADLS              | `containerName` field on the storage connection                                                                |
| Exasol            | Comment access and query usage support                                                                         |
| PowerBI           | Datamart support, per-workspace cache scoping                                                                  |
| dbt               | `includeMetrics` toggle for semantic-layer metrics                                                             |
| Pipeline metadata | `ownershipUpdateMode` (`replace` or `append`) controls how source owners merge with existing pipeline owners   |
| Storage           | Compressed archive support, auto-classification for containers                                                 |

## Connector setup moves to a step-by-step onboarding wizard

**Collate-specific, additive.** This has no OpenMetadata OSS equivalent — it's part of the AI Mode
experience.

Every connector's setup flow is redesigned around a guided, multi-step wizard: add the service,
select the connector, name it, configure the connection and ingestion runner, test the connection,
set filter patterns for what to ingest, then create and deploy. This replaces the single embedded
connection-form widget used in 1.13. The new flow is documented per connector — see any connector's
setup guide under **Connectors** for the full walkthrough with screenshots.

## A generic stale-entity cleanup pattern replaces manual delete loops

**Additive**, changes how connectors should reconcile deletions. `DELETE /v1/tables/deleteStale`
(and 17 other entity types) accepts a `seenFqns` list scoped to a schema or other container; anything
inside the scope not in that list is soft-deleted by default.

<Warning>
  A `seenFqns` list truncated by a partial connector run marks everything else stale. Always run with
  `dryRun` first when wiring this into custom connectors.
</Warning>

## Connector framework internals change for custom connector authors

**Behavioural.** Affects custom connector authors only. mlmodel, metadata, messaging, storage,
search and drive connectors migrate to `BaseConnection`, and `get_connection_dict` is dropped from
it. Duplicated test-connection helpers are consolidated, Athena migrates to the declarative
test-connection framework, and a `ClassifiableEntityAdapter` replaces scattered `isinstance` checks.
Rebuild and re-test custom connectors against the 2.0 ingestion package before upgrading production
agents.

## Full detail

For the full narrative walkthrough of Ingestion & Connectors in Collate 2.0 — including screenshots and step-by-step context — see [Release 2.0: Ingestion & Connectors](/ai-2-0/release-2.0/ingestion-and-connectors).