Google Pub/Sub Connector | Collate Messaging Integration

• GCP Credentials: Google Cloud service account credentials used to authenticate with Pub/Sub. Provide a service account key JSON containing type, projectId, privateKeyId, privateKey, clientEmail, and clientId. For more information, see Creating and managing service account keys.
• Project ID (optional if using inline GCP credentials): GCP Project ID where Pub/Sub topics are located. The project ID is only derived automatically when using inline service account credentials. If you’re using a credentials file path, Application Default Credentials, an external account, or the Pub/Sub emulator, you must set this field explicitly, or ingestion fails with Project ID is required.
• Host and Port (optional): Pub/Sub API endpoint. Defaults to pubsub.googleapis.com. Set to the emulator address (for example, localhost:8085) when using a local emulator.
• Use Emulator (optional): Set to true to connect to a local Pub/Sub emulator instead of the production service.
• Schema Registry Enabled (optional, default: true): Fetch topic schemas from the Pub/Sub Schema Registry. Supports Avro and Protocol Buffer schema types. Set to false if your project does not use the Schema Registry.
• Include Subscriptions (optional, default: true): Fetch subscription details for each topic. Only subscription names are persisted to topic metadata; acknowledgment deadlines, retention durations, push endpoints, and dead letter policies are read but not stored. When a subscription has a BigQuery export configuration, Collate uses it to extract lineage from the Pub/Sub topic to the target BigQuery table, though the export configuration itself is not persisted.
• Include Dead Letter Topics (optional, default: false): Include dead letter topics in metadata extraction. By default, dead letter topics are excluded to keep the topic list focused on primary business topics.

​

Metadata Ingestion Options

• Name: This field refers to the name of ingestion pipeline, you can customize the name or use the generated name.
• Topic Filter Pattern (Optional): Use it to control whether to include topics as part of metadata ingestion.
  • Include: Explicitly include topics by adding a list of comma-separated regular expressions to the ‘Include’ field. OpenMetadata will include all topics with names matching one or more of the supplied regular expressions. All other topics will be excluded.
  • Exclude: Explicitly exclude topics by adding a list of comma-separated regular expressions to the ‘Exclude’ field. OpenMetadata will exclude all topics with names matching one or more of the supplied regular expressions. All other topics will be included.
• Ingest Sample Data (toggle): Set the ‘Ingest Sample Data’ toggle to ingest sample data from the topics.
• Enable Debug Log (toggle): Set the ‘Enable Debug Log’ toggle to set the default log level to debug.
• Mark Deleted Topics (toggle): Set the ‘Mark Deleted Topics’ toggle to flag topics as soft-deleted if they are not present anymore in the source system.