MetadataWorkflow

OpenMetadata Ingestion Framework definition.

Properties

• id: Unique identifier that identifies this pipeline. Refer to ../type/basic.json#/definitions/uuid.
• name: Name that identifies this pipeline instance uniquely. Refer to ../type/basic.json#/definitions/entityName.
• openMetadataWorkflowConfig (object): OpenMetadata Ingestion Workflow Config. Cannot contain additional properties.
  • source: Refer to #/definitions/source.
  • processor: Refer to #/definitions/processor.
  • sink: Refer to #/definitions/sink.
  • stage: Refer to #/definitions/stage.
  • bulkSink: Refer to #/definitions/bulkSink.
  • workflowConfig: Refer to #/definitions/workflowConfig.
  • ingestionPipelineFQN (string): Fully qualified name of ingestion pipeline, used to identify the current ingestion pipeline.
  • pipelineRunId: Unique identifier of pipeline run, used to identify the current pipeline run. Refer to ../type/basic.json#/definitions/uuid.

Definitions

• sourceConfig (object): Additional connection configuration. Cannot contain additional properties.
  • config
    • One of
      • : Refer to databaseServiceMetadataPipeline.json.
      • : Refer to databaseServiceQueryUsagePipeline.json.
      • : Refer to databaseServiceQueryLineagePipeline.json.
      • : Refer to dashboardServiceMetadataPipeline.json.
      • : Refer to messagingServiceMetadataPipeline.json.
      • : Refer to databaseServiceProfilerPipeline.json.
      • : Refer to databaseServiceAutoClassificationPipeline.json.
      • : Refer to pipelineServiceMetadataPipeline.json.
      • : Refer to mlmodelServiceMetadataPipeline.json.
      • : Refer to storageServiceMetadataPipeline.json.
      • : Refer to searchServiceMetadataPipeline.json.
      • : Refer to testSuitePipeline.json.
      • : Refer to metadataToElasticSearchPipeline.json.
      • : Refer to dataInsightPipeline.json.
      • : Refer to dbtPipeline.json.
      • : Refer to applicationPipeline.json.
      • : Refer to apiServiceMetadataPipeline.json.
• source (object): Configuration for Source component in OpenMetadata Ingestion Framework. Cannot contain additional properties.
  • type (string, required): Type of the source connector ex: mysql, snowflake, tableau etc..
  • serviceName (string): Type of the source connector ex: mysql, snowflake, tableau etc..
  • serviceConnection: Connection configuration for the source. ex: mysql , tableau connection. Refer to ../entity/services/connections/serviceConnection.json#/definitions/serviceConnection.
  • sourceConfig: Refer to #/definitions/sourceConfig.
• processor (object): Configuration for Processor Component in the OpenMetadata Ingestion Framework. Cannot contain additional properties.
  • type (string, required): Type of processor component ex: pii-processor.
  • config: Refer to ../type/basic.json#/definitions/componentConfig.
• stage (object): Configuration for Stage Component in the OpenMetadata Ingestion Framework. Cannot contain additional properties.
  • type (string, required): Type of stage component ex: table-usage.
  • config: Refer to ../type/basic.json#/definitions/componentConfig.
• sink (object): Configuration for Sink Component in the OpenMetadata Ingestion Framework. Cannot contain additional properties.
  • type (string, required): Type of sink component ex: metadata.
  • config: Refer to ../type/basic.json#/definitions/componentConfig.
• bulkSink (object): Configuration for BulkSink Component in the OpenMetadata Ingestion Framework. Cannot contain additional properties.
  • type (string, required): Type of BulkSink component ex: metadata-usage.
  • config: Refer to ../type/basic.json#/definitions/componentConfig.
• logLevels (string): Supported logging levels. Must be one of: ["DEBUG", "INFO", "WARN", "ERROR"]. Default: "INFO".
• workflowConfig (object): Configuration for the entire Ingestion Workflow. Cannot contain additional properties.
  • loggerLevel: Refer to #/definitions/logLevels. Default: "INFO".
  • successThreshold (integer): The percentage of successfully processed records that must be achieved for the pipeline to be considered successful. Otherwise, the pipeline will be marked as failed. Minimum: 0. Maximum: 100. Default: 90.
  • openMetadataServerConfig: Refer to ../entity/services/connections/metadata/openMetadataConnection.json.
  • config: Refer to ../type/basic.json#/definitions/componentConfig.

Documentation file automatically generated at 2025-01-15 09:05:25.266839+00:00.