Skip to main content
Azure Data Factory

Azure Data Factory

PROD
In this section, we provide guides and references to use the Azure Data Factory connector. Configure and schedule Azure Data Factory metadata and profiler workflows from the OpenMetadata UI:

​
Requirements

​
Data Factory Versions

The Ingestion framework uses Azure Data Factory APIs to connect to the Data Factory and fetch metadata. You can find further information on the Azure Data Factory connector in the docs.

​
Permissions

Ensure that the service principal or managed identity you’re using has the necessary permissions in the Data Factory resource (Reader, Contributor or Data Factory Contributor role at minimum).

​
Metadata Ingestion

1

Visit the Services Page

Click `Settings` in the side navigation bar and then `Services`. The first step is to ingest the metadata from your sources. To do that, you first need to create a Service connection first. This Service will be the bridge between OpenMetadata and your source system. Once a Service is created, it can be used to configure your ingestion workflows.Visit Services Page
2

Create a New Service

Click on _Add New Service_ to start the Service creation.Create a new Service
3

Select the Service Type

Select Data Factory as the Service type and click _Next_.Select Service
4

Name and Describe your Service

Provide a name and description for your Service.

Service Name

OpenMetadata uniquely identifies Services by their **Service Name**. Provide a name that distinguishes your deployment from other Services, including the other Data Factory Services that you might be ingesting metadata from. Note that when the name is set, it cannot be changed.Add New Service
5

Configure the Service Connection

In this step, we will configure the connection settings required for Data Factory. Please follow the instructions below to properly configure the Service to read from your sources. You will also find helper documentation on the right-hand side panel in the UI.Configure Service connection

​
Connection Details

1

Connection Details

When using a Hybrid Ingestion Runner, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
password: secret:/my/database/password
This applies only to fields marked as secrets in the connection form (these typically mask input and show a visibility toggle icon). For a complete guide on managing secrets in hybrid setups, see the Hybrid Ingestion Runner Secret Management Guide.
  • Subscription ID: Your Azure subscription’s unique identifier. In the Azure portal, navigate to Subscriptions > Your Subscription > Overview. You’ll see the subscription ID listed there.
  • Resource Group name : This is the name of the resource group that contains your Data Factory instance. In the Azure portal, navigate to Resource Groups. Find your resource group, and note the name.
  • Azure Data Factory name : The name of your Data Factory instance. In the Azure portal, navigate to Data Factories and find your Data Factory. The Data Factory name will be listed there.
  • Azure Data Factory pipeline runs day filter : The days range when filtering pipeline runs. It specifies how many days back from the current date to look for pipeline runs, and filter runs within the given period of days. Default is 7 days. Optional

​
Azure Data Factory Configuration

  • Client ID : To get the Client ID (also known as application ID), follow these steps:
  1. Log into Microsoft Azure.
  2. Search for App registrations and select the App registrations link.
  3. Select the Azure AD app you’re using for this connection.
  4. From the Overview section, copy the Application (client) ID.
  • Client Secret : To get the client secret, follow these steps:
  1. Log into Microsoft Azure.
  2. Search for App registrations and select the App registrations link.
  3. Select the Azure AD app you’re using for this connection.
  4. Under Manage, select Certificates & secrets.
  5. Under Client secrets, select New client secret.
  6. In the Add a client secret pop-up window, provide a description for your application secret. Choose when the application should expire, and select Add.
  7. From the Client secrets section, copy the string in the Value column of the newly created application secret.
  • Tenant ID : To get the tenant ID, follow these steps:
  1. Log into Microsoft Azure.
  2. Search for App registrations and select the App registrations link.
  3. Select the Azure AD app you’re using for Power BI.
  4. From the Overview section, copy the Directory (tenant) ID.
  • Account Name : Here are the step-by-step instructions for finding the account name for an Azure Data Lake Storage account:
  1. Sign in to the Azure portal and navigate to the Storage accounts page.
  2. Find the Data Lake Storage account you want to access and click on its name.
  3. In the account overview page, locate the Account name field. This is the unique identifier for the Data Lake Storage account.
  4. You can use this account name to access and manage the resources associated with the account, such as creating and managing containers and directories.
2

Test the Connection

Once the credentials have been added, click on Test Connection and Save the changes.Test Connection
3

Configure Metadata Ingestion

In this step we will configure the metadata ingestion pipeline, Please follow the instructions belowConfigure Metadata Ingestion

​
Metadata Ingestion Options

  • Name: This field refers to the name of ingestion pipeline, you can customize the name or use the generated name.
  • Pipeline Filter Pattern (Optional): Use to pipeline filter patterns to control whether or not to include pipeline as part of metadata ingestion.
    • Include: Explicitly include pipeline by adding a list of comma-separated regular expressions to the Include field. OpenMetadata will include all pipeline with names matching one or more of the supplied regular expressions. All other schemas will be excluded.
    • Exclude: Explicitly exclude pipeline by adding a list of comma-separated regular expressions to the Exclude field. OpenMetadata will exclude all pipeline with names matching one or more of the supplied regular expressions. All other schemas will be included.
  • Include lineage (toggle): Set the Include lineage toggle to control whether to include lineage between pipelines and data sources as part of metadata ingestion.
  • Enable Debug Log (toggle): Set the Enable Debug Log toggle to set the default log level to debug.
  • Mark Deleted Pipelines (toggle): Set the Mark Deleted Pipelines toggle to flag pipelines as soft-deleted if they are not present anymore in the source system.
4

Schedule the Ingestion and Deploy

Scheduling can be set up at an hourly, daily, weekly, or manual cadence. The timezone is in UTC. Select a Start Date to schedule for ingestion. It is optional to add an End Date.Review your configuration settings. If they match what you intended, click Deploy to create the service and schedule metadata ingestion.If something doesn’t look right, click the Back button to return to the appropriate step and change the settings as needed.After configuring the workflow, you can click on Deploy to create the pipeline.Schedule the Workflow
5

View the Ingestion Pipeline

Once the workflow has been successfully deployed, you can view the Ingestion Pipeline running from the Service Page.View Ingestion Pipeline
If AutoPilot is enabled, workflows like usage tracking, data lineage, and similar tasks will be handled automatically. Users don’t need to set up or manage them - AutoPilot takes care of everything in the system.

​
Displaying Lineage Information

Steps to retrieve and display the lineage information for a Data Factory service.
  1. Ingest Source and Sink Database Metadata: Identify both the source and sink database used by the Azure Data Factory service for example Redshift. Ingest metadata for these database.
  2. Ingest Data Factory Service Metadata: Finally, Ingest your DData Factory service. By successfully completing these steps, the lineage information for the service will be displayed.

​
Missing Lineage

If lineage information is not displayed for a Data Factory service, follow these steps to diagnose the issue.
  1. Permissions: Ensure that the service principal or managed identity you’re using has the necessary permissions in the Data Factory resource. (Reader, Contributor or Data Factory Contributor role at minimum).
  2. Metadata Ingestion: Ensure that metadata for both the source and sink database is ingested and passed to the lineage system. This typically involves configuring the relevant connectors to capture and transmit this information.
  3. Run Successful: Ensure that the Pipeline Run is successful.