connectors

No menu items for this category
Home/Connectors/Pipeline/Datafactory
Azure Data Factory
Azure Data Factory
PROD
Available In
Feature List
Pipelines
Pipeline Status
Lineage
Owners
Tags

In this section, we provide guides and references to use the Azure Data Factory connector.

Configure and schedule Azure Data Factory metadata and profiler workflows from the OpenMetadata UI:

Ingestion Deployment

To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with custom Airflow plugins to handle the workflow deployment. If you want to install it manually in an already existing Airflow host, you can follow this guide.

If you don't want to use the OpenMetadata Ingestion container to configure the workflows via the UI, then you can check the following docs to run the Ingestion Framework in any orchestrator externally.

Run Connectors from the OpenMetadata UI

Learn how to manage your deployment to run connectors from the UI

Run the Connector Externally

Get the YAML to run the ingestion externally

External Schedulers

Get more information about running the Ingestion Framework Externally

Requirements

Data Factory Versions

The Ingestion framework uses Azure Data Factory APIs to connect to the Data Factory and fetch metadata.

You can find further information on the Azure Data Factory connector in the docs.

Permissions

Ensure that the service principal or managed identity you’re using has the necessary permissions in the Data Factory resource (Reader, Contributor or Data Factory Contributor role at minimum).

Metadata Ingestion

1. Visit the Services Page

The first step is to ingest the metadata from your sources. To do that, you first need to create a Service connection first.

This Service will be the bridge between OpenMetadata and your source system.

Once a Service is created, it can be used to configure your ingestion workflows.

Visit Services Page

Select your Service Type and Add a New Service

2. Create a New Service

Click on Add New Service to start the Service creation.

Create a new Service

Add a new Service from the Services page

3. Select the Service Type

Select DataFactory as the Service type and click Next.

Select Service

Select your Service from the list

4. Name and Describe your Service

Provide a name and description for your Service.

Service Name

OpenMetadata uniquely identifies Services by their Service Name. Provide a name that distinguishes your deployment from other Services, including the other DataFactory Services that you might be ingesting metadata from.

Note that when the name is set, it cannot be changed.

Add New Service

Provide a Name and description for your Service

5. Configure the Service Connection

In this step, we will configure the connection settings required for DataFactory.

Please follow the instructions below to properly configure the Service to read from your sources. You will also find helper documentation on the right-hand side panel in the UI.

Configure Service connection

Configure the Service connection by filling the form

Connection Details

  • Subscription ID: Your Azure subscription’s unique identifier. In the Azure portal, navigate to Subscriptions > Your Subscription > Overview. You’ll see the subscription ID listed there.

  • Resource Group name : This is the name of the resource group that contains your Data Factory instance. In the Azure portal, navigate to Resource Groups. Find your resource group, and note the name.

  • Azure Data Factory name : The name of your Data Factory instance. In the Azure portal, navigate to Data Factories and find your Data Factory. The Data Factory name will be listed there.

  • Azure Data Factory pipeline runs day filter : The days range when filtering pipeline runs. It specifies how many days back from the current date to look for pipeline runs, and filter runs within the given period of days. Default is 7 days. Optional

Azure Data Factory Configuration

  • Client ID : To get the Client ID (also known as application ID), follow these steps:
  1. Log into Microsoft Azure.
  2. Search for App registrations and select the App registrations link.
  3. Select the Azure AD app you're using for this connection.
  4. From the Overview section, copy the Application (client) ID.
  • Client Secret : To get the client secret, follow these steps:
  1. Log into Microsoft Azure.
  2. Search for App registrations and select the App registrations link.
  3. Select the Azure AD app you're using for this connection.
  4. Under Manage, select Certificates & secrets.
  5. Under Client secrets, select New client secret.
  6. In the Add a client secret pop-up window, provide a description for your application secret. Choose when the application should expire, and select Add.
  7. From the Client secrets section, copy the string in the Value column of the newly created application secret.
  • Tenant ID : To get the tenant ID, follow these steps:
  1. Log into Microsoft Azure.
  2. Search for App registrations and select the App registrations link.
  3. Select the Azure AD app you're using for Power BI.
  4. From the Overview section, copy the Directory (tenant) ID.
  • Account Name : Here are the step-by-step instructions for finding the account name for an Azure Data Lake Storage account:
  1. Sign in to the Azure portal and navigate to the Storage accounts page.
  2. Find the Data Lake Storage account you want to access and click on its name.
  3. In the account overview page, locate the Account name field. This is the unique identifier for the Data Lake Storage account.
  4. You can use this account name to access and manage the resources associated with the account, such as creating and managing containers and directories.

6. Test the Connection

Once the credentials have been added, click on Test Connection and Save the changes.

Test Connection

Test the connection and save the Service

7. Configure Metadata Ingestion

In this step we will configure the metadata ingestion pipeline, Please follow the instructions below

Configure Metadata Ingestion

Configure Metadata Ingestion Page

Metadata Ingestion Options

  • Name: This field refers to the name of ingestion pipeline, you can customize the name or use the generated name.
  • Pipeline Filter Pattern (Optional): Use to pipeline filter patterns to control whether or not to include pipeline as part of metadata ingestion.
    • Include: Explicitly include pipeline by adding a list of comma-separated regular expressions to the Include field. OpenMetadata will include all pipeline with names matching one or more of the supplied regular expressions. All other schemas will be excluded.
    • Exclude: Explicitly exclude pipeline by adding a list of comma-separated regular expressions to the Exclude field. OpenMetadata will exclude all pipeline with names matching one or more of the supplied regular expressions. All other schemas will be included.
  • Include lineage (toggle): Set the Include lineage toggle to control whether to include lineage between pipelines and data sources as part of metadata ingestion.
  • Enable Debug Log (toggle): Set the Enable Debug Log toggle to set the default log level to debug.
  • Mark Deleted Pipelines (toggle): Set the Mark Deleted Pipelines toggle to flag pipelines as soft-deleted if they are not present anymore in the source system.

8. Schedule the Ingestion and Deploy

Scheduling can be set up at an hourly, daily, weekly, or manual cadence. The timezone is in UTC. Select a Start Date to schedule for ingestion. It is optional to add an End Date.

Review your configuration settings. If they match what you intended, click Deploy to create the service and schedule metadata ingestion.

If something doesn't look right, click the Back button to return to the appropriate step and change the settings as needed.

After configuring the workflow, you can click on Deploy to create the pipeline.

Schedule the Workflow

Schedule the Ingestion Pipeline and Deploy

9. View the Ingestion Pipeline

Once the workflow has been successfully deployed, you can view the Ingestion Pipeline running from the Service Page.

View Ingestion Pipeline

View the Ingestion Pipeline from the Service Page

Displaying Lineage Information

Steps to retrieve and display the lineage information for a Data Factory service.

  1. Ingest Source and Sink Database Metadata: Identify both the source and sink database used by the Azure Data Factory service for example Redshift. Ingest metadata for these database.
  2. Ingest Data Factory Service Metadata: Finally, Ingest your DData Factory service.

By successfully completing these steps, the lineage information for the service will be displayed.

Troubleshooting

Workflow Deployment Error

If there were any errors during the workflow deployment process, the Ingestion Pipeline Entity will still be created, but no workflow will be present in the Ingestion container.

  • You can then Edit the Ingestion Pipeline and Deploy it again.
  • From the Connection tab, you can also Edit the Service if needed.

Missing Lineage

If lineage information is not displayed for a Data Factory service, follow these steps to diagnose the issue.

  1. Permissions: Ensure that the service principal or managed identity you’re using has the necessary permissions in the Data Factory resource. (Reader, Contributor or Data Factory Contributor role at minimum).
  2. Metadata Ingestion: Ensure that metadata for both the source and sink database is ingested and passed to the lineage system. This typically involves configuring the relevant connectors to capture and transmit this information.
  3. Run Successful: Ensure that the Pipeline Run is successful.