> ## Documentation Index
> Fetch the complete documentation index at: https://docs.getcollate.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Datafactory | Official Documentation

> Get started with datafactory. Setup instructions, features, and configuration details inside.

export const MetadataIngestionUi = ({connector, selectServicePath, addNewServicePath, serviceConnectionPath}) => {
  return <>
   <p>
      To ingest metadata from your sources, you need to create a service connection.
      The service connects your source system with Collate. Once you create
      a service, you can use it to configure your ingestion workflows.<br />
      <br />
      To create a service connection and ingest your metadata, follow the steps below:
  </p>
  <Steps>
    <Step title="Select the Service">
    <ol>
          <li>
            On the left navigation bar, click <strong>Settings</strong>.
          </li>
          <li>
            On the next page, click <strong>Services</strong>, and then select the service.
            <img src="/public/images/connectors/visit-services-page.png" alt="Visit Services Page" />
          </li>
    </ol>
   </Step>


   <Step title="Create a New Service">
      
       To add a new service connection, click <strong>Add New Service</strong>.
      <img src="/public/images/connectors/create-new-service.png" alt="Create a new Service" />


   </Step>


     <Step title="Select the Connector">
       Select <strong>{connector}</strong> as the service type and click <strong>Next</strong>.


       {selectServicePath && <img src={selectServicePath} alt="Select Service" />}
   </Step>


   <Step title="Name and Describe your Service">
       Enter a unique, descriptive <strong>Service Name</strong> and <strong>Description</strong>.
       <ul>
         <li><strong>Service Name</strong>: Collate identifies services by their service name. Enter a name that distinguishes this deployment from other services, including other {connector} services you are ingesting metadata from.</li>
       </ul>


       <Note>
           The service name cannot be changed after it is set.
       </Note>


       {addNewServicePath && <img src={addNewServicePath} alt="Add New Service" />}
   </Step>


   <Step title="Configure the Service Connection">
       Set up the connection settings required for {connector}. <br /><br />
      
       Configure the following connection options to set up the service and start ingesting metadata from your sources. The right-hand panel displays help documentation for the selected connection type in the product UI

       {serviceConnectionPath && <img src={serviceConnectionPath} alt="Configure Service connection" />}
   </Step>
   </Steps>
   </>;
};

export const ConnectorDetailsHeader = ({name, icon, stage, availableFeatures, unavailableFeatures = [], availableFeaturesCollate = []}) => {
  const showSubHeading = availableFeatures?.length > 0 || unavailableFeatures?.length > 0 || availableFeaturesCollate?.length > 0;
  const totalAvailableFeatures = [...availableFeatures || [], ...availableFeaturesCollate || []];
  return <div className="container">
      <div className="Heading">
        <div className="flex items-center gap-3">
          {icon && <div className="IconContainer">
              <img src={icon} alt={name} noZoom className="ConnectorIcon" />
            </div>}
          <h1 className="ConnectorName">{name}</h1>
          <span className={`StageBadge ${stage === 'PROD' ? 'prod' : 'beta'}`}>
            {stage}
          </span>
        </div>
      </div>
      {showSubHeading && <div className="SubHeading">
          <div className="FeaturesHeading">Feature List</div>
          <div className="FeaturesList">
            {totalAvailableFeatures.map(feature => <div className="FeatureTag AvailableFeature" key={feature}>
                ✓ {feature}
              </div>)}
            {unavailableFeatures.map(feature => <div className="FeatureTag UnavailableFeature" key={feature}>
                ✕ {feature}
              </div>)}
          </div>
        </div>}
    </div>;
};

<ConnectorDetailsHeader icon="/public/images/connectors/datafactory.png" name="Azure Data Factory" stage="PROD" availableFeatures={["Pipelines", "Pipeline Status", "Lineage"]} unavailableFeatures={["Owners", "Tags"]} />

In this section, we provide guides and references to use the Azure Data Factory connector.
Configure and schedule Azure Data Factory metadata workflows from the Collate UI:

* [Requirements](#requirements)
  * [Data Factory Versions](#data-factory-versions)
* [Metadata Ingestion](#metadata-ingestion)
  * [Connection Details](#connection-details)
  * [Metadata Ingestion Options](#metadata-ingestion-options)
* [Troubleshooting](/connectors/pipeline/datafactory/troubleshooting)

## Requirements

### Data Factory Versions

The Ingestion framework uses [Azure Data Factory APIs](https://learn.microsoft.com/en-us/rest/api/datafactory/v2) to connect to the Data Factory and fetch metadata.
You can find further information on the Azure Data Factory connector in the [docs](/connectors/pipeline/datafactory).

## Permissions

Ensure that the service principal or managed identity you’re using has the necessary permissions in the Data Factory resource (Reader, Contributor or Data Factory Contributor role at minimum).

## Metadata Ingestion

<MetadataIngestionUi connector={"Data Factory"} selectServicePath={"/public/images/connectors/datafactory/select-service.png"} addNewServicePath={"/public/images/connectors/datafactory/add-new-service.png"} serviceConnectionPath={"/public/images/connectors/datafactory/service-connection.png"} />

## Connection Details

<Steps>
  <Step title="Connection Details">
    * **Subscription ID**: Your Azure subscription’s unique identifier. In the Azure portal, navigate to Subscriptions > Your Subscription > Overview. You’ll see the subscription ID listed there.
    * **Resource Group name** : This is the name of the resource group that contains your Data Factory instance. In the Azure portal, navigate to Resource Groups. Find your resource group, and note the name.
    * **Azure Data Factory name** : The name of your Data Factory instance. In the Azure portal, navigate to Data Factories and find your Data Factory. The Data Factory name will be listed there.
    * **Azure Data Factory pipeline runs day filter** : The days range when filtering pipeline runs. It specifies how many days back from the current date to look for pipeline runs, and filter runs within the given period of days. Default is `7` days. `Optional`

    ## Azure Data Factory Configuration

    * **Client ID** : To get the Client ID (also known as application ID), follow these steps:

    1. Log into [Microsoft Azure](https://ms.portal.azure.com/#allservices).
    2. Search for `App registrations` and select the `App registrations link`.
    3. Select the `Azure AD` app you're using for this connection.
    4. From the Overview section, copy the `Application (client) ID`.

    * **Client Secret** : To get the client secret, follow these steps:

    1. Log into [Microsoft Azure](https://ms.portal.azure.com/#allservices).
    2. Search for `App registrations` and select the `App registrations link`.
    3. Select the `Azure AD` app you're using for this connection.
    4. Under `Manage`, select `Certificates & secrets`.
    5. Under `Client secrets`, select `New client secret`.
    6. In the `Add a client secret` pop-up window, provide a description for your application secret. Choose when the application should expire, and select `Add`.
    7. From the `Client secrets` section, copy the string in the `Value` column of the newly created application secret.

    * **Tenant ID** : To get the tenant ID, follow these steps:

    1. Log into [Microsoft Azure](https://ms.portal.azure.com/#allservices).
    2. Search for `App registrations` and select the `App registrations link`.
    3. Select the `Azure AD` app you're using for Power BI.
    4. From the `Overview` section, copy the `Directory (tenant) ID`.

    * **Account Name** : Here are the step-by-step instructions for finding the account name for an Azure Data Lake Storage account:

    1. Sign in to the Azure portal and navigate to the `Storage accounts` page.
    2. Find the Data Lake Storage account you want to access and click on its name.
    3. In the account overview page, locate the `Account name` field. This is the unique identifier for the Data Lake Storage account.
    4. You can use this account name to access and manage the resources associated with the account, such as creating and managing containers and directories.
  </Step>

  <Step title="Test the Connection">
    Once the credentials have been added, click on *Test Connection* and *Save* the changes.

    <img src="https://mintcdn.com/collatedocs/L7psA65ao88vmcRI/public/images/connectors/test-connection.png?fit=max&auto=format&n=L7psA65ao88vmcRI&q=85&s=2133f0d65f18df1e57f69d2cc3bdeff4" alt="Test Connection" width="1494" height="310" data-path="public/images/connectors/test-connection.png" />
  </Step>

  <Step title="Configure Metadata Ingestion">
    In this step we will configure the metadata ingestion pipeline,
    Please follow the instructions below

    <img src="https://mintcdn.com/collatedocs/12mtkhFtvTP7FUxZ/public/images/connectors/configure-metadata-ingestion-pipeline.png?fit=max&auto=format&n=12mtkhFtvTP7FUxZ&q=85&s=b310e691eaec850bc9eda3ed0b8a340f" alt="Configure Metadata Ingestion" width="1508" height="1614" data-path="public/images/connectors/configure-metadata-ingestion-pipeline.png" />

    #### Metadata Ingestion Options

    * **Name**: This field refers to the name of ingestion pipeline, you can customize the name or use the generated name.
    * **Pipeline Filter Pattern (Optional)**: Use to pipeline filter patterns to control whether or not to include pipeline as part of metadata ingestion.
      * **Include**: Explicitly include pipeline by adding a list of comma-separated regular expressions to the Include field. OpenMetadata will include all pipeline with names matching one or more of the supplied regular expressions. All other schemas will be excluded.
      * **Exclude**: Explicitly exclude pipeline by adding a list of comma-separated regular expressions to the Exclude field. OpenMetadata will exclude all pipeline with names matching one or more of the supplied regular expressions. All other schemas will be included.
    * **Include lineage (toggle)**: Set the Include lineage toggle to control whether to include lineage between pipelines and data sources as part of metadata ingestion.
    * **Enable Debug Log (toggle)**: Set the Enable Debug Log toggle to set the default log level to debug.
    * **Mark Deleted Pipelines (toggle)**: Set the Mark Deleted Pipelines toggle to flag pipelines as soft-deleted if they are not present anymore in the source system.
  </Step>

  <Step title="Schedule the Ingestion and Deploy">
    Scheduling can be set up at an hourly, daily, weekly, or manual cadence. The
    timezone is in UTC. Select a Start Date to schedule for ingestion. It is
    optional to add an End Date.

    Review your configuration settings. If they match what you intended,
    click Deploy to create the service and schedule metadata ingestion.

    If something doesn't look right, click the Back button to return to the
    appropriate step and change the settings as needed.

    After configuring the workflow, you can click on Deploy to create the
    pipeline.

    <img src="https://mintcdn.com/collatedocs/piJyXg9wW6Ik1lg-/public/images/connectors/schedule.png?fit=max&auto=format&n=piJyXg9wW6Ik1lg-&q=85&s=f1add591824b44456f0e2ff259a21c6f" alt="Schedule the Workflow" width="2733" height="1083" data-path="public/images/connectors/schedule.png" />
  </Step>

  <Step title="View the Ingestion Pipeline">
    Once the workflow has been successfully deployed, you can view the
    Ingestion Pipeline running from the Service Page.

    <img src="https://mintcdn.com/collatedocs/cOe_QuHYxAbkMtTI/public/images/connectors/view-ingestion-pipeline.png?fit=max&auto=format&n=cOe_QuHYxAbkMtTI&q=85&s=8c754af74f99ee70e714f6f707b827e4" alt="View Ingestion Pipeline" width="2733" height="1271" data-path="public/images/connectors/view-ingestion-pipeline.png" />

    <Tip>
      If [AutoPilot](/how-to-guides/admin-guide/applications/autopilot) is enabled, workflows like usage tracking, data lineage, and similar tasks will be handled automatically. Users don’t need to set up or manage them - AutoPilot takes care of everything in the system.
    </Tip>
  </Step>
</Steps>

## Displaying Lineage Information

Steps to retrieve and display the lineage information for a Data Factory service.

1. Ingest Source and Sink Database Metadata: Identify both the source and sink database used by the Azure Data Factory service for example Redshift. Ingest metadata for these database.
2. Ingest Data Factory Service Metadata: Finally, Ingest your Data Factory service.
   By successfully completing these steps, the lineage information for the service will be displayed.

### Missing Lineage

If lineage information is not displayed for a Data Factory service, follow these steps to diagnose the issue.

1. *Permissions*: Ensure that the service principal or managed identity you’re using has the necessary permissions in the Data Factory resource. (Reader, Contributor or Data Factory Contributor role at minimum).
2. *Metadata Ingestion*: Ensure that metadata for both the source and sink database is ingested and passed to the lineage system. This typically involves configuring the relevant connectors to capture and transmit this information.
3. *Run Successful*: Ensure that the Pipeline Run is successful.

## Troubleshooting

<Columns cols={2}>
  <Card title="Data Factory Troubleshooting" href="/connectors/pipeline/datafactory/troubleshooting">
    Learn more about how to troubleshoot common Data Factory connector issues and resolve configuration or ingestion errors.
  </Card>
</Columns>