Skip to main content
Google Drive

Google Drive

BETA
In this section, we provide guides and references to use the Google Drive connector. Configure and schedule Google Drive metadata and profiler workflows from the OpenMetadata UI:

​
How to Run the Connector Externally

To run the Ingestion via the UI you’ll need to use the OpenMetadata Ingestion Container, which comes shipped with custom Airflow plugins to handle the workflow deployment. If, instead, you want to manage your workflows externally on your preferred orchestrator, you can check the following docs to run the Ingestion Framework anywhere.

External Schedulers

Get more information about running the Ingestion Framework Externally

​
Requirements

​
GCP Credentials

To connect to Google Drive, you’ll need to set up a Google Cloud Platform (GCP) service account with the necessary permissions.

​
1. Create a Service Account

  1. Go to the Google Cloud Console
  2. Navigate to IAM & Admin > Service Accounts
  3. Click Create Service Account
  4. Provide a name and description for the service account
  5. Grant the service account the necessary roles (at minimum: Viewer role)
  6. Click Done

​
2. Enable Google Drive API

  1. In the Google Cloud Console, navigate to APIs & Services > Library
  2. Search for “Google Drive API”
  3. Click on it and enable the API

​
3. Create and Download Service Account Key

  1. Navigate back to IAM & Admin > Service Accounts
  2. Click on your newly created service account
  3. Go to the Keys tab
  4. Click Add Key > Create new key
  5. Choose JSON as the key type
  6. Download the JSON key file

​
4. Domain-Wide Delegation (Optional)

If you need to access files owned by other users in your organization:
  1. In the service account details, click Show Domain-Wide Delegation
  2. Enable Enable Google Workspace Domain-wide Delegation
  3. Note the Client ID
  4. In Google Workspace Admin Console:
    • Navigate to Security > API Controls > Domain-wide Delegation
    • Add the Client ID with the following OAuth scopes:
      • https://www.googleapis.com/auth/drive.readonly
      • https://www.googleapis.com/auth/spreadsheets.readonly
  5. Specify the delegated email address in the connection configuration

​
Python Requirements

We have support for Python versions 3.9-3.11
To run the Google Drive ingestion, you will need to install: 
pip3 install "openmetadata-ingestion[googledrive]"

​
Metadata Ingestion

All connectors are defined as JSON Schemas. Here you can find the structure to create a connection to Google Drive. In order to create and run a Metadata Ingestion workflow, we will follow the steps to create a YAML configuration able to connect to the source, process the Entities if needed, and reach the OpenMetadata server. The workflow is modeled around the following JSON Schema

​
1. Define the YAML Config

This is a sample config for Google Drive:

​
2. Run with the CLI

First, we will need to save the YAML file. Afterward, and with all requirements installed, we can run: 
metadata ingest -c <path-to-yaml>
Note that from connector to connector, this recipe will always be the same. By updating the YAML configuration, you will be able to extract metadata from different sources.

​
Data Profiler

The Data Profiler workflow will be using the orm-profiler processor. After running a Metadata Ingestion workflow, we can run the Data Profiler workflow. While the serviceName will be the same to that was used in Metadata Ingestion, so the ingestion bot can get the serviceConnection details from the server.

​
1. Define the YAML Config

This is a sample config for the profiler:
  • You can learn more about how to configure and run the Profiler Workflow to extract Profiler data and execute the Data Quality from here

​
2. Run with the CLI

After saving the YAML config, we will run the command the same way we did for the metadata ingestion: 
metadata profile -c <path-to-yaml>
Note now instead of running ingest, we are using the profile command to select the Profiler workflow.

Data Profiler

Find more information about the Data Profiler here