​

Custom Drive Connector

​

Steps to Set Up a Custom Drive Connector

​

Step 1 - Prepare Your Drive Connector

from metadata.ingestion.api.steps import Source

• prepare
• _iter
• test_connection (optional but recommended so the UI can validate access)

custom_drive/
  my_drive_connector.py

class MyDriveConnector(Source):
    ...
    def _iter(self):
        ...

​

Step 2 - Yield Drive Data as Entities

from metadata.ingestion.api.models import Either, StackTraceError
from metadata.generated.schema.api.data.createDirectory import CreateDirectoryRequest
from metadata.generated.schema.api.data.createFile import CreateFileRequest
from metadata.generated.schema.api.data.createSpreadsheet import CreateSpreadsheetRequest
from metadata.generated.schema.api.data.createWorksheet import CreateWorksheetRequest

for directory in self.drive_client.list_directories():
    try:
        yield Either(
            right=CreateDirectoryRequest(
                name=directory.name,
                description=directory.description,
                parent=directory.parent_fqn,
                href=directory.href,
            )
        )
    except Exception as exc:
        yield Either(
            left=StackTraceError(
                name="Drive Directory Ingestion Error",
                error=str(exc),
                stack_trace=traceback.format_exc(),
            )
        )

for file_metadata in self.drive_client.list_files():
    yield Either(
        right=CreateFileRequest(
            name=file_metadata.name,
            description=file_metadata.description,
            fileFormats=file_metadata.file_formats,
            parent=file_metadata.parent_directory_fqn,
            size=file_metadata.size,
        )
    )

​

Step 3 - Package Your Custom Drive Connector

from setuptools import setup, find_packages

setup(
    name="custom_drive_connector",
    version="0.1",
    packages=find_packages(),
    install_requires=[],
)

python setup.py sdist

​

Step 4 - Update the Ingestion Image

FROM openmetadata/ingestion:2.0.5

WORKDIR ingestion
USER airflow

COPY custom_drive custom_drive
COPY setup.py .
RUN pip install --no-deps .

​

Step 5 - Run Collate with Custom Image

run:
	docker compose -f docker-compose.yml up --build

​

Step 6 - Configure Custom Drive in the Collate UI

1. Navigate to Settings > Services, and select Drive service.
2. Click Add New Service and select Custom.
3. Enter your Source Python Class as:
   
   custom_drive.my_drive_connector.MyDriveConnector
   
   

​

Configure Metadata Agent and Schedule Ingestion

1. Navigate to Settings > Services and select the Drives service.
2. Select your service and click the Agents tab.
3. Click Add Agent and select Metadata from the dropdown. For some services, the dropdown is not available and clicking Add Agent takes you directly to the agent configuration page.
4. On the Configure Ingestion page, do the following and click Next.
   • Name this Ingestion: Enter a unique, recognizable name for this ingestion pipeline.
   • Agent Setup: Configure the core parameters for this agent. The following fields are available:

     Field	Default	Description
     Number of Threads	1	Number of threads to use for parallel drive ingestion.
     Mark Deleted Directories	On	Soft-delete directories in Collate when they are removed from the source. Associated entities like files, spreadsheets, worksheets, and lineage are also deleted.
     Mark Deleted Files	On	Soft-delete files in Collate when they are removed from the source. Associated entities like lineage are also deleted.
     Mark Deleted Spreadsheets	On	Soft-delete spreadsheets in Collate when they are removed from the source. Associated entities like worksheets and lineage are also deleted.
     Mark Deleted Worksheets	On	Soft-delete worksheets in Collate when they are removed from the source. Associated entities like lineage are also deleted.
     Include Directories	On	Fetch directory metadata. Turn off to skip directories entirely.
     Include Files	On	Fetch file metadata. Turn off to skip files entirely.
     Include Spreadsheets	On	Fetch spreadsheet metadata. Turn off to skip spreadsheets entirely.
     Include Worksheets	On	Fetch worksheet metadata. Turn off to skip worksheets entirely.

     
   • Filter Patterns: Apply include or exclude rules to scope which directories, files, spreadsheets, and worksheets this agent ingests. Use fully qualified name (FQN) filtering to apply a regular expression to fully qualified names (for example, service_name.directory_name.file_name) instead of raw names. These follow the same filter options described in Step 5.
   • Scope & Behaviour: Control what metadata to include and how to handle deletions. Toggle each option on or off based on your needs:

     Toggle	Default	Description
     Enable Debug Log	Off	Sets the ingestion log level to DEBUG. Useful for troubleshooting.
     Include Tags	On	Ingest tags from the source.
     Include Owners	Off	Assign owners from the source when the owner email matches a Collate user. Does not overwrite an existing owner.
     Override Metadata	Off	When on, source values overwrite existing descriptions, tags, owners, and display names in Collate. A description or display name that the source doesn’t send never replaces an existing value. If the connector supplies custom properties, the values it sends are merged over the existing ones: properties the source sends overwrite, and properties it doesn’t send are kept. When off, Collate only updates fields that have no existing value.

     
5. On the Schedule Interval page, set when the agent runs:
   • Schedule: Choose a preset interval (Hourly, Daily, Weekly, Monthly) or enter a custom cron expression.
   • On-Demand: No automatic schedule. Trigger the agent manually when needed.
   
6. Click Add to deploy the agent.