Skip to main content
Hex

Hex

BETA
In this section, we provide guides and references to use the Hex connector. Configure and schedule Hex metadata and profiler workflows from the OpenMetadata UI:

​
How to Run the Connector Externally

To run the Ingestion via the UI you’ll need to use the OpenMetadata Ingestion Container, which comes shipped with custom Airflow plugins to handle the workflow deployment. If, instead, you want to manage your workflows externally on your preferred orchestrator, you can check the following docs to run the Ingestion Framework anywhere.

External Schedulers

Get more information about running the Ingestion Framework Externally

​
Requirements

Hex connector supports both personal and workspace tokens for authentication.

​
Hex Account Setup

Follow the steps below to configure the account setup for Hex connector:

​
Step 1: Generate API Token

To generate an API token in Hex:
  1. Log into your Hex account
  2. Navigate to Account Settings
  3. Go to the API Keys section
  4. Click on “Create API Key”
  5. Choose the token type:
    • Personal Token: Provides access to projects you own or have access to
    • Workspace Token: Provides access to all projects in the workspace (requires admin privileges)
  6. Copy and securely store the generated token
API tokens are shown only once when created. Make sure to copy and store them securely.

​
Step 2: Verify API Access

Ensure that the token has appropriate permissions:
  • For personal tokens: Verify you have access to the projects you want to ingest
  • For workspace tokens: Verify admin privileges are granted

​
Python Requirements

We have support for Python versions 3.9-3.11
To run the Hex ingestion, you will need to install: 
pip3 install "openmetadata-ingestion[hex]"

​
Metadata Ingestion

All connectors are defined as JSON Schemas. Here you can find the structure to create a connection to Hex. In order to create and run a Metadata Ingestion workflow, we will follow the steps to create a YAML configuration able to connect to the source, process the Entities if needed, and reach the OpenMetadata server. The workflow is modeled around the following JSON Schema

​
1. Define the YAML Config

This is a sample config for Hex:

​
2. Run with the CLI

First, we will need to save the YAML file. Afterward, and with all requirements installed, we can run: 
metadata ingest -c <path-to-yaml>
Note that from connector to connector, this recipe will always be the same. By updating the YAML configuration, you will be able to extract metadata from different sources.

​
Advanced Configuration

​
Working with Multiple Workspaces

If you need to ingest metadata from multiple Hex workspaces, you can:
  1. Use Workspace Tokens: If you have admin access, use workspace tokens to access all projects in each workspace
  2. Create Multiple Services: Create separate Hex services in OpenMetadata for each workspace
  3. Use Filter Patterns: Apply filter patterns to organize projects by workspace

​
Scheduling Ingestion

For production environments, consider:
  • Setting up regular ingestion schedules (e.g., daily or hourly)
  • Using workflow orchestration tools like Airflow
  • Monitoring ingestion logs for failures
  • Setting up alerts for ingestion issues

​
Performance Optimization

For large Hex deployments:
  • Use filter patterns to limit the scope of ingestion
  • Consider running ingestion during off-peak hours
  • Monitor API rate limits and adjust accordingly
  • Use incremental ingestion where possible

​
Lineage

Hex connector extracts lineage information by:
  • Warehouse Query History: Directly queries data warehouse logs (Snowflake, BigQuery, Databricks, Redshift, etc.) to find queries executed by Hex
  • Hex Metadata Identification: Identifies Hex-originated queries using metadata comments embedded by Hex containing project IDs
  • SQL Parsing: Analyzes the discovered SQL queries to extract upstream table dependencies

​
Lineage Configuration

To configure lineage extraction, add the lineageInformation section with dbServicePrefixes to specify which database services to query: 
sourceConfig:
  config:
    type: DashboardMetadata
    lineageInformation:
      dbServicePrefixes:
        - local_clickhouse            # Example: ClickHouse service
        - prod                        # Matches services starting with "prod"
        - bigquery.dataset1           # Specific BigQuery dataset
        - snowflake.PROD_DB.PUBLIC    # Specific Snowflake schema
        - mysql.openmetadata_db       # Specific MySQL database

​
Troubleshooting

​
Common Issues

​
Authentication Failures

Error: Authentication failed with provided token
Solution:
  • Verify your API token is valid and hasn’t expired
  • Check that the token type matches your access level
  • Ensure the token has necessary permissions

​
Missing Projects

Warning: Expected projects not found in ingestion
Solution:
  • For personal tokens: Verify you have access to the projects
  • For workspace tokens: Confirm admin privileges
  • Review dashboard filter patterns

​
Connection Timeouts

Error: Connection timeout to Hex API
Solution:
  • Verify the hostPort URL is correct
  • Check network connectivity
  • For on-premise installations, ensure firewall rules allow access

​
Rate Limiting

Error: API rate limit exceeded
Solution:
  • Reduce ingestion frequency
  • Contact Hex support for rate limit increases
  • Implement exponential backoff in retry logic

​
Debug Mode

To enable debug logging for troubleshooting: 
workflowConfig:
  loggerLevel: DEBUG
This will provide detailed logs about:
  • API calls being made
  • Response data received
  • Processing steps
  • Any errors or warnings

​
2. Run with the CLI

First, we will need to save the YAML file. Afterward, and with all requirements installed, we can run: 
metadata ingest -c <path-to-yaml>
Note that from connector to connector, this recipe will always be the same. By updating the YAML configuration, you will be able to extract metadata from different sources.