Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.getcollate.io/llms.txt

Use this file to discover all available pages before exploring further.

AWS EKS Hybrid Runner architecture diagram showing the WebSocket connection between Collate SaaS and the Hybrid Runner, and the flow from Argo Workflows through Ingestion Pods to AWS Secrets Manager

​
Requirements

​
Hardware

An EKS cluster with at least 2 worker nodes. Each node requires:
  • 2 x86 vCPUs
  • 8 GiB Memory
  • 64 GiB Storage
Recommended AWS instance type: m6a.large Ingestion workloads can run on Spot Instances to reduce costs, but at least one node must always remain running.
To schedule Collate workloads on dedicated nodes, use Kubernetes taints and tolerations. The Hybrid Runner supports tolerations via custom Helm values.

​
Software

  • Kubernetes 1.28 or later
  • Helm CLI and kubectl installed locally
  • Argo Workflows 3.7 (installed automatically by the Helm chart)

​
Installation

The Helm chart bundles all required Kubernetes resources, including:
  • The Hybrid Runner Server pod
  • Argo Workflows (installed as a dependency)
  • A cron job to periodically renew credentials from Collate’s ECR registry
Helm chart repository: hybrid-ingestion-runner-helm-chart

​
Step 1 — Prerequisites

Before deploying, ensure:
  • Collate has provided you with AWS ECR credentials to pull the required Docker images.
  • You have an EKS cluster running Kubernetes 1.28+.
  • helm and kubectl are installed and configured to target your cluster.
  • A Secrets Store is configured. Connector credentials must be stored in a secrets store — they cannot be entered as plain text in the Collate UI. By default, Kubernetes Secrets are used and the Helm chart installs all required RBAC automatically. Optionally, you can use AWS Secrets Manager — see the Secrets Management section below.

​
Step 2 — Get the Collate Authentication Token

The Hybrid Runner authenticates to the Collate API using a JWT token from the ingestion bot.
  1. Log in to your Collate instance as an administrator.
  2. Navigate to Settings → Bots and search for ingestion.
  3. Open the IngestionBot entry and copy the OpenMetadata JWT Token.
You can also navigate directly to https://<your-instance>.getcollate.io/bots/ingestion-bot.
This token has no expiration date. Store it securely and treat it as a secret.

​
Step 3 — Create the Helm values file

Create a values.yaml file with the minimal required configuration: 
config:
  agentId: "aws-prod"           # A descriptive name shown in the Collate UI
  authToken: <From step 2>
  serverHost: <mycluster>.getcollate.io

ecrRegistryHelper:
  collateCredentials:
    values:
      accessKeyId: <Provided by Collate>
      secretAccessKey: <Provided by Collate>

installArgoWorkflows: true
If you are unsure about any values, reach out to your Collate support contact.

​
Step 4 — Deploy

Add the Helm repository and install the chart: 
helm repo add collate-hybrid https://open-metadata.github.io/hybrid-ingestion-runner-helm-chart
helm repo update
kubectl create namespace argo-workflows
helm upgrade --install collate-prod collate-hybrid/hybrid-ingestion-runner --values values.yaml
If you had to run the previous command multiple times, it’s possible that the helm hooks are not executed and the registry helper is unable to generate the credentials. In that case, the hybrid runner pod will fail with ImagePullBackoff. You can trigger a manual execution of the credentials helper by running kubectl create job --from=cronjob/ecr-registry-helper manual.

​
Step 5 — Validating the Setup

Once deployed, verify the runner is connected from the Collate UI:
  1. Go to Settings → Preferences → Ingestion Runners.
  2. Find your runner by the agentId you configured.
  3. Confirm the status shows Active.

​
Additional Settings

​
Creating a Service with the Hybrid Runner

  1. Navigate to Settings → Services and click Add New Service.
  2. In the Ingestion Runner dropdown, select your Hybrid Runner.
  3. Before filling in connection details, store your connector credentials in your secrets store (see Secrets Management below).

​
Secrets Management

The Hybrid Runner supports two approaches for secrets management. Kubernetes Secrets is the default and requires no additional configuration.

​
Option 1 — Kubernetes Secrets (Default)

By default, the Helm chart uses native Kubernetes Secrets and automatically installs the necessary RBAC permissions on the cluster. This is the most direct approach and works for most customers without any additional setup.

​
Storing a Secret

Create a Kubernetes Secret in the argo-workflows namespace: 
kubectl create secret generic collate-hybrid-ingestion-runner \
  --from-literal=my-db-password='your-secret-value' \
  -n argo-workflows

​
Referencing the Secret in the Collate UI

In any password or secret field in the connection form, enter the secret reference using this format: 
secret:/collate/hybrid-ingestion-runner/<secret-name>
For example: 
secret:/collate/hybrid-ingestion-runner/my-db-password
Secret references only work for fields that are treated as secrets in the connection form — these are the masked fields with a show/hide toggle icon.

​
Option 2 — AWS Secrets Manager (Optional)

If you prefer to use AWS Secrets Manager, configure Pod Identity either via EKS Pod Identity or IRSA to assume an IAM role in your AWS account. The serviceAccount name is ingestion by default. Required IAM permissions: 
secretsmanager:GetSecretValue
secretsmanager:DescribeSecret
secretsmanager:ListSecrets
Permissions can be scoped to a specific path by ARN. Update your Helm values to configure the secrets manager. If using IRSA, also add the role annotation: 
config:
  secretsManager: "managed-aws"
  ingestionPods:
    serviceAccount:
      annotations:
        eks.amazonaws.com/role-arn: arn:aws:iam::<account>:role/<role-name>
Replace <account> and <role-name> with your AWS values. The annotation is only required for IRSA — skip it if using EKS Pod Identity.

​
Storing a Secret

In the AWS Console, go to Secrets Manager → Store a new secret → Other type of secret → Plaintext. Paste the secret value as-is, without quotes or JSON wrapping.

​
Referencing the Secret in the Collate UI

In any password or secret field in the connection form, enter the secret reference using this format: 
secret:/collate/hybrid-ingestion-runner/<secret-name>
For example, if your secret is stored at /collate/hybrid-ingestion-runner/my-db-password, enter: 
secret:/collate/hybrid-ingestion-runner/my-db-password
Secret references only work for fields that are treated as secrets in the connection form — these are the masked fields with a show/hide toggle icon.