Guide to Deploy Collate Binaries On-Premises

This guide will help you start using Collate Docker Images to run the Collate OpenMetadata Application in an on-premises Kubernetes cluster, connecting with Argo Workflows for running ingestion from the Collate OpenMetadata Application itself.

Important: This guide presumes you have an on-premises Kubernetes cluster already set up. All code snippets assume the collate namespace unless otherwise noted.

Architecture

Collate OpenMetadata requires four components:

Collate Server.
Database: Collate Server stores the metadata in a relational database. Collate supports MySQL or Postgres.
- MySQL version 8.0.42 or greater.
- Postgres version 17.6 or greater.
Search Engine: OpenSearch 3.4. ElasticSearch is not supported in Collate BYOC because Collate AI relies on OpenSearch’s vector capabilities for Semantic and Hybrid Search.
Workflow Orchestration: Collate uses Argo Workflows as the orchestrator for ingestion pipelines.

Sizing Requirements

The following sections cover hardware, software, and per-component sizing recommendations for a production-ready deployment.

Hardware Requirements

A Kubernetes Cluster with at least 1 Master Node and 3 Worker Nodes is the required configuration. Each Worker Node should have at least:

4 vCPUs.
16 GiB Memory.
128 GiB Storage capacity.

Note: If you want Collate workloads scheduled on dedicated nodes, use Kubernetes taints and tolerations. Collate OpenMetadata supports tolerations via custom Helm values.

Software Requirements

Collate OpenMetadata supports Kubernetes Cluster version 1.24 or greater.
Collate Docker Images are available via private AWS Elastic Container Registry (ECR). The Collate Team will share credentials and steps to configure Kubernetes to pull Docker Images from AWS ECR.
For Argo Workflows, Collate OpenMetadata is currently compatible with application version 3.4+.

Database Sizing and Capacity

Collate recommends configuring Postgres. For 100,000 Data Assets and 1,000 Users:

8 vCPUs.
64 GiB Memory.
256 GiB Storage Capacity.
3,500 IOPS storage.

Search Client Sizing and Capacity

For 100,000 Data Assets and 1,000 Users:

8 vCPUs.
64 GiB Memory.
256 GiB Storage Capacity.

Argo Workflows Ingestion Runners

The recommended resources are 4 vCPUs and 16 GiB of Memory.

On-Premises Prerequisites

Complete the following object storage setup before installing Argo Workflows and Collate.

Object Storage for Argo Workflows Artifacts

Argo Workflows requires object storage to archive ingestion logs. On-premises deployments can use MinIO as an S3-compatible object store.

Deploy MinIO (If You Don’t Have an Existing Object Store)

Install MinIO using its Helm chart:

helm repo add minio https://charts.min.io/
helm repo update

helm upgrade --install minio minio/minio \
  --namespace minio \
  --create-namespace \
  --set rootUser=admin \
  --set rootPassword=<MINIO_ROOT_PASSWORD> \
  --set persistence.size=50Gi \
  --set service.type=ClusterIP

Create the Argo Workflows artifacts bucket:

kubectl run minio-client --image=minio/mc --rm -it --restart=Never -- \
  /bin/sh -c "mc alias set local http://minio.minio:9000 admin <MINIO_ROOT_PASSWORD> && mc mb local/argo-artifacts"

Create Kubernetes Secret for MinIO Credentials

Create a Kubernetes secret with the MinIO access credentials:

kubectl create secret generic argo-artifact-credentials \
  --from-literal=accessKey=<MINIO_ACCESS_KEY> \
  --from-literal=secretKey=<MINIO_SECRET_KEY> \
  --namespace argo-workflows

Setup AWS ECR

Collate will provide the credentials to pull Docker Images from a private registry located in AWS ECR.

Install AWS CLI

Follow the AWS CLI installation guide to install AWS CLI on your machine.

Configure AWS Credentials

Configure the AWS CLI with the credentials for the ECR profile:

aws configure --profile ecr-collate

The command will prompt for credentials. The Collate team will securely share these via a 1Password link. Confirm the credentials are correctly set:

aws configure list --profile ecr-collate

Kubernetes Docker Registry Secrets for AWS ECR

Create a Kubernetes secret so the cluster can pull images from the private ECR registry:

kubectl create secret docker-registry ecr-registry-creds \
  --docker-server=118146679784.dkr.ecr.eu-west-1.amazonaws.com \
  --docker-username=AWS \
  --docker-password=$(aws ecr get-login-password --profile ecr-collate) \
  --namespace <<NAMESPACE_NAME>>

Note: Replace <<NAMESPACE_NAME>> with the namespace where you want to deploy Collate OpenMetadata Server. If the namespace does not exist yet, create it with kubectl create namespace <<NAMESPACE_NAME>>.

ECR authentication tokens are short-lived, so plan to refresh them periodically.

Important: AWS ECR Token RefreshECR tokens expire after 12 hours. If a pod is rescheduled after 12 hours, you will get an ImagePullBackOff error. Delete the secret and recreate it using the docker-registry secret creation command.

Install Argo Workflows

Install Argo Workflows using its Helm chart, then configure it to work with Collate ingestion.

Add Helm Repository

Add the community Argo Workflows Helm repository:

helm repo add argo https://argoproj.github.io/argo-helm
helm repo update

Create the Argo Namespace

Create a dedicated namespace for Argo Workflows:

kubectl create namespace argo-workflows

Kubernetes Secret for Argo Workflows DB Credentials

Create a Kubernetes secret with the database credentials Argo Workflows will use:

kubectl create secret generic argo-db-credentials \
  --from-literal=username=<DB_USERNAME> \
  --from-literal=password=<DB_PASSWORD> \
  --namespace argo-workflows

Create Custom Helm Values for Argo Workflows

Create a file named argo-workflows.values.yml:

# argo-workflows.values.yml
controller:
  serviceAccount:
    create: true
    name: argo-workflows-controller-sa
  name: workflow-controller
  workflowDefaults:
    spec:
      serviceAccountName: om-role

server:
  serviceAccount:
    create: true
    name: argo-workflows-server-sa
  extraArgs:
    - "--auth-mode=server"
    - "--request-timeout=5m"

persistence:
  archive: true
  postgresql:
    host: <DATABASE_INSTANCE_ENDPOINT>
    database: <DATABASE_NAME>
    tableName: argo_workflows
    userNameSecret:
      name: argo-db-credentials
      key: username
    passwordSecret:
      name: argo-db-credentials
      key: password
    ssl: true
    sslMode: require

useDefaultArtifactRepo: true
useStaticCredentials: true
artifactRepository:
  archiveLogs: true
  s3:
    # MinIO endpoint — replace with your object store endpoint if different
    endpoint: minio.minio:9000
    insecure: true
    bucket: argo-artifacts
    keyFormat: 'workflows/{{workflow.namespace}}/{{workflow.name}}/{{pod.name}}'
    accessKeySecret:
      name: argo-artifact-credentials
      key: accessKey
    secretKeySecret:
      name: argo-artifact-credentials
      key: secretKey

Note: If you are using an existing S3-compatible store (e.g., Ceph, NetApp StorageGRID) instead of MinIO, update endpoint, bucket, and the secret reference to match your environment. Set insecure: false and configure TLS if your store uses HTTPS.

For further customisation, refer to the community Helm chart values.

Deploy Argo Workflows

Collate targets application version 3.7.1 using Helm chart version 0.45.23 (Artifact Hub):

helm upgrade --install argo-workflows argo/argo-workflows \
  --version 0.45.23 \
  --namespace argo-workflows \
  --values argo-workflows.values.yml

Optional: Enable Prometheus Metrics

If you have a Prometheus Application running on your cluster, enable metrics using:

controller:
  serviceMonitor:
    enabled: true
server:
  serviceMonitor:
    enabled: true

Refer to the official Argo Workflows documentation for further configuration.

Install Collate

With Argo Workflows in place, install the Collate OpenMetadata application into the cluster.

Create the Collate Namespace

Create a namespace to host the Collate deployment:

kubectl create namespace collate

Kubernetes Service Account for Ingestion

The Collate OpenMetadata Application communicates with Argo Workflows to dynamically trigger ephemeral pods that run ingestion workloads. Create a dedicated Kubernetes Service Account:

kubectl create serviceaccount om-role -n collate

Create Long-Lived API Token for the ServiceAccount

Create a long-lived token secret for the service account:

kubectl apply -n collate -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: om-role.service-account-token
  annotations:
    kubernetes.io/service-account.name: om-role
type: kubernetes.io/service-account-token
EOF

Configure Kubernetes Roles for the Service Account

Create a file om-argo-role.yml:

# om-argo-role.yml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: om-argo-role
  namespace: collate
rules:
  - verbs: [list, watch, create, update, patch, get, delete]
    apiGroups:
      - argoproj.io
    resources:
      - workflows
  - verbs: [list, watch, patch, get]
    apiGroups:
      - ''
    resources:
      - pods/log
      - pods
  - verbs: [list, watch, create, update, patch, get, delete]
    apiGroups:
      - argoproj.io
    resources:
      - cronworkflows
  - verbs: [create, patch]
    apiGroups:
      - argoproj.io
    resources:
      - workflowtaskresults

Apply the role and create the role binding:

kubectl apply -f om-argo-role.yml

kubectl create rolebinding om-argo-role-binding \
  --role=om-argo-role \
  --serviceaccount=collate:om-role \
  --namespace collate

Install the Collate Helm Chart

Create Kubernetes Secrets for the database connection:

kubectl create secret generic db-credentials \
  --from-literal=password=<<DATABASE_PASSWORD>> \
  --namespace collate

Add the Helm chart repository:

helm repo add open-metadata https://helm.open-metadata.org/
helm repo update

Note: If you plan to use the DeltaLake connector, the ARGO_INGESTION_IMAGE value should be: 118146679784.dkr.ecr.eu-west-1.amazonaws.com/collate-customers-ingestion-eu-west-1:om-1.13.0-cl-1.13.0

Create a file openmetadata.values.yml:

# openmetadata.values.yml
replicaCount: 1
openmetadata:
  config:
    elasticsearch:
      host: ${es_host}
      port: ${es_port}
      scheme: ${es_scheme}
      searchType: opensearch
      auth:
        enabled: true
        username: ${es_username}
        password:
          secretRef: es-credentials
          secretKey: password
    database:
      host: ${db_host}
      port: ${db_port}
      driverClass: org.postgresql.Driver
      dbScheme: postgresql
      auth:
        username: ${db_user}
        password:
          secretRef: db-credentials
          secretKey: password
      dbParams: "allowPublicKeyRetrieval=true&useSSL=true&serverTimezone=UTC"
    pipelineServiceClientConfig:
      className: "io.collate.pipeline.argo.ArgoServiceClient"
      apiEndpoint: "http://argo-workflows-server.argo-workflows:2746"
      metadataApiEndpoint: "http://openmetadata:8585/api"
      auth:
        enabled: false
image:
  repository: 118146679784.dkr.ecr.eu-west-1.amazonaws.com/collate-customers-eu-west-1
  tag: om-1.13.0-cl-1.13.0
  imagePullPolicy: IfNotPresent
imagePullSecrets:
  - name: ecr-registry-creds
extraEnvs:
  - name: ARGO_NAMESPACE
    value: collate
  - name: ARGO_TOKEN
    valueFrom:
      secretKeyRef:
        name: "om-role.service-account-token"
        key: "token"
  - name: ARGO_INGESTION_IMAGE
    value: "118146679784.dkr.ecr.eu-west-1.amazonaws.com/collate-customers-ingestion-slim-eu-west-1:om-1.13.0-cl-1.13.0"
  - name: ARGO_WORKFLOW_EXECUTOR_SERVICE_ACCOUNT_NAME
    value: om-role
  - name: ARGO_IMAGE_PULL_SECRETS
    value: ecr-registry-creds
  - name: ASSET_UPLOADER_PROVIDER
    value: "s3"
  - name: ASSET_UPLOADER_MAX_FILE_SIZE
    value: "10485760"
  - name: ASSET_UPLOADER_S3_ENDPOINT
    value: "http://minio.minio:9000"
  - name: ASSET_UPLOADER_S3_BUCKET_NAME
    value: "argo-artifacts"
  - name: ASSET_UPLOADER_S3_PREFIX_PATH
    value: "assets/collate"
serviceAccount:
  name: "openmetadata"

Install the Collate OpenMetadata Application:

helm upgrade --install openmetadata open-metadata/openmetadata \
  --values openmetadata.values.yml \
  --namespace collate

Optional: Enable Prometheus Metrics

Collate Application exposes Prometheus metrics on port 8586. Enable the integration using:

serviceMonitor:
  enabled: true

Post Installation/Upgrade Steps

Complete the following step after installing or upgrading Collate.

Configure ReIndexing

After installation or upgrade, configure ReIndexing from the Collate UI. For detailed steps, refer to the Reindexing Search guide.

Troubleshooting

Pods Stuck in Pending State

Check for resource constraints or missing secrets:

kubectl describe pod -n collate -l app.kubernetes.io/name=openmetadata

Symptom	Cause	Fix
`ImagePullBackOff`	ECR secret missing or expired	Recreate `ecr-registry-creds` with a fresh ECR token
`Insufficient cpu / memory`	Cluster at capacity	Reduce `resources.requests` in `openmetadata.values.yml` or add nodes
`Pending` on PVC	No default StorageClass	Set a default StorageClass or pass explicit `storageClass` in values

Argo Workflows Cannot Connect to Object Storage

Verify the MinIO service is reachable from the argo-workflows namespace:

kubectl run test --image=curlimages/curl --rm -it --restart=Never -n argo-workflows -- \
  curl -s http://minio.minio:9000/minio/health/live

Expected response: 200 OK. If it fails, check that MinIO is running and the endpoint in argo-workflows.values.yml is correct.

Environment Variables for Collate OpenMetadata Argo

Environment Name	Description	Default Value	Required
`ARGO_IMAGE_PULL_SECRETS`	Image Pull Secret Name to pull Docker Images for Ingestion from a Private Registry. Multiple secrets can be supplied comma-separated.	Empty String	False
`ARGO_INGESTION_IMAGE`	Docker Image and Tag for Ingestion Images	`openmetadata/ingestion-base:1.13.0`	True
`ARGO_NAMESPACE`	Namespace in which Argo Workflows will be executed. Must match the namespace where Collate is deployed.	`argo`	True
`ARGO_SERVER_CERTIFICATE_PATH`	SSL Certificate Path to connect to Argo Server	Empty String	False
`ARGO_TEST_CONNECTION_BACKOFF_TIME`	Backoff retry time in seconds to test the connection	`5`	False
`ARGO_TOKEN`	JWT Token to authenticate with Argo Workflow API	Empty String	True
`ARGO_WORKFLOW_CPU_LIMIT`	Kubernetes CPU Limits for Argo Workflows created with Ingestion	`1000m`	False
`ARGO_WORKFLOW_CPU_REQUEST`	Kubernetes CPU Requests for Argo Workflows created with Ingestion	`200m`	False
`ARGO_WORKFLOW_CUSTOMER_TOLERATION`	Kubernetes Node Toleration to schedule Ingestion Workflow Pods to specific Nodes	`argo`	False
`ARGO_WORKFLOW_EXECUTOR_SERVICE_ACCOUNT_NAME`	Service Account Name to be used for Argo Workflows for Ingestion	`om-role`	True
`ARGO_WORKFLOW_MEMORY_LIMIT`	Kubernetes Memory Limits for Argo Workflows created with Ingestion	`4096Mi`	False
`ARGO_WORKFLOW_MEMORY_REQUEST`	Kubernetes Memory Requests for Argo Workflows created with Ingestion	`256Mi`	False
`ASSET_UPLOADER_ENABLE`	Enable Asset Upload Feature	`True`	False
`ASSET_UPLOADER_PROVIDER`	Asset Upload Provider Name. Can be `s3` or `azure`.	`s3`	False
`ASSET_UPLOADER_MAX_FILE_SIZE`	Max File Size to support for Asset Upload (in bytes)	`5242880`	False
`ASSET_UPLOADER_S3_ENDPOINT`	Custom S3-compatible endpoint (e.g. MinIO)	Empty String	False
`ASSET_UPLOADER_S3_BUCKET_NAME`	Asset Upload S3/MinIO Bucket Name	Empty String	False
`ASSET_UPLOADER_S3_PREFIX_PATH`	Asset Upload S3/MinIO Prefix Path	`assets/default`	False

Deployment

Kubernetes On Premises Deployment

Guide to Deploy Collate Binaries On-Premises

Architecture

Sizing Requirements

Hardware Requirements

Software Requirements

Database Sizing and Capacity

Search Client Sizing and Capacity

Argo Workflows Ingestion Runners

On-Premises Prerequisites

Object Storage for Argo Workflows Artifacts

Deploy MinIO (If You Don’t Have an Existing Object Store)

Create Kubernetes Secret for MinIO Credentials

Setup AWS ECR

Install AWS CLI

Configure AWS Credentials

Kubernetes Docker Registry Secrets for AWS ECR

Install Argo Workflows

Add Helm Repository

Create the Argo Namespace

Kubernetes Secret for Argo Workflows DB Credentials

Create Custom Helm Values for Argo Workflows

Deploy Argo Workflows

Optional: Enable Prometheus Metrics

Install Collate

Create the Collate Namespace

Kubernetes Service Account for Ingestion

Create Long-Lived API Token for the ServiceAccount

Configure Kubernetes Roles for the Service Account

Install the Collate Helm Chart

Optional: Enable Prometheus Metrics

Post Installation/Upgrade Steps

Configure ReIndexing

Troubleshooting

Pods Stuck in Pending State

Argo Workflows Cannot Connect to Object Storage

Environment Variables for Collate OpenMetadata Argo

​Guide to Deploy Collate Binaries On-Premises

​Architecture

​Sizing Requirements

​Hardware Requirements

​Software Requirements

​Database Sizing and Capacity

​Search Client Sizing and Capacity

​Argo Workflows Ingestion Runners

​On-Premises Prerequisites

​Object Storage for Argo Workflows Artifacts

​Deploy MinIO (If You Don’t Have an Existing Object Store)

​Create Kubernetes Secret for MinIO Credentials

​Setup AWS ECR

​Install AWS CLI

​Configure AWS Credentials

​Kubernetes Docker Registry Secrets for AWS ECR

​Install Argo Workflows

​Add Helm Repository

​Create the Argo Namespace

​Kubernetes Secret for Argo Workflows DB Credentials

​Create Custom Helm Values for Argo Workflows

​Deploy Argo Workflows

​Optional: Enable Prometheus Metrics

​Install Collate

​Create the Collate Namespace

​Kubernetes Service Account for Ingestion

​Create Long-Lived API Token for the ServiceAccount

​Configure Kubernetes Roles for the Service Account

​Install the Collate Helm Chart

​Optional: Enable Prometheus Metrics

​Post Installation/Upgrade Steps

​Configure ReIndexing

​Troubleshooting

​Pods Stuck in Pending State

​Argo Workflows Cannot Connect to Object Storage

​Environment Variables for Collate OpenMetadata Argo

Guide to Deploy Collate Binaries On-Premises

Architecture

Sizing Requirements

Hardware Requirements

Software Requirements

Database Sizing and Capacity

Search Client Sizing and Capacity

Argo Workflows Ingestion Runners

On-Premises Prerequisites

Object Storage for Argo Workflows Artifacts

Deploy MinIO (If You Don’t Have an Existing Object Store)

Create Kubernetes Secret for MinIO Credentials

Setup AWS ECR

Install AWS CLI

Configure AWS Credentials

Kubernetes Docker Registry Secrets for AWS ECR

Install Argo Workflows

Add Helm Repository

Create the Argo Namespace

Kubernetes Secret for Argo Workflows DB Credentials

Create Custom Helm Values for Argo Workflows

Deploy Argo Workflows

Optional: Enable Prometheus Metrics

Install Collate

Create the Collate Namespace

Kubernetes Service Account for Ingestion

Create Long-Lived API Token for the ServiceAccount

Configure Kubernetes Roles for the Service Account

Install the Collate Helm Chart

Optional: Enable Prometheus Metrics

Post Installation/Upgrade Steps

Configure ReIndexing

Troubleshooting

Pods Stuck in Pending State

Argo Workflows Cannot Connect to Object Storage

Environment Variables for Collate OpenMetadata Argo