> ## Documentation Index
> Fetch the complete documentation index at: https://docs.getcollate.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Kubernetes On Premises Deployment | Collate

> Deploy Collate Binaries on an on-premises Kubernetes cluster using Argo Workflows for ingestion orchestration.

# Guide to Deploy Collate Binaries On-Premises

This guide will help you start using Collate Docker Images to run the OpenMetadata Application in an on-premises Kubernetes cluster, connecting with Argo Workflows for running ingestion from the OpenMetadata Application itself.

<Warning>
  This guide presumes you have an on-premises Kubernetes cluster already set up. All code snippets assume the `collate` namespace unless otherwise noted.
</Warning>

## Architecture

Collate OpenMetadata requires 4 components:

1. **Collate Server**
2. **Database** — Collate Server stores the metadata in a relational database. We support MySQL or Postgres.
   * MySQL version 8.0.42 or greater
   * Postgres version 17.6 or greater
3. **Search Engine** — **OpenSearch 3.4**. ElasticSearch is not supported in Collate BYOC because Collate AI relies on OpenSearch's vector capabilities for Semantic and Hybrid Search.
4. **Workflow Orchestration** — We use **Argo Workflows** as the orchestrator for ingestion pipelines.

## Sizing Requirements

### Hardware Requirements

A Kubernetes Cluster with at least 1 Master Node and 3 Worker Nodes is the required configuration. Each Worker Node should have at least:

* 4 vCPUs
* 16 GiB Memory
* 128 GiB Storage capacity

<Note>
  If you want Collate workloads scheduled on dedicated nodes, use Kubernetes taints and tolerations. Collate OpenMetadata supports `tolerations` via custom Helm values.
</Note>

### Software Requirements

* Collate OpenMetadata supports Kubernetes Cluster version 1.24 or greater.
* Collate Docker Images are available via private AWS Elastic Container Registry (ECR). The Collate Team will share credentials and steps to configure Kubernetes to pull Docker Images from AWS ECR.
* For Argo Workflows, Collate OpenMetadata is currently compatible with application version 3.4+.

### Database Sizing and Capacity

Our recommendation is to configure PostgreSQL. For 100,000 Data Assets and 1,000 Users:

* 8 vCPUs
* 64 GiB Memory
* 256 GiB Storage Capacity
* 3,500 IOPS storage

### Search Client Sizing and Capacity

For 100,000 Data Assets and 1,000 Users:

* 8 vCPUs
* 64 GiB Memory
* 256 GiB Storage Capacity

### Argo Workflows Ingestion Runners

The recommended resources are **4 vCPUs and 16 GiB of Memory**.

## On-Premises Prerequisites

### Object Storage for Argo Workflows Artifacts

Argo Workflows requires object storage to archive ingestion logs. On-premises deployments can use **MinIO** as an S3-compatible object store.

#### Deploy MinIO (if you don't have an existing object store)

```bash theme={null}
helm repo add minio https://charts.min.io/
helm repo update

helm upgrade --install minio minio/minio \
  --namespace minio \
  --create-namespace \
  --set rootUser=admin \
  --set rootPassword=<MINIO_ROOT_PASSWORD> \
  --set persistence.size=50Gi \
  --set service.type=ClusterIP
```

Create the Argo Workflows artifacts bucket:

```bash theme={null}
kubectl run minio-client --image=minio/mc --rm -it --restart=Never -- \
  /bin/sh -c "mc alias set local http://minio.minio:9000 admin <MINIO_ROOT_PASSWORD> && mc mb local/argo-artifacts"
```

#### Create Kubernetes Secret for MinIO Credentials

```bash theme={null}
kubectl create secret generic argo-artifact-credentials \
  --from-literal=accessKey=<MINIO_ACCESS_KEY> \
  --from-literal=secretKey=<MINIO_SECRET_KEY> \
  --namespace argo-workflows
```

## Setup AWS ECR

Collate will provide the credentials to pull Docker Images from a private registry located in AWS ECR.

### Install AWS CLI

Follow the [AWS CLI installation guide](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) to install AWS CLI on your machine.

### Configure AWS Credentials

```bash theme={null}
aws configure --profile ecr-collate
```

The command will prompt for credentials. The Collate team will securely share these via a 1Password link.

Confirm the credentials are correctly set:

```bash theme={null}
aws configure list --profile ecr-collate
```

### Kubernetes Docker Registry Secrets for AWS ECR

```bash theme={null}
kubectl create secret docker-registry ecr-registry-creds \
  --docker-server=118146679784.dkr.ecr.eu-west-1.amazonaws.com \
  --docker-username=AWS \
  --docker-password=$(aws ecr get-login-password --profile ecr-collate) \
  --namespace <<NAMESPACE_NAME>>
```

<Note>
  Replace `<<NAMESPACE_NAME>>` with the namespace where you want to deploy Collate OpenMetadata Server. If the namespace does not exist yet, create it with `kubectl create namespace <<NAMESPACE_NAME>>`.
</Note>

<Warning>
  **AWS ECR Token Refresh**

  ECR tokens expire after 12 hours. If a pod is rescheduled after 12 hours, you will get an `ImagePullBackOff` error. Delete the secret and recreate it using the command above.
</Warning>

## Install Argo Workflows

### Add Helm Repository

```bash theme={null}
helm repo add argo https://argoproj.github.io/argo-helm
helm repo update
```

### Create the Argo Namespace

```bash theme={null}
kubectl create namespace argo-workflows
```

### Kubernetes Secret for Argo Workflows DB Credentials

```bash theme={null}
kubectl create secret generic argo-db-credentials \
  --from-literal=username=<DB_USERNAME> \
  --from-literal=password=<DB_PASSWORD> \
  --namespace argo-workflows
```

### Create Custom Helm Values for Argo Workflows

Create a file named `argo-workflows.values.yml`:

```yaml theme={null}
# argo-workflows.values.yml
controller:
  serviceAccount:
    create: true
    name: argo-workflows-controller-sa
  name: workflow-controller
  workflowDefaults:
    spec:
      serviceAccountName: om-role

server:
  serviceAccount:
    create: true
    name: argo-workflows-server-sa
  extraArgs:
    - "--auth-mode=server"
    - "--request-timeout=5m"

persistence:
  archive: true
  postgresql:
    host: <DATABASE_INSTANCE_ENDPOINT>
    database: <DATABASE_NAME>
    tableName: argo_workflows
    userNameSecret:
      name: argo-db-credentials
      key: username
    passwordSecret:
      name: argo-db-credentials
      key: password
    ssl: true
    sslMode: require

useDefaultArtifactRepo: true
useStaticCredentials: true
artifactRepository:
  archiveLogs: true
  s3:
    # MinIO endpoint — replace with your object store endpoint if different
    endpoint: minio.minio:9000
    insecure: true
    bucket: argo-artifacts
    keyFormat: 'workflows/{{workflow.namespace}}/{{workflow.name}}/{{pod.name}}'
    accessKeySecret:
      name: argo-artifact-credentials
      key: accessKey
    secretKeySecret:
      name: argo-artifact-credentials
      key: secretKey
```

<Note>
  If you are using an existing S3-compatible store (e.g., Ceph, NetApp StorageGRID) instead of MinIO, update `endpoint`, `bucket`, and the secret reference to match your environment. Set `insecure: false` and configure TLS if your store uses HTTPS.
</Note>

For further customisation, refer to the [community Helm chart values](https://github.com/argoproj/argo-helm/tree/main/charts/argo-workflows).

### Deploy Argo Workflows

We target application version 3.7.1 using Helm chart version 0.45.23 ([Artifact Hub](https://artifacthub.io/packages/helm/argo/argo-workflows/0.45.23)):

```bash theme={null}
helm upgrade --install argo-workflows argo/argo-workflows \
  --version 0.45.23 \
  --namespace argo-workflows \
  --values argo-workflows.values.yml
```

### \[Optional] Enable Prometheus Metrics

If you have a Prometheus Application running on your cluster, enable metrics using:

```yaml theme={null}
controller:
  serviceMonitor:
    enabled: true
server:
  serviceMonitor:
    enabled: true
```

Refer to the [official Argo Workflows documentation](https://argoproj.github.io/argo-workflows/metrics/) for further configuration.

## Install OpenMetadata/Collate

### Create the Collate Namespace

```bash theme={null}
kubectl create namespace collate
```

### Kubernetes Service Account for Ingestion

The OpenMetadata Application communicates with Argo Workflows to dynamically trigger ephemeral pods that run ingestion workloads. Create a dedicated Kubernetes Service Account:

```bash theme={null}
kubectl create serviceaccount om-role -n collate
```

### Create Long-Lived API Token for the ServiceAccount

```bash theme={null}
kubectl apply -n collate -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: om-role.service-account-token
  annotations:
    kubernetes.io/service-account.name: om-role
type: kubernetes.io/service-account-token
EOF
```

### Configure Kubernetes Roles for the Service Account

Create a file `om-argo-role.yml`:

```yaml theme={null}
# om-argo-role.yml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: om-argo-role
  namespace: collate
rules:
  - verbs: [list, watch, create, update, patch, get, delete]
    apiGroups:
      - argoproj.io
    resources:
      - workflows
  - verbs: [list, watch, patch, get]
    apiGroups:
      - ''
    resources:
      - pods/log
      - pods
  - verbs: [list, watch, create, update, patch, get, delete]
    apiGroups:
      - argoproj.io
    resources:
      - cronworkflows
  - verbs: [create, patch]
    apiGroups:
      - argoproj.io
    resources:
      - workflowtaskresults
```

Apply the role and create the role binding:

```bash theme={null}
kubectl apply -f om-argo-role.yml

kubectl create rolebinding om-argo-role-binding \
  --role=om-argo-role \
  --serviceaccount=collate:om-role \
  --namespace collate
```

### Install OpenMetadata Helm Chart

Create Kubernetes Secrets for the database connection:

```bash theme={null}
kubectl create secret generic db-credentials \
  --from-literal=password=<<DATABASE_PASSWORD>> \
  --namespace collate
```

Add the Helm chart repository:

```bash theme={null}
helm repo add open-metadata https://helm.open-metadata.org/
helm repo update
```

<Note>
  If you plan to use the **DeltaLake connector**, the `ARGO_INGESTION_IMAGE` value should be:
  `118146679784.dkr.ecr.eu-west-1.amazonaws.com/collate-customers-ingestion-eu-west-1:om-1.13.0-cl-1.13.0`
</Note>

Create a file `openmetadata.values.yml`:

```yaml theme={null}
# openmetadata.values.yml
replicaCount: 1
openmetadata:
  config:
    elasticsearch:
      host: ${es_host}
      port: ${es_port}
      scheme: ${es_scheme}
      searchType: opensearch
      auth:
        enabled: true
        username: ${es_username}
        password:
          secretRef: es-credentials
          secretKey: password
    database:
      host: ${db_host}
      port: ${db_port}
      driverClass: org.postgresql.Driver
      dbScheme: postgresql
      auth:
        username: ${db_user}
        password:
          secretRef: db-credentials
          secretKey: password
      dbParams: "allowPublicKeyRetrieval=true&useSSL=true&serverTimezone=UTC"
    pipelineServiceClientConfig:
      className: "io.collate.pipeline.argo.ArgoServiceClient"
      apiEndpoint: "http://argo-workflows-server.argo-workflows:2746"
      metadataApiEndpoint: "http://openmetadata:8585/api"
      auth:
        enabled: false
image:
  repository: 118146679784.dkr.ecr.eu-west-1.amazonaws.com/collate-customers-eu-west-1
  tag: om-1.13.0-cl-1.13.0
  imagePullPolicy: IfNotPresent
imagePullSecrets:
  - name: ecr-registry-creds
extraEnvs:
  - name: ARGO_NAMESPACE
    value: collate
  - name: ARGO_TOKEN
    valueFrom:
      secretKeyRef:
        name: "om-role.service-account-token"
        key: "token"
  - name: ARGO_INGESTION_IMAGE
    value: "118146679784.dkr.ecr.eu-west-1.amazonaws.com/collate-customers-ingestion-slim-eu-west-1:om-1.13.0-cl-1.13.0"
  - name: ARGO_WORKFLOW_EXECUTOR_SERVICE_ACCOUNT_NAME
    value: om-role
  - name: ARGO_IMAGE_PULL_SECRETS
    value: ecr-registry-creds
  - name: ASSET_UPLOADER_PROVIDER
    value: "s3"
  - name: ASSET_UPLOADER_MAX_FILE_SIZE
    value: "10485760"
  - name: ASSET_UPLOADER_S3_ENDPOINT
    value: "http://minio.minio:9000"
  - name: ASSET_UPLOADER_S3_BUCKET_NAME
    value: "argo-artifacts"
  - name: ASSET_UPLOADER_S3_PREFIX_PATH
    value: "assets/collate"
serviceAccount:
  name: "openmetadata"
```

Install the Collate OpenMetadata Application:

```bash theme={null}
helm upgrade --install openmetadata open-metadata/openmetadata \
  --values openmetadata.values.yml \
  --namespace collate
```

### \[Optional] Enable Prometheus Metrics

Collate Application exposes Prometheus metrics on port `8586`. Enable the integration using:

```yaml theme={null}
serviceMonitor:
  enabled: true
```

## Post Installation/Upgrade Steps

### Configure ReIndexing

After installation or upgrade, configure ReIndexing from the OpenMetadata UI. For detailed steps, refer to the [OpenMetadata upgrade documentation](https://docs.open-metadata.org/v1.4.x/deployment/upgrade#reindex).

## Troubleshooting

### Pods Stuck in Pending State

Check for resource constraints or missing secrets:

```bash theme={null}
kubectl describe pod -n collate -l app.kubernetes.io/name=openmetadata
```

| Symptom                     | Cause                         | Fix                                                                   |
| --------------------------- | ----------------------------- | --------------------------------------------------------------------- |
| `ImagePullBackOff`          | ECR secret missing or expired | Recreate `ecr-registry-creds` with a fresh ECR token                  |
| `Insufficient cpu / memory` | Cluster at capacity           | Reduce `resources.requests` in `openmetadata.values.yml` or add nodes |
| `Pending` on PVC            | No default StorageClass       | Set a default StorageClass or pass explicit `storageClass` in values  |

### Argo Workflows Cannot Connect to Object Storage

Verify the MinIO service is reachable from the `argo-workflows` namespace:

```bash theme={null}
kubectl run test --image=curlimages/curl --rm -it --restart=Never -n argo-workflows -- \
  curl -s http://minio.minio:9000/minio/health/live
```

Expected response: `200 OK`. If it fails, check that MinIO is running and the endpoint in `argo-workflows.values.yml` is correct.

## Environment Variables for Collate OpenMetadata Argo

| Environment Name                              | Description                                                                                                                           | Default Value                        | Required |
| --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------ | -------- |
| `ARGO_IMAGE_PULL_SECRETS`                     | Image Pull Secret Name to pull Docker Images for Ingestion from a Private Registry. Multiple secrets can be supplied comma-separated. | Empty String                         | False    |
| `ARGO_INGESTION_IMAGE`                        | Docker Image and Tag for Ingestion Images                                                                                             | `openmetadata/ingestion-base:1.13.0` | True     |
| `ARGO_NAMESPACE`                              | Namespace in which Argo Workflows will be executed. Must match the namespace where OpenMetadata is deployed.                          | `argo`                               | True     |
| `ARGO_SERVER_CERTIFICATE_PATH`                | SSL Certificate Path to connect to Argo Server                                                                                        | Empty String                         | False    |
| `ARGO_TEST_CONNECTION_BACKOFF_TIME`           | Backoff retry time in seconds to test the connection                                                                                  | `5`                                  | False    |
| `ARGO_TOKEN`                                  | JWT Token to authenticate with Argo Workflow API                                                                                      | Empty String                         | True     |
| `ARGO_WORKFLOW_CPU_LIMIT`                     | Kubernetes CPU Limits for Argo Workflows created with Ingestion                                                                       | `1000m`                              | False    |
| `ARGO_WORKFLOW_CPU_REQUEST`                   | Kubernetes CPU Requests for Argo Workflows created with Ingestion                                                                     | `200m`                               | False    |
| `ARGO_WORKFLOW_CUSTOMER_TOLERATION`           | Kubernetes Node Toleration to schedule Ingestion Workflow Pods to specific Nodes                                                      | `argo`                               | False    |
| `ARGO_WORKFLOW_EXECUTOR_SERVICE_ACCOUNT_NAME` | Service Account Name to be used for Argo Workflows for Ingestion                                                                      | `om-role`                            | True     |
| `ARGO_WORKFLOW_MEMORY_LIMIT`                  | Kubernetes Memory Limits for Argo Workflows created with Ingestion                                                                    | `4096Mi`                             | False    |
| `ARGO_WORKFLOW_MEMORY_REQUEST`                | Kubernetes Memory Requests for Argo Workflows created with Ingestion                                                                  | `256Mi`                              | False    |
| `ASSET_UPLOADER_ENABLE`                       | Enable Asset Upload Feature                                                                                                           | `True`                               | False    |
| `ASSET_UPLOADER_PROVIDER`                     | Asset Upload Provider Name. Can be `s3` or `azure`.                                                                                   | `s3`                                 | False    |
| `ASSET_UPLOADER_MAX_FILE_SIZE`                | Max File Size to support for Asset Upload (in bytes)                                                                                  | `5242880`                            | False    |
| `ASSET_UPLOADER_S3_ENDPOINT`                  | Custom S3-compatible endpoint (e.g. MinIO)                                                                                            | Empty String                         | False    |
| `ASSET_UPLOADER_S3_BUCKET_NAME`               | Asset Upload S3/MinIO Bucket Name                                                                                                     | Empty String                         | False    |
| `ASSET_UPLOADER_S3_PREFIX_PATH`               | Asset Upload S3/MinIO Prefix Path                                                                                                     | `assets/default`                     | False    |