> ## Documentation Index
> Fetch the complete documentation index at: https://docs.getcollate.io/llms.txt
> Use this file to discover all available pages before exploring further.

# AWS EKS Deployment

> Deploy Collate Binaries on Amazon Elastic Kubernetes Service (EKS) using Argo Workflows for ingestion orchestration.

# Guide to Deploy Collate Binaries in AWS

This guide will help you start using Collate Docker Images to run the Collate OpenMetadata Application in Kubernetes on Amazon EKS, connecting with Argo Workflows for running ingestion from the Collate OpenMetadata Application itself.

## Architecture

Collate OpenMetadata requires four components:

* **Collate Server**.
* **Database**: Collate Server stores the metadata in a relational database. Collate supports MySQL or Postgres. Amazon RDS is recommended for production.
  * MySQL version 8.0.42 or greater.
  * Postgres version 17.6 or greater.
* **Search Engine**: **OpenSearch 3.4** (Amazon OpenSearch Service recommended). ElasticSearch is not supported in Collate BYOC because Collate AI relies on OpenSearch's vector capabilities for Semantic and Hybrid Search.
* **Workflow Orchestration**: Collate uses **Argo Workflows** as the orchestrator for ingestion pipelines.

## Sizing Requirements

The following sections cover hardware, software, and per-component sizing recommendations for a production-ready deployment.

### Hardware Requirements

A Kubernetes Cluster with at least 1 Master Node and 3 Worker Nodes is the required configuration. Each Worker Node should have at least:

* 4 vCPUs.
* 16 GiB Memory.
* 128 GiB Storage capacity.

### Software Requirements

* Collate OpenMetadata supports Kubernetes Cluster version 1.24 or greater.
* Collate Docker Images are available via private AWS Elastic Container Registry (ECR). The Collate Team will share credentials and steps to configure Kubernetes to pull Docker Images from AWS ECR.
* For Argo Workflows, Collate OpenMetadata is currently compatible with application version 3.4+.

### Recommended AWS Instance Types

| Component              | Instance Type         |
| ---------------------- | --------------------- |
| Collate Server         | t4g.large / m6a.large |
| Argo Workflows runners | m7i.large             |

### Database Sizing and Capacity

Collate recommends configuring Amazon RDS Postgres. For 100,000 Data Assets and 1,000 Users:

* 8 vCPUs.
* 64 GiB Memory.
* 256 GiB Storage Capacity.
* 3,500 IOPS storage.

### Search Client Sizing and Capacity

For 100,000 Data Assets and 1,000 Users:

* 8 vCPUs.
* 64 GiB Memory.
* 256 GiB Storage Capacity.

Use Amazon OpenSearch Service for production. Recommend multiple availability zones with a minimum of 2 nodes.

### Argo Workflows Ingestion Runners

The recommended resources are **4 vCPUs and 16 GiB of Memory**. Ingestion workloads can be scheduled on spot instances to reduce costs.

## AWS Prerequisites

Complete the following AWS configuration steps before installing Argo Workflows and Collate.

### Enable EKS OIDC Provider

EKS clusters use an OIDC provider to enable IAM Roles for Service Accounts (IRSA). Check if your cluster already has one:

```bash theme={null}
aws eks describe-cluster --name <CLUSTER_NAME> --query "cluster.identity.oidc.issuer" --output text
```

If no output is returned, associate an OIDC provider:

```bash theme={null}
eksctl utils associate-iam-oidc-provider \
  --cluster <CLUSTER_NAME> \
  --region <AWS_REGION> \
  --approve
```

Retrieve the OIDC issuer URL for use in subsequent steps:

```bash theme={null}
OIDC_ISSUER=$(aws eks describe-cluster --name <CLUSTER_NAME> \
  --query "cluster.identity.oidc.issuer" --output text | sed 's|https://||')
echo "OIDC Issuer: $OIDC_ISSUER"
```

### Create an S3 Bucket for Argo Workflows Artifacts

Argo Workflows archives ingestion logs to S3:

```bash theme={null}
aws s3 mb s3://collate-argo-artifacts-<AWS_REGION> --region <AWS_REGION>
```

### Create IAM Roles for Service Accounts

Four IAM roles are required—one for each service account (Collate Server, Collate Ingestion, Argo Controller, Argo Server).

#### IAM Role for Argo Workflows Controller

Create the trust policy and IAM role for the Argo Workflows controller service account:

```bash theme={null}
cat > argo-controller-trust-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_ISSUER}"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "${OIDC_ISSUER}:sub": "system:serviceaccount:argo-workflows:argo-workflows-controller-sa"
      }
    }
  }]
}
EOF

aws iam create-role \
  --role-name collate-argo-controller-role \
  --assume-role-policy-document file://argo-controller-trust-policy.json

aws iam attach-role-policy \
  --role-name collate-argo-controller-role \
  --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess
```

#### IAM Role for Argo Workflows Server

Create the trust policy and IAM role for the Argo Workflows server service account:

```bash theme={null}
cat > argo-server-trust-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_ISSUER}"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "${OIDC_ISSUER}:sub": "system:serviceaccount:argo-workflows:argo-workflows-server-sa"
      }
    }
  }]
}
EOF

aws iam create-role \
  --role-name collate-argo-server-role \
  --assume-role-policy-document file://argo-server-trust-policy.json

aws iam attach-role-policy \
  --role-name collate-argo-server-role \
  --policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess
```

#### IAM Role for Collate Server Application

Create the trust policy and IAM role for the Collate Server application service account:

```bash theme={null}
cat > collate-server-trust-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_ISSUER}"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "${OIDC_ISSUER}:sub": "system:serviceaccount:collate:openmetadata"
      }
    }
  }]
}
EOF

aws iam create-role \
  --role-name collate-server-role \
  --assume-role-policy-document file://collate-server-trust-policy.json

aws iam attach-role-policy \
  --role-name collate-server-role \
  --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess
```

#### IAM Role for Collate Ingestion

Create the trust policy and IAM role for the Collate Ingestion service account:

```bash theme={null}
cat > collate-ingestion-trust-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_ISSUER}"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "${OIDC_ISSUER}:sub": "system:serviceaccount:collate:om-role"
      }
    }
  }]
}
EOF

aws iam create-role \
  --role-name collate-ingestion-role \
  --assume-role-policy-document file://collate-ingestion-trust-policy.json

aws iam attach-role-policy \
  --role-name collate-ingestion-role \
  --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess
```

## Setup AWS ECR

Collate will provide the credentials to pull Docker Images from a private registry located in AWS ECR.

### Install AWS CLI

Follow the [AWS CLI installation guide](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) to install AWS CLI on your machine.

### Configure AWS Credentials

Configure the AWS CLI with the credentials for the ECR profile:

```bash theme={null}
aws configure --profile ecr-collate
```

The command will prompt for credentials. The Collate team will securely share these via a 1Password link.

Confirm the credentials are correctly set:

```bash theme={null}
aws configure list --profile ecr-collate
```

### Kubernetes Docker Registry Secrets for AWS ECR

Create a Kubernetes secret so the cluster can pull images from the private ECR registry:

```bash theme={null}
kubectl create secret docker-registry ecr-registry-creds \
  --docker-server=118146679784.dkr.ecr.eu-west-1.amazonaws.com \
  --docker-username=AWS \
  --docker-password=$(aws ecr get-login-password --profile ecr-collate) \
  --namespace <<NAMESPACE_NAME>>
```

<Note>
  **Note**: Replace `<<NAMESPACE_NAME>>` with the namespace where you want to deploy Collate OpenMetadata Server. If the namespace does not exist yet, create it first with `kubectl create namespace <<NAMESPACE_NAME>>`.
</Note>

ECR authentication tokens are short-lived, so plan to refresh them periodically.

<Warning>
  **Important**:
  **AWS ECR Token Refresh**

  ECR tokens expire after 12 hours. If a pod is rescheduled to another node after 12 hours, you will get an `ImagePullBackOff` error. Delete the secret and recreate it using the docker-registry secret creation command.
</Warning>

## Install Argo Workflows

Install Argo Workflows using its Helm chart, then configure it to work with Collate ingestion.

### Add Helm Repository

Add the community Argo Workflows Helm repository:

```bash theme={null}
helm repo add argo https://argoproj.github.io/argo-helm
helm repo update
```

### Create the Argo Namespace

Create a dedicated namespace for Argo Workflows:

```bash theme={null}
kubectl create namespace argo-workflows
```

### Kubernetes Secret for Argo Workflows DB Credentials

Create a Kubernetes secret with the database credentials Argo Workflows will use:

```bash theme={null}
kubectl create secret generic argo-db-credentials \
  --from-literal=username=<DB_USERNAME> \
  --from-literal=password=<DB_PASSWORD> \
  --namespace argo-workflows
```

### Create Custom Helm Values for Argo Workflows

Create a file named `argo-workflows.values.yml`:

```yaml theme={null}
# argo-workflows.values.yml
controller:
  resources:
    requests:
      memory: "2Gi"
      cpu: "1"
    limits:
      memory: "4Gi"
      cpu: "2"
  serviceAccount:
    create: true
    name: argo-workflows-controller-sa
    annotations:
      eks.amazonaws.com/role-arn: "arn:aws:iam::<AWS_ACCOUNT_ID>:role/collate-argo-controller-role"
  name: workflow-controller
  workflowDefaults:
    spec:
      serviceAccountName: om-role
  persistence:
    archive: true
    postgresql:
      host: <DATABASE_INSTANCE_ENDPOINT>
      database: <DATABASE_NAME>
      tableName: argo_workflows
      userNameSecret:
        name: argo-db-credentials
        key: username
      passwordSecret:
        name: argo-db-credentials
        key: password
      ssl: true
      sslMode: require

server:
  resources:
    requests:
      memory: "2Gi"
      cpu: "1"
    limits:
      memory: "4Gi"
      cpu: "2"
  serviceAccount:
    create: true
    name: argo-workflows-server-sa
    annotations:
      eks.amazonaws.com/role-arn: "arn:aws:iam::<AWS_ACCOUNT_ID>:role/collate-argo-server-role"
  extraArgs:
    - "--auth-mode=server"
    - "--request-timeout=5m"

useDefaultArtifactRepo: true
useStaticCredentials: false
artifactRepository:
  archiveLogs: true
  s3:
    endpoint: s3.amazonaws.com
    bucket: collate-argo-artifacts-<AWS_REGION>
    keyFormat: 'workflows/{{workflow.namespace}}/{{workflow.name}}/{{pod.name}}'
    insecure: false
    region: <AWS_REGION>
    encryptionOptions:
      enableEncryption: true
```

For further customisation, refer to the [community Helm chart values](https://github.com/argoproj/argo-helm/tree/main/charts/argo-workflows).

### Deploy Argo Workflows

Collate targets application version 3.7.1 using Helm chart version 0.45.23 ([Artifact Hub](https://artifacthub.io/packages/helm/argo/argo-workflows/0.45.23)):

```bash theme={null}
helm upgrade --install argo-workflows argo/argo-workflows \
  --version 0.45.23 \
  --namespace argo-workflows \
  --values argo-workflows.values.yml
```

### Optional: Enable Prometheus Metrics

If you have a Prometheus Application running on your cluster, enable metrics using:

```yaml theme={null}
controller:
  serviceMonitor:
    enabled: true
server:
  serviceMonitor:
    enabled: true
```

Refer to the [official Argo Workflows documentation](https://argoproj.github.io/argo-workflows/metrics/) for further configuration.

## Install Collate

With Argo Workflows in place, install the Collate OpenMetadata application into the cluster.

### Create the Collate Namespace

Create a namespace to host the Collate deployment:

```bash theme={null}
kubectl create namespace collate
```

### Kubernetes Service Account for Ingestion

Create a dedicated Kubernetes service account that Collate uses to trigger ingestion workflows:

```bash theme={null}
kubectl create serviceaccount om-role -n collate
```

### Annotate Service Account with IRSA Role

Annotate the service account so it assumes the IAM role created earlier:

```bash theme={null}
kubectl annotate serviceaccount -n collate om-role \
  eks.amazonaws.com/role-arn=arn:aws:iam::<AWS_ACCOUNT_ID>:role/collate-ingestion-role
```

### Create Long-Lived API Token for the ServiceAccount

Create a long-lived token secret for the service account:

```bash theme={null}
kubectl apply -n collate -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: om-role.service-account-token
  annotations:
    kubernetes.io/service-account.name: om-role
type: kubernetes.io/service-account-token
EOF
```

### Configure Kubernetes Roles for the Service Account

Create a file `om-argo-role.yml`:

```yaml theme={null}
# om-argo-role.yml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: om-argo-role
  namespace: collate
rules:
  - verbs: [list, watch, create, update, patch, get, delete]
    apiGroups:
      - argoproj.io
    resources:
      - workflows
  - verbs: [list, watch, patch, get]
    apiGroups:
      - ''
    resources:
      - pods/log
      - pods
  - verbs: [list, watch, create, update, patch, get, delete]
    apiGroups:
      - argoproj.io
    resources:
      - cronworkflows
  - verbs: [create, patch]
    apiGroups:
      - argoproj.io
    resources:
      - workflowtaskresults
```

Apply the role and create the role binding:

```bash theme={null}
kubectl apply -f om-argo-role.yml

kubectl create rolebinding om-argo-role-binding \
  --role=om-argo-role \
  --serviceaccount=collate:om-role \
  --namespace collate
```

### Install the Collate Helm Chart

Create Kubernetes Secrets for the database connection:

```bash theme={null}
kubectl create secret generic db-credentials \
  --from-literal=password=<<DATABASE_PASSWORD>> \
  --namespace collate
```

Add the Helm chart repository:

```bash theme={null}
helm repo add open-metadata https://helm.open-metadata.org/
helm repo update
```

<Note>
  **Note**: If you plan to use the **DeltaLake connector**, the `ARGO_INGESTION_IMAGE` value should be:
  `118146679784.dkr.ecr.eu-west-1.amazonaws.com/collate-customers-ingestion-eu-west-1:om-1.13.0-cl-1.13.0`
</Note>

Create a file `openmetadata.values.yml`:

```yaml theme={null}
# openmetadata.values.yml
replicaCount: 1
resources:
  limits:
    cpu: 3000m
    memory: 12Gi
  requests:
    cpu: 1000m
    memory: 10Gi
openmetadata:
  config:
    elasticsearch:
      host: ${es_host}
      port: ${es_port}
      scheme: ${es_scheme}
      searchType: opensearch
      auth:
        enabled: true
        username: ${es_username}
        password:
          secretRef: es-credentials
          secretKey: password
    database:
      host: ${db_host}
      port: ${db_port}
      driverClass: org.postgresql.Driver
      dbScheme: postgresql
      auth:
        username: ${db_user}
        password:
          secretRef: db-credentials
          secretKey: password
      dbParams: "allowPublicKeyRetrieval=true&useSSL=true&serverTimezone=UTC"
    pipelineServiceClientConfig:
      enabled: true
      type: "argoWorkflows"
      metadataApiEndpoint: "http://openmetadata:8585/api"
      argoWorkflows:
        namespace: collate
        serviceAccountName: om-role
        ingestionImage: "118146679784.dkr.ecr.eu-west-1.amazonaws.com/collate-customers-ingestion-slim-eu-west-1:om-1.13.0-cl-1.13.0"
        imagePullPolicy: "IfNotPresent"
        imagePullSecrets: "ecr-registry-creds"
        apiEndpoint: "http://argo-workflows-server.argo-workflows:2746"
image:
  repository: 118146679784.dkr.ecr.eu-west-1.amazonaws.com/collate-customers-eu-west-1
  tag: om-1.13.0-cl-1.13.0
  imagePullPolicy: IfNotPresent
imagePullSecrets:
  - name: ecr-registry-creds
extraEnvs:
  - name: ARGO_TOKEN
    valueFrom:
      secretKeyRef:
        name: "om-role.service-account-token"
        key: "token"
  - name: OPENMETADATA_HEAP_OPTS
    value: "-Xmx8G -Xms8G"
  - name: ASSET_UPLOADER_PROVIDER
    value: "s3"
  - name: ASSET_UPLOADER_MAX_FILE_SIZE
    value: "10485760"
  - name: ASSET_UPLOADER_S3_BUCKET_NAME
    value: "<S3_BUCKET_NAME>"
  - name: ASSET_UPLOADER_S3_REGION
    value: "<AWS_REGION>"
  - name: ASSET_UPLOADER_S3_PREFIX_PATH
    value: "assets/collate"
serviceAccount:
  name: "openmetadata"
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<AWS_ACCOUNT_ID>:role/collate-server-role
```

Install the Collate OpenMetadata Application:

```bash theme={null}
helm upgrade --install openmetadata open-metadata/openmetadata \
  --values openmetadata.values.yml \
  --namespace collate
```

### Optional: Enable Prometheus Metrics

Collate Application exposes Prometheus metrics on port `8586`. Enable the integration using:

```yaml theme={null}
serviceMonitor:
  enabled: true
```

## Post Installation/Upgrade Steps

Complete the following step after installing or upgrading Collate.

### Configure ReIndexing

After installation or upgrade, configure ReIndexing from the Collate UI. For detailed steps, refer to the [Reindexing Search guide](/ai-2-0/admin-guide/reindexing-search).

## Environment Variables for Collate OpenMetadata Argo

| Environment Name                              | Description                                                                                                                           | Default Value                        | Required |
| --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------ | -------- |
| `ARGO_IMAGE_PULL_SECRETS`                     | Image Pull Secret Name to pull Docker Images for Ingestion from a Private Registry. Multiple secrets can be supplied comma-separated. | Empty String                         | False    |
| `ARGO_INGESTION_IMAGE`                        | Docker Image and Tag for Ingestion Images                                                                                             | `openmetadata/ingestion-base:1.13.0` | True     |
| `ARGO_NAMESPACE`                              | Namespace in which Argo Workflows will be executed. Must match the namespace where Collate is deployed.                               | `argo`                               | True     |
| `ARGO_SERVER_CERTIFICATE_PATH`                | SSL Certificate Path to connect to Argo Server                                                                                        | Empty String                         | False    |
| `ARGO_TEST_CONNECTION_BACKOFF_TIME`           | Backoff retry time in seconds to test the connection                                                                                  | `5`                                  | False    |
| `ARGO_TOKEN`                                  | JWT Token to authenticate with Argo Workflow API                                                                                      | Empty String                         | True     |
| `ARGO_WORKFLOW_CPU_LIMIT`                     | Kubernetes CPU Limits for Argo Workflows created with Ingestion                                                                       | `1000m`                              | False    |
| `ARGO_WORKFLOW_CPU_REQUEST`                   | Kubernetes CPU Requests for Argo Workflows created with Ingestion                                                                     | `200m`                               | False    |
| `ARGO_WORKFLOW_CUSTOMER_TOLERATION`           | Kubernetes Node Toleration to schedule Ingestion Workflow Pods to specific Nodes                                                      | `argo`                               | False    |
| `ARGO_WORKFLOW_EXECUTOR_SERVICE_ACCOUNT_NAME` | Service Account Name to be used for Argo Workflows for Ingestion                                                                      | `om-role`                            | True     |
| `ARGO_WORKFLOW_MEMORY_LIMIT`                  | Kubernetes Memory Limits for Argo Workflows created with Ingestion                                                                    | `4096Mi`                             | False    |
| `ARGO_WORKFLOW_MEMORY_REQUEST`                | Kubernetes Memory Requests for Argo Workflows created with Ingestion                                                                  | `256Mi`                              | False    |
| `ASSET_UPLOADER_ENABLE`                       | Enable Asset Upload Feature                                                                                                           | `True`                               | False    |
| `ASSET_UPLOADER_PROVIDER`                     | Asset Upload Provider Name. Can be `s3` or `azure`.                                                                                   | `s3`                                 | False    |
| `ASSET_UPLOADER_MAX_FILE_SIZE`                | Max File Size to support for Asset Upload (in bytes)                                                                                  | `5242880`                            | False    |
| `ASSET_UPLOADER_S3_BUCKET_NAME`               | Asset Upload S3 Bucket Name                                                                                                           | Empty String                         | False    |
| `ASSET_UPLOADER_S3_REGION`                    | Asset Upload S3 Region                                                                                                                | Empty String                         | False    |
| `ASSET_UPLOADER_S3_PREFIX_PATH`               | Asset Upload S3 Prefix Path                                                                                                           | `assets/default`                     | False    |

## Appendix: List of AWS ECR Public IPs

If your company policy blocks access to external resources, ensure the public IPs of AWS ECR are reachable from your cluster.

### Using Terraform

Query the AWS IP ranges for the ECR region using Terraform:

```hcl theme={null}
data "aws_ip_ranges" "ip_ranges" {
  regions  = ["eu-west-1"]
  services = ["amazon"]
}

output "ireland_ip_ranges" {
  value = data.aws_ip_ranges.ip_ranges.cidr_blocks
}
```

### Using curl and jq

Query the AWS IP ranges for the ECR region using curl and jq:

```bash theme={null}
curl -s https://ip-ranges.amazonaws.com/ip-ranges.json | jq '.prefixes[] | select(.region=="eu-west-1") | select(.service=="AMAZON")'
```