Deploy Argo Workflows

Argo Workflows orchestrates Collate ingestion pipelines. It requires special configuration on OpenShift due to Security Context Constraints. The recommended version is 3.5.4 (Helm chart 0.40.8).

Note: Complete Prerequisites before this step to have the OIDC endpoint and S3 bucket values ready.

helm repo add argo https://argoproj.github.io/argo-helm
helm repo update

IAM Roles (ROSA)

Note: This section is ROSA/AWS-specific. On non-AWS OpenShift, skip to Create the Argo Namespace. Configure the artifactRepository in the Helm values to point at your own S3-compatible store, or disable artifact archiving by setting useDefaultArtifactRepo: false.

IRSA for Argo Workflows Controller

The Argo controller needs read/write access to S3 to archive workflow logs.

cat > policy.json <<EOF
{
  "Statement": [
    {
      "Action": "s3:ListBucket",
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::${BUCKET_NAME}",
      "Sid": "ListBuckets"
    },
    {
      "Action": ["s3:PutObject", "s3:GetObject", "s3:DeleteObject",
                 "s3:PutObjectAcl", "s3:PutObjectVersionAcl"],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::${BUCKET_NAME}/workflows/openmetadata/*",
      "Sid": "S3RW"
    }
  ],
  "Version": "2012-10-17"
}
EOF

aws iam create-policy \
  --policy-name collate-argowf-controller-policy \
  --policy-document file://policy.json

cat > assume-role-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_ENDPOINT}"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "${OIDC_ENDPOINT}:sub": "system:serviceaccount:argo:argo-workflows-controller-sa"
      }
    }
  }]
}
EOF

aws iam create-role \
  --role-name collate-argowf-controller-role \
  --assume-role-policy-document file://assume-role-policy.json

aws iam attach-role-policy \
  --role-name collate-argowf-controller-role \
  --policy-arn arn:aws:iam::${AWS_ACCOUNT_ID}:policy/collate-argowf-controller-policy

IRSA for Argo Workflows Server

The Argo server needs read-only access to S3 to fetch and display workflow logs in the UI.

cat > policy.json <<EOF
{
  "Statement": [
    {
      "Action": "s3:ListBucket",
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::${BUCKET_NAME}",
      "Sid": "ListBuckets"
    },
    {
      "Action": ["s3:GetObject"],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::${BUCKET_NAME}/*",
      "Sid": "S3RO"
    }
  ],
  "Version": "2012-10-17"
}
EOF

aws iam create-policy \
  --policy-name collate-argowf-server-policy \
  --policy-document file://policy.json

cat > assume-role-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_ENDPOINT}"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "${OIDC_ENDPOINT}:sub": "system:serviceaccount:argo:argo-workflows-server-sa"
      }
    }
  }]
}
EOF

aws iam create-role \
  --role-name collate-argowf-server-role \
  --assume-role-policy-document file://assume-role-policy.json

aws iam attach-role-policy \
  --role-name collate-argowf-server-role \
  --policy-arn arn:aws:iam::${AWS_ACCOUNT_ID}:policy/collate-argowf-server-policy

Create the Argo Namespace

oc new-project argo

Create Database Credentials Secret

Argo Workflows requires a database for workflow archiving.

oc create secret generic argo-db-credentials \
  --from-literal=username=<DB_USERNAME> \
  --from-literal=password=<DB_PASSWORD> \
  --namespace argo

Grant Required SCCs

OpenShift’s default restricted-v2 SCC prevents Argo system components from running. Grant anyuid to the Argo service accounts before deploying:

oc adm policy add-scc-to-user anyuid \
  system:serviceaccount:argo:argo-workflows-controller-sa

oc adm policy add-scc-to-user anyuid \
  system:serviceaccount:argo:argo-workflows-server-sa

Note: anyuid allows pods to run as any UID including root. This is required for Argo system components only. Collate’s own pods run under restricted-v2 and do not require anyuid.

Helm Values (`argo-workflows.values.yml`)

controller:
  serviceAccount:
    create: true
    name: argo-workflows-controller-sa
  annotations:
    # [ROSA] Remove on non-AWS OpenShift
    eks.amazonaws.com/role-arn: arn:aws:iam::<AWS_ACCOUNT_ID>:role/collate-argowf-controller-role

server:
  serviceAccount:
    create: true
    name: argo-workflows-server-sa
  annotations:
    # [ROSA] Remove on non-AWS OpenShift
    eks.amazonaws.com/role-arn: arn:aws:iam::<AWS_ACCOUNT_ID>:role/collate-argowf-server-role
  extraArgs:
    - "--auth-mode=server"
    - "--request-timeout=5m"

persistence:
  archive: true
  postgresql:
    host: <DB_INSTANCE_ENDPOINT>
    database: <DB_DATABASE_NAME>
    tableName: argo_workflows
    userNameSecret:
      name: argo-db-credentials
      key: username
    passwordSecret:
      name: argo-db-credentials
      key: password
    ssl: true
    sslMode: require

# [ROSA] On non-AWS OpenShift, point at your own S3-compatible store
# or set useDefaultArtifactRepo: false to disable artifact archiving.
useDefaultArtifactRepo: true
useStaticCredentials: false
artifactRepository:
  archiveLogs: true
  s3:
    endpoint: s3.amazonaws.com
    bucket: <AWS_S3_BUCKET_NAME>
    keyFormat: workflows/{{workflow.namespace}}/{{workflow.name}}/{{pod.name}}
    insecure: false
    region: <AWS_REGION>
    encryptionOptions:
      enableEncryption: true

Deploy

helm upgrade --install argo-workflows argo/argo-workflows \
  --version 0.40.8 \
  --namespace argo \
  --values argo-workflows.values.yml

Verify

oc get pods -n argo
oc get route -n argo

Optional: Enable Prometheus Metrics

controller:
  serviceMonitor:
    enabled: true
server:
  serviceMonitor:
    enabled: true

Next Step

Deploy Collate

Create the Collate IAM role, configure ECR credentials, and install Collate with the openmetadata Helm chart.

Deployment

Deploy Argo Workflows | OpenShift (ROSA) Deployment

Deploy Argo Workflows

IAM Roles (ROSA)

IRSA for Argo Workflows Controller

IRSA for Argo Workflows Server

Create the Argo Namespace

Create Database Credentials Secret

Grant Required SCCs

Helm Values (`argo-workflows.values.yml`)

Deploy

Verify

Optional: Enable Prometheus Metrics

Next Step

Deploy Collate

​Deploy Argo Workflows

​IAM Roles (ROSA)

​IRSA for Argo Workflows Controller

​IRSA for Argo Workflows Server

​Create the Argo Namespace

​Create Database Credentials Secret

​Grant Required SCCs

​Helm Values (argo-workflows.values.yml)

​Deploy

​Verify

​Optional: Enable Prometheus Metrics

​Next Step

Deploy Collate

Deploy Argo Workflows

IAM Roles (ROSA)

IRSA for Argo Workflows Controller

IRSA for Argo Workflows Server

Create the Argo Namespace

Create Database Credentials Secret

Grant Required SCCs

Helm Values (`argo-workflows.values.yml`)

Deploy

Verify

Optional: Enable Prometheus Metrics

Next Step