Skip to main content

​
Deploy Argo Workflows

Argo Workflows orchestrates Collate ingestion pipelines. It requires special configuration on OpenShift due to Security Context Constraints. The recommended version is 3.5.4 (Helm chart 0.40.8).
Complete Prerequisites before this step to have the OIDC endpoint and S3 bucket values ready.
helm repo add argo https://argoproj.github.io/argo-helm
helm repo update

​
IAM Roles (ROSA)

This section is ROSA/AWS-specific. On non-AWS OpenShift, skip to Create the Argo Namespace. Configure the artifactRepository in the Helm values to point at your own S3-compatible store, or disable artifact archiving by setting useDefaultArtifactRepo: false.

​
IRSA for Argo Workflows Controller

The Argo controller needs read/write access to S3 to archive workflow logs. 
cat > policy.json <<EOF
{
  "Statement": [
    {
      "Action": "s3:ListBucket",
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::${BUCKET_NAME}",
      "Sid": "ListBuckets"
    },
    {
      "Action": ["s3:PutObject", "s3:GetObject", "s3:DeleteObject",
                 "s3:PutObjectAcl", "s3:PutObjectVersionAcl"],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::${BUCKET_NAME}/workflows/openmetadata/*",
      "Sid": "S3RW"
    }
  ],
  "Version": "2012-10-17"
}
EOF

aws iam create-policy \
  --policy-name collate-argowf-controller-policy \
  --policy-document file://policy.json

cat > assume-role-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_ENDPOINT}"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "${OIDC_ENDPOINT}:sub": "system:serviceaccount:argo:argo-workflows-controller-sa"
      }
    }
  }]
}
EOF

aws iam create-role \
  --role-name collate-argowf-controller-role \
  --assume-role-policy-document file://assume-role-policy.json

aws iam attach-role-policy \
  --role-name collate-argowf-controller-role \
  --policy-arn arn:aws:iam::${AWS_ACCOUNT_ID}:policy/collate-argowf-controller-policy

​
IRSA for Argo Workflows Server

The Argo server needs read-only access to S3 to fetch and display workflow logs in the UI. 
cat > policy.json <<EOF
{
  "Statement": [
    {
      "Action": "s3:ListBucket",
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::${BUCKET_NAME}",
      "Sid": "ListBuckets"
    },
    {
      "Action": ["s3:GetObject"],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::${BUCKET_NAME}/*",
      "Sid": "S3RO"
    }
  ],
  "Version": "2012-10-17"
}
EOF

aws iam create-policy \
  --policy-name collate-argowf-server-policy \
  --policy-document file://policy.json

cat > assume-role-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_ENDPOINT}"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "${OIDC_ENDPOINT}:sub": "system:serviceaccount:argo:argo-workflows-server-sa"
      }
    }
  }]
}
EOF

aws iam create-role \
  --role-name collate-argowf-server-role \
  --assume-role-policy-document file://assume-role-policy.json

aws iam attach-role-policy \
  --role-name collate-argowf-server-role \
  --policy-arn arn:aws:iam::${AWS_ACCOUNT_ID}:policy/collate-argowf-server-policy

​
Create the Argo Namespace

oc new-project argo

​
Create Database Credentials Secret

Argo Workflows requires a database for workflow archiving. 
oc create secret generic argo-db-credentials \
  --from-literal=username=<DB_USERNAME> \
  --from-literal=password=<DB_PASSWORD> \
  --namespace argo

​
Grant Required SCCs

OpenShift’s default restricted-v2 SCC prevents Argo system components from running. Grant anyuid to the Argo service accounts before deploying: 
oc adm policy add-scc-to-user anyuid \
  system:serviceaccount:argo:argo-workflows-controller-sa

oc adm policy add-scc-to-user anyuid \
  system:serviceaccount:argo:argo-workflows-server-sa
anyuid allows pods to run as any UID including root. This is required for Argo system components only. Collate’s own pods run under restricted-v2 and do not require anyuid.

​
Helm Values (argo-workflows.values.yml)

controller:
  serviceAccount:
    create: true
    name: argo-workflows-controller-sa
  annotations:
    # [ROSA] Remove on non-AWS OpenShift
    eks.amazonaws.com/role-arn: arn:aws:iam::<AWS_ACCOUNT_ID>:role/collate-argowf-controller-role

server:
  serviceAccount:
    create: true
    name: argo-workflows-server-sa
  annotations:
    # [ROSA] Remove on non-AWS OpenShift
    eks.amazonaws.com/role-arn: arn:aws:iam::<AWS_ACCOUNT_ID>:role/collate-argowf-server-role
  extraArgs:
    - "--auth-mode=server"
    - "--request-timeout=5m"

persistence:
  archive: true
  postgresql:
    host: <DB_INSTANCE_ENDPOINT>
    database: <DB_DATABASE_NAME>
    tableName: argo_workflows
    userNameSecret:
      name: argo-db-credentials
      key: username
    passwordSecret:
      name: argo-db-credentials
      key: password
    ssl: true
    sslMode: require

# [ROSA] On non-AWS OpenShift, point at your own S3-compatible store
# or set useDefaultArtifactRepo: false to disable artifact archiving.
useDefaultArtifactRepo: true
useStaticCredentials: false
artifactRepository:
  archiveLogs: true
  s3:
    endpoint: s3.amazonaws.com
    bucket: <AWS_S3_BUCKET_NAME>
    keyFormat: workflows/{{workflow.namespace}}/{{workflow.name}}/{{pod.name}}
    insecure: false
    region: <AWS_REGION>
    encryptionOptions:
      enableEncryption: true

​
Deploy

helm upgrade --install argo-workflows argo/argo-workflows \
  --version 0.40.8 \
  --namespace argo \
  --values argo-workflows.values.yml

​
Verify

oc get pods -n argo
oc get route -n argo

​
Optional: Enable Prometheus Metrics

controller:
  serviceMonitor:
    enabled: true
server:
  serviceMonitor:
    enabled: true

​
Next Step

Deploy Collate

Create the Collate IAM role, configure ECR credentials, and install Collate with the openmetadata Helm chart.