​

Troubleshooting

​

Could Not Get the Secret Value or Forbidden

ERROR (metadata.utils.kubernetes_secrets_manager:159) - Could not get the secret
value of <path>
Reason: Forbidden

kubectl logs -l app.kubernetes.io/name=hybrid-ingestion-runner,app.kubernetes.io/instance=collate-prod | grep secretsManager

​

Cause 1 — Missing IAM or Workload Identity

• Verify the annotation on the ingestion service account.
• Confirm the cloud IAM binding is in place for your provider.

​

Cause 2 — Secret Name Mismatch

• Open your secrets store (AWS Secrets Manager, Azure Key Vault, or GCP Secret Manager) and confirm the exact name the secret is stored under.
• In the Collate UI connection form, verify the masked field contains secret:<secret-name>, where <secret-name> matches the name in your secrets store character for character.
• Check for typos, extra slashes, or path segments that aren’t part of the stored secret name.

​

Cause 3 — Missing secretsManager Helm Value

1. Open your values.yaml.
2. Confirm config.secretsManager is set to the correct value for your provider (managed-aws, gcp, or managed-azure-kv).
3. Run helm upgrade to apply the change.

​

Runner Shows as ERROR in the Collate UI

kubectl logs -n argo-workflows -l app.kubernetes.io/name=hybrid-ingestion-runner,app.kubernetes.io/instance=<release-name> | grep "Authentication rejected"

​

Runner Shows as Inactive in the Collate UI

• Check that the authToken in values.yaml is the correct JWT from the IngestionBot.
• Verify outbound TLS (port 443) is allowed from your cluster to <your-instance>.getcollate.io.
• Confirm the pod is running: kubectl get pods.
• Check the Runner pod logs for connection or authentication errors:
  kubectl logs -l app.kubernetes.io/name=hybrid-ingestion-runner,app.kubernetes.io/instance=collate-prod
  

​

ImagePullBackOff on the Runner Pod

kubectl create job --from=cronjob/ecr-registry-helper manual

​

Test Connection or Ingestion Job Rejected: Memory Requests Must Equal Limits

io.kubernetes.client.openapi.ApiException: Message:
HTTP response code: 400
... admission webhook "validate.kyverno.svc-fail" denied the request:
resource Job/<namespace>/automation-<id> was blocked due to the following policies
memory-requests-equal-limits:
  autogen-memory-requests-equal-limits: resources.requests.memory must be equal to
    resources.limits.memory

• The ingestion image is never pulled or started. The image tag in the log (for example om-1.13.4-cl-1.13.4) is not the cause.
• No ingestion pod exists to inspect. kubectl get pods and kubectl logs show nothing for the run, because nothing was scheduled.

1. Confirm the runner version and executor. Check the runner pod logs for the executor in use (SimpleK8s or Argo):
   kubectl logs -l app.kubernetes.io/name=hybrid-ingestion-runner,app.kubernetes.io/instance=<release-name> | grep -iE "SimpleK8s|Argo"
   
2. Check the memory configuration the runner actually loaded:
   kubectl exec <runner-pod> -- printenv | grep -iE "MEMORY"
   
3. Compare the request and limit the runner submits against the policy. Because the rejected Job is never created, read the submitted values from the webhook message (the requests.memory and limits.memory it reports) rather than from kubectl describe job, which has nothing to show. To confirm, review the admission events:
   kubectl get events -n <namespace> --sort-by='.lastTimestamp' | grep -i kyverno
   
4. Set the request and limit to the same value using the variables that match your executor:
   • Argo Workflows: ARGO_WORKFLOW_MEMORY_REQUEST and ARGO_WORKFLOW_MEMORY_LIMIT
   • Simple Kubernetes: INGESTION_REQUEST_MEMORY and INGESTION_LIMIT_MEMORY
   For example, set both to 4Gi. See Configuring Ingestion Pod Memory for full examples.
5. Run helm upgrade to apply the change.
6. Restart the runner pod.

​

Ingestion Pod Not Found — Diagnostics Unavailable

Could not retrieve pod diagnostics (pod may be deleted, missing RBAC
permissions, or other Kubernetes errors).

WARNING - No main pod found for workflow <workflow-id>
WARNING - Could not find main pod for workflow <workflow-id> - skipping diagnostics

• Cluster autoscaling scaled down the node running the ingestion pod.
• A pod cleanup policy or TTL controller removed the pod.
• The node was rotated or replaced during the ingestion run.

1. Check the pod state immediately after the next failure:
   kubectl get pods -n argo-workflows
   kubectl describe pod <ingestion-pod-name> -n argo-workflows
   
2. Review cluster events around the time of failure:
   kubectl get events -n argo-workflows --sort-by='.lastTimestamp'
   
3. Once identified, work with your infrastructure team to address the cause — for example, configuring scale-down protection for ingestion workloads or excluding the argo-workflows namespace from pod cleanup policies.