ArgoCD — operations, sync semantics, and the SMP / checksum gotchas

ArgoCD is the GitOps controller every cluster you operate eventually runs. The basics — Application, ApplicationSet, Project, kubectl apply from a Git repo — are well documented. This page is for the parts that bite operators in production: sync semantics, the strategic-merge-patch orphan-field problem, the checksum/cm annotation rollouts, and how to actually operate the controllers.

Architecture, with what each pod does

The hot path on every sync: application-controller asks repo-server to render the chart at the current Git SHA, diffs the rendered manifests against live cluster state, applies the diff. Almost every weird sync issue traces back to either repo-server (rendering errors) or application-controller (diff/apply logic).

App, ApplicationSet, Project — at a glance

Application — a single deployable unit, points at one Git repo / path / revision.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: gpu-operator
  namespace: argocd
spec:
  project: platform
  source:
    repoURL: https://github.com/coreweave/cluster-manager
    path: charts/gpu-operator
    targetRevision: main
    helm:
      valueFiles:
        - values.yaml
        - values-tenant-foo.yaml
  destination:
    server: https://kubernetes.default.svc
    namespace: gpu-operator
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true

ApplicationSet — a generator that produces many Applications.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: per-tenant-gpu-operator
spec:
  generators:
    - list:
        elements:
          - tenant: tenant-foo
            cluster: gpu-cluster-1
          - tenant: tenant-bar
            cluster: gpu-cluster-2
  template:
    metadata:
      name: '{{tenant}}-gpu-operator'
    spec:
      ...

Project — RBAC + source/destination allowlists. Apps live in projects. A project named tenant-foo says "apps in this project may only sync from these repos to these namespaces".

App-of-apps

The pattern where one root Application deploys a directory of Application manifests, which themselves deploy charts. This is how you boot a new cluster from one kubectl apply:

charts/
  apps/
    gpu-operator-app.yaml         # Application -> charts/gpu-operator
    network-operator-app.yaml     # Application -> charts/network-operator
    sunk-app.yaml                 # Application -> charts/sunk
    weka-csi-app.yaml             # Application -> charts/weka-csi

bootstrap/
  root-app.yaml                   # Application -> charts/apps

kubectl apply -f bootstrap/root-app.yaml and ArgoCD recursively brings up everything. Order is not sync-wave-deterministic by default — use argocd.argoproj.io/sync-wave: "-2" annotations on apps that must come up first (CRDs, operators).

Helm via ArgoCD

Three patterns:

1. Single-source, valueFiles in repo:

source:
  repoURL: https://github.com/coreweave/cluster-manager
  path: charts/gpu-operator
  helm:
    valueFiles:
      - values.yaml
      - values-prod.yaml

2. External chart + values file in another repo ($values):

sources:
  - repoURL: https://helm.ngc.nvidia.com/nvidia
    chart: gpu-operator
    targetRevision: v24.6.1
    helm:
      valueFiles:
        - $values/charts/gpu-operator/values.yaml
  - repoURL: https://github.com/coreweave/cluster-manager
    targetRevision: main
    ref: values

The ref: values source is mounted as $values inside the helm template render. This keeps your values files in your config repo and the chart pristine.

3. Inline values:

helm:
  values: |
    driver:
      enabled: false
    toolkit:
      enabled: true

Inline is fine for small overrides; valueFiles is what you want for anything substantial.

Sync options — what each one means

Default is regular kubectl apply with three-way strategic-merge-patch. This is fine until you hit the orphan-field problem.

The strategic-merge-patch orphan-field problem

This is one of the most under-documented gotchas in ArgoCD operations.

The rule of three-way SMP:

• field is in live state, AND
• field is NOT in kubectl.kubernetes.io/last-applied-configuration, AND
• field is NOT in the desired (Git-rendered) manifest

→ then the field is preserved in the live state. It is treated as somebody else's change that we shouldn't disturb.

This is intentional and usually correct — it's why kubectl apply doesn't trample changes made by controllers/webhooks. But it bites when:

Concrete example: chart major-version revert

A cluster runs argocd-helm chart 9.x (which renders an init container with command: [sh, -c] and args: [...long shell snippet...]). For some reason you revert to chart 7.x (which renders command: [/bin/cp, -n, ...] with no args).

What happens on next sync:

• Live state: init container has both command (from 9.x) and args (from 9.x).
• last-applied: written by chart 9.x, has both command and args.
• Desired (chart 7.x): has command but NO args.

3-way SMP looks at args:

• live: [...]
• last-applied: [...]
• desired: not present
• → SMP says "the user wants to remove args" (because last-applied had it, desired doesn't) → args IS removed.

But the other direction — when last-applied gets out of sync with reality (e.g., somebody hand-patched live, or a previous reconcile failed) — the orphan rule kicks in and args gets kept. Now command: [/bin/cp, -n, ...] runs with stray args: [...long shell snippet...] appended, which causes:

/bin/cp: extra operand 'sh -c "..."'
Try '/bin/cp --help' for more information.

Init:CrashLoopBackOff. The repo-server pod (or whatever was being deployed) won't start. The fix is not rolling back further — it's forcing SMP to drop the orphan.

The fix: Replace=true on the resource

Annotate the specific resource (in helm: via chart values that map to deploymentAnnotations):

deploymentAnnotations:
  argocd.argoproj.io/sync-options: Replace=true

Replace=true ignores last-applied entirely and uses kubectl replace semantics: live state is reset to exactly the desired manifest. Orphan args gets dropped.

After one successful sync with Replace=true, you can remove the annotation — the next regular sync will write a fresh last-applied, and SMP resumes correctly.

Watch for this pattern when reverting a chart major version, or after manual kubectl edit on a resource that ArgoCD manages. The symptoms are usually weird init-container failures or "this field shouldn't be here".

The Helm chart 7.x vs 9.x init-container variant

The argocd-helm chart's copyutil init container changed shape across versions:

# Chart 7.x
initContainers:
  - name: copyutil
    image: argoproj/argocd:v2.14.9
    command: [/bin/cp, -n, /usr/local/bin/argocd, /var/run/argocd/argocd-cmp-server]
    # no args

# Chart 9.x
initContainers:
  - name: copyutil
    image: argoproj/argocd:v2.x
    command: [sh, -c]
    args:
      - |
        cp -n /usr/local/bin/argocd /var/run/argocd/argocd-cmp-server && \
        echo done

If a cluster's last-applied was written by 9.x but the desired (after a chart pin downgrade) is 7.x, the live pod has both command and args from 9.x in the resource, and SMP preserves the orphan args. The result on a busybox/Alpine-based image: cp: unknown option: --update=none or extra operand, Init:CrashLoopBackOff.

The healthy old ReplicaSet keeps serving (1/2 replicas), so the cluster looks "Synced + Degraded" — service still up, but the new pod can't roll. kubectl describe pod on the failing pod reveals the bad command line.

selfHeal and prune semantics

syncPolicy:
  automated:
    prune: true        # delete resources that are removed from Git
    selfHeal: true     # re-sync when live state drifts from Git
  syncOptions:
    - PrunePropagationPolicy=foreground
    - PruneLast=true

• selfHeal=true — every reconcile (~3 min default), if live ≠ desired, sync. Without this, manual kubectl edit "wins" until the next Git change. With this, manual edits get reverted within minutes.
• prune=true — resources deleted from Git get deleted from the cluster. Without this, removed-from-Git resources sit forever as orphans.
• PruneLast=true — prune happens after apply, not before. Avoids the "delete old before new is up" downtime.

Default to both on for platform components (operators, CRDs); leave selfHeal off for stateful tenant resources where you want manual control.

The stale checksum/cm rolling update gotcha

Pods rendered by Helm charts often have annotations like:

spec:
  template:
    metadata:
      annotations:
        checksum/cm: "abc123def..."         # SHA of associated ConfigMap
        checksum/cmd-params: "456789..."     # SHA of cmd-params CM

These annotations are part of the pod template. Changing them mutates the pod template hash, which forces a new ReplicaSet, which triggers a rolling update.

The trap: if a pod has been running for 100+ days and the chart has been re-rendered occasionally (but the Deployment never rolled because the rest of the spec didn't change), the live checksum/cm is whatever was current 100 days ago. The next sync that does mutate the pod template — even an unrelated values.yaml change — refreshes those checksums to current values.

The result: a values.yaml change you expected to be a no-op rolling-update-wise causes a full Deployment rollout. Pods that have been up for 6 months start cycling. If they had stuck file descriptors, lingering connections, or any "works because it's been running forever" state, they break.

This was caused by latent stale-checksum drift, not by the change you just pushed. The same rollout would have happened on the next ConfigMap-touching sync, whenever that was.

Mitigations:

• Audit long-uptime pods (kubectl get pods -A --sort-by=.status.startTime | head) before any major sync window.
• For pods you absolutely don't want to roll right now, add argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true or temporarily disable selfHeal on that app.
• After any major chart bump, expect ALL associated pods to roll. Plan for it.

Common operational issues

argocd-repo-server Init:CrashLoopBackOff

99% of the time this is the chart-version SMP orphan problem above. The previous chart version put fields in the init container that the new render doesn't include, SMP preserved them, and the resulting command line is invalid.

Triage:

kubectl -n argocd describe pod -l app.kubernetes.io/name=argocd-repo-server | grep -A 20 'Init Containers:'
# Look for command + args together when the chart version has only command (or vice versa)

kubectl -n argocd get deploy argocd-repo-server -o yaml \
  | yq '.spec.template.spec.initContainers[0]'

Fix: kubectl patch to remove the orphan, or apply Replace=true annotation via Helm values and re-sync.

kubectl -n argocd patch deploy argocd-repo-server --type=json \
  -p='[{"op":"remove","path":"/spec/template/spec/initContainers/0/args"}]'

This is non-destructive (changes the Deployment, not the live ReplicaSet that's serving), and the chart's next render either confirms the patch (Synced) or applies the proper fix.

Application stuck OutOfSync

argocd app diff <app>          # what would change
argocd app sync <app> --dry-run

Common causes:

• Resource has a controller-injected field that ArgoCD considers a diff. Add ignoreDifferences for that field.
• Last sync failed and the Application is in Failed state. Check argocd app get <app> for the error.
• A required parameter (e.g., a Helm value) is missing or wrong; repo-server can't render. Check repo-server logs.

Synced + Degraded

The resource is what Git says, but the running workload is unhealthy. The Application controller doesn't roll back automatically — degraded means "this is what you asked for, but it's not working". Investigate the underlying pod/Deployment.

argocd app sync succeeds but nothing happens

Check the actual sync result:

argocd app get <app> --refresh
argocd app sync <app> --force

--force is Force=true — it reapplies even unchanged resources. Useful when ArgoCD's diff cache is stale.

Operating ArgoCD itself

# Health check
kubectl -n argocd get pods
# argocd-application-controller-0   1/1   Running   0   12d
# argocd-applicationset-controller-...    1/1   Running   0   12d
# argocd-dex-server-...                   1/1   Running   0   12d
# argocd-notifications-controller-...     1/1   Running   0   12d
# argocd-redis-ha-...                     ...
# argocd-repo-server-...           1/1   Running   0   12d
# argocd-server-...                1/1   Running   0   12d

# Tail what it's actually doing
kubectl -n argocd logs -f deploy/argocd-application-controller \
  | grep -E 'sync|reconcil'

# CLI login (use SSO if available)
argocd login argocd.example.internal

# Inspect an Application end-to-end
argocd app get gpu-operator
argocd app history gpu-operator
argocd app rollback gpu-operator <revision>

See also

• RKE2 — typical bootstrap flow with ArgoCD
• GPU Operator — common ArgoCD-managed chart
• Reservations — managed as ArgoCD apps
• Operations: troubleshooting — generic K8s pod debug

External:

• argo-cd.readthedocs.io
• ArgoCD sync options: argo-cd.readthedocs.io/en/stable/user-guide/sync-options/
• Three-way SMP details: kubernetes.io/docs/tasks/manage-kubernetes-objects/declarative-config/