Skip to content

Deploying SPOG via ArgoCD

This guide walks you through deploying SPOG's control plane and instrumentation agents using ArgoCD in a GitOps workflow. By the end, you will understand why SPOG requires two small accommodations for ArgoCD's rendering model, how to enable them with a single values switch, and how to deploy glass-instrumentation either with plain Helm Applications or with a helmfile-based fleet Application.

Prerequisites:

  • kubectl configured against the target cluster
  • ArgoCD CLI (argocd) installed (for status checks)
  • SSH access to your Git repository hosting the SPOG chart sources
  • Helm 3 (used locally to inspect charts; ArgoCD renders them server-side)

Overview

ArgoCD renders Helm charts by running helm template rather than helm install. That single difference has two consequences that SPOG handles explicitly:

No Helm release secret

When helm install runs it writes a sh.helm.release.v1.* secret to the cluster. The SPOG secret generator jobs (which create the NATS nkeys, passwords, JWT tokens, and any playlist token secrets you have configured) use that secret to register an owner reference on the secrets they produce. Owner references mean Kubernetes garbage-collects the generated secrets when the Helm release is uninstalled.

Under ArgoCD, no release secret exists. The generators detect this at run time, log a warning, and proceed to create their secrets without an owner reference instead of failing. The deployment succeeds, but you must be aware of the consequence:

Generated secrets are not garbage-collected on uninstall

Because there is no owner reference, the secrets created by the generator jobs — nkeys, passwords, JWT tokens, and playlist tokens — will not be deleted when you remove the ArgoCD Application. NATS user secrets (nkeys, passwords, JWT tokens) follow the naming pattern glass-<name>, where <name> is the key used in the NATS user values; playlist token secrets are named by each token's generate.secretName field. All of them carry the same labels, so a label selector finds (and can delete) the complete set regardless of name:

Bash
1
2
3
4
5
kubectl get secrets -n controlplane \
  -l app.kubernetes.io/managed-by=Helm,app.kubernetes.io/instance=glass-ui
# after reviewing the list:
kubectl delete secrets -n controlplane \
  -l app.kubernetes.io/managed-by=Helm,app.kubernetes.io/instance=glass-ui

TTL-deleted jobs cause OutOfSync drift

The generator jobs carry ttlSecondsAfterFinished: 60. After they complete, Kubernetes deletes them automatically. ArgoCD normally tracks every rendered resource and flags missing ones as OutOfSync. Without intervention, every sync would show permanent drift the moment a job cleaned itself up.

The global.argocd.enabled: true chart switch solves this by annotating the jobs as ArgoCD Sync hooks, which excludes them from sync-status comparison entirely.

The global.argocd.enabled Values Switch

The glass-ui chart provides a single top-level switch that enables all ArgoCD-specific behaviour at once:

YAML
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
# Example: Deploying glass-ui through ArgoCD
#
# ArgoCD renders charts with "helm template", so no Helm release secret
# exists in the cluster. The chart adapts to this with a single switch:
#
#   global.argocd.enabled: true
#
# Effects (plain helm deployments are unaffected when the switch is off):
#   - The secret generator Jobs become wave-less ArgoCD Sync hooks: they are
#     applied together with all other resources (Kubernetes resolves the
#     start-up ordering), but ArgoCD excludes them from sync-status
#     comparison, so their TTL-based self-cleanup causes no drift.
#   - The generator binaries themselves detect the missing
#     sh.helm.release.v1.* secret and create their secrets without an owner
#     reference instead of failing.

# Single switch for all ArgoCD-specific chart behaviour
global:
  argocd:
    enabled: true

# Generated secrets need the generator images; point them (and everything
# else) at your registry as usual.
images:
  registry: "cloudcontrol-registry.local:5000"
  project: "spog"

When global.argocd.enabled: true is set, each secret generator job receives these annotations:

YAML
1
2
argocd.argoproj.io/hook: Sync
argocd.argoproj.io/hook-delete-policy: BeforeHookCreation

What hook: Sync does: ArgoCD Sync hooks are resources that participate in a sync operation but are excluded from the ongoing sync-status comparison. The job runs in the default wave (wave 0) together with all other resources, so no explicit ordering is imposed. Kubernetes resolves the start-up ordering naturally: pods that mount a generated secret stay in ContainerCreating with a FailedMount event until the secret exists; once the generator job creates it, the kubelet's mount retry picks it up and the pod proceeds. In practice this resolves within seconds. Because Sync hooks are excluded from sync-status comparison, the TTL cleanup causes no drift.

What hook-delete-policy: BeforeHookCreation does: If a previous sync left a completed job that has not yet been deleted by its TTL (for example, you re-sync within 60 seconds), ArgoCD removes the old job before creating the new one. This prevents name collisions and ensures the generators always run fresh.

Idempotency: The generators skip secret creation when a secret with the expected name already exists. Subsequent syncs are therefore cheap — the job starts, checks for the existing secret, exits immediately, and is cleaned up by TTL.

glass-instrumentation does not need this switch

The glass-instrumentation chart does not include secret generator jobs. The global.argocd.enabled: true switch is only needed in the glass-ui (control plane) values.

Plain Helm or Helmfile?

The control plane (glass-ui) is always a single ArgoCD Application using ArgoCD's native Helm support. For the instrumentation agents the picture is different: glass-instrumentation is deployed once per CloudControl userplane cluster namespace (for example auth-eu-east, recursor-eu-east-000), so the number of releases grows with your fleet. You have two strategies for managing that fan-out.

Plain Helm means one ArgoCD Application per userplane cluster, each pointing at the helm/glass-instrumentation chart with that cluster's values. Every cluster gets its own sync history, its own rollback point, and its own diff view.

Helmfile is a declarative wrapper around Helm: a single specification file describes a whole set of Helm releases — which chart, which namespace, which values for each — and can generate that set from a template. Instead of maintaining one Application manifest per cluster, you maintain one helmfile that fans out a release per userplane namespace from a region list, and a single ArgoCD Application renders it through a Config Management Plugin (CMP). Adding a region or cluster becomes a one-line change.

When to choose helmfile: lots of userplane clusters with a uniform topology. The break-even point is roughly ten clusters — below that, maintaining individual Application manifests is simpler and gives you per-cluster rollback; above that, hand-maintained manifests become operationally heavy and the helmfile's region list is the better source of truth. Choose plain Helm when clusters have materially different configurations, or when you want independent rollback and per-cluster diff visibility.

Concern Plain Helm Helmfile
Number of Applications One per cluster One for the fleet
Independent rollback Yes, per cluster No, fleet-wide
Per-cluster diff visibility Yes, per Application Single large diff
Handles non-uniform config Yes Only via template logic
Scales past ~10 clusters Operationally heavy Designed for this
Requires helmfile CMP No Yes

You can combine both

The two strategies coexist happily — for example, helmfile for the uniform bulk of the fleet and a plain Helm Application for the few clusters with special configuration needs.

Pick the strategy for this walkthrough; the instructions below adapt accordingly:

How will you deploy glass-instrumentation?

Installing ArgoCD

Install ArgoCD

Bash
1
2
3
kubectl create namespace argocd
kubectl apply -n argocd --server-side \
  -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Server-side apply (--server-side) is required because the ArgoCD CRDs are large enough to exceed the annotation size limit used by client-side apply.

Install the helmfile CMP sidecar

The helmfile CMP is only needed for the helmfile fleet strategy — with plain Helm Applications you can skip this step entirely.

The fleet Application uses the helmfile CMP to render helmfile templates. The plugin is a ConfigMap that tells ArgoCD how to discover and render helmfile-based sources:

YAML
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
# argocd/install/helmfile-cmp.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: helmfile-cmp-config
  namespace: argocd
data:
  plugin.yaml: |
    apiVersion: argoproj.io/v1alpha1
    kind: ConfigManagementPlugin
    metadata:
      name: helmfile
    spec:
      version: v1.0
      generate:
        command: [sh, -c]
        args:
          - helmfile template --include-crds -q
      discover:
        fileName: "./helmfile.yaml*"

ArgoCD resolves the plugin identifier for Applications as <metadata.name>-<spec.version>, so this ConfigMap produces helmfile-v1.0 — the value used later in spec.source.plugin.name. Apply the ConfigMap, then patch the repo-server deployment to add the sidecar container:

Bash
1
2
3
kubectl apply -f argocd/install/helmfile-cmp.yaml
kubectl -n argocd patch deployment argocd-repo-server \
  --patch-file argocd/install/repo-server-helmfile-sidecar-patch.yaml

The sidecar patch adds a container running ghcr.io/helmfile/helmfile:v1.1.3 as the CMP server process, with Helm cache directories re-pointed to /tmp so the container can run as a non-root user (runAsNonRoot: true, runAsUser: 999 — the ArgoCD CMP server convention). It also mounts the plugin ConfigMap so the process knows how to handle helmfile sources.

Create repository credentials

ArgoCD needs SSH credentials to pull from your Git repository. Generate a dedicated SSH key pair (do not reuse personal or CI keys) and add the public key to your Git server's authorized keys.

Bash
1
2
3
4
5
6
7
8
9
# The secret name is free-form; ArgoCD finds the secret through the
# argocd.argoproj.io/secret-type=repository label.
kubectl -n argocd create secret generic repo-spog \
  --from-literal=type=git \
  --from-literal=url=ssh://git@your-git-host/spog.git \
  --from-literal=insecure=true \
  --from-file=sshPrivateKey=/path/to/spog-argocd-key
kubectl -n argocd label secret repo-spog \
  argocd.argoproj.io/secret-type=repository

insecure=true is for development environments only

The insecure=true entry shown above skips SSH host-key verification, matching the in-repository development setup (argocd/README.md). This is acceptable in a local development cluster where you control both sides of the connection. Never use it against a production Git server — drop that line and let ArgoCD verify the host key normally.

Deploying the Control Plane (glass-ui)

The glass-ui Application uses ArgoCD's native Helm support. A single Application manifest points at the helm/glass-ui chart path inside the repository:

YAML
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
# argocd/apps/glass-ui.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: glass-ui
  namespace: argocd
spec:
  project: default
  source:
    repoURL: ssh://git@your-git-host/spog.git
    targetRevision: main
    path: helm/glass-ui
    helm:
      releaseName: glass-ui
      valueFiles:
        - values-argocd.yaml
  destination:
    server: https://kubernetes.default.svc
    namespace: controlplane
  syncPolicy:
    syncOptions:
      - ServerSideApply=true

ServerSideApply=true is recommended for glass-ui because several of its resources carry large annotations; server-side apply avoids the kubectl.kubernetes.io/last-applied-configuration annotation size limit.

Check targetRevision before applying the in-repository manifests

The manifests under argocd/apps/ in the repository target the development environment and may pin targetRevision to a feature branch and repoURL to a development Git host. Update both to your production repository and branch (for example main or a release tag) before applying.

This Application must be single-source

Notice that valueFiles references values-argocd.yaml, a file that lives inside the chart directory (helm/glass-ui/values-argocd.yaml), not a separate $values repository source. This is intentional.

ArgoCD silently ignores Sync hooks (such as the secret generator jobs) for multi-source Applications. If you split the chart source and the values source into two sources: entries, the argocd.argoproj.io/hook annotations will have no effect and the generator jobs will be tracked as ordinary resources — meaning TTL cleanup will cause persistent OutOfSync drift and every sync will re-run the generators unconditionally.

Merge all ArgoCD-specific overrides into helm/glass-ui/values-argocd.yaml inside the chart directory to keep the Application single-source.

Apply the Application:

Bash
1
kubectl apply -f argocd/apps/glass-ui.yaml

Deploying the Instrumentation Agents (glass-instrumentation)

With the plain Helm strategy, each userplane cluster gets its own ArgoCD Application. Each Application points directly at the helm/glass-instrumentation chart and supplies cluster-specific values:

YAML
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
# Example: argocd/apps/glass-instrumentation-auth-eu-east.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: glass-instrumentation-auth-eu-east
  namespace: argocd
spec:
  project: default
  source:
    repoURL: ssh://git@your-git-host/spog.git
    targetRevision: main
    path: helm/glass-instrumentation
    helm:
      releaseName: glass-instrumentation
      valuesObject:
        clusterId: "auth-eu-east"
        labels:
          region: "eu-east"
          role: "authoritative"
  destination:
    server: https://kubernetes.default.svc
    namespace: auth-eu-east
  syncPolicy:
    syncOptions:
      - ServerSideApply=true

Use valuesObject to inline small per-cluster overrides directly in the Application manifest, or use valueFiles to point at a committed values file such as:

YAML
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
# Example: glass-instrumentation values for a plain-helm ArgoCD Application
#
# Use this variant when you manage a handful of clusters: one ArgoCD
# Application per CloudControl cluster namespace, each with its own small
# values file like this one. For large, uniform fleets prefer the helmfile
# variant (see the "Deploying SPOG via ArgoCD" article).

# Identifies the cluster this instrumentation announces itself as
clusterId: "auth-eu-east"

# Labels drive dashboards, filtering and REGO permissions
labels:
  region: "eu-east"
  role: "authoritative"
  environment: "production"

# Image source for all instrumentation services
images:
  registry: "cloudcontrol-registry.local:5000"
  project: "spog"
  pullPolicy: "IfNotPresent"

Repeat the Application manifest for every userplane cluster, adjusting metadata.name, destination.namespace, and the values.

With the helmfile strategy, a single Application manages every instrumentation release. The helmfile CMP renders argocd/glass-instrumentation/helmfile.yaml.gotmpl, which fans out one Helm release per CloudControl namespace:

YAML
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
# argocd/apps/glass-instrumentation.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: glass-instrumentation
  namespace: argocd
spec:
  project: default
  source:
    repoURL: ssh://git@your-git-host/spog.git
    targetRevision: main
    path: argocd/glass-instrumentation
    plugin:
      name: helmfile-v1.0
  # No destination.namespace: every chart resource sets an explicit
  # metadata.namespace, and a resource missing one should fail the sync
  # loudly instead of silently landing in a fallback namespace.
  destination:
    server: https://kubernetes.default.svc
  syncPolicy:
    syncOptions:
      - ServerSideApply=true

The Application tile shows no namespace — that is expected

The ArgoCD UI displays destination.namespace on the Application card. For this fleet Application the field is intentionally unset because resources land in many namespaces at once; use the Application's resource tree (or argocd app resources glass-instrumentation) to see the actual per-namespace spread.

The helmfile template that drives this Application fans out across all region and recursor namespaces:

YAML
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
# argocd/glass-instrumentation/helmfile.yaml.gotmpl (excerpt)
{{- $regions := env "ARGOCD_ENV_REGIONS" | default "eu-east,eu-west,us-east,us-west" | splitList "," }}
{{- $recursorsPerRegion := env "ARGOCD_ENV_RECURSORS_PER_REGION" | default "2" | atoi }}

releases:
  {{- range $region := $regions }}
  - name: glass-instrumentation-auth-{{ $region }}
    namespace: auth-{{ $region }}
    chart: ../../helm/glass-instrumentation
    createNamespace: false
    values:
      - ../values/glass-instrumentation-images.yaml
      - ../../helmfiles/helm-glass/values/glass-instrumentation.yaml
      - clusterId: "auth-{{ $region }}"
      - labels:
          region: {{ $region }}
          role: "authoritative"
  {{- end }}
  # ... recursor releases follow the same pattern

Every rendered resource carries an explicit metadata.namespace, so despite the single Application destination, resources land in the correct namespaces.

Customising the region topology without modifying the helmfile directly: set plugin env vars on the Application's spec.source.plugin.env. ArgoCD automatically prefixes each name with ARGOCD_ENV_ before passing it to the CMP process, so you write the short name in the manifest:

YAML
1
2
3
4
5
6
7
    plugin:
      name: helmfile-v1.0
      env:
        - name: REGIONS                  # becomes ARGOCD_ENV_REGIONS inside the CMP
          value: "us-east,us-west,eu-west"
        - name: RECURSORS_PER_REGION     # becomes ARGOCD_ENV_RECURSORS_PER_REGION
          value: "3"

The helmfile template reads them with the Sprig env function (env "ARGOCD_ENV_REGIONS") and falls back to its built-in defaults when the var is absent.

The entire fleet syncs together

A misconfiguration affects all clusters at once, and rollback affects all clusters at once. If some clusters need special handling, manage those with individual plain Helm Applications alongside the fleet.

Applying the Applications

Once ArgoCD is running and credentials are in place, apply the manifests:

Bash
1
2
3
4
5
# Control plane
kubectl apply -f argocd/apps/glass-ui.yaml

# Instrumentation — one Application per cluster:
kubectl apply -f argocd/apps/glass-instrumentation-auth-eu-east.yaml

Watch the sync progress in the ArgoCD UI or via the CLI:

Bash
1
2
argocd app get glass-ui
argocd app get glass-instrumentation-auth-eu-east
Bash
1
2
3
4
5
# Control plane
kubectl apply -f argocd/apps/glass-ui.yaml

# Instrumentation — one Application for the fleet:
kubectl apply -f argocd/apps/glass-instrumentation.yaml

Watch the sync progress in the ArgoCD UI or via the CLI:

Bash
1
2
argocd app get glass-ui
argocd app get glass-instrumentation

The first sync runs the secret generator jobs. After they complete and are cleaned up by TTL, the Applications should show Synced and Healthy.