Self-Hosted k3s on Azure - Part 3: GitOps with Argo CD

This is Part 3 of the Self-Hosted k3s on Azure series. Part 2 made the cluster itself a command. This post makes the contents of the cluster a command too, with Argo CD reconciling everything from Git.

Placeholders: example.com for the domain, example/gitops for the repo.

Pull mode: one Argo CD per cluster

Argo CD can run in two shapes: one central instance pushing to many clusters, or one instance per cluster pulling from Git. I use pull mode: each cluster runs its own Argo CD that watches the same GitOps repo and only deploys to itself. The payoff shows up the day you add a cluster at home behind NAT: it just needs outbound access to GitHub, and a central controller never has to reach into your LAN.

push code (app repo)
  -> CI builds image, pushes to registry
  -> CI bumps the image tag in the GitOps repo
  -> each cluster's Argo CD sees the GitOps change
  -> Argo syncs it to that cluster

Installing Argo CD (use server-side apply)

Install into its own namespace, and use --server-side from the very first apply:

kubectl create namespace argocd
kubectl apply -n argocd --server-side -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Why server-side matters: the client-side apply stuffs the whole manifest into a last-applied-configuration annotation, and the applicationsets CRD is larger than the 256 KB annotation limit, so it errors out. Server-side apply lets the API server do the merge without that annotation.

The related gotcha: if you start client-side and later switch to server-side, you hit a field-manager conflict (the old kubectl-client-side-apply manager versus the new server-side one). The clean fix is to delete and reinstall server-side from the start, so there is no legacy manager to conflict with. Start as you mean to continue.

Connecting the private GitOps repo

Argo needs credentials to read a private repo. Create a fine-grained token scoped to just that repo with Contents: Read-only, then connect it. One non-obvious failure mode: if the repo is connected under the default project but the apps live in another, Argo reports authentication required: Repository not found. The repo credential is project-scoped and must match the apps’ project:

kubectl create secret generic repo-gitops -n argocd `
  --from-literal=type=git `
  --from-literal=url=https://github.com/example/gitops.git `
  --from-literal=username=<user> `
  --from-literal=password='<read-only PAT>' `
  --from-literal=project=homelab
kubectl label secret repo-gitops -n argocd argocd.argoproj.io/secret-type=repository

AppProject: the permission boundary

An AppProject says which repos a group of apps may pull from and which cluster and namespaces they may deploy to. It is the seatbelt that stops a bad manifest from doing something out of scope:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: homelab
  namespace: argocd
spec:
  sourceRepos:
    - https://github.com/example/gitops.git
    - https://charts.jetstack.io          # allow-listed Helm repos
    - https://charts.authelia.com
  destinations:
    - server: https://kubernetes.default.svc
      namespace: '*'

App-of-Apps: bootstrap the whole cluster with one apply

A single root Application watches the argocd/applications/<cluster>/ folder and creates every child Application it finds. Adding an app later is just committing a file into that folder.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root-cluster
  namespace: argocd
spec:
  project: homelab
  source:
    repoURL: https://github.com/example/gitops.git
    targetRevision: master
    path: argocd/applications/prod
    directory:
      recurse: true
  destination: { server: https://kubernetes.default.svc, namespace: argocd }
  syncPolicy:
    automated: { prune: true, selfHeal: true }

Bootstrap order matters, and there is a chicken-and-egg wrinkle:

kubectl apply -f argocd/projects/homelab.yaml         # 1) the permission boundary first
kubectl apply -f argocd/bootstrap/prod.yaml           # 2) then the root app

The AppProject is applied by hand, not through GitOps. It cannot be managed by an app that lives inside it (the root app depends on the project existing before it is allowed to sync anything). So the rule I keep in my head is: projects are applied manually, applications sync automatically. When you add a new Helm repo to sourceRepos, remember to re-apply the project or Argo will refuse the new source with repository not permitted.

Kustomize base plus overlay

Each app has a cluster-agnostic base/ and a per-cluster overlays/<cluster>/ that patches in the differences. Small deep edits (a hostname, a replica count) are cleanest as inline JSON6902 patches:

# overlays/prod/kustomization.yaml
namespace: sample-web-prod
resources:
  - ../../base
patches:
  - target: { kind: Deployment, name: sample-web }
    patch: |-
      - op: replace
        path: /spec/replicas
        value: 3
  - target: { kind: Ingress, name: sample-web }
    patch: |-
      - op: replace
        path: /spec/rules/0/host
        value: sample.example.com
images:
  - name: ghcr.io/example/sample-web
    newTag: <commit-sha>          # CI writes this

Argo runs kustomize itself; you just point the Application at the overlay path.

CI: build the image, bump the tag

The app repo’s CI has two jobs. The first builds and pushes the image (tagged with the commit SHA). The second checks out the GitOps repo and runs kustomize edit set image to bump the tag, then commits. A nice property: the CI loops over every directory under overlays/, so which clusters an app deploys to is decided by which overlays exist, not by editing the workflow. Add a cluster by adding an overlay.

The token the CI uses to write the GitOps repo is a separate fine-grained token with Contents: Read and write (distinct from Argo’s read-only one).

Private image pulls

The first deploy of a private image gives ImagePullBackOff because the cluster has no registry credentials. There are a few options, roughly in order of blast radius:

Option	Scope	Notes
Make the image public	n/a	simplest, fine for non-secret images
imagePullSecret per namespace	one namespace	explicit, repetitive
registries.yaml baked into cloud-init	whole cluster	one place, but a node-level change
reflector controller	all namespaces	copies one secret everywhere

For a fully-GitOps approach, put the registry dockerconfigjson in a SealedSecret (Part 5) and reference it as an imagePullSecret. For cluster-wide with no per-app config, bake registries.yaml into cloud-init. I started with public images to get moving and tightened later.

Argo CD managing itself

The final piece: Argo manages its own installation, so upgrading Argo is a version bump in Git.

There is a bootstrap paradox: to have “Argo manage Argo,” you first need an Argo running, so the first install is always manual (Part 2’s runbook does the helm install). After that, an argocd Application points at the Argo Helm chart, and the freshly installed Argo starts reconciling itself. Self-management is not installing twice; it is the running instance adopting itself and staying aligned with Git.

Two things I learned doing this:

• Bootstrap and self-management must use the same install method. I first installed Argo from raw manifests and then tried to adopt it with the Helm chart. That produced a stream of ownership conflicts (a redis ServiceAccount field owned by the old Deployment, cluster-scoped CRDs and ClusterRoles with no Helm ownership metadata). The clean answer is to install via Helm from the start, so the self-management chart matches what is already there.
• Run Argo in insecure mode behind Traefik. The chart serves HTTPS by default. Put that behind a TLS-terminating ingress and you get the classic redirect loop: Traefik forwards HTTP, argocd-server redirects to HTTPS, forever. The fix is server.insecure: true so argocd-server speaks plain HTTP and Traefik terminates TLS once at the edge. This “no double TLS” rule comes up again in Part 4.

# the argocd self-management Application (Helm values, trimmed)
configs:
  params:
    server.insecure: true
server:
  ingress:
    enabled: true
    ingressClassName: traefik
    hostname: argocd.example.com

Verifying a sync

kubectl get applications -n argocd          # each should be Synced / Healthy
kubectl get all -n sample-web-prod

To skip the poll interval and sync immediately, nudge the Application object:

kubectl annotate application root-cluster -n argocd argocd.argoproj.io/refresh=hard --overwrite

Argo consumes and removes that annotation; it is a one-shot trigger. normal re-reads Git and diffs; hard also invalidates the repo-server cache.

What’s next

Part 4 gives every app HTTPS: one wildcard certificate for the whole domain, and a custom cert-manager DNS-01 webhook for a DNS provider that has no built-in solver, plus making Traefik itself highly available.