Self-Hosted k3s on Azure - Part 1: Architecture & Why GitOps

This is Part 1 of the Self-Hosted k3s on Azure series, where I document how I run my own apps (password manager, RSS reader, dashboards, and more) on a small, highly-available Kubernetes cluster that I can destroy and rebuild in half an hour.

Throughout the series I use example.com as a stand-in for my real domain, and placeholders like <public-ip> / <subscription-id> for anything sensitive. Everything else (the manifests, the Bicep, the scripts) is real, though snippets are trimmed to the part that matters. This series is a design walkthrough of the reasoning and the gotchas behind each choice, not a copy-paste tutorial.

What I was after

A few goals shaped every decision:

• Self-hosted, but not hand-fed. I wanted real apps (Vaultwarden, Miniflux, Headlamp, …) with automatic HTTPS and single sign-on, not a pile of docker run commands I’d forget in a month.
• Fully declarative. The entire system (infrastructure and apps) lives in Git. If a node dies, if I want to move regions, or if I break something, I can rebuild from scratch and get an identical result.
• Cheap. This is a personal cluster. It runs on burstable VMs and a burstable managed database, and the whole thing costs about 130 USD/month (I break that number down and trim it in Part 9 and Part 12).
• HA where it’s cheap, disposable everywhere else. Three control-plane nodes with embedded etcd for real high availability, but zero precious state on any node; the cluster itself is cattle.

The high-level architecture

flowchart TB
    user[Browser / clients]
    gh[GitHub<br/>GitOps repo]

    subgraph azure[Azure]
        subgraph rgshared[rg-shared - outlives any cluster]
            pip[Public IP<br/>fixed VIP]
            kv[Key Vault<br/>k3s join token]
            uami[Managed Identity<br/>UAMI]
            st[(Storage<br/>etcd snapshots + lease lock)]
        end
        subgraph rgcluster[rg-cluster - disposable]
            lb[Standard Load Balancer]
            subgraph cluster[k3s HA - 3 nodes, no public IP]
                traefik[Traefik ingress<br/>wildcard TLS]
                argo[Argo CD]
                sso[Authelia + LLDAP<br/>SSO]
                apps[apps: Vaultwarden,<br/>Miniflux, Headlamp, ...]
                vmagent[vmagent<br/>metrics collector]
            end
        end
        pg[(Azure Postgres<br/>Flexible Server)]
    end
    gc[Grafana Cloud<br/>metrics + dashboards]

    user -->|443 / 6443 / SSH| lb
    lb -. claims .-> pip
    lb --> traefik
    traefik --> apps
    traefik --> argo
    traefik --> sso
    argo -->|pulls desired state| gh
    cluster -->|token + snapshots| kv
    cluster --> st
    cluster -.->|node identity| uami
    apps -->|app data| pg
    sso -->|users / sessions| pg
    vmagent -->|remote_write| gc

The moving parts:

• Compute: 3 Azure VMs running k3s as servers with the embedded etcd datastore, a proper HA control plane. Nodes have no public IP.
• One Standard Load Balancer fronts everything: 6443 (Kubernetes API), 80/443 (web via Traefik), SSH (inbound NAT per node), and outbound egress. It owns a single fixed public IP (the cluster VIP) that survives cluster rebuilds because it lives in a separate resource group (more on that below).
• Ingress: Traefik (ships with k3s) terminates TLS at the edge and routes by hostname. A single wildcard certificate (*.example.com) means a new app needs zero certificate configuration.
• GitOps engine: Argo CD watches a Git repo and continuously reconciles the cluster to match it.
• State lives outside the cluster: a managed Azure Postgres Flexible Server holds app data (users, sessions, feeds, …). The cluster nodes stay stateless and disposable.
• Identity: LLDAP (a tiny user directory) + Authelia (2FA forward-auth and an OIDC provider) give every app single sign-on.
• Observability: a light collector (vmagent, node-exporter, kube-state-metrics) runs in the cluster and ships metrics to Grafana Cloud, so no time-series database or Grafana competes with the apps for memory.
• Scale-to-zero: idle interactive apps are scaled to zero by Sablier and woken on the first request, so they hold no memory when nobody is using them.

I’ll build each of these up over the series. This post is the map.

Why GitOps

The core idea: Git is the single source of truth for the desired state of the whole system. You don’t kubectl apply things by hand; you commit YAML, and a controller (Argo CD) makes reality match Git, continuously.

What that buys you:

• Reproducibility. The cluster is a function of the repo. Rebuild = re-point Argo at the repo and everything comes back.
• Auditability. Every change is a commit. git log is your change history; git revert is your rollback.
• Self-healing. If something drifts (a manual change, a crashed controller recreating an object), Argo reverts it to what Git says.
• No snowflakes. There’s no “that one setting I changed on the server and forgot.” If it’s not in Git, it doesn’t exist.

Two Argo features do the heavy lifting:

• App-of-Apps. One root Application points at a folder of child Applications. Apply the root once and Argo discovers and creates everything else. Bootstrapping the whole cluster is a single kubectl apply.
• AppProject. A permission boundary: which Git repos and which cluster namespaces a group of apps is allowed to touch. It keeps a runaway manifest from doing something it shouldn’t.

Two repositories

I split things into two kinds of repo:

• The GitOps repo: all declarative manifests: app kustomizations, cluster infrastructure, Argo CD Applications, and the Azure IaC. This is what Argo watches. It contains no application source code.
• App repos: each application’s source code + Dockerfile + CI. CI builds and pushes a container image; the GitOps repo pins the exact image tag.

Why separate? The GitOps repo changes when deployment changes; app repos change when code changes. Keeping them apart means a code push doesn’t churn your deployment history, and your cluster’s desired state has one clean home.

The GitOps repo layout

apps/                         # per-app Kustomize
  <app>/
    base/                     # deployment, service, ingress, kustomization
    overlays/<cluster>/       # per-cluster patches (hostname, replicas, image tag)
infra/                        # cluster infrastructure (directory-synced apps)
  lldap/ authelia-config/ cert-manager-config/ ...
argocd/
  applications/<cluster>/     # one Argo Application per app/infra component
  projects/                   # AppProject (the permission boundary)
  bootstrap/<cluster>.yaml    # the App-of-Apps root
provisioning/<cluster>/       # Azure IaC: bicep + cloud-init + deploy.ps1 + runbook

Two conventions I lean on everywhere:

• Kustomize base + overlay. The base/ holds cluster-agnostic manifests with placeholder hostnames; each overlays/<cluster>/ patches in the real hostname, replica count, and image tag. Adding a second cluster later is “copy the overlay, change the patches.”
• One Argo Application per component, all living under argocd/applications/<cluster>/, all discovered by the App-of-Apps root.

A typical app Application is tiny; it just points Argo at an overlay path:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: sample-web
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: homelab
  source:
    repoURL: https://github.com/example/gitops.git
    targetRevision: master
    path: apps/sample-web/overlays/prod   # the overlay, not the base
  destination:
    server: https://kubernetes.default.svc
    namespace: sample-web-prod
  syncPolicy:
    automated:
      prune: true       # remove from cluster what's removed from Git
      selfHeal: true    # revert manual drift back to Git
    syncOptions:
      - CreateNamespace=true

Choosing the ingress / HTTPS approach

Getting HTTPS traffic to the cluster has a few common options. I weighed:

Approach	Verdict for me
Cloud-managed ingress controller (per-service cloud LB)	A public IP per service gets expensive and hard to manage
NodePort + DNS to nodes	Nodes would need public IPs; ugly ports; no clean TLS
Traefik (built into k3s) behind one LB	✅ One VIP, host-based routing, TLS at the edge
Cloudflare Tunnel	Nice for zero open ports, but adds a dependency; kept as an optional edge later

The decision: one Standard Load Balancer → Traefik → services, plus a wildcard certificate so every new subdomain (app.example.com) works with no per-app certificate wiring. The nodes never get a public IP; the only public surface is the LB’s single VIP. Cloudflare’s free proxy can sit in front later for edge caching/DDoS, but it’s optional.

Design principles that recur

These show up again and again in later parts, so I’ll state them once:

1. Infrastructure as Code, reimage-safe. The cluster is provisioned by Bicep + cloud-init (Part 2). Any node can be reimaged, and the whole cluster can be destroyed and rebuilt, without losing anything that matters.
2. State lives outside the cluster. App data goes to managed Postgres; the one truly irreplaceable local secret (the SealedSecrets master key) is backed up offline. Everything else is reconstructable from Git. This is what makes the cluster disposable.
3. Least privilege. Dedicated service accounts, scoped AppProjects, read-only Git tokens, a read-only LDAP bind user. Each component gets exactly what it needs.
4. Cost-conscious by default. Burstable compute, a burstable database, a single small cluster. I measure real cost and trim it in Parts 9 and 12, and reclaim memory from idle apps in Part 13 instead of buying another node.

The series

This is a bottom-up build. Each part is self-contained, but they stack:

1. Architecture & Why GitOps (this post): the map and the principles.
2. Provisioning with Bicep & cloud-init: the single-LB network, burstable VMs, and a cloud-init that bootstraps a k3s HA cluster with state on a separate data disk so reimaging is safe (Part 12 later folds that disk into a single Premium OS disk).
3. GitOps with Argo CD: installing Argo, the App-of-Apps bootstrap, Kustomize overlays, private image pulls, and having Argo manage itself.
4. Wildcard TLS & a custom DNS-01 webhook: one wildcard cert for every subdomain, plus writing a cert-manager webhook when your DNS provider has no built-in solver.
5. Committing secrets with SealedSecrets: how to put encrypted secrets in Git, and the one key you must back up.
6. Stateful apps & external Postgres: why I keep state out of the cluster, one database per app, and the storage options I considered.
7. Single sign-on with LLDAP & Authelia: a user directory plus forward-auth and OIDC, so every app shares one login.
8. Rolling out apps: Headlamp, Vaultwarden, and Miniflux, and the patterns that make new apps a copy-paste.
9. DR, migration & cost control: destroying and rebuilding the whole thing, moving regions, and keeping the bill in check.
10. Observability with VictoriaMetrics: self-hosting a metrics pipeline on k3s, adapting to a single-process control plane, and cutting ingest by ninety percent.
11. Grafana Cloud & right-sizing: moving storage and dashboards off the cluster to a managed backend, and trimming CPU throttling and an unbounded controller heap.
12. Trimming memory, disk & a reserved-instance dead end: reading the Azure bill precisely, a disk-billing reversal, bounding the k3s heap, and why reserved instances were a dead end here.
13. Scale-to-zero for idle apps: reclaiming memory by letting Sablier scale idle interactive apps to zero, and making Traefik and Argo CD cooperate with a controller that owns the replica count.

There is also a companion postmortem on how a single portal click reimaged all three nodes of my first, hand-built cluster; it is the reason the provisioning in Part 2 looks the way it does.

If you want the fastest mental model to carry through all of it: Git describes the system, Argo makes it real, and the cluster underneath is disposable.

Next up is Part 2, where the cluster actually gets built.