Self-Hosted k3s on Azure - Part 8: Rolling Out Apps

This is Part 8 of the Self-Hosted k3s on Azure series. With GitOps, TLS, secrets, storage, and SSO all in place, adding an app becomes a rhythm. This post rolls out a few and pulls out the patterns worth reusing.

Placeholders: example.com for the domain.

What’s actually running

By now the cluster runs a full set. Split by role, infrastructure (built in earlier parts) and the apps themselves:

• Infrastructure: sealed-secrets (Part 5); cert-manager, the DNS-01 webhook, and Traefik config (Part 4); Argo CD managing itself (Part 3); LLDAP and Authelia (Part 7).
• Apps: a demo static site (sample-web, the running example throughout the series), IT-Tools, Miniflux, Vaultwarden, Headlamp, and a small custom Go service of my own (covered below).

Everything below is an app; the patterns are what make adding the next one cheap.

Two templates cover almost everything

Most apps fall into one of two shapes:

• Stateless static site (a base with deployment, service, ingress, and a per-cluster overlay patching the hostname). It gets HTTPS for free from the wildcard cert (Part 4) and 2FA for free from forward-auth (Part 7). No database, no secret, nothing to back up.
• Shared-Postgres app (the same, plus a SealedSecret holding a connection string, plus a one-time CREATE DATABASE / CREATE ROLE / GRANT on the shared instance from Part 6).

A new static app is: copy the folder, change the name and image, patch the hostname:

# overlays/prod/kustomization.yaml (the only per-cluster file)
namespace: it-tools
resources: [../../base]
patches:
  - target: { kind: Ingress, name: it-tools }
    patch: |-
      - op: replace
        path: /spec/rules/0/host
        value: tools.example.com
images:
  - name: corentinth/it-tools
    newTag: <pinned-version>     # pin a real tag, not latest

Headlamp: delegate authorization to the API server

Headlamp is a Kubernetes web UI. The interesting decision is authentication. It sits behind the global forward-auth already, so only an authenticated user reaches it. I let it run with the pod’s ServiceAccount and skip its own login screen:

config:
  unsafeUseServiceAccountToken: true   # safe ONLY because forward-auth gates it
serviceAccount: { create: true }
clusterRoleBinding: { create: true, clusterRoleName: cluster-admin }

Headlamp has no RBAC of its own, by design. Kubernetes authorization only ever happens in the API server. Headlamp is a UI proxy; whatever it can do is decided by the token it presents to the API server. In this mode every user shares the pod’s ServiceAccount, so it is effectively single-admin (fine for me). Real per-user RBAC would mean giving Headlamp OIDC and configuring the API server to trust the same issuer, so it authorizes each user’s token individually. The flag is literally named unsafe because it is only safe with an auth proxy in front; never expose it without one.

Vaultwarden: stateless, without logging users out

Vaultwarden (a Bitwarden-compatible server) is the app I most wanted to be both stateless and durable. Two things make it interesting.

First, it must bypass forward-auth, because the phone apps and browser extensions are machine clients hitting /api and /identity and cannot complete a 2FA page. So the domain is bypassed at the proxy, and authentication is Vaultwarden’s own master password (I gate only /admin with 2FA). The vault is end-to-end encrypted anyway, so the server only ever holds ciphertext.

Second, the session keys. Vaultwarden signs session JWTs with an RSA key it stores in its data folder. With no persistent volume, that key regenerates on every pod restart and logs everyone out. But I do not want a node-pinned PVC (Part 6). The solution: put the RSA key in the SealedSecret and seed it into an emptyDir with an init container.

initContainers:
  - name: seed-rsa-key
    image: vaultwarden/server:<pinned>
    command: ["sh", "-c", "cp /seed/rsa_key.pem /data/rsa_key.pem"]
    volumeMounts:
      - { name: rsa-seed, mountPath: /seed }
      - { name: data, mountPath: /data }
volumes:
  - name: data
    emptyDir: {}                      # writable scratch, regenerable
  - name: rsa-seed
    secret:
      secretName: vaultwarden-secret
      items: [ { key: rsa_key.pem, path: rsa_key.pem } ]

Now the key is stable, seeded from Git-stored ciphertext on every start, and the pod has no PVC, so it reschedules to any node. Verified during a full cluster rebuild, the key’s hash is identical before and after, so nobody is logged out even after nuking the cluster. Vault data itself lives in the shared Postgres, so disaster recovery is just Postgres plus the master key. To keep the data folder free of anything irreplaceable, I also disable attachments (USER_ATTACHMENT_LIMIT=0).

Miniflux: OIDC login and an Azure extension gotcha

Miniflux (a minimalist RSS reader) is Postgres-only, which suits the shared instance, and it speaks OIDC, so it plugs straight into Authelia for single sign-on:

env:
  - { name: OAUTH2_PROVIDER, value: oidc }
  - { name: OAUTH2_OIDC_DISCOVERY_ENDPOINT, value: https://auth.example.com }
  - { name: OAUTH2_USER_CREATION, value: "1" }   # first OIDC login becomes admin
  - { name: DATABASE_MAX_CONNS, value: "3" }      # respect the shared connection budget
  - { name: RUN_MIGRATIONS, value: "1" }
  - { name: PORT, value: "8080" }                 # else it binds 127.0.0.1 only

There is an Azure-specific trap. A recent Miniflux migration runs DROP EXTENSION IF EXISTS hstore (it dropped its old hstore dependency). On Azure Postgres, any DDL touching an extension is rejected unless that extension is on the server’s allow-list, even a DROP IF EXISTS on a fresh database that never had it. So migrations fail and Miniflux will not start. Two fixes: allow-list the extension on the shared server (a server-parameter change), or, without touching shared config, skip that one migration by bumping the schema version in the app’s own database:

UPDATE schema_version SET version = '<the failing version>';

On a brand-new database that never had hstore, skipping the drop is a no-op with zero consequences. Miniflux has no config flag to skip a migration; the version bump is the manual escape hatch.

A custom app: when a background worker cannot scale out

Not everything is off-the-shelf. One of the apps is a small Go service I wrote: a date-reminder tool with a web dashboard, an iCal feed you can subscribe to, and a background scheduler that sends push notifications. It surfaces a design constraint that off-the-shelf web apps usually hide.

The GitOps shape is familiar by now:

• Config in a ConfigMap, secrets in a SealedSecret. The reminder definitions live in a ConfigMap (generated by kustomize, so editing them is just a Git commit); the push credentials are a SealedSecret (Part 5).
• The iCal endpoint bypasses forward-auth. A calendar client fetching /cal.ics is a machine client that cannot complete a 2FA page, so that single path is bypassed at the proxy (the machine-client pattern from Part 7), while the dashboard stays protected.

The third property is the one that matters: it runs a scheduler in-process, which makes it non-horizontally-scalable. The scheduler wakes on an interval and fires due reminders, keeping an in-memory record of what it already sent so it does not double-send. Run two replicas and both timers fire, both miss the other’s in-memory state, and every reminder goes out twice. So the deployment is pinned to a single replica with a Recreate strategy:

spec:
  replicas: 1
  strategy: { type: Recreate }   # a second replica would double-fire the scheduler

It is easy to reach for replicas: 3 without thinking. Any app that combines serving with an in-process periodic job has this split: the web tier scales horizontally, the scheduler does not. The honest options are to accept a single replica (what I did, since a brief gap during a restart is harmless for reminders), or to lift the coordination out of memory: move the “already sent” state into the shared Postgres, or take a leader-election lease so only one replica’s scheduler is active. Both turn a stateful singleton back into something you can run several of. For a personal reminder service, one replica is the right amount of engineering; the point is to make the constraint explicit in the manifest rather than discover it as duplicate notifications.

Backup design for the Postgres apps

Azure’s managed Postgres already does point-in-time recovery, but I want a second, portable copy I control. The pattern is a CronJob that runs pg_dump, gzips, and PUTs the result to blob storage with a write-only SAS token (stored in a SealedSecret). pg_dump does not lock the database (it uses an MVCC snapshot and only conflicts with DDL). These databases are a few MB, so a backup is a couple of seconds of light I/O scheduled off-peak. SQLite apps get the same effect from Litestream (Part 6). PITR handles “roll back to a moment,” the dump handles “an independent copy I can download.”

What’s next

The system is complete and full of apps. Part 9, the finale, is about resilience and economics: destroying and rebuilding the whole thing, moving it across regions, and reading and trimming the Azure bill.