Self-Hosted k3s on Azure - Part 6: Stateful Apps & External Postgres

This is Part 6 of the Self-Hosted k3s on Azure series. The cluster is disposable by design, which raises the obvious question: where does the data go? This post is the storage strategy.

Pods are ephemeral, and local-path pins them down

A pod can be rescheduled to any node. Its storage needs to survive that. A PersistentVolumeClaim outlives the pod, so a crashed or restarted pod re-attaches and keeps its data. But k3s’s default provisioner, local-path, backs a volume with a directory on one specific node, and pins the pod there with node affinity:

• Same-node restart: data is there.
• That node is gone: data is gone and the pod is stuck Pending (it is only allowed on the dead node).

So local-path gives you persistence across pod restarts, but not across losing a node. For a disposable-cluster design, that is the wrong default for anything precious.

The storage landscape

The general options, for context:

Category	Examples	Idea
Cloud block storage (CSI)	Azure Disk, EBS, GCP PD	volume is a cloud resource, re-attaches when the pod moves
Distributed / replicated	Longhorn, Rook-Ceph, OpenEBS	replicate across nodes, survive node loss
Shared file (RWX)	NFS, Azure Files	many nodes mount it; not for databases
External managed service	Azure Database, RDS, object storage	push state out, keep the cluster stateless
App-level replication	Postgres (CNPG/Patroni), etcd	the app replicates across its own replicas
Backup / restore	Velero, snapshots, dumps	orthogonal safety net, always do it

Comparing the three that were realistic for me:

	local-path	Azure Disk CSI	Longhorn
data location	one node’s disk	Azure-managed disk	replicated across nodes
pod moves with data	no (pinned)	yes (disk re-attaches)	yes (replica elsewhere)
survives node loss	no	yes	yes (replicas >= 2)
cost	tiny	per-disk	in-cluster processes + N x disk

Longhorn deserves a warning: it is not one pod, it is a set of components on every node (manager DaemonSet, an instance-manager per node, CSI plugins, sidecars, a UI). On three nodes it idles around 1 to 1.5 GiB of RAM total. It also reserves a slice of each node’s CPU for the instance-manager by default, and stores N copies of every volume. On 2 vCPU / 4 GiB burstable nodes that is “possible but tight.” For “a few small apps,” it is overkill.

How this cluster maps

• etcd, the most critical state, already uses app-level replication plus backup. Three nodes, Raft replication, snapshots to blob storage. That is the database-HA pattern, not shared storage.
• Everything else follows the same instinct: push state out of the cluster. Which leads to one managed database.

One shared managed Postgres

Surveying the apps I wanted to run, the dependency on Postgres was dense: some require it (Miniflux, Umami), many recommend it (Authelia, Wiki.js, n8n, Gatus, Grafana, Vaultwarden). Bundling a Postgres inside each app would spin up a dozen database instances and eat all the memory. Instead: one shared managed Postgres, with one database and one role per app. A single instance hosts many isolated databases cleanly.

Creating an app’s database follows a fixed recipe, and there is a PG15+ gotcha baked into it:

CREATE ROLE app LOGIN PASSWORD '<alnum-password>';
CREATE DATABASE app OWNER app;
\c app
-- PG15+: the public schema no longer lets everyone CREATE, and on Azure
-- Flexible Server its owner is azure_pg_admin, so the app role gets
-- "permission denied for schema public" on its first table without this:
GRANT USAGE, CREATE ON SCHEMA public TO app;

Two more things I standardized on:

• TLS is mandatory. Azure Postgres Flexible Server enforces require_secure_transport=ON, so a plaintext connection is rejected at the pg_hba layer with a no pg_hba.conf entry ... no encryption error that reads like a firewall problem but is not. Every app must set sslmode=require in its connection string or enable its TLS toggle; watch for Helm charts that default TLS off.
• A small connection budget. The burstable tier caps connections (mine allows 50). Each app’s pool has to be modest; one app defaulted to a pool of 20, which alone would have eaten most of the budget, so I set it to 3. Watch the connection count, not the CPU.

Why no Redis

None of the apps I picked require Redis. The one that tempted me, Authelia, only needs it in HA (for shared sessions). Its session store is code-limited to memory or Redis, with no SQL option, so a single Authelia replica with in-memory sessions is fine. Running a single replica against the shared Postgres makes Authelia stateless and freely reschedulable, which is worth more to me than shared sessions. No Redis, no extra moving part.

The three-way storage strategy

I sort every app into one of three buckets by how precious its data is:

Bucket	Apps	Storage
Precious / needs SQL	Authelia, Vaultwarden, Miniflux, Umami	shared Postgres, one database each
Low-risk SQLite	Uptime Kuma, Memos	SQLite plus Litestream replication to blob
Stateless	IT-Tools, Homepage, Headlamp	no storage at all

The middle bucket is worth explaining. SQLite is an embedded single-file database; there is no server to manage, and no cloud offers a “managed SQLite.” The right place to solve it is the storage layer: Litestream runs as a sidecar and continuously streams the .db file to blob storage, and an init container restores it on startup. The app is unaware, it costs a few MB, and it is effectively free. The catch is an asynchronous RPO of about a second, and single-writer only. That is exactly why anything precious (passwords, 2FA) goes to Postgres instead.

Picking the managed Postgres tier

• Managed over self-hosted (CNPG). I preferred low maintenance. Managed gives automated backups, PITR, patching, and an SLA; the price is money and no hot-standby HA on the cheap tier.
• Burstable B1ms (1 vCore / 2 GiB) is inexpensive, on the order of ten dollars a month including 32 GB of storage. True HA needs the general-purpose tier at roughly ten times the price, which is not worth it for personal use; a single instance with managed PITR is plenty.
• Enable storage autogrow (it only grows, never shrinks), and set retention/cleanup on the chatty apps so they do not balloon it.
• Skip the built-in connection pooler at first; 50 connections is enough. Turn it on if you actually hit the limit, and be aware transaction pooling breaks session features like LISTEN/NOTIFY, so pool per app.

One hard rule I set for myself: the shared data layer (database, key vault, identity, storage, public IP) is shared infrastructure, so any change to it (firewall, parameters, deletion) is done by hand, deliberately, and never by automation.

What’s next

State is now outside the cluster, which is what makes the cluster throwaway. Part 7 builds identity: a user directory and single sign-on so every app shares one login, with both forward-auth and OIDC.