Self-Hosted k3s on Azure - Part 9: DR, Migration & Cost Control

This is Part 9, the finale of the Self-Hosted k3s on Azure series. Everything so far led to one property: the cluster is disposable. This post puts that to use: disaster recovery, a region move, and getting the bill under control.

Placeholders: example.com for the domain, <public-ip> for the VIP, generic resource-group names.

The disaster-recovery model: one precious key

The whole design (state in external Postgres from Part 6, secrets sealed in Git from Part 5, infrastructure as code from Part 2) collapses disaster recovery to a single question: what cannot be reconstructed?

Thing	Where it lives	Lost on rebuild?	How it comes back
all manifests and IaC	Git	no	clone
all secret ciphertext	Git (sealed)	no	decrypted by the master key
SealedSecrets master key	cluster, not in Git	yes, and unrecoverable	offline backup only
app data (users, sessions, feeds)	external Postgres	no	Azure-managed, PITR
TOTP encryption key	Git (sealed)	no	master key (kept verbatim, or all 2FA breaks)
wildcard cert	cluster Secret	yes	cert-manager re-issues automatically
etcd / node disks	nodes	yes	fresh bootstrap; state is elsewhere

So the conclusion is blunt: as long as the SealedSecrets master key has an offline backup, a rebuild needs no other backup. That the cluster can be treated as cattle is exactly the lesson from the reimage postmortem that kicked off this whole redesign.

The rebuild runbook

Rebuilding is eight steps, and the ordering has one hard rule (restore the master key before Argo installs the sealed-secrets controller, or it generates a new key and every sealed secret becomes undecryptable):

# 1. Backup (from the live cluster) - the master key is the one that matters
kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key=active -o yaml `
  | Out-File "$Vault\sealed-secrets-master.key.yaml" -Encoding ascii

# 2. Destroy compute (shared RG with Postgres + VIP is untouched)
az group delete -n rg-cluster --yes

# 3. Deploy fresh (Bicep + cloud-init from Part 2)
.\deploy.ps1 -ResourceGroup rg-cluster -FreshCluster

# 4. Fetch kubeconfig (SAN includes the VIP)
#    ssh to a node, rewrite 127.0.0.1 -> <public-ip>, save to ~/.kube/config

# 5. Restore the master key BEFORE installing Argo
kubectl apply -f "$Vault\sealed-secrets-master.key.yaml"

# 6. Install Argo, connect the repo, bootstrap
helm install argocd argo/argo-cd -n argocd --create-namespace
kubectl apply -f argocd/projects/homelab.yaml
kubectl apply -f argocd/bootstrap/prod.yaml

# 7. Verify
kubectl get applications -n argocd     # all Synced / Healthy
kubectl get sealedsecret -A            # all decrypted

# 8. DNS only if the VIP changed - it does not, on an in-place rebuild

The VIP does not change because the public IP lives in the shared resource group (Part 2), so DNS needs no edit.

Verified twice, including a disk swap

I have run this for real more than once. The most recent was not even a disaster. I wanted to change the node disks from Premium to Standard SSD, and since a Bicep re-run only updates the scale-set model (existing instances keep their disks), the clean way to apply it is a full rebuild. So I rebuilt on purpose. Result: three-node HA back, all disks on the new tier, all 17 Argo Applications Synced/Healthy, every sealed secret decrypted, wildcard cert ready. And the detail I was most curious about: Vaultwarden’s RSA key hash was identical before and after (Part 8), so nobody was logged out despite the entire cluster being destroyed and recreated. Destroying the cluster is now a routine operation, not an emergency.

Cross-region migration

I also moved the whole stack from one region to another. Because everything is code plus external state, migration is just “build the new one beside the old one, verify, switch DNS, delete the old one.” A parallel build with a clean rollback:

1. Build the shared layer in the new region (public IP, key vault, identity, storage).
2. Change the region and shared-RG name in the IaC.
3. Back up the three things (master key, cert, kubeconfig) from the old cluster.
4. Deploy the new cluster (old one still serving).
5. Restore and bootstrap on the new cluster.
6. Switch DNS to the new VIP once verified.
7. Delete the old resource groups after a day of watching.

Two migration-specific gotchas:

• Why move at all: the dev subscription was blocked from provisioning Postgres Flexible Server in the original region (an offer-level regional block, not an Azure Policy, and there is no API to enumerate it, it only fails at create time). Compute capacity was also tighter there. Cost was a wash; the same VM size is the same price across the main regions, so “move to save money” is outdated for compute.
• Key Vault names are globally unique. My preferred name returned AlreadyExists but was not in my deleted-vaults list, meaning another tenant had taken it (a too-guessable name). Purging does not help (you can only purge your own soft-deleted vaults). The fix is just a more distinctive name.

Reading the Azure bill (it is not obvious)

Getting real numbers took some fighting:

• az consumption usage list returns pretaxCost = None on a Visual Studio / credit subscription, so it is useless for cost.
• The costmanagement CLI extension only has export and show-operation-result, no query.
• What works is the Cost Management REST API (the one the portal uses) via az rest, read-only, grouped by resource group or resource:

az rest --method post `
  --url "https://management.azure.com/subscriptions/<sub>/providers/Microsoft.CostManagement/query?api-version=2023-11-01" `
  --body "@query.json"

• For deterministic estimates, the retail prices API (prices.azure.com/api/retail/prices) is public and unauthenticated; filter by region and SKU.

And a trap that fooled me first: historical cost badly understates freshly-created resources. A newly built cluster showed about 2 USD over 31 days, which is nonsense; the resources had not existed for the full period and Cost Management lags a day or more, more for brand-new resources. The right method is a run-rate: take a recent full day’s per-resource cost, or just compute retail price times hours. That gives the honest number, around 145 USD/month for my setup, with the three VMs and the load balancer as the big line items.

Trimming: Premium to Standard SSD disks

The run-rate was a few dollars over my 140 USD target, and the biggest easy win was the disks. Node disks were Premium SSD, which is priced for high-IOPS databases, not for a k3s node’s OS and image cache. Standard SSD is billed by tier and roughly half the price:

Disk	Size	Premium	Standard SSD
3 x OS	30 GB	~$14.4/mo	~$7.2/mo
3 x data	16 GB	~$7.2/mo	~$3.6/mo

That is about 10 USD/month saved, which brought the total to ~133 USD, under target. I checked the data disk was not undersized first: the real usage of /var/lib/rancher (etcd plus the container image cache) was about 2.6 GB out of 16 GB, so 16 GB is comfortable even after many more apps. I did not shrink the OS disk below 30 GB because the Ubuntu image mandates that minimum, and the saving would have been trivial. Applying the change is exactly the “rebuild to change the model” from earlier: flip the disk type in the parameters, destroy, redeploy.

The end

Nine parts in, the shape is simple: Git describes the system, Argo makes it real, external state and one backed-up key make the cluster disposable, and the bill is a hundred and thirty-odd dollars a month. That up-front declarative work pays off concretely: the scariest operations, losing a node, moving regions, changing infrastructure, rebuilding from nothing, are now ordinary, boring, and safe.

Thanks for reading the series.