How a Portal Click Reimaged My k3s Cluster - A Postmortem

This is a companion story to the Self-Hosted k3s on Azure series. It is the incident that convinced me to make the whole cluster reimage-safe and provisioned as code. If you run k3s (or anything stateful) on an Azure VM Scale Set, this one is worth your time.

Names and versions are generic; the mechanism is exactly what happened.

The setup

Three Azure VMs in a VM Scale Set, running k3s as HA servers with embedded etcd. I had built them by hand over an afternoon, in the older Uniform orchestration mode (the reimage-safe rebuild in Part 2 later moves to Flexible). They had been up for about ten days. The image reference in the scale set model was, unthinkingly, version: latest.

The click that broke it

I wanted the three nodes behind a load balancer, so I did the obvious thing: opened the portal, went to the load balancer, and used the “add to backend pool” wizard to attach the scale set.

The wizard finished. Then SSH stopped working. New host keys. uptime reset to seconds. /etc/machine-id had changed. All three nodes had been reimaged: their OS disks rebuilt from the base image. etcd was gone on all three at once. The cluster only came back because every manifest was in Git and I could bootstrap a fresh control plane and re-sync.

Nothing in that wizard said “this will reimage your nodes.”

What actually happened

The “add to backend pool” wizard is not an incremental change. Under the hood it submits a scale-set update that does a full PUT of the entire VM profile, and that profile includes imageReference: version: latest.

Two things then combine:

1. A full PUT re-resolves latest. Azure evaluates latest at apply time to the newest available build. My instances were created ten days earlier against an older build, so now the model’s image and the running image differ.
2. The rolling upgrade policy applies model differences automatically, one instance at a time. And the way an image-version difference gets applied is by reimage: rebuild the OS disk from the new image.

So a change I thought was “attach a NIC to a backend pool” was really “rewrite the whole VM profile, re-resolve the image to something newer, and roll that out by wiping every OS disk.”

Reproducing it, both directions

I did not want to believe it, so I reproduced it deliberately:

# pin to a specific (different) build
az vmss update -g rg-cluster -n vmss-cluster --set virtualMachineProfile.storageProfile.imageReference.version=<older-build>

Under a rolling policy this command hangs (it is driving a rolling upgrade), and when it finishes the instances have new SSH host keys and a new exactVersion. That is a reimage. Changing the pinned version back to latest reproduces it again. Both directions confirm the mechanism: any change to the resolved image version is applied by reimaging.

The control group is just as important. Incremental changes that do not touch imageReference (adding to a backend pool via CLI --set/--add, editing tags, update-instances) return instantly and do not reimage. I also falsified one red herring: flipping diskControllerType from null to SCSI does not cause it.

Why “Manual” was not enough on its own

The intuitive fix is to set the upgrade policy to Manual so Azure stops auto-applying model changes. I tried it. It is necessary but not sufficient.

With Manual, the model change does not roll out automatically, but the moment you run update-instances (which you must, eventually, for a legitimate change like adding a backend pool at creation time), Azure applies the whole model difference to those instances. If the model still says latest and it has drifted from the running build, that update-instances reimages them.

So Manual only delays the trigger. The thing that actually removes the danger is pinning the image to an exact version, so there is no image difference to apply, ever.

How to tell a reimage happened

Handy because none of it needs your application data:

• uptime -s (boot time) jumps to now.
• /etc/machine-id changes.
• SSH host key changes (you get the scary mismatch warning).
• Any marker file you dropped on the OS disk is gone.
• az vm show ... storageProfile.imageReference.exactVersion shows the build the instance is actually running, which changes only on reimage.

What is actually safe

The reassuring flip side: normal OS patching does not reimage. Running apt upgrade inside the VM, or Azure’s AutomaticByPlatform patching, installs packages in place on the existing OS disk. They do not change exactVersion and they do not touch etcd. exactVersion only changes when the OS disk is rebuilt (a reimage). Day-to-day patching is fine; it is the scale-set model image resolution that bites.

The fix (four changes)

I rebuilt the cluster as code (Bicep + cloud-init, covered in Part 2 of the series) with four protections:

1. Pin imageReference.version to an exact build, never latest. This is the one that actually matters. No image difference means no reimage, no matter what triggers an update-instances.
2. upgradePolicy: Manual. A useful backstop so nothing rolls out on its own.
3. Keep etcd on a separate data disk, mounted by label and formatted only when blank, with periodic etcd snapshots uploaded to blob storage. Now even a reimage does not lose state.
4. Attach the scale set to the backend pool at creation time, in the template, so I never need the portal’s full-PUT wizard on a running, stateful scale set.

Verified: it now self-heals

With those in place, I deliberately reimaged one node. The OS disk was rebuilt, but the data disk (with etcd) survived because it is mounted by label and never reformatted. cloud-init detected the existing etcd database and took the rejoin path. The node came back with its state, and the other two held quorum throughout. The exact failure that started this is now a non-event.

Takeaways

• latest in a scale-set image reference is unsafe. A full-profile update re-resolves it and rolls it out by reimaging. Pin exact versions.
• The portal’s convenience wizards can do full-profile PUTs. On stateful scale sets, prefer incremental CLI changes, or better, define everything in a template so you never click.
• Treat nodes as disposable and keep real state elsewhere (a separate data disk, an external database, Git). Then a reimage is an inconvenience, not a disaster.
• Reproduce before you conclude. Bidirectional reproduction plus a control group turned “I think the wizard did it” into “here is exactly the mechanism.”

The rest of the story (how the reimage-safe cluster is actually built, and how the whole thing is recovered from scratch) is in the series, Parts 2 and 9.