Self-Hosted k3s on Azure - Part 2: Provisioning with Bicep & cloud-init

This is Part 2 of the Self-Hosted k3s on Azure series. Part 1 laid out the architecture and the “the cluster is disposable” principle. This post makes that real: the entire cluster (network, load balancer, VMs, and node bootstrap) is provisioned by Bicep + cloud-init, so I can destroy and recreate it on demand and get the exact same thing back.

Placeholders as always: <public-ip> for the VIP, generic names like rg-cluster / rg-shared / kv-shared for resources. Region westus3 is real.

Why IaC instead of clicking around the portal

I originally hand-built the three nodes. That ended badly (I wrote up the full postmortem separately), but the short version: a routine change in the portal triggered a rolling reimage of all three VMs and wiped their OS disks. The cluster survived only because everything was in Git.

The lesson wasn’t “be more careful in the portal.” It was “stop touching the portal.” If the cluster is defined by code, then:

• Rebuilding is a command, not an afternoon.
• There are no undocumented clicks.
• Dangerous defaults (like an image tag of latest, more on that later) get pinned once and stay pinned.

The network: one Load Balancer for everything

A common instinct is a public IP per node, or a cloud LB per service. Both get expensive and hard to manage. Instead, a single Standard Load Balancer with one static public IP (the VIP) carries every kind of traffic:

                          6443  (Kubernetes API)
Internet --> LB VIP  -->   80 / 443  (web, into Traefik)
<public-ip>               SSH via inbound NAT (ports 50001+, one per node)
                          outbound SNAT (nodes have no public IP; egress via the LB)
                                    |
                        3x VMSS nodes (control-plane + etcd), no public IP

Two consequences worth calling out:

• Nodes have no public IP. Inbound traffic comes through the LB; outbound goes through an LB outbound rule (SNAT). This shrinks the public attack surface to a single IP.
• Node-to-node k3s join does not use the VIP. Azure has a hairpin limitation: a backend VM cannot reach the frontend VIP of the load balancer it sits behind. So cluster registration happens over the private subnet. cloud-init scans the subnet for a live peer on 6443 and joins that directly. The VIP is only for external clients (my kubectl, web visitors). The API server’s --tls-san includes the VIP so external kubectl https://<public-ip>:6443 still validates.

Why burstable B-series VMs

This is a personal cluster that sits near-idle most of the time, with occasional bursts (an image pull, a sync, a page load). That is exactly the workload B-series burstable VMs are priced for: you bank CPU credits while idle and spend them during spikes. Three 2 vCPU / 4 GiB burstable nodes run the whole thing, and (as Part 9 shows) the CPU credit balance basically never runs dry.

The shared layer (a separate resource group)

A handful of resources should outlive any given cluster, so they live in their own resource group and the cluster Bicep references them as existing:

Resource	Purpose
Key Vault	stores the k3s join token
User-Assigned Managed Identity (UAMI)	the node identity; granted read on Key Vault + read/write on the storage container
Storage account (HNS on)	leader-election lock + etcd snapshots
Public IP (Standard, Static)	the fixed VIP; the LB claims it, so the IP survives cluster rebuilds

Because the cluster Bicep only references these, deleting and recreating the cluster resource group never touches them, and the role assignments do not need to be redone. That static public IP is the reason a rebuild needs zero DNS changes.

// cluster Bicep references the shared layer, it does not create it
resource sharedPip 'Microsoft.Network/publicIPAddresses@2023-09-01' existing = {
  name: 'pipv4-shared'
  scope: resourceGroup('rg-shared')
}
resource uami 'Microsoft.ManagedIdentity/userAssignedIdentities@2023-01-31' existing = {
  name: 'uami-shared'
  scope: resourceGroup('rg-shared')
}

The Bicep: LB + VMSS

The load balancer wires the VIP to the backend pool with health probes and rules for each port, plus inbound NAT for SSH and an outbound rule for egress:

resource lb 'Microsoft.Network/loadBalancers@2023-09-01' = {
  name: 'lb-cluster'
  location: location
  sku: { name: 'Standard' }
  properties: {
    frontendIPConfigurations: [
      { name: 'fe', properties: { publicIPAddress: { id: sharedPip.id } } }
    ]
    backendAddressPools: [ { name: 'be' } ]
    probes: [
      { name: 'p6443', properties: { protocol: 'Tcp', port: 6443, intervalInSeconds: 5, numberOfProbes: 2 } }
      { name: 'p443',  properties: { protocol: 'Tcp', port: 443,  intervalInSeconds: 5, numberOfProbes: 2 } }
    ]
    loadBalancingRules: [
      // 6443 API, 80/443 web -> backend pool, each referencing a probe
    ]
    inboundNatRules: []       // SSH: a frontend port range 50001+ maps to :22 per instance
    outboundRules: [
      // SNAT egress for nodes that have no public IP
    ]
  }
}

The VMSS is where the two hard-won safety settings live:

resource vmss 'Microsoft.Compute/virtualMachineScaleSets@2023-09-01' = {
  name: 'vmss-cluster'
  location: location
  sku: { name: vmSku, capacity: instanceCount }   // Standard_B2als_v2, 3
  properties: {
    orchestrationMode: 'Flexible'
    upgradePolicy: { mode: 'Manual' }             // never auto-apply model changes
    virtualMachineProfile: {
      storageProfile: {
        imageReference: {
          publisher: 'canonical'
          offer: 'ubuntu-24_04-lts'
          sku: 'server'
          version: imageVersion                   // PINNED exact version, never 'latest'
        }
        osDisk: { managedDisk: { storageAccountType: osDiskType } }
        dataDisks: [
          { lun: 0, createOption: 'Empty', diskSizeGB: dataDiskSizeGB, managedDisk: { storageAccountType: dataDiskType } }
        ]
      }
      osProfile: {
        customData: loadFileAsBase64('cloud-init.yaml')   // node bootstrap
        linuxConfiguration: { disablePasswordAuthentication: true /* SSH key only */ }
      }
      // identity: the shared UAMI, so nodes can read Key Vault + storage without stored creds
    }
  }
}

Two settings there are not cosmetic:

• version: imageVersion pinned to an exact build, never latest. This is the single most important line. A latest image reference plus a full model update re-resolves to a newer build, and the rolling engine applies that difference by reimaging. Pinning removes the drift entirely.
• upgradePolicy: Manual. It stops Azure from auto-applying model changes. It is a useful backstop, but on its own it is not enough (the postmortem shows why); pinning the image is what actually protects you.

The parameters carry no secrets, just the SSH public key, SKU, count, disk types, and the pinned image version:

param location string = 'westus3'
param vmSku string = 'Standard_B2als_v2'
param instanceCount int = 3
param imageVersion string          // e.g. '24.04.20260714' - bump deliberately on rebuild
param osDiskType string = 'StandardSSD_LRS'
param dataDiskType string = 'StandardSSD_LRS'
param dataDiskSizeGB int = 16

cloud-init: mount the data disk, then bootstrap k3s

Every node runs the same cloud-init. It does two things.

1. Mount the data disk (format only if blank)

The data disk holds /var/lib/rancher (etcd database, snapshots, and the containerd image cache). The mount script is deliberately idempotent: if the disk already has a filesystem, it mounts it without reformatting. That is what makes reimaging safe: wipe the OS disk, and the etcd state on the data disk is picked back up.

LABEL="k3sdata"; MOUNT="/var/lib/rancher"; DEV="/dev/disk/azure/scsi1/lun0"
if blkid "$DEV"; then
  echo "existing filesystem -> mount, do NOT format"
else
  mkfs.ext4 -L "$LABEL" "$DEV"       # only on a blank disk
fi
mkdir -p "$MOUNT"
grep -q "LABEL=$LABEL" /etc/fstab || echo "LABEL=$LABEL $MOUNT ext4 defaults,nofail 0 2" >> /etc/fstab
mountpoint -q "$MOUNT" || mount "$MOUNT"

Mounting by LABEL (not device path) matters: after a reimage the disk can come back on a different device path, but the label is stable.

2. Bootstrap the HA control plane

k3s HA with embedded etcd needs exactly one node to run --cluster-init and the rest to join it. With three identical nodes booting at once, “who initializes?” is a race. I solve it with a blob lease as a distributed lock:

1. Each node fetches a UAMI token from the Azure Instance Metadata Service (IMDS), then reads the k3s join token from Key Vault.
2. Each node tries to acquire a lease on a well-known blob. Exactly one wins.
3. The winner runs k3s server --cluster-init and becomes the first server.
4. The others wait, then join by scanning the private subnet for a live 6443 and registering against it (round-robin, never the VIP, because of the hairpin limitation).
5. If the data disk already contains an etcd database (a reimaged node coming back), skip all of the above and simply rejoin.

# acquire the leader lease (Blob REST). Note the required empty-body header.
curl -X PUT "https://<storage>.blob.core.windows.net/boot/lock?comp=lease" \
  -H "Authorization: Bearer $TOKEN" -H "x-ms-lease-action: acquire" \
  -H "x-ms-lease-duration: 60" -H "Content-Length: 0"    # 411 without this

The lease request must send an explicit Content-Length: 0. Without it Blob storage returns 411 Length Required, no leader is elected, and every node deadlocks waiting for a cluster that never initializes.

The lock also fails safe. The lease is short (60 seconds), and the winner writes an “initialized” marker blob only after its own API server is actually up, then releases the lease. So if the winner dies mid-init, the lease simply expires, another node acquires it, sees no marker, and takes over the --cluster-init itself. A dead leader delays the bootstrap; it does not wedge it.

The API server is started with --tls-san <public-ip> so external kubectl against the VIP validates, and etcd snapshots are periodically uploaded to the same storage container for an extra offline copy.

Reimage safety, verified

Here is the payoff. Reimage one node: the OS disk is rebuilt (uptime resets, SSH host key changes), but the data disk survives because it is mounted by label and never reformatted. cloud-init sees an existing etcd database and takes the “rejoin” path. The node comes back with its state intact, and with three-node HA the other two held quorum the whole time. The disaster that started this whole redesign now self-heals.

Control-plane memory and topology (why three full servers)

Once it was running I looked at memory, expecting the apps to be the cost. They aren’t. On a 4 GiB node, the dominant consumer is the k3s server process itself at roughly 1.6 GiB. k3s packs the entire control plane (API server, controller-manager, scheduler, datastore) into one process. The big chunk is the API server’s in-memory watch cache, plus the OpenAPI schema for every CRD (Argo, cert-manager, Traefik, and sealed-secrets add up). All three nodes run a full API server; they do not share.

I considered lighter topologies:

Topology	API servers	Auto failover	Control-plane RAM
3 full servers (current)	3	yes	~4.8 GiB total
2 full + 1 etcd-only	2	yes	~3.7 GiB
1 full + 2 etcd-only	1	manual	~2.6 GiB

The measured available memory per node is comfortable, so I kept three full servers for real HA and left the alternatives documented for the day memory gets tight. The apps themselves are tiny; a static site pod idles at single-digit MB.

The deploy script and first boot

deploy.ps1 is thin: create the resource group, (for a fresh cluster) clear the stale bootstrap markers in storage, then run the Bicep deployment.

# brand-new cluster (clears the leader-election markers first)
.\deploy.ps1 -ResourceGroup rg-cluster -FreshCluster
# re-apply against an existing running cluster (keeps markers)
.\deploy.ps1 -ResourceGroup rg-cluster

That -FreshCluster flag matters: the leader-election lock and init marker live in the shared storage account, which is not deleted with the cluster. If you rebuild without clearing them, every blank node thinks the cluster already exists, nobody runs --cluster-init, and they all deadlock. Clearing the markers on a fresh build avoids that.

A few operational notes I hit along the way:

• SKU capacity. Standard_B2als_v2 occasionally returns SkuNotAvailable (real-time regional capacity, not a config error). Wait and retry, or temporarily override the SKU; the design is SKU-agnostic.
• Scaling down means deleting the VMSS instance and running kubectl delete node <name> so k3s removes the stale etcd member and quorum math stays correct.
• Never use the portal’s “full PUT” wizards on a VMSS running stateful workloads. Use incremental CLI changes. (See the postmortem.)

What’s next

Part 3 puts GitOps on top of this: installing Argo CD, connecting a private repo, the App-of-Apps bootstrap, Kustomize overlays, private image pulls, and having Argo manage itself. The infrastructure is now a command; next we make the contents of the cluster a command too.