Self-Hosted k3s on Azure - Part 12: Trimming Memory, Disk, and a Reserved-Instance Dead End

Part 9 got the bill to about 130 USD a month, and Part 11 right-sized the pieces that were quietly wasting resources. This part starts from a concrete goal: add a fourth node without breaking a 150 USD ceiling. Chasing that budget turned up three things worth writing down: how to read the Azure bill precisely, a disk-billing detail that reverses a decision from Part 9, and a way to shrink the single largest memory consumer on every node.

Placeholders: example.com for the domain, generic resource-group and subscription names.

Reading the bill precisely

The first task is knowing what you actually spend, and that is harder than it should be on a credit-based subscription. az consumption usage list returns null costs, so it is useless here. What works is the Cost Management REST API, the same one the portal uses, queried with az rest and grouped by resource group, service, or meter.

Two traps make the number misleading if you are not careful:

• Trailing costs from deleted resource groups. A seven-day window showed charges for resource groups that no longer existed. They were the tail end of a region migration; the groups had been deleted, but their costs kept appearing in the window. The fix is to compute a run-rate from a recent full day across only the resource groups that still exist, not a multi-day sum that includes ghosts.
• New resources are understated. The mirror image of the same problem: a freshly created resource shows almost nothing, because it did not exist for the whole period and Cost Management lags. Again, a single recent day is the honest measure.

With that method, the run-rate came out at roughly 147 USD a month: about 127 for the cluster resource group and 20 for the shared layer. Adding a fourth node would push it well over the ceiling. So the rest of the work was finding that money elsewhere.

The disk-billing surprise

Breaking the cluster resource group down by meter turned up an unexpected line item: disk operations, about 17 USD a month. Storage capacity was cheap; the operations were not.

Azure Monitor showed where they came from. The data disks, which hold etcd under /var/lib/rancher, were doing around 40 IO operations per second each, roughly 11 million per day across three nodes. That is not a busy cluster; it is etcd doing what etcd does, calling fsync on every write for durability. The OS disks, by contrast, were nearly idle.

The billing detail is the important part. Standard SSD managed disks charge per transaction. Premium SSD disks do not. Premium bills a flat monthly rate by tier, with provisioned IOPS included and no per-operation charge. For a small, high-IOPS disk like an etcd volume, that inverts the usual intuition: Premium is cheaper than Standard SSD once you count the transactions.

So the data disk went away entirely. Instead of an OS disk plus a separate Standard SSD data disk, each node now has a single Premium disk holding the OS, etcd, and the image cache. The transaction charge dropped from about 17 USD to near zero, the extra capacity cost a few dollars, and the net saving was around 12 to 15 USD a month. etcd also got Premium’s steadier latency for free. Host caching on that disk is set to read-only, not read-write: reads are cached, but writes go straight to storage, so etcd’s fsync stays durable across a host failure.

This reverses a decision from Part 9, where I moved disks from Premium to Standard SSD to save money. That change only counted capacity price. It ignored transaction cost, and for the etcd disk specifically it was a net loss. The lesson is to price all the dimensions, capacity and transactions and the latency you actually need, not just dollars per gigabyte.

The trade-off worth naming: a single OS-resident etcd no longer survives a bare reimage the way a separate persistent data disk did. But with the image version pinned, three-node HA, etcd snapshots to blob storage, and application state in external Postgres, that survival was already redundant. A reimaged node rejoins and re-syncs; a full rebuild restores from snapshot or rebuilds from Git.

Bounding the k3s control-plane heap

The single largest memory consumer on each node is not an app. It is the k3s-server process itself, sitting at 1.2 to 1.6 GB. k3s runs the entire control plane, the API server, embedded etcd, scheduler, controller-manager, and kubelet, in one process, so that number is all of them combined.

The cache side of that memory is intrinsic, but the Go runtime side is not. By default Go holds on to freed heap rather than returning it to the operating system. Setting GOMEMLIMIT changes that: it is a soft memory target that makes the garbage collector return memory to the OS as usage approaches the limit. I applied it through a systemd drop-in on the k3s service:

# /etc/systemd/system/k3s.service.d/gomemlimit.conf
[Service]
Environment=GOMEMLIMIT=900MiB

The method matters more than the value. I tested it live on one node first, where deleting the file reverts the change, rather than committing it blind. Two things are easy to get wrong:

• A fresh restart reads low. After restarting k3s, the process starts small and grows as it rebuilds etcd’s keyspace and the API server’s watch caches. Comparing that early number to the old steady state is meaningless. You have to let it run an hour or so to reach a real plateau.
• Watch the load average. If the limit is below the working set, the collector runs constantly to chase an impossible target, and CPU climbs. That is the signal to loosen it.

At 900 MiB the result was a steady-state drop from about 1264 MB to 990 MB per node, roughly 270 MB saved, with the load average staying near zero. No thrashing. Applied across three nodes and baked into cloud-init, that is close to 800 MB of cluster memory returned.

One conceptual point, because the numbers look contradictory: the resident memory settled at 990 MB, above the 900 MiB limit. That is expected. GOMEMLIMIT bounds the Go heap, not total resident memory. The extra 90 MB is goroutine stacks, the memory-mapped etcd database, and the runtime, none of which the garbage collector governs. And because it is a soft limit, it never triggers an out-of-memory kill; the worst case is more GC work. Go even caps garbage collection at half a CPU to avoid a death spiral, so setting the limit too low makes memory exceed it rather than melt the node. That safety is exactly why it is reasonable to apply to a process as critical as the control plane, as long as you leave headroom above etcd’s working set.

The reserved-instance dead end

With disk savings banked and memory trimmed, the arithmetic still did not close. A fourth node of a usable size adds about 32 USD a month, and the run-rate after the disk change was about 132, so a fourth node lands near 164, over the ceiling by roughly 14.

The obvious lever is a reserved instance. A one-year reservation on the burstable VM size is around 40 percent off, paid monthly at no extra cost, and it applies to any matching running VM in the region, which suits a cluster that gets rebuilt. Four reserved nodes would cost less than three on-demand ones.

It is not available here. This is a Visual Studio subscription, and the portal rejects the purchase outright: reservations apply only to Enterprise Agreement, Pay-As-You-Go, CSP, and Microsoft Customer Agreement offers. Azure Savings Plans have the same eligibility, so they are out too. Converting to Pay-As-You-Go would unlock reservations but forfeit the monthly credit the whole thing runs on. The single most effective cost lever was simply closed.

Worth checking before you plan around it: try the purchase in the portal first. It fails at the eligibility stage, before any charge, so it is a safe way to find out.

The honest conclusion

Without reservations, on demand, a full fourth 4 GB node does not fit under 150. The cheapest 4 GB size is already the one in use; going smaller means dropping to 2 GB, and a 2 GB node cannot be a server, because k3s-server alone wants 1.2 GB. It could be an agent, a worker with no control plane, but that only adds about 1.5 GB of schedulable memory and needs a mixed server/agent bootstrap.

So the useful takeaways are not “here is how I added a node.” They are:

• Exhaust the zero-cost optimizations first. The disk refactor and the memory trim cost nothing and bought back both dollars and headroom. Those come before buying hardware.
• Price every dimension. The disk decision hinged on transactions, not capacity. The obvious number was the wrong one.
• Check purchasing eligibility before you design around a discount. A reservation that saves 40 percent is worthless if the subscription cannot buy it.
• The most expensive dollar can be the one you keep. Moving the managed database off Azure would have funded the node, but it is the very thing that makes the cluster disposable. Some costs are load-bearing.

The cluster runs leaner now, and the path to a fourth node is clear even if the budget is not: reclaim more by dropping control-plane HA to a single server plus agents, and back the redundancy with an external datastore. That is a larger change, and a topic for another day.

Thanks for reading.