Self-Hosted k3s on Azure - Part 11: Grafana Cloud & Right-Sizing

Part 10 got a working metrics pipeline running entirely on the cluster. It worked, but it asked a lot of three small nodes: an operator, a time-series database, and a Grafana, all competing for memory with the apps they were supposed to watch. This part moves the storage and the dashboards off the cluster to a managed backend, leaving only a light collector behind, and then right-sizes a few things that were wasting resources without anyone noticing.

Placeholders: example.com for the domain, <cloud-endpoint> for the managed backend, generic namespace names.

Why move, and what stays

Two reasons pushed the change. First, the umbrella chart is heavy. Rendering it produced thousands of lines of manifests on every sync, which the repo-server had to process each time. Second, a managed backend removes the local database and the local Grafana, and its free tier is enough for a cluster this size once the cardinality work from Part 10 is in place.

The end state keeps only the collection side on the cluster: vmagent, node-exporter, and kube-state-metrics. No operator, no CRDs, no local database, no Grafana. Storage and dashboards live in the managed backend. The manifests Argo has to render drop from thousands of lines to a few hundred.

One decision I want to justify, because it looks like it should change: I kept vmagent rather than switching to the backend’s native agent. The ingestion endpoint is the standard Prometheus remote_write protocol, so the backend does not care who sends the data. What it does care about is that all of my cardinality control, the metric keep-list and the stream aggregation that collapses request counters into a handful of series, is written in vmagent’s own configuration. Keeping vmagent means that work carries over unchanged. Switching agents would mean rewriting all of it.

Validate before you cut over

The risky part of this migration is the scrape configuration. Moving off the operator means replacing its custom resources with a hand-written static scrape_configs block that reproduces exactly what the operator was collecting. Get a job wrong and you silently lose a set of metrics. So the whole migration was built to validate against production without touching it, and the cutover to the cloud was the very last step.

The technique that made this safe was isolating the new pipeline with a label. I deployed the new vmagent alongside the old one, still writing to the local database, but stamped every series it produced with a distinct cluster label. Old and new data then sat in the same database, separated by that label, and I could compare them directly:

count(count by (__name__) ({cluster="prod-test"}))
count(count by (__name__) ({cluster="prod"}))

Comparing the two metric-name sets side by side is what caught a job I had missed entirely: CoreDNS was absent from the rewritten config. Nothing about a plain target count would have shown that; only diffing the actual set of metric names did. Once the two sides matched, I switched vmagent’s remote_write from the local database to the cloud endpoint and removed the temporary database. Every earlier step happened locally.

One thing that looks alarming during this kind of transition but is not: when you swap agents, the old one’s series stay queryable for a few minutes inside the database’s staleness window. A count that should read three can briefly read six while both generations overlap. It settles on its own once the stale series expire.

Retiring the old stack cleanly

Disabling the self-hosted stack is a good example of doing a teardown in a way you can undo. The wrong way is to edit the umbrella chart’s values to disable its sub-components, because that leaves a half-configured release in Git. The approach I used instead:

• Restore the chart’s values to their last known-good state by checking them out from the commit before the migration, so the release stays internally consistent.
• Rename the Applications to *.yaml.disabled. The App-of-Apps root ignores files with that suffix, so Argo prunes the whole stack. Renaming them back is a one-step re-enable, with the configuration fully preserved.

Disabling the umbrella also prunes the components it owned, including node-exporter and kube-state-metrics, so those became their own small Applications using the official upstream charts. Adding a new chart repository means updating the AppProject’s allowed source list, which is the one file in this setup that is applied by hand rather than through GitOps.

The new collection stack also got its own namespace. Leaving it in the old grafana namespace after Grafana was gone would have been misleading. The house convention is one namespace per logical group, so vmagent, node-exporter, kube-state-metrics, and their credential now share a monitoring namespace. Changing the namespace touched the obvious places, and one less obvious one: the sealed credential for the cloud backend.

Re-sealing a secret across namespaces

The credential that authenticates writes to the cloud backend is a SealedSecret, and by default those are bound tightly to both a name and a namespace. Moving it to the new namespace meant re-sealing it. The safe way, without ever putting the plaintext in Git or on screen, is to seal it directly from the live secret:

kubectl get secret cloud-rw -n old-ns -o json |
  # rewrite metadata.namespace to the new namespace, drop runtime fields
  kubeseal --format yaml > sealed-cloud-rw.yaml

The secret values move from the running secret through the sealing step and into the new encrypted file. They are never printed. This is the same discipline from Part 5: the plaintext of a secret has exactly one moment of existence, and it is never a commit.

With the migration done, the local disk driver had no users left, since nothing on the cluster mounted a persistent volume any more. So it was disabled the same way, by renaming its Applications to .yaml.disabled, removing a DaemonSet and a controller from every node. The trade-off is explicit and documented: re-enabling anything that needs a persistent volume means re-enabling the driver first.

Right-sizing, part one: CPU throttling on idle pods

With the migration settled, the new dashboards surfaced something I had never looked at: a couple of pods showed continuous CPU throttling, around ten percent of the time, even though their average CPU use was almost nothing. This is a common and misunderstood effect, so it is worth explaining properly.

The Linux CFS scheduler turns a CPU limit into a quota per 100-millisecond period. A limit of 100m means the container may run for 10 milliseconds in each 100-millisecond window. A limit of 150m means 15 milliseconds. The pods in question were nearly idle on average, but they did short bursts of work: a session store answering replication pings and health checks, a small service running a scheduler tick and garbage collection. Any burst that needs more than its 10 or 15 milliseconds inside a single window gets throttled, even though the average across seconds is tiny. That is exactly what the dashboard showed.

The effect is usually harmless. Throttling only delays those brief bursts by a few milliseconds, which does not matter for services that are not latency-critical. But the fix is simple and low-risk: raise the CPU limit well above the burst peak while leaving the request untouched.

resources:
  requests:
    cpu: 25m      # unchanged; the scheduling floor
  limits:
    cpu: 300m     # was 150m; headroom so bursts are not clipped

Because average use is so low, the pods never actually consume the higher limit. The change just stops CFS from clipping their bursts, and the throttling dropped to near zero. I deliberately raised the limits rather than removing them. On two-core nodes, an unbounded limit invites a burst to starve its neighbors, which is the failure mode the memory incident in Part 10 was a version of. A generous ceiling is safer than no ceiling.

Right-sizing, part two: an unbounded controller heap

Sorting pods by memory turned up a clear outlier. Argo CD’s application-controller was using around 600 MB, six times the next-largest pod, and the whole Argo namespace accounted for close to a gigabyte. The controller’s memory is mostly its cache of every resource it manages, plus the Go runtime’s heap. The cache side was already trimmed as far as it goes, by excluding the churny resource kinds it does not need to track.

The remaining lever is the Go runtime. The controller had no memory limit and no soft-limit hint, so the Go runtime, which by default holds on to freed memory rather than returning it to the operating system, simply kept it. Setting a soft memory limit changes that. It makes the garbage collector run more aggressively as usage approaches the limit and return memory to the OS, and a hard limit slightly above it acts as a safety ceiling so the collector fires before anything is killed:

controller:
  env:
    - name: GOMEMLIMIT
      value: "512MiB"
  resources:
    requests:
      memory: 320Mi
    limits:
      memory: 640Mi

After the controller restarted with these settings, its memory dropped from about 600 MB to around 330 MB, freeing roughly 280 MB across the cluster at no measurable CPU cost. One caution, because this component manages everything else: set the soft limit too low and the garbage collector thrashes or the process is killed, which is disruptive when the thing being killed is your reconciler. The values here are deliberately conservative, and the right response to seeing it restart under memory pressure is to raise both numbers together, not to lower them.

The trade-offs, stated plainly

Moving to a managed backend is not free of downsides, and it is worth being honest about them:

• Retention is shorter. The free tier keeps roughly two weeks of data, where the self-hosted database kept a year. Long-term trends now need a paid tier.
• The data leaves the cluster. For a homelab that is an acceptable trade for not running a database on nodes that are short on memory. For something with stricter data requirements it might not be.
• You depend on an external service. The collector buffers briefly if the backend is unreachable, but sustained connectivity problems mean gaps.

Against those, the cluster stopped running an operator, a database, and a Grafana; the repo-server stopped rendering thousands of lines on every sync; and the memory those pieces used went back to the apps. For this setup that was the right call.

Closing the series, again

Part 9 ended the build arc, and in a sense it was an ending: it proved the cluster was disposable. These two parts are what you build once the platform is stable enough to ask how it is actually behaving. The shape of the answer is familiar by now. Collect efficiently, understand what each number means before you trust it, size things against the hardware you actually have, and keep the whole configuration in Git so any of it can be rebuilt or reversed. The cluster is disposable, and now it is observable too.

Thanks for reading.