Self-Hosted k3s on Azure - Part 10: Observability with VictoriaMetrics

The series reached a natural stopping point at Part 9: the cluster was disposable, the bill was under control, and everything ran from Git. But there was still no way to answer a basic operational question: is anything unhealthy right now, and was it different an hour ago? This part adds observability. It turned out to be the densest engineering of the whole project, mostly because k3s makes some assumptions that standard Kubernetes monitoring does not expect.

Placeholders: example.com for the domain, generic namespace and resource-group names. Snippets are trimmed to the part that matters.

Why VictoriaMetrics, and why the umbrella chart

The obvious default is Prometheus. I chose VictoriaMetrics instead. It speaks the same language (PromQL for queries, the Prometheus remote_write protocol for ingestion), so nothing downstream has to change, but it uses noticeably less memory and disk for the same data. On a cluster of burstable 4 GB nodes, that efficiency is the deciding factor.

For packaging I started with the victoria-metrics-k8s-stack umbrella chart. One Helm release gives you the whole pipeline: the VictoriaMetrics operator, a single-node TSDB, the scraper, kube-state-metrics, node-exporter, the scrape configuration, and a set of dashboards. The cost is one extra operator pod. The benefit is that you do not hand-write scrape configuration, which, as the rest of this post shows, is where most of the traps live.

The pieces and how data flows

flowchart LR
    subgraph targets[Scrape targets]
        ne[node-exporter]
        ksm[kube-state-metrics]
        kubelet[kubelet / cAdvisor]
        api[apiserver]
        etcd[etcd :2381]
    end
    vmagent[vmagent<br/>the only scraper]
    vmsingle[(vmsingle<br/>TSDB)]
    grafana[Grafana]

    targets -->|scrape /metrics| vmagent
    vmagent -->|remote_write| vmsingle
    vmsingle -->|PromQL| grafana

Two roles are worth separating clearly, because confusing them wastes debugging time:

• vmagent is the only component that scrapes. If a target is missing, you look here, at its /targets page.
• vmsingle only stores and serves. It never scrapes. It holds the time series and answers queries.

The operator model means you do not write Deployments directly. You write custom resources (VMSingle, VMAgent, VMServiceScrape, VMStaticScrape) and the operator renders the underlying workloads. It is declarative and upgrades cleanly, but it introduces two Argo CD frictions worth calling out up front:

• The operator’s CRDs are large. Argo’s default client-side apply tries to store the whole CRD in an annotation and hits metadata.annotations: Too long. The fix is syncOptions: [ServerSideApply=true] on the Application.
• The release name has to match. The scrape CRs select their targets by the Helm release name, while Argo overrides the instance label with the Application name. If the two differ, scraping silently finds nothing. Setting the chart’s release-name override to equal the Application name lines them up.

k3s does not run a normal control plane

This is the first place standard monitoring guides stop working. In upstream Kubernetes, etcd, the controller-manager, and the scheduler are static pods, and kube-proxy is a DaemonSet. Each has its own /metrics endpoint. k3s collapses all of them into a single k3s server process. There are no pod objects for them, and their metrics ports bind to 127.0.0.1 by default.

Getting to them takes two steps. First, open the ports at the node level, in each server’s /etc/rancher/k3s/config.yaml and in cloud-init:

Component	Port	Notes
etcd	2381	plain HTTP, a read-only metrics port (not the 2379 data port)
controller-manager	10257	HTTPS, needs a service-account token
scheduler	10259	HTTPS, needs a service-account token
kube-proxy	10249	plain HTTP

etcd deserves a note. You scrape it on 2381, never on 2379. Port 2379 is the data API and speaks mutual TLS; scraping it would mean handing the etcd client certificate, effectively the master key to all cluster data, to the scraper. Port 2381 exists precisely so metrics can be collected safely. The apiserver, kubelet, and CoreDNS are all scrapable out of the box on k3s and need no changes.

The second step is the scrape configuration, and it hides a subtle GitOps trap.

The manual Endpoints trap

Because the control-plane components are not pods, the chart cannot select them with a label selector. The pattern it falls back to is a Service with no selector plus a manually defined Endpoints object pointing at the node IPs. That is a normal Kubernetes technique. It just collides with an Argo CD default.

Argo’s out-of-the-box configuration excludes Endpoints and EndpointSlice from the resources it manages. The reason is sound: those objects churn constantly as pods come and go, and watching them cluster-wide is expensive. The consequence here is that Argo silently drops the manually defined Endpoints. The Service gets created, the scrape CR exists, and the target count sits at zero. The tell is that kubectl get endpoints <svc> returns NotFound.

There are three ways out:

Option	Approach	Trade-off
VMStaticScrape	declare the target IPs directly in a static scrape CR; no Endpoints object is created	one file, immune to the exclusion; fewer labels
Un-exclude Endpoints	override Argo’s exclusion list to keep watching Endpoints	full labels, but Argo now watches all Endpoints cluster-wide
hostNetwork placeholder pods	run a dummy pod per node so Kubernetes builds the Endpoints for you	works, but you are maintaining fake pods

I initially un-excluded Endpoints, measured the impact, and found it small: the cluster had only about seventy Endpoints objects, and the controller’s memory did not move. That let the chart’s native configuration work with full labels. As the next section explains, I later reversed most of this anyway.

The reversal: k3s stores the same metrics five times

After scraping was working, I broke the series count down by job label and found the same metrics stored five times over:

count by (job) (apiserver_request_total)
# apiserver, controller-manager, scheduler, kube-proxy, kubelet all report it

The kubelet endpoint was even returning apiserver_* and etcd_* metrics, which have nothing to do with the kubelet. The cause is the same single-process design from earlier. Because one k3s server process hosts the entire control plane, every metrics port exposes the same merged registry. Scraping five ports stored five identical copies that differed only by their job label.

Only three sources hold anything unique: the apiserver (the shared registry), the kubelet (its cAdvisor container_* metrics), and etcd on its separate 2381 port. So I cut the rest:

• apiserver: kept as the single source of the shared registry. It is a first-class target that authenticates with a service-account token, its labels are semantically correct, and community dashboards expect apiserver_* under job="apiserver".
• kubelet: kept, but filtered down to only its own container_*, kubelet_*, and machine_* metrics. cAdvisor is genuinely unique to it.
• etcd: kept, on its own port.
• controller-manager, scheduler, kube-proxy: turned off entirely. Every metric they exposed was already present in the apiserver’s copy.

The lesson generalizes to any distribution that packs multiple components into one process: verify by job whether you are collecting duplicate registries before you decide how many ports to scrape. Copying the upstream “one target per control-plane component” pattern onto k3s just stores the same data several times.

Cutting ingest by ninety percent

Duplication was not the only problem. The first full scrape produced roughly 30,000 samples per second and over 600,000 active series, far more than a small TSDB should hold. Breaking cardinality down by metric name pointed at one culprit: histogram _bucket series.

A histogram stores one series per bucket boundary, multiplied by every label combination (verb, resource, code, and so on). A single request-duration histogram easily becomes tens of thousands of series. Those buckets exist to compute latency percentiles, which a homelab almost never needs. The _count and _sum series, which still give you averages and rates, are cheap.

Dropping them per-scrape would mean editing several jobs and missing some. The clean fix is a single relabel rule on vmagent’s remote_write, so every sample from every job passes through it once before storage:

remoteWrite:
  - url: http://vmsingle:8428/api/v1/write
    inlineUrlRelabelConfig:
      - sourceLabels: [__name__]
        regex: (apiserver_.+|etcd_request_duration_seconds|workqueue_.+|scheduler_.+_duration_seconds|kubelet_.+_duration_seconds|rest_client_request_duration_seconds)_bucket
        action: drop

Combined with the control-plane dedup, ingest fell in two stages:

Stage	Ingest	Method
Original	~30,000/s	—
Drop histogram buckets	~7,400/s	one relabel rule on remote_write
Dedup control plane + trim kubelet	~2,900/s	turn off duplicate jobs

That is about a ninety percent reduction, and the real active-series count dropped from around 600,000 to roughly 64,000. One caveat when you verify this: the TSDB’s reported total-series number lags, because series already written stay in the index until they age out. To see the live figure, count series that received a sample in the last couple of minutes rather than reading the cumulative total.

The incident: one pod’s memory took down GitOps

VictoriaMetrics is memory-bound. Its query and index caches live in RAM, and, importantly, it grows its cache to fill a large fraction of its memory limit. It treats the limit as a target, not just a ceiling. I learned this the hard way.

To fix slow queries I had raised the TSDB’s memory limit generously. On a 2 vCPU / 4 GB control-plane node, the result was a cascade:

1. The TSDB grew past 900 MB and pushed its node into NotReady.
2. Its storage is a single read-write-once volume with a Recreate strategy, so the pod rescheduled onto a second node and did the same thing there.
3. As nodes went under load, CoreDNS slowed down. Argo’s repo-server timed out resolving DNS, could no longer render manifests, and every application flipped to ComparisonError. Even self-heal stopped working, because self-heal depends on the repo-server being able to render.

A single stateful app’s memory had taken down the entire GitOps control loop, through the chain of node pressure to DNS to the repo-server. Recovery was manual: SSH to each NotReady node and restart k3s to release the memory, one node at a time to keep etcd quorum.

The permanent fix was to tighten the limit in Git, but that had its own twist worth remembering. While the cluster was unhealthy, a manual kubectl patch to lower the limit was pointless. The moment DNS recovered and self-heal resumed, Argo reverted the patch back to the value in the Application’s inline values. The correct order is to fix the value in Git, let the root Application sync so the new limit flows through the chart to the operator to the pod, and only then is live state consistent with Git and stable.

Two durable lessons came out of it:

• On small nodes, a memory-hungry component’s limit must be sized against the node’s actual free memory. A generous limit is not a safety margin; it is an instruction to fill the node, and the read-write-once reschedule spreads the failure to the next node.
• Under GitOps, kubectl patch is a probe, not a repair. The real fix goes through Git, or self-heal undoes it.

Grafana on top, kept stateless

With the pipeline healthy, Grafana went on top to visualize it. I ran it stateless: no persistent volume, dashboards defined declaratively, and authentication delegated to the existing forward-auth layer from Part 7 rather than local accounts. That keeps it disposable, in line with the rest of the cluster. Newer Grafana releases have a heavy cold start, so the probes need a generous startup window, and the memory limit has to reflect real usage rather than an optimistic guess, but none of that changes the stateless posture.

Where this leaves us

The self-hosted stack worked and cost only a couple of dollars a month in disk. But it also meant running an operator, a TSDB, and a Grafana on nodes that were already tight on memory, and every deployment briefly stressed whichever node absorbed it. That set up the next question: could the storage and the dashboards move off the cluster entirely, leaving only a light collector behind? That is Part 11, where the metrics go to Grafana Cloud and the whole footprint gets right-sized.