Self-Hosted k3s on Azure - Part 13: Scale-to-Zero for Idle Apps

Part 12 ended on an honest note: on this subscription, under a 150 USD ceiling, a fourth node does not fit. The cheapest useful size is already in use, and k3s-server alone wants more memory than a 2 GB node has. So if I want to run more, the memory has to come from somewhere other than new hardware.

It turns out some of it was already there, held by apps that do nothing most of the day. An editor, a throwaway browser, a PDF tool: each is interactive, each is used for a few minutes at a time, and each sits resident around the clock holding memory it is not using. This part reclaims that memory by scaling those apps to zero when idle and waking them on demand, and then works through the two integration problems that make scale-to-zero non-trivial on this stack.

Placeholders: example.com for the domain, generic namespace names. Snippets are trimmed to the part that matters.

Scale to zero, and why it fits here

Scale-to-zero is exactly what it sounds like: an idle workload is scaled down to zero replicas, and the first request that arrives scales it back up. For a request-serving API this is a bad trade, because the first user after an idle period pays the entire cold-start latency. For a human-facing interactive app it is almost free, because the person clicking a bookmark already expects a moment before a desktop or an editor appears, and a few seconds of boot disappears into that expectation.

The apps I picked share that property. None of them is a dependency of anything else; nothing calls them machine-to-machine; they are opened by a person, used, and abandoned. That is the profile where the memory saving is real and the cost is a wait nobody notices. Databases, auth, ingress, the metrics collector all stay resident, because something is always talking to them.

The tool is Sablier, a small controller that watches for HTTP activity through a reverse-proxy middleware and scales a workload up when a request arrives, down when it goes quiet. It is not an autoscaler in the HPA sense; it does not look at CPU or a queue depth. It looks at whether anyone is currently using the thing.

How the wake works

Sablier sits behind the reverse proxy as a middleware. Every request for a managed app passes through it first. The mechanics are worth stating precisely, because the failure modes later all follow from them.

• Each managed workload carries a label marking it as Sablier-managed and a group name. Sablier discovers it and, when idle, holds it at zero replicas.
• A request arrives for app.example.com. The middleware intercepts it, tells Sablier the group was just used, and Sablier patches the workload’s scale subresource from 0 to 1.
• While the pod is starting, the middleware serves a small waiting page that auto-refreshes. This is Sablier’s dynamic strategy. On Kubernetes it is the only one that works; the alternative blocking strategy, which holds the request open until the app is ready, is not supported here.
• Each request resets a sliding session window (sessionDuration). As long as someone is active, the timer keeps moving. A background loop (expiration-interval) checks periodically, and once the window has elapsed with no activity, it scales the workload back to zero.

Two consequences fall out of this that are easy to trip over. First, the session is idle time, not a hard lease: an app in active use never gets pulled out from under you, and an abandoned one goes away on its own. Second, Sablier is lazy about stopping. It only scales something down when its session expires; it does not proactively hunt for running-but-unmanaged replicas and stop them. A freshly deployed workload therefore starts at one replica and stays there until you scale it to zero once by hand to hand it over. After that first handoff, Sablier owns it. If you want to end a session early rather than wait out the idle window, you scale it to zero yourself; the next request wakes it exactly as if the timer had run out.

Making Traefik cooperate

The proxy here is Traefik, and it needs two things.

The middleware is installed as a Traefik plugin and enabled globally on the entrypoint, then attached to each managed app’s ingress by annotation:

annotations:
  traefik.ingress.kubernetes.io/router.middlewares: app-namespace-sablier@kubernetescrd

The non-obvious setting is this one, on the ingress provider:

providers:
  kubernetesIngress:
    allowEmptyServices: true

Here is why it matters. When an app is scaled to zero, its Service has no endpoints. By default Traefik treats a Service with no endpoints as a route that cannot be served and drops it. But dropping the route is precisely the wrong move: the request that would have woken the app now hits nothing, so the app can never come back. allowEmptyServices keeps the router alive with zero backends, so the request still reaches the Sablier middleware, which is what triggers the wake. Without it, scale-to-zero deadlocks the moment it succeeds.

There is a happy interaction with auth. This cluster runs a global forward-auth middleware on the HTTPS entrypoint, so every request is checked against the identity provider before anything else. That check runs before the Sablier middleware, which means an unauthenticated visitor sees the login page, not the wake page, and only an authenticated user can spin an app up. Scale-to-zero and “don’t let strangers start my workloads” fall out of the same ordering for free.

Making Argo CD let go

This is the integration that actually bites, and it is specific to running scale-to-zero under GitOps.

Argo CD’s whole job is to make the cluster match Git. Self-heal is on, so when the live state drifts from the manifest, Argo corrects it. Now put Sablier in the same cluster. Sablier’s job is to change the replica count in response to traffic. To Argo, that change looks like drift from the declared spec, and it “corrects” it, fighting the very controller that is supposed to own that field. Two controllers writing the same field is a loop.

The resolution is to make one field authoritatively Sablier’s. Three pieces, all required:

1. Omit replicas from the Deployment manifest. If Git declares replicas: 1, Argo has a concrete target to enforce. Leaving it out means Git has no opinion on the count.

2. Tell Argo to ignore the field:

   ignoreDifferences:
     - group: apps
       kind: Deployment
       name: app
       namespace: app
       jsonPointers:
         - /spec/replicas
   

3. Make the ignore apply during sync, not just drift detection:

   syncPolicy:
     syncOptions:
       - RespectIgnoreDifferences=true
   

The third one is the piece that is easy to miss. Without RespectIgnoreDifferences, ignoreDifferences only affects what Argo reports as out-of-sync; a sync can still write the field back. With it, the field is genuinely left alone through a sync. All three together, plus the one-time manual scale to zero to bootstrap, and Sablier owns the replica count without Argo undoing it.

A symptom worth naming, because it looks like the fix failed: right after enabling an app, I scaled it to zero and it bounced back to one within seconds. The cause was not Argo losing the fight; it was propagation timing. The app had only just been created and was still mid-rollout when I scaled it, and the two operations raced. Scaling it again once it had settled stuck immediately. When something looks like a controller conflict, check whether you are just watching two operations overlap in time before you go changing configuration.

A waiting page

The dynamic strategy serves an HTML page while the pod boots, and it is themable. One small packaging wrinkle: the Helm chart has no way to mount an arbitrary file, so the theme lives in its own ConfigMap generated by kustomize with a stable name, mounted into the controller as a volume, and pointed at with a custom-themes-path flag. The template gets a few variables from the middleware, a display name and the session duration, which is enough to tell a visitor what they just woke and roughly how long it will stay. It is cosmetic, but a branded “starting up, back to sleep after N idle” beats a raw spinner, and it is the one thing a user actually sees, so it is worth the ten minutes.

Takeaways

• Idle interactive apps are free memory waiting to be reclaimed. Anything a human opens for a few minutes and forgets is a scale-to-zero candidate; the cold start hides inside the click. Anything that serves other machines is not.
• A scale-to-zero route must survive having zero backends. The request that wakes the app has to reach the proxy while the app is down. On Traefik that is allowEmptyServices; drop it and the design deadlocks the instant it works.
• Two controllers cannot own one field. Under GitOps, hand the replica count to the scaler explicitly: omit it from Git, ignore it in the app, and make the ignore hold during sync, not just in the diff.

The cluster runs three more apps than it did, at no steady memory cost, and the path from Part 12 is now a little clearer: before you buy a node, check what the idle ones are holding.

Thanks for reading.