Self-Hosted k3s on Azure - Part 5: Committing Secrets with SealedSecrets

This is Part 5 of the Self-Hosted k3s on Azure series. Everything is in Git now, except the one thing you cannot put in Git as-is: secrets. This post closes that gap with SealedSecrets.

The problem: base64 is not encryption

A Kubernetes Secret stores its data as base64, which is encoding, not encryption. Anyone who can read the YAML can read the secret. So a plain Secret must never be committed. Before this, my app credentials were applied by hand with kubectl create secret, living outside GitOps and undocumented. That is exactly the kind of snowflake the whole project is trying to eliminate.

How SealedSecrets works

SealedSecrets fixes this with asymmetric crypto:

• A controller runs in the cluster and generates an RSA keypair on first start. The public key encrypts; the private key never leaves the cluster and only the controller uses it to decrypt.
• Locally, kubeseal fetches the public key and encrypts a plain Secret into a SealedSecret (a custom resource). The ciphertext can only be decrypted by that cluster’s private key, so it is safe to commit.
• The controller watches for SealedSecret objects, decrypts them, and produces a normal Secret with an ownerReference back to the SealedSecret (so their lifecycles are linked).

The mental split: a SealedSecret in Git is opaque ciphertext; the real Secret only ever exists inside the cluster.

Installing the controller

I run it as an Argo Application (early sync-wave) into kube-system, with two values that matter:

# Helm values for the sealed-secrets chart
fullnameOverride: sealed-secrets-controller   # matches kubeseal's defaults, so no extra flags
keyrenewperiod: "0"                            # disable key rotation (see backup section)

fullnameOverride makes the controller name and namespace line up with kubeseal’s defaults, so sealing commands need no extra arguments. Remember to add the chart’s Helm repo to your AppProject sourceRepos, or Argo refuses to pull it.

Sealing a secret

Generate the Secret locally with --dry-run (never sent to the cluster), pipe it through kubeseal, and write the result into your kustomize base:

kubectl create secret generic app-secret -n app-prod `
  --from-literal="DATABASE_URL=$dbUrl" `
  --dry-run=client -o yaml `
| kubeseal --controller-name sealed-secrets-controller --controller-namespace kube-system --format yaml `
| Out-File apps\app\base\sealed-app-secret.yaml -Encoding ascii

Two things to get right:

• Name plus namespace are baked into the ciphertext. By default SealedSecrets uses strict scope, so the encryption is bound to the exact name and namespace. If the final deployment uses different ones, it will not decrypt. Get them right at seal time.
• On Windows, do not use > redirection. PowerShell’s > writes UTF-16, which then breaks kubectl, kustomize, and git parsing. Use Out-File -Encoding ascii (SealedSecret content is all base64/ASCII, so ASCII is safe and BOM-free).

Then add the sealed file to the base kustomization.yaml resources, and Argo deploys it like anything else.

Handing off from a manual secret

If a manually-created Secret of the same name already exists (with no ownerReference), the controller refuses to clobber it:

Resource "app-secret" already exists and is not managed by SealedSecret

The fix has a subtlety. Delete the manual secret, but also nudge the controller, because it is watch-driven and only wakes on SealedSecret changes; deleting the target Secret is not such a change, so its status stays stuck on the old error:

kubectl delete secret app-secret -n app-prod
kubectl annotate sealedsecret app-secret -n app-prod nudge=1 --overwrite   # manufacture an update event
kubectl get secret app-secret -n app-prod -o jsonpath="{.metadata.ownerReferences[0].kind}"   # -> SealedSecret

Deleting the target Secret does not disturb a running pod; the values are already mounted, so this is safe to do live.

The one key you must back up

The controller’s private key is the only thing that can decrypt every SealedSecret in Git. It is the single irreplaceable local secret in the whole cluster. Back it up offline:

kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key=active `
  -o yaml | Out-File sealed-secrets-master.key.yaml -Encoding ascii

Store that in a password manager or offline vault, and never commit it. To restore into a rebuilt cluster, apply it before the controller starts, so the controller reuses the old key instead of generating a new one:

kubectl apply -f sealed-secrets-master.key.yaml   # then install/start the controller

About rotation: the controller rotates keys every 30 days by default (new active key, old keys retained to decrypt old ciphertext). That means a backup taken before a rotation lacks the newest key, so you would re-back-up after each rotation. I set keyrenewperiod: "0" to disable rotation, so the key never changes and one backup is valid forever. The tradeoff is losing rotation’s forward secrecy, which is acceptable at personal scale. Because of this, a backup taken today still decrypts secrets I seal months from now.

You can even decrypt offline, without a cluster, straight from the backup:

kubeseal --recovery-unseal --recovery-private-key master.key -f apps\app\base\sealed-app-secret.yaml -o yaml

One placement rule

Keep a SealedSecret in the same app and the same sync-wave as the workload that consumes it. If the secret lands in a later wave than its pod, a fresh cluster bootstrap starts the pod before the secret exists, and it CrashLoops until the secret shows up. For a webhook whose secret has to arrive with it, I put the SealedSecret right inside that component’s chart templates/ (as plain static YAML, no templating), so they deploy together.

What’s next

Secrets are now in Git, encrypted, with a clear backup story. Part 6 tackles state: why I keep it out of the cluster entirely, on a shared managed Postgres, and the storage options I weighed for the rest.