Self-Hosted k3s on Azure - Part 4: Wildcard TLS & a Custom DNS-01 Webhook

This is Part 4 of the Self-Hosted k3s on Azure series. Apps are now deployed by Argo (Part 3). This post gets them all onto trusted HTTPS with a single wildcard certificate, then removes the ingress single point of failure.

Placeholders: example.com for the domain, <public-ip> for the VIP. My DNS is hosted at a provider that has no built-in cert-manager solver, so I wrote my own webhook; I keep the provider generic below.

The idea: one wildcard cert, zero per-app work

Instead of issuing a certificate per app, I issue one wildcard, *.example.com, and make Traefik serve it as the cluster default. After that, a new app gets HTTPS by simply existing under the domain. No per-app certificate, no per-app issuer, nothing.

The catch: a wildcard can only be validated by ACME DNS-01 (proving control of the domain by writing a _acme-challenge TXT record). HTTP-01 cannot issue wildcards. And DNS-01 needs cert-manager to be able to create that TXT record through your DNS provider’s API.

When your DNS provider has no solver: write a webhook

cert-manager ships solvers for the big clouds. Mine is not one of them. cert-manager’s answer for that is a webhook solver: a tiny API server that implements two operations, Present (create the challenge TXT record) and CleanUp (delete it). You write it once for your provider.

The interface is small. In Go it is essentially:

type customSolver struct{ /* provider API client */ }

func (s *customSolver) Name() string { return "mydns" }

func (s *customSolver) Present(ch *acme.ChallengeRequest) error {
    // create TXT record: _acme-challenge.<zone>  ==  ch.Key
    return s.client.AddTXT(ch.ResolvedFQDN, ch.Key)
}

func (s *customSolver) CleanUp(ch *acme.ChallengeRequest) error {
    return s.client.DeleteTXT(ch.ResolvedFQDN, ch.Key)
}

func main() {
    cmd.RunWebhookServer(GroupName, &customSolver{ /* ... */ })
}

The provider credentials come from environment variables (fed by a Secret), not from the issuer config, so the issuer stays free of secrets. The webhook is packaged as its own small Helm chart and deployed by Argo like any other app, with CI bumping its image tag.

Wiring the issuer and the certificate

Two ClusterIssuers (staging and prod) reference the webhook as their DNS-01 solver:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata: { name: letsencrypt-prod }
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef: { name: letsencrypt-prod-account-key }
    solvers:
      - dns01:
          webhook:
            groupName: mydns.acme.example.com
            solverName: mydns

The Certificate itself lives in kube-system, not in an app namespace, so its lifecycle follows the platform and not any app that might get deleted:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata: { name: wildcard-example, namespace: kube-system }
spec:
  secretName: wildcard-example-tls
  issuerRef: { name: letsencrypt-prod, kind: ClusterIssuer }
  commonName: '*.example.com'
  dnsNames: ['*.example.com', 'example.com']

Always exercise staging first. Let’s Encrypt production is rate-limited; the staging issuer is not, and it validates the whole DNS-01 path without burning quota. Once staging works end to end, flip issuerRef to prod and let cert-manager re-issue.

Serving one cert everywhere: the Traefik default TLSStore

The certificate produces a Secret, and Secrets are namespaced, but an Ingress can only reference a Secret in its own namespace. Copying the wildcard into every namespace is ugly. Traefik solves it with a default TLSStore: a TLSStore named default in kube-system that points at the wildcard Secret and becomes the cluster-wide default certificate.

apiVersion: traefik.io/v1alpha1
kind: TLSStore
metadata: { name: default, namespace: kube-system }
spec:
  defaultCertificate:
    secretName: wildcard-example-tls

After that, an app’s Ingress just needs a tls block with no secretName. That declares “serve me over 443” and Traefik uses the default certificate:

spec:
  tls:
    - hosts: [app.example.com]     # no secretName on purpose
  rules:
    - host: app.example.com
      http: { paths: [ { path: /, pathType: Prefix, backend: { service: { name: app, port: { number: 80 } } } } ] }

Note there is no double TLS here. The Ingress tls block is an instruction to Traefik (terminate TLS once, at the edge). Traefik then talks plain HTTP to the pod. The moment a backend also insists on doing its own TLS and HTTPS redirect, you get the redirect loop from Part 3; keep backends on plain HTTP.

Making Traefik itself highly available

With three nodes behind one load balancer, I assumed the web path was HA. It was not. Traefik ships as a single replica, so it is a hidden single point of failure. I found out by rebooting the node running it. The Kubernetes API stayed up the whole time (etcd held quorum, the control plane really is HA), but the web path was down for roughly a minute until the Traefik pod came back. From the load balancer’s health probe view, all three nodes’ 80/443 probes went red at once, because they all forward to the same (now empty) Traefik Service.

k3s renders Traefik from a HelmChart via its built-in helm-controller, so you cannot just kubectl scale it (the controller reverts you). Instead you drop in a same-named HelmChartConfig whose values get deep-merged and re-applied:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata: { name: traefik, namespace: kube-system }
spec:
  valuesContent: |-
    deployment:
      replicas: 3
    affinity:                        # one per node, soft anti-affinity
      podAntiAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              topologyKey: kubernetes.io/hostname
              labelSelector: { matchLabels: { app.kubernetes.io/name: traefik } }
    podDisruptionBudget:
      enabled: true
      minAvailable: 2

Three replicas spread one-per-node, plus a PDB of minAvailable: 2, means losing any node leaves two Traefik endpoints, the surviving nodes’ probes stay green, and the web path has zero downtime.

Global HTTP to HTTPS redirect (and a lesson in silent failure)

I wanted all port-80 traffic to 308-redirect to 443, done once at the entrypoint rather than per-Ingress:

    ports:
      web:
        http:
          redirections:
            entryPoint:
              to: websecure
              scheme: https
              permanent: true      # 308

My first attempt used an older key, ports.web.redirectTo, which the chart had removed in a newer major version. Helm ran, no error, no pod restart, and port 80 still returned 200. The ports map is a merge target, so an unknown key is silently ignored. The lesson: when a Helm values change has no effect and no error, open the chart’s own values.yaml for that version and confirm the key, do not guess. Verify with:

curl.exe -sI --resolve app.example.com:80:<public-ip> http://app.example.com | Select-String "HTTP/|[Ll]ocation"
# HTTP/1.1 308 Permanent Redirect + Location: https://app.example.com/

What’s next

Every app now gets trusted HTTPS for free. Part 5 tackles the other half of committing your cluster to Git: secrets, with SealedSecrets, and the one key you must never lose.