1 min read lxd, canonical, containers

Wiring up snapctl set-health for LXD cluster refreshes

A PR I got merged into LXD today that reports snap health during cluster-wide refreshes, so a member stuck waiting for the rest of the cluster to catch up shows up as "waiting" instead of silently looking fine.

canonical/lxd#18856 merged today, closing canonical/lxd#15721. It adds snapctl set-health reporting during cluster-wide snap refreshes, so a member that’s stuck waiting for the rest of the cluster to align shows up as waiting instead of looking perfectly healthy while it does nothing.

The problem

Simon filed the issue after noticing that a cluster-wide refresh gives you almost no signal about what’s happening. When a member’s snap gets refreshed, LXD has to wait until every other member is running a compatible schema/API version before it can rejoin the cluster and start serving again. That wait can take a while on a big cluster, and from the outside — snap health lxd, monitoring that watches it, anything checking on the daemon — there was nothing to distinguish “mid-wait” from “just running normally.” The daemon looked healthy the entire time it was blocked.

snapctl set-health is exactly the hook for this: a snap can report its own status (okay, waiting, blocked, error) to snapd, and that shows up in snap health and anything else watching it.

The solution

The fix is a small SnapSetHealth helper in the new shared/snap.go, wrapping snapctl set-health with four status constants (okay, waiting, blocked, error) and a no-op fallback when LXD isn’t running inside the snap at all. It’s called from two spots: runUpdate() in lxd/cluster/upgrade.go sets waiting right before triggering a member’s refresh and error if the update fails, and init() in lxd/daemon.go sets waiting while blocked on other members aligning their schema/API versions, then okay once startup actually finishes.