Shaping GitHub Actions Fleets Without a Proxy: Inside Strict’s Auto Load-Balancer
GitHub Actions assigns queued jobs directly to whichever matching
self-hosted runner polls first—with zero awareness of a machine’s free
disk, memory pressure, or CPU saturation.
Instead of rewriting workflows to route webhooks through a custom job broker, Strict
steers placement by dynamically pausing and re-admitting idle runner slots on each host.
Here is the exact algorithm running inside computeFleetDirective() and
reconcile_drain().
1. Why supply-side control beats proxying webhooks
A job proxy sits in the critical path of every workflow dispatch: if the webhook
consumer stalls or rate-limits, CI queues hang. By letting hosts run standard
persistent actions-runner listeners and shaping
available runner capacity cleanly via
launchctl bootout/bootstrap (macOS) and
systemctl --user stop/start (Linux):
-
Workflow YAML needs zero changes—GitHub matches
runs-onlabels natively against whichever slots are awake. - When every node is healthy, Strict adds zero scheduling latency.
- If the Strict server reboots or becomes unreachable, hosts automatically recover their own runners after 5 minutes.
2. Interactive Load-Balancer Simulator
Experiment with the exact TypeScript rules from computeFleetDirective().
Click a scenario preset or drag any host’s sliders to observe hysteresis, repo-pool
safety guards, proportional load-skew shedding, and deterministic
tie-breaking live.
Live Fleet Directive Evaluator
lowGiB=20 · targetGiB=40 · 2 slots/host
3. Four balance signals with enter/exit hysteresis
Only online hosts (seen within FRESH_MS = 105,000 ms) participate in
fleet evaluation. To keep runners from flapping back and forth near a threshold, every
signal in basePressure() and computeFleetDirective() uses
separate enter and exit bands:
| Reason | Enters pressure when | Stays latched until cleared at | Scope drained |
|---|---|---|---|
disk |
diskFree < lowGiB |
diskFree >= targetGiB |
All covered slots + runCleanup |
memory |
memUsedPct >= 85% |
memUsedPct <= 75% |
All covered slots |
cpu |
load1 / cpuCores >= 1.4 |
load1 / cpuCores <= 0.9 |
All covered slots |
load-skew |
busyCount >= 2 & idle healthy peer has
>= 8 GiB more free RAM
|
busyCount === 0 or peer free RAM gap < 4 GiB
|
Half of covered slots (floor(n/2)) |
4. Pool matching, headroom checks & anti-thrash tie-breaking
Draining the only Mac capable of building acme/release just because its
disk is filling up would strand jobs indefinitely in GitHub’s queue. Strict refuses to
drain unless peerCoversPool() confirms another online peer covers that
slot cleanly:
-
Repo-pool identity (
poolKey): Runners are grouped by their configured repository (runnerRepos[runnerName]), falling back to stripping any<deviceId>-host prefix. Peers only count as coverage if they hold anidleorbusynon-draining runner in the same pool. -
Never cascade onto overloaded peers (
peerHasHigherHeadroom): Strict never routes spillover onto a peer experiencingmemoryorcpupressure. Ondisk-only pressure, a disk-pressured peer is only eligible if it has strictly more free disk. For all active reasons, the target peer must have strictly better headroom on every active pressure axis. -
Deterministic symmetry breaking: If two hosts in the same pool hit
the exact same free disk and free memory simultaneously, comparing
peer.id < device.idensures only the lexically higher ID drains—the lower ID stays online to absorb the pool so both machines never mutually take themselves offline.
5. Proportional shedding & agent-side safety guarantees
Once directives reach the host agent, three local invariants keep running jobs and fleet capacity safe:
-
Proportional shedding under
load-skew: When a host has multiple slots busy and a roomier idle peer is waiting, Strict wants to balance concurrency—not empty the busy host. Underload-skewalone, it sheds onlyMath.floor(coveredCandidates.length / 2)slots, choosing candidates viadrainCandidatePriority(): already-draining (0), thenidle(1), thenstarting(2), thenbusy(3). -
In-flight jobs are never aborted: Before executing
launchctl bootoutorsystemctl --user stop,reconcile_drain()runsis_runner_active()to inspect liveRunner.Workerprocesses. Active slots finish their current GitHub job untouched and are drained only once they go idle. -
drained.jsonstopsself_heal()interference: Intentionally drained runners are recorded indrained.jsonso the maintenance timer’s crash-recovery loop doesn’t immediately restart a runner the load balancer just paused. -
5-minute offline recovery fail-safe: If a host loses connectivity
to the Strict control plane for more than 300 seconds (measured against
last-sync-ok.jsonor drain timestamps),recover_offline_drain()clears drains and brings runners back online automatically—provided free disk is at leastRESTART_SPACE(5 GiB).