Preventing Silent Self-Hosted Runner Failures: 7-Tier Disk Cleanup, Preflight Doctor & Risk-Aware Alerts
Dedicated build machines rarely fail all at once—they erode quietly.
DerivedData eats 180 GB, macOS puts the Ethernet port to sleep, one Mac
auto-updates Xcode behind the others, or a Tailscale node key expires on a Friday night.
1. Progressive 7-tier disk reclamation
Whenever free disk drops below lowGiB (and autoCleanup is
enabled), strict_maintenance.py runs
reclaim(target_space) inside maintenance.lock and stops
immediately after whichever tier restores free_space() >= targetGiB.
Warm caches stay warm as long as headroom is healthy:
Every host also computes a 300-second cached
storageBreakdown (estimate_reclaimable()) across
runnerCaches, pkgCaches, xcode, and
trash so operators can see reclaimable GiB before running cleanup—and
invalidates that cache immediately after reclaim() runs. Opt-in Tiers 6
(deep-macos) and 7 (trash) stay off until explicitly enabled
per host.
2. Nine preflight doctor checks & 12-toolchain drift
strict_doctor.py runs read-only, zero-sudo diagnostics cached for 300s
(or on demand via strict doctor) across 9 failure modes that commonly
break self-hosted fleets:
| Check ID | Platform | What it catches before builds fail |
|---|---|---|
dns-github |
macOS & Linux | Flags unresolvable GitHub API endpoints (e.g., EC2/office DNS issues) using a 3s timeout |
disk-headroom |
macOS & Linux | Warns below lowGiB and fails below critical disk margin |
apfs-unallocated |
macOS |
Parses diskutil apfs list to detect unexpanded EBS/APFS container
partitions
|
macos-sleep |
macOS |
Inspects pmset sleep settings so build hosts don’t nap
mid-workflow
|
macos-autologin |
macOS |
Checks com.apple.loginwindow.plist so user
launchd runners resume after power loss
|
linux-linger |
Linux |
Verifies loginctl enable-linger so
systemd --user runners survive SSH logout
|
runner-hooks |
macOS & Linux |
Verifies ACTIONS_RUNNER_HOOK_JOB_STARTED points to
before-job.sh on every runner
|
xcode-license |
macOS |
Runs xcodebuild -checkFirstLaunchStatus to catch post-update
license prompts
|
tailscale-health |
macOS & Linux | Checks daemon state and flags expired or < 14-day expiring Tailscale node keys |
In parallel, collect_toolchains() inventories 12 binaries
(xcode, swift, node, bun,
python, rust, go, docker,
gh, ruby, java, runner), and
computeToolchainDrift() compares same-OS hosts within repo pools plus
xcode/swift across all macOS nodes.
3. Calibrated issue, at-risk & fleet alerting
Early self-hosted alert systems page on too little (only "machine offline") or too
much (spamming every macOS warning on a developer Mac). Strict separates triggers
cleanly in state.ts and alert-email.ts:
-
Machine issues (critical):
offline(missed heartbeats > 105s),disk(free disk <lowGiB), andrunner(any runner service stopped or unavailable). -
Machines at risk (warning):
pressure(RAM ≥ 90%, sustained 5m loadload5 / cpuCores >= 2.0, compute-pressure draining, or disk within 5 GiB abovelowGiB),doctor(triggers strictly onfailCount > 0so dev-Mac warnings don’t spam), andtailscale(node key expired, expiring within 14 days, or Tailscale disconnected on an online host). -
Fleet issues (
__fleet__):fleettriggers a warning onxcode/swifttoolchain drift, or a critical alert across ≥ 2 reporting hosts if 0 fleet runners are ready or ≤ 50% of hosts are online.
Alerts dispatch transactional HTML emails from noreply@strict.ai via
Cloudflare Email Sending (native Worker EMAIL binding in Cloud mode, or
encrypted REST credentials in Standalone mode) and Slack/Discord/generic
webhooks—complete with per-trigger opt-outs, failed-delivery state rollback for clean
retries, and recovery notices once the host or fleet returns to green.