Strict Blog
Reliability & Alerts 6 min read · Updated November 2026

Preventing Silent Self-Hosted Runner Failures: 7-Tier Disk Cleanup, Preflight Doctor & Risk-Aware Alerts

Dedicated build machines rarely fail all at once—they erode quietly. DerivedData eats 180 GB, macOS puts the Ethernet port to sleep, one Mac auto-updates Xcode behind the others, or a Tailscale node key expires on a Friday night.

1. Progressive 7-tier disk reclamation

Whenever free disk drops below lowGiB (and autoCleanup is enabled), strict_maintenance.py runs reclaim(target_space) inside maintenance.lock and stops immediately after whichever tier restores free_space() >= targetGiB. Warm caches stay warm as long as headroom is healthy:

Strict 7-tier progressive disk reclamation stopping at targetGiB Tier 1 _diag logs _work/_temp Tier 2 npm / bun / cargo / brew Tier 3 Git-ignored workspaces Tier 4 Stale Xcode DerivedData Tier 5 Docker prune >24h idle Tier 6 (opt) APFS snaps, DeviceSupport Tier 7 (opt) User Trash sweep Early return as soon as free_space() >= targetGiB · Active Runner.Worker workspaces are always skipped

Every host also computes a 300-second cached storageBreakdown (estimate_reclaimable()) across runnerCaches, pkgCaches, xcode, and trash so operators can see reclaimable GiB before running cleanup—and invalidates that cache immediately after reclaim() runs. Opt-in Tiers 6 (deep-macos) and 7 (trash) stay off until explicitly enabled per host.

2. Nine preflight doctor checks & 12-toolchain drift

strict_doctor.py runs read-only, zero-sudo diagnostics cached for 300s (or on demand via strict doctor) across 9 failure modes that commonly break self-hosted fleets:

Check ID Platform What it catches before builds fail
dns-github macOS & Linux Flags unresolvable GitHub API endpoints (e.g., EC2/office DNS issues) using a 3s timeout
disk-headroom macOS & Linux Warns below lowGiB and fails below critical disk margin
apfs-unallocated macOS Parses diskutil apfs list to detect unexpanded EBS/APFS container partitions
macos-sleep macOS Inspects pmset sleep settings so build hosts don’t nap mid-workflow
macos-autologin macOS Checks com.apple.loginwindow.plist so user launchd runners resume after power loss
linux-linger Linux Verifies loginctl enable-linger so systemd --user runners survive SSH logout
runner-hooks macOS & Linux Verifies ACTIONS_RUNNER_HOOK_JOB_STARTED points to before-job.sh on every runner
xcode-license macOS Runs xcodebuild -checkFirstLaunchStatus to catch post-update license prompts
tailscale-health macOS & Linux Checks daemon state and flags expired or < 14-day expiring Tailscale node keys

In parallel, collect_toolchains() inventories 12 binaries (xcode, swift, node, bun, python, rust, go, docker, gh, ruby, java, runner), and computeToolchainDrift() compares same-OS hosts within repo pools plus xcode/swift across all macOS nodes.

3. Calibrated issue, at-risk & fleet alerting

Early self-hosted alert systems page on too little (only "machine offline") or too much (spamming every macOS warning on a developer Mac). Strict separates triggers cleanly in state.ts and alert-email.ts:

  • Machine issues (critical): offline (missed heartbeats > 105s), disk (free disk < lowGiB), and runner (any runner service stopped or unavailable).
  • Machines at risk (warning): pressure (RAM ≥ 90%, sustained 5m load load5 / cpuCores >= 2.0, compute-pressure draining, or disk within 5 GiB above lowGiB), doctor (triggers strictly on failCount > 0 so dev-Mac warnings don’t spam), and tailscale (node key expired, expiring within 14 days, or Tailscale disconnected on an online host).
  • Fleet issues (__fleet__): fleet triggers a warning on xcode/swift toolchain drift, or a critical alert across ≥ 2 reporting hosts if 0 fleet runners are ready or ≤ 50% of hosts are online.

Alerts dispatch transactional HTML emails from noreply@strict.ai via Cloudflare Email Sending (native Worker EMAIL binding in Cloud mode, or encrypted REST credentials in Standalone mode) and Slack/Discord/generic webhooks—complete with per-trigger opt-outs, failed-delivery state rollback for clean retries, and recovery notices once the host or fleet returns to green.