Troubleshooting Guide

Common Issues

“project root not found”

Bosun searches upward for bosun/ or manifest/ directory.

“connect to docker: …”

“sops decrypt failed”

Bosun classifies failures without printing raw SOPS errors, key identifiers, encrypted values, or decrypted MACs:

Set BOSUN_LOG_LEVEL=debug for the sanitized failure category and file context. Bosun never logs the raw upstream decryption error, even at debug level.

“docker compose: command not found”

Bosun requires Docker Compose v2:

“docker compose up timed out”

BOSUN_COMPOSE_UP_TIMEOUT bounds each compose-up operation (default 10m). During a local deploy each compose file gets its own deadline, measured from when that file starts, and the daemon’s BOSUN_RECONCILE_TIMEOUT clock pauses while compose up runs, so a slow file cannot run later files out of time (#727). BOSUN_RECONCILE_MAX_TIMEOUT (default 30m) still caps the cycle, compose up included. When it fires, the file running at that moment fails, and later files are neither started nor rolled back (their containers were never touched); the orphan pass is skipped. Every backup extraction and rollback in the cycle, including the health-gate rollback and cleanup that started before the cap, ends by the cap’s deadline plus BOSUN_RECONCILE_ROLLBACK_GRACE (default 5m). A cycle therefore ends within the cap plus the grace. Errors that say compose up not started: reconcile cycle deadline passed mean the cap was reached: raise it, or find the slow files. Otherwise, several files timing out within seconds of each other no longer points to a shared budget: check what each named file waits on (image pulls, dependencies). When that deadline expires, Bosun signals the Docker CLI and Compose plugin so they cancel the daemon request, waits up to five seconds for a graceful exit, and then force-kills unresponsive local processes. The command can therefore return a few seconds after the configured timeout, but container startup should not continue in the background after Bosun reports the failure.

If startup continues, capture docker compose version, the Bosun error, and daemon events from docker events --since <timestamp> when reporting the bug.

Daemon shutdown waits on a reconciliation

On SIGTERM, SIGINT, or parent-context cancellation, Bosun cancels reconciles accepted through webhooks, the Unix socket, TCP, and /api/trigger, then waits up to BOSUN_SHUTDOWN_TIMEOUT (default 30s) for their tracked goroutines to unwind. New trigger requests receive 503 after shutdown starts. If cleanup exceeds the timeout, Bosun logs Shutdown timeout waiting for background goroutines and finishes shutdown rather than hanging indefinitely; inspect the preceding reconcile logs to find the operation that ignored cancellation.

If shutdown arrives during a local file sync, Bosun stops before the next directory creation, atomic file replacement, or managed stale-file deletion. It may wait briefly to flush and verify a file whose atomic rename already completed; later files and deletions remain untouched. A temp-file copy that is still in progress is interrupted and cleaned up. After the daemon restarts, rerun the reconcile normally—the content-hash pass safely skips files that already reached their intended bytes and resumes the remaining work. If Bosun was staging a managed file-to-directory or directory-to-file replacement, it also removes that private transition stage on cancellation so the retry is not blocked by a leftover .bosun-transition-stage path. In standard atomic-swap mode, cancellation after the live directory is moved aside restores that original directory before returning and removes the unpublished temp tree.

Cancellation during template rendering also stops before the next staging directory creation or rendered-output rename. Bosun removes an unpublished template temp file and preserves any prior rendered output; if cancellation arrives after the old staging tree was cleared, the empty staging slot is not recreated and the next reconcile renders it from scratch.

SSH connection failures

Deploy reports success but files unchanged

If bosun reconcile returns success: true and docker compose up exits 0, but the destination files at /mnt/user/appdata/<path> haven’t been updated (compare mtimes, or grep for an expected token from the new template), one of two invariant errors will now surface the cause instead of letting the deploy claim success silently. Both were added in response to GH#214.

Invariant 1 — declared-state invariant: no declared services in staging compose directory

The render step produced no parseable services in <staging>/compose/. Either templates failed to write to the expected location, the compose dir is genuinely empty, or all files in it are unparseable YAML.

Invariant 2 — deploy destination paths are present, fresh, and type-correct

The deploy sync step claimed success but the destination doesn’t reflect it. Three shapes trip this: (1) a created directory or written file is missing or has an mtime older than the reconcile start, (2) a recorded path no longer has its expected type — directory or regular file — including when a symlink occupies it, or (3) no regular file was written against a source containing regular files and a source file is missing from — or holds stale bytes at — the destination. The third check still runs when the change set contains only newly created directories, so an empty-directory write cannot mask a silent file-sync failure. It uses SHA-256 content equality rather than existence: a destination that already byte-matches the source is a legitimate no-op and does not fail (fixed in GH#330), but a file that exists at the right path with outdated content does fail. The mismatch=… field names the first absent-or-differing destination path. Symlinks in the source are skipped — they are never deployed, so they impose no requirement.

To debug:

BOSUN_LOG_LEVEL=debug bosun reconcile

The per-file logs from internal/fileutil will show wrote src=… dst=… bytes=N for every actual write and skipped src=… dst=… reason=hash_match for every skip. Compare against the destination’s mtime on disk.

Emergency escape hatch (do NOT leave on):

BOSUN_SKIP_DEPLOY_INVARIANT=true bosun reconcile

The reconciler will log a Warn with override=true so the override is visible in monitoring. File a bug if you needed this — it indicates the invariant is misfiring or the underlying sync bug is reproducing.

Removed post-sync hooks still run

On current versions, removing the root post_sync_hooks key, setting it to [], or committing a valid empty bosun.yaml clears file-sourced hooks on the next successful reload. A missing or malformed config intentionally retains the last effective hooks because it is not a valid snapshot. Check the reload log for hooks_outcome, hooks_source, and target; command arguments are intentionally redacted.

For target hooks, an omitted key inherits root and explicit post_sync_hooks: [] disables inheritance. Removing a target descriptor drops its operational hook override immediately, but restart the daemon to remove the target from the running topology. BOSUN_POST_SYNC_HOOKS and target hooks supplied by BOSUN_TARGETS are environment-owned; remove or change those environment values and restart Bosun rather than editing bosun.yaml.

Post-sync hook never runs after deploy paths change

Look for Deploy paths changed but no post-sync hook patterns matched. The warning reports distinct, duplicate, empty, and missing pattern counts plus at most five pattern and staging-relative path samples; evaluated and matched-path counts make the zero-match outcome explicit. This usually exposes a typo or a missing prefix such as appdata/. Absolute or traversal paths are redacted. No deploy paths changed; post-sync hooks have nothing to evaluate is a separate informational outcome and does not indicate a pattern problem.

Git sync is wedged

Symptom: a reconcile cycle runs far longer than any configured bound, holding the reconcile lock and queueing triggers behind it. A failure alert may arrive naming a timeout much shorter than the run’s actual duration.

Which phases are bounded. Bosun bounds what it can reach:

Phase Bounded by Enforced
TCP dial GitSSHDialTimeout (30s), capped by the operation’s remaining budget Yes
Between protocol steps GitFetchTimeout (2m) / GitCloneTimeout (5m) Yes
SSH handshake — No
Packfile transfer — No

The last two are a dependency limit, not an oversight. Bounding them needs a deadline on the connection itself, which needs a custom transport.Transport, and go-git’s only implementation of that session layer lives in an internal/ package bosun cannot import. Setting ssh.ClientConfig.Timeout does not help: the transport applies it to the dial context and then hands the raw connection to ssh.NewClientConn, which honors no deadline. Tracked in #655.

Identifying it. A bounded timeout logs at error level with the measured elapsed time next to the bound that expired:

{"level":"error","component":"git","operation":"fetch","url":"ssh://git@github.com/owner/repo.git",
 "branch":"main","elapsed_ms":30004,"timeout_ms":30000,"message":"Git fetch timed out"}

An unbounded stall is the absence of that line: the cycle simply does not finish. Compare the run’s duration_ms against the configured bounds — duration_ms covers the whole run, not just the fetch, so a large value alone does not prove where the time went. The elapsed_ms field is what settles it.

docker logs --since 2h --timestamps bosun | grep -E 'timed out|duration_ms'

Clearing it. Restart the daemon: docker restart bosun. The reconcile lock is released on process exit, and the next cycle starts clean. Nothing is lost — a wedged fetch never reached the deploy stage.

Note that an error naming a timeout is not by itself evidence the timeout was enforced. Before this behaviour existed, git fetch timed out after 2m0s appeared on a run lasting 16m30s, and git clone timed out after 5m0s could appear when GitCloneTimeout had never been applied at all. Errors now report measured elapsed time and name the bound that actually expired.

A webhook request you cannot account for

The request log carries two address fields, and they mean different things:

They are never collapsed and the header is never preferred. BOSUN_LISTEN_ADDR binds all interfaces by design, so any host that can reach the daemon may send a well-formed X-Forwarded-For; parsing it as an IP does not make it true.

Set BOSUN_TRUSTED_PROXIES to a comma-separated list of IP addresses or CIDR prefixes to enable forwarded_for. It defaults to empty — trust nothing — and an unparseable entry is refused rather than ignored, so a typo cannot silently disable attribution.

To find who sent an unexpected request, grep the path rather than the field name; once the field exists it matches every request line:

docker logs --since 1h bosun | grep '"url":"/some/unexpected/path"'

Upgrade script verdicts

scripts/upgrade-bosun.sh ends with one VERDICT: line and a matching exit code. Exit 1 means the upgrade was rolled back, not “a check failed”. The NAS keeps one line per run in /mnt/user/appdata/bosun-upgrade/history.log. Failure detail does not live beside it — see Where failure detail lands below; the run prints the path it used.

Exit Verdict What happened Next step
0 UPGRADED The candidate completed its first reconcile with no error Nothing
0 ALREADY-CURRENT The running digest already equals the pin Nothing
0 RENDER-IDENTICAL / RENDER-DIFFERS / RENDER-OK-NO-BASELINE (dry run) Shadow render only; nothing changed Review any listed differences, then run without --dry-run
0 DECLINED at … You answered no at the cutover prompt Nothing changed
1 ROLLED-BACK The candidate failed its watch; the previous version is back and healthy Revert the pin PR in homelab, then delete rollback.override.yml on the NAS. Read the run’s <run>-candidate.log (path printed by the run)
2 FAULT-NOT-UPGRADE The previous version also fails after rollback The upgrade is not the cause. Follow homelab docs/runbooks/bosun-deploys-blocked.md
3 HALF-CHANGED The rollback itself did not start Run the printed docker compose … up -d command on the NAS. The anchor image is bosun:rollback-<version>; do not prune it
4 HARNESS-INVALID The running version failed its own shadow render The repo or the harness is broken, not the candidate. Read the run’s <run>-shadow-incumbent.log (path printed by the run)
5 CANDIDATE-FAILED The candidate failed its shadow render; nothing changed Read the run’s <run>-shadow-candidate.log (path printed by the run)
5 CANDIDATE-FAILED (provenance) No attestation from bosun’s release workflow for that digest Check the pin names a real release digest. Never bypass this for an upgrade
64 usage or config Unpinned image, bad flag, missing container or compose file, malformed state file, or --dry-run while an interrupted upgrade is recorded Fix what the error names; nothing changed. A --dry-run never resumes a cutover: run without it to finish
5 CANDIDATE-FAILED (tag mismatch) The pin’s X.Y.Z tag and the image’s bosun --version disagree The digest belongs to another release. Fix the pin; nothing changed
3 HALF-CHANGED (pin moved) The pin changed to a third image while an upgrade was interrupted Decide which image should run, recreate by hand, then remove the state file beside history.log. A pin reverted to the old version is not this case: the re-run rolls back on its own
75 transient / LOCKED / CONNECTION-LOST / INTERRUPTED-BEFORE-CUTOVER Pull failed, repo moved mid-render, a live run holds the lock, ssh dropped, or a run stopped before the old container was replaced Re-run. LOCKED names the live run’s pid: after a dropped connection the NAS side keeps going and finishes on its own, so wait for it. A lock whose owner is gone (killed, or the NAS rebooted) is reclaimed automatically, and the re-run resumes any recorded cutover or rollback
129/130/143 INTERRUPTED by HUP/INT/TERM The remote script got a signal Re-run the wrapper; a recorded cutover or rollback resumes

A history line that ends in [provenance skipped: drill] came from a --skip-provenance-for-drill run. One ending in [provenance: not checked by wrapper] came from running the NAS script directly, which skips the Mac-side provenance check.

Where failure detail lands

Both kinds — a failed shadow render’s last 60 log lines, and the daemon-side detail after a failed watch (daemon-status, restart count, the last 40 log lines) — go to a directory the run creates for itself as /tmp/bosun-canary-failures.XXXXXX, where the suffix is random per run.

The path is not fixed, so read it from the run’s output — the run prints the file it wrote. Listing /tmp/bosun-canary-failures.* finds it too, along with any left by earlier runs. The name is unguessable on purpose: /tmp is shared and the script runs as root, so a fixed name would let any local user pre-create the directory, keep ownership, and read these files.

That is RAM on Unraid, so it is gone at reboot. Copy a file elsewhere yourself if you need it to survive. The reason is that either can quote a rendered secret: a template error carries the value it failed on, and the daemon runs the same template pipeline the shadow does. /mnt/user is array-backed and copied by the appdata backup, so it is the wrong home for them.

After a hard kill. The script cleans up on exit and on HUP/INT/TERM, but not on SIGKILL. If a run is killed outright, remove the rendered tree by hand: rm -rf /tmp/bosun-canary.*. The next run’s preflight also sweeps it.

Downgrades. A older release is a valid, signed image, so provenance accepts it. The script prints WARNING this is a DOWNGRADE and always asks, even with --yes.

Debug Mode

Set verbose output:

bosun --verbose provision mystack

Getting Help