Troubleshooting Guide
Common Issues
“project root not found”
Bosun searches upward for bosun/ or manifest/ directory.
- Ensure you’re inside a bosun project
- Or specify path:
bosun --root /path/to/project
“connect to docker: …”
- Check Docker is running:
docker ps - Check Docker socket permissions
- On Linux:
sudo usermod -aG docker $USER
“sops decrypt failed”
Bosun classifies failures without printing raw SOPS errors, key identifiers, encrypted values, or decrypted MACs:
- SOPS integrity verification failed — the file or its MAC may have been modified. Restore the encrypted file from a trusted source or re-encrypt it; rotating the Age key will not repair corrupted ciphertext.
- SOPS decryption key unavailable — verify that
SOPS_AGE_KEYorSOPS_AGE_KEY_FILEcontains an identity matching the file recipients and that the key file is a regular, non-empty file containing a parseable Age identity. Bosun rejects an invalid key path before calling SOPS. If a container path is a directory, pre-create the host key file before mounting it; Docker can create a directory when a bind-mount source is missing. - Malformed SOPS encrypted data — validate or re-encrypt the file with SOPS; an encrypted value or metadata field is not decodable.
- SOPS decryption failed — validate the file with SOPS and verify the Age key when the failure cannot be safely classified further.
Set BOSUN_LOG_LEVEL=debug for the sanitized failure category and file context.
Bosun never logs the raw upstream decryption error, even at debug level.
“docker compose: command not found”
Bosun requires Docker Compose v2:
- Install: https://docs.docker.com/compose/install/
- Verify:
docker compose version
“docker compose up timed out”
BOSUN_COMPOSE_UP_TIMEOUT bounds each compose-up operation (default 10m).
During a local deploy each compose file gets its own deadline, measured from
when that file starts, and the daemon’s BOSUN_RECONCILE_TIMEOUT clock pauses
while compose up runs, so a slow file cannot run later files out of time (#727).
BOSUN_RECONCILE_MAX_TIMEOUT (default 30m) still caps the cycle, compose up
included. When it fires, the file running at that moment fails, and later
files are neither started nor rolled back (their containers were never
touched); the orphan pass is skipped. Every backup extraction and rollback in
the cycle, including the health-gate rollback and cleanup that started before
the cap, ends by the cap’s deadline plus BOSUN_RECONCILE_ROLLBACK_GRACE
(default 5m). A cycle therefore ends within the cap plus the grace. Errors that say compose up not started: reconcile cycle deadline passed mean the cap was reached: raise it, or find the slow
files. Otherwise, several files timing out within seconds of each other no
longer points to a shared budget: check what each named file waits on (image
pulls, dependencies).
When that deadline expires, Bosun signals the Docker CLI and Compose plugin so
they cancel the daemon request, waits up to five seconds for a graceful exit,
and then force-kills unresponsive local processes. The command can therefore
return a few seconds after the configured timeout, but container startup should
not continue in the background after Bosun reports the failure.
If startup continues, capture docker compose version, the Bosun error, and
daemon events from docker events --since <timestamp> when reporting the bug.
Daemon shutdown waits on a reconciliation
On SIGTERM, SIGINT, or parent-context cancellation, Bosun cancels reconciles
accepted through webhooks, the Unix socket, TCP, and /api/trigger, then waits
up to BOSUN_SHUTDOWN_TIMEOUT (default 30s) for their tracked goroutines to
unwind. New trigger requests receive 503 after shutdown starts. If cleanup
exceeds the timeout, Bosun logs Shutdown timeout waiting for background goroutines and finishes shutdown rather than hanging indefinitely; inspect the
preceding reconcile logs to find the operation that ignored cancellation.
If shutdown arrives during a local file sync, Bosun stops before the next
directory creation, atomic file replacement, or managed stale-file deletion.
It may wait briefly to flush and verify a file whose atomic rename already
completed; later files and deletions remain untouched. A temp-file copy that is
still in progress is interrupted and cleaned up. After the daemon restarts,
rerun the reconcile normally—the content-hash pass safely skips files that
already reached their intended bytes and resumes the remaining work. If Bosun
was staging a managed file-to-directory or directory-to-file replacement, it
also removes that private transition stage on cancellation so the retry is not
blocked by a leftover .bosun-transition-stage path. In standard atomic-swap
mode, cancellation after the live directory is moved aside restores that
original directory before returning and removes the unpublished temp tree.
Cancellation during template rendering also stops before the next staging directory creation or rendered-output rename. Bosun removes an unpublished template temp file and preserves any prior rendered output; if cancellation arrives after the old staging tree was cleared, the empty staging slot is not recreated and the next reconcile renders it from scratch.
SSH connection failures
- Test manually:
ssh user@host exit - Check SSH key is loaded:
ssh-add -l - For an SSH Git repository, Bosun tries
SSH_AUTH_SOCKfirst, thenBOSUN_SSH_KEYand the conventional key paths. An agent takes precedence only when it returns at least one signer; Bosun closes an empty or unreadable agent connection and continues to key files. The repository’s SSH username is preserved for both SCP-like andssh://URLs. - When
BOSUN_SSH_KEYis set, it must name a regular, non-empty, parseable private key file. Bosun fails before Git network access if it is missing, a directory, empty, or malformed. If an existing conventional key candidate such as/config/deploy-keyis unusable, Bosun tries later candidates and reports the invalid path when none succeeds. Pre-create Docker bind-mount source files; mounting a missing host source can create a directory at the container path. - If no agent or key is usable, Bosun rejects the SSH repository configuration before network access instead of allowing go-git to continue with nil auth.
- Configure host-key trust before connecting: set
BOSUN_SSH_KNOWN_HOSTSor mount/config/known_hosts. Git fails before network access when neither exists or the selected file is malformed; remote deploy refuses an unpinned host with strict OpenSSH checking. UseBOSUN_SSH_INSECURE_HOST_KEY=trueonly for a deliberate one-time bootstrap. - Verify host is reachable:
ping host
Deploy reports success but files unchanged
If bosun reconcile returns success: true and docker compose up exits 0, but the destination files at /mnt/user/appdata/<path> haven’t been updated (compare mtimes, or grep for an expected token from the new template), one of two invariant errors will now surface the cause instead of letting the deploy claim success silently. Both were added in response to GH#214.
Invariant 1 — declared-state invariant: no declared services in staging compose directory
The render step produced no parseable services in <staging>/compose/. Either templates failed to write to the expected location, the compose dir is genuinely empty, or all files in it are unparseable YAML.
-
For genuinely empty repos: set
BOSUN_ALLOW_EMPTY_DECLARED_STATE=trueto opt out — the reconciler will log atWarnlevel (withoverride=true) and continue. -
For misconfigured staging paths (compose dir missing entirely): the error is unconditionally fatal — no override applies. Check
BOSUN_INFRA_DIRand the rendered staging tree. When the configured infra dir has nocompose/but a sibling directory does, the error now names the candidate and suggests the fix, e.g.:declared-state invariant: staging compose directory does not exist: /app/staging/compose (compose/ found under sibling dir(s): unraid) — did you mean BOSUN_INFRA_DIR="unraid"?This is the GH#214 root cause:
BOSUN_INFRA_DIR="."whilecompose/andappdata/live underunraid/. SetBOSUN_INFRA_DIRto the named directory so render, discovery, and deploy all resolve the same infra root.
Invariant 2 — deploy destination paths are present, fresh, and type-correct
The deploy sync step claimed success but the destination doesn’t reflect it. Three shapes trip this: (1) a created directory or written file is missing or has an mtime older than the reconcile start, (2) a recorded path no longer has its expected type — directory or regular file — including when a symlink occupies it, or (3) no regular file was written against a source containing regular files and a source file is missing from — or holds stale bytes at — the destination. The third check still runs when the change set contains only newly created directories, so an empty-directory write cannot mask a silent file-sync failure. It uses SHA-256 content equality rather than existence: a destination that already byte-matches the source is a legitimate no-op and does not fail (fixed in GH#330), but a file that exists at the right path with outdated content does fail. The mismatch=… field names the first absent-or-differing destination path. Symlinks in the source are skipped — they are never deployed, so they impose no requirement.
To debug:
BOSUN_LOG_LEVEL=debug bosun reconcile
The per-file logs from internal/fileutil will show wrote src=… dst=… bytes=N for every actual write and skipped src=… dst=… reason=hash_match for every skip. Compare against the destination’s mtime on disk.
Emergency escape hatch (do NOT leave on):
BOSUN_SKIP_DEPLOY_INVARIANT=true bosun reconcile
The reconciler will log a Warn with override=true so the override is visible in monitoring. File a bug if you needed this — it indicates the invariant is misfiring or the underlying sync bug is reproducing.
Removed post-sync hooks still run
On current versions, removing the root post_sync_hooks key, setting it to [], or committing a valid empty bosun.yaml clears file-sourced hooks on the next successful reload. A missing or malformed config intentionally retains the last effective hooks because it is not a valid snapshot. Check the reload log for hooks_outcome, hooks_source, and target; command arguments are intentionally redacted.
For target hooks, an omitted key inherits root and explicit post_sync_hooks: [] disables inheritance. Removing a target descriptor drops its operational hook override immediately, but restart the daemon to remove the target from the running topology. BOSUN_POST_SYNC_HOOKS and target hooks supplied by BOSUN_TARGETS are environment-owned; remove or change those environment values and restart Bosun rather than editing bosun.yaml.
Post-sync hook never runs after deploy paths change
Look for Deploy paths changed but no post-sync hook patterns matched. The warning
reports distinct, duplicate, empty, and missing pattern counts plus at most five
pattern and staging-relative path samples; evaluated and matched-path counts
make the zero-match outcome explicit. This usually exposes a typo or a missing
prefix such as appdata/. Absolute or traversal paths are redacted. No deploy paths changed; post-sync hooks have nothing to evaluate is a separate informational
outcome and does not indicate a pattern problem.
Git sync is wedged
Symptom: a reconcile cycle runs far longer than any configured bound, holding the reconcile lock and queueing triggers behind it. A failure alert may arrive naming a timeout much shorter than the run’s actual duration.
Which phases are bounded. Bosun bounds what it can reach:
| Phase | Bounded by | Enforced |
|---|---|---|
| TCP dial | GitSSHDialTimeout (30s), capped by the operation’s remaining budget |
Yes |
| Between protocol steps | GitFetchTimeout (2m) / GitCloneTimeout (5m) |
Yes |
| SSH handshake | — | No |
| Packfile transfer | — | No |
The last two are a dependency limit, not an oversight. Bounding them needs a
deadline on the connection itself, which needs a custom transport.Transport,
and go-git’s only implementation of that session layer lives in an internal/
package bosun cannot import. Setting ssh.ClientConfig.Timeout does not help:
the transport applies it to the dial context and then hands the raw connection
to ssh.NewClientConn, which honors no deadline. Tracked in
#655.
Identifying it. A bounded timeout logs at error level with the measured elapsed time next to the bound that expired:
{"level":"error","component":"git","operation":"fetch","url":"ssh://git@github.com/owner/repo.git",
"branch":"main","elapsed_ms":30004,"timeout_ms":30000,"message":"Git fetch timed out"}
An unbounded stall is the absence of that line: the cycle simply does not
finish. Compare the run’s duration_ms against the configured bounds —
duration_ms covers the whole run, not just the fetch, so a large value alone
does not prove where the time went. The elapsed_ms field is what settles it.
docker logs --since 2h --timestamps bosun | grep -E 'timed out|duration_ms'
Clearing it. Restart the daemon: docker restart bosun. The reconcile lock
is released on process exit, and the next cycle starts clean. Nothing is lost —
a wedged fetch never reached the deploy stage.
Note that an error naming a timeout is not by itself evidence the timeout was
enforced. Before this behaviour existed, git fetch timed out after 2m0s
appeared on a run lasting 16m30s, and git clone timed out after 5m0s could
appear when GitCloneTimeout had never been applied at all. Errors now report
measured elapsed time and name the bound that actually expired.
A webhook request you cannot account for
The request log carries two address fields, and they mean different things:
remote_addr— always present, always the observed connection peer. A fact.forwarded_for— the firstX-Forwarded-Forelement, recorded only whenremote_addris a configured trusted proxy. A claim.
They are never collapsed and the header is never preferred. BOSUN_LISTEN_ADDR
binds all interfaces by design, so any host that can reach the daemon may send a
well-formed X-Forwarded-For; parsing it as an IP does not make it true.
Set BOSUN_TRUSTED_PROXIES to a comma-separated list of IP addresses or CIDR
prefixes to enable forwarded_for. It defaults to empty — trust nothing — and
an unparseable entry is refused rather than ignored, so a typo cannot silently
disable attribution.
To find who sent an unexpected request, grep the path rather than the field name; once the field exists it matches every request line:
docker logs --since 1h bosun | grep '"url":"/some/unexpected/path"'
Upgrade script verdicts
scripts/upgrade-bosun.sh ends with one VERDICT: line and a matching exit code. Exit 1 means the upgrade was rolled back, not “a check failed”. The NAS keeps one line per run in /mnt/user/appdata/bosun-upgrade/history.log. Failure detail does not live beside it — see Where failure detail lands below; the run prints the path it used.
| Exit | Verdict | What happened | Next step |
|---|---|---|---|
| 0 | UPGRADED |
The candidate completed its first reconcile with no error | Nothing |
| 0 | ALREADY-CURRENT |
The running digest already equals the pin | Nothing |
| 0 | RENDER-IDENTICAL / RENDER-DIFFERS / RENDER-OK-NO-BASELINE (dry run) |
Shadow render only; nothing changed | Review any listed differences, then run without --dry-run |
| 0 | DECLINED at … |
You answered no at the cutover prompt | Nothing changed |
| 1 | ROLLED-BACK |
The candidate failed its watch; the previous version is back and healthy | Revert the pin PR in homelab, then delete rollback.override.yml on the NAS. Read the run’s <run>-candidate.log (path printed by the run) |
| 2 | FAULT-NOT-UPGRADE |
The previous version also fails after rollback | The upgrade is not the cause. Follow homelab docs/runbooks/bosun-deploys-blocked.md |
| 3 | HALF-CHANGED |
The rollback itself did not start | Run the printed docker compose … up -d command on the NAS. The anchor image is bosun:rollback-<version>; do not prune it |
| 4 | HARNESS-INVALID |
The running version failed its own shadow render | The repo or the harness is broken, not the candidate. Read the run’s <run>-shadow-incumbent.log (path printed by the run) |
| 5 | CANDIDATE-FAILED |
The candidate failed its shadow render; nothing changed | Read the run’s <run>-shadow-candidate.log (path printed by the run) |
| 5 | CANDIDATE-FAILED (provenance) |
No attestation from bosun’s release workflow for that digest | Check the pin names a real release digest. Never bypass this for an upgrade |
| 64 | usage or config | Unpinned image, bad flag, missing container or compose file, malformed state file, or --dry-run while an interrupted upgrade is recorded |
Fix what the error names; nothing changed. A --dry-run never resumes a cutover: run without it to finish |
| 5 | CANDIDATE-FAILED (tag mismatch) |
The pin’s X.Y.Z tag and the image’s bosun --version disagree |
The digest belongs to another release. Fix the pin; nothing changed |
| 3 | HALF-CHANGED (pin moved) |
The pin changed to a third image while an upgrade was interrupted | Decide which image should run, recreate by hand, then remove the state file beside history.log. A pin reverted to the old version is not this case: the re-run rolls back on its own |
| 75 | transient / LOCKED / CONNECTION-LOST / INTERRUPTED-BEFORE-CUTOVER |
Pull failed, repo moved mid-render, a live run holds the lock, ssh dropped, or a run stopped before the old container was replaced | Re-run. LOCKED names the live run’s pid: after a dropped connection the NAS side keeps going and finishes on its own, so wait for it. A lock whose owner is gone (killed, or the NAS rebooted) is reclaimed automatically, and the re-run resumes any recorded cutover or rollback |
| 129/130/143 | INTERRUPTED by HUP/INT/TERM |
The remote script got a signal | Re-run the wrapper; a recorded cutover or rollback resumes |
A history line that ends in [provenance skipped: drill] came from a --skip-provenance-for-drill run. One ending in [provenance: not checked by wrapper] came from running the NAS script directly, which skips the Mac-side provenance check.
Where failure detail lands
Both kinds — a failed shadow render’s last 60 log lines, and the daemon-side detail after a failed watch (daemon-status, restart count, the last 40 log lines) — go to a directory the run creates for itself as /tmp/bosun-canary-failures.XXXXXX, where the suffix is random per run.
The path is not fixed, so read it from the run’s output — the run prints the file it wrote. Listing /tmp/bosun-canary-failures.* finds it too, along with any left by earlier runs. The name is unguessable on purpose: /tmp is shared and the script runs as root, so a fixed name would let any local user pre-create the directory, keep ownership, and read these files.
That is RAM on Unraid, so it is gone at reboot. Copy a file elsewhere yourself if you need it to survive. The reason is that either can quote a rendered secret: a template error carries the value it failed on, and the daemon runs the same template pipeline the shadow does. /mnt/user is array-backed and copied by the appdata backup, so it is the wrong home for them.
After a hard kill. The script cleans up on exit and on HUP/INT/TERM, but not on SIGKILL. If a run is killed outright, remove the rendered tree by hand: rm -rf /tmp/bosun-canary.*. The next run’s preflight also sweeps it.
Downgrades. A older release is a valid, signed image, so provenance accepts it. The script prints WARNING this is a DOWNGRADE and always asks, even with --yes.
Debug Mode
Set verbose output:
bosun --verbose provision mystack
Getting Help
- GitHub Issues: https://github.com/cameronsjo/bosun/issues
- Run diagnostics:
bosun doctor