GitOps Reconciliation System
This document describes the bosun GitOps reconciliation system, which automates infrastructure deployment by syncing configuration from a Git repository to target systems.
Overview
Bosun supports two deployment modes:
| Mode | Description | Use Case |
|---|---|---|
| Daemon | Long-running service with Unix socket API | Production, automated GitOps |
| One-shot | Single bosun reconcile execution |
Manual deploys, testing |
The reconcile system implements a GitOps workflow that:
- Monitors a Git repository for changes
- Decrypts secrets using SOPS with age encryption
- Renders templates using Go’s native
text/templateengine with Sprig functions - Deploys configuration files to local or remote targets
- Restarts affected services
When to Use
- Continuous deployment: Run the daemon with webhooks or polling for automated GitOps
- Manual deployment: Run
bosun reconcileto apply the latest configuration immediately - Testing changes: Use
--dry-runto preview what would change before applying
Daemon Mode
For production deployments, run bosun as a long-running daemon:
bosun daemon
Unix Socket API
The daemon exposes a Unix socket at /var/run/bosun.sock (configurable) for local control:
| Endpoint | Method | Description |
|---|---|---|
/trigger |
POST | Trigger reconciliation |
/status |
GET | Operator status with sanitized reconcile diagnostics |
/health |
GET | Public liveness JSON: status, ready, and uptime |
/ready |
GET | Readiness check |
/config |
GET | Get current config |
/ping |
GET | Simple ping |
/api/drift |
GET | Get drift status (declared vs actual) |
/api/containers |
GET | List all containers with summary |
/api/trigger |
POST | Trigger reconciliation (WebUI) |
/api/status |
GET | Extended status for WebUI |
/health has the same bounded response on the webhook listener, TCP API, and
Unix socket. It returns 200 when healthy and 503 otherwise, does not vary
when an Authorization header is present, and rejects non-GET methods with 405.
It never exposes reconcile errors, repository paths, subsystem messages, or
circuit-breaker state. Use the bearer-protected TCP /status endpoint or the
local Unix-socket /status endpoint for operator diagnostics.
Example usage:
# Trigger reconciliation
curl --unix-socket /var/run/bosun.sock http://localhost/trigger -X POST
# Check status
curl --unix-socket /var/run/bosun.sock http://localhost/status
Or use the CLI:
bosun trigger # Trigger via socket
bosun daemon-status # Get daemon status
bosun validate # Validate config and connectivity
Webhook Providers
The daemon itself serves two webhook endpoints:
| Endpoint | Path | Signature Header |
|---|---|---|
| GitHub | /webhook/github |
X-Hub-Signature-256 |
| Generic | /webhook |
X-Signature, or X-Hub-Signature-256 |
Both daemon endpoints act only on pushes to the tracked branch (BOSUN_REPO_BRANCH). After the signature check, the generic endpoint reads an optional top-level ref from a JSON body: a reconcile starts only when it equals refs/heads/<branch>. Other branches, tags, and a bare branch name get 200 with {"status":"ignored"} so the sender does not retry. A request with an X-GitHub-Event header other than push is ignored the same way, as is a GitHub push with no JSON ref (set the hook content type to application/json). A body with no ref still triggers, so a plain signed POST keeps working. Point GitHub at /webhook/github.
GitLab, Gitea, and Bitbucket are not handled directly by the daemon. To receive
them, run the standalone bosun webhook receiver, which understands each provider’s
signature scheme and forwards normalized triggers to the daemon over its Unix socket:
| Provider | Path (receiver) | Signature Header |
|---|---|---|
| GitHub | /webhook/github |
X-Hub-Signature-256 |
| GitLab | /webhook/gitlab |
X-Gitlab-Token |
| Gitea | /webhook/gitea |
X-Gitea-Signature |
| Bitbucket | /webhook/bitbucket |
X-Hub-Signature |
Do not point a Gitea/GitLab/Bitbucket webhook directly at the daemon’s /webhook
endpoint — its generic handler expects X-Signature/X-Hub-Signature-256, so a
provider-specific signature header will fail validation. Signatures are validated
using HMAC-SHA256 (or SHA1 for legacy) with constant-time comparison.
The receiver’s trigger endpoints fail closed, matching the daemon: with no
secret resolved (--secret, WEBHOOK_SECRET, GITHUB_WEBHOOK_SECRET, or
--fetch-secret) every one of them returns 403 unless
BOSUN_ALLOW_UNAUTHENTICATED_WEBHOOK=true is set. /health and /ready stay
open. --fetch-secret needs the receiver to run as an authorized socket peer,
since the daemon’s GET /config is peer-credential checked.
The receiver’s generic /webhook applies the same branch filter as the daemon’s. Its provider endpoints filter on the same branch after their signature and push-event checks: GitHub, GitLab, and Gitea pushes forward only when ref is refs/heads/<branch>, and Bitbucket pushes only when a push.changes[].new entry is a branch with that name. Other branches, tags, and deletions get 200 ignored. The receiver learns the branch from --branch, then BOSUN_REPO_BRANCH, then the daemon’s GET /config; with none known, the ref check is off on every endpoint and startup warns.
A push that deletes the tracked branch (deleted: true or an all-zero after) gets 200 ignored with a warning. A delivery ID (X-GitHub-Delivery, X-Gitea-Delivery, GitLab webhook-id/Idempotency-Key/X-Gitlab-Event-UUID, Bitbucket X-Request-UUID) or a non-empty request body seen in the last hour also gets 200 ignored, so a captured body re-sent under a new delivery ID is dropped too. The record is checked only after the signature or token passes, holds at most 1024 keys, and drops both keys of a delivery whose forward failed so the provider’s retry goes through. An empty-body signed trigger can still be repeated. The daemon’s own /webhook and /webhook/github apply both checks too. A Bitbucket push made only of deletions is ignored even with no tracked branch. The daemon’s /webhook/manual accepts only an empty body or {"force","source","nonce"} (anything else is 400) and answers 409 to a body a webhook endpoint recorded in the last hour; add a fresh nonce to retrigger.
Polling Mode
Enable periodic reconciliation with --poll-interval:
bosun daemon --poll-interval 3600 # Check every hour
Set to 0 to disable polling (webhook-only mode).
Concurrency
The daemon uses single-flight reconciliation with counted trigger coalescing:
- If a trigger arrives during reconciliation, increment the pending count and retain its source
- Return immediately (HTTP 202 Accepted)
- After reconciliation completes, atomically drain the pending batch
- Run one follow-up reconciliation with sorted distinct-source attribution and sticky
force - Triggers arriving during that follow-up form another batch
This prevents concurrent docker compose operations while ensuring no trigger batch or source attribution is lost.
Security
- Socket permissions: 0660 (owner and group only)
- SO_PEERCRED authorization: On Linux,
POST /triggerandGET /configrequire the kernel-reported peer UID to match the daemon’s effective UID orBOSUN_SOCKET_ALLOWED_UIDS; missing credentials fail closed with403unlessBOSUN_ALLOW_UNAUTHENTICATED_SOCKET=trueexplicitly opts out./configis checked despite being a read because it returns the webhook secret, which authorizes a trigger on the HTTP port - Bearer auth: Optional TCP API requires
Authorization: Bearer <token> - Secret injection: Webhook secret fetched from daemon, never on disk — the receiver must run as an authorized socket peer to fetch it
See docs/architecture/daemon-split.md for the full security model.
Architecture
+------------------+ +------------------+ +------------------+
| Git Repository |---->| Reconciler |---->| Target System |
| (dotfiles repo) | | | | (Unraid/Docker) |
+------------------+ +------------------+ +------------------+
|
v
+--------------------+
| Component Stack |
+--------------------+
| GitOps - go-git library (in-process)
| SOPSOps - go-sops library (in-process)
| TemplateOps - Go text/template + Sprig
| DeployOps - native file copy / tar-over-SSH
+--------------------+
Data Flow
Before any target starts, Bosun canonicalizes every effective staging path, rejects equal or nested slots, and secures or deletes pre-existing evidence. A slot that can be neither protected nor deleted aborts the whole cycle before Git sync or secret decryption.
1. Lock Acquisition
|
v
2. Git Sync (clone or pull)
|
v
3. State-Based Skip Logic
(compare deployed commit vs current, circuit breaker check)
|
v (if new commit, force, or state mismatch)
4. SOPS Decryption
|
v
5. Template Rendering (Go text/template + Sprig)
|
v
6. Extract Declared State (parse rendered compose files;
fatal if compose dir missing; fatal on zero services unless
BOSUN_ALLOW_EMPTY_DECLARED_STATE=true)
|
v
7. Backup Creation (tar.gz of current configs)
|
v
8. Deployment (native file copy or tar-over-SSH)
|
v
9. Deploy-Sync Invariant Check
(per-target: created directories and written files in WrittenFiles
must exist at destination with their expected type and
mtime >= reconcile start; when no regular file was written, every
regular source file must already byte-match the destination;
bypass via BOSUN_SKIP_DEPLOY_INVARIANT)
|
v
10. Service Reload (docker compose up, SIGHUP)
|
v
11. Critical Container Health Gate (with rollback)
|
v
12. Post-Sync Hooks
|
v
13. Post-Deploy Verification (drift check against Docker)
|
v
14. Remove Verified Staging
|
v
15. Record Deploy State (commit, declared services, timestamp)
|
v
16. Release Lock
Invariant gates at stages 6 and 9 prevent the silent-success failure mode where a deploy reports success but expected content did not land on disk. See Deploy-Sync Invariants in the skill resource and the troubleshooting guide for operator-facing detail on
BOSUN_ALLOW_EMPTY_DECLARED_STATEandBOSUN_SKIP_DEPLOY_INVARIANT.
Deploy State Tracking
Before upgrading to filesystem confinement, stop Bosun and ensure the configured
Unix lock directory is persistent, daemon-owned and mode 0700; an existing
lock file must be daemon-owned, regular and mode 0600. Repair the owner/mode
or symlinked path named by the error before restarting. Use physical template
source/include directory paths, and replace symlinked .tmpl entries with
regular template files. Directory rendering now refuses a symlinked template
with its path and that remedy, preventing deployment or stale-file pruning from
a partial render. Windows lock behavior is unchanged.
The reconciler maintains persistent state in a JSON file at /var/lib/bosun/deploy-state.json (configurable via BOSUN_STATE_DIR). This state drives skip logic, circuit breaking, and drift detection. Both daemon and one-shot modes create the state file’s parent directory before a real reconciliation; one-shot multi-target runs prepare and verify each target’s state directory independently and fail that target before deployment if the directory cannot be written. Multi-target configuration is rejected when two effective state-file paths collide after path cleaning; case-only variants also collide on Windows and macOS. Daemon startup performs this validation before binding API listeners or announcing readiness. A one-shot dry run works against a temporary copy of existing state, preserving skip and circuit-breaker decisions without creating or updating the configured state path.
Container deployments must persist /var/lib/bosun. The shipped Compose and
Unraid definitions bind that directory to /mnt/user/appdata/bosun/state; keep
an equivalent writable mount when defining a custom deployment. Before starting
the container, create the host directory for Bosun’s fixed UID/GID 1000:
install -d -o 1000 -g 1000 -m 0700 /mnt/user/appdata/bosun/state
The shipped Compose definition refuses to auto-create a missing source directory,
which avoids Docker creating it as root:root. Bosun also write-probes the state
directory and fails startup before binding API listeners when it is not writable.
Without the mount, container replacement discards drift history, skip state, and
circuit-breaker state, forcing the next run to reconcile the full declared estate.
State File Schema (v2)
{
"schema_version": 2,
"last_deployed_commit": "abc123...",
"deployed_at": "2025-01-15T14:30:22Z",
"source": "webhook:github",
"last_attempted_commit": "abc123...",
"attempt_count": 0,
"declared_services": [
{"name": "web", "image": "nginx:1.25"},
{"name": "api", "image": "myapp:v2"}
],
"drift_checked_at": "2025-01-15T14:35:00Z",
"drift_items": []
}
State-Based Skip Logic
The reconciler compares the last deployed commit (not last fetched) against the current HEAD. This distinction matters: if a deploy fails halfway, the state file still records the previous successful commit, ensuring the pipeline re-runs on the next trigger.
Circuit Breaker
After 3 consecutive failures on the same commit, the reconciler stops retrying:
Attempt 1: deploy fails → attempt_count=1, continues
Attempt 2: deploy fails → attempt_count=2, continues
Attempt 3: deploy fails → attempt_count=3, stops
Next trigger: skips unless --force or new commit arrives
Use --force to override the circuit breaker.
Atomic State Writes
State is written using the crash-safe pattern: write temp file (same directory) → fsync temp → rename → fsync directory. This prevents corruption from power loss or crashes during write.
Fail-Open Behavior
Missing or corrupt state files are treated as “never deployed,” which triggers a full deploy. This is correct fail-open behavior — it’s better to deploy redundantly than to silently skip.
Drift Detection
Drift detection compares declared state (services defined in rendered compose files) against actual state (running Docker containers). This closes the feedback loop in the GitOps model — after deploying, verify the deployment took effect.
How It Works
- Declared state is extracted from rendered compose files during reconciliation (step 6) and saved in the deploy state file.
- Actual state is collected from Docker by querying all containers and filtering by compose project labels.
- Comparison checks each declared service against actual containers.
Drift Types
| Type | Severity | Condition |
|---|---|---|
missing |
Critical | Service not running, or running but state is not running (e.g., exited) |
unhealthy |
Critical | Service running but Docker health check reports unhealthy |
image_mismatch |
Warning | Running image tag differs from declared (e.g., nginx:1.24 vs nginx:1.25) |
Container Matching
Containers are matched to declared services using Docker Compose v2 labels:
com.docker.compose.project— the compose project namecom.docker.compose.service— the service name within the project
Labels are preferred because they are authoritative and unambiguous. For containers without labels, a name-based fallback parses the <project>-<service>-<replica> convention.
Image Comparison
Images are normalized before comparison: bare names (e.g., nginx) are treated as nginx:latest. Digest references (nginx@sha256:...) are compared exactly.
Daemon Drift Loop
When running as a daemon, drift checks run on a configurable interval (default: 5 minutes, set via BOSUN_DRIFT_INTERVAL). The loop:
- Skips if a reconciliation is in progress (avoids state file race conditions)
- Loads declared state from the deploy state file
- Queries Docker for actual state
- Filters through the debounce layer (if enabled via
BOSUN_DRIFT_ALERT_DEBOUNCE) - Filters through the dedup layer (per-item cooldown via
BOSUN_DRIFT_ALERT_COOLDOWN) - Updates the state file with drift results and debounce/dedup timestamps
- Sends an alert if critical drift persists past both layers
Set BOSUN_DRIFT_INTERVAL=0 to disable periodic drift checks.
Restart Circuit Breaker
The restart circuit breaker samples container identity and restart counts during each local target’s drift check. Once restart counts begin increasing, Bosun preserves the earliest unresolved baseline until a clean sample observes no new restarts, so sustained slow loops still accumulate toward BOSUN_RESTART_THRESHOLD even when checks are farther apart than BOSUN_RESTART_WINDOW. A deploy can recreate a container and reset Docker’s restart count; Bosun treats the changed identity as recreation, keeps an existing trip active, and resolves it only after the same container identity has no additional restarts across the next drift-check interval. Missing containers retain the trip but restart the stability grace when they return; transient inspect failures preserve the last persisted observation and cannot count as recovery. The breaker stops containers only within the effective Compose project for the target being checked. Each candidate must carry Compose labels naming that project and its service; a container matched only by a <project>- name prefix is never stopped. A named local target without project_name uses the existing compose fallback, and because any stack started from a directory named compose gets that name, the breaker then also requires the container’s com.docker.compose.project.working_dir label to be the target’s <local appdata>/compose directory (symlinks on either path are resolved before comparing); a trip stops the container by the ID that was inspected, not by name; an implicit default target with no project name fails closed before inspection or stopping and preserves its restart tracking. When the breaker is enabled in that unresolved implicit-default condition, daemon startup emits one warning and bosun doctor reports the same condition; set project_name on the implicit default target (a target named default in bosun.yaml) or in BOSUN_TARGETS to restore protection from startup. A root-level project_name in bosun.yaml scopes the breaker only after a reconcile loads it from the repository; until then the breaker stops nothing, and startup and bosun doctor still warn and say why. Multiple targets keep independent project scopes and state files. Keep BOSUN_DRIFT_INTERVAL at or below BOSUN_RESTART_WINDOW for timely detection; daemon configuration and bosun doctor warn when the sampling interval is longer.
Drift Self-Heal Breaker
With BOSUN_DRIFT_SELF_HEAL=true, periodic drift may trigger reconciliation, but one stable set of drifted service:type items receives at most BOSUN_DRIFT_SELF_HEAL_MAX_ATTEMPTS attempts (default 3). The attempt count and cooldown timestamp are stored in each effective target’s deploy state file, so daemon restarts and signature churn cannot bypass the bound or cooldown. Continued drift after the final attempt marks the signature exhausted and sends one self-heal-exhausted alert; a changed signature gets a fresh budget, and resolved drift re-arms a later recurrence. Breaker logs and alerts use a short opaque signature identifier plus counts rather than service names or image values.
Drift Alert Debounce
Transient drift from I/O pressure, image updates, or daemon restarts generates alert noise that self-resolves within minutes. The debounce layer suppresses alerts until drift persists beyond a configurable window.
The alert pipeline flows: detect -> debounce filter -> dedup (per-item cooldown) -> send.
BOSUN_DRIFT_ALERT_DEBOUNCE(env var) ordrift_alert_debounce(bosun.yaml): Duration before first alert fires. Default0(disabled, alerts fire immediately). Recommended starting value:5m.- First detection records the item in
drift_debounce_itemswithout alerting. - If drift resolves before the window expires, the item is silently removed (no alert).
- If drift persists past the window, the item graduates to the dedup/cooldown layer.
- Resolution alerts bypass debounce: they fire immediately for previously alerted items, but not for items that were still in debounce (never alerted).
- A critical type transition (
unhealthytomissing, or the reverse) remains active drift for that service. The replacement type follows normal debounce/dedup handling, and Bosun does not send a resolution until the service has no critical drift. - Adding an ignore rule for active drift removes its prior alert state silently; Bosun sends a resolution only when the service’s critical drift is absent from the unfiltered Docker comparison.
- Debounce state persists across daemon restarts via the state file.
Post-Deploy Verification
After each successful deployment, the reconciler performs an immediate drift check:
- Waits for the startup grace period (default: 30 seconds) to let containers start and pass health checks
- Collects actual state from Docker
- Compares against declared state
- Updates drift results in the state file
- Logs warnings for any drift items (does not fail the reconciliation)
The earlier health gate is a separate, rollback-capable check controlled by
health_gate_scope and BOSUN_HEALTH_GATE_TIMEOUT. Its timeout defaults to
60s; set BOSUN_HEALTH_GATE_TIMEOUT=0 to disable the gate without changing
the configured scope or container list.
CLI Access
bosun drift # Show cached drift status
bosun drift --live # Fresh check against Docker
bosun drift --json # Machine-readable output
bosun drift --project core # Filter by compose project
API Access
The daemon exposes drift status at GET /api/drift:
{
"status": "clean",
"checked_at": "2025-01-15T14:35:00Z",
"declared_count": 12,
"drift_item_count": 0,
"items": []
}
Status values: clean (no drift), drifted (items detected), unknown (no deployment recorded).
Configuration
Environment Variables
| Variable | Required | Default | Description |
|---|---|---|---|
REPO_URL |
Yes | - | Git repository URL |
REPO_BRANCH |
No | main |
Branch to track |
BOSUN_GIT_USERNAME |
No | - | Private HTTPS Git Basic-auth username; requires BOSUN_GIT_TOKEN |
BOSUN_GIT_TOKEN |
No | - | Private HTTPS Git Basic-auth password/token; requires BOSUN_GIT_USERNAME |
BOSUN_SSH_KEY |
No | conventional SSH paths | Private key fallback for SSH Git URLs; the SSH agent takes precedence |
REPO_DIR |
No | /app/repo |
Local clone directory |
STAGING_DIR |
No | /app/staging |
Secret-bearing rendered staging and single-slot failure evidence directory |
BACKUP_DIR |
No | /app/backups |
Configuration backups |
LOG_DIR |
No | /app/logs |
Log files directory |
LOCAL_APPDATA |
No | /mnt/appdata |
Local appdata path |
REMOTE_APPDATA |
No | /mnt/user/appdata |
Remote appdata path |
DEPLOY_TARGET |
No | - | SSH target (e.g., root@192.168.1.8) |
SECRETS_FILES |
No | - | Comma-separated SOPS files |
DRY_RUN |
No | false |
Preview mode |
FORCE |
No | false |
Deploy even without changes |
BOSUN_STATE_DIR |
No | /var/lib/bosun |
Directory for deploy state file |
BOSUN_DRIFT_INTERVAL |
No | 5m |
Drift check interval (0 to disable) |
BOSUN_RESTART_BREAKER |
No | true |
Enable restart-loop protection |
BOSUN_RESTART_THRESHOLD |
No | 5 |
Accumulated restart count that trips the breaker (must be positive) |
BOSUN_RESTART_WINDOW |
No | 10m |
Restart observation window; keep at least as long as the drift interval |
BOSUN_DRIFT_ALERT_COOLDOWN |
No | 1h |
Cooldown between repeated drift alerts per item |
BOSUN_DRIFT_ALERT_DEBOUNCE |
No | 0 |
Debounce window before first drift alert (0 = disabled) |
BOSUN_DRIFT_RESOLVE_ALERTS |
No | true |
Send “drift resolved” notifications |
BOSUN_HEALTH_GATE_TIMEOUT |
No | 60s |
Health gate polling timeout (0 disables the gate) |
Private HTTPS repositories use BOSUN_GIT_USERNAME and BOSUN_GIT_TOKEN as
one required pair for both clone and fetch. With both unset, HTTPS remains
anonymous. Bosun rejects partial pairs, non-HTTPS credential use, credentials
embedded in the repository URL, HTTPS-to-HTTP redirects, and redirects to a
different host or effective port. The pair has no legacy aliases and applies
to the effective repository URL after BOSUN_REPO_URL overrides REPO_URL.
Credentials remain process-environment state: remove any URL userinfo, rotate
the environment values, and restart Bosun; bosun.yaml reload cannot rotate
them.
SSH Git repositories use SSH_AUTH_SOCK first. Bosun accepts the agent only
after it returns at least one signer; an empty or unreadable agent connection is
closed before Bosun falls back to BOSUN_SSH_KEY, followed by its conventional
key paths. The username parsed from either SCP-like syntax or an ssh:// URL is
preserved. An explicit BOSUN_SSH_KEY must be a regular, non-empty, parseable
private key file; missing, directory, empty, and malformed mounts fail before
Git network access. An unusable existing conventional candidate is skipped if
a later key works, otherwise Bosun reports its path instead of passing nil
authentication to go-git. A recognized SSH URL with no usable authentication
fails startup rather than reaching the network. HTTPS and public local
repository paths ignore unrelated SSH key settings. For containers, pre-create
a key bind-mount source as a file because Docker can create a directory when the
host source is missing.
SSH Host-Key Verification
Upgrade required: Before upgrading, provision a persistent known_hosts file inside Bosun at
/config/known_hosts, or set BOSUN_SSH_KNOWN_HOSTS to its mounted path. Include the Git
and deploy hosts; copy any previously verified deploy-host entries from the container’s
~/.ssh/known_hosts into this file before recreating the container. Git does not read
~/.ssh/known_hosts. Collect new candidate keys with ssh-keyscan -p 22 HOST > known_hosts.candidate, inspect them with ssh-keygen -lf known_hosts.candidate, and
verify each retained SHA256 fingerprint against the host console or administrator
through an independent trusted channel before installing the entries. ssh-keyscan alone
does not authenticate a host. Deploys without a Bosun trust file still use strict
OpenSSH default stores; container-only pins must be persisted.
Both SSH channels resolve configured trust files in this order:
BOSUN_SSH_KNOWN_HOSTS(explicit override)/config/known_hosts(container convention)
Git fails closed before network access when neither candidate exists or the
selected file is malformed. Remote ssh and scp operations use
StrictHostKeyChecking=yes with the selected file; when no configured file
exists, they leave OpenSSH’s normal trust stores available but refuse an
unpinned host. Bosun never uses accept-new as an implicit trust decision.
~/.ssh/known_hosts is intentionally excluded from Git because container user
profiles are ephemeral. Set BOSUN_SSH_INSECURE_HOST_KEY=true only for an
intentional bootstrap opt-out; it disables host-key verification for both
channels.
Command-Line Flags
bosun reconcile [flags]
Flags:
-n, --dry-run Show what would be done without making changes
-f, --force Force deployment even if no changes detected
-l, --local Force local deployment mode
-r, --remote string Target host for remote deployment (e.g., root@192.168.1.8)
Example Configuration
# Container deployment
export REPO_URL="git@github.com:user/dotfiles.git"
export REPO_BRANCH="main"
export SECRETS_FILES="infrastructure/secrets.yaml"
export DEPLOY_TARGET="root@192.168.1.8"
bosun reconcile
# Local testing with dry-run
export REPO_URL="git@github.com:user/dotfiles.git"
export SECRETS_FILES="infrastructure/secrets.yaml"
bosun reconcile --dry-run --local
Git Operations
The git subsystem (internal/reconcile/git.go) handles repository synchronization using the go-git library for pure Go, in-process Git operations:
Clone Behavior
- Shallow clone: Uses depth 1 by default to minimize bandwidth;
BOSUN_GIT_FETCH_DEPTHconfigures a deeper positive depth - Single branch: Only fetches the configured branch
- Cleanup on failure: Removes partial clones if the operation fails
- In-process: No external
gitbinary required
Pull Behavior
- Shallow fetch: Uses the same configured depth as clone and deepens an existing shallow checkout when needed
- Hard reset: Resets to
origin/<branch>to ensure clean state - Change detection: Compares commit hashes before/after to detect changes
- Missing diff history fails safe: An unavailable prior commit is reported explicitly; deploy-path checks run a full deploy and post-sync hooks all fire
Post-Sync Hook Snapshots
After each pull, bosun treats a successfully decoded project config as an authoritative hook snapshot. Omitting post_sync_hooks or setting it to [] clears file-sourced hooks; omitting hook_settle_delay retains the effective delay, while explicit 0s disables it. Missing, unreadable, malformed, unknown-field, and invalid-hook configs retain the previous hook/delay state. BOSUN_POST_SYNC_HOOKS, BOSUN_HOOK_SETTLE_DELAY, and target hooks supplied by BOSUN_TARGETS remain authoritative environment replacements.
Root and per-target hooks validate before either hooks or delay change. An omitted target hook key inherits root, explicit [] clears inheritance, and removing a target descriptor drops its stale operational hook override while structural removal waits for daemon restart. Logs expose source, target, outcome, and counts, never hook command arguments.
Hook globs use staging-relative paths such as appdata/traefik/**. An empty content-hash deploy result is authoritative no-change; standard-copy mode uses non-empty written or deleted path evidence directly and falls back to a diff from DeployState.LastDeployedCommit only when both lists are empty. Fallback paths have the infra-directory prefix stripped before matching. Remote deploys ignore both path lists and fire every configured hook. A failed pipeline cannot advance the hook diff base or silently lose its changes on retry.
DeployState.DeployedFiles remains a regular-file-only ownership manifest. Empty-directory ownership is not persisted, so the inverse transition from a previously managed empty directory to a file requires a separate ownership and rollback contract.
When deploy paths changed but no hook pattern matches, bosun emits a warning with distinct/duplicate/empty pattern counts, at most five pattern samples, the evaluated-path count, an explicit zero matched-path count, and at most five staging-relative path samples. Absolute or traversal paths are redacted. The distinct no-change case is an info log, not a mismatch warning. These diagnostics include bounded path metadata only, never file contents or hook command arguments.
Timeouts
| Operation | Timeout |
|---|---|
| Clone | 5 minutes |
| Fetch | 2 minutes |
| Local operations | 30 seconds |
State Tracking
The Sync() method returns:
changed bool- Whether the repository was updatedbefore string- Commit hash before sync (empty for fresh clones)after string- Commit hash after sync
Secrets Management
The SOPS subsystem (internal/reconcile/sops.go) handles encrypted secrets using the go-sops library with age encryption. All decryption happens in-process without requiring an external sops binary.
Secrets-file format is inferred from the filename. Bosun supports YAML (.yaml, .yml), JSON (.json), dotenv (.env), and INI (.ini), including legacy names such as secrets.yaml.sops. Other extensions fail with a clear error; SOPS binary files are not supported because Bosun must merge decrypted secrets into a key/value template map.
Age Key Resolution
The system checks for age keys in this order:
SOPS_AGE_KEYenvironment variable (key content directly)SOPS_AGE_KEY_FILEenvironment variable (path to key file)- Default location:
~/.config/sops/age/keys.txt
SOPS_AGE_KEY retains precedence over file settings. When Bosun resolves a
file path, it requires a regular, non-empty file containing at least one
parseable Age identity before decryption begins. Missing, directory, empty,
and malformed paths fail with the configured path and setup guidance. For
container mounts, pre-create the host key file; otherwise Docker can replace a
missing file source with a directory at the container path.
When at least one secrets file is configured, both the daemon and one-shot
bosun reconcile validate the Age identity before Git authentication. The
daemon performs this check before binding its Unix, TCP, or HTTP listeners. A
reconcile with no secrets files does not require an Age identity and ignores an
unrelated invalid Age mount.
Key Setup
# Generate a new age key
age-keygen -o ~/.config/sops/age/keys.txt
# Or use an existing key via environment
export SOPS_AGE_KEY="AGE-SECRET-KEY-1..."
# Or specify a custom key file
export SOPS_AGE_KEY_FILE="/path/to/my/key"
SOPS File Validation
Before decryption, the system validates:
- File exists
- Valid YAML syntax
- Contains a
sopsmetadata mapping - Contains a non-empty MAC and a valid RFC3339
lastmodifiedtimestamp - Contains at least one key recipient with a non-empty encrypted data key
After structural validation, decryption failures are reported as one of four sanitized categories: integrity verification, key unavailable, malformed encrypted data, or an unclassified decryption failure. Integrity failures tell operators to restore or re-encrypt the file rather than rotate an unrelated key. Raw SOPS errors are not returned or logged because they can contain decrypted MACs, encrypted values, key identifiers, or additional local paths; the requested secrets-file path remains in the surrounding error context.
Decryption Flow
1. Validate SOPS file structure
|
v
2. Check age key availability
|
v
3. In-process decryption via go-sops library
|
v
4. Parse decrypted content to map[string]any
|
v
5. Merge multiple files (later files override earlier)
Multiple Secrets Files
When multiple secrets files are specified, they are decrypted and merged:
export SECRETS_FILES="common/secrets.yaml,infrastructure/secrets.yaml"
Keys in later files override earlier ones. Nested maps are recursively merged.
Templating
The template subsystem (internal/reconcile/template.go) uses Go’s native text/template engine with Sprig functions for template rendering. This provides the same Go template syntax without requiring an external binary.
Template File Convention
- Files ending in
.tmplare processed as templates - The
.tmplextension is removed in the output - Non-template files are copied as-is
Accessing Secrets in Templates
Bosun decrypts secrets and passes the resulting map directly as the template’s root data. Access values without reading the secrets file:
{{ .network.unraid_ip }}
For non-secret supporting data, include and fromJsonFile may read files
inside the configured include subtree. This example uses an absolute allowlist
root so the file path is independent of the process working directory:
export BOSUN_TEMPLATE_INCLUDE_DIR=/srv/bosun/includes
{{ $defaults := fromJsonFile "/srv/bosun/includes/defaults.json" }}
{{ $defaults.timezone }}
Available Template Functions
The template engine provides Go’s standard template functions plus all Sprig functions. Commonly used:
| Function | Description |
|---|---|
env "VAR" |
Get environment variable |
include "path" |
Read an allowlisted file as text |
fromJsonFile "path" |
Read and parse an allowlisted JSON file |
fromJson "..." |
Parse JSON string |
toJson . |
Convert to JSON |
quote . |
Quote a string |
default "val" . |
Default value if empty |
upper . |
Uppercase string |
lower . |
Lowercase string |
trim . |
Trim whitespace |
b64enc . |
Base64 encode |
b64dec . |
Base64 decode |
Template Processing
1. Find all .tmpl files in source directory
|
v
2. For each template:
a. Read template content
b. Parse template with Sprig functions
c. Execute with secrets data
d. Write output to staging directory
|
v
3. Copy non-template files as-is
Template staging honors reconcile cancellation before directory creation and before publishing each atomic rendered output. Cancellation interrupts writes to the private template temp file, removes that partial file, and leaves an existing output untouched. If cancellation follows staging cleanup, Bosun does not recreate the staging root for a render that will no longer run.
Environment Filtering
For security, only safe environment variables are exposed to templates:
Allowed prefixes: PATH=, HOME=, USER=, LANG=, LC_, TERM=, XDG_, TMPDIR=, TMP=, TEMP=
Blocked patterns:
- Prefixes:
SOPS_,AWS_,AZURE_,GCP_,GOOGLE_,CLOUDFLARE_, etc. - Suffixes:
_TOKEN,_SECRET,_KEY,_PASSWORD,_AUTH, etc. - Exact matches:
GITHUB_TOKEN,GITLAB_TOKEN,NPM_TOKEN, etc.
Deployment
The deploy subsystem (internal/reconcile/deploy.go) handles file synchronization to local or remote targets.
Deployment Modes
Local Mode: Used when appdata is mounted locally (e.g., container with volume mount)
bosun reconcile --local
Remote Mode: Uses tar-over-SSH for file transfer
bosun reconcile --remote root@192.168.1.8
Mode Detection
If --local is not specified, the system auto-detects:
- If
--remoteorDEPLOY_TARGETis set, use remote mode - If
LocalAppdataPathexists on filesystem, use local mode - Otherwise, attempt remote mode using
network.unraid_ipfrom secrets
Local Deployment
Uses native Go file operations to sync directories:
Staging Target
staging/unraid/appdata/ -> /mnt/appdata/
Local copy walks honor the reconcile context before every directory creation, file replacement, and managed stale-file deletion. Cancellation also interrupts copying into an atomic temp file, so a cancelled reconcile stops before the next live-target mutation. Files whose atomic rename already completed are still flushed and verified before the cancellation error returns; later files and stale deletions are left untouched. In standard atomic-swap mode, cancellation after the existing target is moved aside restores that original target and removes the unpublished temp tree.
Remote Deployment
Uses tar-over-SSH for efficient file transfer:
Staging Target
staging/unraid/appdata/ -> root@host:/mnt/user/appdata/
The remote deployment creates a tar archive locally, streams it over SSH, and extracts it into a temp directory on the remote host. This avoids requiring rsync on the remote system.
The staged tree is then promoted with a retain-old rename-swap: the live target directory is moved aside to <target>.bosun-old.<timestamp> (never deleted first), the new tree is moved into place, and the retained copy is removed only after the swap succeeds. If the move-in fails, the swap restores the retained copy. An interrupted deploy therefore always retains at least one complete tree — the old or the new. A crash between the move-aside and the move-in can leave the target path temporarily absent (the old tree stays intact at its .bosun-old.<timestamp> sibling) until the next deploy self-heals it by promoting the newest retained copy; orphaned copies are cleaned up at the same time. On Unraid’s /mnt/user FUSE mount the swap is safe (retain-old) rather than kernel-atomic, and a settle delay follows the move so writes propagate before docker compose up reads them.
Deployed Paths
| Source | Destination |
|---|---|
staging/unraid/appdata/traefik/ |
appdata/traefik/ |
staging/unraid/appdata/authelia/configuration.yml |
appdata/authelia/configuration.yml |
staging/unraid/appdata/agentgateway/config.yaml |
appdata/agentgateway/config.yaml |
staging/unraid/appdata/gatus/config.yaml |
appdata/gatus/config.yaml |
staging/unraid/appdata/tailscale-gateway/serve.json |
appdata/tailscale-gateway/serve.json |
staging/unraid/compose/ |
appdata/compose/ |
Service Reload
After deployment:
docker compose up -d --remove-orphans --waiton core.ymldocker kill --signal=SIGHUP agentgatewayto reload config
Timeouts
| Operation | Timeout |
|---|---|
| SSH connect | 5 seconds |
| SSH commands | 30 seconds |
| File sync | 5 minutes |
| docker compose up | 10 minutes |
Retry Logic
SSH and file sync operations retry on transient errors with exponential backoff:
- Max retries: 3
- Backoff sequence: 1s, 2s, 4s
Retryable errors:
- Connection refused/reset
- Connection timed out
- Network unreachable
- No route to host
- Host is down
- I/O timeout
- Temporary failure
Locking
The reconciler uses file-based locking to prevent concurrent runs.
Lock File
Path: /var/run/bosun/reconcile.lock
The directory must be owned by the daemon uid with mode 0700. The image’s
directory satisfies this for its default user (uid 1000). A container run as
root or another uid needs its own 0700 directory at /run/bosun, such as
tmpfs: ["/run/bosun:mode=0700,size=1m"]; see
Security § Filesystem Confinement for the full rule.
Unix Implementation
Uses flock(2) system call with LOCK_EX | LOCK_NB for non-blocking exclusive lock.
syscall.Flock(fd, syscall.LOCK_EX|syscall.LOCK_NB)
Windows Implementation
Uses LockFileEx with LOCKFILE_EXCLUSIVE_LOCK | LOCKFILE_FAIL_IMMEDIATELY.
Behavior
- If lock is held by another process, the reconciliation is skipped gracefully
- Lock is released on normal completion, error, or process termination
- Stale locks (from crashed processes) are automatically cleaned up by the OS
Backup System
Before deployment, the system creates timestamped backups of current configurations.
Backup Format
backups/
backup-20240115-143022/
configs.tar.gz
Backed Up Paths
During normal rendered-target enumeration, Bosun backs up regular staging files at the corresponding appdata destinations; symlinks and irregular entries are excluded from that managed-file list. If deploy-target discovery itself fails, Bosun attempts a bounded full-appdata backup rather than treating the footprint as empty. That fallback preserves symlink entries without following them.
Remote Backup
For a non-empty remote footprint, Bosun runs tar -czf - over SSH and streams
the archive to the local backup directory. An empty footprint performs no SSH
work.
Backup Verification
After creation, backups are verified:
- Archive file exists
- Archive is non-empty
- Archive is listable with
tar -tzf - Archive contains at least one entry
- Every member can be decompressed and read to EOF under the caller deadline and the total 10 GiB verification bound
Rollback Archive Admission
Every local and remote rollback path reads configs.tar.gz with Bosun’s
in-process archive/tar extractor into a fresh temporary root. The complete
archive must pass before the root becomes usable. Bosun rejects member-name
traversal, absolute or escaping symlink targets, and escaping hardlink targets;
confined relative symlinks and archive-root-relative hardlinks remain supported.
Validation, corruption, I/O, size-bound, and extraction-context failures remove
the partial root before any live managed file is copied or removed and before a
backup path can reach compose or the orphan-reconciliation pass. The 10 GiB
whole-decompressed-stream bound includes tar headers, padding, skipped entry
bodies, and trailing data; extraction drains to true gzip EOF so trailer
checksum failures are rejected.
Local rollback extraction retains reconcile logging metadata but runs under a background-derived independent compose timeout. Cancellation of the failed deployment’s outer context does not suppress the recovery attempt; cancellation or expiry of the independent extraction context stops it. Full-tree rollback returns rollback-not-attempted while retaining the archive cause. Per-file rollback retains the original compose failure, logs the archive cause, reports the file as not rolled back, and excludes it from orphan reconciliation.
An incomplete managed-footprint enumeration aborts before backup or deploy work, even when an older verified backup exists: that older archive may omit a path the current deploy would mutate. Other creation, transport, verification, and timeout failures may reuse the most recent previously verified backup as a rollback anchor; without one, the reconcile aborts before mutation. A successfully enumerated empty footprint is not a failure and proceeds without an anchor.
Retention
By default, keeps the 5 most recent valid backups — those that pass the same
verification used to select a rollback anchor. Pruning is deferred until the
current deploy clears its health checks, post-sync hooks, and post-deploy
verification gates. With a retention count of 1, the prior known-good backup
therefore remains available alongside the fresh pre-deploy snapshot throughout
the deploy; a failed deploy keeps both, while a successful deploy prunes to the
configured count.
Corrupt or partial backup directories (a missing, unlistable, or truncated configs.tar.gz)
do not count toward the retention limit and are removed outright, so a broken backup can
never occupy a keep slot and evict an older good one. When a deploy has no managed files to
back up (a fresh host’s first deploy), no backup is created and no rollback anchor is recorded.
BackupTimeout (BOSUN_BACKUP_TIMEOUT, default 5m) applies as two independent
deadlines: once to pre-deploy creation + verification and again to post-success
retention verification + cleanup. If retention reaches that deadline, Bosun
warns, marks the cleanup telemetry span as an error, preserves every backup it
has not already classified and removed, and still records the verified deploy
as successful.
cfg.BackupsToKeep = 5 // Default
Error Recovery
Git Failures
- Clone failures clean up partial directories
- Timeout errors provide specific timeout duration
- Network errors include stderr output for debugging
Secrets Failures
- Missing key files provide setup instructions
- Invalid SOPS files suggest encryption command
- Decryption errors include the secrets-file path and a sanitized category; integrity failures are distinct from key and malformed-data failures
- Raw SOPS errors are never returned or logged, including at debug level
Template Failures
- stderr is sanitized to avoid leaking secrets (truncated to 500 chars)
- Template errors include file path
- Missing map keys stop rendering and identify the missing key; for optional
keys, use
get . "key" | default "value"so the lookup is explicit - Missing directories are created automatically
Deployment Failures
- SSH errors are parsed into actionable messages:
- Permission denied: Check authorized_keys
- Connection refused: Check SSH service
- Host key verification: Run ssh-keyscan
- No route to host: Check network/host status
- Connection timeout: Check firewall rules
- DNS failure: Check hostname
Compose Failures
- Isolated compose deployment can restore a failed file from a fully admitted backup archive; archive rejection leaves the original compose error visible
- Failed compose operations log warnings but don’t abort the entire reconciliation
- Container health is verified after compose up
Partial Failures
Some operations log warnings but continue:
- Backup creation, transport, verification, or timeout failure when an older verified rollback anchor exists and the current managed footprint was enumerated completely
- tailscale-gateway sync failure (warns, continues)
- agentgateway reload failure (warns, continues)
- Staging cleanup failure when owner-only retention succeeds (warns)
Security Considerations
Secret Handling
- In-memory processing: Secrets are decrypted and processed entirely in memory using the go-sops library
- No external processes: Template rendering uses native Go text/template, avoiding secrets in environment variables
- Environment filtering: Only safe env vars are exposed to templates
- Error sanitization: Output is truncated to avoid leaking secrets in logs
- Memory: Secrets are stored in Go maps, garbage collected after use
SSH Security
- Host validation: Rejects hosts with shell metacharacters (
;,&,|,$, etc.) - Option injection prevention: Rejects hosts starting with
- - BatchMode: Uses
-o BatchMode=yesto prevent password prompts - Connection timeout: 5 second timeout prevents hanging
Input Validation
All user inputs are validated before use:
| Input | Validation |
|---|---|
| SSH host | Regex pattern, no shell metacharacters, no - prefix |
| Git branch | Regex pattern, no shell metacharacters, no - prefix |
| Container name | Regex pattern, no shell metacharacters, alphanumeric start |
| Docker signal | Allowlist: SIGHUP, SIGTERM, SIGKILL, SIGUSR1, SIGUSR2 |
Temp File Security
- The effective staging root is
0700before decrypted output is written; payload temp files stay0600until atomic rename. Active descendants keep their established deploy modes so local copy and remote tar do not change destination permissions. - A failed render or any later failed pipeline stage leaves one diagnostic slot
at the target’s effective
StagingDir. Retained directories are0700and regular files are0600; symlinks and irregular files make Bosun delete the slot without following them. - A subsequent render replaces that same slot rather than accumulating archives. Git sync, config reload, decryption, and deploy-mode failures before rendering leave the prior secured evidence unchanged.
- Dry runs retain the secured render. Verified real deployments remove staging only after health checks, hooks, and post-deploy verification. If removal fails, Bosun warns and retains a proven owner-only tree; inability to harden or delete it is a security error and prevents success from being recorded.
- Multi-target staging paths must be canonically disjoint. Bosun validates and preflights every target before executing any target, including when a one-shot run selects only one target.
- Backup archives use timestamped directories to prevent conflicts
Container Security
When running as a container:
- Mount SSH keys as read-only volume
- Mount age keys as read-only volume or use environment variable
- Use non-root user if possible
- Limit network access to Git server and target hosts
Dry-Run Mode
Use --dry-run to:
- Preview deploy changes without applying
- Skip service restarts
- Skip backup creation
- Verify configuration without risk
# Safe way to test configuration changes
bosun reconcile --dry-run
Known Limitations
Hardcoded Deploy Paths
The deploy step (deployLocal/deployRemote) syncs a fixed set of directories: traefik, authelia, agentgateway, gatus, tailscale-gateway, and compose. Adding a new service to the reconciliation pipeline currently requires a code change to Bosun itself.
The manifest/provision system is generic, but the reconciler’s deploy step is not — it targets specific paths. A future version will make deploy paths data-driven via a deploy manifest. See bosun-ciy.
Compose Up Blast Radius
docker compose up -d --remove-orphans --wait applies to the entire compose file. If one service in core.yml has a broken image tag, the entire compose operation fails. Rollback restores previous configs and re-runs compose up, but there is no per-service granularity.
For homelabs with 10-20 containers in one compose file, a bad image tag on one service affects all services. See bosun-22q.
Remote Deploy Partial Failure
Remote deployments use tar-over-SSH for config sync followed by docker compose up over SSH. If the SSH connection succeeds for config sync but fails during compose up, the configs on disk are ahead of the running containers. The state file will not record this as a successful deploy (correct behavior), but a manual intervention or retry may be needed. See bosun-4t4.
Full Sync on Every Reconcile
The reconciler syncs all deploy paths on every run — it does not diff “what changed in this commit” to selectively sync affected services. For homelab scale this is fast enough, and Docker Compose won’t restart unchanged containers (the --wait flag handles this). This is a deliberate simplicity tradeoff, not a bug.
Backup Scope
Backups capture configuration files only (traefik routes, authelia config, gatus endpoints, compose files). Volume data — databases, application state, media files — is not included. If a compose up triggers a database migration (e.g., new PostgreSQL version), the rollback restores old configs but cannot restore the database to its previous state.
This is inherent to the Docker Compose deployment model. Volume backup/restore is the responsibility of a dedicated backup tool (e.g., Duplicati, Restic, Borg).
Template File-Read Functions
The include and fromJsonFile functions apply a subtree allowlist before
reading. The default root is <infraDir>/templates; template_include_dir or
BOSUN_TEMPLATE_INCLUDE_DIR can select another root. Lexical traversal and
paths whose resolved symlink target leaves that root are rejected with an error
that names the allowed directory.
During reconciliation, Bosun opens the source tree and configured include root
through descriptor-bound traversal before walking them. Each template source,
include, and fromJsonFile read uses the already selected root namespace, and an
ancestor or final-entry replacement fails closed instead of redirecting the
read. The standalone bosun render command pins the selected template’s
parent for the source read and pins the configured include root for each helper
read. Missing, non-directory, symlinked, or escaping roots fail closed;
symlinks that resolve within an allowed root retain normal os.Root semantics.
Templates from untrusted sources remain unsupported.
Template Rendering Scope
Template rendering walks the entire source directory for .tmpl files, while non-template file copying is limited to the infrastructure subdirectory. A .tmpl file placed outside the infra subdirectory will still be rendered. In practice this is harmless (you control the repo), but it expands the rendering surface beyond the intended scope.