Bosun Security Documentation
This document describes the security architecture, practices, and controls implemented in Bosun.
Security Principles
Bosun follows a defense-in-depth approach with these core principles:
- Least Privilege: Operations use minimal permissions required
- Secure by Default: Security measures are always enabled (no opt-out)
- Fail Secure: Errors fail closed, never exposing sensitive data
- Input Validation: All external inputs are validated before use
- Secret Isolation: Secrets are isolated from logs, environment variables, and error messages
Secrets Management
SOPS Integration
Bosun uses SOPS (Secrets OPerationS) for encrypting sensitive configuration files. SOPS provides:
- Encryption at rest for YAML/JSON files
- Support for multiple key management backends
- Partial encryption (only values are encrypted, keys remain readable)
- Git-friendly encrypted file format
Implementation: internal/internal/reconcile/sops.go
// SOPSOps provides SOPS decryption operations
type SOPSOps struct{}
// Decrypt decrypts a SOPS-encrypted file and returns the plaintext bytes
func (s *SOPSOps) Decrypt(ctx context.Context, file string) ([]byte, error)
Age Encryption
Bosun uses age as the encryption backend for SOPS. Age provides:
- Modern, audited cryptography (X25519 + ChaCha20-Poly1305)
- Simple key format (single line text files)
- No external dependencies or key servers
Key Location Priority (checked in order):
SOPS_AGE_KEYenvironment variable (inline key)SOPS_AGE_KEY_FILEenvironment variable (path to key file)- Default:
~/.config/sops/age/keys.txt
Inline key content takes precedence. Resolved files must be regular, non-empty, and contain at least one parseable Age identity. Bosun rejects an invalid file before decryption and explains that Docker can create a directory when a file bind-mount source is missing. When secrets files are configured, daemon and one-shot startup perform this validation before Git; the daemon binds no API listeners on failure.
Key Generation and Storage
Implementation: internal/internal/cmd/init.go
During bosun init, age keys are generated with secure defaults:
// Create key directory with restricted permissions
if err := os.MkdirAll(keyDir, 0700); err != nil {
return "", fmt.Errorf("create key directory: %w", err)
}
// Generate key using age-keygen
keygen := exec.Command("age-keygen", "-o", ageKeyFile)
// Set secure permissions on key file
if err := os.Chmod(ageKeyFile, 0600); err != nil {
return "", fmt.Errorf("set key permissions: %w", err)
}
Security Controls:
| Resource | Permission | Rationale |
|---|---|---|
Key directory (~/.config/sops/age/) |
0700 |
Owner-only access |
Key file (keys.txt) |
0600 |
Owner read/write only |
Key Rotation Procedures
To rotate age keys:
-
Generate new key:
age-keygen -o ~/.config/sops/age/keys-new.txt -
Update
.sops.yamlwith the new public key:creation_rules: - path_regex: .*\.sops\.yaml$ age: age1newpublickeyhere... -
Re-encrypt existing secrets:
# For each encrypted file sops updatekeys secrets.sops.yaml -
Verify decryption works with new key
-
Archive old key securely (do not delete immediately - needed for backups)
Secret Handling in Templates
In-Process Template Data
Implementation: internal/reconcile/sops.go and
internal/reconcile/template.go
Bosun decrypts SOPS input into an in-memory map and passes that map directly as
the Go template root. Templates access a value such as network.unraid_ip with
{{ .network.unraid_ip }}; no plaintext secrets file path is injected into the
template environment. This avoids exposing plaintext secret values in process
arguments, shell history, or a temporary interchange file.
Templates can intentionally materialize secrets in rendered configuration. Access to the staging and deployment filesystems is therefore part of the security boundary; do not treat rendered output as non-sensitive by default.
File Permission Standards
| File Type | Permission | Rationale |
|---|---|---|
| Rendered output files | 0644 |
Current renderer mode; output may contain secrets, so restrict host access accordingly |
| Output directories | 0755 |
Standard directory permissions |
| Staging directories | 0755 |
Standard directory permissions |
Cleanup Procedures
Decrypted secret data is held in Go values; Bosun does not create a plaintext secret interchange file to remove or promise explicit memory zeroing. Each rendered template output produced in the reconciler’s staging directory is first written to a same-directory temporary file. The reconciler removes that temporary file if rendering fails and atomically renames it to the final staging path on success.
bosun render --output has different semantics: it directly creates or
truncates the requested output file and executes the template into it. A render
failure can therefore leave a partial output file. The CLI does not currently
provide the reconciler’s atomic staging-write guarantee.
Reconciliation clears the staging directory before rendering and removes it
after a successful non-dry-run deployment. A failed render or deployment, and a
dry run, can leave rendered staging files for diagnosis. Those files may contain
secrets and must be protected and removed according to the operator’s retention
policy. Successfully written files from bosun render --output are requested
output, not temporary state, and remain until the operator removes them.
Filesystem Confinement
Local deploy copies open the configured application-data root once and resolve directory creation, temporary files, replacements, directory syncs, content hash comparisons, post-write verification, and managed stale-file removals through that trusted root. A swapped ancestor or escaping destination fails the deploy instead of redirecting a privileged write or delete outside the root. The shared appdata root is the boundary, rather than each service directory: an in-root symlink raced into place after type checks can still redirect a service’s write or prune into a sibling service’s tree.
Template sources are opened and read from the descriptor that was validated.
During a directory render, the source tree and the configured
template_include_dir are pinned for the render operation. include and
fromJsonFile preserve the existing IncludeDir allowlist and refuse paths that
leave its root. Rollback archives are extracted beneath a fresh pinned
temporary root and reject realized escapes through chained symlinks or
hardlinks.
Before upgrading, use physical template source and include directory paths;
arbitrary symlinked roots and ancestors are refused, apart from validated system
aliases. Replace symlinked .tmpl files with regular template files. Directory
rendering refuses these links with the template path and replacement remedy,
including links introduced after discovery, so an omitted template cannot
silently turn its previous output into a stale-file prune candidate.
Standard atomic-swap deployment creates its temporary tree directly in the target’s existing parent and uses a same-parent rename sequence. That mode retains its narrower parent-rename guarantee; the root-handle guarantee applies to content-hash destination operations and managed pruning. The target’s parent and its ancestors must therefore remain trusted and non-adversarial while the standard swap selects and renames its sibling temporary tree.
On Unix, reconcile traverses lock ancestors through directory handles, accepts
root- or daemon-owned ancestors (including a root-owned sticky system temporary
ancestor), and requires the final directory to be daemon-owned 0700. It opens
the basename through that final handle, creates new lock files as 0600, and
tightens existing daemon-owned regular files to 0600. Failure to establish
those properties stops reconciliation with the affected path and a
chown/chmod fix. Windows retains its current LockFileEx path and inherited
ACL behavior; Windows reparse-point and ACL trust remain operator
responsibilities.
Stop Bosun before upgrading or repairing its Unix lock path. Provision a
persistent directory for the configured lock, owned by the daemon UID/GID with
mode 0700 (chown DAEMON_UID:DAEMON_GID LOCK_DIR and chmod 0700 LOCK_DIR).
An existing lock file must be a daemon-owned regular file with mode 0600.
Repair unsafe ancestors or symlinked lock paths as named by the error before
restarting Bosun.
The published image creates /var/run/bosun owned by bosun:bosun (uid 1000)
with mode 0700, which satisfies this check for the image’s default user. Images
before 0.46.3 created it 0755, which the check refuses even for that user. A
container run as any other uid, including user: "0:0", is still refused
(final directory is owned by uid 1000, need daemon uid 0), and the lock path is
not configurable. Mount a directory owned by the run uid with mode 0700 at
/run/bosun (/var/run is a symlink to /run in the image). A tmpfs is the
simplest choice, because the lock only needs to live as long as the container:
services:
bosun:
user: "0:0"
tmpfs:
- "/run/bosun:mode=0700,size=1m"
A tmpfs is owned by root; for another uid, add uid=<n>,gid=<n> to its options.
An existing named volume mounted at /var/run/bosun keeps the mode it was
created with, because Docker copies the image’s ownership and mode only into a
new, empty volume. chmod 0700 it by hand, or replace it with the tmpfs.
SSH Security
Upgrading to Strict Host Trust
Before upgrading, provision a persistent known_hosts file inside Bosun at
/config/known_hosts, or set BOSUN_SSH_KNOWN_HOSTS to its mounted path. Include the Git
and deploy hosts; copy any previously verified deploy-host entries from the container’s
~/.ssh/known_hosts into this file before recreating the container. Git does not read
~/.ssh/known_hosts. Collect new candidate keys with ssh-keyscan -p 22 HOST > known_hosts.candidate, inspect them with ssh-keygen -lf known_hosts.candidate, and
verify each retained SHA256 fingerprint against the host console or administrator
through an independent trusted channel before installing the entries. ssh-keyscan alone
does not authenticate a host. Deploys without a Bosun trust file still use strict
OpenSSH default stores; container-only pins must be persisted.
Connection Validation
Implementation: internal/internal/reconcile/deploy.go
SSH connections include multiple security controls:
// SSH connection with security options
cmd := exec.CommandContext(ctx, "ssh",
"-o", "ConnectTimeout=5", // Prevent hanging on unreachable hosts
"-o", "BatchMode=yes", // Disable password prompts (key-only)
host, "exit", "0",
)
Security Options:
| Option | Value | Purpose |
|---|---|---|
ConnectTimeout |
5 seconds | Prevent DoS via slow hosts |
BatchMode |
yes | Disable interactive prompts, enforce key auth |
Host-Key Verification
Bosun resolves SSH trust files in this order:
BOSUN_SSH_KNOWN_HOSTS/config/known_hosts
Git authentication fails before network access when no candidate exists or the
selected file is malformed. Remote ssh and scp operations use
StrictHostKeyChecking=yes with the selected file; without a configured file,
OpenSSH’s normal trust stores remain available for already-pinned hosts, while
an unknown host is refused. Bosun does not use accept-new as an implicit
trust decision, and Git does not consult ~/.ssh/known_hosts.
BOSUN_SSH_INSECURE_HOST_KEY=true is the only opt-out. It accepts any host key
for both channels and is intended only for a one-time bootstrap while a
parseable known_hosts file is provisioned.
Retry on Transient Errors
Bosun implements exponential backoff retry for transient SSH errors:
// Transient error patterns that trigger retry
transientPatterns := []string{
"connection refused",
"connection reset",
"connection timed out",
"network is unreachable",
"no route to host",
"host is down",
"operation timed out",
"i/o timeout",
"temporary failure",
}
Non-transient errors (authentication failures, host key verification) fail immediately.
Timeout Controls
Operation Timeouts (defined in internal/internal/reconcile/deploy.go):
| Operation | Timeout | Rationale |
|---|---|---|
| SSH Connect | 5 seconds | Quick failure detection |
| SSH Commands | 30 seconds | Reasonable for remote ops |
| Rsync Transfer | 5 minutes | Large file transfers |
| Docker Compose Up | 10 minutes | Container pulls/startup |
Host Validation
Implementation: internal/internal/reconcile/validation.go
SSH hosts are validated to prevent command injection:
// Reject SSH option injection (arguments starting with -)
if strings.HasPrefix(host, "-") {
return fmt.Errorf("invalid host: cannot start with '-' (potential SSH option injection)")
}
// Reject shell metacharacters
shellMetachars := []string{";", "&", "|", "$", "`", "(", ")", "{", "}", "<", ">", "\\", "\n", "\r", "'", "\""}
for _, char := range shellMetachars {
if strings.Contains(host, char) {
return fmt.Errorf("invalid host: contains shell metacharacter %q", char)
}
}
// Validate format with regex
hostPattern = regexp.MustCompile(`^([a-zA-Z0-9_-]+@)?[a-zA-Z0-9.-]+$`)
Environment Variable Filtering
Blocked Variables
Implementation: internal/internal/reconcile/template.go
Environment variables are filtered to prevent secret leakage to child processes:
Excluded Prefixes:
| Prefix | Reason |
|---|---|
SOPS_ |
Contains encryption keys |
AWS_ |
Cloud credentials |
AZURE_ |
Cloud credentials |
GCP_, GOOGLE_ |
Cloud credentials |
DO_ |
DigitalOcean credentials |
LINODE_ |
Linode credentials |
VULTR_ |
Vultr credentials |
CLOUDFLARE_ |
Cloudflare credentials |
HETZNER_ |
Hetzner credentials |
OVH_ |
OVH credentials |
API_KEY |
Generic API keys |
SECRET |
Generic secrets |
TOKEN |
Generic tokens |
PASSWORD |
Passwords |
CREDENTIAL |
Credentials |
Excluded Suffixes:
| Suffix | Reason |
|---|---|
_TOKEN |
Auth tokens |
_SECRET |
Secret values |
_KEY |
API/encryption keys |
_PASS, _PASSWORD |
Passwords |
_AUTH |
Auth credentials |
_CREDENTIAL, _CREDENTIALS |
Credentials |
Excluded Exact Matches:
| Variable | Reason |
|---|---|
GITHUB_TOKEN |
CI/CD token |
GITLAB_TOKEN |
CI/CD token |
NPM_TOKEN |
Registry auth |
DOCKER_AUTH |
Registry auth |
REGISTRY_AUTH |
Registry auth |
SSH_AUTH_SOCK |
SSH agent socket |
GPG_TTY |
GPG signing |
Safe Variables (Allowed)
Only these prefixes are passed to child processes:
PATH=- Required for command executionHOME=- User home directoryUSER=- Current userLANG=- Locale settingsLC_- Locale categoriesTERM=- Terminal typeXDG_- XDG base directoriesTMPDIR=,TMP=,TEMP=- Temp directories
Error Output Sanitization
Template errors are sanitized to prevent secret leakage:
func sanitizeStderr(stderr string) string {
// Truncate long output that might contain secrets
const maxLen = 500
if len(stderr) > maxLen {
stderr = stderr[:maxLen] + "... (truncated)"
}
return stderr
}
File Locking
Lock File Implementation
Implementation: internal/lock/lock.go
Bosun uses file-based locking to prevent concurrent operations:
// Lock structure
type Lock struct {
path string // e.g., .bosun/locks/provision.lock
file *os.File
}
// Acquire exclusive lock (non-blocking)
if err := syscall.Flock(int(f.Fd()), syscall.LOCK_EX|syscall.LOCK_NB); err != nil {
// Another process holds the lock
}
Platform Support:
- Unix: Uses
flock(2)system call - Windows: Uses
LockFileExAPI
Lock File Locations
| Lock | Path | Purpose |
|---|---|---|
| Provision | .bosun/locks/provision.lock |
Prevent concurrent renders |
| Reconcile | /var/run/bosun/reconcile.lock |
Prevent concurrent deploys |
Lock File Contents
Lock files contain the PID of the holding process for debugging:
// Write PID to lock file for debugging
f.Truncate(0)
f.Seek(0, 0)
fmt.Fprintf(f, "%d\n", os.Getpid())
The lock file itself is persistent so every process locks the same filesystem inode. Its existence does not mean the lock is held; kernel lock state does. Do not delete the file while a process might hold it, because a replacement file would have a different inode and could be locked concurrently.
Lock Release
Locks are automatically released when:
- The process calls
Release() - The process terminates (kernel releases flock)
- The file descriptor is closed
Path Traversal Protection
Tar Extraction Validation
Implementations: internal/reconcile/remote_rollback.go for reconcile
rollback archives and internal/cmd/emergency.go for emergency restore
archives.
All reconcile rollback consumers use the same in-process archive/tar reader.
It maps each member into a fresh temporary root and validates the realized
destination before writing. It rejects member-name traversal, absolute or
escaping symlink targets, and escaping archive-root-relative hardlink targets
before they can create or redirect content outside the root. Confined relative
symlinks and hardlinks remain supported.
The complete archive must pass before the extracted root is returned. A validation, corruption, I/O, whole-decompressed-stream size, or independent-context failure deletes the partial temporary tree, so local rollback cannot copy or remove live content or pass a partial backup path to compose or orphan cleanup. The stream bound covers tar headers, padding, skipped bodies, and trailing data, and a drain to true gzip EOF validates the trailer checksum. Local rollback’s extraction timeout is derived from a fresh background context, so cancellation of the failed deployment does not suppress recovery; the extractor still honors cancellation or expiry of that independent context.
Emergency restore separately validates archive paths before writing:
// Sanitize path to prevent directory traversal
target := filepath.Join(destDir, header.Name)
if !strings.HasPrefix(target, filepath.Clean(destDir)+string(os.PathSeparator)) {
return fmt.Errorf("invalid file path in archive: %s", header.Name)
}
These controls prevent malicious archives containing paths like:
../../../etc/passwd/etc/shadowfoo/../../bar
File Size Limits
Extracted files are limited to prevent resource exhaustion:
// Limit copy size as a security measure
const maxFileSize = 100 * 1024 * 1024 // 100MB max per file
if _, err := io.CopyN(outFile, tr, maxFileSize); err != nil && err != io.EOF {
return err
}
Input Validation
Validated Inputs
Implementation: internal/internal/reconcile/validation.go
| Input | Pattern | Rejects |
|---|---|---|
| SSH Host | ^([a-zA-Z0-9_-]+@)?[a-zA-Z0-9.-]+$ |
Shell metacharacters, option injection |
| Git Branch | ^[a-zA-Z0-9_/.-]+$ |
Shell metacharacters, option injection |
| Container Name | ^[a-zA-Z0-9][a-zA-Z0-9_.-]*$ |
Shell metacharacters, option injection |
| Docker Signal | Allowlist only | Arbitrary signals |
Shell Metacharacter Rejection
All validated inputs reject these characters:
shellMetachars = []string{
";", "&", "|", "$", "`",
"(", ")", "{", "}",
"<", ">", "\\",
"\n", "\r", "'", "\""
}
Option Injection Prevention
Inputs starting with - are rejected to prevent:
- SSH option injection:
ssh -oProxyCommand=... evil - Git option injection:
git clone --upload-pack=evil ... - Docker option injection:
docker --config=evil ...
Template Security
include Function Scope
The include and fromJsonFile template functions read files from the local
filesystem. Reads are confined to a subtree allowlist — by default
<infraDir>/templates — so requested paths are limited to that subtree during
normal rendering rather than permitting arbitrary filesystem reads:
The implementation validates the allowlist path, then opens it through the
pinned os.Root; include and fromJsonFile read from that returned
descriptor. Validation or descriptor-open errors are returned to the template
renderer.
Threat model: Templates come from your own Git repository, which you
control. The allowlist is a defense-in-depth control so that even a mistaken or
malicious template cannot read the SOPS secrets file, age keys, or bosun.yaml
that live in the infra root alongside (but above) templates/.
Enforcement (internal/reconcile/template.go, shared by reconciliation and
bosun render): the requested path is resolved (filepath.Abs), checked for
lexical containment via filepath.Rel
(a ..-escape, an absolute-outside path, or a Rel error all fail closed), and
then re-checked after filepath.EvalSymlinks so a symlink whose target escapes
the subtree is rejected. A {{ include "/etc/shadow" }} or a read of a sibling
*.sops.yaml is now rejected with an error that names the allowed root. The
allowlist confines by location: a hardlink whose path is inside templates/
remains readable. Reconciliation pins the source directory and include root
before walking them; each source/include read is then opened and read through
that descriptor-bound namespace. The standalone bosun render command pins
each selected template’s parent for the source read and the configured include
root for helper reads. A root or ancestor that is missing,
non-directory, or symlinked is refused, so a concurrent replacement cannot
redirect a read outside the selected tree. The final guarantee is the pinned
namespace; symlinks that resolve within that namespace remain subject to the
usual os.Root semantics.
Configuration: the allowlist root is <infraDir>/templates by default and
is overridable with the template_include_dir config field or
BOSUN_TEMPLATE_INCLUDE_DIR (relative values resolve against the infra dir,
absolute values are used as-is). This is a breaking change for configs that
included files from outside templates/; move those files under the subtree or
point the override at their location. For bosun render, the infra directory is
the discovered project root plus BOSUN_INFRA_DIR; outside a Bosun project it
is the current directory.
Template Rendering Scope
The template renderer walks the entire cloned repository directory for .tmpl files. Non-template files are only copied from the infrastructure subdirectory. This means a .tmpl file placed outside the expected path will still be rendered, potentially producing unexpected output files.
Mitigation: Limit .tmpl files to the infrastructure subdirectory in your repository. Consider code review rules that flag .tmpl files in unexpected locations.
Auth Ingress Chain
The auth stack — Traefik, Authelia, and Tailscale gateway — forms a dependency chain for external access. A partial compose up failure where Authelia is down but Traefik is up could serve routes without authentication middleware.
Current mitigations:
- Traefik’s
forwardAuthmiddleware fails closed — if Authelia is unreachable, requests receive 502 errors rather than passing through unauthenticated - Post-deploy drift verification catches missing or unhealthy containers and logs warnings
- All external routes defined via provisions include
forwardAuthmiddleware by default
Planned: A dedicated health gate that verifies the full auth chain is healthy before declaring a deploy successful. See bosun-r9n.
Daemon Webhook Authentication
The daemon’s HTTP trigger endpoints (/webhook, /webhook/github,
/webhook/manual) authenticate requests with an HMAC-SHA256 signature over the
request body, validated with constant-time comparison against WEBHOOK_SECRET
(GitHub convention: X-Hub-Signature-256; generic/manual: X-Signature).
Webhook auth fails closed. When no webhook secret is configured, every
trigger request is rejected with 403 and no reconcile runs. This is the
default posture; a missing secret MUST NOT silently grant anonymous callers
deploy control. The daemon logs a loud warning at startup naming the active
posture.
Operators on isolated, trusted networks MAY opt out with
BOSUN_ALLOW_UNAUTHENTICATED_WEBHOOK=true (strict lowercase match, like the
other escape hatches). With the opt-out active:
- The daemon warns at startup that unauthenticated triggers are enabled
- Every accepted unauthenticated request logs a
SECURITY:warning with the caller’s remote address
The opt-out never bypasses a configured secret — when WEBHOOK_SECRET is set,
signature validation always runs.
The standalone bosun webhook receiver fails closed the same way. It
forwards to the daemon over the Unix socket, which authorizes by peer
credential and never re-applies the daemon’s own HTTP webhook gate — so the
receiver’s HTTP port needs its own. When no secret is resolved (--secret,
WEBHOOK_SECRET, GITHUB_WEBHOOK_SECRET, or --fetch-secret), every receiver
trigger endpoint (/webhook, /webhook/github, /webhook/gitlab,
/webhook/gitea, /webhook/bitbucket) rejects requests with 403. The
receiver reads the same BOSUN_ALLOW_UNAUTHENTICATED_WEBHOOK=true opt-out,
warns loudly at startup about whichever posture is active, and logs a
SECURITY: warning per accepted unauthenticated request. /health and
/ready change nothing and stay open.
This matters most on the implicit daemon-fetch path: a receiver started before the daemon gets no secret, and previously served with signature validation silently disabled while still forwarding triggers.
Webhook attribution is treated as untrusted even after signature validation.
Both the daemon endpoint and the standalone webhook receiver strip control,
formatting, and line-separator characters and cap the remaining value at 256
Unicode code points before writing it to logs or using it in the reconcile
source propagated to tracing and Sentry. The receiver applies this to the
pusher name and the pushed ref of every provider it accepts — GitHub, GitLab,
Gitea, and Bitbucket — and the daemon’s Unix socket /trigger handler applies
it again to any caller-supplied source, so the property does not depend on
each client sanitizing before it forwards. The same sanitization applies when
the explicit unauthenticated-webhook opt-out is active.
The same helper neutralizes container health-check output, which is the stdout and stderr of a command running inside a monitored container and is therefore controlled by anyone with code execution there. Both the drift printout and the health-gate error path strip before capping, so a control character cannot survive by sitting beyond the truncation point.
All daemon HTTP transports — the webhook listener, Unix socket API, and
optional bearer-authenticated TCP API — allow at most 5 seconds to receive
request headers and set a 32 KiB request-header parsing limit. These
pre-handler limits reduce exposure to slow-header and oversized-header
denial-of-service attacks without changing the existing per-operation
BOSUN_API_TIMEOUT behavior.
Unix socket publication and permissions
The daemon publishes its Unix control socket at mode 0660 by default and
honors an explicitly configured daemon.Config.SocketMode. On Unix, Bosun
binds the listener inside a private 0700 staging directory, applies the
configured permission bits there, and atomically publishes the socket at its
final path without replacing a racing entry. The public path therefore never
exists at the operating system’s more-permissive default mode, and Bosun does
not change the process-global umask while other goroutines may be creating
files.
On Unix, startup refuses to delete a stale-path entry unless it is actually a Unix socket; symlinks, regular files, and directories are left untouched. Shutdown removes the socket only when the path still identifies the inode Bosun published, so a replacement entry is never deleted as though Bosun owned it. Windows builds retain AF_UNIX compatibility, but Unix permission bits are not an access-control boundary there; platform ACLs and request authorization remain authoritative.
Unix socket peer authorization
Socket file mode controls which processes can connect, but does not by itself
authorize a deployment. On Linux, Bosun reads SO_PEERCRED for each accepted
connection and authorizes privileged socket requests only when the peer UID is
the daemon’s effective UID or appears in BOSUN_SOCKET_ALLOWED_UIDS. The
allowlist is a comma-separated list of numeric UIDs; malformed, negative, or
out-of-range entries fail configuration validation instead of partially
applying the list.
Missing peer credentials and non-Linux platforms fail closed: POST /trigger
returns 403 and does not reconcile. Operators who deliberately use socket
permissions as their entire trust boundary can set
BOSUN_ALLOW_UNAUTHENTICATED_SOCKET=true (strict lowercase match). This
security opt-out is logged at startup and for every accepted unauthenticated
request the check admits.
GET /config carries the same peer check, despite being a read. It returns
the webhook secret for the daemon-injected-secrets pattern, and that secret is
exactly what the HTTP listener’s trigger endpoints accept — so a peer refused
by POST /trigger could otherwise read the secret here and sign a forced
trigger against the HTTP port instead. The boundary is drawn on effect, not on
HTTP verb. BOSUN_ALLOW_UNAUTHENTICATED_SOCKET=true also disables this
/config peer check: any process able to connect can then retrieve the webhook
secret as well as trigger reconciliation. GET /status and GET /health return
no credential and remain governed by the socket’s filesystem permissions.
A bosun webhook receiver using --fetch-secret must therefore run as the
daemon’s UID or as a BOSUN_SOCKET_ALLOWED_UIDS member. A receiver that fails
the check could never have forwarded a trigger anyway, so this costs a
correctly configured deployment nothing.
Public Health and Operator Diagnostics
GET /health stays unauthenticated on the webhook HTTP listener, TCP API, and
Unix socket so container and load-balancer liveness probes do not need a control
credential. Its JSON response is deliberately bounded to exactly status,
ready, and uptime. It returns 200 when healthy, 503 when degraded or
unhealthy, and 405 for non-GET requests.
The public response never contains reconcile errors or timestamps, repository URLs or filesystem paths, subsystem names or messages, or circuit-breaker state. Supplying an Authorization header does not expand the response, which prevents caches and proxies from mixing public and credential-bearing variants.
Operators obtain sanitized last-error and last-reconcile diagnostics from
GET /status: the TCP API requires its bearer token, while Unix-socket access is
governed by the local socket trust boundary. Subsystem detail remains in daemon
logs. The standalone bosun webhook receiver preserves the bounded /health
schema and rejects non-GET /health and /ready requests.
Daemon Metrics and Widget Authentication
The /metrics (Prometheus) and /api/widget (Homepage) endpoints disclose the
deployed commit and daemon health — information disclosure on an all-interfaces
bind. They authenticate with a read-scope bearer token, kept deliberately
separate from the control bearer.
BOSUN_METRICS_TOKENis the read-scope credential a scraper presents asAuthorization: Bearer <token>. It authorizes only these read endpoints.BOSUN_BEARER_TOKEN(the TCP control bearer) also authorizes/triggerand/api/restart. Because it is strictly more privileged, it is also accepted on the read endpoints — but a scraper MUST NOT be given it, since that would hand a control credential to a monitoring system. Provision scrapers withBOSUN_METRICS_TOKENalone.
Metrics auth fails closed. With neither token configured, /metrics and
/api/widget reject every request with 403 and the daemon logs a loud warning
at startup. Operators on isolated, trusted networks MAY opt out with
BOSUN_ALLOW_UNAUTHENTICATED_METRICS=true (strict lowercase match); with the
opt-out active the daemon warns at startup and logs a SECURITY: warning per
accepted unauthenticated request. A configured token always takes precedence
over the opt-out.
The bind address defaults to all interfaces because container-side callers
(reverse proxy, metrics scrapers, dashboards) reach the daemon over the docker
bridge, not loopback. BOSUN_LISTEN_ADDR narrows the bind where the network
topology allows it; the bind address is not an authentication control — the
Unix socket trigger, the webhook endpoints, and the /metrics and /api/widget
endpoints each enforce their own auth as described above.
Operator Escape Hatches as Risk Surface
Bosun ships several “escape hatch” environment variables that disable safety checks for legitimate operational scenarios (genuinely empty repos, diagnostic deploys, transient infrastructure conditions). These flags reduce the system’s defense-in-depth and SHOULD be treated as a risk surface in their own right.
Inventory
| Env var | What it disables | Default | Intended use |
|---|---|---|---|
BOSUN_ALLOW_EMPTY_DECLARED_STATE |
ErrNoDeclaredServices gate at pipeline stage 6 (no parseable services in staging compose dir) |
false (strict) |
Genuinely empty repos, scaffolding |
BOSUN_SKIP_DEPLOY_INVARIANT |
Post-deploy created/written path existence, type, fresh-mtime, and no-file-write content gates at pipeline stage 9 | false (strict) |
Diagnostic deploys, repro of intermittent issues |
BOSUN_SSH_INSECURE_HOST_KEY |
SSH host-key verification | false (strict) |
Initial bootstrap before known_hosts is populated |
BOSUN_ALLOW_UNAUTHENTICATED_WEBHOOK |
Fail-closed webhook authentication (secret-less trigger requests rejected with 403) |
false (strict) |
Trusted, isolated networks where a webhook secret is impractical |
BOSUN_ALLOW_UNAUTHENTICATED_METRICS |
Fail-closed metrics/widget authentication (token-less /metrics and /api/widget requests rejected with 403) |
false (strict) |
Trusted, isolated networks where a metrics token is impractical |
BOSUN_ALLOW_UNAUTHENTICATED_SOCKET |
Fail-closed peer-credential authorization for mutating Unix socket requests and secret-bearing /config |
false (strict) |
Platforms without peer credentials, when socket permissions are an intentionally sufficient trust boundary |
ErrComposeDirMissing (stage 6) has no escape hatch by design — a missing staging compose directory always indicates a misconfigured deploy path, never an intentional state.
Attack scenarios these hatches enable
- Tampered-staging persistence: an attacker who can write to the staging directory but cannot reach the destination filesystem (e.g., compromised renderer, leaked SOPS key without root) cannot persist tampered content past stage 9 by default. If
BOSUN_SKIP_DEPLOY_INVARIANT=trueis set in the daemon environment, the post-deploy gate is disabled and tampered content can land if the attacker also disables content-hash sync. Theoverride=truelog line is the canary. - Misconfigured-staging masking: an attacker who can manipulate
BOSUN_INFRA_DIR(or the rendered staging path) to point at an empty directory would, by default, tripErrNoDeclaredServicesand halt the reconcile. WithBOSUN_ALLOW_EMPTY_DECLARED_STATE=trueset, the deploy continues against an empty service set, which can be used to silently take services down by removing them from the apparent declared state. - Bootstrap MITM:
BOSUN_SSH_INSECURE_HOST_KEY=trueaccepts any host key, opening a window for MITM during the bootstrap deploy. If this flag persists past first-deploy, an in-network attacker can substitute the deploy target.
Operator guidance
- Never set escape hatches in the daemon’s persistent environment (systemd unit, Docker
env_file, etc.). They should be set on the command line for a single one-shotbosun reconcileinvocation and never persist. - Alert on
override=truelog lines — the reconcile pipeline emits aWarn-level log withoverride=trueevery time a strict gate is bypassed. A monitoring rule on this field surfaces accidental persistence within one deploy cycle. - Treat any deploy with an escape hatch active as suspect for that cycle — re-run a strict deploy as soon as the underlying condition (empty repo, FUSE issue, missing
known_hosts) is resolved. bosun doctorflags persistent escape hatches — if any of these vars is set in the daemon environment at boot time, it appears in the diagnostic output as aWARNING(notERROR— these are legitimate operational tools, just risky if forgotten).
Why we ship escape hatches anyway
The alternative is forcing operators to patch and rebuild Bosun for every edge case. That moves the risk surface from a one-line env var (visible in monitoring, easy to roll back) to a forked binary (invisible, hard to roll back). Escape hatches make the unsafe path the visible path.
Best Practices
Key Management
- Generate unique keys per environment (dev, staging, production)
- Store production keys in HSM or cloud KMS when possible
- Rotate keys annually or after personnel changes
- Never commit private keys to version control
- Use
age-keygenrather than importing existing keys
Secret Rotation
- Rotate secrets regularly (quarterly minimum)
- Update encrypted files when rotating secrets
- Use unique secrets per service (no shared credentials)
- Audit secret access via git history of
.sops.yamlfiles
Audit Logging
- Git history tracks all secret file changes
- Lock files contain PIDs for debugging
- SSH connections use BatchMode (logged by SSH daemon)
- Timeouts prevent silent failures
Deployment Security
- Use SSH keys with passphrases
- Configure known_hosts before first deployment
- Set ConnectTimeout to prevent hanging
- Verify host keys to prevent MITM attacks
Security Checklist
Before deploying with Bosun:
- Age keys generated with
0600permissions -
.sops.yamlconfigured with correct public key - SSH keys configured on target hosts
-
known_hostspopulated for all targets - No secrets in environment variables
- No secrets in git history (use
git-secretsor similar) - Production keys stored securely (not on developer machines)
- Audit log retention configured