About systemd-sentry.

01The Genesis: Moving Beyond Blind Restarts

System supervision in Linux has historically been trapped between two extremes: either a crude Restart=always directive in a systemd unit file that blindly restarts a crashing service until system resources are exhausted, or an over-engineered observability stack requiring hundreds of megabytes of RAM, external agents, and complex network telemetry pipelines.

systemd-sentry was designed from first principles to bridge this gap: an autonomous, zero-trust host supervisor that detects failures in real time, extracts rich kernel and journal telemetry, synthesizes root cause diagnostics, and applies deterministic circuit breaking to permanently stop flapping failure storms.

02Unix Philosophy: Mechanism vs. Policy

Following classical Unix design principles codified by McIlroy, Thompson, and Ritchie:

• Separation of Mechanism & Policy. The telemetry drivers (sentry-driver) and diagnostic engine (sentry-diagnostic) provide the mechanism of discovering why a failure occurred. The declarative policy engine (sentry-safety) enforces the policy of what actions are permissible.
• Silence on Success. When managed services are healthy, systemd-sentry produces zero log noise and sleeps inside the kernel epoll_wait event loop with 0.0% CPU utilization.
• Composable Streams & Standard Exit Codes. All CLI commands emit clean, parseable text or structured JSON (via --json), and return standard exit codes (e.g. EX_CONFIG = 78 for pre-flight syntax errors).

03Linus Torvalds Pragmatism

Engineered for real-world production Linux servers with zero patience for bloat:

• Zero Unnecessary Abstractions. Direct interaction with Linux kernel subsystems via rustix syscalls: cgroups v2 (/sys/fs/cgroup/), Pressure Stall Information (/proc/pressure/), and raw Journal Export streams.
• Zero Panics in Daemon Paths. Strictly zero runtime calls to .unwrap() or .expect() on active daemon event paths. All I/O and protocol errors propagate through structured error hierarchies or safely fall back to deterministic heuristics.
• Strict <15MB RSS Memory Budget. Bounded data structures throughout: 100-item incident ring buffer, 512-unit circuit breaker registry, 64 KiB maximum policy file size, and automatic load shedding when memory pressure is detected.

04systemd Design Alignment

systemd-sentry integrates as a native, first-class citizen of modern systemd environments:

• Socket Activation. Inherits listening sockets via $LISTEN_FDS (file descriptors 3..N), allowing the systemd manager to buffer client requests before the daemon even starts.
• Drop-In Configuration Directories. Full support for /etc/systemd-sentry/policy.d/*.toml drop-in files, merged in strict alphanumeric order to allow seamless configuration management via Ansible, Puppet, or Nix.
• Credential Security. Integrates with systemd-creds for encrypted API keys and credential passing, eliminating plaintext secrets in configuration files or process command-line arguments.
• Privilege Dropping & Hardened Sandboxing. Runs as unprivileged user sentry in the systemd-journal group with ProtectSystem=strict, ProtectHome=yes, and MemoryDenyWriteExecute=yes.

05Codebase Architecture

The workspace is partitioned into specialized, single-responsibility crates with strict file length constraints (≤256 LOC per file):

systemd-sentry/
├── crates/
│   ├── sentry-core/        Domain models, cgroups v2, PSI, coredumps, error types
│   ├── sentry-driver/      Pure Rust socket activation, sd_notify, journal, D-Bus
│   ├── sentry-diagnostic/  LLM clients (routerd, llama.cpp, OpenAI) & fallback triage
│   ├── sentry-mcp/         Model Context Protocol stdio server for AI agents
│   └── sentry-safety/      Sliding-window circuit breaker & drop-in policy gatekeeper
├── qa/
│   ├── unit/               1:1 Unit QA test suite (200 tests)
│   ├── e2e/                4-tier E2E feature verification suite (87 tests)
│   ├── fixtures/           Crashing mock binaries (segfault, OOM, flapper)
│   └── fuzz/               cargo-fuzz targets (journal, D-Bus, PSI, triage)
├── systemd/                Unit files (.service, .socket, sysusers.d, tmpfiles.d, dbus-1)
└── site/                   Static documentation & GitHub Pages website