javascript.best

devstack

~ by

devstack — write-up

An overview of every part of the system: what it is, why it's there, and (where the history records it) how it got that way. For the full reasoning, DESIGN.md is authoritative; section numbers below (§n) point into it. Diagram: structure.svg.

What it is, in one paragraph

devstack is a single Python CLI that turns one config file (devstack.toml) into one hardened incus system container (Debian 13) on a Linux host. That container is both a dev box (VSCode Remote-SSH in, project folder shared from the host) and a local copy of production (Caddy with local TLS → hardened Node app → Postgres). The core idea is a layer between untrusted code (npm packages, extensions, coding agents) and the host: the code can do whatever it likes inside the container, while what reaches the host is narrow and enforced by the OS rather than by convention.

How it got here (git history)

Date Commit What changed
2026-10-03 1d9ebdb v1. The tool, six phases, journal + snapshot undo, dnsmasq DNS, Caddy (with its own CA), the project scaffold, guide.html.
2026-10-04 5a5d817 "security fixes": host-made name-constrained CA, devstack-requires.toml (config split), undo path checks, config risk review (typed yes), IANA TLD check, first tests, CI, README/SECURITY. guide.html → index.html.
2026-10-04 8a77bb9 v2 "more container hardening", after testing a v1 container (§1.4): fixed service users, nftables egress profiles, runners, limited sudo + devstack-sync, incus network ACL, Caddy admin socket, app data volume, test DB. Trust zones and "decisions" tables added to DESIGN.md.
2026-10-05 6271923 SBOM (syft in container, grype on host), devstack upgrade, AGENTS.md. Hand-written SBOM.md replaced by generated CycloneDX files. Pin verification job (verify_pins.py).
2026-10-10 b6a76bd "improve integrity": checkpoints, devstack browse (confined Chromium), dev servers, Caddy same-origin rule, all of .git read-only (was only hooks/config).
2026-10-10 6515d4e gitleaks (leaks scan) and the project lint/type toolchain (ESLint, tsc, pnpm supply-chain settings). Trivy and Semgrep considered and rejected.
uncommitted — README owner's note, table reformatting; the TOML writer handles inline tables so the requirements diff can show runners (+ test).

The big turn was v1 → v2. v1 relied on settings that the OS didn't actually enforce. The clearest case is systemd's IPAddressDeny, which is silently ignored in an unprivileged container. v2's rule: only rely on what the OS enforces, and never describe a protection that isn't enforced (§1.4).

1. The repository (the tool itself)

Path What / why Origin
devstack Bash launcher. Finds a Python ≥ 3.11 (needed for tomllib), trying /usr/bin/python3 before PATH because pyenv/conda shims often shadow it. v1
devstack.py The whole tool, ~7.2k lines, stdlib only (no pip, so nothing extra to trust or install). Sections: helpers · config schema · logging · runner/state/journal · host+container helpers · phases · orchestration/undo · commands. v1, grown every commit
devstack.example.toml Every config key, with its default and help text. Generated from the SCHEMA table (./devstack init -n -c devstack.example.toml --force), so docs and code can't drift. v1
templates/project/ Scaffold copied into a new project (never overwrites existing files). v1; toolchain files added in 6515d4e
templates/system/ Files rendered into the container: sshd drop-in, app unit, runner socket/service, egress.nft, devstack-egress.service, devstack-sync, Postgres conf. v1; isolation files in 8a77bb9
data/iana-tlds.txt Snapshot of real TLDs. dns.tld may not be one, because address=/com/… in dnsmasq would hijack every .com name on the host. 5a5d817
tests/test_devstack.py ~215 unit tests that need no host, root or incus: config, rendering, quoting, journal/undo with a fake runner, CA constraints (real openssl), ACL rules, SBOM normalizer, gitleaks argv, checkpoints on a temp git repo. 5a5d817, grown since
tests/verify_pins.py Downloads every pinned artifact and checks its sha256. Network-only, so it runs weekly in CI and never on PRs. 6271923
.github/workflows/ci.yml bash -n, py_compile, config check, unittest; plus the weekly pins job. This CI is for devstack only. It is not a template for projects. 5a5d817, pins job 6271923
DESIGN.md Maintainer/agent doc and the authoritative spec. Includes "decisions (reopen only with new facts)" and "security details that look simplifiable but aren't" (§10.1). v1, restructured in v2
README.md / SECURITY.md Quick start, security model summary, the trust-roots table / threat model and what's in or out of scope. 5a5d817
index.html + .woff2 font End-user manual ("devstack Field Guide"): setup, phases, undo, security and troubleshooting. The font is self-hosted for it. guide.html in v1, renamed 5a5d817

Template convention: templates hold content, devstack.py holds logic. Placeholders are @@UPPER_SNAKE@@, because ${}/{} clash with TypeScript and Caddyfile syntax. Rendering fails if any placeholder is left unreplaced.

2. Configuration model

Piece What / why
SCHEMA (in devstack.py) Single source of truth. Each key has a type, default, help text and validator. init, config explain, validation and the example file all come from it. Unknown keys are errors, with "did you mean" suggestions.
devstack.toml Private per-user config (git-ignored): host paths, ssh, DNS and mount sources. Treated like a script: risky settings (mounting $HOME/~/.ssh//etc, full sudo, open app profile, LAN exceptions, new runners) need a typed yes/--yes.
Derived names Every name that could collide (container, TLD, volumes, ssh key, dnsmasq file, units, users…) defaults to {stack.name}…, so a second stack only needs a new name. Preflight flags conflicts with anything this stack doesn't own.
<project>/infra/devstack-requires.toml The project's share of the config, committed with the project: container-side keys only (image, Node, Caddy, app, runners, Postgres, GPU on/off) plus required mounts by name and path, never by host source. It is read with an allowlist, so a cloned repo can't steer the host. Any change from what this stack last wrote needs approval. Added in 5a5d817 so configs (with host details) never land in public repos.

3. Execution model

Piece What / why
Phases base → app → proxy → standards → isolation → postgres. Each one adds a single concern (contain, serve, proxy, client-side standards, server-side isolation, data). Each is a list of steps with describe()/is_done()/apply(), followed by verify checks. Steps are idempotent, so up can be re-run safely.
preflight / plan Read-only. They check host deps, incus, name conflicts and config risks, and list every host path that would be written. Run them first.
Undo: incus snapshots A devstack-pre-<phase> snapshot is taken before each phase after base. Undoing a phase restores it, which reverts packages, files, users and devices without inverse logic for each command (§7.1).
Undo: journal Everything outside the container is written to journal.jsonl before it's changed (write-ahead). Undo uses a fixed registry of entry kinds, never shell strings. Undo also re-checks each stored path against what the current config would produce, so a tampered journal can't redirect sudo rm (§7.2).
Failure handling run.on_failure = rollback (default) undoes the failed phase; stop keeps it for debugging. fcntl lock: one run at a time. The design goal is "no half-finished state you can't get out of".
Checkpoints (b6a76bd) Undo after provisioning (an agent's changes, a bad install). Taken with the container stopped: container snapshot + volume snapshots + project as a commit on a hidden ref refs/devstack/checkpoints/<name> + ignored files as a tar. Branches and HEAD are untouched. git runs hermetically, so project-written config can't execute on the host.
State dir ~/.local/state/devstack/<stack>/: state.json, journal, lock, log, backups/, ca/, reports, browser profile.
Logging Console at info; the file always gets debug (every command, exit code, output). Secrets only go through stdin or files, never argv, and a redaction filter is a second line of defence.
What survives what Data volumes (Postgres, app data) survive undo and destroy unless --purge-data, with an auto-backup first. Re-running up reattaches them.

4. On the host

Component What / why Origin
dnsmasq drop-in One file per stack: *.<tld> → container IP. This lets the app run at its real domain plus a dev suffix (app.example.com.webstack). The host must already resolve through dnsmasq (a one-time manual setup, since it changes host-wide DNS). devstack never touches resolv.conf. v1
Local CA Root + key are created on the host and X.509 name-constrained to *.<tld>. The container only gets an intermediate. A key stolen from the container can only sign dev names. (v1 used Caddy's own CA, whose key lived in the container and was trusted for every site.) 5a5d817
ssh Per-stack ed25519 key, a marked block in ~/.ssh/config, and a per-stack known_hosts read via incus exec, so there's no trust-on-first-use. Key-only login, no agent or X11 forwarding. v1
incus network ACL Attached to the container's NIC and enforced on the host. Rejects traffic to the host and LAN except DNS/DHCP to the gateway. Added because in v1 nothing stopped container → host/LAN (§1.4). base verifies it with its own temporary host listener. 8a77bb9
Read-only .git All of .git is a read-only disk device. The host runs git there, so hooks or core.fsmonitor would be host code execution, and refs/grafts decide what git log shows. Commits happen on the host. Started with hooks/config only (5a5d817); widened to all of .git in b6a76bd. 5a5d817→b6a76bd
Data volumes <stack>-pgdata and <stack>-appdata are incus custom volumes. They aren't part of instance snapshots on purpose: rolling back the container never rolls back data. v1 / 8a77bb9
devstack browse (b6a76bd) Chromium with its own profile and NSS db (the only browser that trusts the dev CA). Everything except the dev domain goes to a dead-end proxy, including loopback. The reason: container pages in your everyday browser could reach host loopback services and the LAN, which the ACL can't stop. tls.browser_trust therefore defaults to no. b6a76bd
grype, gitleaks Run on the host against files the container produced: "the container is the artifact, the host is the auditor". The scanner stays out of the zone it audits, and the container never needs the ~1 GB vuln DB. 6271923, 6515d4e

5. Inside the container

Trust zones (§1.3). The login user and every agent share one uid, so they can't be told apart. The rule is: nothing the login user can edit runs as root, and security-relevant changes come from the host (devstack up).

Component What / why Origin
Login user dev uid = host uid, so files on the shared folder keep the right owner. Has internet access and DB access (as the app role) by design. No general sudo: in v1 the user had passwordless sudo, which made every agent root. The name is generic so it doesn't leak into committed files. v1, limited in 8a77bb9
devstack-sync + sudoers The only command sudo allows. Fixed verbs: packages (reads names on stdin, so root never opens a user-controlled file; exact Debian names only; services prevented from starting and new units masked), restart app/runners, validate/reload caddy (as the caddy user), status. Lets agents install apt packages through a reviewable list in git. 8a77bb9
sshd drop-in Key-only, AllowUsers dev, no root login, no agent/X11 forwarding. TCP forwarding stays on because Remote-SSH needs it. v1
Node + pnpm The Node tarball is verified with gpgv against the Node release keyring, pinned by commit + sha256. pnpm comes via corepack at a pinned version at least a week old. v1, pins 5a5d817
Caddy TLS (leaf certs from the intermediate), static files, /api/* → app. Two Caddyfiles: root-owned /etc/caddy/Caddyfile holds global options (admin API on a unix socket, not TCP 2019) and the same-origin snippet, then imports the project's infra/Caddyfile. A project file can't redefine global options because a second global block fails to parse (fails closed). v1; admin socket 8a77bb9; same-origin b6a76bd
Same-origin rule Caddy returns 403 for requests a browser marks as cross-site/same-site or with a foreign Origin, so other sites can't call the app with your cookies. b6a76bd
App service Hardened systemd unit (ProtectSystem=strict, syscall filter, no capabilities…). Fixed user <stack>-app; binds 127.0.0.1 only. It only accepts requests carrying X-Internal-Token, which only Caddy adds. v1 used DynamicUser, which left no stable uid to firewall or own files. v1, reworked 8a77bb9
Egress profiles egress.nft matches on socket owner uid (meta skuid). The app gets local (loopback only, no DNS, to prevent DNS tunnelling); runners get public internet only. Loaded by devstack-egress.service, and units Require= it, so there are no rules → no app (fails closed). It replaced systemd IPAddressDeny, which silently didn't work. 8a77bb9
Runners Socket-activated jobs for tools that need the internet or parse untrusted files (fetchers, ffmpeg…). One short-lived process per connection; own user; no secrets; no access to app data except incoming/. 8a77bb9
ufw Inbound deny by default; allows 22/80/443. Debian's nftables.service is deliberately not used, because its flush ruleset would wipe ufw's rules. v1
Postgres From the PGDG repo (key fingerprint pinned). Listens on localhost, scram auth, peer-only superuser. App role + DB + <db>_test. Data on the persistent volume; postgres uid fixed (500) so a rebuilt container still owns the data. db restore runs as the app role in one transaction. v1, test DB 8a77bb9
Secrets /etc/app-secrets/: internal token and postgres.env, generated once with secrets.token_hex and never passed on argv. The dev env file is written as the user, so root never writes through a path the user controls. v1 / 8a77bb9
syft Pinned binary in the container. Generates the SBOM where the real dpkg DB and node_modules are. 6271923

6. The scaffolded project

Created only if missing; existing files are skipped and reported. The toolchain files (ESLint, pnpm workspace, lint script) are only added to new projects, because adding them under an existing lockfile could fail a rebuild.

File What / why Origin
server.ts, static/, error/ A zero-dependency TypeScript smoke-test app (/api/health, token middleware; fails closed in production without a token) and its static page and error page. A proof the stack works, not a framework. v1
package.json, tsconfig.json ESM, pnpm pinned through packageManager; strict + noUncheckedIndexedAccess. TypeScript is held at 6.0 until typescript-eslint supports 7. v1, 6515d4e
eslint.config.js strict-type-checked, security + no-unsanitized plugins, no eval, child_process only in one wrapper. Stands in for a SAST tool; Semgrep was rejected because it can't be pinned as a single binary and phones home. 6515d4e
pnpm-workspace.yaml Supply-chain settings: 7-day minimumReleaseAge, blockExoticSubdeps, trustPolicy: no-downgrade, strictDepBuilds. 6515d4e
scripts/sync-env.sh sudo devstack-sync packages < infra/apt-packages.txt, then pnpm install --frozen-lockfile. 8a77bb9
scripts/lint-changed.sh Lints what changed. A command run in the container, not a git hook: a hook would be project code executed on the host (§8.7). 6515d4e
infra/apt-packages.txt System packages the project needs, one per line. Each addition is a reviewable diff. 8a77bb9
infra/Caddyfile Routes, CSP and headers. Generated (managed) by default; a project can own it. v1
infra/compress-static.sh Writes .br/.gz copies so Caddy can serve precompressed files. v1
infra/sbom/{container,app}.cdx.json CycloneDX inventory, committed. Normalized (no timestamps, ids or file paths, sorted) so a diff always means a real dependency change. Replaced the hand-written SBOM.md from v1. 6271923
infra/devstack-requires.toml See §2. 5a5d817
AGENTS.md Tells coding agents what the container enforces (no root, .git read-only, how to add packages, secrets rules), so they don't learn it by hitting walls. 6271923
README.md, .gitignore Human orientation and dependency policy; ignores .env, keys and precompressed copies. v1

7. Supply chain and auditing

Piece What / why
Trust roots The Caddy and PGDG apt key fingerprints, the Node keyring commit + sha256, and syft/grype/gitleaks version + sha256 are constants in devstack.py. A unit test fails if a pin is missing from README's table. A plain sha256 was preferred over cosign as a much smaller trust root.
sbom generate/check/scan syft in the container → normalize → commit. grype on the host scans it. The gate counts only fixable findings: on a real Debian system all 554 critical findings had no fix, and a permanently red gate gets switched off. Exit 5 = "gate says no", distinct from tool errors. Not a 7th phase (§8.7 explains why).
upgrade Patches Debian packages from the host, not via devstack-sync, so the login user gains no new capability. Uses --force-confold, otherwise a caddy upgrade could restore the vendor Caddyfile and put the admin API back on TCP. It only lets already-running services restart and masks new units.
leaks scan gitleaks on the host, over history plus a copy of the working tree and staged files. Every project-controlled input (.gitleaks.toml, .gitleaksignore, inline gitleaks:allow, .gitattributes -diff) is replaced or bypassed, because the container can write all of them. Fails closed on gitleaks' silent git errors.

8. Deliberately not there

Recorded in DESIGN.md §1.5 so they aren't re-added without new facts:

  • Git hooks (lint, SBOM, leaks): host-executed project code, not cloned, skippable with --no-verify.
  • CI files for projects: devstack gives commands and exit codes; which CI a project uses is its own choice.
  • Domain allow-lists / egress proxy: can't be enforced at the netfilter layer. Listed as an open item.
  • Trivy: overlaps syft + grype, has weaker secret rules, and its release pipeline was compromised in 2026.
  • Semgrep: pip-distributed (no single binary to pin) and sends metrics. To revisit as Opengrep with vendored rules.
  • Automatic host DNS setup: it changes how the whole host resolves names, so it's documented as a manual step.

Known limits (accepted)

The login user (and so every agent) can read the real database and app data and has internet access, so it can send that data out. An app that passes arbitrary URLs to a runner gets internet through it. Membership of incus-admin is root-equivalent on the host. Per the README's owner note, the generated code hasn't been fully reviewed yet; treat the project as a reference, not a hardened product.