mirror of
https://github.com/garrytan/gbrain.git
synced 2026-08-14 17:02:19 +00:00
Compare commits
60
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
771acdf752 | ||
|
|
2d30c3e29b | ||
|
|
a30add4e53 | ||
|
|
d351a5fa10 | ||
|
|
f43e626c71 | ||
|
|
d2fb2023b8 | ||
|
|
375f14a06f | ||
|
|
60f7710c31 | ||
|
|
d542f8ad8d | ||
|
|
95d3d55fbd | ||
|
|
234c2a417f | ||
|
|
0708e6baa9 | ||
|
|
0f1089e836 | ||
|
|
329d99ccbe | ||
|
|
bf063757ab | ||
|
|
1134c27b0f | ||
|
|
ce781d3b95 | ||
|
|
2f6b817a28 | ||
|
|
99e15f0a10 | ||
|
|
6fb6790b2a | ||
|
|
5183e9f039 | ||
|
|
f55af9da7b | ||
|
|
56dd1438e5 | ||
|
|
d72047caff | ||
|
|
25b798d612 | ||
|
|
7816981ac0 | ||
|
|
2437a60784 | ||
|
|
004434e886 | ||
|
|
7c9e9fd331 | ||
|
|
aeb14df35c | ||
|
|
3cf666e288 | ||
|
|
eacc98b6a3 | ||
|
|
c1d7a61a6d | ||
|
|
989794701e | ||
|
|
f94b9ef316 | ||
|
|
64e70c53a0 | ||
|
|
ecc2c378d1 | ||
|
|
4627c7a047 | ||
|
|
83817561e6 | ||
|
|
59ba03afcb | ||
|
|
ce350250e2 | ||
|
|
12cb232470 | ||
|
|
bdab11e524 | ||
|
|
9f28e66bfe | ||
|
|
5374fc8519 | ||
|
|
ffb4af81a2 | ||
|
|
15d8107b28 | ||
|
|
6ab29f3e31 | ||
|
|
3fb54d0d9e | ||
|
|
9e92c4b0c5 | ||
|
|
503e58a9f5 | ||
|
|
0004f280c3 | ||
|
|
3f64a6d467 | ||
|
|
1129fa062c | ||
|
|
75e30015a7 | ||
|
|
b66bd8d155 | ||
|
|
59a6ce091a | ||
|
|
1b66e911f9 | ||
|
|
619d502a04 | ||
|
|
75e708b745 |
@@ -1,4 +1,4 @@
|
||||
<!-- gbrain-runbook-stamp: 0.45.3.0 -->
|
||||
<!-- gbrain-runbook-stamp: 0.45.5.0 -->
|
||||
<!-- This stamp must equal the VERSION file at every release; CI enforces it
|
||||
(scripts/check-bootstrap-tag.sh). `gbrain bootstrap status` compares it to
|
||||
the installed binary and warns on skew. -->
|
||||
|
||||
+106
@@ -2,6 +2,112 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.45.5.0] - 2026-08-12
|
||||
|
||||
Brain currency, part one: a brain is only useful if it's CURRENT, and until now
|
||||
the machinery keeping it current could die without anyone noticing. This release
|
||||
makes autopilot's health honest end-to-end — status that reads the heartbeat,
|
||||
a daemon that takes itself out of rotation when its repo vanishes, migrations
|
||||
that pause it instead of racing it, and staleness reporting that can no longer
|
||||
say "fresh" forever.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **A dead autopilot can no longer report healthy.** `gbrain autopilot --status`
|
||||
now reads the daemon's heartbeat instead of checking that install artifacts
|
||||
exist, and gains real exit codes for cron and CI gates: 0 fresh (or nothing
|
||||
installed), 1 needs attention (stale heartbeat, never ran, or paused), 2 the
|
||||
daemon took itself out of rotation. Status runs without touching the
|
||||
database, so it keeps working during the exact outages it exists to
|
||||
diagnose. Staleness tolerance scales with the tick interval and accounts
|
||||
for the adaptive scheduler's longer healthy-brain sleeps, and a garbage
|
||||
interval value can no longer silence the alarm.
|
||||
- **Content-relative staleness now has a wall-clock ceiling.** A source whose
|
||||
content stopped moving (or whose local clone vanished) previously reported
|
||||
fresh forever off the stored content timestamp. `sync_freshness`,
|
||||
`federation_health`, and `gbrain status` now ramp toward stale past a
|
||||
ceiling (default 72h; `GBRAIN_STALENESS_CEILING_HOURS` to tune) — ramping,
|
||||
not stepping, so the warn tier still fires before the fail tier instead of
|
||||
both alarms tripping at once.
|
||||
- **The documented agent-scheduler chain works on keyless brains.**
|
||||
`gbrain sync --repo <path> && gbrain embed --stale` used to exit 1 on every
|
||||
brain installed without an embedding key, breaking the always-current cron
|
||||
for external agent schedulers. A bare stale embed now refuses cleanly
|
||||
(exit 0, stderr hint); explicit embed requests (a slug, a slugs list, the
|
||||
all flag) still exit 1.
|
||||
- **Engine migrations and the autopilot daemon no longer race.**
|
||||
`gbrain migrate --to <engine>` claims a cooperative pause marker before
|
||||
touching the target — the marker doubles as a migration mutex, so a second
|
||||
concurrent migrate refuses to run instead of corrupting the first one's
|
||||
resume state, and a marker it cannot write refuses the migration outright
|
||||
rather than running unfenced. Background job workers stop picking up new
|
||||
work while the marker is parked. It then waits for in-flight
|
||||
sync/embed/cycle work and running jobs to actually drain (watching the DB
|
||||
lock table, capped by `GBRAIN_MIGRATE_QUIESCE_SECONDS`) instead of
|
||||
sleeping a blind grace period.
|
||||
The marker is released even when the migration fails or is killed: cleanup
|
||||
registers the moment the claim lands, adoption of a dead run's orphan is
|
||||
pid-liveness-checked (a live migrate's marker is never stolen), and the
|
||||
daemon itself clears an orphan whose owning process died. After a clean
|
||||
flip the daemon detects the engine change on its next tick and relaunches
|
||||
onto the new engine — previously it kept syncing into the abandoned source
|
||||
engine until its process happened to restart — and the migration warns if
|
||||
an exported connection-string env var would override the new config.
|
||||
- **Self-disable requires three consecutive misses.** A repo on an external
|
||||
or cloud-synced volume that is briefly absent at login no longer
|
||||
permanently takes the daemon out of rotation; one successful probe resets
|
||||
the strike counter.
|
||||
- **A cron'd status monitor no longer reads as an install.** Machines whose
|
||||
only crontab reference is the recommended health-gate line stop reporting
|
||||
"installed but never ran".
|
||||
- **Malformed connection URLs stop the daemon immediately** with a clear
|
||||
config verdict instead of spending the whole reconnect budget retrying a
|
||||
value only the operator can fix.
|
||||
- **Sync no longer silently drops git typechange and unmerged statuses.**
|
||||
Replacing an indexed file's content in a way git reports as `T` or `U`
|
||||
now imports as a modification instead of never reaching the index; a
|
||||
copy status imports its destination path.
|
||||
- **A wedged sync can no longer read as "in progress" forever.** A sync
|
||||
lock holder that keeps heartbeating past the staleness ceiling without
|
||||
finishing now fails `gbrain doctor`'s freshness check, naming the holder
|
||||
and the exact `gbrain sync --break-lock --source <id>` remedy.
|
||||
|
||||
### Added
|
||||
|
||||
- **Autopilot self-disable guard.** The generated wrapper now stops the daemon
|
||||
for real when its `--repo` path vanishes: it writes an explanatory marker,
|
||||
then boots the job out of the supervisor (`launchctl bootout` on macOS,
|
||||
`systemctl --user disable --now` on systemd) — a bare `exit 0` under
|
||||
KeepAlive/Restart=always is just a quieter respawn loop. `--status` explains
|
||||
why it stopped; a reinstall against a restored path clears the marker;
|
||||
`--uninstall` clears it too.
|
||||
- **`paused` status state.** A daemon parked by a migration (or by an orphaned
|
||||
pause marker) now reports `paused` with exit 1 and the marker path, instead
|
||||
of "running" off its still-fresh heartbeat.
|
||||
- **Harness e2e tier.** A real-launchd lifecycle test on macOS (install →
|
||||
load → self-disable → status, under a per-run unique label) plus a
|
||||
shimmed-supervisor lifecycle that runs on every platform, and an
|
||||
agent-scheduler contract test that drives the documented sync-and-embed
|
||||
shell chain end-to-end against a keyless brain — including the
|
||||
pull-failure case that must break the chain.
|
||||
- **Honest staleness numbers in `gbrain status`.** Source rows now carry
|
||||
`hours_since_last_sync` (raw wall-clock truth) alongside the
|
||||
threshold-relative `staleness_hours` that drives the fresh/stale/severe
|
||||
class, so the escalation ordering and the human-facing number stop being
|
||||
the same field.
|
||||
- **Shared numeric env resolver.** The doctor and staleness-threshold
|
||||
`GBRAIN_*` numeric env vars now resolve through one warn-once helper
|
||||
(`src/core/env-number.ts`), so a typo'd value falls back loudly exactly
|
||||
once instead of NaN-ing a threshold silently.
|
||||
|
||||
### To take advantage of v0.45.5.0
|
||||
|
||||
- `gbrain upgrade`, then wire your scheduler's health gate to
|
||||
`gbrain autopilot --status` — the exit code is now trustworthy.
|
||||
- If autopilot is installed, reinstall once (`gbrain autopilot --install
|
||||
--repo <path>`) so the generated wrapper picks up the self-disable guard.
|
||||
- Keyless installs: your sync-and-embed cron chain now exits 0; no action
|
||||
needed beyond upgrading.
|
||||
## [0.45.3.0] - 2026-08-12
|
||||
|
||||
**Codex installs stop asking a question Codex can't honor.** The bootstrap used to offer every install a choice of MCP scope — this folder only, or the whole machine — but Codex has no per-folder registrations, so picking "this folder" led to a confusing round-trip where the agent asked permission to keep what it had already done. Now each harness gets the honest version: Claude Code records your scope choice during the interview (where it actually sticks), and Codex simply tells you the truth — its registration reaches the whole machine, read and write — along with the exact commands to remove it (just the registration, or the whole install).
|
||||
|
||||
@@ -251,7 +251,13 @@ Set up using your platform's scheduler (OpenClaw cron, Railway cron, crontab), o
|
||||
platform glue entirely with `gbrain autopilot --install` (built-in self-maintaining daemon):
|
||||
|
||||
- **Live sync** (every 15 min): `gbrain sync --repo ~/brain && gbrain embed --stale`
|
||||
— or `gbrain sync --watch` for a continuous loop.
|
||||
— or `gbrain sync --watch` for a continuous loop. Safe on keyless brains:
|
||||
a bare `gbrain embed --stale` exits 0 with a stderr note when embeddings
|
||||
are disabled, so the chain doesn't break.
|
||||
- **Health gate** (daily): `gbrain autopilot --status` — exit 0 fresh (or
|
||||
nothing installed), 1 needs attention (stale heartbeat, never ran, or
|
||||
paused), 2 the daemon took itself out of rotation. Filesystem-only, so it
|
||||
works during DB outages.
|
||||
- **Auto-update** (daily): `gbrain check-update --json` (tell user, never auto-install).
|
||||
- **Dream cycle** (nightly): `gbrain dream` runs the 8-phase overnight maintenance cycle.
|
||||
Entity sweep, citation fixes, memory consolidation, plus (v0.23+) overnight conversation
|
||||
|
||||
@@ -1,5 +1,65 @@
|
||||
# TODOS
|
||||
|
||||
## Brain-currency harness-e2e follow-ups (filed with the PR-A wave)
|
||||
|
||||
- [ ] **P1 — Extend engine-identity convergence to the other long-lived planes.**
|
||||
The autopilot daemon now detects a post-migration engine flip
|
||||
(`autopilotEngineIdentity` per-tick compare → clean exit for supervisor
|
||||
relaunch), but `gbrain serve` (MCP) and a standalone `gbrain jobs work`
|
||||
worker hold their engine handle indefinitely and keep writing into the
|
||||
abandoned source engine after a flip — the same silent-divergence class,
|
||||
still open on those planes (adversarial-review catch). Fix shape: the same
|
||||
boot-identity compare in their main loops.
|
||||
- [ ] **P2 — DB-visible pause for cross-host workers.** The pause marker now
|
||||
fences local job pickup (pre-claim check + post-claim release-back in
|
||||
`src/core/minions/worker.ts`), but the marker is a local file: a worker on
|
||||
ANOTHER host or container pointed at the same Postgres brain never sees it
|
||||
and keeps claiming jobs during a migration copy (its in-flight work IS
|
||||
visible to the drain via `minion_jobs`/lock rows; new claims are the gap).
|
||||
Fix shape: a row in a control table (or a pause flag in `gbrain_cycle_locks`)
|
||||
that the claim query itself honors — atomic with claiming, visible
|
||||
cluster-wide.
|
||||
- [ ] **P2 — Route file→symlink typechanges to delete.** `buildSyncManifest`
|
||||
maps git status `T` to modified, but import-file deliberately SKIPS symlinks
|
||||
(the exfil guard), so replacing an indexed file with a symlink leaves the
|
||||
old content indexed forever with no delete. Fix shape: when the post-change
|
||||
path is a symlink, emit a delete instead of a modify.
|
||||
- [ ] **P3 — Surface daemon-internal degradation in status.** A daemon stuck
|
||||
in the reconnect-retry loop (crash-classified errors) keeps heartbeating,
|
||||
so `--status` reads fresh while zero work happens. Fix shape: a breadcrumb
|
||||
file with consecutive-failure count that showStatus reads.
|
||||
|
||||
- [ ] **P3 — Extract a shared `seedBrain` test helper.** The keyless-PGLite +
|
||||
tmp-HOME + shimmed-PATH setup is duplicated between
|
||||
`test/autopilot-launchd-lifecycle.serial.test.ts` and
|
||||
`test/agent-scheduler-contract.serial.test.ts` (review-army maintainability
|
||||
finding). A third harness-e2e file (the PR-B tier) should force the
|
||||
extraction into `test/helpers/`; don't extract before then — two instances
|
||||
is a coincidence, three is a pattern.
|
||||
- [ ] **P3 — Name the quiesce protocol's magic numbers.** `migrate-engine.ts`
|
||||
and `autopilot.ts` share three constants by value, not by name: the 600s
|
||||
heartbeat-freshness window, the 35s default grace, and the daemon's paused
|
||||
fast-poll interval. Hoist into `src/core/autopilot-paths.ts` (the shared
|
||||
leaf) as named exports so the two planes can't drift.
|
||||
- [ ] **P3 — Migration manifest rows don't carry content_hash.** A resume
|
||||
trusts `(source_id, slug)` membership in `completed_slugs`; a page edited
|
||||
BETWEEN the failed run and the resume is skipped with its stale copy left on
|
||||
the target (review-army data-migration finding; pre-existing design, not a
|
||||
regression). Fix shape: stamp `content_hash` per completed entry and re-copy
|
||||
on mismatch during resume.
|
||||
|
||||
- [ ] **P2 — Keyless `gbrain dream` contract test.** The documented nightly cron
|
||||
(INSTALL_FOR_AGENTS.md Step 7) runs `gbrain dream` unconditionally, and the cycle's
|
||||
embed phase hits the same `EmbeddingDisabledError` class that broke the documented
|
||||
sync-and-embed chain on keyless brains (fixed in `runEmbed` for the `--stale`
|
||||
spelling; `test/agent-scheduler-contract.serial.test.ts` pins it). Nobody has verified that a
|
||||
full keyless dream exits 0 — if any phase surfaces the disabled-embeddings error as a
|
||||
phase failure, the documented nightly cron is broken identically for every
|
||||
`init --no-embedding` install. **Where to start:** `src/core/cycle.ts` embed phase +
|
||||
`src/commands/dream.ts` exit-code handling; test shape mirrors
|
||||
`test/agent-scheduler-contract.serial.test.ts` (keyless PGLite brain, real CLI spawn, exit-code
|
||||
assertion). Surfaced by the harness-e2e outside-voice review.
|
||||
|
||||
## BrainBench follow-ups (filed v0.44.0.0, Cathedral 2)
|
||||
|
||||
Deferred from the BrainBench wave (eng-reviewed; plan + GSTACK REVIEW REPORT at
|
||||
|
||||
+6
-1
@@ -18,4 +18,9 @@ timeout = 60_000
|
||||
# runs, so audit-emitting code paths (content-sanity, shell-audit, etc.)
|
||||
# can't leak fixture events into the operator's real ~/.gbrain/audit/. See
|
||||
# test/helpers/audit-dir-preload.ts for the full rationale.
|
||||
preload = ["./test/helpers/legacy-embedding-preload.ts", "./test/helpers/audit-dir-preload.ts"]
|
||||
#
|
||||
# Same treatment for the sync failure ledger: broken-fixture import/sync tests
|
||||
# were appending rows into the operator's real ~/.gbrain/sync-failures.jsonl,
|
||||
# which `gbrain doctor` reads and warns on. See
|
||||
# test/helpers/sync-failures-preload.ts.
|
||||
preload = ["./test/helpers/legacy-embedding-preload.ts", "./test/helpers/audit-dir-preload.ts", "./test/helpers/sync-failures-preload.ts"]
|
||||
|
||||
@@ -177,6 +177,25 @@ live in `test/postgres-engine-rls-scope.test.ts`.
|
||||
|
||||
**Migration:** `gbrain migrate --to supabase` exports everything (pages, chunks, embeddings, links, tags, timeline) and imports into Supabase. `gbrain migrate --to pglite` goes the other direction. Bidirectional, lossless.
|
||||
|
||||
The migration and the autopilot daemon do not race: `migrate --to` claims a
|
||||
cooperative pause marker before touching the target. The marker doubles as a
|
||||
migration mutex — a second concurrent migrate refuses to run, and a marker
|
||||
that cannot be written refuses the migration outright. Background job workers
|
||||
stop picking up new work while it is parked, and the migration waits for
|
||||
in-flight sync/embed/cycle work and running jobs to actually drain (watching
|
||||
the DB lock table, capped by `GBRAIN_MIGRATE_QUIESCE_SECONDS` — default 300;
|
||||
`0` skips the wait). Cleanup registers the moment the claim lands, so the
|
||||
marker is released on failure and on catchable signals; a marker orphaned by
|
||||
an uncleanly killed run is adopted by a later migrate only after a
|
||||
pid-liveness check (a live migrate's marker is never stolen), and the daemon
|
||||
clears an orphan whose owning process died on its next poll. `gbrain
|
||||
autopilot --status` reports `paused` (exit 1) while the marker is parked and
|
||||
prints the marker path; on a host with no daemon running to self-heal,
|
||||
remove an orphan by hand only after confirming the pid it names is dead.
|
||||
After a clean flip the daemon detects the engine change on its next
|
||||
tick and relaunches onto the new engine, and the migration warns if an
|
||||
exported connection-string env var would override the new config.
|
||||
|
||||
### Troubleshooting: startup abort (`RuntimeError: Aborted()`)
|
||||
|
||||
**Symptom:** every PGLite-touching command dies at startup with
|
||||
|
||||
@@ -175,6 +175,8 @@ Unit tests and what they cover:
|
||||
- `test/volunteer-context.test.ts` — push-based context core (#2095), hermetic in-memory PGLite: `parseWindow` lenient `user:`/`assistant:` parsing, multi-turn window extraction, confidence-gated volunteering (arm confidences, multi-turn/newest-turn boosts, `min_confidence` gate, max-pages cap), slug-only suppression, privacy (rationales are deterministic templates; synopses pass the takes/facts fence), and the approximate usage-stats join.
|
||||
- `test/watch-command.test.ts` — `gbrain watch` push transport (#2095): streaming loop, rolling window, session dedupe, `--json` JSONL shape, `channel: 'watch'` event logging, clean EOF return. Hermetic PGLite + injected line/write deps (no subprocess, no real stdin).
|
||||
- `test/watch-sigint.serial.test.ts` — `gbrain watch` SIGINT lifecycle against a real spawned CLI subprocess with a tmpdir brain. SERIAL: parallel unit shards flake on concurrent subprocess spawns (same rationale as `apply-migrations-pglite-spawn.serial.test.ts`).
|
||||
- `test/autopilot-launchd-lifecycle.serial.test.ts` — autopilot lifecycle behavior, not generated-string assertions: the full install → self-disable → status → reinstall → uninstall arc with `launchctl` replaced by an argv recorder and the generated wrapper executed by a REAL bash against a genuinely deleted repo (every platform), plus a darwin-only fail-SKIP describe against the real launchd under a per-run unique label (`GBRAIN_AUTOPILOT_LABEL`) so it can never collide with — or tear down — a real install on the host. Serial: spawns subprocesses and pins HOME/GBRAIN_HOME for the whole file.
|
||||
- `test/agent-scheduler-contract.serial.test.ts` — the documented external agent-scheduler shell chain (`gbrain sync --repo X && gbrain embed --stale`, live-sync.md / INSTALL_FOR_AGENTS.md Step 7) driven end-to-end through a real `/bin/sh` against a keyless PGLite brain: the `&&` short-circuit IS the contract (argv arrays can't exercise it), the keyless bare stale embed exits 0, and the pull-failure case that must break the chain does. Anti-vacuity: the fixture commits a real page and every read-back asserts pages >= 1. Serial: real spawned CLI + tmpdir HOME.
|
||||
- `test/cli-format-volunteer.test.ts` — `formatResult`'s `volunteer_context` human rendering: pointer lines with confidence/arm/rationale, the empty-result message, the approximate stats summary.
|
||||
- `test/config.test.ts` — config redaction.
|
||||
- `test/files.test.ts` — MIME/hash.
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,406 @@
|
||||
# Brain currency — fix the incident, then build the ladder
|
||||
Generated by /plan-ceo-review on 2026-08-10
|
||||
Rev 3, after two adversarial spec-review rounds (6/10 → 7/10) and an independent outside voice.
|
||||
Branch: garrytan/gbrain-commit-indexing | Mode: SELECTIVE EXPANSION
|
||||
Repo: garrytan/gbrain
|
||||
|
||||
**Citation convention:** repo-relative paths. `src/core/sync.ts` (540 lines) and
|
||||
`src/commands/sync.ts` (5804 lines) are different files; both are cited.
|
||||
|
||||
## Origin
|
||||
|
||||
An investigation into "how does gbrain pick up new commits from GitHub" found it never
|
||||
talks to GitHub. It diffs `git diff last_commit..HEAD` against a **local checkout**
|
||||
(`src/core/sync-delta.ts:113`). Getting remote commits into that checkout is a separate,
|
||||
opt-in concern.
|
||||
|
||||
It then found worse: on the founder's machine `gbrain autopilot` was installed, died
|
||||
2026-05-31, and stayed dead **71 days** while three surfaces reported healthy.
|
||||
|
||||
**1. `autopilot --status` is an artifact-presence check.**
|
||||
`src/commands/autopilot.ts:1775-1786` — plist `existsSync` on darwin, crontab grep
|
||||
elsewhere. Never asks whether the job is loaded, the process alive, the baked `--repo`
|
||||
present, or the log fresh. Always exits 0.
|
||||
|
||||
**2. `doctor`'s `sync_freshness` computes the 71-day number and throws it away.**
|
||||
`src/core/source-health.ts:182-194`:
|
||||
|
||||
```ts
|
||||
const wallClockSeconds = Math.floor((nowMs - lastSyncMs) / 1000); // ← the 71 days
|
||||
if (wallClockSeconds < 0) return wallClockSeconds;
|
||||
if (contentMs !== null && Number.isFinite(contentMs)) {
|
||||
return contentMs <= lastSyncMs ? 0 : wallClockSeconds; // ← discarded
|
||||
}
|
||||
```
|
||||
|
||||
When the clone is unreachable, `src/commands/doctor.ts:4306-4344` routes the verdict here.
|
||||
The function measures *drain completeness*, not *staleness*. "We caught up when we last
|
||||
looked" and "we have not looked in 71 days" both return 0.
|
||||
|
||||
**3. `gbrain status` inherits it.** `src/commands/sync.ts:5440-5453` → `'fresh'` beside a
|
||||
71-day-old date, exit 0. (`gbrain sources status` does report the real lag in its LAG
|
||||
column, but has no warn line for it and no exit contract.)
|
||||
|
||||
**Root cause of the death:** `src/commands/migrate-engine.ts` (22,733 bytes) contains
|
||||
**zero** autopilot references. The Supabase-to-PGLite migration rewrote
|
||||
`~/.gbrain/config.json` while a daemon built on the old config kept running and died on
|
||||
`config.database_url`.
|
||||
|
||||
## The key insight the reviews converged on
|
||||
|
||||
The content comparison in #2 is not a bug someone forgot. `src/commands/doctor.ts:4288-4305`
|
||||
documents why it exists:
|
||||
|
||||
> a container restart wipes `local_path` ... **and since a no-op sync doesn't advance
|
||||
> `last_sync_at`**, every QUIET source read as stale/FAIL after a restart (score-sinking
|
||||
> alert storm; observed live: 16-source brain, 12 clones gone after a config-update
|
||||
> restart, doctor 70→30).
|
||||
|
||||
**The premise in bold was invalidated after that code was written.** v0.42.52.0 added a
|
||||
heartbeat at `src/commands/sync.ts:2287-2298`:
|
||||
|
||||
```ts
|
||||
// bump last_sync_at as a heartbeat on every successful 0-changes sync...
|
||||
if (opts.sourceId) {
|
||||
await engine.executeRaw(`UPDATE sources SET last_sync_at = now() WHERE id = $1`, [opts.sourceId]);
|
||||
}
|
||||
```
|
||||
|
||||
A no-op sync **does** advance `last_sync_at` now. So a quiet source that is being checked
|
||||
has a recent `last_sync_at` and survives a wall-clock ceiling; the 71-day case has an old
|
||||
one because **no sync ran at all**. The two cases are now distinguishable, and the
|
||||
fallback's justification has expired.
|
||||
|
||||
That is the whole incident: a wall-clock ceiling on the discard branch, in one pure
|
||||
function that `doctor`, `gbrain status`, and `sources status` all call. It fixes all three
|
||||
by construction, with no new table, no new command, and no migration.
|
||||
|
||||
It also means **the heartbeat this plan originally proposed to build already ships.** A
|
||||
separate `live_ticks` table would be a fourth status surface on a fifth data source,
|
||||
curing "three surfaces disagreed" by adding one more that can disagree.
|
||||
|
||||
## Base branch
|
||||
|
||||
The whole wave (PR-A, PR-B, PR-C) is based on
|
||||
`garrytan/codex-as-agent-default-install`, not `master`. That branch carries the
|
||||
bootstrap surface (`src/core/bootstrap/{host-specs,hooks}.ts`, `detectHarness()`) that
|
||||
PR-B's harness tier needs, so **PR-B is not blocked** — an earlier revision of this doc
|
||||
assumed it was.
|
||||
|
||||
That branch moves frequently; re-fetch before comparing anything against it. A stale
|
||||
remote-tracking ref is an easy way to reach a confidently wrong conclusion here.
|
||||
|
||||
## Sequencing (decided)
|
||||
|
||||
Three PRs. Nothing is cut; the order changed.
|
||||
|
||||
### PR-A — close the incident (ships first)
|
||||
|
||||
1. **Wall-clock ceiling** in `lagFromContentMs` (`src/core/source-health.ts:189`): return
|
||||
`wallClockSeconds` once it exceeds an absolute bound regardless of the content
|
||||
comparison. Bound is a named env knob per repo convention
|
||||
(`GBRAIN_STALENESS_CEILING_HOURS`, default 72, matching the existing
|
||||
`GBRAIN_SYNC_FRESHNESS_FAIL_HOURS`).
|
||||
2. **Regression test** (acceptance criterion 1 below).
|
||||
3. **E3** — `src/commands/migrate-engine.ts` reconciles the running daemon.
|
||||
4. **Wrapper self-disable** — `src/commands/autopilot.ts:1314-1359`. Adapted from
|
||||
`src/core/brain-repo-durability.ts:509-512`, NOT copied: two corrections the
|
||||
engineering review established.
|
||||
- Predicate is `[ ! -d "$repo" ]`, not `[ ! -d "$repo/.git" ]`. `--repo` may be a
|
||||
subdirectory of the checkout (sync resolves the root itself by walking up), and
|
||||
`.git` is a FILE in worktrees and submodules — either shape would self-disable a
|
||||
healthy install.
|
||||
- `exit 0` is correct for the durability wrapper because launchd fires it on
|
||||
`StartInterval` (one shot). Autopilot runs under `KeepAlive=true` +
|
||||
`ThrottleInterval=60` and systemd `Restart=always`, where exiting disables nothing
|
||||
and instead produces a silent respawn-every-60s loop. The wrapper must
|
||||
`launchctl bootout` / `systemctl --user disable --now` itself on those targets and
|
||||
drop a marker that `--status` surfaces.
|
||||
5. **Reconnect classifier** — `src/commands/autopilot.ts:58-78`; a JS `TypeError` must not
|
||||
substring-match as a config verdict.
|
||||
6. **`autopilot --status` reads the heartbeat** instead of `existsSync`, and exits nonzero
|
||||
when stale.
|
||||
7. **E8 hygiene** — test-run pollution of `~/.gbrain/sync-failures.jsonl`;
|
||||
`buildSyncManifest` (`src/core/sync.ts:105-140`) dropping git **`T`** (typechange).
|
||||
Narrowed: `src/core/sync-delta.ts:130` passes `-M` only, so `C` is unreachable without
|
||||
`--find-copies` and `U` needs a conflicted worktree. `C`/`U` handled defensively.
|
||||
|
||||
**Not in PR-A:** the lockfile-leak fix. Removing the leaked `~/.gbrain/autopilot.lock`
|
||||
deletes the signal that distinguishes *crashed* from *never installed*
|
||||
(`src/commands/status.ts:595-598`) before its replacement exists. It lands in PR-C
|
||||
alongside `live status`.
|
||||
|
||||
### PR-B — the `harness` tier alone
|
||||
|
||||
The harness tier is the only tier the modal gbrain user can actually run (PGLite default,
|
||||
desktop harness, behind NAT), so it ships alone and early rather than buried inside the
|
||||
cathedral. Reuses `src/core/bootstrap/{host-specs,hooks}.ts`, which the base branch
|
||||
already provides.
|
||||
|
||||
### PR-C — the ladder
|
||||
|
||||
`live` command family, `live.mode` bundle, shape detection, `cron`/`daemon`/`webhook`
|
||||
tiers, advisor collector, `init` offer, watch tier, shared `os-scheduler.ts`, and the
|
||||
lockfile-leak fix. **`live_ticks` is re-examined here against the shipped
|
||||
`last_sync_at` heartbeat rather than assumed** — the burden is on the new table to justify
|
||||
itself.
|
||||
|
||||
## The constraint being satisfied (quoted so it can be checked)
|
||||
|
||||
`docs/designs/AGENT_BOOTSTRAP_PLAN.md` on `origin/garrytan/codex-as-agent-default-install`,
|
||||
decision **D9**:
|
||||
|
||||
> **D9 Scheduling: almost nothing on by default.** ON: SessionEnd push (event-driven, no
|
||||
> daemon). OPT-IN: 15-min harden cron. **Autopilot NOT default on PGLite** (verified: its
|
||||
> sync/embed children would contend with every live serve for the single-writer lock, and
|
||||
> nothing handles `LiveServeLockError` politely today) — **recommended on Postgres**; any
|
||||
> future scheduled job must treat lock-held as skip-silently-and-log.
|
||||
|
||||
*Reconciliation:* D9 says "15-min harden cron"; the shipped default is **1800s / 30 min**
|
||||
(`src/core/brain-repo-durability.ts:76`, `:659`). D9's figure is stale. This plan uses 30.
|
||||
|
||||
This plan's decisions are labelled **L1..L14** to avoid collision with that document.
|
||||
|
||||
## PR-C design (carried forward, not yet committed to a diff)
|
||||
|
||||
### Tiers — five active plus `off`
|
||||
|
||||
| tier | mechanism | expected cadence | engine gate |
|
||||
|---|---|---|---|
|
||||
| `off` | nothing | n/a — `live status` exits **0** | — |
|
||||
| `harness` | agent hook / session boundary | event-driven, **age-exempt** | any (incl. Windows, containers) |
|
||||
| `webhook` | HMAC push from GitHub | event-driven, **age-exempt**; paired keepalive `cron` supplies the age signal | any + reachable `serve --http` |
|
||||
| `cron` | OS scheduler | declared `expected_cadence_seconds` | any; **PGLite floor 1800s + lock-aware skip** |
|
||||
| `daemon` | resident autopilot, `runCycle` | 300s | **Postgres only** (D9) |
|
||||
| `watch` | daemon + chokidar | **floor 300s for freshness purposes**, not the ~1s event latency | **Postgres only** (D9) |
|
||||
|
||||
Event-driven tiers are exempt from age-based failure; a webhook repo with no pushes for
|
||||
three days is healthy, not failed. `watch`'s freshness cadence is decoupled from its event
|
||||
latency so a GC pause is not a FAIL.
|
||||
|
||||
`off` is a first-class bundle member with `enabled: false`, copied from
|
||||
`src/core/pace-mode.ts:65-71`.
|
||||
|
||||
### L1 — Shape detection predicate
|
||||
|
||||
| Signal | Source | Meaning |
|
||||
|---|---|---|
|
||||
| engine | `config.engine` | `postgres` required for `daemon`/`watch` |
|
||||
| interactive desktop harness | `CLAUDECODE`, `CLAUDE_CODE_ENTRYPOINT`, `CODEX_HOME`, `CODEX_SANDBOX`, `CODEX_CI` (**env only**) | any present → cap at `harness` |
|
||||
| long-lived host | `detectInstallTarget()` ∈ {`macos`, `linux-systemd`, `ephemeral-container`+injection point} | a reboot-surviving scheduler exists |
|
||||
| server posture | `serve --http` configured, or `minion_mode != 'off'` | corroborating, never sufficient alone |
|
||||
|
||||
`macos` is in the long-lived row deliberately: `detectInstallTarget()` returns `'macos'`
|
||||
unconditionally on darwin (`src/commands/autopilot.ts:1277`), and darwin is the platform
|
||||
of the origin incident. Omitting it would make the incident host permanently
|
||||
shape-ineligible.
|
||||
|
||||
**No filesystem probes for harness identity.** The `~/.claude/hooks/...` class of probe
|
||||
(`src/commands/autopilot.ts:1304`) is what false-positives today. Env vars only.
|
||||
|
||||
Any inconclusive read falls to `harness`, never `daemon`.
|
||||
|
||||
### L2 — `live status` exit codes
|
||||
|
||||
| Condition | Exit |
|
||||
|---|---|
|
||||
| fresh, or `live.mode == off` | 0 |
|
||||
| PGLite lock held by a live `serve` (`blocked_by_serve`) | 0 |
|
||||
| tier enabled + heartbeat missing or stale | 1 |
|
||||
| drifted install, or DB **connect failure** | 2 |
|
||||
|
||||
`live.mode == off` exiting 0 is load-bearing: otherwise every fresh install exits nonzero,
|
||||
which is the `cycle_freshness` #2540 lesson (never-configured must not turn the surface
|
||||
red). And lock-held is **not** an outage: `src/core/pglite-engine.ts:444` acquires the file
|
||||
lock on every `connect()` and throws if it fails, so on the default engine with a resident
|
||||
`serve`, treating that as exit 2 would make FAIL the steady state.
|
||||
|
||||
### L3 — `skipped_locked` semantics
|
||||
|
||||
A tick that cannot acquire the PGLite lock **does not satisfy freshness and does not
|
||||
degrade it**. It is neutral: logged, not recorded as work-done, and not counted toward
|
||||
staleness for a grace window of 3 consecutive skips, after which the surface reports
|
||||
`blocked_by_serve` with the remediation inline. Treating it as work-done rebuilds the
|
||||
71-day false-green; treating it as failure makes the default engine permanently red.
|
||||
|
||||
### L4 — Scheduler ownership
|
||||
|
||||
Ownership lives in a sidecar `~/.gbrain/live-ownership.json`, **not** in an entry comment.
|
||||
On darwin both harden and autopilot install launchd **plists** (files, not comment-bearing
|
||||
crontab lines), so the `# gbrain:autopilot v0.11.0` marker convention does not generalize.
|
||||
The sidecar covers all install targets uniformly.
|
||||
|
||||
Three enumerated cases:
|
||||
|
||||
1. **Harden cron exists + pull opted in** → rewrite through `os-scheduler.ts`,
|
||||
`ownership=live-adopted`.
|
||||
2. **Harden cron exists + pull declined** → leave it entirely alone; install a separately
|
||||
labelled `live` entry. **This is the default and lands first**, so PR-C's `live on`
|
||||
never meets an existing harden cron without a rule.
|
||||
3. **Neither exists** → install a `live` entry, `ownership=live`.
|
||||
|
||||
`live off` removes only entries `live` created and reverts adopted ones to harden.
|
||||
|
||||
Pre-existing `gbrain autopilot` installs are **migrated, not orphaned**: first `live
|
||||
status` after upgrade reports `tier: daemon (legacy autopilot)` and offers one-time
|
||||
adoption.
|
||||
|
||||
### L5 — Op scopes
|
||||
|
||||
| Op | scope | localOnly | remote |
|
||||
|---|---|---|---|
|
||||
| `live_status` | `read` | no | allowed; omits `local_path`, scheduler artifact paths, and log tail |
|
||||
| `live_tick` | `write` | **yes** | reject |
|
||||
| `live_on` / `live_off` | `admin` | **yes** | reject |
|
||||
| `live_self_heal` | `admin` | **yes** | reject |
|
||||
|
||||
Self-heal walks a **DB-supplied** `local_path` and then writes a scheduler entry.
|
||||
`src/commands/doctor.ts` already gates its git short-circuit on `localOnly === true`
|
||||
(*"a remote-callable code path must NOT walk DB-supplied `local_path` values with
|
||||
subprocess calls"*). Self-heal honors that and additionally requires a realpath match
|
||||
against the anchor via `isAnchorOwnedSyncPath` (`src/commands/sync.ts:1296`).
|
||||
|
||||
**Bootstrap paradox, acknowledged:** if the broken thing is the scheduler entry, a
|
||||
scheduled self-heal never runs. Non-scheduled triggers are the `harness` tier (PR-B) and
|
||||
an explicit `gbrain live doctor`. PR-C ships self-heal with both, not with a scheduled
|
||||
trigger alone.
|
||||
|
||||
### L6 — Revert
|
||||
|
||||
A code revert leaves plists, crontab lines, systemd units, and (E1) a GitHub webhook
|
||||
installed and unowned. Therefore:
|
||||
|
||||
- **Revert requires `gbrain live off` first** on any enabled host. Stated in the PR body.
|
||||
- The generated wrapper self-disables on a **marker file** written by `live on` and removed
|
||||
by `live off`. Not a `gbrain live --help` probe: that adds a process spawn per tick and
|
||||
assumes an exit code the CLI does not guarantee.
|
||||
- The migration, if `live_ticks` survives PR-C's re-examination, is additive and uses the
|
||||
**next free version at implementation time** (125 is the current max; two waves may land
|
||||
first).
|
||||
|
||||
### L7 — E5 must not use `nag-state.ts`
|
||||
|
||||
`src/core/skillpack/nag-state.ts` is skillpack-scoped (schema `gbrain-skillpack-nag-v1`,
|
||||
entries keyed on `pack_version`, `DEFAULT_NAG_CEILING = 3`, suppressed thereafter). Wiring
|
||||
a dead-sync alarm through it means a genuinely broken brain goes silent after three
|
||||
notices, which is a suppression mechanism for the exact failure mode whose defining
|
||||
property was 71 days of silence.
|
||||
|
||||
E5 instead uses a **rate limit, not a ceiling**: at most once per session, never
|
||||
suppressed permanently, escalating in terseness rather than disappearing.
|
||||
|
||||
### L8 — E1 webhook dependencies (previously unpriced)
|
||||
|
||||
Creating a GitHub webhook programmatically needs an `admin:repo_hook` token. No
|
||||
acquisition, storage, scope, or rotation story existed. Therefore E1 ships in **manual
|
||||
mode only**: `live on --tier webhook` generates the secret, resolves and prints the payload
|
||||
URL, and the user pastes it into GitHub, matching what `gbrain sources webhook set`
|
||||
(`src/commands/sources.ts:909-916`) already does. No token, no remote hook creation, no
|
||||
`live off` remote deletion problem.
|
||||
|
||||
The "verified test ping" must originate **from GitHub**, not locally. A local ping proves
|
||||
nothing through NAT and would be an artifact-presence check, the precise anti-pattern in
|
||||
the Origin section.
|
||||
|
||||
### L9 — `live_ticks` retention
|
||||
|
||||
If the table survives PR-C, the sweep runs **inside `live tick`** (bounded best-effort
|
||||
DELETE on a TTL), not only in the cycle's `purge` phase. `purge` is a `runCycle` phase
|
||||
(`src/core/cycle.ts:1434`), and `runCycle` runs only on `daemon`/`watch` — the `cron`,
|
||||
`webhook`, and `harness` tiers would accumulate forever.
|
||||
|
||||
## Scope decisions (all accepted; PR assignment added)
|
||||
|
||||
| # | Item | PR | Note |
|
||||
|---|---|---|---|
|
||||
| L10 | Approach C: full ladder | A/B/C | user chose the cathedral; resequenced, not cut |
|
||||
| L11 | Tier default keys on deployment shape, not vendor | C | Hermes has zero detectable signal |
|
||||
| E1 | Webhook tier, **manual mode** (L8) | C | |
|
||||
| E2 | Self-heal with `.tmp`+rename+`.bak` rollback | C | bootstrap paradox handled per L5 |
|
||||
| E3 | `migrate-engine` reconciles the daemon | **A** | the literal root cause |
|
||||
| E4 | Pull cron adoption per L4, separate opt-in per L12 | C | |
|
||||
| E5 | Agent-facing staleness, rate-limited not nag-ceilinged (L7) | C | |
|
||||
| E6 | Windows hard error naming `--tier harness` | C | `detectInstallTarget()` has no win32 branch |
|
||||
| E7 | `live_ticks` — **re-examined, not assumed** | C | the shipped `last_sync_at` heartbeat may suffice |
|
||||
| E8 | Hygiene, narrowed to git `T` | **A** | |
|
||||
|
||||
### L12 — E4's pull cron is an autonomy question
|
||||
|
||||
`docs/guides/upgrades-auto-update.md:41-43` states *"`auto` is deliberately NOT a default
|
||||
anywhere — it's an explicit autonomy grant, because applying code from GitHub unattended
|
||||
is, by design, remote code execution."* This plan does **not** flip `self_upgrade.mode`.
|
||||
|
||||
E4 schedules `git pull` every 30 minutes. That is content, not code, and durability keeps
|
||||
gbrain's hooks local and untracked so a pulled commit cannot rewrite executable hook code.
|
||||
But it is still unattended network fetch into a directory gbrain runs tooling against.
|
||||
Therefore the pull cron is a **separate opt-in from the tier**, proposed and explained by
|
||||
`live on`, never silently bundled.
|
||||
|
||||
### L13 — The directive's internal tension, stated
|
||||
|
||||
"OpenClaw and Hermes default to always-up-to-date" sits against L1's "shape detection
|
||||
recommends, never installs" and D9's "almost nothing on by default." These are reconciled
|
||||
by scope: shape detection sets the **recommended tier** and pre-selects it in the `init`
|
||||
consent prompt, so a shape-matching host is one keystroke from always-on rather than
|
||||
silently converted. Whether that consent is required on **upgrade** as well as fresh
|
||||
install is **open decision F1** below.
|
||||
|
||||
## L14 — Acceptance criteria
|
||||
|
||||
1. **Three-surface honesty.** A source whose `local_path` is deleted, whose `last_sync_at`
|
||||
is 71 days old, whose `newest_content_at` is **non-NULL**, and whose `chunker_version`
|
||||
**matches** must report stale/fail from `doctor` and `gbrain status`, and must surface
|
||||
the lag in `sources status`. Both fixture preconditions are required: a NULL
|
||||
`newest_content_at` already falls through to wall-clock
|
||||
(`src/commands/doctor.ts:4335-4342`) and a chunker mismatch already disables the
|
||||
fallback (`:4318`), so a naive fixture passes against unfixed code.
|
||||
*`sources status` is held to output, not exit code — it has no exit contract today and
|
||||
adding one is an undeclared breaking change to a read-only dashboard.*
|
||||
2. **Quiet-source non-regression.** A source with a recent `last_sync_at`, an unreachable
|
||||
clone, and no new content must still report **OK**. This is the 16-source / doctor
|
||||
70→30 incident; the ceiling must not re-light it.
|
||||
3. **Install honesty** (PR-C). `live on --tier cron` verifies the job loaded and exits
|
||||
nonzero if not; deleting the repo makes `live status` exit nonzero and name the path;
|
||||
`live off` leaves nothing.
|
||||
4. **Concurrency** (PR-C, **Postgres only**). Two tiers ticking produce one import and one
|
||||
neutral skip record. On PGLite the second process cannot open the DB at all, so the
|
||||
defined outcome is a log line and no row.
|
||||
5. **Watch tier** (PR-C). E2E expects **queued-job-failure**, not synchronous rejection —
|
||||
`ingest_capture` enqueues and returns.
|
||||
6. **Engine parity** (PR-C, if `live_ticks` survives). DDL identical in both engines,
|
||||
pinned by `test/e2e/engine-parity.test.ts`; bootstrap probe-set entry pinned by
|
||||
`test/schema-bootstrap-coverage.test.ts`.
|
||||
|
||||
## Open decisions (unanswered — do not silently default)
|
||||
|
||||
- **F1.** Does shape-detected always-on apply on **upgrade** as well as fresh install?
|
||||
Codebase precedent (`src/commands/upgrade.ts:513-516`, `mcp.publish_skills`) is
|
||||
new-installs-only with a one-time prompt for existing. Gates PR-C only.
|
||||
- **F2.** Command noun and config key: `gbrain live` + `live.mode` (requires renaming the
|
||||
existing `liveSyncStatus` helper at `src/core/db-lock.ts:749` to `syncInProgress`, two
|
||||
call sites) vs `gbrain sync live` + `sync.live.mode`. Gates PR-C only.
|
||||
|
||||
## Deferred to TODOS.md
|
||||
|
||||
- Full Windows `schtasks` tier — no test machine; `harness` covers it
|
||||
- Per-tier cost meter for `daemon` / `watch`
|
||||
- Cross-OS scheduler probing as a `live status` diagnostic (TODO-V19-D stays open; the
|
||||
heartbeat makes it optional rather than load-bearing)
|
||||
- Centralize the three freshness call sites onto one `freshnessVerdict()` helper
|
||||
(existing filed P3, now partially satisfied by PR-A's single-function fix)
|
||||
|
||||
## Dream state delta
|
||||
|
||||
PR-A leaves brain currency *honest*. PR-B leaves it *workable for the modal user*. PR-C
|
||||
leaves it *a product feature*. Remaining gap to the 12-month ideal: currency is still
|
||||
something the user turns on, not something simply true of a configured brain. F1 is the
|
||||
decision that closes or preserves that gap.
|
||||
|
||||
## Reviewer concerns (unresolved after 3 iterations)
|
||||
|
||||
- **Scope, from both reviewers:** PR-C remains large (command family, mode bundle, shape
|
||||
detector, three tiers, advisor collector, init prompt, webhook, watch tier, scheduler
|
||||
extraction). The PR-A/B/C split answers the sequencing objection but not the size of C
|
||||
itself. Revisit at PR-C planning with the incident already fixed.
|
||||
- **`live_ticks` necessity** is explicitly unresolved and assigned to PR-C rather than
|
||||
decided here.
|
||||
@@ -65,6 +65,12 @@ For scheduling `sync` + `embed --stale` specifically, the home doc is
|
||||
# Brain health — weekly Mondays at 6 AM
|
||||
0 6 * * 1 gbrain doctor --json >> /tmp/gbrain-health.log 2>&1 && gbrain embed --stale
|
||||
|
||||
# Autopilot health gate — daily at 7 AM. The exit code is the signal:
|
||||
# 0 fresh (or nothing installed), 1 needs attention (stale heartbeat,
|
||||
# never ran, or paused), 2 the daemon took itself out of rotation.
|
||||
# Status is filesystem-only, so it works even during a DB outage.
|
||||
0 7 * * * gbrain autopilot --status >> /tmp/gbrain-autopilot-health.log 2>&1 || your-notify "gbrain autopilot needs attention"
|
||||
|
||||
# Dream cycle — nightly at 2 AM
|
||||
0 2 * * * /path/to/dream-cycle.sh
|
||||
```
|
||||
|
||||
@@ -51,6 +51,10 @@ gbrain sync --repo /path/to/brain && gbrain embed --stale
|
||||
[spend controls](../operations/spend-controls.md).
|
||||
- `gbrain embed --stale` -- backfill embeddings for any chunks that don't have
|
||||
them. Safety net for large syncs (>100 files) or prior `--no-embed` runs.
|
||||
On a keyless brain (installed with `--no-embedding`), a bare stale embed
|
||||
refuses cleanly — exit 0 with a stderr note — so this chain is safe to
|
||||
schedule on keyless installs; keyword search keeps working. Explicit embed
|
||||
requests (a slug, `--slugs`, `--all`) still exit 1 on a keyless brain.
|
||||
- `gbrain sync --watch --repo <path>` -- foreground polling loop, every 60s
|
||||
(configurable with `--interval N`). Embeds inline for small changesets. Exits
|
||||
after 5 consecutive failures, so run under a process manager or pair with a
|
||||
@@ -151,7 +155,17 @@ vars — incident-time escape hatches, not everyday knobs.
|
||||
history rewrite still hard-blocks even with `--skip-failed`. Run
|
||||
`gbrain sync --skip-failed` to acknowledge a known-bad set yourself.
|
||||
|
||||
5. **Import checkpoints name the import target, not the caller's CWD.**
|
||||
5. **Staleness can't read "fresh" forever.** A source whose content stopped
|
||||
moving (or whose local clone vanished) used to report fresh indefinitely
|
||||
off the stored content timestamp. Content-relative staleness now ramps
|
||||
toward stale once wall-clock time since the last sync passes a ceiling
|
||||
(default 72h; `GBRAIN_STALENESS_CEILING_HOURS` to tune — it tracks
|
||||
`GBRAIN_SYNC_FRESHNESS_FAIL_HOURS` unless set). The ramp is gradual, so
|
||||
the warn tier still fires before the fail tier. `gbrain status` source
|
||||
rows carry `hours_since_last_sync` (raw wall-clock truth) alongside the
|
||||
threshold-relative `staleness_hours` that drives the fresh/stale class.
|
||||
|
||||
6. **Import checkpoints name the import target, not the caller's CWD.**
|
||||
Interrupted `gbrain import <dir>` runs may leave
|
||||
`~/.gbrain/import-checkpoint.json` so the next import can resume. The
|
||||
checkpoint `dir` is the absolute, resolved import target captured when
|
||||
@@ -178,6 +192,15 @@ vars — incident-time escape hatches, not everyday knobs.
|
||||
`gbrain embed --stale` isn't running after sync, leaving chunks invisible
|
||||
to vector search.
|
||||
|
||||
4. **Gate on the daemon's heartbeat.** If the built-in daemon runs your sync
|
||||
(`gbrain autopilot --install`), wire your scheduler's health check to
|
||||
`gbrain autopilot --status`. The exit code is the signal: 0 fresh (or
|
||||
nothing installed), 1 needs attention (stale heartbeat, never ran, or
|
||||
paused by a migration), 2 the daemon took itself out of rotation.
|
||||
`--json` emits the full report, including `heartbeat_age_seconds`. Status
|
||||
reads only the filesystem — no database connection — so it keeps working
|
||||
during the exact outages it exists to diagnose.
|
||||
|
||||
---
|
||||
|
||||
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*
|
||||
|
||||
@@ -121,6 +121,9 @@ cat ~/.gbrain/preferences.json
|
||||
cat ~/.gbrain/migrations/completed.jsonl
|
||||
|
||||
# 3. Autopilot is supervising a Minions worker child
|
||||
# (v0.46+: the exit code is the verdict — 0 fresh, 1 needs attention,
|
||||
# 2 self-disabled — so a nonzero exit here IS the finding, not a
|
||||
# broken verify step. Under `set -e`, append `|| true` to keep going.)
|
||||
gbrain autopilot --status
|
||||
ps aux | grep 'jobs work'
|
||||
|
||||
|
||||
@@ -61,7 +61,7 @@ The runtime `gbrain init --force` re-runs the init flow against the now-populate
|
||||
|
||||
## Pattern 3: No key, ever (keyless mode)
|
||||
|
||||
`--no-embedding` isn't only a deferral — it's also the install shape for **keyless mode**, a first-class supported end state (not a broken one). With zero provider keys, gbrain runs keyword-only (BM25) search and takes memory from agent-authored `## Facts` fences and write ops; embedding and extraction paths refuse cleanly instead of failing silently.
|
||||
`--no-embedding` isn't only a deferral — it's also the install shape for **keyless mode**, a first-class supported end state (not a broken one). With zero provider keys, gbrain runs keyword-only (BM25) search and takes memory from agent-authored `## Facts` fences and write ops; embedding and extraction paths refuse cleanly instead of failing silently. Concretely: the documented always-current chain (`gbrain sync --repo <path> && gbrain embed --stale`) is safe to schedule on a keyless brain — a bare stale embed exits 0 with a stderr note instead of breaking the chain, while explicit embed requests (a slug, `--slugs`, `--all`) still exit 1.
|
||||
|
||||
```dockerfile
|
||||
FROM oven/bun:1
|
||||
|
||||
+59
-2
@@ -1264,7 +1264,13 @@ Set up using your platform's scheduler (OpenClaw cron, Railway cron, crontab), o
|
||||
platform glue entirely with `gbrain autopilot --install` (built-in self-maintaining daemon):
|
||||
|
||||
- **Live sync** (every 15 min): `gbrain sync --repo ~/brain && gbrain embed --stale`
|
||||
— or `gbrain sync --watch` for a continuous loop.
|
||||
— or `gbrain sync --watch` for a continuous loop. Safe on keyless brains:
|
||||
a bare `gbrain embed --stale` exits 0 with a stderr note when embeddings
|
||||
are disabled, so the chain doesn't break.
|
||||
- **Health gate** (daily): `gbrain autopilot --status` — exit 0 fresh (or
|
||||
nothing installed), 1 needs attention (stale heartbeat, never ran, or
|
||||
paused), 2 the daemon took itself out of rotation. Filesystem-only, so it
|
||||
works during DB outages.
|
||||
- **Auto-update** (daily): `gbrain check-update --json` (tell user, never auto-install).
|
||||
- **Dream cycle** (nightly): `gbrain dream` runs the 8-phase overnight maintenance cycle.
|
||||
Entity sweep, citation fixes, memory consolidation, plus (v0.23+) overnight conversation
|
||||
@@ -2230,6 +2236,25 @@ live in `test/postgres-engine-rls-scope.test.ts`.
|
||||
|
||||
**Migration:** `gbrain migrate --to supabase` exports everything (pages, chunks, embeddings, links, tags, timeline) and imports into Supabase. `gbrain migrate --to pglite` goes the other direction. Bidirectional, lossless.
|
||||
|
||||
The migration and the autopilot daemon do not race: `migrate --to` claims a
|
||||
cooperative pause marker before touching the target. The marker doubles as a
|
||||
migration mutex — a second concurrent migrate refuses to run, and a marker
|
||||
that cannot be written refuses the migration outright. Background job workers
|
||||
stop picking up new work while it is parked, and the migration waits for
|
||||
in-flight sync/embed/cycle work and running jobs to actually drain (watching
|
||||
the DB lock table, capped by `GBRAIN_MIGRATE_QUIESCE_SECONDS` — default 300;
|
||||
`0` skips the wait). Cleanup registers the moment the claim lands, so the
|
||||
marker is released on failure and on catchable signals; a marker orphaned by
|
||||
an uncleanly killed run is adopted by a later migrate only after a
|
||||
pid-liveness check (a live migrate's marker is never stolen), and the daemon
|
||||
clears an orphan whose owning process died on its next poll. `gbrain
|
||||
autopilot --status` reports `paused` (exit 1) while the marker is parked and
|
||||
prints the marker path; on a host with no daemon running to self-heal,
|
||||
remove an orphan by hand only after confirming the pid it names is dead.
|
||||
After a clean flip the daemon detects the engine change on its next
|
||||
tick and relaunches onto the new engine, and the migration warns if an
|
||||
exported connection-string env var would override the new config.
|
||||
|
||||
### Troubleshooting: startup abort (`RuntimeError: Aborted()`)
|
||||
|
||||
**Symptom:** every PGLite-touching command dies at startup with
|
||||
@@ -2859,6 +2884,10 @@ gbrain sync --repo /path/to/brain && gbrain embed --stale
|
||||
[spend controls](../operations/spend-controls.md).
|
||||
- `gbrain embed --stale` -- backfill embeddings for any chunks that don't have
|
||||
them. Safety net for large syncs (>100 files) or prior `--no-embed` runs.
|
||||
On a keyless brain (installed with `--no-embedding`), a bare stale embed
|
||||
refuses cleanly — exit 0 with a stderr note — so this chain is safe to
|
||||
schedule on keyless installs; keyword search keeps working. Explicit embed
|
||||
requests (a slug, `--slugs`, `--all`) still exit 1 on a keyless brain.
|
||||
- `gbrain sync --watch --repo <path>` -- foreground polling loop, every 60s
|
||||
(configurable with `--interval N`). Embeds inline for small changesets. Exits
|
||||
after 5 consecutive failures, so run under a process manager or pair with a
|
||||
@@ -2959,7 +2988,17 @@ vars — incident-time escape hatches, not everyday knobs.
|
||||
history rewrite still hard-blocks even with `--skip-failed`. Run
|
||||
`gbrain sync --skip-failed` to acknowledge a known-bad set yourself.
|
||||
|
||||
5. **Import checkpoints name the import target, not the caller's CWD.**
|
||||
5. **Staleness can't read "fresh" forever.** A source whose content stopped
|
||||
moving (or whose local clone vanished) used to report fresh indefinitely
|
||||
off the stored content timestamp. Content-relative staleness now ramps
|
||||
toward stale once wall-clock time since the last sync passes a ceiling
|
||||
(default 72h; `GBRAIN_STALENESS_CEILING_HOURS` to tune — it tracks
|
||||
`GBRAIN_SYNC_FRESHNESS_FAIL_HOURS` unless set). The ramp is gradual, so
|
||||
the warn tier still fires before the fail tier. `gbrain status` source
|
||||
rows carry `hours_since_last_sync` (raw wall-clock truth) alongside the
|
||||
threshold-relative `staleness_hours` that drives the fresh/stale class.
|
||||
|
||||
6. **Import checkpoints name the import target, not the caller's CWD.**
|
||||
Interrupted `gbrain import <dir>` runs may leave
|
||||
`~/.gbrain/import-checkpoint.json` so the next import can resume. The
|
||||
checkpoint `dir` is the absolute, resolved import target captured when
|
||||
@@ -2986,6 +3025,15 @@ vars — incident-time escape hatches, not everyday knobs.
|
||||
`gbrain embed --stale` isn't running after sync, leaving chunks invisible
|
||||
to vector search.
|
||||
|
||||
4. **Gate on the daemon's heartbeat.** If the built-in daemon runs your sync
|
||||
(`gbrain autopilot --install`), wire your scheduler's health check to
|
||||
`gbrain autopilot --status`. The exit code is the signal: 0 fresh (or
|
||||
nothing installed), 1 needs attention (stale heartbeat, never ran, or
|
||||
paused by a migration), 2 the daemon took itself out of rotation.
|
||||
`--json` emits the full report, including `heartbeat_age_seconds`. Status
|
||||
reads only the filesystem — no database connection — so it keeps working
|
||||
during the exact outages it exists to diagnose.
|
||||
|
||||
---
|
||||
|
||||
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*
|
||||
@@ -3063,6 +3111,12 @@ For scheduling `sync` + `embed --stale` specifically, the home doc is
|
||||
# Brain health — weekly Mondays at 6 AM
|
||||
0 6 * * 1 gbrain doctor --json >> /tmp/gbrain-health.log 2>&1 && gbrain embed --stale
|
||||
|
||||
# Autopilot health gate — daily at 7 AM. The exit code is the signal:
|
||||
# 0 fresh (or nothing installed), 1 needs attention (stale heartbeat,
|
||||
# never ran, or paused), 2 the daemon took itself out of rotation.
|
||||
# Status is filesystem-only, so it works even during a DB outage.
|
||||
0 7 * * * gbrain autopilot --status >> /tmp/gbrain-autopilot-health.log 2>&1 || your-notify "gbrain autopilot needs attention"
|
||||
|
||||
# Dream cycle — nightly at 2 AM
|
||||
0 2 * * * /path/to/dream-cycle.sh
|
||||
```
|
||||
@@ -4908,6 +4962,9 @@ cat ~/.gbrain/preferences.json
|
||||
cat ~/.gbrain/migrations/completed.jsonl
|
||||
|
||||
# 3. Autopilot is supervising a Minions worker child
|
||||
# (v0.46+: the exit code is the verdict — 0 fresh, 1 needs attention,
|
||||
# 2 self-disabled — so a nonzero exit here IS the finding, not a
|
||||
# broken verify step. Under `set -e`, append `|| true` to keep going.)
|
||||
gbrain autopilot --status
|
||||
ps aux | grep 'jobs work'
|
||||
|
||||
|
||||
+1
-1
@@ -152,7 +152,7 @@
|
||||
"bun": ">=1.3.10"
|
||||
},
|
||||
"license": "MIT",
|
||||
"version": "0.45.3.0",
|
||||
"version": "0.45.5.0",
|
||||
"overrides": {
|
||||
"@hono/node-server": "^2.0.5",
|
||||
"fast-uri": "^3.1.5",
|
||||
|
||||
@@ -195,6 +195,13 @@ Verify autopilot is running:
|
||||
```bash
|
||||
gbrain autopilot --status
|
||||
```
|
||||
The exit code is trustworthy for gating: 0 fresh (or nothing installed),
|
||||
1 needs attention (stale heartbeat, never ran, or paused by a migration),
|
||||
2 the daemon disabled itself (its repo path vanished). `--json` emits the
|
||||
full report (`state`, `heartbeat_age_seconds`, `paused_reason`,
|
||||
`disabled_reason`). Status reads only the filesystem, so it works even
|
||||
when the database is down.
|
||||
|
||||
If not running, install it:
|
||||
```bash
|
||||
gbrain autopilot --install --repo ~/brain
|
||||
|
||||
@@ -57,7 +57,7 @@
|
||||
"idea-lineage/routing-eval.jsonl": "ee2e00704b9accb7dd58bb8f126a3bc04a2c40be499180fa505dbf6d5061cd41",
|
||||
"ingest/SKILL.md": "dc40ecc0072806fb8c7bb6ab9cf1f103842e05653eb55d67632d7e3ffc4dd7d2",
|
||||
"install/SKILL.md": "881bd0a422f34c6df4642aae66c51e2a4cc18ad5ca6d0b52d44b4de93512a3c4",
|
||||
"maintain/SKILL.md": "e80cf5bb170c979b773a67a18e0d880c086e0b983db9a576a207074b0112c4f8",
|
||||
"maintain/SKILL.md": "762e20ce7e064709a31a843704600679fd39e6734c9f14a67a98fb8c081b7513",
|
||||
"manifest.json": "8fbdd5fd3c5c8f9b4ebf4c5af18f2396af6f3f638bf180e8ca0b0c35584dc6dd",
|
||||
"media-ingest/SKILL.md": "33db12830ed31a4ff4a6a58c4f126bf2596ee28d54cee0027680c83fee648a20",
|
||||
"meeting-ingestion/SKILL.md": "7767334c63ff3bd8e60cd4d7cd1d1b44f0d6b0a7e0ac411529a1d59cc0a7781b",
|
||||
|
||||
+12
@@ -2251,6 +2251,18 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
}
|
||||
}
|
||||
|
||||
// Autopilot status + uninstall are filesystem-only verdicts and MUST stay
|
||||
// engine-free: a running PGLite daemon holds the exclusive DB lock, so an
|
||||
// engine-bound status would fail to connect in exactly the scenarios the
|
||||
// exit-code contract exists to diagnose (live daemon, DB outage). Order
|
||||
// mirrors runAutopilot's own flag precedence (uninstall before status).
|
||||
if (command === 'autopilot' && (args.includes('--uninstall') || args.includes('--status'))) {
|
||||
const { runAutopilotStatus, uninstallDaemon } = await import('./commands/autopilot.ts');
|
||||
if (args.includes('--uninstall')) uninstallDaemon();
|
||||
else runAutopilotStatus(args);
|
||||
return;
|
||||
}
|
||||
|
||||
// All remaining CLI-only commands need a DB connection
|
||||
const engine = await connectEngine();
|
||||
try {
|
||||
|
||||
+456
-43
@@ -17,13 +17,13 @@
|
||||
* gbrain autopilot --status [--json]
|
||||
*/
|
||||
|
||||
import { existsSync, readFileSync, writeFileSync, mkdirSync, appendFileSync, utimesSync, unlinkSync, chmodSync } from 'fs';
|
||||
import { existsSync, readFileSync, writeFileSync, mkdirSync, appendFileSync, utimesSync, unlinkSync, chmodSync, statSync } from 'fs';
|
||||
import { setCliExitVerdict } from '../core/cli-force-exit.ts';
|
||||
import { join, dirname } from 'path';
|
||||
import { execSync } from 'child_process';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { loadPreferences } from '../core/preferences.ts';
|
||||
import { loadConfig, saveConfig, gbrainPath as gbrainHomePath } from '../core/config.ts';
|
||||
import { loadConfig, loadConfigFileOnly, saveConfig, gbrainPath as gbrainHomePath } from '../core/config.ts';
|
||||
import { ChildWorkerSupervisor } from '../core/minions/child-worker-supervisor.ts';
|
||||
import { VERSION } from '../version.ts';
|
||||
import {
|
||||
@@ -40,6 +40,20 @@ import { evaluateQuietHours } from '../core/minions/quiet-hours.ts';
|
||||
import { inspectLock } from '../core/db-lock.ts';
|
||||
import { registerCleanup } from '../core/process-cleanup.ts';
|
||||
import { resolveAutopilotDispatchTimeoutMs } from './autopilot-timeout.ts';
|
||||
// Path helpers live in a LEAF core module so other commands (gbrain migrate)
|
||||
// can read the daemon's state files without importing this one — a dynamic
|
||||
// import of a command module drags its whole flag surface into the importer's
|
||||
// CLI allowlist. Re-exported here so existing importers keep working.
|
||||
import {
|
||||
autopilotLockPath,
|
||||
autopilotDisabledMarkerPath,
|
||||
autopilotPausedMarkerPath,
|
||||
autopilotDisableStrikesPath,
|
||||
autopilotLaunchdLabel,
|
||||
markerHolderAlive,
|
||||
MIGRATE_PAUSE_MARKER_PREFIX,
|
||||
} from '../core/autopilot-paths.ts';
|
||||
export { autopilotLockPath, autopilotDisabledMarkerPath, autopilotPausedMarkerPath, autopilotLaunchdLabel };
|
||||
|
||||
/**
|
||||
* v0.37.7.0 #1162 — classify autopilot reconnect-loop errors.
|
||||
@@ -52,11 +66,42 @@ import { resolveAutopilotDispatchTimeoutMs } from './autopilot-timeout.ts';
|
||||
* config file unreadable): exit immediately so launchd's 60s
|
||||
* `ThrottleInterval` backs off the relaunch instead of thrashing.
|
||||
*
|
||||
* Exported (string-based signature) so tests drive it without needing
|
||||
* `crash` (a JS TypeError from dereferencing an undefined object): a BUG, not a
|
||||
* verdict about the operator's configuration. Treated as recoverable so a code
|
||||
* defect cannot permanently kill the daemon, but logged distinctly so it is not
|
||||
* silently misfiled as "you forgot to set a URL."
|
||||
*
|
||||
* Exported (string-based signature preserved) so tests drive it without needing
|
||||
* a real reconnect error.
|
||||
*/
|
||||
export function classifyReconnectError(err: unknown): 'recoverable' | 'unrecoverable' {
|
||||
export function classifyReconnectError(err: unknown): 'recoverable' | 'unrecoverable' | 'crash' {
|
||||
// Type check FIRST. Bun/V8 renders a null-deref as
|
||||
// "undefined is not an object (evaluating 'config.database_url')"
|
||||
// which, lowercased, contains BOTH "database_url" and "undefined" — so the
|
||||
// substring rule below classified a crash as a config verdict and exited the
|
||||
// daemon permanently. That is exactly how a 71-day outage started: an engine
|
||||
// migration rewrote config.json, the running daemon crashed on a stale object,
|
||||
// and the crash was reported as "database_url not set".
|
||||
const earlyMsg = (err instanceof Error ? err.message : String(err ?? '')).toLowerCase();
|
||||
// Invalid-URL errors ARE TypeErrors in JS (`new URL('garbage')`), but they
|
||||
// are an operator-config verdict, not a code defect — test the message
|
||||
// pattern BEFORE the blanket TypeError-means-crash rule, or a malformed
|
||||
// database_url spends the whole reconnect budget before exiting.
|
||||
if (earlyMsg.includes('invalid url') || earlyMsg.includes('malformed') || earlyMsg.includes('parse url')) {
|
||||
return 'unrecoverable';
|
||||
}
|
||||
if (err instanceof Error && err.name === 'TypeError') return 'crash';
|
||||
const msg = (err instanceof Error ? err.message : String(err ?? '')).toLowerCase();
|
||||
// Same shape, for hosts where the error arrives as a plain string/serialized
|
||||
// object and the `name` is gone.
|
||||
if (
|
||||
msg.includes('is not an object')
|
||||
|| msg.includes('is not a function')
|
||||
|| msg.includes('cannot read propert')
|
||||
|| msg.includes('undefined is not')
|
||||
) {
|
||||
return 'crash';
|
||||
}
|
||||
if (msg.includes('database_url') && (msg.includes('undefined') || msg.includes('missing') || msg.includes('empty') || msg.includes('not set'))) {
|
||||
return 'unrecoverable';
|
||||
}
|
||||
@@ -381,12 +426,15 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
|
||||
return;
|
||||
}
|
||||
if (args.includes('--status')) {
|
||||
showStatus(args.includes('--json'));
|
||||
runAutopilotStatus(args);
|
||||
return;
|
||||
}
|
||||
|
||||
const repoPath = parseArg(args, '--repo') || await engine.getConfig('sync.repo_path');
|
||||
const baseInterval = parseInt(parseArg(args, '--interval') || '300', 10);
|
||||
// Same NaN guard as the status path: a typo'd interval would otherwise
|
||||
// reach setTimeout(NaN) → 0ms and busy-loop the daemon against the DB.
|
||||
const rawBaseInterval = parseInt(parseArg(args, '--interval') || '300', 10);
|
||||
const baseInterval = Number.isFinite(rawBaseInterval) && rawBaseInterval > 0 ? rawBaseInterval : 300;
|
||||
const jsonMode = args.includes('--json');
|
||||
const forceInline = args.includes('--inline');
|
||||
const noWorker = !shouldSpawnAutopilotWorker(args);
|
||||
@@ -402,7 +450,7 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
|
||||
// two brains sharing GBRAIN_HOME=different-paths still wrote to the
|
||||
// same global lockfile and one would silently respawn the other
|
||||
// forever.
|
||||
const lockPath = gbrainHomePath('autopilot.lock');
|
||||
const lockPath = autopilotLockPath();
|
||||
try {
|
||||
mkdirSync(gbrainHomePath(), { recursive: true });
|
||||
const decision = decideLockAcquisition(lockPath, process.pid);
|
||||
@@ -427,6 +475,15 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
|
||||
const useMinionsDispatch = mode !== 'off' && engineType === 'postgres' && !forceInline;
|
||||
const spawnManagedWorker = useMinionsDispatch && !noWorker;
|
||||
|
||||
// Engine identity at boot, re-checked every tick. A cross-engine migration
|
||||
// flips config.json at the END of its copy; this long-lived process would
|
||||
// otherwise keep syncing into the ABANDONED source engine indefinitely —
|
||||
// the health probe keeps succeeding (the old engine stays alive as the
|
||||
// preserved backup) and reconnect() deliberately restores the config
|
||||
// captured at connect() (#2034), never the new file. Same silent-divergence
|
||||
// class as the dead-daemon incident, moved to after the flip.
|
||||
const engineIdentityAtBoot = autopilotEngineIdentity(loadConfigFileOnly());
|
||||
|
||||
// v0.42 self-upgrade: if a prior tick swapped the binary and exited for
|
||||
// relaunch, we're now the relaunched process — reconcile the breadcrumb so a
|
||||
// crash-on-launch is recorded known-bad and a success is confirmed.
|
||||
@@ -592,6 +649,8 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
|
||||
// runs the full cycle (phase-coupling exercise) before settling into
|
||||
// targeted-submit mode.
|
||||
let lastFullCycleAt = 0;
|
||||
// Log the pause/resume transition once each, not every poll.
|
||||
let pausedAnnounced = false;
|
||||
|
||||
while (!stopping) {
|
||||
const cycleStart = Date.now();
|
||||
@@ -601,6 +660,60 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
|
||||
// declare the instance stale after 10 minutes (Codex C).
|
||||
try { utimesSync(lockPath, new Date(), new Date()); } catch { /* best-effort */ }
|
||||
|
||||
// Post-migration convergence: if the file-plane engine identity changed
|
||||
// since boot, this process is connected to the wrong engine. Exit through
|
||||
// the clean shutdown path (engine close matters for PGLite WAL) so the
|
||||
// supervisor relaunches on the new config; the same relaunch contract the
|
||||
// self-upgrade swap relies on. Cron and one-shot targets simply pick up
|
||||
// the new config on their next run.
|
||||
// A torn or failed read (concurrent config write, transient EACCES) must
|
||||
// not restart the daemon: skip the comparison unless the file read
|
||||
// actually produced a config — a genuine migration flip never yields null.
|
||||
let identityNow: string | null = null;
|
||||
try {
|
||||
const fileCfg = loadConfigFileOnly();
|
||||
identityNow = fileCfg ? autopilotEngineIdentity(fileCfg) : null;
|
||||
} catch { /* torn read mid-write; check again next tick */ }
|
||||
if (identityNow !== null && identityNow !== engineIdentityAtBoot) {
|
||||
console.log('[autopilot] engine config changed on disk (migration?) — exiting for relaunch on the new engine.');
|
||||
await shutdown('engine-config-changed');
|
||||
return;
|
||||
}
|
||||
|
||||
// Cooperative pause (see autopilotPausedMarkerPath). Checked AFTER the
|
||||
// heartbeat so a paused daemon still reads as alive, and BEFORE any DB
|
||||
// work so a cross-engine migration is not racing our writes into an
|
||||
// engine that is about to stop being the configured one.
|
||||
if (existsSync(autopilotPausedMarkerPath())) {
|
||||
// Self-heal an orphan: a migrate-owned marker whose recorded pid is dead
|
||||
// was leaked by a killed migration (SIGKILL, power loss — anything its
|
||||
// own cleanup could not catch). Nothing else ever deletes it, and an
|
||||
// orphan parks this daemon forever. An operator's manual hold (no
|
||||
// migrate signature) is never touched, and a live migrate's marker
|
||||
// reads alive and is honored.
|
||||
let orphaned = false;
|
||||
try {
|
||||
const body = readFileSync(autopilotPausedMarkerPath(), 'utf-8');
|
||||
orphaned = body.startsWith(MIGRATE_PAUSE_MARKER_PREFIX) && markerHolderAlive(body) === 'dead';
|
||||
} catch { /* vanished or unreadable: fall through to the normal pause */ }
|
||||
if (orphaned) {
|
||||
console.log('[autopilot] clearing an orphaned pause marker (its migrate process is dead); resuming.');
|
||||
try { unlinkSync(autopilotPausedMarkerPath()); } catch { /* already gone */ }
|
||||
} else {
|
||||
if (!pausedAnnounced) {
|
||||
console.log('[autopilot] paused (autopilot-paused marker present) — skipping cycles until it clears.');
|
||||
pausedAnnounced = true;
|
||||
}
|
||||
// Poll faster than a normal tick so a migration's quiesce window is short.
|
||||
await new Promise((r) => setTimeout(r, Math.min(baseInterval, 30) * 1000));
|
||||
continue;
|
||||
}
|
||||
}
|
||||
if (pausedAnnounced) {
|
||||
console.log('[autopilot] resumed — pause marker cleared.');
|
||||
pausedAnnounced = false;
|
||||
}
|
||||
|
||||
// DB health check (reconnect if needed).
|
||||
//
|
||||
// v0.37.7.0 #1162: classify reconnect failures. Pre-fix, the
|
||||
@@ -633,7 +746,17 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
|
||||
logError('reconnect', e);
|
||||
autopilotReconnectFails++;
|
||||
const klass = classifyReconnectError(e);
|
||||
if (klass === 'unrecoverable') {
|
||||
if (klass === 'crash') {
|
||||
// A gbrain BUG, not an operator misconfiguration. Say so plainly
|
||||
// instead of blaming the config, and keep retrying: a code defect must
|
||||
// not permanently disable the daemon. The consecutive-failure cap below
|
||||
// still bounds it.
|
||||
console.error(
|
||||
`[autopilot] BUG: internal error during reconnect (${(e as Error).message ?? 'unknown'}). ` +
|
||||
`This is a gbrain defect, not a configuration problem — please report it. ` +
|
||||
`Retrying (${autopilotReconnectFails}/${AUTOPILOT_MAX_RECONNECT_FAILS}).`,
|
||||
);
|
||||
} else if (klass === 'unrecoverable') {
|
||||
console.error(
|
||||
`[autopilot] FATAL: unrecoverable DB error (${(e as Error).message ?? 'unknown'}). ` +
|
||||
`Exiting so launchd ThrottleInterval can apply backoff.`,
|
||||
@@ -1248,11 +1371,11 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
|
||||
// --- Install/Uninstall ---
|
||||
|
||||
function plistPath(): string {
|
||||
return join(process.env.HOME || '', 'Library', 'LaunchAgents', 'com.gbrain.autopilot.plist');
|
||||
return join(process.env.HOME || '', 'Library', 'LaunchAgents', `${autopilotLaunchdLabel()}.plist`);
|
||||
}
|
||||
|
||||
function systemdUnitPath(): string {
|
||||
return join(process.env.HOME || '', '.config', 'systemd', 'user', 'gbrain-autopilot.service');
|
||||
return join(process.env.HOME || '', '.config', 'systemd', 'user', AUTOPILOT_SYSTEMD_UNIT);
|
||||
}
|
||||
|
||||
function ephemeralStartScriptPath(): string {
|
||||
@@ -1311,9 +1434,79 @@ function detectOpenClaw(): { detected: boolean; bootstrapCandidates: string[] }
|
||||
return { detected: signal, bootstrapCandidates: existing };
|
||||
}
|
||||
|
||||
function writeWrapperScript(repoPath: string): string {
|
||||
const home = process.env.HOME || '';
|
||||
const gbrainDir = join(home, '.gbrain');
|
||||
/** systemd unit name. The launchd label lives in `autopilotLaunchdLabel()`
|
||||
* (core/autopilot-paths.ts) so installer, uninstaller, status, and the
|
||||
* wrapper's self-disable can never name different jobs. */
|
||||
export const AUTOPILOT_SYSTEMD_UNIT = 'gbrain-autopilot.service';
|
||||
|
||||
|
||||
/**
|
||||
* Bash block that stops the daemon for good when its captured `--repo` is gone.
|
||||
*
|
||||
* Two things this must get right, both learned the hard way:
|
||||
*
|
||||
* 1. PREDICATE. Test the repo DIRECTORY, not `$repo/.git`. `--repo` may be a
|
||||
* subdirectory of the checkout (sync resolves the root itself by walking up
|
||||
* with `git rev-parse` show-toplevel), and in worktrees and submodules
|
||||
* `.git` is a FILE, not a directory. Testing `.git/` would self-disable a
|
||||
* perfectly healthy install in both shapes.
|
||||
*
|
||||
* 2. MECHANISM. `exit 0` is the right answer for the sibling cron wrapper in
|
||||
* `brain-repo-durability.ts` because launchd fires that one on StartInterval —
|
||||
* one shot, so exiting ends it. Autopilot runs under `KeepAlive=true` +
|
||||
* `ThrottleInterval=60` (and systemd `Restart=always` / `RestartSec=30`),
|
||||
* where exiting 0 disables NOTHING: it converts a dead install into a silent
|
||||
* respawn-every-60s log-append loop that runs forever. On those two targets
|
||||
* the wrapper has to actually take the job out of rotation.
|
||||
*
|
||||
* `linux-cron` and `ephemeral-container` are periodic/one-shot, so a plain
|
||||
* exit is correct and sufficient there.
|
||||
*
|
||||
* Exported pure so tests can assert the emitted shape per target without
|
||||
* installing a daemon.
|
||||
*/
|
||||
export function generateSelfDisableGuard(repoPath: string, target: InstallTarget): string {
|
||||
const q = (s: string) => s.replace(/'/g, "'\\''");
|
||||
const marker = autopilotDisabledMarkerPath();
|
||||
const disableCmd =
|
||||
target === 'macos'
|
||||
? ` launchctl bootout "gui/$(id -u)/${autopilotLaunchdLabel()}" 2>/dev/null || true\n`
|
||||
: target === 'linux-systemd'
|
||||
? ` systemctl --user disable --now ${AUTOPILOT_SYSTEMD_UNIT} 2>/dev/null || true\n`
|
||||
: '';
|
||||
const strikes = autopilotDisableStrikesPath();
|
||||
return `# Self-disable if the captured checkout is gone (rename / relocation / deletion).
|
||||
# Tests the repo DIRECTORY: --repo may be a subdirectory of the checkout, and
|
||||
# .git is a FILE in worktrees and submodules, so [ ! -d "$repo/.git" ] would
|
||||
# false-positive on healthy installs.
|
||||
# THREE consecutive misses before disabling: repos on external volumes, NFS,
|
||||
# or cloud-synced folders are routinely absent for the first launch after
|
||||
# login, and one transient miss must not permanently kill the install. Any
|
||||
# successful probe resets the strike counter.
|
||||
if [ ! -d '${q(repoPath)}' ]; then
|
||||
_strikes=$(($(cat '${q(strikes)}' 2>/dev/null || echo 0) + 1))
|
||||
echo "$_strikes" > '${q(strikes)}' 2>/dev/null || true
|
||||
if [ "$_strikes" -lt 3 ]; then
|
||||
echo "$(date -u +%FT%TZ) [autopilot] repo path missing (strike $_strikes of 3, disabling at 3):" '${q(repoPath)}'
|
||||
exit 0
|
||||
fi
|
||||
echo "$(date -u +%FT%TZ) [autopilot] repo path gone, disabling:" '${q(repoPath)}'
|
||||
printf '%s\\n' 'repo path gone: ${q(repoPath)}' > '${q(marker)}' 2>/dev/null || true
|
||||
rm -f '${q(strikes)}' 2>/dev/null || true
|
||||
${disableCmd} exit 0
|
||||
fi
|
||||
rm -f '${q(strikes)}' 2>/dev/null || true
|
||||
`;
|
||||
}
|
||||
|
||||
function writeWrapperScript(repoPath: string, target: InstallTarget): string {
|
||||
// gbrainHomePath, not raw $HOME: the daemon writes its lock/markers through
|
||||
// it and the status command reads through it, so a GBRAIN_HOME install must
|
||||
// keep its wrapper (and the start-script detection that looks for it) in
|
||||
// the same directory. Identical to the old behavior when GBRAIN_HOME is
|
||||
// unset. The env var is also baked into the wrapper below — launchd does
|
||||
// not pass the installer's environment to the spawned job.
|
||||
const gbrainDir = gbrainHomePath();
|
||||
mkdirSync(gbrainDir, { recursive: true });
|
||||
|
||||
// Wrapper sources the user's shell profile for API keys so nothing is
|
||||
@@ -1352,7 +1545,8 @@ source ~/.zshrc 2>/dev/null || source ~/.bashrc 2>/dev/null || true
|
||||
# fallback, keeps the wrapper self-contained regardless of where bun is installed
|
||||
# or which init file the OS loaded.
|
||||
export PATH=${runtimePathPrefix}"$HOME/.bun/bin:$PATH"
|
||||
exec '${safeGbrainPath}' autopilot --repo '${safeRepoPath}'
|
||||
${process.env.GBRAIN_HOME ? `# Baked at install: the supervisor does not pass the installer's env, and\n# without this the daemon would read/write a different home than the\n# install that configured it.\nexport GBRAIN_HOME='${(process.env.GBRAIN_HOME).replace(/'/g, "'\\''")}'\n` : ''}
|
||||
${generateSelfDisableGuard(repoPath, target)}exec '${safeGbrainPath}' autopilot --repo '${safeRepoPath}'
|
||||
`;
|
||||
writeFileSync(wrapperPath, wrapper, { mode: 0o755 });
|
||||
return wrapperPath;
|
||||
@@ -1371,7 +1565,12 @@ async function installDaemon(engine: BrainEngine, args: string[]) {
|
||||
const injectBootstrap = args.includes('--inject-bootstrap');
|
||||
const noInject = args.includes('--no-inject');
|
||||
|
||||
const wrapperPath = writeWrapperScript(repoPath);
|
||||
const wrapperPath = writeWrapperScript(repoPath, target);
|
||||
// A fresh install clears any prior self-disable AND any leaked pause, so a
|
||||
// reinstall does not report "disabled" forever or park itself from day one
|
||||
// on a marker some dead migration left behind.
|
||||
try { unlinkSync(autopilotDisabledMarkerPath()); } catch { /* not disabled */ }
|
||||
try { unlinkSync(autopilotPausedMarkerPath()); } catch { /* not paused */ }
|
||||
const home = process.env.HOME || '';
|
||||
|
||||
switch (target) {
|
||||
@@ -1401,7 +1600,7 @@ export function generateLaunchdPlist(wrapperPath: string, home: string): string
|
||||
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
|
||||
<plist version="1.0">
|
||||
<dict>
|
||||
<key>Label</key><string>com.gbrain.autopilot</string>
|
||||
<key>Label</key><string>${escapeXml(autopilotLaunchdLabel())}</string>
|
||||
<key>ProgramArguments</key><array>
|
||||
<string>${escapeXml(wrapperPath)}</string>
|
||||
</array>
|
||||
@@ -1438,7 +1637,7 @@ function installLaunchd(wrapperPath: string, home: string, repoPath: string) {
|
||||
// normalize unconditionally.
|
||||
chmodSync(plistPath(), 0o644);
|
||||
execSync(`launchctl load "${plistPath()}"`, { stdio: 'pipe' });
|
||||
console.log('Installed launchd service: com.gbrain.autopilot');
|
||||
console.log(`Installed launchd service: ${autopilotLaunchdLabel()}`);
|
||||
console.log(` Repo: ${repoPath}`);
|
||||
console.log(` Log: ~/.gbrain/autopilot.log`);
|
||||
console.log(' Uninstall: gbrain autopilot --uninstall');
|
||||
@@ -1552,8 +1751,8 @@ function installSystemd(wrapperPath: string, repoPath: string) {
|
||||
// world-writable units); mode only applies on create, so normalize.
|
||||
chmodSync(unitPath, 0o644);
|
||||
execSync('systemctl --user daemon-reload', { stdio: 'pipe', timeout: 10_000 });
|
||||
execSync('systemctl --user enable --now gbrain-autopilot.service', { stdio: 'pipe', timeout: 15_000 });
|
||||
console.log('Installed systemd user service: gbrain-autopilot.service');
|
||||
execSync(`systemctl --user enable --now ${AUTOPILOT_SYSTEMD_UNIT}`, { stdio: 'pipe', timeout: 15_000 });
|
||||
console.log(`Installed systemd user service: ${AUTOPILOT_SYSTEMD_UNIT}`);
|
||||
console.log(` Repo: ${repoPath}`);
|
||||
console.log(' Log: ~/.gbrain/autopilot.log');
|
||||
console.log(' Uninstall: gbrain autopilot --uninstall');
|
||||
@@ -1657,9 +1856,29 @@ function installCrontab(wrapperPath: string, home: string) {
|
||||
}
|
||||
}
|
||||
|
||||
function uninstallDaemon() {
|
||||
/**
|
||||
* The status verdict, engine-free. Dispatched BEFORE connectEngine in cli.ts:
|
||||
* a running PGLite daemon holds the exclusive DB lock, so an engine-bound
|
||||
* status could not run against a healthy live install — and a DB outage would
|
||||
* take down the very alarm meant to diagnose it. Everything it reads is
|
||||
* filesystem (lock mtime, markers, plist/unit/crontab, log tail).
|
||||
*/
|
||||
export function runAutopilotStatus(args: string[]): void {
|
||||
// An INSTALLED daemon always runs the default interval — the generated
|
||||
// wrapper execs `autopilot --repo <path>` with no --interval. The flag is
|
||||
// honored here for the manual foreground case. Garbage input must not
|
||||
// become NaN: staleAfter = NaN makes every age comparison false, which
|
||||
// reads a 71-day-dead daemon as 'fresh' with exit 0 — a typo'd flag would
|
||||
// silently disable the very alarm this exit code exists to be.
|
||||
const rawInterval = parseInt(parseArg(args, '--interval') || '300', 10);
|
||||
showStatus(args.includes('--json'), Number.isFinite(rawInterval) && rawInterval > 0 ? rawInterval : 300);
|
||||
}
|
||||
|
||||
export function uninstallDaemon() {
|
||||
const home = process.env.HOME || '';
|
||||
const wrapperPath = join(home, '.gbrain', 'autopilot-run.sh');
|
||||
// Same resolution as writeWrapperScript — a GBRAIN_HOME install must
|
||||
// uninstall the wrapper it actually wrote, not a sibling under raw $HOME.
|
||||
const wrapperPath = join(gbrainHomePath(), 'autopilot-run.sh');
|
||||
|
||||
// Always try all four targets — the user might have run `--install` under
|
||||
// one target earlier and moved hosts (e.g. macOS laptop → Linux server).
|
||||
@@ -1672,7 +1891,7 @@ function uninstallDaemon() {
|
||||
try {
|
||||
execSync(`launchctl unload "${plistPath()}" 2>/dev/null || true`, { stdio: 'pipe' });
|
||||
unlinkSync(plistPath());
|
||||
console.log('Removed launchd service: com.gbrain.autopilot');
|
||||
console.log(`Removed launchd service: ${autopilotLaunchdLabel()}`);
|
||||
removed++;
|
||||
} catch (e) {
|
||||
console.error(` [warn] launchd: ${e instanceof Error ? e.message : e}`);
|
||||
@@ -1682,7 +1901,7 @@ function uninstallDaemon() {
|
||||
// Linux systemd user unit
|
||||
if (existsSync(systemdUnitPath())) {
|
||||
try {
|
||||
execSync('systemctl --user disable --now gbrain-autopilot.service 2>/dev/null || true', { stdio: 'pipe', timeout: 10_000 });
|
||||
execSync(`systemctl --user disable --now ${AUTOPILOT_SYSTEMD_UNIT} 2>/dev/null || true`, { stdio: 'pipe', timeout: 10_000 });
|
||||
unlinkSync(systemdUnitPath());
|
||||
try { execSync('systemctl --user daemon-reload', { stdio: 'pipe', timeout: 5_000 }); } catch { /* best-effort */ }
|
||||
console.log('Removed systemd user service: gbrain-autopilot.service');
|
||||
@@ -1761,36 +1980,230 @@ function uninstallDaemon() {
|
||||
if (removed === 0) {
|
||||
console.log('No autopilot install found on this host. Nothing to uninstall.');
|
||||
}
|
||||
|
||||
// A deliberate uninstall ends the disabled/paused story: without this, a
|
||||
// self-disabled install that is then uninstalled keeps reporting
|
||||
// "DISABLED — repo path gone" with exit 2 forever on a machine with nothing
|
||||
// installed ('disabled' outranks 'not_installed' in the classifier).
|
||||
try { unlinkSync(autopilotDisabledMarkerPath()); } catch { /* not present */ }
|
||||
try { unlinkSync(autopilotPausedMarkerPath()); } catch { /* not present */ }
|
||||
}
|
||||
|
||||
function showStatus(json: boolean) {
|
||||
const logFile = join(process.env.HOME || '', '.gbrain', 'autopilot.log');
|
||||
let lastLine = '';
|
||||
try {
|
||||
const content = readFileSync(logFile, 'utf-8');
|
||||
const lines = content.trim().split('\n');
|
||||
lastLine = lines[lines.length - 1] || '';
|
||||
} catch { /* no log */ }
|
||||
/**
|
||||
* The daemon's view of WHICH engine the file-plane config points at. Pure and
|
||||
* deliberately narrow: only the fields an engine migration flips participate,
|
||||
* so unrelated config edits (models, search knobs, spend gates) never trigger
|
||||
* a restart. Compared at boot vs every tick by the daemon loop.
|
||||
*/
|
||||
export function autopilotEngineIdentity(
|
||||
cfg: { engine?: string; database_url?: string; database_path?: string } | null,
|
||||
): string {
|
||||
return JSON.stringify({
|
||||
engine: cfg?.engine ?? 'pglite',
|
||||
url: cfg?.database_url ?? null,
|
||||
path: cfg?.database_path ?? null,
|
||||
});
|
||||
}
|
||||
|
||||
let installed = false;
|
||||
if (process.platform === 'darwin') {
|
||||
installed = existsSync(plistPath());
|
||||
} else {
|
||||
export type AutopilotState = 'not_installed' | 'disabled' | 'paused' | 'never_run' | 'stale' | 'fresh';
|
||||
|
||||
export interface AutopilotStatusReport {
|
||||
installed: boolean;
|
||||
install_target: InstallTarget | null;
|
||||
state: AutopilotState;
|
||||
disabled_reason: string | null;
|
||||
paused_reason: string | null;
|
||||
heartbeat_age_seconds: number | null;
|
||||
stale_after_seconds: number;
|
||||
last_log: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Exit codes for `gbrain autopilot --status`, so cron and CI can gate on it.
|
||||
* 0 — fresh, or nothing installed (nothing claimed, nothing broken)
|
||||
* 1 — installed but not syncing (stale heartbeat, never ran, or parked on a
|
||||
* cooperative pause marker — a live migrate, or one that died without
|
||||
* cleaning up; either way the brain is not being kept current)
|
||||
* 2 — the daemon took itself out of rotation (repo gone)
|
||||
*/
|
||||
export function autopilotStatusExitCode(state: AutopilotState): number {
|
||||
if (state === 'disabled') return 2;
|
||||
if (state === 'stale' || state === 'never_run' || state === 'paused') return 1;
|
||||
return 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* Pure classifier so the tri-state is testable without an installed daemon.
|
||||
*
|
||||
* The heartbeat is the lock mtime, which the tick loop already refreshes every
|
||||
* pass (`utimesSync(lockPath, ...)`). Deliberately NOT a new artifact: a second
|
||||
* one could disagree with the first, and a daemon still running a pre-upgrade
|
||||
* binary would never write it, so a healthy install would report stale until it
|
||||
* happened to relaunch.
|
||||
*/
|
||||
export function classifyAutopilotStatus(input: {
|
||||
installed: boolean;
|
||||
installTarget: InstallTarget | null;
|
||||
disabledReason: string | null;
|
||||
pausedReason?: string | null;
|
||||
heartbeatAgeSeconds: number | null;
|
||||
intervalSeconds: number;
|
||||
lastLog: string;
|
||||
}): AutopilotStatusReport {
|
||||
// Tolerance = 6 intervals. The adaptive scheduler sleeps TWO intervals
|
||||
// between ticks on the healthiest brains (score >= 90), and the heartbeat
|
||||
// only refreshes at tick top — so a healthy gap is cycle_duration + 2x
|
||||
// interval, and a 3x tolerance would flap 'stale' exit-1 alarms on exactly
|
||||
// the installs doing best. 6x still catches the dead-daemon incident in
|
||||
// 30 minutes at the default interval instead of 71 days. A non-finite or
|
||||
// non-positive interval (a typo'd flag upstream) would make staleAfter NaN
|
||||
// and every age comparison false — reading a long-dead daemon as 'fresh' —
|
||||
// so the pure layer defends itself too.
|
||||
const staleAfter = Number.isFinite(input.intervalSeconds) && input.intervalSeconds > 0
|
||||
? input.intervalSeconds * 6
|
||||
: 1800;
|
||||
const pausedReason = input.pausedReason ?? null;
|
||||
let state: AutopilotState;
|
||||
if (input.disabledReason !== null) state = 'disabled';
|
||||
else if (!input.installed) state = 'not_installed';
|
||||
// Paused outranks the heartbeat states: the tick loop refreshes its
|
||||
// heartbeat BEFORE honoring the pause marker, so a parked daemon looks
|
||||
// 'fresh' by mtime while doing no work. Without this state a pause marker
|
||||
// orphaned by a dead migrate is invisible — the daemon idles forever and
|
||||
// status swears everything is fine (the 71-day incident's shape again).
|
||||
else if (pausedReason !== null) state = 'paused';
|
||||
else if (input.heartbeatAgeSeconds === null) state = 'never_run';
|
||||
else state = input.heartbeatAgeSeconds > staleAfter ? 'stale' : 'fresh';
|
||||
|
||||
return {
|
||||
installed: input.installed,
|
||||
install_target: input.installTarget,
|
||||
state,
|
||||
disabled_reason: input.disabledReason,
|
||||
paused_reason: pausedReason,
|
||||
heartbeat_age_seconds: input.heartbeatAgeSeconds,
|
||||
stale_after_seconds: staleAfter,
|
||||
last_log: input.lastLog,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Which supervisor, if any, currently holds an autopilot install.
|
||||
*
|
||||
* Checks every target `installDaemon` can produce. The prior version grepped
|
||||
* crontab ONLY on non-darwin, so systemd-user and ephemeral-container installs
|
||||
* read as "not installed" — cosmetic while status always exited 0, but a hard
|
||||
* false failure once the exit code became load-bearing.
|
||||
*/
|
||||
function detectInstalledTarget(): InstallTarget | null {
|
||||
if (process.platform === 'darwin' && existsSync(plistPath())) return 'macos';
|
||||
if (existsSync(systemdUnitPath())) return 'linux-systemd';
|
||||
if (existsSync(join(gbrainHomePath(), 'start-autopilot.sh'))) return 'ephemeral-container';
|
||||
try {
|
||||
const crontab = execSync('crontab -l 2>/dev/null || true', { encoding: 'utf-8' });
|
||||
if (crontabIndicatesAutopilotInstall(crontab)) {
|
||||
return 'linux-cron';
|
||||
}
|
||||
} catch { /* no crontab */ }
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Does this crontab contain an autopilot INSTALL line? The installed line
|
||||
* invokes the generated wrapper (autopilot-run.sh); older installs called
|
||||
* `gbrain autopilot` directly — match either. But the docs also recommend
|
||||
* cron-ing `gbrain autopilot --status` as a health monitor, and counting THAT
|
||||
* line as an install makes a monitor-only machine report installed/never_run
|
||||
* with exit 1 forever. Comments never count. Pure and exported for tests.
|
||||
*/
|
||||
export function crontabIndicatesAutopilotInstall(crontab: string): boolean {
|
||||
return crontab.split('\n').some((line) => {
|
||||
if (line.trimStart().startsWith('#')) return false;
|
||||
if (line.includes('autopilot-run.sh')) return true;
|
||||
return line.includes('gbrain autopilot') && !line.includes('--status');
|
||||
});
|
||||
}
|
||||
|
||||
function showStatus(json: boolean, intervalSeconds: number) {
|
||||
// gbrainHomePath, not raw HOME: the daemon writes its lock through
|
||||
// gbrainHomePath() (#1226), so a GBRAIN_HOME install had status reading one
|
||||
// directory while the daemon wrote another — a permanent false "stale".
|
||||
const home = gbrainHomePath();
|
||||
let lastLine = '';
|
||||
for (const logPath of [join(home, 'autopilot.log'), join(process.env.HOME || '', '.gbrain', 'autopilot.log')]) {
|
||||
try {
|
||||
const crontab = execSync('crontab -l 2>/dev/null || true', { encoding: 'utf-8' });
|
||||
// The installed cron line invokes the generated wrapper (…/autopilot-run.sh);
|
||||
// older installs called `gbrain autopilot` directly. Match either so status
|
||||
// isn't a false negative after the wrapper indirection landed.
|
||||
installed = crontab.includes('autopilot-run.sh') || crontab.includes('gbrain autopilot');
|
||||
} catch { /* no crontab */ }
|
||||
const content = readFileSync(logPath, 'utf-8');
|
||||
const lines = content.trim().split('\n');
|
||||
lastLine = lines[lines.length - 1] || '';
|
||||
break;
|
||||
} catch { /* try the next home; supervisor log redirects bake raw $HOME */ }
|
||||
}
|
||||
|
||||
let disabledReason: string | null = null;
|
||||
try {
|
||||
disabledReason = readFileSync(autopilotDisabledMarkerPath(), 'utf-8').trim() || null;
|
||||
} catch { /* not self-disabled */ }
|
||||
|
||||
let pausedReason: string | null = null;
|
||||
try {
|
||||
pausedReason = readFileSync(autopilotPausedMarkerPath(), 'utf-8').trim() || 'pause marker present (no reason recorded)';
|
||||
} catch { /* not paused */ }
|
||||
|
||||
let heartbeatAgeSeconds: number | null = null;
|
||||
try {
|
||||
const { mtimeMs } = statSync(autopilotLockPath());
|
||||
heartbeatAgeSeconds = Math.max(0, Math.floor((Date.now() - mtimeMs) / 1000));
|
||||
} catch { /* never ran, or already cleaned up */ }
|
||||
|
||||
const installTarget = detectInstalledTarget();
|
||||
const report = classifyAutopilotStatus({
|
||||
installed: installTarget !== null,
|
||||
installTarget,
|
||||
disabledReason,
|
||||
pausedReason,
|
||||
heartbeatAgeSeconds,
|
||||
intervalSeconds,
|
||||
lastLog: lastLine,
|
||||
});
|
||||
|
||||
if (json) {
|
||||
console.log(JSON.stringify({ installed, last_log: lastLine }));
|
||||
console.log(JSON.stringify(report));
|
||||
} else {
|
||||
console.log(`Autopilot: ${installed ? 'installed' : 'not installed'}`);
|
||||
switch (report.state) {
|
||||
case 'not_installed':
|
||||
console.log('Autopilot: not installed. Install with `gbrain autopilot --install`.');
|
||||
break;
|
||||
case 'disabled':
|
||||
console.log(`Autopilot: DISABLED — ${report.disabled_reason}`);
|
||||
console.log(' It stopped itself. Fix the path, then `gbrain autopilot --install --repo <path>`.');
|
||||
break;
|
||||
case 'paused':
|
||||
console.log(`Autopilot: PAUSED — ${report.paused_reason}`);
|
||||
console.log(' A pause marker is parked at ' + autopilotPausedMarkerPath() + '.');
|
||||
console.log(' Normal while `gbrain migrate` runs. A marker orphaned by a dead migration');
|
||||
console.log(' clears itself on the daemon\'s next poll; only remove it by hand if the');
|
||||
console.log(' pid it names is dead and no daemon is running to clean it up.');
|
||||
break;
|
||||
case 'never_run':
|
||||
console.log(`Autopilot: installed (${report.install_target}) but has NEVER run.`);
|
||||
break;
|
||||
case 'stale':
|
||||
console.log(
|
||||
`Autopilot: installed (${report.install_target}) but NOT ticking — last heartbeat ` +
|
||||
`${report.heartbeat_age_seconds}s ago (stale after ${report.stale_after_seconds}s).`,
|
||||
);
|
||||
break;
|
||||
case 'fresh':
|
||||
console.log(
|
||||
`Autopilot: running (${report.install_target}) — last heartbeat ` +
|
||||
`${report.heartbeat_age_seconds}s ago.`,
|
||||
);
|
||||
break;
|
||||
}
|
||||
if (lastLine) console.log(`Last log: ${lastLine}`);
|
||||
}
|
||||
|
||||
setCliExitVerdict(autopilotStatusExitCode(report.state));
|
||||
}
|
||||
|
||||
function escapeXml(s: string): string {
|
||||
|
||||
+52
-35
@@ -48,7 +48,8 @@ import {
|
||||
import { probeSourceGitState } from '../core/git-head.ts';
|
||||
// v0.41.32.0: remote staleness reads the stored newest_content_at column via
|
||||
// this pure comparator (no git subprocess on the HTTP MCP doctor path).
|
||||
import { lagFromContentMs } from '../core/source-health.ts';
|
||||
import { lagFromContentMs, resolveStalenessCeilingSeconds } from '../core/source-health.ts';
|
||||
import { resolveEnvNumber, resolveHoursEnv, warnOnceForEnv } from '../core/env-number.ts';
|
||||
import { CHUNKER_VERSION } from '../core/chunkers/code.ts';
|
||||
import { LINK_EXTRACTOR_VERSION_TS } from '../core/link-extraction.ts';
|
||||
import { isUndefinedColumnError } from '../core/utils.ts';
|
||||
@@ -3322,10 +3323,6 @@ export async function checkSubagentCapability(engine: BrainEngine): Promise<Chec
|
||||
const checkSubagentProvider = checkSubagentCapability;
|
||||
void checkSubagentProvider;
|
||||
|
||||
// Module-scoped set so each invalid-env-var warning fires once per process,
|
||||
// per variable name (v0.42.7 #1696: was a single bool shared across all vars).
|
||||
const _envNumberWarned = new Set<string>();
|
||||
|
||||
/**
|
||||
* v0.42.7 (#1696): single source of truth for the extraction-lag warn
|
||||
* threshold (percent). Both the `links_extraction_lag` doctor check AND the
|
||||
@@ -3339,31 +3336,19 @@ export const EXTRACTION_LAG_WARN_PCT_DEFAULT = 20;
|
||||
export const EXTRACTION_LAG_MIN_PAGES = 100;
|
||||
|
||||
/**
|
||||
* v0.42.7 (#1696, C1): generic "read a positive number from an env var, warn
|
||||
* once + fall back on garbage." Extracted from _resolveSyncFreshnessHours so
|
||||
* the percent-threshold doctor checks don't reuse a `...Hours`-named helper.
|
||||
* `opts.unit` is purely cosmetic for the warning string ('h', '%', '').
|
||||
* Exported (D3) so the sync nudge resolves the threshold the same way.
|
||||
* Re-exported from `src/core/env-number.ts`, which now owns the implementation
|
||||
* AND the warn-once memo. `source-health.ts` needs the hours resolver for the
|
||||
* staleness ceiling, and doctor already imports from source-health — so the
|
||||
* helper had to move to core or the import graph would cycle.
|
||||
*
|
||||
* The `_resolveEnvNumber` name is kept because `sync.ts:5730` dynamically
|
||||
* imports it from this module.
|
||||
*/
|
||||
export function _resolveEnvNumber(varName: string, fallback: number, opts?: { unit?: string }): number {
|
||||
const raw = process.env[varName];
|
||||
if (raw === undefined || raw === '') return fallback;
|
||||
const n = Number(raw);
|
||||
if (!Number.isFinite(n) || n <= 0) {
|
||||
if (!_envNumberWarned.has(varName)) {
|
||||
_envNumberWarned.add(varName);
|
||||
console.warn(
|
||||
`[gbrain doctor] Ignoring invalid ${varName}=${raw}; using default ${fallback}${opts?.unit ?? ''}.`,
|
||||
);
|
||||
}
|
||||
return fallback;
|
||||
}
|
||||
return n;
|
||||
}
|
||||
export { resolveEnvNumber as _resolveEnvNumber };
|
||||
|
||||
function _resolveSyncFreshnessHours(varName: string, fallback: number): number {
|
||||
return _resolveEnvNumber(varName, fallback, { unit: 'h' });
|
||||
}
|
||||
/** Local aliases; the shared memo lives in core so it can't fork per module. */
|
||||
const _resolveEnvNumber = resolveEnvNumber;
|
||||
const _resolveSyncFreshnessHours = resolveHoursEnv;
|
||||
|
||||
/**
|
||||
* Sync freshness check (v0.32.4) — verify that sources with local_path have
|
||||
@@ -3757,9 +3742,11 @@ export async function checkLinksExtractionLag(
|
||||
const n = Number(failRaw);
|
||||
if (Number.isFinite(n) && n > 0) {
|
||||
failPct = n;
|
||||
} else if (!_envNumberWarned.has('GBRAIN_EXTRACTION_LAG_FAIL_PCT')) {
|
||||
_envNumberWarned.add('GBRAIN_EXTRACTION_LAG_FAIL_PCT');
|
||||
console.warn(`[gbrain doctor] Ignoring invalid GBRAIN_EXTRACTION_LAG_FAIL_PCT=${failRaw}; hard-fail stays disabled.`);
|
||||
} else {
|
||||
warnOnceForEnv(
|
||||
'GBRAIN_EXTRACTION_LAG_FAIL_PCT',
|
||||
`[gbrain] Ignoring invalid GBRAIN_EXTRACTION_LAG_FAIL_PCT=${failRaw}; hard-fail stays disabled.`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -4349,7 +4336,7 @@ export async function checkSyncFreshness(
|
||||
// bucket). Empty when nothing is syncing — keeps the steady-state messages
|
||||
// byte-for-byte unchanged.
|
||||
const inProgress: string[] = [];
|
||||
let liveSyncSnap: (sourceId: string) => Promise<{ holder_pid: number; holder_host: string } | null> =
|
||||
let liveSyncSnap: (sourceId: string) => Promise<{ holder_pid: number; holder_host: string; age_ms: number } | null> =
|
||||
async () => null;
|
||||
try {
|
||||
const { inspectLock, syncLockId } = await import('../core/db-lock.ts');
|
||||
@@ -4357,7 +4344,7 @@ export async function checkSyncFreshness(
|
||||
try {
|
||||
const snap = await inspectLock(engine, syncLockId(sourceId));
|
||||
return snap && !snap.ttl_expired
|
||||
? { holder_pid: snap.holder_pid, holder_host: snap.holder_host }
|
||||
? { holder_pid: snap.holder_pid, holder_host: snap.holder_host, age_ms: snap.age_ms }
|
||||
: null;
|
||||
} catch {
|
||||
return null;
|
||||
@@ -4367,6 +4354,10 @@ export async function checkSyncFreshness(
|
||||
/* db-lock unavailable — skip in-progress detection, staleness stands. */
|
||||
}
|
||||
|
||||
// One ceiling for the whole report: hoisted out of the loop so every
|
||||
// source is judged against the same number (and the env read + warn-once
|
||||
// machinery runs once, not once per source).
|
||||
const stalenessCeilingSeconds = resolveStalenessCeilingSeconds();
|
||||
for (const source of sources) {
|
||||
// Embed source.id in user-visible messages so `gbrain sync --source <id>`
|
||||
// matches what the user copy-pastes. Show display name in parens when set.
|
||||
@@ -4376,10 +4367,35 @@ export async function checkSyncFreshness(
|
||||
|
||||
// BUG 4: actively syncing (live lock) → healthy, count as synced_recently
|
||||
// and skip the staleness checks. Keeps the 3-bucket invariant intact.
|
||||
//
|
||||
// ...but ONLY up to the staleness ceiling. `withRefreshingLock` bumps the
|
||||
// heartbeat on its own timer regardless of whether the import is making
|
||||
// forward progress (`liveSyncStatus`'s docstring is explicit: callers may
|
||||
// report "running", NOT "healthy"). So a holder blocked inside a query
|
||||
// keeps refreshing forever, and an uncapped in-progress verdict would
|
||||
// mask that source from every staleness check indefinitely — the same
|
||||
// invisible-failure class this whole pass exists to close, just reached
|
||||
// through the lock table instead of the freshness column.
|
||||
const liveSnap = await liveSyncSnap(source.id);
|
||||
if (liveSnap) {
|
||||
inProgress.push(`${display} sync in progress (pid ${liveSnap.holder_pid} on ${liveSnap.holder_host})`);
|
||||
synced_recently_count++;
|
||||
const ceilingMs = stalenessCeilingSeconds * 1000;
|
||||
if (liveSnap.age_ms <= ceilingMs) {
|
||||
inProgress.push(`${display} sync in progress (pid ${liveSnap.holder_pid} on ${liveSnap.holder_host})`);
|
||||
synced_recently_count++;
|
||||
continue;
|
||||
}
|
||||
// Sub-hour ceilings are legal (fractional env override), and an alarm
|
||||
// that names a zero duration ("held the lock for 0h") reads as broken.
|
||||
const heldFor = liveSnap.age_ms >= 3600_000
|
||||
? `${Math.floor(liveSnap.age_ms / 3600_000)}h`
|
||||
: `${Math.max(1, Math.floor(liveSnap.age_ms / 60_000))}m`;
|
||||
issues.push(
|
||||
`Source ${display} has held the sync lock for ${heldFor} ` +
|
||||
`(pid ${liveSnap.holder_pid} on ${liveSnap.holder_host}) — heartbeating but not finishing. ` +
|
||||
`Run \`gbrain sync --break-lock --source ${source.id}\` after confirming the holder is wedged.`,
|
||||
);
|
||||
hasFailures = true;
|
||||
stale_count++;
|
||||
continue;
|
||||
}
|
||||
|
||||
@@ -4468,6 +4484,7 @@ export async function checkSyncFreshness(
|
||||
contentMs !== null && Number.isFinite(contentMs) ? contentMs : null,
|
||||
lastSync,
|
||||
now,
|
||||
stalenessCeilingSeconds,
|
||||
);
|
||||
thresholdAgeMs = lagSec === null ? ageMs : lagSec * 1000;
|
||||
}
|
||||
|
||||
@@ -492,7 +492,46 @@ export function parsePaceArgs(
|
||||
return { ...(perCallMode !== undefined && { perCallMode }), ...(perCall && { perCall }) };
|
||||
}
|
||||
|
||||
/**
|
||||
* Keyless brains: does this invocation qualify for the stale-mode CLEAN
|
||||
* refusal (zero-failure result → exit 0)? The documented always-current chain
|
||||
* for external agent schedulers is `gbrain sync ... && gbrain embed --stale`
|
||||
* (docs/guides/live-sync.md, INSTALL_FOR_AGENTS.md Step 7) — a hard exit 1
|
||||
* here broke that chain on every brain installed keyless (init's no-embedding
|
||||
* mode), which docs/operations/headless-install.md calls a first-class
|
||||
* supported end state whose embed paths "refuse cleanly". Same class as
|
||||
* sync's resolveNoEmbed sentinel (sync.ts), same remedy. Deliberately NARROW:
|
||||
* only the stale spelling (the chain's). An explicit slug, a slugs list, or
|
||||
* the all flag is an explicit request for something impossible on a keyless
|
||||
* brain and keeps exiting 1 via EmbeddingDisabledError. Note the slugs-list
|
||||
* exclusion mirrors the dispatch precedence below: a slugs flag wins over
|
||||
* stale, so a combined invocation is an explicit-slugs run, not a stale run.
|
||||
*/
|
||||
export function isKeylessStaleRefusal(args: string[], embeddingDisabled: boolean | undefined): boolean {
|
||||
return args.includes('--stale')
|
||||
&& !args.includes('--all')
|
||||
&& !args.includes('--slugs')
|
||||
&& !args.includes('--dry-run')
|
||||
&& embeddingDisabled === true;
|
||||
}
|
||||
|
||||
export async function runEmbed(engine: BrainEngine, args: string[]): Promise<EmbedResult | undefined> {
|
||||
// Keyless clean refusal — see isKeylessStaleRefusal. Checked BEFORE the
|
||||
// background block so we never queue a job that can only fail. stderr only;
|
||||
// stdout stays empty like every other embed outcome (embed has no JSON
|
||||
// result surface — do not invent one here).
|
||||
if (isKeylessStaleRefusal(args, loadConfig()?.embedding_disabled)) {
|
||||
process.stderr.write(
|
||||
'[embed] Embeddings are disabled on this brain (keyless install). '
|
||||
+ 'Nothing to backfill; keyword search keeps working. '
|
||||
+ 'Enable later: set embedding_model via gbrain config, then re-run gbrain init with the force flag.\n',
|
||||
);
|
||||
return {
|
||||
embedded: 0, skipped: 0, would_embed: 0, total_chunks: 0,
|
||||
pages_processed: 0, failures: 0, failure_samples: [], dryRun: false,
|
||||
};
|
||||
}
|
||||
|
||||
// v0.36+ T7: --background submits via Minion queue, returns job_id to
|
||||
// stdout, exits. Same semantics in TTY and cron (D9).
|
||||
if (args.includes('--background')) {
|
||||
|
||||
+405
-123
@@ -11,12 +11,19 @@ import { createEngine } from '../core/engine-factory.ts';
|
||||
import { loadConfig, saveConfig, toEngineConfig, gbrainPath, effectiveEnvDatabaseUrl, type GBrainConfig } from '../core/config.ts';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import type { EngineConfig, Page } from '../core/types.ts';
|
||||
import { writeFileSync, readFileSync, existsSync, unlinkSync } from 'fs';
|
||||
import { writeFileSync, readFileSync, existsSync, unlinkSync, statSync, mkdirSync, renameSync } from 'fs';
|
||||
import { createHash } from 'crypto';
|
||||
import { resolve } from 'path';
|
||||
import { resolve, dirname } from 'path';
|
||||
import { createProgress } from '../core/progress.ts';
|
||||
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
|
||||
import { setCliExitVerdict } from '../core/cli-force-exit.ts';
|
||||
import { registerCleanup } from '../core/process-cleanup.ts';
|
||||
// LEAF module, deliberately NOT './autopilot.ts': the flag-registry generator
|
||||
// follows a command's dynamic imports and would fold autopilot's entire flag
|
||||
// surface into `migrate`'s accepted-flag allowlist.
|
||||
import { autopilotPausedMarkerPath, autopilotLockPath, markerHolderAlive, MIGRATE_PAUSE_MARKER_PREFIX } from '../core/autopilot-paths.ts';
|
||||
export { MIGRATE_PAUSE_MARKER_PREFIX };
|
||||
import { listLiveLocks } from '../core/db-lock.ts';
|
||||
|
||||
interface MigrateOpts {
|
||||
targetEngine: 'postgres' | 'pglite';
|
||||
@@ -237,6 +244,225 @@ export interface MigratePageFailure {
|
||||
reason: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Quiesce a running autopilot daemon for the duration of the copy.
|
||||
*
|
||||
* Why this brackets the copy instead of trailing it: `saveConfig` runs at the
|
||||
* very END of the migration, gated on zero failures. A live daemon therefore
|
||||
* keeps syncing and embedding into the SOURCE engine for the entire copy window,
|
||||
* and everything it writes after page enumeration is silently lost the moment
|
||||
* the config flips. `migrate-engine.ts` previously contained ZERO references to
|
||||
* autopilot, which is the root cause of the observed incident: an engine
|
||||
* migration rewrote `config.json` while a daemon built on the old config kept
|
||||
* running, crashed on the stale object, and nothing linked the two events.
|
||||
*
|
||||
* Cooperative marker, not a supervisor stop — see `autopilotPausedMarkerPath`.
|
||||
* Returns a resume function that is safe to call twice.
|
||||
*/
|
||||
/**
|
||||
* A twice-safe, ownership-conditional unlinker for the pause marker: it
|
||||
* deletes the marker ONLY while the content is still the exact body this
|
||||
* process wrote. If the marker changed hands (an adoption race resolved
|
||||
* against us), deleting by path would un-pause the daemon in the middle of
|
||||
* the new owner's copy window — the precise failure the fence exists to
|
||||
* stop. (The read-then-unlink pair has a microscopic window of its own;
|
||||
* every layer here shrinks the race multiplicatively rather than claiming
|
||||
* to erase it.)
|
||||
*/
|
||||
function markerRelease(marker: string, ownBody: string): () => void {
|
||||
let released = false;
|
||||
return () => {
|
||||
if (released) return;
|
||||
released = true;
|
||||
try {
|
||||
if (readFileSync(marker, 'utf8') !== ownBody) {
|
||||
console.log('[migrate] pause marker changed hands; leaving it for its new owner.');
|
||||
return;
|
||||
}
|
||||
} catch { return; /* already gone */ }
|
||||
try { unlinkSync(marker); console.log('[migrate] autopilot resumed.'); }
|
||||
catch { /* already gone */ }
|
||||
};
|
||||
}
|
||||
|
||||
/** Our marker body: prefix + ownership pid + timestamp. */
|
||||
function markerBody(): string {
|
||||
return `${MIGRATE_PAUSE_MARKER_PREFIX} (pid ${process.pid}) at ${new Date().toISOString()}\n`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Drain cap in seconds. Empty/unset → default; `0` disables the wait
|
||||
* entirely; garbage warns and falls back (an env typo must not silently
|
||||
* remove the drain).
|
||||
*/
|
||||
function resolveDrainCapSeconds(): number {
|
||||
const raw = process.env.GBRAIN_MIGRATE_QUIESCE_SECONDS;
|
||||
if (raw === undefined || raw === '') return 300;
|
||||
if (raw === '0') return 0;
|
||||
const n = Number(raw);
|
||||
if (!Number.isFinite(n) || n < 0) {
|
||||
console.warn(`[migrate] ignoring invalid GBRAIN_MIGRATE_QUIESCE_SECONDS=${JSON.stringify(raw)}; using 300.`);
|
||||
return 300;
|
||||
}
|
||||
return n;
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns a resume closure, or NULL when the pause is owned by someone else —
|
||||
* in which case the caller must ABORT: proceeding without owning the pause
|
||||
* means the real owner's cleanup un-pauses the daemon in the middle of OUR
|
||||
* copy window, and two migrations racing one manifest/target corrupt both.
|
||||
*/
|
||||
export async function quiesceAutopilot(engine?: BrainEngine): Promise<(() => void) | null> {
|
||||
const marker = autopilotPausedMarkerPath();
|
||||
|
||||
// The marker is written UNCONDITIONALLY — not gated on the daemon's lock
|
||||
// file existing. The lock is unlinked on every clean daemon exit (self
|
||||
// upgrade swap, supervisor restart, SIGTERM) and the supervisor relaunches
|
||||
// within a minute, so "no lock right now" recurs routinely on healthy
|
||||
// installs; cron installs have no lock BETWEEN runs, which is most of the
|
||||
// time. A migrate that skipped the marker in such a window would let the
|
||||
// (re)started daemon run full cycles into the source engine for the whole
|
||||
// copy — all silently lost at the config flip. A marker on a machine with
|
||||
// no autopilot at all costs one tiny file, removed by the finally.
|
||||
mkdirSync(dirname(marker), { recursive: true });
|
||||
|
||||
// Claim it atomically ('wx' = fail if it exists) — an existsSync pre-check
|
||||
// would race a concurrent migrate/pauser between check and write. The pid
|
||||
// in the body is the ownership handle for orphan adoption below.
|
||||
let claimed = false;
|
||||
const ownBody = markerBody();
|
||||
try {
|
||||
writeFileSync(marker, ownBody, { flag: 'wx' });
|
||||
claimed = true;
|
||||
} catch (e) {
|
||||
if ((e as NodeJS.ErrnoException)?.code === 'EEXIST') {
|
||||
// A marker already exists. Adopt it ONLY when it was written by a
|
||||
// migrate whose process is provably dead — refusing to adopt a dead
|
||||
// run's orphan would make it permanent (every retry defers to it and
|
||||
// the daemon sits paused forever), while adopting a LIVE run's marker
|
||||
// would be worse: our resume would un-pause the daemon in the middle
|
||||
// of the other migrate's copy window. Content prefix alone cannot tell
|
||||
// those apart; the pid can.
|
||||
try {
|
||||
const body = readFileSync(marker, 'utf8');
|
||||
if (body.startsWith(MIGRATE_PAUSE_MARKER_PREFIX) && markerHolderAlive(body) === 'dead') {
|
||||
// Take OWNERSHIP atomically. Two migrates racing the same orphan
|
||||
// both read the dead pid; a plain rewrite would let BOTH claim it
|
||||
// and one's cleanup would un-pause the daemon mid-copy for the
|
||||
// other. rename(2) is the compare-and-claim: exactly one racer
|
||||
// moves the orphan aside; the loser gets ENOENT and aborts. The
|
||||
// fresh write then goes through the same exclusive 'wx' gate, so a
|
||||
// third claimant sneaking into the gap loses cleanly too.
|
||||
const tomb = `${marker}.reclaim.${process.pid}`;
|
||||
renameSync(marker, tomb);
|
||||
// Verify identity AFTER the rename: between our read above and the
|
||||
// rename, a racing adopter may have completed the whole
|
||||
// adopt-and-rewrite — in which case we just renamed THEIR live
|
||||
// marker, not the orphan. Restore it no-clobber and bow out.
|
||||
let tombBody = '';
|
||||
try { tombBody = readFileSync(tomb, 'utf8'); } catch { /* vanished */ }
|
||||
if (tombBody.startsWith(MIGRATE_PAUSE_MARKER_PREFIX) && markerHolderAlive(tombBody) === 'dead') {
|
||||
try { unlinkSync(tomb); } catch { /* best-effort */ }
|
||||
writeFileSync(marker, ownBody, { flag: 'wx' });
|
||||
console.log('[migrate] adopting a pause marker left by a dead migrate run.');
|
||||
claimed = true;
|
||||
} else {
|
||||
try { writeFileSync(marker, tombBody, { flag: 'wx' }); } catch { /* a racer claimed the gap; theirs now */ }
|
||||
try { unlinkSync(tomb); } catch { /* best-effort */ }
|
||||
}
|
||||
}
|
||||
} catch { /* lost an adoption race, or unreadable: treat as foreign */ }
|
||||
if (!claimed) {
|
||||
// Live migrate, operator hold, or unreadable: NOT ours. Abort rather
|
||||
// than run unowned — see the null contract above.
|
||||
return null;
|
||||
}
|
||||
} else {
|
||||
// EACCES / EROFS / ENOSPC: fail CLOSED, same as the foreign-marker
|
||||
// case. Proceeding unquiesced here would let a running (or supervisor
|
||||
// relaunched) daemon write into the source for the entire copy — all
|
||||
// silently lost at the flip, the exact incident this marker prevents.
|
||||
console.error(`[migrate] cannot write the autopilot pause marker at ${marker}: ${e instanceof Error ? e.message : String(e)}`);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
// Cleanup registered IMMEDIATELY after the claim — a SIGTERM during the
|
||||
// drain below must not orphan the marker. (The daemon also self-clears a
|
||||
// migrate-prefixed marker whose recorded pid is dead, so even a SIGKILL
|
||||
// here heals within one daemon poll.)
|
||||
const release = markerRelease(marker, ownBody);
|
||||
const deregister = registerCleanup('migrate-autopilot-resume', async () => release());
|
||||
const resume = () => { release(); deregister(); };
|
||||
|
||||
let heartbeatAgeMs = Infinity;
|
||||
try { heartbeatAgeMs = Date.now() - statSync(autopilotLockPath()).mtimeMs; }
|
||||
catch { /* no lock: no live daemon */ }
|
||||
console.log(heartbeatAgeMs < 600_000
|
||||
? '[migrate] autopilot detected and paused for the duration of the copy.'
|
||||
: '[migrate] autopilot pause marker parked for the copy window (no live daemon detected; covers one starting mid-copy).');
|
||||
|
||||
// Drain in-flight writers instead of sleeping a blind grace: cycle runs,
|
||||
// sync imports, and embed backfills all hold live rows in the DB lock
|
||||
// table while they work, so "no live locks" is the actual quiesced signal.
|
||||
// An idle daemon holds nothing and the drain returns immediately (faster
|
||||
// than any fixed sleep); a mid-cycle daemon is waited on up to the cap.
|
||||
// Fail-open: a probe error must never block a migration — the marker still
|
||||
// parks the NEXT cycle, which was the whole pre-drain guarantee.
|
||||
const capMs = resolveDrainCapSeconds() * 1000;
|
||||
if (engine && capMs > 0) {
|
||||
const started = Date.now();
|
||||
let lastLog = 0;
|
||||
for (;;) {
|
||||
let live: Awaited<ReturnType<typeof listLiveLocks>>;
|
||||
try {
|
||||
live = (await listLiveLocks(engine))
|
||||
// The supervisor lock is PRESENCE, not work: a healthy minion
|
||||
// supervisor holds it continuously, so counting it would park
|
||||
// every drain at the cap and cry wolf. Actual work (cycle, sync,
|
||||
// embed backfill, unify, ...) holds its own rows.
|
||||
.filter((l) => !l.id.startsWith('gbrain-supervisor:'));
|
||||
} catch (e) {
|
||||
console.warn(`[migrate] could not probe the lock table (${e instanceof Error ? e.message : String(e)}); relying on the pause marker alone.`);
|
||||
break;
|
||||
}
|
||||
// A claimed minion job writes through the worker process; count healthy
|
||||
// active jobs as writers too. Fail-open — a missing queue table (older
|
||||
// schema) must not block the migration.
|
||||
try {
|
||||
const rows = await engine.executeRaw(
|
||||
`SELECT count(*)::text AS count FROM minion_jobs WHERE status = 'active' AND lock_until > now()`,
|
||||
) as Array<{ count?: string }>;
|
||||
const activeJobs = Number(rows?.[0]?.count ?? 0);
|
||||
if (Number.isFinite(activeJobs) && activeJobs > 0) {
|
||||
live = [...live, { id: `minion-jobs-active:${activeJobs}` } as (typeof live)[number]];
|
||||
}
|
||||
} catch { /* no queue table: nothing to count */ }
|
||||
if (live.length === 0) {
|
||||
if (started !== lastLog && lastLog !== 0) console.log('[migrate] in-flight work drained; starting the copy.');
|
||||
break;
|
||||
}
|
||||
const waited = Date.now() - started;
|
||||
if (waited >= capMs) {
|
||||
console.warn(
|
||||
`[migrate] ${live.length} writer lock(s) still live after ${Math.round(waited / 1000)}s (${live.map((l) => l.id).join(', ')}). ` +
|
||||
'Proceeding — anything they write during the copy will NOT be carried to the target. ' +
|
||||
'For large migrations, stop autopilot and idle the job queue first.',
|
||||
);
|
||||
break;
|
||||
}
|
||||
if (waited - lastLog >= 30_000) {
|
||||
lastLog = waited;
|
||||
console.log(`[migrate] waiting for in-flight work to drain: ${live.map((l) => l.id).join(', ')} (${Math.round((capMs - waited) / 1000)}s left)...`);
|
||||
}
|
||||
await new Promise((r) => setTimeout(r, 3_000));
|
||||
}
|
||||
}
|
||||
|
||||
return resume;
|
||||
}
|
||||
|
||||
export async function runMigrateEngine(sourceEngine: BrainEngine, args: string[]): Promise<void> {
|
||||
const opts = parseArgs(args);
|
||||
const config = loadConfig();
|
||||
@@ -251,6 +477,20 @@ export async function runMigrateEngine(sourceEngine: BrainEngine, args: string[]
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Quiesce FIRST — before the target is even connected. The pause marker
|
||||
// doubles as the migrate mutex: a second migrate aborts here instead of
|
||||
// racing this one's manifest and target (a concurrent force run could
|
||||
// delete pages the first run already copied). It also means everything
|
||||
// below (target initSchema, manifest handling, the copy) runs with the
|
||||
// daemon parked. Every process.exit in this function is above this line;
|
||||
// the paths below it return through the finally instead.
|
||||
const resumeAutopilot = await quiesceAutopilot(sourceEngine);
|
||||
if (!resumeAutopilot) {
|
||||
console.error('Could not take ownership of the autopilot pause (a concurrent migrate, an operator hold, or an unwritable marker — see above).');
|
||||
console.error(`Refusing to migrate without it: a daemon this migration cannot pause keeps writing into the source engine, and those writes are lost at the flip. Inspect ${autopilotPausedMarkerPath()} if you are sure nothing is running.`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Build target config
|
||||
const targetConfig: EngineConfig = { engine: opts.targetEngine };
|
||||
if (opts.targetEngine === 'postgres') {
|
||||
@@ -258,7 +498,11 @@ export async function runMigrateEngine(sourceEngine: BrainEngine, args: string[]
|
||||
targetConfig.database_url = opts.targetUrl || effectiveEnvDatabaseUrl();
|
||||
if (!targetConfig.database_url) {
|
||||
console.error('Target is Supabase but no connection string provided. Use: --url <connection_string>');
|
||||
process.exit(1);
|
||||
// Not process.exit: that would skip the resume in the finally below and
|
||||
// leave the daemon parked on the pause marker.
|
||||
setCliExitVerdict(1);
|
||||
resumeAutopilot();
|
||||
return;
|
||||
}
|
||||
} else {
|
||||
targetConfig.database_path = opts.targetPath || gbrainPath('brain.pglite');
|
||||
@@ -307,7 +551,10 @@ export async function runMigrateEngine(sourceEngine: BrainEngine, args: string[]
|
||||
console.error(`Target brain is not empty (${targetStats.page_count} pages).`);
|
||||
console.error('Run with --force to overwrite, or migrate to an empty brain.');
|
||||
await targetEngine.disconnect();
|
||||
process.exit(1);
|
||||
// Not process.exit: the resume must run (see the quiesce block above).
|
||||
setCliExitVerdict(1);
|
||||
resumeAutopilot();
|
||||
return;
|
||||
} else if (targetStats.page_count > 0 && resumingMatchingManifest) {
|
||||
console.log(`Resuming previous migration: ${manifest!.completed_slugs.length} page(s) already copied.`);
|
||||
}
|
||||
@@ -339,133 +586,168 @@ export async function runMigrateEngine(sourceEngine: BrainEngine, args: string[]
|
||||
// Pages.source_id is a foreign key. Copy the complete source catalog first,
|
||||
// including archived rows and sync/routing metadata, so every page write has
|
||||
// a valid parent and the target preserves source behavior.
|
||||
await copyMigrationSources(sourceEngine, targetEngine);
|
||||
|
||||
// Get all source pages
|
||||
const sourceStats = await sourceEngine.getStats();
|
||||
const allPages = await sourceEngine.listPages({ limit: 100000 });
|
||||
const pagesToMigrate = allPages.filter(p => !completedSet.has(makeManifestKey(p.source_id, p.slug)));
|
||||
|
||||
console.log(`Migrating ${pagesToMigrate.length} pages (${allPages.length} total, ${completedSet.size} already done)...`);
|
||||
|
||||
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
|
||||
progress.start('migrate.copy_pages', pagesToMigrate.length);
|
||||
|
||||
// v0.32.8 F8: thread source_id end-to-end so multi-source pages migrate
|
||||
// intact. Pre-fix: putPage / getTags / getTimeline / getRawData / getLinks
|
||||
// all silently defaulted to source_id='default', so non-default-source
|
||||
// tags / timeline / raw / links were either dropped or attached to the
|
||||
// wrong row.
|
||||
//
|
||||
// The pause marker must not outlive this process. `registerCleanup` covers
|
||||
// signals and UNhandled rejections, but a caught-and-rethrown error (or any
|
||||
// handled failure path) exits through normal unwinding — without this
|
||||
// finally, that path leaks the marker and the daemon idles forever while
|
||||
// the status command keeps reporting fresh. On success, finally still runs
|
||||
// AFTER the config flip above, preserving the resume-after-flip ordering:
|
||||
// resuming any earlier would let the daemon write into the source engine
|
||||
// again, and those writes would be the exact ones the migration cannot
|
||||
// carry over.
|
||||
//
|
||||
// Known residual: the finally fires before verifyTarget below, so a fast
|
||||
// supervisor relaunch can touch the target during verification — verify is
|
||||
// read-only, so the cost is at most a noisy count, not corruption.
|
||||
//
|
||||
// NOTE the daemon still holds the PRE-migration config in memory, and its
|
||||
// health probe keeps succeeding against the old engine (the backup stays
|
||||
// alive), so it would never reconnect on its own. Its tick loop compares
|
||||
// the file-plane engine identity against what it booted with
|
||||
// (`autopilotEngineIdentity`) and exits cleanly for supervisor relaunch on
|
||||
// the new config — that check, not reconnect(), is what converges it.
|
||||
let sourceStats!: Awaited<ReturnType<BrainEngine['getStats']>>;
|
||||
let pagesToMigrate: Page[] = [];
|
||||
let migrated = 0;
|
||||
const failures: MigratePageFailure[] = [];
|
||||
for (const page of pagesToMigrate) {
|
||||
try {
|
||||
await copyPageToTarget(sourceEngine, targetEngine, page);
|
||||
// Track progress with composite key so multi-source resume is correct.
|
||||
manifest!.completed_slugs.push(makeManifestKey(page.source_id, page.slug));
|
||||
saveManifest(manifest!);
|
||||
migrated++;
|
||||
} catch (e) {
|
||||
// #3194: a per-page write failure must never be swallowed into the
|
||||
// success count. Leave it OUT of completed_slugs (a resume retries
|
||||
// it — putPage/upsertChunks/etc. are all upserts, so re-running the
|
||||
// whole page copy is safe) and surface it in the final summary below
|
||||
// instead of letting "N pages copied" imply everything landed.
|
||||
failures.push({
|
||||
source_id: page.source_id,
|
||||
slug: page.slug,
|
||||
reason: e instanceof Error ? e.message : String(e),
|
||||
});
|
||||
}
|
||||
progress.tick(1, page.slug);
|
||||
}
|
||||
progress.finish();
|
||||
try {
|
||||
await copyMigrationSources(sourceEngine, targetEngine);
|
||||
|
||||
if (failures.length > 0) {
|
||||
console.error(`\n${failures.length} of ${pagesToMigrate.length} page(s) FAILED to copy and were NOT migrated:`);
|
||||
for (const f of failures) {
|
||||
const key = f.source_id === 'default' ? f.slug : `${f.source_id}::${f.slug}`;
|
||||
console.error(` - ${key}: ${f.reason}`);
|
||||
}
|
||||
console.error('Re-run `gbrain migrate` to retry the failed pages (already-copied pages resume via the manifest).');
|
||||
// Non-fatal so the run still copies links + config for everything that
|
||||
// DID land, but the process must exit non-zero — a partial migration
|
||||
// must never look identical to a clean one.
|
||||
setCliExitVerdict(1);
|
||||
}
|
||||
// Get all source pages
|
||||
sourceStats = await sourceEngine.getStats();
|
||||
const allPages = await sourceEngine.listPages({ limit: 100000 });
|
||||
pagesToMigrate = allPages.filter(p => !completedSet.has(makeManifestKey(p.source_id, p.slug)));
|
||||
|
||||
// Copy links (after all pages exist in target).
|
||||
// v0.32.8 F8: thread source_id so cross-source links migrate correctly.
|
||||
// #3194: a page that failed to copy above does NOT exist on the target,
|
||||
// so any link touching it would violate the target's FK and abort this
|
||||
// whole phase (the exact "addLink failed: page ... not found" crash from
|
||||
// the original report). Skip links on either end of a known-failed page —
|
||||
// a retry that successfully copies the page also re-copies its links.
|
||||
const failedKeys = new Set(failures.map(f => makeManifestKey(f.source_id, f.slug)));
|
||||
console.log('Copying links...');
|
||||
progress.start('migrate.copy_links', allPages.length);
|
||||
for (const page of allPages) {
|
||||
if (failedKeys.has(makeManifestKey(page.source_id, page.slug))) {
|
||||
console.log(`Migrating ${pagesToMigrate.length} pages (${allPages.length} total, ${completedSet.size} already done)...`);
|
||||
|
||||
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
|
||||
progress.start('migrate.copy_pages', pagesToMigrate.length);
|
||||
|
||||
// v0.32.8 F8: thread source_id end-to-end so multi-source pages migrate
|
||||
// intact. Pre-fix: putPage / getTags / getTimeline / getRawData / getLinks
|
||||
// all silently defaulted to source_id='default', so non-default-source
|
||||
// tags / timeline / raw / links were either dropped or attached to the
|
||||
// wrong row.
|
||||
for (const page of pagesToMigrate) {
|
||||
try {
|
||||
await copyPageToTarget(sourceEngine, targetEngine, page);
|
||||
// Track progress with composite key so multi-source resume is correct.
|
||||
manifest!.completed_slugs.push(makeManifestKey(page.source_id, page.slug));
|
||||
saveManifest(manifest!);
|
||||
migrated++;
|
||||
} catch (e) {
|
||||
// #3194: a per-page write failure must never be swallowed into the
|
||||
// success count. Leave it OUT of completed_slugs (a resume retries
|
||||
// it — putPage/upsertChunks/etc. are all upserts, so re-running the
|
||||
// whole page copy is safe) and surface it in the final summary below
|
||||
// instead of letting "N pages copied" imply everything landed.
|
||||
failures.push({
|
||||
source_id: page.source_id,
|
||||
slug: page.slug,
|
||||
reason: e instanceof Error ? e.message : String(e),
|
||||
});
|
||||
}
|
||||
progress.tick(1, page.slug);
|
||||
}
|
||||
progress.finish();
|
||||
|
||||
if (failures.length > 0) {
|
||||
console.error(`\n${failures.length} of ${pagesToMigrate.length} page(s) FAILED to copy and were NOT migrated:`);
|
||||
for (const f of failures) {
|
||||
const key = f.source_id === 'default' ? f.slug : `${f.source_id}::${f.slug}`;
|
||||
console.error(` - ${key}: ${f.reason}`);
|
||||
}
|
||||
console.error('Re-run `gbrain migrate` to retry the failed pages (already-copied pages resume via the manifest).');
|
||||
// Non-fatal so the run still copies links + config for everything that
|
||||
// DID land, but the process must exit non-zero — a partial migration
|
||||
// must never look identical to a clean one.
|
||||
setCliExitVerdict(1);
|
||||
}
|
||||
|
||||
// Copy links (after all pages exist in target).
|
||||
// v0.32.8 F8: thread source_id so cross-source links migrate correctly.
|
||||
// #3194: a page that failed to copy above does NOT exist on the target,
|
||||
// so any link touching it would violate the target's FK and abort this
|
||||
// whole phase (the exact "addLink failed: page ... not found" crash from
|
||||
// the original report). Skip links on either end of a known-failed page —
|
||||
// a retry that successfully copies the page also re-copies its links.
|
||||
const failedKeys = new Set(failures.map(f => makeManifestKey(f.source_id, f.slug)));
|
||||
console.log('Copying links...');
|
||||
progress.start('migrate.copy_links', allPages.length);
|
||||
for (const page of allPages) {
|
||||
if (failedKeys.has(makeManifestKey(page.source_id, page.slug))) {
|
||||
progress.tick(1);
|
||||
continue;
|
||||
}
|
||||
const sourceOpts = { sourceId: page.source_id };
|
||||
const links = await sourceEngine.getLinks(page.slug, sourceOpts);
|
||||
for (const link of links) {
|
||||
if (failedKeys.has(makeManifestKey(page.source_id, link.to_slug))) continue;
|
||||
await targetEngine.addLink(
|
||||
link.from_slug, link.to_slug,
|
||||
link.context, link.link_type,
|
||||
undefined, undefined, undefined,
|
||||
{ fromSourceId: page.source_id, toSourceId: page.source_id },
|
||||
);
|
||||
}
|
||||
progress.tick(1);
|
||||
continue;
|
||||
}
|
||||
const sourceOpts = { sourceId: page.source_id };
|
||||
const links = await sourceEngine.getLinks(page.slug, sourceOpts);
|
||||
for (const link of links) {
|
||||
if (failedKeys.has(makeManifestKey(page.source_id, link.to_slug))) continue;
|
||||
await targetEngine.addLink(
|
||||
link.from_slug, link.to_slug,
|
||||
link.context, link.link_type,
|
||||
undefined, undefined, undefined,
|
||||
{ fromSourceId: page.source_id, toSourceId: page.source_id },
|
||||
);
|
||||
progress.finish();
|
||||
|
||||
// Copy config (selective).
|
||||
//
|
||||
// v0.37 fix wave Lane C.4: these DB-plane writes are SCHEMA METADATA for
|
||||
// the target engine — they record "the schema was sized using this
|
||||
// embedding model + dimension." They are NOT the runtime gateway config
|
||||
// (which lives in the file plane via `~/.gbrain/config.json`). When this
|
||||
// function copies them, it's preserving the schema-applied state across
|
||||
// the migration, not re-pointing the gateway. The newConfig below
|
||||
// doesn't carry these fields because the user's existing file config
|
||||
// already has them (or didn't, in which case the file plane should stay
|
||||
// unset and re-read from gateway defaults).
|
||||
const configKeys = ['embedding_model', 'embedding_dimensions', 'chunk_strategy'];
|
||||
for (const key of configKeys) {
|
||||
const val = await sourceEngine.getConfig(key);
|
||||
if (val) await targetEngine.setConfig(key, val);
|
||||
}
|
||||
progress.tick(1);
|
||||
}
|
||||
progress.finish();
|
||||
|
||||
// Copy config (selective).
|
||||
//
|
||||
// v0.37 fix wave Lane C.4: these DB-plane writes are SCHEMA METADATA for
|
||||
// the target engine — they record "the schema was sized using this
|
||||
// embedding model + dimension." They are NOT the runtime gateway config
|
||||
// (which lives in the file plane via `~/.gbrain/config.json`). When this
|
||||
// function copies them, it's preserving the schema-applied state across
|
||||
// the migration, not re-pointing the gateway. The newConfig below
|
||||
// doesn't carry these fields because the user's existing file config
|
||||
// already has them (or didn't, in which case the file plane should stay
|
||||
// unset and re-read from gateway defaults).
|
||||
const configKeys = ['embedding_model', 'embedding_dimensions', 'chunk_strategy'];
|
||||
for (const key of configKeys) {
|
||||
const val = await sourceEngine.getConfig(key);
|
||||
if (val) await targetEngine.setConfig(key, val);
|
||||
}
|
||||
|
||||
// Update local config. v0.37 fix wave: preserve existing file-plane
|
||||
// embedding/expansion/chat config across the engine migration; only
|
||||
// the engine + connection target should change.
|
||||
//
|
||||
// #3194: only flip the ACTIVE config when the migration is fully clean.
|
||||
// A partial migration leaves the target's data incomplete; auto-switching
|
||||
// every subsequent `gbrain` invocation onto that incomplete target would
|
||||
// (a) make the failure invisible behind otherwise-normal usage and (b)
|
||||
// break the natural retry — `gbrain migrate --to X` again would hit the
|
||||
// "Already using X engine" guard even though the migration never actually
|
||||
// finished. Leaving the file-plane config untouched keeps the source the
|
||||
// active engine, so a retry (which resumes via the still-intact manifest)
|
||||
// is a same-shaped command, not a special case.
|
||||
if (failures.length === 0) {
|
||||
const existingFile = (await import('../core/config.ts')).loadConfigFileOnly() ?? ({} as GBrainConfig);
|
||||
const newConfig: GBrainConfig = {
|
||||
...existingFile,
|
||||
engine: opts.targetEngine,
|
||||
...(opts.targetEngine === 'postgres'
|
||||
? { database_url: targetConfig.database_url, database_path: undefined }
|
||||
: { database_path: targetConfig.database_path, database_url: undefined }),
|
||||
};
|
||||
saveConfig(newConfig);
|
||||
// Clean up the resume manifest — only safe once nothing is left pending.
|
||||
clearManifest();
|
||||
// Update local config. v0.37 fix wave: preserve existing file-plane
|
||||
// embedding/expansion/chat config across the engine migration; only
|
||||
// the engine + connection target should change.
|
||||
//
|
||||
// #3194: only flip the ACTIVE config when the migration is fully clean.
|
||||
// A partial migration leaves the target's data incomplete; auto-switching
|
||||
// every subsequent `gbrain` invocation onto that incomplete target would
|
||||
// (a) make the failure invisible behind otherwise-normal usage and (b)
|
||||
// break the natural retry — `gbrain migrate --to X` again would hit the
|
||||
// "Already using X engine" guard even though the migration never actually
|
||||
// finished. Leaving the file-plane config untouched keeps the source the
|
||||
// active engine, so a retry (which resumes via the still-intact manifest)
|
||||
// is a same-shaped command, not a special case.
|
||||
if (failures.length === 0) {
|
||||
const existingFile = (await import('../core/config.ts')).loadConfigFileOnly() ?? ({} as GBrainConfig);
|
||||
const newConfig: GBrainConfig = {
|
||||
...existingFile,
|
||||
engine: opts.targetEngine,
|
||||
...(opts.targetEngine === 'postgres'
|
||||
? { database_url: targetConfig.database_url, database_path: undefined }
|
||||
: { database_path: targetConfig.database_path, database_url: undefined }),
|
||||
};
|
||||
saveConfig(newConfig);
|
||||
// An exported connection-string env var overrides the file config for
|
||||
// every gbrain process (including the relaunched daemon), which would
|
||||
// silently keep the whole brain on the OLD engine after this flip.
|
||||
const envUrl = effectiveEnvDatabaseUrl();
|
||||
if (envUrl && envUrl !== targetConfig.database_url) {
|
||||
console.warn('WARNING: a DATABASE_URL-style env var is exported and does not match the migration target.');
|
||||
console.warn('It overrides config.json for every gbrain process — unset it (shell profile, launchd plist, systemd unit) or the brain stays on the old engine.');
|
||||
}
|
||||
// Clean up the resume manifest — only safe once nothing is left pending.
|
||||
clearManifest();
|
||||
}
|
||||
} finally {
|
||||
resumeAutopilot();
|
||||
}
|
||||
|
||||
if (failures.length > 0) {
|
||||
|
||||
@@ -5271,6 +5271,14 @@ export interface SyncStatusReportSource {
|
||||
local_path: string | null;
|
||||
sync_enabled: boolean;
|
||||
last_sync_at: string | null;
|
||||
/** Raw wall-clock hours since the last successful sync — the honest human
|
||||
* number. Distinct from staleness_hours, which is threshold-relative. */
|
||||
hours_since_last_sync: number | null;
|
||||
/** Threshold-relative lag driving staleness_class. For a source whose
|
||||
* content is OLDER than its last sync this is the ceiling-ramped value
|
||||
* (see lagFromContentMs), which deliberately under-reads raw wall-clock so
|
||||
* the warn tier fires before the fail tier — display hours_since_last_sync
|
||||
* when a human asks "how long since we synced". */
|
||||
staleness_hours: number | null;
|
||||
staleness_class: 'fresh' | 'stale' | 'severe' | 'unknown';
|
||||
last_commit: string | null;
|
||||
@@ -5446,6 +5454,9 @@ export async function buildSyncStatusReport(
|
||||
now,
|
||||
);
|
||||
const stalenessHours = lagSeconds === null ? null : lagSeconds / 3600;
|
||||
const hoursSinceLastSync = lastSyncMs !== null && Number.isFinite(lastSyncMs)
|
||||
? Math.round(((now - lastSyncMs) / 3600_000) * 10) / 10
|
||||
: null;
|
||||
let stalenessClass: 'fresh' | 'stale' | 'severe' | 'unknown' = 'unknown';
|
||||
if (stalenessHours !== null) {
|
||||
if (stalenessHours < 24) stalenessClass = 'fresh';
|
||||
@@ -5464,6 +5475,7 @@ export async function buildSyncStatusReport(
|
||||
local_path: src.local_path,
|
||||
sync_enabled: cfgEntry.syncEnabled !== false,
|
||||
last_sync_at: lastSyncIso,
|
||||
hours_since_last_sync: hoursSinceLastSync,
|
||||
staleness_hours: stalenessHours === null ? null : Math.round(stalenessHours * 10) / 10,
|
||||
staleness_class: stalenessClass,
|
||||
last_commit: row.last_commit,
|
||||
|
||||
@@ -0,0 +1,115 @@
|
||||
/**
|
||||
* Filesystem coordination points for the autopilot daemon.
|
||||
*
|
||||
* A LEAF module (imports only `path` + `config`) so other commands can read the
|
||||
* daemon's state files WITHOUT importing `src/commands/autopilot.ts`.
|
||||
*
|
||||
* Why that matters concretely: the CLI flag-registry generator follows a
|
||||
* command's dynamic `import('./x.ts')` one level deep and harvests every flag
|
||||
* literal it finds. A single `await import()` of the autopilot command module inside
|
||||
* `migrate-engine.ts` therefore folded autopilot's ENTIRE flag surface (install,
|
||||
* uninstall, interval, no-worker, status, repo, ...) into the migrate command's
|
||||
* accepted-flag allowlist, which would have made `gbrain migrate` silently
|
||||
* accept and ignore any of them. Pinned by `test/cli-flag-validation.test.ts`.
|
||||
*
|
||||
* Flag names appear here WITHOUT leading dashes on purpose — the generator
|
||||
* scans comments too, so writing them literally recreates the very bug this
|
||||
* comment describes.
|
||||
*
|
||||
* Everything here resolves through `gbrainPath` (NOT raw `process.env.HOME`)
|
||||
* because that is where the daemon itself writes; a `GBRAIN_HOME` install
|
||||
* otherwise has readers looking in a different directory than the writer.
|
||||
*/
|
||||
import { join } from 'path';
|
||||
import { gbrainPath } from './config.ts';
|
||||
|
||||
/**
|
||||
* The daemon's lock file. Its mtime IS the liveness heartbeat — the tick loop
|
||||
* refreshes it every pass — so "is autopilot alive?" is `now - mtime` against
|
||||
* the expected interval, with no scheduler probing required.
|
||||
*/
|
||||
export function autopilotLockPath(): string {
|
||||
return gbrainPath('autopilot.lock');
|
||||
}
|
||||
|
||||
/**
|
||||
* Written by the generated wrapper when it self-disables (its captured repo path
|
||||
* is gone). Read by the status command so a stopped daemon explains itself
|
||||
* instead of looking merely idle.
|
||||
*/
|
||||
export function autopilotDisabledMarkerPath(): string {
|
||||
return join(gbrainPath(), 'autopilot-disabled');
|
||||
}
|
||||
|
||||
/**
|
||||
* The launchd job label / plist basename (systemd keeps its own unit name).
|
||||
*
|
||||
* The env override is a TEST SEAM, not a user knob: it exists so the real-launchd
|
||||
* lifecycle e2e can load a genuinely unique job on a dev Mac without colliding
|
||||
* with — or tearing down — the machine's actual autopilot install. It is read at
|
||||
* CALL time, so every invocation touching the same install (install, status,
|
||||
* uninstall, the generated wrapper's self-disable) must run with the same value;
|
||||
* a mismatched pair reports a false not-installed or misses the plist.
|
||||
*
|
||||
* The grammar check is load-bearing: the label reaches a plist FILENAME and a
|
||||
* double-quoted shell line inside the generated wrapper. `escapeXml` protects
|
||||
* only the XML site, so slashes, quotes, whitespace, `$`, backticks, and
|
||||
* dot-dot must be rejected here.
|
||||
*/
|
||||
export function autopilotLaunchdLabel(): string {
|
||||
const label = process.env.GBRAIN_AUTOPILOT_LABEL ?? 'com.gbrain.autopilot';
|
||||
if (!/^[A-Za-z0-9._-]+$/.test(label) || label.includes('..')) {
|
||||
throw new Error(`invalid GBRAIN_AUTOPILOT_LABEL: ${JSON.stringify(label)}`);
|
||||
}
|
||||
return label;
|
||||
}
|
||||
|
||||
/**
|
||||
* Cooperative pause. While present the tick loop keeps heartbeating but does no
|
||||
* work. `gbrain migrate` uses it to quiesce the daemon for the
|
||||
* duration of a cross-engine copy, so the daemon cannot keep writing into an
|
||||
* engine that is about to stop being the configured one.
|
||||
*
|
||||
* A marker rather than stopping the supervisor: ephemeral-container installs
|
||||
* have no supervisor to restart afterwards, a crash mid-migration leaves a
|
||||
* removable file rather than an uninstalled daemon, and it is one code path
|
||||
* across launchd / systemd / cron / container.
|
||||
*/
|
||||
export function autopilotPausedMarkerPath(): string {
|
||||
return join(gbrainPath(), 'autopilot-paused');
|
||||
}
|
||||
|
||||
/**
|
||||
* Consecutive-miss counter for the wrapper's self-disable guard. A repo on an
|
||||
* external volume, NFS, or a cloud-synced folder is routinely absent for the
|
||||
* first daemon launch after login, and a single missed probe must not
|
||||
* permanently take the install out of rotation — the guard requires several
|
||||
* consecutive strikes before disabling, and any successful probe resets this
|
||||
* file.
|
||||
*/
|
||||
export function autopilotDisableStrikesPath(): string {
|
||||
return join(gbrainPath(), 'autopilot-disable-strikes');
|
||||
}
|
||||
|
||||
/**
|
||||
* First line of every pause marker written by `gbrain migrate` — the ownership
|
||||
* signature that separates a migrate-owned pause (safe to adopt when its pid
|
||||
* is dead) from an operator's manual hold (never touched).
|
||||
*/
|
||||
export const MIGRATE_PAUSE_MARKER_PREFIX = 'paused by gbrain migrate';
|
||||
|
||||
/**
|
||||
* Is the pid recorded in a pause-marker body still a live process?
|
||||
* ESRCH = provably dead. EPERM = exists but owned by another user: alive.
|
||||
* No parseable pid = unknown (treat as a foreign hold; adoption needs proof).
|
||||
*/
|
||||
export function markerHolderAlive(body: string): 'alive' | 'dead' | 'unknown' {
|
||||
const m = /\(pid (\d+)\)/.exec(body);
|
||||
if (!m) return 'unknown';
|
||||
try {
|
||||
process.kill(Number(m[1]), 0);
|
||||
return 'alive';
|
||||
} catch (e) {
|
||||
return (e as NodeJS.ErrnoException)?.code === 'ESRCH' ? 'dead' : 'alive';
|
||||
}
|
||||
}
|
||||
@@ -13,7 +13,7 @@ export const CLI_FLAG_REGISTRY: Record<string, readonly string[]> = {
|
||||
'anomalies': ['--aliases', '--all', '--brain', '--fast', '--force', '--from-pages', '--help', '--http', '--include-null-signature', '--json', '--lookback-days', '--mcp-only', '--no-embedding', '--no-extract', '--pattern', '--pending', '--reset', '--resolve', '--sigma', '--since', '--source', '--stale', '--supersessions', '--surface', '--thin', '--timeout'],
|
||||
'apply-migrations': ['--ab', '--all', '--auto-update', '--brain', '--break-lock', '--build-index', '--by-mention', '--compile', '--days', '--dry-run', '--exclusive', '--fast', '--force', '--force-all', '--force-orchestrator', '--force-retry', '--force-schema', '--from-meetings', '--from-pages', '--help', '--history', '--host-dir', '--http', '--json', '--lang', '--list', '--locks', '--markdown', '--max-age', '--migrate-only', '--migration', '--mode', '--multimodal', '--no-autopilot-install', '--no-embedding', '--no-extract', '--non-interactive', '--phase', '--priority', '--refresh-unqualified', '--remediate', '--rollback', '--skip-verify', '--source', '--stale', '--surface', '--undo-wave', '--use-captured-snapshot', '--with-calibration', '--yes'],
|
||||
'auth': ['--aliases', '--all', '--bound-brain', '--bound-max-concurrent', '--bound-slug-prefixes', '--bound-source', '--bound-tools', '--brain', '--budget-usd-per-day', '--enable-dcr', '--enable-dcr-insecure', '--fast', '--federated-read', '--force', '--from-pages', '--grant-types', '--help', '--http', '--include-null-signature', '--json', '--no-embedding', '--no-extract', '--pattern', '--pending', '--redirect-uri', '--reset', '--resolve', '--scopes', '--source', '--stale', '--supersessions', '--surface', '--takes-holders', '--thin', '--token', '--token-endpoint-auth-method', '--yes'],
|
||||
'autopilot': ['--aliases', '--all', '--auto-fix', '--batch', '--brain', '--break-lock', '--by-type', '--check', '--dimensions', '--dir', '--drain', '--embedding-dimensions', '--embedding-model', '--explain', '--fast', '--ff-only', '--fix', '--force', '--force-break-lock', '--force-retry', '--from-pages', '--full', '--help', '--http', '--include-null-signature', '--inject-bootstrap', '--inline', '--input', '--install', '--interval', '--json', '--markdown', '--max-age', '--max-rss', '--max-usd', '--migrate-only', '--model', '--multimodal', '--no', '--no-embed', '--no-embedding', '--no-extract', '--no-inject', '--no-mutate', '--no-worker', '--non-interactive', '--now', '--once', '--output', '--path', '--pattern', '--pending', '--phase', '--refresh-cache', '--remediate', '--remediation-plan', '--repo', '--reset', '--resolve', '--source', '--stale', '--status', '--supersessions', '--surface', '--swap-only', '--target', '--target-score', '--thin', '--timeout', '--to', '--uninstall', '--unsafe-bypass-dream-guard', '--user', '--version', '--yes'],
|
||||
'autopilot': ['--aliases', '--all', '--auto-fix', '--batch', '--brain', '--break-lock', '--by-type', '--check', '--dimensions', '--dir', '--drain', '--embedding-dimensions', '--embedding-model', '--explain', '--fast', '--ff-only', '--fix', '--force', '--force-break-lock', '--force-retry', '--from-pages', '--help', '--http', '--include-null-signature', '--inject-bootstrap', '--inline', '--input', '--install', '--interval', '--json', '--markdown', '--max-age', '--max-rss', '--max-usd', '--migrate-only', '--model', '--multimodal', '--no', '--no-embed', '--no-embedding', '--no-extract', '--no-inject', '--no-mutate', '--no-worker', '--non-interactive', '--now', '--once', '--output', '--path', '--pattern', '--pending', '--phase', '--refresh-cache', '--remediate', '--remediation-plan', '--repo', '--reset', '--resolve', '--source', '--stale', '--status', '--supersessions', '--surface', '--swap-only', '--target', '--target-score', '--thin', '--timeout', '--to', '--uninstall', '--unsafe-bypass-dream-guard', '--user', '--version', '--yes'],
|
||||
'backfill': ['--aliases', '--all', '--batch-size', '--brain', '--concurrency', '--dry-run', '--fresh', '--help', '--include-null-signature', '--json', '--keep-index', '--list', '--max-errors', '--max-rows', '--no-extract', '--pattern', '--pending', '--reset', '--resolve', '--resume', '--source', '--stale', '--supersessions', '--thin'],
|
||||
'bench': ['--baseline', '--brain', '--explain', '--force', '--from', '--help', '--json', '--label', '--lang', '--limit', '--markdown', '--multimodal', '--near-symbol', '--restore-only', '--source', '--stale', '--symbol-kind', '--thin', '--threshold-jaccard', '--threshold-latency-multiplier', '--threshold-top1', '--to', '--tool'],
|
||||
'book-mirror': ['--aliases', '--all', '--allow-empty', '--apply', '--asof', '--author', '--auto', '--background', '--bound-max-concurrent', '--bound-slug-prefixes', '--bound-source', '--bound-tools', '--brain', '--brain-wide-max-cost-usd', '--budget-usd-per-day', '--by-mention', '--chapters-dir', '--content', '--context-file', '--date', '--days', '--dry-run', '--entities', '--explain', '--fast', '--federated', '--file', '--follow', '--force', '--from-pages', '--help', '--http', '--image', '--include-null-signature', '--json', '--kind', '--limit', '--max-turns', '--max-usd', '--mode', '--model', '--multimodal', '--no-confirm', '--no-embedding', '--no-extract', '--no-follow', '--offset', '--path', '--pattern', '--pending', '--progress-interval', '--progress-json', '--quiet', '--remediate', '--reset', '--resolve', '--save', '--session', '--since', '--slug', '--slugs', '--source', '--stale', '--stats', '--supersessions', '--surface', '--thin', '--timeout', '--timeout-ms', '--title', '--trusted-extraction', '--url', '--with-db', '--yes'],
|
||||
@@ -21,7 +21,7 @@ export const CLI_FLAG_REGISTRY: Record<string, readonly string[]> = {
|
||||
'brainstorm': ['--aliases', '--all', '--brain', '--chunker-debug', '--code', '--compile', '--fast', '--fix', '--force', '--force-rechunk', '--force-resume', '--from-pages', '--full', '--help', '--http', '--include-null-signature', '--json', '--judge-model', '--lang', '--limit', '--list-runs', '--markdown', '--max-cost', '--max-far-set', '--max-ideas-per-judge-call', '--model', '--no-embed', '--no-embedding', '--no-extract', '--no-save', '--pattern', '--pending', '--reset', '--resolve', '--resume', '--retry-failed', '--retry-judge', '--save', '--source', '--stale', '--strict-budget', '--supersessions', '--surface', '--thin', '--timeout', '--yes'],
|
||||
'cache': ['--brain', '--fast', '--force', '--from-pages', '--help', '--http', '--json', '--no-embedding', '--source', '--surface', '--yes'],
|
||||
'calibration': ['--ab', '--aliases', '--all', '--allow-empty', '--apply', '--asof', '--auto', '--bound-max-concurrent', '--bound-slug-prefixes', '--bound-source', '--bound-tools', '--brain', '--budget-usd-per-day', '--by-mention', '--content', '--date', '--days', '--dry-run', '--entities', '--explain', '--fast', '--federated', '--file', '--follow', '--force', '--from-pages', '--help', '--holder', '--http', '--image', '--include-null-signature', '--json', '--key-prefix', '--kind', '--lang', '--limit', '--markdown', '--max-usd', '--mode', '--multimodal', '--near-symbol', '--no-embedding', '--no-extract', '--no-federated', '--offset', '--path', '--pattern', '--pending', '--phase', '--progress-interval', '--progress-json', '--quiet', '--regenerate', '--repo', '--reset', '--resolve', '--restore-only', '--save', '--scrub-gstack', '--session', '--since', '--slug', '--slugs', '--source', '--stale', '--stats', '--supersessions', '--surface', '--symbol-kind', '--thin', '--trusted-extraction', '--undo-wave', '--url', '--with-calibration', '--with-db', '--yes'],
|
||||
'call': ['--aliases', '--all', '--all-sources', '--as-context', '--auto-fix', '--background', '--brain', '--by-mention', '--catch-up', '--concurrency', '--confirm-destructive', '--content', '--cost-estimate', '--count', '--days', '--depth', '--dim', '--dir', '--direction', '--enable-dcr', '--enable-dcr-insecure', '--explain', '--fast', '--federated', '--file', '--fix', '--follow', '--force', '--from', '--from-meetings', '--grant-types', '--grep', '--hard-deadline', '--help', '--http', '--image', '--include-expired', '--include-frontmatter', '--include-null-signature', '--infer-dates', '--install', '--interval', '--json', '--key', '--kind', '--lang', '--limit', '--link-source', '--link-type', '--llm', '--migrate-only', '--missing-path', '--multimodal', '--ner', '--no-embed', '--no-expand', '--no-extract', '--no-federated', '--no-hard-deadline', '--no-retry-connect', '--no-save', '--older-than', '--page', '--param', '--params', '--password', '--path', '--pattern', '--pending', '--pglite', '--port', '--progress-interval', '--progress-json', '--public-url', '--queue', '--quiet', '--reenrich-after', '--refresh-cache', '--remediate', '--remediation-plan', '--repo', '--reset', '--resolve', '--restore-only', '--save', '--scopes', '--session', '--sigma', '--since', '--slug', '--slug-prefix', '--source', '--source-id', '--stale', '--status', '--stdin', '--strategy', '--suites', '--supabase', '--supersessions', '--surface', '--symbol-kind', '--synthesize', '--tag', '--thin', '--timeout', '--to', '--today', '--token', '--token-ttl', '--tools-json', '--type', '--url', '--version', '--watch', '--with-calibration', '--workers', '--yes'],
|
||||
'call': ['--aliases', '--all', '--all-sources', '--as-context', '--auto-fix', '--background', '--brain', '--by-mention', '--catch-up', '--concurrency', '--confirm-destructive', '--content', '--cost-estimate', '--count', '--days', '--depth', '--dim', '--dir', '--direction', '--enable-dcr', '--enable-dcr-insecure', '--explain', '--fast', '--federated', '--file', '--fix', '--follow', '--force', '--from', '--from-meetings', '--grant-types', '--grep', '--hard-deadline', '--help', '--http', '--image', '--include-expired', '--include-frontmatter', '--include-null-signature', '--infer-dates', '--install', '--interval', '--json', '--key', '--kind', '--lang', '--limit', '--link-source', '--link-type', '--llm', '--migrate-only', '--missing-path', '--multimodal', '--ner', '--no-embed', '--no-expand', '--no-extract', '--no-federated', '--no-hard-deadline', '--no-retry-connect', '--no-save', '--older-than', '--page', '--param', '--params', '--password', '--path', '--pattern', '--pending', '--pglite', '--port', '--progress-interval', '--progress-json', '--public-url', '--queue', '--quiet', '--reenrich-after', '--refresh-cache', '--remediate', '--remediation-plan', '--repo', '--reset', '--resolve', '--restore-only', '--save', '--scopes', '--session', '--sigma', '--since', '--slug', '--slug-prefix', '--source', '--source-id', '--stale', '--status', '--stdin', '--strategy', '--suites', '--supabase', '--supersessions', '--surface', '--symbol-kind', '--synthesize', '--tag', '--thin', '--timeout', '--to', '--today', '--token', '--token-ttl', '--tools-json', '--type', '--uninstall', '--url', '--version', '--watch', '--with-calibration', '--workers', '--yes'],
|
||||
'capture': ['--aliases', '--all', '--allow-empty', '--apply', '--asof', '--auto', '--bound-max-concurrent', '--bound-slug-prefixes', '--bound-source', '--bound-tools', '--brain', '--budget-usd-per-day', '--by-mention', '--content', '--date', '--days', '--depth', '--entities', '--fast', '--federated', '--file', '--follow', '--force', '--from-pages', '--help', '--http', '--image', '--include-null-signature', '--json', '--kind', '--limit', '--max-usd', '--mcp-only', '--mode', '--multimodal', '--no-embedding', '--no-extract', '--no-federated', '--offset', '--path', '--pattern', '--pending', '--progress-interval', '--progress-json', '--quiet', '--repo', '--reset', '--resolve', '--restore-only', '--save', '--scopes', '--session', '--since', '--slug', '--slugs', '--source', '--stale', '--stats', '--stdin', '--supersessions', '--surface', '--thin', '--timeout', '--trusted-extraction', '--type', '--url', '--what', '--where', '--who', '--with-db', '--yes'],
|
||||
'check-backlinks': ['--background', '--brain', '--brain-wide-max-cost-usd', '--dir', '--dry-run', '--explain', '--follow', '--help', '--include-frontmatter', '--json', '--progress-interval', '--progress-json', '--quiet', '--remediate', '--source', '--stale', '--timeout', '--type'],
|
||||
'check-resolvable': ['--brain', '--dry-run', '--fix', '--help', '--json', '--skills-dir', '--source', '--strict', '--verbose'],
|
||||
@@ -55,11 +55,11 @@ export const CLI_FLAG_REGISTRY: Record<string, readonly string[]> = {
|
||||
'init': ['--all', '--brain', '--chat-model', '--check', '--ctx-size', '--embedding-dimensions', '--embedding-model', '--entity', '--expansion-model', '--fast', '--force', '--from-pages', '--grant-types', '--help', '--http', '--issuer-url', '--json', '--judge-model', '--key', '--mcp-only', '--mcp-url', '--migrate-only', '--model', '--multimodal', '--no', '--no-embed', '--no-embedding', '--non-interactive', '--oauth-client-id', '--oauth-client-secret', '--path', '--pglite', '--provenance', '--schema-pack', '--scopes', '--skip-embed-check', '--source', '--stale', '--supabase', '--surface', '--target', '--to', '--touchpoint', '--url', '--version'],
|
||||
'integrations': ['--auto', '--brain', '--dry-run', '--embeddings', '--fast', '--force', '--from-pages', '--help', '--http', '--json', '--no-embedding', '--overwrite', '--refresh', '--reranking', '--source', '--surface', '--target'],
|
||||
'integrity': ['--aliases', '--all', '--auto', '--backend', '--background', '--brain', '--brain-wide-max-cost-usd', '--check', '--confidence', '--cost', '--dry-run', '--explain', '--fast', '--follow', '--force', '--fresh', '--from-pages', '--help', '--http', '--include-null-signature', '--json', '--limit', '--no-embedding', '--no-extract', '--pattern', '--pending', '--progress-interval', '--progress-json', '--quiet', '--remediate', '--reset', '--resolve', '--review-lower', '--skip-bare-tweet', '--skip-urls', '--source', '--stale', '--supabase', '--supersessions', '--surface', '--thin', '--timeout', '--type', '--url'],
|
||||
'jobs': ['--abbrev-ref', '--aliases', '--all', '--allow-empty', '--allow-protected', '--allow-shell-jobs', '--apply', '--asof', '--auto', '--auto-with-prompt', '--background', '--backoff-delay', '--backoff-jitter', '--backoff-type', '--batch-size', '--bound-max-concurrent', '--bound-slug-prefixes', '--bound-source', '--bound-tools', '--brain', '--break-lock', '--budget-usd', '--budget-usd-per-day', '--by-mention', '--cached', '--catch-up', '--check', '--cli-path', '--cluster', '--cluster-errors', '--code', '--concurrency', '--confidence', '--confirm-destructive', '--content', '--date', '--days', '--delay', '--detach', '--diff-filter', '--dir', '--drain', '--dry-run', '--embedding-dimensions', '--embedding-model', '--empty', '--entities', '--exclude', '--exclude-standard', '--explain', '--fast', '--federated', '--federated-read', '--file', '--fix', '--follow', '--force', '--force-break-lock', '--force-retry', '--format', '--fresh', '--from-meetings', '--from-pages', '--full', '--hard-deadline', '--health-interval', '--held-out', '--help', '--http', '--idempotency-key', '--image', '--include-frontmatter', '--include-gitignored', '--include-null-signature', '--infer-dates', '--inject-bootstrap', '--inline', '--input', '--install', '--interval', '--is-ancestor', '--json', '--kind', '--lang', '--limit', '--lock', '--markdown', '--max-age', '--max-attempts', '--max-cost-usd', '--max-crashes', '--max-rss', '--max-runtime-min', '--max-sources', '--max-stalled', '--max-usd', '--max-waiting', '--mcp-only', '--min-context', '--missing-path', '--mode', '--model', '--multimodal', '--name-only', '--name-status', '--near-symbol', '--ner', '--nice', '--no', '--no-auto-embed', '--no-embed', '--no-embedding', '--no-extract', '--no-federate', '--no-gpg-sign', '--no-hard-deadline', '--no-inject', '--no-mutate', '--no-pull', '--no-renames', '--no-schema-pack', '--no-verify', '--no-worker', '--now', '--offset', '--older-than', '--once', '--order', '--others', '--override-disabled', '--pace', '--pace-max-concurrency', '--pack', '--parallel', '--params', '--path', '--pattern', '--pending', '--phase', '--pid-file', '--priority', '--progress-interval', '--progress-json', '--queue', '--quiet', '--redact-secrets', '--reenrich-after', '--refresh-ms', '--remediate', '--repo', '--reset', '--resolve', '--respect-gitignore', '--restore-only', '--resume', '--retry-failed', '--review-lower', '--run-id', '--save', '--segment-limit', '--serial', '--session', '--short', '--show-toplevel', '--sigkill-rescue', '--since', '--skip-bare-tweet', '--skip-failed', '--skip-urls', '--sleep', '--slug', '--slugs', '--source', '--source-id', '--src-subpath', '--stale', '--stats', '--status', '--strategy', '--supersessions', '--surface', '--swap-only', '--symbol-kind', '--target', '--thin', '--thin-threshold', '--timeout', '--timeout-ms', '--trusted-extraction', '--type', '--types', '--uninstall', '--unsafe-bypass-dream-guard', '--url', '--user', '--verbose', '--version', '--watch', '--wedge-rescue', '--with-db', '--workers', '--yes'],
|
||||
'jobs': ['--abbrev-ref', '--aliases', '--all', '--allow-empty', '--allow-protected', '--allow-shell-jobs', '--apply', '--asof', '--auto', '--auto-fix', '--auto-with-prompt', '--background', '--backoff-delay', '--backoff-jitter', '--backoff-type', '--batch', '--batch-size', '--bound-max-concurrent', '--bound-slug-prefixes', '--bound-source', '--bound-tools', '--brain', '--break-lock', '--budget-usd', '--budget-usd-per-day', '--by-mention', '--by-type', '--cached', '--catch-up', '--check', '--cli-path', '--cluster', '--cluster-errors', '--code', '--concurrency', '--confidence', '--confirm-destructive', '--content', '--date', '--days', '--delay', '--detach', '--diff-filter', '--dimensions', '--dir', '--drain', '--dry-run', '--embedding-dimensions', '--embedding-model', '--empty', '--entities', '--exclude', '--exclude-standard', '--explain', '--fast', '--federated', '--federated-read', '--ff-only', '--file', '--fix', '--follow', '--force', '--force-break-lock', '--force-retry', '--format', '--fresh', '--from-meetings', '--from-pages', '--full', '--hard-deadline', '--health-interval', '--held-out', '--help', '--http', '--idempotency-key', '--image', '--include-frontmatter', '--include-gitignored', '--include-null-signature', '--infer-dates', '--inject-bootstrap', '--inline', '--input', '--install', '--interval', '--is-ancestor', '--json', '--kind', '--lang', '--limit', '--lock', '--markdown', '--max-age', '--max-attempts', '--max-cost-usd', '--max-crashes', '--max-rss', '--max-runtime-min', '--max-sources', '--max-stalled', '--max-usd', '--max-waiting', '--mcp-only', '--migrate-only', '--min-context', '--missing-path', '--mode', '--model', '--multimodal', '--name-only', '--name-status', '--near-symbol', '--ner', '--nice', '--no', '--no-auto-embed', '--no-embed', '--no-embedding', '--no-extract', '--no-federate', '--no-gpg-sign', '--no-hard-deadline', '--no-inject', '--no-mutate', '--no-pull', '--no-renames', '--no-schema-pack', '--no-verify', '--no-worker', '--non-interactive', '--now', '--offset', '--older-than', '--once', '--order', '--others', '--output', '--override-disabled', '--pace', '--pace-max-concurrency', '--pack', '--parallel', '--params', '--path', '--pattern', '--pending', '--phase', '--pid-file', '--priority', '--progress-interval', '--progress-json', '--queue', '--quiet', '--redact-secrets', '--reenrich-after', '--refresh-cache', '--refresh-ms', '--remediate', '--remediation-plan', '--repo', '--reset', '--resolve', '--respect-gitignore', '--restore-only', '--resume', '--retry-failed', '--review-lower', '--run-id', '--save', '--segment-limit', '--serial', '--session', '--short', '--show-toplevel', '--sigkill-rescue', '--since', '--skip-bare-tweet', '--skip-failed', '--skip-urls', '--sleep', '--slug', '--slugs', '--source', '--source-id', '--src-subpath', '--stale', '--stats', '--status', '--strategy', '--supersessions', '--surface', '--swap-only', '--symbol-kind', '--target', '--target-score', '--thin', '--thin-threshold', '--timeout', '--timeout-ms', '--to', '--trusted-extraction', '--type', '--types', '--uninstall', '--unsafe-bypass-dream-guard', '--url', '--user', '--verbose', '--version', '--watch', '--wedge-rescue', '--with-db', '--workers', '--yes'],
|
||||
'lint': ['--aliases', '--all', '--background', '--brain', '--brain-wide-max-cost-usd', '--dry-run', '--exclude', '--explain', '--fast', '--fix', '--follow', '--force', '--from-pages', '--help', '--http', '--include-null-signature', '--json', '--no-embedding', '--no-extract', '--pattern', '--pending', '--progress-interval', '--progress-json', '--quiet', '--remediate', '--reset', '--resolve', '--source', '--stale', '--supersessions', '--surface', '--thin', '--timeout'],
|
||||
'lsd': ['--brain', '--force-resume', '--help', '--json', '--judge-model', '--limit', '--list-runs', '--max-cost', '--max-far-set', '--max-ideas-per-judge-call', '--no-save', '--resume', '--retry-judge', '--save', '--source', '--strict-budget', '--yes'],
|
||||
'maintain': ['--aliases', '--all', '--background', '--brain', '--break-lock', '--by-mention', '--catch-up', '--column', '--concurrency', '--content-audit', '--count', '--detach', '--dim', '--dir', '--drain', '--dry-run', '--embedding-dimensions', '--embedding-model', '--explain', '--fast', '--fix', '--force', '--force-retry', '--force-schema', '--from-meetings', '--full', '--help', '--include-flagged', '--include-frontmatter', '--include-null-signature', '--index-audit', '--infer-dates', '--input', '--json', '--kind', '--lang', '--locks', '--markdown', '--max-cost', '--max-cost-usd', '--max-jobs', '--max-rss', '--max-usd', '--migrate-only', '--multimodal', '--near-symbol', '--ner', '--nice', '--no-extract', '--no-mutate', '--older-than', '--once', '--pack', '--parallel', '--params', '--path', '--pattern', '--pending', '--pglite', '--phase', '--pid-file', '--porcelain', '--progress-json', '--query', '--queue', '--quiet', '--rebuild-rollup', '--regenerate', '--remediate', '--remediation-plan', '--reset', '--resolve', '--restore-only', '--resume', '--run-id', '--safe', '--scope', '--since', '--skills-dir', '--skip-failed', '--slugs', '--source', '--source-id', '--stale', '--status', '--supabase', '--supersessions', '--symbol-kind', '--target', '--target-score', '--thin', '--to', '--top-k', '--type', '--unsafe-bypass-dream-guard', '--url', '--verbose', '--window', '--workers', '--yes'],
|
||||
'migrate': ['--ab', '--aliases', '--all', '--auto-update', '--background', '--batch-size', '--brain', '--brain-wide-max-cost-usd', '--break-lock', '--build-index', '--by-mention', '--catch-up', '--compile', '--days', '--dim', '--dry-run', '--embedding-dimensions', '--embedding-model', '--embeddings', '--exclusive', '--explain', '--fast', '--follow', '--force', '--force-retry', '--force-schema', '--from-meetings', '--from-pages', '--help', '--history', '--http', '--ignore-env-override', '--ignore-missing-key', '--include-null-signature', '--json', '--lang', '--locks', '--markdown', '--max-age', '--model', '--multimodal', '--name', '--near-symbol', '--no', '--no-embed', '--no-embedding', '--no-extract', '--non-interactive', '--pace', '--pace-max-concurrency', '--parallel', '--path', '--pattern', '--pending', '--phase', '--prefix', '--priority', '--progress-interval', '--progress-json', '--quiet', '--refresh-unqualified', '--remediate', '--reranking', '--reset', '--resolve', '--restore-only', '--resume', '--rollback', '--skip-verify', '--slugs', '--source', '--stale', '--supersessions', '--surface', '--symbol-kind', '--thin', '--timeout', '--to', '--undo', '--undo-wave', '--url', '--use-captured-snapshot', '--version', '--with-calibration', '--yes'],
|
||||
'migrate': ['--ab', '--aliases', '--all', '--auto-update', '--background', '--batch-size', '--brain', '--brain-wide-max-cost-usd', '--break-lock', '--build-index', '--by-mention', '--catch-up', '--compile', '--days', '--dim', '--dry-run', '--embedding-dimensions', '--embedding-model', '--embeddings', '--exclusive', '--explain', '--fast', '--fix', '--follow', '--force', '--force-break-lock', '--force-retry', '--force-schema', '--from-meetings', '--from-pages', '--help', '--history', '--http', '--ignore-env-override', '--ignore-missing-key', '--include-null-signature', '--json', '--lang', '--locks', '--markdown', '--max-age', '--model', '--multimodal', '--name', '--near-symbol', '--no', '--no-embed', '--no-embedding', '--no-extract', '--non-interactive', '--pace', '--pace-max-concurrency', '--parallel', '--path', '--pattern', '--pending', '--phase', '--prefix', '--priority', '--progress-interval', '--progress-json', '--quiet', '--refresh-unqualified', '--remediate', '--reranking', '--reset', '--resolve', '--restore-only', '--resume', '--rollback', '--skip-verify', '--slugs', '--source', '--stale', '--supersessions', '--surface', '--symbol-kind', '--thin', '--timeout', '--to', '--undo', '--undo-wave', '--url', '--use-captured-snapshot', '--version', '--with-calibration', '--yes'],
|
||||
'models': ['--aliases', '--all', '--brain', '--embedding-dimensions', '--embedding-model', '--embeddings', '--fast', '--force', '--from-pages', '--help', '--http', '--include-null-signature', '--json', '--judge-model', '--model', '--multimodal', '--no', '--no-embed', '--no-embedding', '--no-extract', '--pattern', '--pending', '--reranking', '--reset', '--resolve', '--skip', '--source', '--stale', '--supersessions', '--surface', '--thin', '--undo', '--version'],
|
||||
'mounts': ['--alias', '--brain', '--cache', '--database-path', '--database-url', '--db-path', '--db-url', '--engine', '--explain', '--help', '--id', '--json', '--lang', '--lock', '--markdown', '--mcp-url', '--multimodal', '--near-symbol', '--path', '--restore-only', '--skills-dir', '--source', '--stale', '--symbol-kind', '--thin', '--verbose'],
|
||||
'notability-eval': ['--aliases', '--all', '--brain', '--embedding-dimensions', '--embedding-model', '--help', '--in', '--include-null-signature', '--json', '--model', '--multimodal', '--no', '--no-embed', '--no-extract', '--out', '--pattern', '--pending', '--repo', '--reset', '--resolve', '--skip-llm', '--source', '--stale', '--supersessions', '--target-high', '--target-low', '--target-medium', '--thin', '--version'],
|
||||
|
||||
@@ -467,6 +467,22 @@ export async function listStaleLocks(engine: BrainEngine): Promise<LockSnapshot[
|
||||
return selectLockRows(engine, { staleOnly: true });
|
||||
}
|
||||
|
||||
/**
|
||||
* Every lock whose holder still looks live (TTL unexpired / holder judged
|
||||
* alive by `isLockHolderLive`). This is the "is anything writing right now?"
|
||||
* probe: cycle runs, sync imports, and embed backfills all hold rows in
|
||||
* `gbrain_cycle_locks` while they work, so an empty result means the write
|
||||
* planes this table guards are drained. Used by `gbrain migrate`'s quiesce
|
||||
* to wait for in-flight work instead of sleeping a blind fixed grace.
|
||||
*/
|
||||
export async function listLiveLocks(
|
||||
engine: BrainEngine,
|
||||
ttlMinutes: number = DEFAULT_TTL_MINUTES,
|
||||
): Promise<LockSnapshot[]> {
|
||||
const rows = await selectLockRows(engine);
|
||||
return rows.filter((snap) => isLockHolderLive(snap, ttlMinutes));
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.41.6.0 D3: atomic verify-and-delete for `gbrain sync --break-lock`.
|
||||
*
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
/**
|
||||
* Numeric env-var resolution with a shared warn-once memo.
|
||||
*
|
||||
* Relocated from `src/commands/doctor.ts` (where `_resolveEnvNumber` was already
|
||||
* exported but `_resolveSyncFreshnessHours` was module-private). `src/core/source-health.ts`
|
||||
* needs the hours resolver for the staleness ceiling, and `doctor.ts:51` already imports
|
||||
* FROM `source-health.ts` — so having core reach back into commands would close an import
|
||||
* cycle. Duplicating the helper instead would create a SECOND `_envNumberWarned` set, so
|
||||
* the same bad env var would warn once per module and the "warn exactly once" contract
|
||||
* would quietly become "warn twice".
|
||||
*
|
||||
* One module, one memo, imported by both planes.
|
||||
*
|
||||
* The warning prefix is `[gbrain]` rather than `[gbrain doctor]` because these vars are
|
||||
* now read outside doctor. No test asserted the old prefix (verified before the move).
|
||||
*/
|
||||
|
||||
/** Warn-once memo, keyed by env var name. Module-level so every caller shares it. */
|
||||
const _envNumberWarned = new Set<string>();
|
||||
|
||||
/**
|
||||
* Read a POSITIVE number from an env var; warn once and fall back on garbage.
|
||||
*
|
||||
* `opts.unit` is cosmetic, used only in the warning string ('h', '%', '').
|
||||
* Zero and negative values are treated as garbage: every consumer of this helper wants
|
||||
* a positive threshold, and `0` is far more often an unset-variable accident than an
|
||||
* intentional "disable".
|
||||
*/
|
||||
export function resolveEnvNumber(
|
||||
varName: string,
|
||||
fallback: number,
|
||||
opts?: { unit?: string },
|
||||
): number {
|
||||
const raw = process.env[varName];
|
||||
if (raw === undefined || raw === '') return fallback;
|
||||
const n = Number(raw);
|
||||
if (!Number.isFinite(n) || n <= 0) {
|
||||
if (!_envNumberWarned.has(varName)) {
|
||||
_envNumberWarned.add(varName);
|
||||
console.warn(
|
||||
`[gbrain] Ignoring invalid ${varName}=${raw}; using default ${fallback}${opts?.unit ?? ''}.`,
|
||||
);
|
||||
}
|
||||
return fallback;
|
||||
}
|
||||
return n;
|
||||
}
|
||||
|
||||
/** `resolveEnvNumber` with the hours unit pre-applied. */
|
||||
export function resolveHoursEnv(varName: string, fallback: number): number {
|
||||
return resolveEnvNumber(varName, fallback, { unit: 'h' });
|
||||
}
|
||||
|
||||
/**
|
||||
* Warn once for `varName` using the SAME memo as `resolveEnvNumber`.
|
||||
*
|
||||
* For env vars that are disabled-unless-set, where `resolveEnvNumber`'s
|
||||
* "fall back to a default" shape does not apply (e.g.
|
||||
* `GBRAIN_EXTRACTION_LAG_FAIL_PCT`, whose absence means "no hard fail"), but
|
||||
* whose invalid-value warning must still fire exactly once per process.
|
||||
* Exposed instead of the raw Set so the memo stays encapsulated.
|
||||
*/
|
||||
export function warnOnceForEnv(varName: string, message: string): void {
|
||||
if (_envNumberWarned.has(varName)) return;
|
||||
_envNumberWarned.add(varName);
|
||||
console.warn(message);
|
||||
}
|
||||
|
||||
/**
|
||||
* Test seam: clear the warn-once memo so a suite can assert the warning fires.
|
||||
* Production code must never call this.
|
||||
*/
|
||||
export function _resetEnvNumberWarnedForTests(): void {
|
||||
_envNumberWarned.clear();
|
||||
}
|
||||
@@ -11,6 +11,8 @@
|
||||
* await worker.start(); // polls until SIGTERM
|
||||
*/
|
||||
|
||||
import { existsSync } from 'fs';
|
||||
import { autopilotPausedMarkerPath } from '../autopilot-paths.ts';
|
||||
import type { BrainEngine } from '../engine.ts';
|
||||
import type {
|
||||
MinionJob, MinionJobContext, MinionHandler, MinionWorkerOpts,
|
||||
@@ -152,6 +154,8 @@ export class MinionWorker extends EventEmitter {
|
||||
private queue: MinionQueue;
|
||||
private handlers = new Map<string, MinionHandler>();
|
||||
private running = false;
|
||||
/** Log the pause/resume transition once each, not every poll. */
|
||||
private pausedByMarkerAnnounced = false;
|
||||
private inFlight = new Map<number, InFlightJob>();
|
||||
private workerId = randomUUID();
|
||||
|
||||
@@ -535,7 +539,24 @@ export class MinionWorker extends EventEmitter {
|
||||
}
|
||||
}
|
||||
|
||||
// Claim jobs up to concurrency limit
|
||||
// Claim jobs up to concurrency limit — unless the system-wide pause
|
||||
// marker is parked. `gbrain migrate` quiesces writers for the copy
|
||||
// window; the marker stops the autopilot dispatch loop, and gating
|
||||
// the CLAIM here extends that fence to queued jobs (an already
|
||||
// in-flight job finishes and is waited on by the migrate drain).
|
||||
// Checked at claim time only: one existsSync per poll tick.
|
||||
if (existsSync(autopilotPausedMarkerPath())) {
|
||||
if (!this.pausedByMarkerAnnounced) {
|
||||
console.log('[worker] pause marker present — not claiming new jobs until it clears.');
|
||||
this.pausedByMarkerAnnounced = true;
|
||||
}
|
||||
await new Promise(resolve => setTimeout(resolve, this.opts.pollInterval));
|
||||
continue;
|
||||
}
|
||||
if (this.pausedByMarkerAnnounced) {
|
||||
console.log('[worker] pause marker cleared — resuming job claims.');
|
||||
this.pausedByMarkerAnnounced = false;
|
||||
}
|
||||
if (this.inFlight.size < this.opts.concurrency) {
|
||||
const lockToken = `${this.workerId}:${Date.now()}`;
|
||||
let job: MinionJob | null;
|
||||
@@ -563,6 +584,16 @@ export class MinionWorker extends EventEmitter {
|
||||
}
|
||||
|
||||
if (job) {
|
||||
// Post-claim fence re-check: the pre-claim marker check above
|
||||
// races migrate's marker write — this claim may have committed
|
||||
// after migrate's drain probe counted zero active jobs. A job
|
||||
// claimed into that window is released back (delayed, un-run)
|
||||
// instead of executed; the poll loop then parks on the marker.
|
||||
if (existsSync(autopilotPausedMarkerPath())) {
|
||||
console.log(`[worker] pause marker appeared after claim — releasing ${job.name} (id=${job.id}) un-run.`);
|
||||
await this.releaseClaimForPause(job, lockToken);
|
||||
continue;
|
||||
}
|
||||
// Quiet-hours gate: evaluated at claim time, not dispatch.
|
||||
// Config lives on the job record (jsonb column added in
|
||||
// schema migration v12). Worker releases the job back to the
|
||||
@@ -629,6 +660,31 @@ export class MinionWorker extends EventEmitter {
|
||||
* 'skip' → status='cancelled', final_status='skipped_quiet_hours'. The
|
||||
* event is dropped.
|
||||
*/
|
||||
/**
|
||||
* Release a just-claimed job back to the queue un-run because the
|
||||
* system-wide pause marker appeared between our pre-claim check and the
|
||||
* claim committing. Same conditional-release SQL shape as the quiet-hours
|
||||
* defer, but with a short delay: the poll loop parks on the marker, so the
|
||||
* job re-enters waiting and is picked up as soon as the pause clears.
|
||||
*/
|
||||
private async releaseClaimForPause(job: MinionJob, lockToken: string): Promise<void> {
|
||||
try {
|
||||
await this.engine.executeRaw(
|
||||
`UPDATE minion_jobs
|
||||
SET status = 'delayed', lock_token = NULL, lock_until = NULL,
|
||||
delay_until = now() + interval '1 minute',
|
||||
updated_at = now()
|
||||
WHERE id = $1 AND lock_token = $2`,
|
||||
[job.id, lockToken],
|
||||
);
|
||||
} catch (e) {
|
||||
// Fail-open: if the release UPDATE itself fails, the claim lock simply
|
||||
// expires and the stall detector requeues the row — slower, same end
|
||||
// state, and never a reason to crash the worker.
|
||||
console.error(`[worker] pause release failed for job ${job.id}: ${e instanceof Error ? e.message : String(e)}`);
|
||||
}
|
||||
}
|
||||
|
||||
private async handleQuietHoursDefer(job: MinionJob, lockToken: string, verdict: 'skip' | 'defer'): Promise<void> {
|
||||
try {
|
||||
if (verdict === 'skip') {
|
||||
|
||||
@@ -4,10 +4,15 @@
|
||||
* v0.41.6.0 D5 — registry + signal handlers so abnormal termination
|
||||
* (SIGTERM/SIGHUP/SIGPIPE, EPIPE on stdout, uncaughtException) releases
|
||||
* locks instead of leaking them for up to 30 minutes until the TTL
|
||||
* expires. Pre-v0.41.6.0, `gbrain sync --full | head -20` would SIGPIPE
|
||||
* gbrain, finally blocks wouldn't run, and the next sync would report
|
||||
* expires. Pre-v0.41.6.0, a full `gbrain sync` piped into `head` would
|
||||
* SIGPIPE gbrain, finally blocks wouldn't run, and the next sync would report
|
||||
* "Another sync is in progress" because the lock row was orphaned.
|
||||
*
|
||||
* (The flag name is spelled out rather than written literally: the CLI
|
||||
* flag-registry generator scans comments, so a bare double-dash token here
|
||||
* would be inherited as an accepted flag by every command that imports this
|
||||
* module. Pinned by `test/cli-flag-validation.test.ts`.)
|
||||
*
|
||||
* Design (per eng-review D7 + outside-voice F9-F11, 2026-05-24):
|
||||
*
|
||||
* - Signal scope: SIGTERM, SIGHUP, SIGPIPE, uncaughtException,
|
||||
|
||||
+80
-12
@@ -25,6 +25,7 @@ import { execFileSync } from 'child_process';
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
import { parseSourceConfig, type SourceRow } from './sources-load.ts';
|
||||
import { isSourceUnchangedSinceSync } from './git-head.ts';
|
||||
import { resolveHoursEnv } from './env-number.ts';
|
||||
|
||||
export interface SourceMetrics {
|
||||
source_id: string;
|
||||
@@ -165,17 +166,56 @@ export function commitTimeMs(localPath: string | null, sha: string | null): numb
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve the staleness-ceiling in seconds.
|
||||
*
|
||||
* `GBRAIN_STALENESS_CEILING_HOURS` overrides; otherwise it tracks
|
||||
* `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS` (default 72) so the ceiling and the check
|
||||
* that reads it cannot drift apart by default, while still being separable
|
||||
* during an incident.
|
||||
*/
|
||||
export function resolveStalenessCeilingSeconds(): number {
|
||||
const base = resolveHoursEnv('GBRAIN_SYNC_FRESHNESS_FAIL_HOURS', 72);
|
||||
return Math.floor(resolveHoursEnv('GBRAIN_STALENESS_CEILING_HOURS', base) * 3600);
|
||||
}
|
||||
|
||||
/**
|
||||
* Commit-relative lag in seconds from a STORED content timestamp (the
|
||||
* `newest_content_at` column), for REMOTE consumers that cannot shell out:
|
||||
* `newest_content_at` column), for consumers that cannot shell out:
|
||||
* - `null` when `lastSyncMs` is unknown.
|
||||
* - Negative wall-clock (future `last_sync_at`) is surfaced as-is so upstream
|
||||
* clock-skew detection still fires.
|
||||
* - `0` when the stored content is at or before the last sync (caught up).
|
||||
* - Caught up (content at or before the last sync) → `0` UNTIL wall-clock
|
||||
* passes the ceiling, then a monotonic ramp (see below).
|
||||
* - Wall-clock `now - lastSync` when content is newer, or when `contentMs` is
|
||||
* null (no column value / pre-migration) — detection never regresses.
|
||||
*
|
||||
* Pure. The LOCAL path does NOT use this — it keys off the live commit hash via
|
||||
* The core logic is pure; only the DEFAULT `ceilingSeconds` reads env. Callers
|
||||
* that need determinism pass the ceiling explicitly.
|
||||
*
|
||||
* WHY THE RAMP (the 71-day bug):
|
||||
*
|
||||
* Before, "caught up" returned a hard 0 forever, so a source whose clone had
|
||||
* vanished reported fresh indefinitely — the wall-clock number was computed on
|
||||
* the line above and thrown away. But the caught-up branch is not a mistake: it
|
||||
* exists (see `checkSyncFreshness`'s clone-unavailable path) because a
|
||||
* container restart wipes `local_path`, and every QUIET source would otherwise
|
||||
* read stale after a restart. That justification assumed a no-op sync does not
|
||||
* advance `last_sync_at`, which stopped being true in v0.42.52.0 — a 0-change
|
||||
* sync now heartbeats the column. So a quiet source that is genuinely being
|
||||
* checked has a RECENT `last_sync_at` and stays at 0; only a source nobody has
|
||||
* looked at in a very long time crosses the ceiling.
|
||||
*
|
||||
* It ramps rather than steps because three consumers read this one value at
|
||||
* different thresholds: `federation_health` fails at 24h, `sync_freshness`
|
||||
* warns at 24h / fails at 72h, and `buildSyncStatusReport` buckets at a
|
||||
* hardcoded 24/72. A step to the ceiling would cross all of them in the same
|
||||
* instant, firing two checks at once and skipping the warn tier entirely —
|
||||
* which is the alert-storm shape the caught-up branch was written to prevent.
|
||||
* `max(0, wallClock - ceiling)` grows continuously, so warn still precedes
|
||||
* fail and the surfaces escalate in order.
|
||||
*
|
||||
* The LOCAL path does NOT use this — it keys off the live commit hash via
|
||||
* `isSourceUnchangedSinceSync` (robust against HEAD moving to an old-dated
|
||||
* commit, which a timestamp comparison would miss).
|
||||
*/
|
||||
@@ -183,12 +223,17 @@ export function lagFromContentMs(
|
||||
contentMs: number | null,
|
||||
lastSyncMs: number | null,
|
||||
nowMs: number,
|
||||
ceilingSeconds: number = resolveStalenessCeilingSeconds(),
|
||||
): number | null {
|
||||
if (lastSyncMs === null || !Number.isFinite(lastSyncMs)) return null;
|
||||
const wallClockSeconds = Math.floor((nowMs - lastSyncMs) / 1000);
|
||||
if (wallClockSeconds < 0) return wallClockSeconds; // clock skew passthrough
|
||||
if (contentMs !== null && Number.isFinite(contentMs)) {
|
||||
return contentMs <= lastSyncMs ? 0 : wallClockSeconds;
|
||||
// Caught up: 0 while recently checked, then ramp once nobody has looked
|
||||
// for longer than the ceiling.
|
||||
return contentMs <= lastSyncMs
|
||||
? Math.max(0, wallClockSeconds - ceilingSeconds)
|
||||
: wallClockSeconds;
|
||||
}
|
||||
return wallClockSeconds; // no stored content signal — wall-clock fallback
|
||||
}
|
||||
@@ -220,6 +265,9 @@ export async function computeAllSourceMetrics(
|
||||
// commit-hash probe; the REMOTE federation_health path leaves it off and
|
||||
// reads the stored column (no subprocess on a DB-supplied local_path).
|
||||
const probeContent = opts?.probeContent === true;
|
||||
// One ceiling for the whole report (hoisted out of the per-source map):
|
||||
// every source gets the same number, and the env read runs once.
|
||||
const stalenessCeilingSeconds = resolveStalenessCeilingSeconds();
|
||||
|
||||
return sources.map((src) => {
|
||||
const cfg = parseSourceConfig(src.config);
|
||||
@@ -232,13 +280,33 @@ export async function computeAllSourceMetrics(
|
||||
: Math.round((chunkStats.embedded / chunkStats.total) * 1000) / 10;
|
||||
|
||||
const lastMs = src.last_sync_at ? new Date(src.last_sync_at).getTime() : null;
|
||||
// v0.41.32.0: commit-relative lag.
|
||||
// LOCAL (probeContent): caught up iff HEAD == last_commit AND no tracked
|
||||
// working-tree changes (untracked ignored) → lag 0; else wall-clock.
|
||||
// Uses the live commit hash so a HEAD that moved to an old-dated commit
|
||||
// is correctly NOT caught up. NULL last_commit → not caught up → wall-clock.
|
||||
// REMOTE (default): read the stored newest_content_at column via
|
||||
// lagFromContentMs — no git subprocess (v0.41.27.0 trust boundary).
|
||||
// v0.41.32.0: commit-relative lag. TWO implementations, and the split is
|
||||
// load-bearing rather than accidental duplication — see the diagram.
|
||||
//
|
||||
// computeAllSourceMetrics
|
||||
// │
|
||||
// ├─ probeContent: true ──► isSourceUnchangedSinceSync (git subprocess)
|
||||
// │ LOCAL only. Accurate: keys off the live
|
||||
// │ commit hash, so a HEAD moved to an
|
||||
// │ old-dated commit is correctly NOT caught
|
||||
// │ up. NULL last_commit → wall-clock.
|
||||
// │ Caller: `gbrain sources status`.
|
||||
// │
|
||||
// └─ probeContent: false ──► lagFromContentMs (column read)
|
||||
// REMOTE-SAFE: no subprocess, because a
|
||||
// remote-callable path must never shell
|
||||
// out against a DB-supplied local_path
|
||||
// (v0.41.27.0 trust boundary). That makes
|
||||
// it structurally less accurate than the
|
||||
// probe, which is exactly why it needs the
|
||||
// staleness ceiling to stay honest.
|
||||
// Callers: doctor `federation_health`,
|
||||
// doctor `sync_freshness` (via its own
|
||||
// clone-unavailable branch), and
|
||||
// `buildSyncStatusReport` (gbrain status).
|
||||
//
|
||||
// Do NOT "unify" these behind one flag: collapsing the trust boundary into a
|
||||
// boolean is how a remote caller eventually acquires a subprocess.
|
||||
let lagSeconds: number | null;
|
||||
if (lastMs === null) {
|
||||
lagSeconds = null;
|
||||
@@ -251,7 +319,7 @@ export async function computeAllSourceMetrics(
|
||||
const contentMs = src.newest_content_at
|
||||
? new Date(src.newest_content_at).getTime()
|
||||
: null;
|
||||
lagSeconds = lagFromContentMs(contentMs, lastMs, now);
|
||||
lagSeconds = lagFromContentMs(contentMs, lastMs, now, stalenessCeilingSeconds);
|
||||
}
|
||||
|
||||
return {
|
||||
|
||||
@@ -206,8 +206,24 @@ export function formatCodeBreakdown(
|
||||
return summary.map(s => ` ${s.code}: ${s.count}`).join('\n');
|
||||
}
|
||||
|
||||
/**
|
||||
* Where `sync-failures.jsonl` lives. Defaults to the gbrain home.
|
||||
*
|
||||
* `GBRAIN_SYNC_FAILURES_DIR` exists for the same reason `GBRAIN_AUDIT_DIR` does
|
||||
* (#2823): the test suite exercises import/sync failure paths with deliberately
|
||||
* broken fixtures, and without an override those fixture rows append into the
|
||||
* operator's REAL ledger — the one `gbrain doctor` reads and warns on until it
|
||||
* is cleaned up. A stray `srcE / notes/bad.md SLUG_MISMATCH` row in a live
|
||||
* brain came from exactly this.
|
||||
*
|
||||
* A dedicated var rather than leaning on `GBRAIN_HOME`: pointing GBRAIN_HOME at
|
||||
* a scratch dir for the whole suite also makes `loadConfig()` return null for
|
||||
* every test that reads the real config, which is a far wider blast radius than
|
||||
* this problem needs (and `test/gbrain-home-isolation.test.ts` asserts the
|
||||
* unset-fallback behavior directly).
|
||||
*/
|
||||
function _failuresDir(): string {
|
||||
return _gbrainPath();
|
||||
return process.env.GBRAIN_SYNC_FAILURES_DIR || _gbrainPath();
|
||||
}
|
||||
|
||||
export function syncFailuresPath(): string {
|
||||
|
||||
+37
-7
@@ -99,8 +99,31 @@ const CODE_EXTENSIONS = new Set<string>([
|
||||
* Input format (tab-separated):
|
||||
* A path/to/new-file.md
|
||||
* M path/to/modified-file.md
|
||||
* T path/to/retyped-file.md (file <-> symlink; content changed)
|
||||
* D path/to/deleted-file.md
|
||||
* R100 old/path.md new/path.md
|
||||
*
|
||||
* Status coverage, against the options gbrain actually passes (name-status with
|
||||
* `-M` only, see `sync-delta.ts` — note NO `-C` copy detection):
|
||||
* A/M/D/R — the everyday statuses.
|
||||
* T — TYPECHANGE. Reachable in normal operation: replacing a file with
|
||||
* a symlink (or vice versa) emits `T`, and it was silently dropped,
|
||||
* so the change never reached the index until something else
|
||||
* touched the path. Treated as `modified`: the path still exists and
|
||||
* its blob changed, which is exactly what re-import handles. (When a
|
||||
* file becomes a symlink, import-file SKIPS symlinks by design —
|
||||
* the exfil guard — so the old regular-file content stays indexed;
|
||||
* routing the typechange to delete-then-skip is a filed follow-up.)
|
||||
* C — COPY. Unreachable without `-C`, handled defensively.
|
||||
* U — UNMERGED. Only in a conflicted worktree, which sync does not run
|
||||
* against; handled defensively so a conflicted tree degrades to
|
||||
* "re-import this path" rather than silently skipping it.
|
||||
*
|
||||
* NOTE for future editors: git option names appear here WITHOUT the leading
|
||||
* double-dash on purpose. The CLI flag-registry generator string-scans module
|
||||
* source including comments, so a bare double-dash token in prose registers as
|
||||
* a real, accepted CLI flag on every command that imports this file. Pinned by
|
||||
* `test/cli-flag-validation.test.ts`.
|
||||
*/
|
||||
export function buildSyncManifest(gitDiffOutput: string): SyncManifest {
|
||||
const manifest: SyncManifest = {
|
||||
@@ -120,20 +143,27 @@ export function buildSyncManifest(gitDiffOutput: string): SyncManifest {
|
||||
if (parts.length < 2) continue;
|
||||
|
||||
const action = parts[0];
|
||||
const path = parts[parts.length === 3 ? 2 : 1]; // For renames, new path is 3rd column
|
||||
|
||||
if (action === 'A') {
|
||||
manifest.added.push(path);
|
||||
} else if (action === 'M') {
|
||||
manifest.modified.push(path);
|
||||
manifest.added.push(parts[1]);
|
||||
} else if (action === 'M' || action === 'T' || action === 'U') {
|
||||
// T (typechange) and U (unmerged) both mean "this path exists and its
|
||||
// content is not what we imported" — the same remedy as M.
|
||||
manifest.modified.push(parts[1]);
|
||||
} else if (action === 'D') {
|
||||
manifest.deleted.push(parts[1]);
|
||||
} else if (action.startsWith('R')) {
|
||||
// Rename: R100\told-path\tnew-path
|
||||
} else if (action.startsWith('R') || action.startsWith('C')) {
|
||||
// Rename/copy: R100\told-path\tnew-path. Copy is unreachable without -C,
|
||||
// but if the flags ever change, the destination must still be imported.
|
||||
const oldPath = parts[1];
|
||||
const newPath = parts[2];
|
||||
if (oldPath && newPath) {
|
||||
manifest.renamed.push({ from: oldPath, to: newPath });
|
||||
if (action.startsWith('C')) {
|
||||
// A copy leaves the source in place — only the destination is new.
|
||||
manifest.added.push(newPath);
|
||||
} else {
|
||||
manifest.renamed.push({ from: oldPath, to: newPath });
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# gbrain agent workspace — template
|
||||
|
||||
<!-- gbrain-template-stamp: 0.45.3.0 -->
|
||||
<!-- gbrain-template-stamp: 0.45.5.0 -->
|
||||
|
||||
This repository is the **"Use this template"** distribution artifact for a
|
||||
[gbrain](https://github.com/garrytan/gbrain) personal-agent workspace — the same
|
||||
|
||||
@@ -0,0 +1,207 @@
|
||||
/**
|
||||
* Agent-scheduler contract — the documented shell-chain surface, pinned.
|
||||
*
|
||||
* SCOPE, stated honestly: the downstream agent platforms documented at
|
||||
* docs/guides/live-sync.md:62-81 (the cron-registration examples) have zero
|
||||
* detection code in gbrain and no runner in this repo. Their ENTIRE
|
||||
* integration is an external cron that shells out to the gbrain CLI and reads
|
||||
* exit codes — "/cron add ... gbrain sync --repo X && gbrain embed --stale"
|
||||
* (live-sync.md) and INSTALL_FOR_AGENTS.md Step 7. This file pins that
|
||||
* consumer contract through a REAL /bin/sh, because the `&&` short-circuit IS
|
||||
* the contract; argv arrays cannot exercise it.
|
||||
*
|
||||
* (Naming note: the specific downstream platform names live in the docs, which
|
||||
* carry their own scrub policy; scripts/check-test-real-names.sh bans agent
|
||||
* fork names inside test fixtures, so this file uses the generic term.)
|
||||
*
|
||||
* Anti-vacuity: the fixture commits a real page and every read-back asserts
|
||||
* pages >= 1 — an empty repo would satisfy every other assertion here.
|
||||
*
|
||||
* Explicitly OUT of scope (same docs, separate surfaces): `gbrain
|
||||
* check-update --json` (network), and the nightly keyless `gbrain dream`
|
||||
* contract — the cycle's embed phase has the same EmbeddingDisabledError
|
||||
* class risk this file's chain case pins for embed; filed as a TODO rather
|
||||
* than smoke-tested here.
|
||||
*/
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { mkdtempSync, mkdirSync, rmSync, writeFileSync, existsSync, chmodSync } from 'fs';
|
||||
import { tmpdir } from 'os';
|
||||
import { join, resolve, dirname } from 'path';
|
||||
import { spawnSync, execFileSync } from 'child_process';
|
||||
|
||||
const REPO = resolve(import.meta.dir, '..');
|
||||
const CLI = join(REPO, 'src', 'cli.ts');
|
||||
const SKIP = process.env.GBRAIN_SKIP_SUBPROCESS_TESTS === '1';
|
||||
|
||||
interface ShellResult { exitCode: number; stdout: string; stderr: string }
|
||||
|
||||
describe.skipIf(SKIP)('agent-scheduler shell-chain contract', () => {
|
||||
let home = '';
|
||||
let tmpbin = '';
|
||||
let repoDir = '';
|
||||
let cloneDir = '';
|
||||
let env: Record<string, string>;
|
||||
|
||||
/** Run a string through a REAL POSIX shell — the && is the thing under test. */
|
||||
function sh(command: string, timeoutMs: number, extraEnv: Record<string, string> = {}): ShellResult {
|
||||
const res = spawnSync('/bin/sh', ['-c', command], {
|
||||
cwd: REPO, // near-repo cwd keeps Bun's transpile cache warm for the shim
|
||||
env: { ...env, ...extraEnv },
|
||||
encoding: 'utf8',
|
||||
timeout: timeoutMs,
|
||||
});
|
||||
return { exitCode: res.status ?? -1, stdout: res.stdout ?? '', stderr: res.stderr ?? '' };
|
||||
}
|
||||
|
||||
function statusSyncSection(): { last_sync_at: string | null; staleness_class: string; pages: number } {
|
||||
const r = sh('gbrain status --json --section sync', 90_000);
|
||||
expect(r.exitCode).toBe(0);
|
||||
const report = JSON.parse(r.stdout.trim().split('\n').pop()!);
|
||||
const src = report.sync.sources.find((s: { source_id: string }) => s.source_id === 'default')
|
||||
?? report.sync.sources[0];
|
||||
expect(src).toBeDefined();
|
||||
return { last_sync_at: src.last_sync_at, staleness_class: src.staleness_class, pages: src.pages };
|
||||
}
|
||||
|
||||
function git(dir: string, args: string[]): void {
|
||||
execFileSync('git', ['-C', dir, ...args], {
|
||||
stdio: ['ignore', 'pipe', 'pipe'],
|
||||
timeout: 30_000,
|
||||
env: {
|
||||
...env,
|
||||
GIT_AUTHOR_DATE: '2026-08-01T00:00:00Z',
|
||||
GIT_COMMITTER_DATE: '2026-08-01T00:00:00Z',
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
beforeAll(() => {
|
||||
home = mkdtempSync(join(tmpdir(), 'gb-sched-'));
|
||||
tmpbin = mkdtempSync(join(tmpdir(), 'gb-sched-bin-'));
|
||||
repoDir = join(home, 'brain');
|
||||
cloneDir = join(home, 'brain-clone');
|
||||
|
||||
// The shim is what makes `gbrain` a real command for /bin/sh.
|
||||
writeFileSync(join(tmpbin, 'gbrain'), `#!/bin/sh\nexec bun run '${CLI}' "$@"\n`, { mode: 0o755 });
|
||||
chmodSync(join(tmpbin, 'gbrain'), 0o755);
|
||||
|
||||
const bunDir = dirname(process.execPath || '/usr/local/bin');
|
||||
env = {
|
||||
PATH: `${tmpbin}:${bunDir}:/usr/bin:/bin:/usr/sbin:/sbin`,
|
||||
HOME: home,
|
||||
GBRAIN_HOME: home,
|
||||
TMPDIR: tmpdir(),
|
||||
// Repo paths cross the shell boundary via env, never string interpolation.
|
||||
SCHED_REPO: repoDir,
|
||||
SCHED_CLONE: cloneDir,
|
||||
};
|
||||
|
||||
// Keyless brain — the install shape whose chain used to exit 1.
|
||||
mkdirSync(join(home, '.gbrain'), { recursive: true });
|
||||
writeFileSync(
|
||||
join(home, '.gbrain', 'config.json'),
|
||||
JSON.stringify({
|
||||
engine: 'pglite',
|
||||
database_path: join(home, '.gbrain', 'brain.pglite'),
|
||||
embedding_disabled: true,
|
||||
}) + '\n',
|
||||
);
|
||||
const init = spawnSync('bun', ['run', CLI, 'init', '--migrate-only'], {
|
||||
cwd: REPO, env, encoding: 'utf8', timeout: 120_000,
|
||||
});
|
||||
if (init.status !== 0) throw new Error(`init --migrate-only failed:\n${(init.stderr ?? '').slice(-2000)}`);
|
||||
|
||||
// Fixture repo with ONE REAL PAGE (anti-vacuity) + repo-local identity
|
||||
// (HOME=tmp has no global gitconfig — commits fail outright without it).
|
||||
mkdirSync(join(repoDir, 'people'), { recursive: true });
|
||||
git(repoDir, ['init', '-q', '-b', 'main']);
|
||||
git(repoDir, ['config', 'user.email', 'test@example.com']);
|
||||
git(repoDir, ['config', 'user.name', 'Sched Test']);
|
||||
writeFileSync(join(repoDir, 'people', 'alice-example.md'), '# Alice Example\n\nA real page so nothing here passes vacuously.\n');
|
||||
git(repoDir, ['add', '-A']);
|
||||
git(repoDir, ['commit', '-q', '-m', 'seed page']);
|
||||
}, 180_000);
|
||||
|
||||
afterAll(() => {
|
||||
rmSync(home, { recursive: true, force: true });
|
||||
rmSync(tmpbin, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
let firstSyncAt: string | null = null;
|
||||
|
||||
test('1. the documented chain exits 0 on a keyless brain, with the keyless hint on stderr', () => {
|
||||
// Verified against unfixed embed.ts during implementation: this chain
|
||||
// exited 1 (EmbeddingDisabledError -> blanket process.exit(1)), breaking
|
||||
// the documented agent-scheduler cron on every keyless brain. The fix is
|
||||
// load-bearing for this exact spelling.
|
||||
const r = sh('gbrain sync --repo "$SCHED_REPO" --no-pull && gbrain embed --stale', 150_000);
|
||||
expect(r.exitCode).toBe(0);
|
||||
expect(r.stderr).toContain('Embeddings are disabled');
|
||||
|
||||
const s = statusSyncSection();
|
||||
expect(s.pages).toBeGreaterThanOrEqual(1); // anti-vacuity: the page landed
|
||||
expect(s.staleness_class).toBe('fresh');
|
||||
firstSyncAt = s.last_sync_at;
|
||||
expect(firstSyncAt).not.toBeNull();
|
||||
}, 300_000);
|
||||
|
||||
test('2. quiet re-run: sync itself reports up_to_date; heartbeat strictly advances', async () => {
|
||||
// Cross-second so the strict > below cannot alias on same-second stamps.
|
||||
await new Promise((r) => setTimeout(r, 1_100));
|
||||
const r = sh('gbrain sync --repo "$SCHED_REPO" --no-pull && gbrain embed --stale', 150_000);
|
||||
expect(r.exitCode).toBe(0);
|
||||
// The run's OWN verdict, not a timestamp tautology: a crashed second run
|
||||
// would satisfy t2 >= t1; it cannot print "Already up to date."
|
||||
expect(r.stdout).toContain('Already up to date');
|
||||
|
||||
const s = statusSyncSection();
|
||||
expect(s.staleness_class).toBe('fresh');
|
||||
// v0.42.52.0 heartbeat: a 0-change sync still bumps last_sync_at.
|
||||
expect(Date.parse(s.last_sync_at!)).toBeGreaterThan(Date.parse(firstSyncAt!));
|
||||
}, 300_000);
|
||||
|
||||
test('3. the documented weekly chain (doctor --json && embed --stale) exits 0 keyless', () => {
|
||||
// Runs BEFORE the poisoned-clone case: that one appends real rows to this
|
||||
// brain's sync-failures ledger, which doctor reads and may warn on.
|
||||
const r = sh('gbrain doctor --json && gbrain embed --stale', 240_000);
|
||||
if (r.exitCode !== 0) {
|
||||
// A keyless-first-class bug in its own right — surface the failing checks.
|
||||
const doc = (() => { try { return JSON.parse(r.stdout.trim().split('\n').pop()!); } catch { return null; } })();
|
||||
const fails = doc?.checks?.filter((c: { status: string }) => c.status === 'fail') ?? [];
|
||||
throw new Error(`keyless weekly chain exited ${r.exitCode}; failing checks: ${JSON.stringify(fails.map((f: { name: string; message: string }) => `${f.name}: ${f.message}`), null, 2)}`);
|
||||
}
|
||||
expect(r.exitCode).toBe(0);
|
||||
}, 330_000);
|
||||
|
||||
test('4. pull failure breaks the chain: sync exits 1, embed never runs', () => {
|
||||
// A clone whose origin is a local filesystem path: pullRepo's SSRF flag
|
||||
// (protocol.file.allow=never) rejects it deterministically on every pull.
|
||||
execFileSync('git', ['clone', '-q', '--no-hardlinks', repoDir, cloneDir], {
|
||||
env: { ...env, GIT_ALLOW_PROTOCOL: 'file' },
|
||||
stdio: ['ignore', 'pipe', 'pipe'],
|
||||
timeout: 60_000,
|
||||
});
|
||||
git(cloneDir, ['config', 'user.email', 'test@example.com']);
|
||||
git(cloneDir, ['config', 'user.name', 'Sched Test']);
|
||||
|
||||
// Because cases 1-2 already synced the SAME commit for source `default`,
|
||||
// the clone's HEAD equals the stored anchor — so this run is ALREADY the
|
||||
// quiet case: zero imports + pullFailed (protocol.file rejected) →
|
||||
// partial/pull_failed → exit 1 (sync.ts:4994, the #3068 contract), and the
|
||||
// && short-circuits: the keyless embed hint must NOT appear. (On a FRESH
|
||||
// source the first run imports the working tree and exits 0
|
||||
// warn-and-continue; it takes a second, quiet run to surface the wedge —
|
||||
// both shapes converge on "a permanently failing pull cannot stay green".)
|
||||
const run1 = sh('gbrain sync --repo "$SCHED_CLONE" --source default && gbrain embed --stale', 150_000);
|
||||
if (run1.exitCode !== 1) {
|
||||
throw new Error(`expected pull_failed exit 1, got ${run1.exitCode}\nstderr:\n${run1.stderr.slice(-2500)}\nstdout:\n${run1.stdout.slice(-800)}`);
|
||||
}
|
||||
expect(run1.exitCode).toBe(1);
|
||||
expect(run1.stderr).not.toContain('Embeddings are disabled');
|
||||
|
||||
// And it stays red on repeat — a wedged pull is not a one-off blip.
|
||||
const run2 = sh('gbrain sync --repo "$SCHED_CLONE" --source default && gbrain embed --stale', 150_000);
|
||||
expect(run2.exitCode).toBe(1);
|
||||
expect(run2.stderr).not.toContain('Embeddings are disabled');
|
||||
}, 450_000);
|
||||
});
|
||||
@@ -137,8 +137,14 @@ describe('autopilot wrapper script — bun PATH export (v0.42.x regression)', ()
|
||||
// that installed via the wrapper indirection.
|
||||
describe('autopilot showStatus — wrapper-path detection', () => {
|
||||
test('status detects the autopilot-run.sh wrapper line', async () => {
|
||||
// The inline crontab.includes check became the pure, unit-tested
|
||||
// crontabIndicatesAutopilotInstall (a cron'd status-monitor line must not
|
||||
// read as an install). Pin the behavior through the function itself and
|
||||
// the detect path's wiring to it.
|
||||
const { crontabIndicatesAutopilotInstall } = await import('../src/commands/autopilot.ts');
|
||||
expect(crontabIndicatesAutopilotInstall("*/5 * * * * '/h/.gbrain/autopilot-run.sh' >> log 2>&1")).toBe(true);
|
||||
const { readFileSync } = await import('fs');
|
||||
const src = readFileSync('src/commands/autopilot.ts', 'utf8');
|
||||
expect(src).toMatch(/crontab\.includes\('autopilot-run\.sh'\)/);
|
||||
expect(src).toMatch(/crontabIndicatesAutopilotInstall\(crontab\)/);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,387 @@
|
||||
/**
|
||||
* Autopilot launchd lifecycle — the e2e that closes the artifact-presence gap.
|
||||
*
|
||||
* PR-A rewrote `autopilot --status` because the old one answered "installed"
|
||||
* from a bare plist existsSync while the daemon had been dead for 71 days. The
|
||||
* unit tests for the fix assert the STRINGS we generate (guard text, plist
|
||||
* XML) — the same anti-pattern one level up. This file asserts BEHAVIOR:
|
||||
*
|
||||
* Describe A (every platform, incl. ubuntu CI): the full install →
|
||||
* self-disable → status → reinstall → uninstall arc, with `launchctl`
|
||||
* replaced by an argv recorder, and the generated wrapper executed by a
|
||||
* REAL bash against a genuinely deleted repo.
|
||||
*
|
||||
* Describe B (darwin only, fail-SKIP): the same self-disable against the
|
||||
* REAL launchd — job provably loaded (RunAtLoad ran our wrapper),
|
||||
* `launchctl kickstart -k` forces the guard after the repo vanishes, and
|
||||
* the job is provably gone afterwards. Non-vacuous by construction: each
|
||||
* link is asserted separately, so "job absent" can't be true for the
|
||||
* wrong reason.
|
||||
*
|
||||
* Label safety: every spawn sets GBRAIN_AUTOPILOT_LABEL to a unique test
|
||||
* label, so this can never collide with — or tear down — a real
|
||||
* com.gbrain.autopilot install on the machine running the tests. Cleanup is
|
||||
* OWN-LABEL ONLY: a prefix sweep would boot out a concurrent Conductor
|
||||
* workspace's live test run on the same Mac.
|
||||
*
|
||||
* Known real-machine flake vector (describe B): the launchd-spawned wrapper
|
||||
* runs with the REAL user's HOME and sources their zshenv/zshrc before the
|
||||
* guard (launchd does not inherit our test env). A dotfile that execs another
|
||||
* shell or stalls can eat the wrapper. The marker path and label are baked as
|
||||
* absolute strings at install time so they cannot be misrouted; if the poll
|
||||
* times out, the failure dump names this vector. Escape hatch:
|
||||
* GBRAIN_SKIP_LAUNCHD_E2E=1.
|
||||
*/
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { mkdtempSync, mkdirSync, rmSync, writeFileSync, readFileSync, existsSync, chmodSync } from 'fs';
|
||||
import { tmpdir } from 'os';
|
||||
import { join, resolve, dirname } from 'path';
|
||||
import { spawnSync } from 'child_process';
|
||||
import { withEnv } from './helpers/with-env.ts';
|
||||
import { autopilotLaunchdLabel } from '../src/core/autopilot-paths.ts';
|
||||
|
||||
const REPO = resolve(import.meta.dir, '..');
|
||||
const CLI = join(REPO, 'src', 'cli.ts');
|
||||
const SKIP_SUBPROCESS = process.env.GBRAIN_SKIP_SUBPROCESS_TESTS === '1';
|
||||
|
||||
// ── label seam unit tests (serial file on purpose: env mutation would
|
||||
// interleave with other autopilot tests in the parallel lane) ─────────────
|
||||
|
||||
describe('autopilotLaunchdLabel', () => {
|
||||
test('default label, env unset', async () => {
|
||||
await withEnv({ GBRAIN_AUTOPILOT_LABEL: undefined }, async () => {
|
||||
expect(autopilotLaunchdLabel()).toBe('com.gbrain.autopilot');
|
||||
});
|
||||
});
|
||||
|
||||
test('override lands in the generated guard bootout line', async () => {
|
||||
await withEnv({ GBRAIN_AUTOPILOT_LABEL: 'com.gbrain.autopilot.test.override' }, async () => {
|
||||
const { generateSelfDisableGuard } = await import('../src/commands/autopilot.ts');
|
||||
const guard = generateSelfDisableGuard('/data/brain', 'macos');
|
||||
expect(guard).toContain('gui/$(id -u)/com.gbrain.autopilot.test.override');
|
||||
});
|
||||
});
|
||||
|
||||
test('grammar: shell/filesystem metacharacters are rejected', async () => {
|
||||
// The label reaches a plist FILENAME and a double-quoted shell line;
|
||||
// escapeXml protects only the XML site.
|
||||
for (const evil of ['a/b', 'a b', 'a"b', "a'b", 'a$b', 'a`b', 'a..b', '']) {
|
||||
await withEnv({ GBRAIN_AUTOPILOT_LABEL: evil }, async () => {
|
||||
expect(() => autopilotLaunchdLabel()).toThrow(/invalid GBRAIN_AUTOPILOT_LABEL/);
|
||||
});
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
// ── keyless explicit-slug contract (spawn-free; the deliberate exit-1 half
|
||||
// of the embed keyless fix) ───────────────────────────────────────────────
|
||||
|
||||
describe('embed keyless: explicit requests still fail loudly', () => {
|
||||
test('runEmbedCore throws EmbeddingDisabledError for a named slug on a keyless brain', async () => {
|
||||
// The --stale path refuses cleanly (exit 0) because the documented agent
|
||||
// chain depends on it. A user who NAMED a page asked for something
|
||||
// impossible on a keyless brain — that must stay a structured failure.
|
||||
const { assertEmbeddingEnabled, EmbeddingDisabledError } = await import('../src/core/embedding-dim-check.ts');
|
||||
expect(() => assertEmbeddingEnabled({ embedding_disabled: true } as never))
|
||||
.toThrow(EmbeddingDisabledError);
|
||||
});
|
||||
});
|
||||
|
||||
// ── shared subprocess helpers ────────────────────────────────────────────────
|
||||
|
||||
interface CliResult { exitCode: number; stdout: string; stderr: string }
|
||||
|
||||
function runCli(args: string[], env: Record<string, string>, timeoutMs: number): CliResult {
|
||||
const res = spawnSync('bun', ['run', CLI, ...args], {
|
||||
cwd: REPO, // near-repo cwd keeps Bun's transpile cache warm (5-20s/spawn otherwise)
|
||||
env,
|
||||
encoding: 'utf8',
|
||||
timeout: timeoutMs,
|
||||
});
|
||||
return { exitCode: res.status ?? -1, stdout: res.stdout ?? '', stderr: res.stderr ?? '' };
|
||||
}
|
||||
|
||||
/** Minimal PATH: tmpbin first, then the running bun's dir, then the OS basics. */
|
||||
function basePath(tmpbin: string): string {
|
||||
const bunDir = dirname(process.execPath || '/usr/local/bin');
|
||||
return `${tmpbin}:${bunDir}:/usr/bin:/bin:/usr/sbin:/sbin`;
|
||||
}
|
||||
|
||||
/** Write the keyless PGLite config by hand (fast path; skips init's provider dance). */
|
||||
function seedBrain(home: string, env: Record<string, string>): void {
|
||||
mkdirSync(join(home, '.gbrain'), { recursive: true });
|
||||
writeFileSync(
|
||||
join(home, '.gbrain', 'config.json'),
|
||||
JSON.stringify({
|
||||
engine: 'pglite',
|
||||
database_path: join(home, '.gbrain', 'brain.pglite'),
|
||||
embedding_disabled: true,
|
||||
}) + '\n',
|
||||
);
|
||||
const r = runCli(['init', '--migrate-only'], env, 120_000);
|
||||
if (r.exitCode !== 0) {
|
||||
throw new Error(`seedBrain init --migrate-only failed (${r.exitCode}):\n${r.stderr.slice(-2000)}`);
|
||||
}
|
||||
}
|
||||
|
||||
// ── Describe A: shimmed lifecycle (all platforms) ───────────────────────────
|
||||
|
||||
describe.skipIf(SKIP_SUBPROCESS)('autopilot launchd lifecycle — shimmed (all platforms)', () => {
|
||||
const label = `com.gbrain.autopilot.test.${process.pid}`;
|
||||
let home = '';
|
||||
let tmpbin = '';
|
||||
let repoDir = '';
|
||||
let recordFile = '';
|
||||
let breadcrumbFile = '';
|
||||
let env: Record<string, string>;
|
||||
|
||||
const plist = () => join(home, 'Library', 'LaunchAgents', `${label}.plist`);
|
||||
const wrapper = () => join(home, '.gbrain', 'autopilot-run.sh');
|
||||
const marker = () => join(home, '.gbrain', 'autopilot-disabled');
|
||||
const recorded = () => (existsSync(recordFile) ? readFileSync(recordFile, 'utf8') : '');
|
||||
|
||||
beforeAll(() => {
|
||||
home = mkdtempSync(join(tmpdir(), 'gb-launchd-a-'));
|
||||
tmpbin = mkdtempSync(join(tmpdir(), 'gb-launchd-bin-'));
|
||||
repoDir = join(home, 'brain-repo');
|
||||
mkdirSync(repoDir, { recursive: true });
|
||||
recordFile = join(tmpbin, 'record.log');
|
||||
breadcrumbFile = join(tmpbin, 'execed.log');
|
||||
|
||||
// launchctl argv recorder — exit 0 so install's execSync succeeds.
|
||||
writeFileSync(join(tmpbin, 'launchctl'), `#!/bin/sh\necho "launchctl $*" >> '${recordFile}'\nexit 0\n`, { mode: 0o755 });
|
||||
// crontab no-op — detectInstalledTarget falls through to the REAL
|
||||
// `crontab -l` on non-darwin; a dev's real gbrain cron entry would flip
|
||||
// the verdict to linux-cron.
|
||||
writeFileSync(join(tmpbin, 'crontab'), `#!/bin/sh\nexit 0\n`, { mode: 0o755 });
|
||||
// gbrain breadcrumb — `which gbrain` at install time bakes THIS into the
|
||||
// wrapper's exec line. Records context, exits 0. NOTE: asserts "a process
|
||||
// execed gbrain from PATH", not "the daemon started" — assert on context.
|
||||
writeFileSync(join(tmpbin, 'gbrain'), `#!/bin/sh\necho "gbrain $*" >> '${breadcrumbFile}'\nexit 0\n`, { mode: 0o755 });
|
||||
for (const f of ['launchctl', 'crontab', 'gbrain']) chmodSync(join(tmpbin, f), 0o755);
|
||||
|
||||
env = {
|
||||
PATH: basePath(tmpbin),
|
||||
HOME: home,
|
||||
GBRAIN_HOME: home,
|
||||
GBRAIN_AUTOPILOT_LABEL: label,
|
||||
TMPDIR: tmpdir(),
|
||||
};
|
||||
seedBrain(home, env);
|
||||
}, 180_000);
|
||||
|
||||
afterAll(() => {
|
||||
rmSync(home, { recursive: true, force: true });
|
||||
rmSync(tmpbin, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
test('1. virgin status → not_installed, exit 0', () => {
|
||||
const r = runCli(['autopilot', '--status', '--json'], env, 90_000);
|
||||
expect(r.exitCode).toBe(0);
|
||||
const report = JSON.parse(r.stdout.trim().split('\n').pop()!);
|
||||
expect(report.state).toBe('not_installed');
|
||||
}, 120_000);
|
||||
|
||||
test('2. install --target macos: plist + wrapper + recorded load', () => {
|
||||
const r = runCli(['autopilot', '--install', '--target', 'macos', '--repo', repoDir], env, 90_000);
|
||||
expect(r.exitCode).toBe(0);
|
||||
expect(existsSync(plist())).toBe(true);
|
||||
const xml = readFileSync(plist(), 'utf8');
|
||||
expect(xml).toContain(`<string>${label}</string>`);
|
||||
expect(recorded()).toContain(`launchctl load ${plist()}`);
|
||||
const w = readFileSync(wrapper(), 'utf8');
|
||||
// Guard precedes exec — the self-disable must run before the daemon could start.
|
||||
expect(w.indexOf('repo path gone')).toBeGreaterThan(-1);
|
||||
expect(w.indexOf('repo path gone')).toBeLessThan(w.indexOf('exec '));
|
||||
expect(w).toContain(`gui/$(id -u)/${label}`);
|
||||
}, 120_000);
|
||||
|
||||
test('3. repo deleted → three strikes, THEN marker + recorded bootout', () => {
|
||||
rmSync(repoDir, { recursive: true, force: true });
|
||||
// One transient miss must not kill the install (repos on external or
|
||||
// cloud-synced volumes are routinely absent right after login) — the
|
||||
// guard requires three consecutive misses. Runs 1 and 2 exit 0 with no
|
||||
// marker and no bootout; run 3 disables for real.
|
||||
for (const run of [1, 2]) {
|
||||
const res = spawnSync('bash', [wrapper()], { env, encoding: 'utf8', timeout: 30_000 });
|
||||
expect(res.status).toBe(0);
|
||||
expect(existsSync(marker())).toBe(false);
|
||||
expect(recorded()).not.toContain('launchctl bootout gui/');
|
||||
expect(run).toBeGreaterThan(0);
|
||||
}
|
||||
const res = spawnSync('bash', [wrapper()], { env, encoding: 'utf8', timeout: 30_000 });
|
||||
expect(res.status).toBe(0);
|
||||
expect(existsSync(marker())).toBe(true);
|
||||
expect(readFileSync(marker(), 'utf8')).toContain(repoDir);
|
||||
expect(recorded()).toContain(`launchctl bootout gui/`);
|
||||
expect(recorded()).toContain(label);
|
||||
}, 120_000);
|
||||
|
||||
test('4. status after self-disable → disabled, names the path, exit 2', () => {
|
||||
const r = runCli(['autopilot', '--status', '--json'], env, 90_000);
|
||||
expect(r.exitCode).toBe(2);
|
||||
const report = JSON.parse(r.stdout.trim().split('\n').pop()!);
|
||||
expect(report.state).toBe('disabled');
|
||||
expect(report.disabled_reason).toContain(repoDir);
|
||||
}, 120_000);
|
||||
|
||||
test('5. reinstall against a RECREATED repo clears the marker', () => {
|
||||
// Recreate first — reinstalling against the deleted path would only prove
|
||||
// marker-clearing while leaving an immediately-doomed install.
|
||||
mkdirSync(repoDir, { recursive: true });
|
||||
const r = runCli(['autopilot', '--install', '--target', 'macos', '--repo', repoDir], env, 90_000);
|
||||
expect(r.exitCode).toBe(0);
|
||||
expect(existsSync(marker())).toBe(false);
|
||||
}, 120_000);
|
||||
|
||||
test('6. uninstall: plist gone, recorded unload, markers cleared', () => {
|
||||
// Seed a disabled marker first: without the uninstall-time clear, a
|
||||
// self-disabled-then-uninstalled machine reports "DISABLED" with exit 2
|
||||
// forever ('disabled' outranks 'not_installed' in the classifier).
|
||||
writeFileSync(marker(), 'repo path gone: /somewhere\n');
|
||||
const r = runCli(['autopilot', '--uninstall'], env, 90_000);
|
||||
expect(r.exitCode).toBe(0);
|
||||
expect(existsSync(plist())).toBe(false);
|
||||
expect(recorded()).toContain(`launchctl unload ${plist()}`);
|
||||
expect(existsSync(marker())).toBe(false);
|
||||
|
||||
const status = runCli(['autopilot', '--status', '--json'], env, 90_000);
|
||||
expect(JSON.parse(status.stdout).state).toBe('not_installed');
|
||||
expect(status.exitCode).toBe(0);
|
||||
}, 240_000);
|
||||
|
||||
test('7. the daemon never started: breadcrumb has no autopilot exec', () => {
|
||||
const crumbs = existsSync(breadcrumbFile) ? readFileSync(breadcrumbFile, 'utf8') : '';
|
||||
expect(crumbs).not.toContain('autopilot --repo');
|
||||
});
|
||||
});
|
||||
|
||||
// ── Describe B: REAL launchd (darwin, fail-SKIP) ────────────────────────────
|
||||
|
||||
function canUseLaunchd(): boolean {
|
||||
if (process.platform !== 'darwin') return false;
|
||||
if (process.env.GBRAIN_SKIP_LAUNCHD_E2E === '1') return false;
|
||||
if (typeof process.getuid !== 'function') return false;
|
||||
const probe = spawnSync('launchctl', ['print', `gui/${process.getuid()}`], { encoding: 'utf8', timeout: 10_000 });
|
||||
return probe.status === 0;
|
||||
}
|
||||
|
||||
const LAUNCHD_OK = canUseLaunchd();
|
||||
if (!LAUNCHD_OK && process.env.GBRAIN_REQUIRE_LAUNCHD === '1') {
|
||||
throw new Error('[autopilot-launchd-lifecycle] GBRAIN_REQUIRE_LAUNCHD=1 but the launchd GUI domain is unavailable');
|
||||
}
|
||||
|
||||
describe.skipIf(SKIP_SUBPROCESS || !LAUNCHD_OK)('autopilot launchd lifecycle — REAL launchd (darwin)', () => {
|
||||
const uid = typeof process.getuid === 'function' ? process.getuid() : 0;
|
||||
const label = `com.gbrain.autopilot.e2e.${process.pid}.${Date.now()}`;
|
||||
let home = '';
|
||||
let tmpbin = '';
|
||||
let repoDir = '';
|
||||
let breadcrumbFile = '';
|
||||
let env: Record<string, string>;
|
||||
|
||||
const plist = () => join(home, 'Library', 'LaunchAgents', `${label}.plist`);
|
||||
const marker = () => join(home, '.gbrain', 'autopilot-disabled');
|
||||
const printJob = () => spawnSync('launchctl', ['print', `gui/${uid}/${label}`], { encoding: 'utf8', timeout: 10_000 });
|
||||
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
|
||||
|
||||
beforeAll(() => {
|
||||
home = mkdtempSync(join(tmpdir(), 'gb-launchd-b-'));
|
||||
tmpbin = mkdtempSync(join(tmpdir(), 'gb-launchd-b-bin-'));
|
||||
repoDir = join(home, 'brain-repo');
|
||||
mkdirSync(repoDir, { recursive: true });
|
||||
breadcrumbFile = join(tmpbin, 'execed.log');
|
||||
|
||||
// gbrain breadcrumb only — NO launchctl shim here: install must reach the
|
||||
// real launchd. Breadcrumb path baked ABSOLUTE: launchd will not pass our
|
||||
// test env to the wrapper it spawns.
|
||||
writeFileSync(join(tmpbin, 'gbrain'), `#!/bin/sh\necho "gbrain $*" >> '${breadcrumbFile}'\nexit 0\n`, { mode: 0o755 });
|
||||
|
||||
env = {
|
||||
PATH: basePath(tmpbin),
|
||||
HOME: home,
|
||||
GBRAIN_HOME: home,
|
||||
GBRAIN_AUTOPILOT_LABEL: label,
|
||||
TMPDIR: tmpdir(),
|
||||
};
|
||||
seedBrain(home, env);
|
||||
}, 180_000);
|
||||
|
||||
afterAll(() => {
|
||||
// Own-label only. Bootout BEFORE removing files: launchd never rescans a
|
||||
// tmp-HOME LaunchAgents dir, so a leftover job dies at logout regardless,
|
||||
// but booting first keeps the teardown quiet.
|
||||
spawnSync('launchctl', ['bootout', `gui/${uid}/${label}`], { timeout: 10_000 });
|
||||
rmSync(home, { recursive: true, force: true });
|
||||
rmSync(tmpbin, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
test('1. install loads a real job and RunAtLoad runs our wrapper', async () => {
|
||||
const r = runCli(['autopilot', '--install', '--target', 'macos', '--repo', repoDir], env, 90_000);
|
||||
expect(r.exitCode).toBe(0);
|
||||
expect(existsSync(plist())).toBe(true);
|
||||
|
||||
// Breadcrumb = launchd genuinely spawned the wrapper AND the wrapper ran
|
||||
// end-to-end to its exec line (repo exists, guard passed). This is the
|
||||
// "install verified nothing" gap actually closing.
|
||||
let sawBreadcrumb = false;
|
||||
for (let i = 0; i < 60; i++) {
|
||||
if (existsSync(breadcrumbFile) && readFileSync(breadcrumbFile, 'utf8').includes('autopilot --repo')) {
|
||||
sawBreadcrumb = true;
|
||||
break;
|
||||
}
|
||||
await sleep(500);
|
||||
}
|
||||
if (!sawBreadcrumb) {
|
||||
const p = printJob();
|
||||
throw new Error(
|
||||
`wrapper never ran under launchd within 30s.\n`
|
||||
+ `Likely: a dotfile in the REAL user zshenv/zshrc exec'd another shell or stalled `
|
||||
+ `(launchd runs the wrapper with your real HOME). Escape hatch: GBRAIN_SKIP_LAUNCHD_E2E=1.\n`
|
||||
+ `launchctl print (exit ${p.status}):\n${(p.stdout || p.stderr).slice(0, 1500)}`,
|
||||
);
|
||||
}
|
||||
expect(printJob().status).toBe(0); // job registered; KeepAlive keeps it so
|
||||
}, 180_000);
|
||||
|
||||
test('2. repo deleted → guard self-disables: marker written, job gone', async () => {
|
||||
rmSync(repoDir, { recursive: true, force: true });
|
||||
// The guard needs three consecutive misses before disabling; under real
|
||||
// launchd that is three ThrottleInterval windows (~3min). Pre-seed two
|
||||
// strikes so the NEXT respawned run crosses the threshold — this still
|
||||
// exercises the real counter read + threshold + bootout path while
|
||||
// keeping the poll inside one throttle window.
|
||||
writeFileSync(join(home, '.gbrain', 'autopilot-disable-strikes'), '2\n');
|
||||
// Best-effort accelerant only. Observed live: kickstart BLOCKS (not
|
||||
// errors) when the job is inside its ThrottleInterval window after the
|
||||
// shim's quick exits, so we must not assert its exit code or depend on
|
||||
// it at all. KeepAlive + ThrottleInterval=60 guarantees launchd respawns
|
||||
// the wrapper naturally within one throttle window, and the respawned
|
||||
// wrapper hits the guard. The poll therefore spans >60s.
|
||||
spawnSync('launchctl', ['kickstart', '-k', `gui/${uid}/${label}`], { encoding: 'utf8', timeout: 10_000 });
|
||||
|
||||
let ok = false;
|
||||
for (let i = 0; i < 180; i++) { // 90s: one full throttle window + slack
|
||||
if (existsSync(marker()) && printJob().status !== 0) { ok = true; break; }
|
||||
await sleep(500);
|
||||
}
|
||||
if (!ok) {
|
||||
const p = printJob();
|
||||
const log = join(home, '.gbrain', 'autopilot.log');
|
||||
const err = join(home, '.gbrain', 'autopilot.err');
|
||||
throw new Error(
|
||||
`self-disable did not converge in 90s: marker=${existsSync(marker())} printExit=${p.status}\n`
|
||||
+ `autopilot.log:\n${existsSync(log) ? readFileSync(log, 'utf8').slice(-1200) : '(absent)'}\n`
|
||||
+ `autopilot.err:\n${existsSync(err) ? readFileSync(err, 'utf8').slice(-1200) : '(absent)'}`,
|
||||
);
|
||||
}
|
||||
expect(readFileSync(marker(), 'utf8')).toContain(repoDir);
|
||||
}, 240_000);
|
||||
|
||||
test('3. status reports disabled with exit 2', () => {
|
||||
const r = runCli(['autopilot', '--status', '--json'], env, 90_000);
|
||||
expect(r.exitCode).toBe(2);
|
||||
const report = JSON.parse(r.stdout.trim().split('\n').pop()!);
|
||||
expect(report.state).toBe('disabled');
|
||||
}, 120_000);
|
||||
});
|
||||
@@ -0,0 +1,245 @@
|
||||
/**
|
||||
* PR-A / A3: the cooperative pause marker that lets `gbrain migrate --to <engine>`
|
||||
* quiesce a running daemon for the duration of a cross-engine copy.
|
||||
*
|
||||
* The incident this closes: `migrate-engine.ts` contained ZERO references to
|
||||
* autopilot, so an engine migration rewrote `config.json` while a live daemon
|
||||
* kept writing into the SOURCE engine. Those writes were lost, and the daemon
|
||||
* then died on a stale config object with no operator-visible link between the
|
||||
* two events.
|
||||
*
|
||||
* EVERY test here runs against a scratch GBRAIN_HOME. The paths resolve through
|
||||
* gbrainHomePath() at call time, so without the override this file would write
|
||||
* and — worse — unconditionally DELETE markers in the developer's real
|
||||
* ~/.gbrain, un-quiescing a live migration on the machine running the tests.
|
||||
*/
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { existsSync, writeFileSync, unlinkSync, mkdirSync, readFileSync, utimesSync } from 'fs';
|
||||
import { spawnSync } from 'child_process';
|
||||
import { dirname } from 'path';
|
||||
import { withEnv, emptyHome } from './helpers/with-env.ts';
|
||||
import {
|
||||
autopilotPausedMarkerPath,
|
||||
autopilotDisabledMarkerPath,
|
||||
autopilotLockPath,
|
||||
} from '../src/core/autopilot-paths.ts';
|
||||
import { quiesceAutopilot, MIGRATE_PAUSE_MARKER_PREFIX } from '../src/commands/migrate-engine.ts';
|
||||
|
||||
/** Scratch home + no grace sleep: the default 35s would stall the suite. */
|
||||
function quiesceEnv(extra: Record<string, string | undefined> = {}) {
|
||||
return { GBRAIN_HOME: emptyHome(), GBRAIN_MIGRATE_QUIESCE_SECONDS: '0', ...extra };
|
||||
}
|
||||
|
||||
function touchLock(mtime?: Date) {
|
||||
const lock = autopilotLockPath();
|
||||
mkdirSync(dirname(lock), { recursive: true });
|
||||
writeFileSync(lock, `${process.pid}\n`);
|
||||
if (mtime) utimesSync(lock, mtime, mtime);
|
||||
}
|
||||
|
||||
describe('autopilot pause marker (A3 quiesce)', () => {
|
||||
test('resolves inside the gbrain home, not raw $HOME', async () => {
|
||||
await withEnv({ GBRAIN_HOME: emptyHome() }, () => {
|
||||
// The daemon writes its lock through gbrainHomePath(); a marker resolved
|
||||
// from raw process.env.HOME would land in a different directory under
|
||||
// GBRAIN_HOME and the pause would silently never be observed.
|
||||
const p = autopilotPausedMarkerPath();
|
||||
expect(p.endsWith('autopilot-paused')).toBe(true);
|
||||
expect(p).toContain(process.env.GBRAIN_HOME!);
|
||||
});
|
||||
});
|
||||
|
||||
test('pause and disabled markers are distinct files', async () => {
|
||||
await withEnv({ GBRAIN_HOME: emptyHome() }, () => {
|
||||
// They mean different things: paused = "a migration is running, hold";
|
||||
// disabled = "your repo is gone, I stopped myself". Collapsing them would
|
||||
// make a migration look like a fault in `--status`.
|
||||
expect(autopilotPausedMarkerPath()).not.toBe(autopilotDisabledMarkerPath());
|
||||
});
|
||||
});
|
||||
|
||||
test('presence is the whole protocol — create then remove', async () => {
|
||||
await withEnv({ GBRAIN_HOME: emptyHome() }, () => {
|
||||
const p = autopilotPausedMarkerPath();
|
||||
mkdirSync(dirname(p), { recursive: true });
|
||||
expect(existsSync(p)).toBe(false);
|
||||
writeFileSync(p, `${MIGRATE_PAUSE_MARKER_PREFIX} at 2026-08-11T00:00:00.000Z\n`);
|
||||
expect(existsSync(p)).toBe(true);
|
||||
unlinkSync(p);
|
||||
expect(existsSync(p)).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
test('marker content is human-readable, so an orphan explains itself', () => {
|
||||
// If a migration crashes hard the marker survives. Whoever finds it must be
|
||||
// able to tell what wrote it and when, rather than guessing why autopilot
|
||||
// stopped doing work.
|
||||
const body = `${MIGRATE_PAUSE_MARKER_PREFIX} at ${new Date(0).toISOString()}\n`;
|
||||
expect(body).toContain('gbrain migrate');
|
||||
expect(body).toMatch(/\d{4}-\d{2}-\d{2}T/);
|
||||
});
|
||||
});
|
||||
|
||||
describe('quiesceAutopilot (migrate-side of the protocol)', () => {
|
||||
test('no daemon lock → marker STILL written (red-team: the no-lock window is routine)', async () => {
|
||||
await withEnv(quiesceEnv(), async () => {
|
||||
// The daemon unlinks its lock on every clean exit and the supervisor
|
||||
// relaunches it within a minute; cron installs have no lock between
|
||||
// runs. A migrate that skipped the marker in such a window would let a
|
||||
// daemon that STARTS mid-copy write into the source engine — all lost
|
||||
// at the flip. So the marker is unconditional; only the grace sleep
|
||||
// keys on a live heartbeat.
|
||||
const resume = await quiesceAutopilot();
|
||||
expect(resume).not.toBeNull();
|
||||
expect(existsSync(autopilotPausedMarkerPath())).toBe(true);
|
||||
resume!();
|
||||
expect(existsSync(autopilotPausedMarkerPath())).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
test('live lock → marker written with owned prefix + pid; resume removes it, twice-safe', async () => {
|
||||
await withEnv(quiesceEnv(), async () => {
|
||||
touchLock();
|
||||
const resume = await quiesceAutopilot();
|
||||
expect(resume).not.toBeNull();
|
||||
const marker = autopilotPausedMarkerPath();
|
||||
expect(existsSync(marker)).toBe(true);
|
||||
const body = readFileSync(marker, 'utf8');
|
||||
expect(body.startsWith(MIGRATE_PAUSE_MARKER_PREFIX)).toBe(true);
|
||||
expect(body).toContain(`(pid ${process.pid})`);
|
||||
resume!();
|
||||
expect(existsSync(marker)).toBe(false);
|
||||
resume!(); // idempotent — the finally + cleanup backstop may both fire
|
||||
expect(existsSync(marker)).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
test('foreign marker → null (caller must ABORT), and the marker is untouched', async () => {
|
||||
await withEnv(quiesceEnv(), async () => {
|
||||
touchLock();
|
||||
const marker = autopilotPausedMarkerPath();
|
||||
mkdirSync(dirname(marker), { recursive: true });
|
||||
writeFileSync(marker, 'paused by an operator for maintenance\n');
|
||||
// Codex catch: the old contract returned a no-op resume and let the
|
||||
// migration proceed UNOWNED — when the real owner finished, its cleanup
|
||||
// un-paused the daemon in the middle of our copy window. Null forces
|
||||
// the caller to refuse to run instead.
|
||||
expect(await quiesceAutopilot()).toBeNull();
|
||||
expect(existsSync(marker)).toBe(true);
|
||||
expect(readFileSync(marker, 'utf8')).toContain('operator');
|
||||
});
|
||||
});
|
||||
|
||||
test('adopts a DEAD migrate\'s orphan: the marker does not become permanent', async () => {
|
||||
await withEnv(quiesceEnv(), async () => {
|
||||
touchLock();
|
||||
const marker = autopilotPausedMarkerPath();
|
||||
mkdirSync(dirname(marker), { recursive: true });
|
||||
// A prior migrate wrote this and died before its finally could run.
|
||||
// Refusing to adopt would park the daemon forever: every retry would
|
||||
// defer to the "foreign" marker, and nothing else ever deletes it.
|
||||
// A real reaped pid, so the liveness probe sees ESRCH.
|
||||
const dead = spawnSync('true');
|
||||
writeFileSync(marker, `${MIGRATE_PAUSE_MARKER_PREFIX} (pid ${dead.pid}) at ${new Date(0).toISOString()}\n`);
|
||||
const resume = await quiesceAutopilot();
|
||||
expect(resume).not.toBeNull();
|
||||
expect(existsSync(marker)).toBe(true); // still paused during the copy
|
||||
// Ownership is TAKEN, not shared: the body now carries OUR pid, so a
|
||||
// third migrate reads a live holder and aborts instead of also adopting
|
||||
// (double-adoption would un-pause the daemon mid-copy).
|
||||
expect(readFileSync(marker, 'utf8')).toContain(`(pid ${process.pid})`);
|
||||
resume!();
|
||||
expect(existsSync(marker)).toBe(false); // and released after
|
||||
});
|
||||
});
|
||||
|
||||
test('does NOT adopt a LIVE migrate\'s marker — resume must not un-pause a concurrent copy', async () => {
|
||||
await withEnv(quiesceEnv(), async () => {
|
||||
touchLock();
|
||||
const marker = autopilotPausedMarkerPath();
|
||||
mkdirSync(dirname(marker), { recursive: true });
|
||||
// Same prefix as ours, but the holder pid is alive (this very process).
|
||||
// Red-team catch: prefix-only adoption let migrate B adopt migrate A's
|
||||
// live marker; B's finally then deleted it mid-copy and the daemon
|
||||
// resumed inside A's copy window — the exact incident this closes.
|
||||
writeFileSync(marker, `${MIGRATE_PAUSE_MARKER_PREFIX} (pid ${process.pid}) at ${new Date(0).toISOString()}\n`);
|
||||
expect(await quiesceAutopilot()).toBeNull();
|
||||
expect(existsSync(marker)).toBe(true); // untouched — the live holder owns it
|
||||
});
|
||||
});
|
||||
|
||||
test('a prefix-matching marker WITHOUT a pid is treated as foreign, not adopted', async () => {
|
||||
await withEnv(quiesceEnv(), async () => {
|
||||
touchLock();
|
||||
const marker = autopilotPausedMarkerPath();
|
||||
mkdirSync(dirname(marker), { recursive: true });
|
||||
// No ownership handle → cannot prove the holder is dead → defer.
|
||||
writeFileSync(marker, `${MIGRATE_PAUSE_MARKER_PREFIX} at ${new Date(0).toISOString()}\n`);
|
||||
expect(await quiesceAutopilot()).toBeNull();
|
||||
expect(existsSync(marker)).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
test('garbage grace env falls back with a warning instead of silently disabling', async () => {
|
||||
await withEnv(quiesceEnv({ GBRAIN_MIGRATE_QUIESCE_SECONDS: 'garbage' }), async () => {
|
||||
// Stale lock (aged past the heartbeat window) so no code path can sleep:
|
||||
// the marker still gets written — quiesce keys on lock EXISTENCE — but
|
||||
// the grace is skipped, letting this assert the NaN warning without a
|
||||
// 35s stall. Pre-fix, Number('garbage') = NaN made `NaN > 0` false and
|
||||
// the grace vanished with no signal at all.
|
||||
touchLock(new Date(Date.now() - 3600_000));
|
||||
const warns: string[] = [];
|
||||
const origWarn = console.warn;
|
||||
console.warn = (...a: unknown[]) => { warns.push(a.join(' ')); };
|
||||
try {
|
||||
const resume = await quiesceAutopilot();
|
||||
expect(resume).not.toBeNull();
|
||||
expect(existsSync(autopilotPausedMarkerPath())).toBe(true);
|
||||
resume!();
|
||||
} finally {
|
||||
console.warn = origWarn;
|
||||
}
|
||||
expect(warns.join('\n')).toContain('GBRAIN_MIGRATE_QUIESCE_SECONDS');
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('release is ownership-conditional', () => {
|
||||
test('resume does NOT delete a marker that changed hands', async () => {
|
||||
await withEnv(quiesceEnv(), async () => {
|
||||
// Round-4 review catch: an adoption race resolved against us means the
|
||||
// marker at this PATH now belongs to another migrate. A path-only
|
||||
// unlink would un-pause the daemon in the middle of the new owner's
|
||||
// copy window; release must compare content first.
|
||||
const resume = await quiesceAutopilot();
|
||||
expect(resume).not.toBeNull();
|
||||
const marker = autopilotPausedMarkerPath();
|
||||
writeFileSync(marker, `${MIGRATE_PAUSE_MARKER_PREFIX} (pid 999999) at ${new Date(0).toISOString()}\n`);
|
||||
resume!();
|
||||
expect(existsSync(marker)).toBe(true); // still there — not ours anymore
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('pause marker fences minion job pickup', () => {
|
||||
test('worker claim loop checks the marker BEFORE queue.claim (static wiring guard)', async () => {
|
||||
// Third review round: the marker parked the autopilot dispatch loop but a
|
||||
// minion worker kept CLAIMING queued jobs mid-copy — writes the migration
|
||||
// silently lost at the flip. No in-process worker harness exists, so pin
|
||||
// the wiring statically (the autopilot-supervisor-wiring pattern): the
|
||||
// gate must sit in the poll loop ahead of the claim call.
|
||||
const { readFileSync } = await import('fs');
|
||||
const { resolve } = await import('path');
|
||||
const src = readFileSync(resolve(import.meta.dir, '../src/core/minions/worker.ts'), 'utf8');
|
||||
const gateIdx = src.indexOf('existsSync(autopilotPausedMarkerPath())');
|
||||
const claimIdx = src.indexOf('this.queue.claim(');
|
||||
expect(gateIdx).toBeGreaterThan(-1);
|
||||
expect(claimIdx).toBeGreaterThan(gateIdx);
|
||||
// And the POST-claim re-check (round 4): a claim can commit after
|
||||
// migrate's drain probe; a job claimed into that window is released
|
||||
// back un-run rather than executed.
|
||||
const postGateIdx = src.indexOf('existsSync(autopilotPausedMarkerPath())', claimIdx);
|
||||
expect(postGateIdx).toBeGreaterThan(claimIdx);
|
||||
expect(src.indexOf('releaseClaimForPause(', postGateIdx)).toBeGreaterThan(postGateIdx);
|
||||
});
|
||||
});
|
||||
@@ -21,6 +21,42 @@ describe('classifyReconnectError (#1162)', () => {
|
||||
expect(classifyReconnectError(err)).toBe('unrecoverable');
|
||||
});
|
||||
|
||||
describe('crash vs config verdict — the 71-day-outage class', () => {
|
||||
test('real Bun null-deref → crash, NOT unrecoverable', () => {
|
||||
// Verbatim Bun/V8 text. Lowercased it contains BOTH "database_url" and
|
||||
// "undefined", so the substring rules below used to classify it as a
|
||||
// config verdict and exit the daemon permanently. It is a gbrain bug.
|
||||
const err = new TypeError("undefined is not an object (evaluating 'config.database_url')");
|
||||
expect(classifyReconnectError(err)).toBe('crash');
|
||||
});
|
||||
|
||||
test('any TypeError → crash, by error type', () => {
|
||||
expect(classifyReconnectError(new TypeError('cfg.foo is not a function'))).toBe('crash');
|
||||
});
|
||||
|
||||
test('V8-worded null-deref arriving as a plain Error → crash, by message shape', () => {
|
||||
// Some hosts lose the error class across a serialization boundary.
|
||||
expect(classifyReconnectError(new Error("Cannot read properties of undefined (reading 'database_url')")))
|
||||
.toBe('crash');
|
||||
});
|
||||
|
||||
test('a genuine missing-config error is still unrecoverable', () => {
|
||||
// Guard against over-broadening: the crash rules must not swallow the
|
||||
// real "operator never set a URL" case, which SHOULD stop the daemon.
|
||||
expect(classifyReconnectError(new Error('GBRAIN_DATABASE_URL is not set'))).toBe('unrecoverable');
|
||||
expect(classifyReconnectError(new Error('No brain configured. Run: gbrain init'))).toBe('unrecoverable');
|
||||
});
|
||||
|
||||
test('a malformed URL is a config verdict even though JS throws it as a TypeError', () => {
|
||||
// `new URL('garbage')` throws TypeError('Invalid URL'). The blanket
|
||||
// TypeError-means-crash rule would spend the whole 30-attempt reconnect
|
||||
// budget on an error only the operator can fix (adversarial review
|
||||
// catch); the message pattern is tested first.
|
||||
expect(classifyReconnectError(new TypeError('Invalid URL'))).toBe('unrecoverable');
|
||||
expect(classifyReconnectError(new Error('failed to parse URL from postgres://'))).toBe('unrecoverable');
|
||||
});
|
||||
});
|
||||
|
||||
test('database_url empty → unrecoverable', () => {
|
||||
expect(classifyReconnectError(new Error('database_url is empty'))).toBe('unrecoverable');
|
||||
});
|
||||
|
||||
@@ -0,0 +1,229 @@
|
||||
/**
|
||||
* PR-A: the two surfaces that make a dead autopilot visible.
|
||||
*
|
||||
* generateSelfDisableGuard — stops a daemon whose --repo vanished, for real,
|
||||
* under a supervisor that restarts it on every exit.
|
||||
* classifyAutopilotStatus — tri-state verdict + exit code, so `--status` can
|
||||
* no longer answer "installed" for a daemon that died 71 days ago.
|
||||
*
|
||||
* Both are pure, so no daemon is installed here.
|
||||
*/
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import {
|
||||
generateSelfDisableGuard,
|
||||
crontabIndicatesAutopilotInstall,
|
||||
classifyAutopilotStatus,
|
||||
autopilotStatusExitCode,
|
||||
autopilotLaunchdLabel,
|
||||
autopilotEngineIdentity,
|
||||
AUTOPILOT_SYSTEMD_UNIT,
|
||||
} from '../src/commands/autopilot.ts';
|
||||
|
||||
describe('generateSelfDisableGuard', () => {
|
||||
test('tests the repo DIRECTORY, never $repo/.git', () => {
|
||||
const guard = generateSelfDisableGuard('/data/brain', 'macos');
|
||||
expect(guard).toContain("[ ! -d '/data/brain' ]");
|
||||
// .git is a FILE in worktrees and submodules, and --repo may be a
|
||||
// subdirectory of the checkout — either shape would false-positive and
|
||||
// disable a healthy install. Assert on the executable line only; the
|
||||
// surrounding comment mentions .git precisely to explain the trap.
|
||||
const testLine = guard.split('\n').find((l) => l.startsWith('if [ '))!;
|
||||
expect(testLine).toBe("if [ ! -d '/data/brain' ]; then");
|
||||
expect(testLine).not.toContain('.git');
|
||||
});
|
||||
|
||||
test('macOS actually boots the job out — exit 0 alone is a respawn loop', () => {
|
||||
// KeepAlive=true means launchd restarts on ANY exit, so "detect and exit"
|
||||
// disables nothing; it just makes the failure quieter and more frequent.
|
||||
const guard = generateSelfDisableGuard('/data/brain', 'macos');
|
||||
expect(guard).toContain('launchctl bootout');
|
||||
expect(guard).toContain(autopilotLaunchdLabel());
|
||||
});
|
||||
|
||||
test('systemd disables the unit — Restart=always has the same problem', () => {
|
||||
const guard = generateSelfDisableGuard('/data/brain', 'linux-systemd');
|
||||
expect(guard).toContain('systemctl --user disable --now');
|
||||
expect(guard).toContain(AUTOPILOT_SYSTEMD_UNIT);
|
||||
});
|
||||
|
||||
test('cron and container just exit — they are periodic/one-shot, not KeepAlive', () => {
|
||||
for (const target of ['linux-cron', 'ephemeral-container'] as const) {
|
||||
const guard = generateSelfDisableGuard('/data/brain', target);
|
||||
expect(guard).toContain('exit 0');
|
||||
expect(guard).not.toContain('launchctl bootout');
|
||||
expect(guard).not.toContain('systemctl --user disable');
|
||||
}
|
||||
});
|
||||
|
||||
test('writes a marker so --status can explain WHY it stopped', () => {
|
||||
expect(generateSelfDisableGuard('/data/brain', 'macos')).toContain('autopilot-disabled');
|
||||
});
|
||||
|
||||
test('single-quotes in the repo path cannot break out of the shell literal', () => {
|
||||
const guard = generateSelfDisableGuard("/data/o'brien/brain", 'macos');
|
||||
expect(guard).toContain("'/data/o'\\''brien/brain'");
|
||||
});
|
||||
|
||||
test('one missed probe is a strike, not a disable — three consecutive misses required', () => {
|
||||
// Repos on external volumes, NFS, or cloud-synced folders are routinely
|
||||
// absent for the first launch after login. A single-probe guard would
|
||||
// permanently bootout a healthy install on one transient miss (adversarial
|
||||
// review catch); the counter makes disable require persistence.
|
||||
const guard = generateSelfDisableGuard('/data/brain', 'macos');
|
||||
expect(guard).toContain('autopilot-disable-strikes');
|
||||
expect(guard).toContain('-lt 3');
|
||||
// Below the threshold the guard exits WITHOUT writing the disabled marker
|
||||
// or booting the job out: the early-exit line precedes both.
|
||||
const strikeExit = guard.indexOf('exit 0');
|
||||
expect(strikeExit).toBeGreaterThan(-1);
|
||||
expect(strikeExit).toBeLessThan(guard.indexOf('autopilot-disabled'));
|
||||
expect(strikeExit).toBeLessThan(guard.indexOf('launchctl bootout'));
|
||||
// A successful probe resets the counter, or three misses spread over
|
||||
// months would accumulate into a false disable.
|
||||
expect(guard.trimEnd().endsWith("2>/dev/null || true")).toBe(true);
|
||||
expect(guard.split('rm -f').length).toBeGreaterThanOrEqual(3);
|
||||
});
|
||||
});
|
||||
|
||||
describe('crontabIndicatesAutopilotInstall', () => {
|
||||
test('wrapper line and direct daemon line count as installs', () => {
|
||||
expect(crontabIndicatesAutopilotInstall("*/5 * * * * '/h/.gbrain/autopilot-run.sh' >> log 2>&1")).toBe(true);
|
||||
expect(crontabIndicatesAutopilotInstall('*/5 * * * * gbrain autopilot --repo /data/brain')).toBe(true);
|
||||
});
|
||||
|
||||
test('a status-monitor cron line is NOT an install', () => {
|
||||
// docs recommend cron-ing the status command as a health gate; counting
|
||||
// it as an install makes a monitor-only machine report installed and
|
||||
// never_run with exit 1 forever (adversarial review catch).
|
||||
expect(crontabIndicatesAutopilotInstall('*/10 * * * * gbrain autopilot --status --json >> health.log')).toBe(false);
|
||||
});
|
||||
|
||||
test('comments and unrelated lines never count', () => {
|
||||
expect(crontabIndicatesAutopilotInstall('# gbrain autopilot --repo /data/brain (disabled)')).toBe(false);
|
||||
expect(crontabIndicatesAutopilotInstall('0 3 * * * /usr/local/bin/backup.sh')).toBe(false);
|
||||
expect(crontabIndicatesAutopilotInstall('')).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('classifyAutopilotStatus', () => {
|
||||
const base = {
|
||||
installed: true,
|
||||
installTarget: 'macos' as const,
|
||||
disabledReason: null as string | null,
|
||||
pausedReason: null as string | null,
|
||||
heartbeatAgeSeconds: 10,
|
||||
intervalSeconds: 300,
|
||||
lastLog: '',
|
||||
};
|
||||
|
||||
test('THE INCIDENT: installed with a 71-day-old heartbeat → stale, exit 1', () => {
|
||||
// This is the case that reported "Autopilot: installed" and exited 0 for
|
||||
// 71 days while the daemon was dead.
|
||||
const r = classifyAutopilotStatus({ ...base, heartbeatAgeSeconds: 71 * 86400 });
|
||||
expect(r.state).toBe('stale');
|
||||
expect(autopilotStatusExitCode(r.state)).toBe(1);
|
||||
});
|
||||
|
||||
test('fresh heartbeat → fresh, exit 0', () => {
|
||||
const r = classifyAutopilotStatus(base);
|
||||
expect(r.state).toBe('fresh');
|
||||
expect(autopilotStatusExitCode(r.state)).toBe(0);
|
||||
});
|
||||
|
||||
test('nothing installed → exit 0 (nothing claimed, nothing broken)', () => {
|
||||
const r = classifyAutopilotStatus({ ...base, installed: false, installTarget: null, heartbeatAgeSeconds: null });
|
||||
expect(r.state).toBe('not_installed');
|
||||
expect(autopilotStatusExitCode(r.state)).toBe(0);
|
||||
});
|
||||
|
||||
test('installed but never ticked → never_run, exit 1', () => {
|
||||
const r = classifyAutopilotStatus({ ...base, heartbeatAgeSeconds: null });
|
||||
expect(r.state).toBe('never_run');
|
||||
expect(autopilotStatusExitCode(r.state)).toBe(1);
|
||||
});
|
||||
|
||||
test('self-disabled wins over every other signal, exit 2', () => {
|
||||
const r = classifyAutopilotStatus({ ...base, disabledReason: 'repo path gone: /data/brain' });
|
||||
expect(r.state).toBe('disabled');
|
||||
expect(r.disabled_reason).toContain('/data/brain');
|
||||
expect(autopilotStatusExitCode(r.state)).toBe(2);
|
||||
});
|
||||
|
||||
test('a healthy adaptive-schedule gap does not flip the verdict — a real outage does', () => {
|
||||
// The adaptive scheduler sleeps TWO intervals between ticks on the
|
||||
// healthiest brains and the heartbeat only refreshes at tick top, so a
|
||||
// normal healthy gap is cycle_duration + 2x interval. Tolerance is 6x —
|
||||
// review-army catch: at 3x, the best-behaved installs flapped stale
|
||||
// exit-1 alarms in the exact cron gates this exit code ships for.
|
||||
expect(classifyAutopilotStatus({ ...base, heartbeatAgeSeconds: 301 }).state).toBe('fresh');
|
||||
expect(classifyAutopilotStatus({ ...base, heartbeatAgeSeconds: 1799 }).state).toBe('fresh');
|
||||
expect(classifyAutopilotStatus({ ...base, heartbeatAgeSeconds: 1801 }).state).toBe('stale');
|
||||
});
|
||||
|
||||
test('tolerance scales with the interval, not a hardcoded constant', () => {
|
||||
const slow = classifyAutopilotStatus({ ...base, intervalSeconds: 3600, heartbeatAgeSeconds: 5000 });
|
||||
expect(slow.stale_after_seconds).toBe(21600);
|
||||
expect(slow.state).toBe('fresh');
|
||||
});
|
||||
|
||||
test('a garbage interval cannot silence the alarm — NaN staleAfter would read dead as fresh', () => {
|
||||
// Pre-fix: `--interval garbage` parsed to NaN, staleAfter = NaN, and every
|
||||
// age comparison came back false — the 71-day-dead daemon reported 'fresh'
|
||||
// with exit 0. The pure layer now falls back to the default tolerance.
|
||||
const r = classifyAutopilotStatus({ ...base, intervalSeconds: NaN, heartbeatAgeSeconds: 71 * 86400 });
|
||||
expect(r.stale_after_seconds).toBe(1800);
|
||||
expect(r.state).toBe('stale');
|
||||
expect(autopilotStatusExitCode(r.state)).toBe(1);
|
||||
expect(classifyAutopilotStatus({ ...base, intervalSeconds: 0 }).stale_after_seconds).toBe(1800);
|
||||
});
|
||||
|
||||
test('pause marker → paused, exit 1: a parked daemon must not report running', () => {
|
||||
// The tick loop refreshes its heartbeat BEFORE honoring the pause marker,
|
||||
// so a quiesced daemon looks 'fresh' by mtime while doing no work. Without
|
||||
// this state, a marker orphaned by a dead migrate is invisible — the
|
||||
// 71-day incident's shape again, reached through the pause path.
|
||||
const r = classifyAutopilotStatus({ ...base, pausedReason: 'paused by gbrain migrate at 2026-08-11T00:00:00Z' });
|
||||
expect(r.state).toBe('paused');
|
||||
expect(r.paused_reason).toContain('gbrain migrate');
|
||||
expect(autopilotStatusExitCode(r.state)).toBe(1);
|
||||
});
|
||||
|
||||
test('disabled outranks paused; a leftover marker with nothing installed stays quiet', () => {
|
||||
// disabled is the stronger, permanent claim (exit 2). And with no install
|
||||
// at all, a stray marker pauses nothing — alarming on it would make a
|
||||
// clean machine fail its cron gate forever.
|
||||
const both = classifyAutopilotStatus({ ...base, disabledReason: 'repo path gone', pausedReason: 'paused' });
|
||||
expect(both.state).toBe('disabled');
|
||||
const stray = classifyAutopilotStatus({
|
||||
...base, installed: false, installTarget: null, heartbeatAgeSeconds: null, pausedReason: 'paused',
|
||||
});
|
||||
expect(stray.state).toBe('not_installed');
|
||||
expect(autopilotStatusExitCode(stray.state)).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('autopilotEngineIdentity (post-migration convergence)', () => {
|
||||
// Red-team catch: after a clean migration flipped config.json, the daemon
|
||||
// never converged — its health probe kept succeeding against the OLD engine
|
||||
// (the preserved backup stays alive) and reconnect() deliberately restores
|
||||
// the config captured at connect(). The tick loop now compares this
|
||||
// identity against its boot value and exits for supervisor relaunch.
|
||||
test('changes when the engine kind or connection target changes', () => {
|
||||
const pglite = autopilotEngineIdentity({ engine: 'pglite', database_path: '/a/brain.pglite' });
|
||||
expect(autopilotEngineIdentity({ engine: 'postgres', database_url: 'postgres://x/db' })).not.toBe(pglite);
|
||||
expect(autopilotEngineIdentity({ engine: 'pglite', database_path: '/b/brain.pglite' })).not.toBe(pglite);
|
||||
});
|
||||
|
||||
test('ignores everything a migration does not flip — unrelated config edits must not restart the daemon', () => {
|
||||
const before = autopilotEngineIdentity({ engine: 'pglite', database_path: '/a/brain.pglite' });
|
||||
const after = autopilotEngineIdentity({
|
||||
engine: 'pglite', database_path: '/a/brain.pglite',
|
||||
embedding_model: 'voyage-3', search: { mode: 'tokenmax' },
|
||||
} as Parameters<typeof autopilotEngineIdentity>[0]);
|
||||
expect(after).toBe(before);
|
||||
});
|
||||
|
||||
test('a missing config file reads as the pglite default, not a crash', () => {
|
||||
expect(autopilotEngineIdentity(null)).toBe(autopilotEngineIdentity({ engine: 'pglite' }));
|
||||
});
|
||||
});
|
||||
+57
-2
@@ -1097,7 +1097,7 @@ describe('v0.41.32.0 — commit-relative staleness', () => {
|
||||
expect(result.details).toEqual({ unchanged_count: 1, synced_recently_count: 0, stale_count: 0 });
|
||||
});
|
||||
|
||||
test('T2: REMOTE (no localOnly) reads column, quiet repo → ok, NO git subprocess', async () => {
|
||||
test('T2: REMOTE (no localOnly) reads column, quiet + recently synced → ok, NO git subprocess', async () => {
|
||||
const { checkSyncFreshness } = await import('../src/commands/doctor.ts');
|
||||
const { _setGitHeadProbeForTests, _setGitCleanProbeForTests } =
|
||||
await import('../src/core/git-head.ts');
|
||||
@@ -1107,7 +1107,9 @@ describe('v0.41.32.0 — commit-relative staleness', () => {
|
||||
|
||||
const result = await checkSyncFreshness(makeStubEngine([
|
||||
{ id: 'remote', name: '', local_path: '/tmp/remote',
|
||||
last_sync_at: agoMs(100 * 60 * 60 * 1000),
|
||||
// Synced 1h ago: caught up AND being checked. Must stay ok — this is the
|
||||
// quiet-source contract the content comparison exists to protect.
|
||||
last_sync_at: agoMs(1 * 60 * 60 * 1000),
|
||||
last_commit: 'x', chunker_version: currentChunkerVersion,
|
||||
// Content committed BEFORE the last sync → caught up.
|
||||
newest_content_at: agoMs(200 * 60 * 60 * 1000) },
|
||||
@@ -1119,6 +1121,30 @@ describe('v0.41.32.0 — commit-relative staleness', () => {
|
||||
expect(result.details).toEqual({ unchanged_count: 0, synced_recently_count: 1, stale_count: 0 });
|
||||
});
|
||||
|
||||
test('T2-ceiling: REMOTE, caught up but unsynced past the ceiling → NOT ok', async () => {
|
||||
// The 71-day bug at the doctor layer. This shape (caught up, synced 100h ago)
|
||||
// previously reported ok forever, so a dead daemon was invisible. The ceiling
|
||||
// ramps it to ~28h, which crosses the 24h warn line but NOT the 72h fail line —
|
||||
// proving the escalation is ordered rather than a single jump to fail.
|
||||
const { checkSyncFreshness } = await import('../src/commands/doctor.ts');
|
||||
const { _setGitHeadProbeForTests, _setGitCleanProbeForTests } =
|
||||
await import('../src/core/git-head.ts');
|
||||
let headCalls = 0;
|
||||
_setGitHeadProbeForTests(() => { headCalls++; return 'x'; });
|
||||
_setGitCleanProbeForTests(() => true);
|
||||
|
||||
const result = await checkSyncFreshness(makeStubEngine([
|
||||
{ id: 'abandoned', name: '', local_path: '/tmp/abandoned',
|
||||
last_sync_at: agoMs(100 * 60 * 60 * 1000),
|
||||
last_commit: 'x', chunker_version: currentChunkerVersion,
|
||||
newest_content_at: agoMs(200 * 60 * 60 * 1000) },
|
||||
]));
|
||||
|
||||
expect(headCalls).toBe(0); // still no subprocess on the remote path
|
||||
expect(result.status).toBe('warn'); // 28h: past warn, short of fail
|
||||
expect(result.details).toEqual({ unchanged_count: 0, synced_recently_count: 0, stale_count: 1 });
|
||||
});
|
||||
|
||||
test('T2b: REMOTE + NULL column → wall-clock fallback → stale (no git subprocess)', async () => {
|
||||
const { checkSyncFreshness } = await import('../src/commands/doctor.ts');
|
||||
const { _setGitHeadProbeForTests, _setGitCleanProbeForTests } =
|
||||
@@ -1621,6 +1647,35 @@ describe('BUG 4 — in-progress sync via live lock, not stale freshness', () =>
|
||||
expect(result.status).toBe('fail');
|
||||
});
|
||||
|
||||
test('a lock held PAST the staleness ceiling is wedged, not in-progress → fail', async () => {
|
||||
// `withRefreshingLock` bumps the heartbeat on its own timer regardless of
|
||||
// whether the import is making forward progress, so a holder blocked inside
|
||||
// a query refreshes forever. An uncapped in-progress verdict would mask that
|
||||
// source from every staleness check indefinitely — the same invisible-failure
|
||||
// class as the freshness bug, reached through the lock table instead.
|
||||
const { checkSyncFreshness } = await import('../src/commands/doctor.ts');
|
||||
await addSource('wiki', staleDate());
|
||||
// Live, actively-refreshing lock (TTL 30 min in the future) — but acquired
|
||||
// 100h ago, well past the 72h default ceiling.
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO gbrain_cycle_locks (id, holder_pid, holder_host, acquired_at, ttl_expires_at, last_refreshed_at)
|
||||
VALUES ($1, 4242, 'testhost', now() - interval '100 hours', now() + interval '30 minutes', now())`,
|
||||
[syncLockId('wiki')],
|
||||
);
|
||||
const result = await checkSyncFreshness(engine);
|
||||
expect(result.status).toBe('fail');
|
||||
expect(result.message).toContain('held the sync lock');
|
||||
expect(result.message).toContain('break-lock');
|
||||
});
|
||||
|
||||
test('a freshly-acquired live lock is still in-progress → ok (cap does not over-fire)', async () => {
|
||||
const { checkSyncFreshness } = await import('../src/commands/doctor.ts');
|
||||
await addSource('wiki', staleDate());
|
||||
await holdLock('wiki', 30); // acquired_at = now(), well under the ceiling
|
||||
const result = await checkSyncFreshness(engine);
|
||||
expect(result.status).toBe('ok');
|
||||
});
|
||||
|
||||
test('blocked source with banked checkpoint rows but NO live lock → still fail', async () => {
|
||||
const { checkSyncFreshness } = await import('../src/commands/doctor.ts');
|
||||
await addSource('wiki', staleDate());
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
/**
|
||||
* The keyless clean-refusal predicate for `gbrain embed`.
|
||||
*
|
||||
* The documented always-current chain for external agent schedulers is
|
||||
* `gbrain sync ... && gbrain embed --stale`; on a keyless brain that chain
|
||||
* must exit 0 (nothing to backfill is not a failure). Every OTHER spelling is
|
||||
* an explicit request for something impossible on a keyless brain and keeps
|
||||
* exiting 1 via EmbeddingDisabledError — the predicate's exclusions mirror
|
||||
* runEmbed's dispatch precedence, where a slugs flag wins over stale.
|
||||
*/
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { isKeylessStaleRefusal } from '../src/commands/embed.ts';
|
||||
|
||||
describe('isKeylessStaleRefusal', () => {
|
||||
test('the documented chain spelling refuses cleanly on a keyless brain', () => {
|
||||
expect(isKeylessStaleRefusal(['--stale'], true)).toBe(true);
|
||||
});
|
||||
|
||||
test('never fires on a brain with embeddings enabled (or unknown)', () => {
|
||||
expect(isKeylessStaleRefusal(['--stale'], false)).toBe(false);
|
||||
expect(isKeylessStaleRefusal(['--stale'], undefined)).toBe(false);
|
||||
});
|
||||
|
||||
test('a slugs list keeps exiting 1 — dispatch gives slugs precedence over stale', () => {
|
||||
// Review-army catch: the guard originally excluded only the all flag, so
|
||||
// an explicit `embed --slugs a b --stale` on a keyless brain exited 0
|
||||
// while doing nothing — contradicting the guard's own contract.
|
||||
expect(isKeylessStaleRefusal(['--slugs', 'a', 'b', '--stale'], true)).toBe(false);
|
||||
});
|
||||
|
||||
test('the all flag and dry-run are excluded — explicit asks stay loud', () => {
|
||||
expect(isKeylessStaleRefusal(['--all', '--stale'], true)).toBe(false);
|
||||
expect(isKeylessStaleRefusal(['--stale', '--dry-run'], true)).toBe(false);
|
||||
});
|
||||
|
||||
test('a bare stale run with routing flags still qualifies', () => {
|
||||
// Scoping flags do not change WHAT is being asked, only where.
|
||||
expect(isKeylessStaleRefusal(['--stale', '--source', 'wiki'], true)).toBe(true);
|
||||
});
|
||||
|
||||
test('no stale flag, no refusal — an explicit slug is not a stale run', () => {
|
||||
expect(isKeylessStaleRefusal(['my-page'], true)).toBe(false);
|
||||
expect(isKeylessStaleRefusal([], true)).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,107 @@
|
||||
/**
|
||||
* Direct coverage for the shared numeric env-var resolver.
|
||||
*
|
||||
* This module exists so doctor.ts and source-health.ts share ONE warn-once
|
||||
* memo (a duplicate helper would warn once per module — "warn exactly once"
|
||||
* quietly becoming "warn twice"). It backs the staleness-ceiling resolution
|
||||
* that closed the 71-day false-green, so its garbage-handling is load-bearing:
|
||||
* an env typo that silently produced NaN here would resurface the incident.
|
||||
*/
|
||||
import { describe, test, expect, beforeEach } from 'bun:test';
|
||||
import { withEnv } from './helpers/with-env.ts';
|
||||
import {
|
||||
resolveEnvNumber,
|
||||
resolveHoursEnv,
|
||||
warnOnceForEnv,
|
||||
_resetEnvNumberWarnedForTests,
|
||||
} from '../src/core/env-number.ts';
|
||||
|
||||
const VAR = 'GBRAIN_ENV_NUMBER_TEST_VAR';
|
||||
|
||||
function captureWarns(fn: () => void): string[] {
|
||||
const warns: string[] = [];
|
||||
const orig = console.warn;
|
||||
console.warn = (...a: unknown[]) => { warns.push(a.join(' ')); };
|
||||
try { fn(); } finally { console.warn = orig; }
|
||||
return warns;
|
||||
}
|
||||
|
||||
beforeEach(() => _resetEnvNumberWarnedForTests());
|
||||
|
||||
describe('resolveEnvNumber', () => {
|
||||
test('unset and empty both mean fallback, with no warning', async () => {
|
||||
await withEnv({ [VAR]: undefined }, () => {
|
||||
const warns = captureWarns(() => {
|
||||
expect(resolveEnvNumber(VAR, 72)).toBe(72);
|
||||
});
|
||||
expect(warns).toEqual([]);
|
||||
});
|
||||
await withEnv({ [VAR]: '' }, () => {
|
||||
expect(resolveEnvNumber(VAR, 72)).toBe(72);
|
||||
});
|
||||
});
|
||||
|
||||
test('a valid positive number wins, including fractional', async () => {
|
||||
await withEnv({ [VAR]: '36' }, () => {
|
||||
expect(resolveEnvNumber(VAR, 72)).toBe(36);
|
||||
});
|
||||
await withEnv({ [VAR]: '0.5' }, () => {
|
||||
expect(resolveEnvNumber(VAR, 72)).toBe(0.5);
|
||||
});
|
||||
});
|
||||
|
||||
test('garbage, zero, and negatives fall back — zero is an unset accident, not a disable', async () => {
|
||||
for (const bad of ['garbage', 'NaN', 'Infinity', '-5', '0']) {
|
||||
_resetEnvNumberWarnedForTests();
|
||||
await withEnv({ [VAR]: bad }, () => {
|
||||
expect(resolveEnvNumber(VAR, 72)).toBe(72);
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
test('warns exactly once per var per process, unit string included', async () => {
|
||||
await withEnv({ [VAR]: 'garbage' }, () => {
|
||||
const warns = captureWarns(() => {
|
||||
resolveEnvNumber(VAR, 72, { unit: 'h' });
|
||||
resolveEnvNumber(VAR, 72, { unit: 'h' });
|
||||
resolveEnvNumber(VAR, 72, { unit: 'h' });
|
||||
});
|
||||
expect(warns).toHaveLength(1);
|
||||
expect(warns[0]).toContain(VAR);
|
||||
expect(warns[0]).toContain('72h');
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('resolveHoursEnv', () => {
|
||||
test('is resolveEnvNumber with the hours unit pre-applied', async () => {
|
||||
await withEnv({ [VAR]: 'bogus' }, () => {
|
||||
const warns = captureWarns(() => {
|
||||
expect(resolveHoursEnv(VAR, 24)).toBe(24);
|
||||
});
|
||||
expect(warns[0]).toContain('24h');
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('warnOnceForEnv', () => {
|
||||
test('shares the SAME memo as resolveEnvNumber — the whole reason this module exists', async () => {
|
||||
await withEnv({ [VAR]: 'garbage' }, () => {
|
||||
const warns = captureWarns(() => {
|
||||
resolveEnvNumber(VAR, 72); // seeds the memo for VAR
|
||||
warnOnceForEnv(VAR, 'should be suppressed');
|
||||
warnOnceForEnv(VAR, 'should be suppressed');
|
||||
});
|
||||
expect(warns).toHaveLength(1);
|
||||
expect(warns[0]).not.toBe('should be suppressed');
|
||||
});
|
||||
});
|
||||
|
||||
test('fires its message once for a var resolveEnvNumber never touched', () => {
|
||||
const warns = captureWarns(() => {
|
||||
warnOnceForEnv('GBRAIN_ENV_NUMBER_OTHER_VAR', 'custom disabled-unless-set warning');
|
||||
warnOnceForEnv('GBRAIN_ENV_NUMBER_OTHER_VAR', 'custom disabled-unless-set warning');
|
||||
});
|
||||
expect(warns).toEqual(['custom disabled-unless-set warning']);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,42 @@
|
||||
/**
|
||||
* Pre-test setup: redirect the sync failure ledger to a per-run scratch
|
||||
* directory instead of the operator's real `~/.gbrain/sync-failures.jsonl`.
|
||||
*
|
||||
* Why this exists: the suite deliberately exercises broken-fixture import and
|
||||
* sync paths (bad frontmatter, slug mismatches, unparseable YAML). Every one of
|
||||
* those records a row through `src/core/sync-failure-ledger.ts`, which resolved
|
||||
* to the gbrain home with no override — so fixture failures appended straight
|
||||
* into the operator's live ledger. That file is not inert: `gbrain doctor` reads
|
||||
* it and keeps warning until each entry is fixed or acknowledged, and the rows
|
||||
* carry commit attribution, so a fabricated `srcE / notes/bad.md` row looks
|
||||
* exactly like a real un-indexed page.
|
||||
*
|
||||
* Same shape and rationale as `audit-dir-preload.ts` (#2823), and deliberately a
|
||||
* DEDICATED var rather than a global `GBRAIN_HOME` redirect: pointing
|
||||
* GBRAIN_HOME at a scratch dir for the whole suite would also make
|
||||
* `loadConfig()` return null for every test that reads the real config — a much
|
||||
* wider blast radius than this problem needs, and
|
||||
* `test/gbrain-home-isolation.test.ts` asserts the unset-fallback behavior
|
||||
* directly.
|
||||
*
|
||||
* Each `scripts/run-unit-shard.sh` shard is its own `bun test` process, so each
|
||||
* gets its own scratch dir — no cross-shard collision, no cleanup needed (the
|
||||
* OS reaps tmp, the same tradeoff `test/helpers/with-env.ts` documents).
|
||||
*
|
||||
* Only sets the var if it isn't already set, so a developer who exports it to
|
||||
* inspect ledger output after a local run keeps their override. Tests that
|
||||
* manage their own value per-test via `withEnv` are unaffected.
|
||||
*
|
||||
* Imported by `bunfig.toml` via `preload`.
|
||||
*/
|
||||
import { mkdtempSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
|
||||
if (!process.env.GBRAIN_SYNC_FAILURES_DIR) {
|
||||
const dir = mkdtempSync(join(tmpdir(), 'gbrain-test-syncfail-'));
|
||||
process.env.GBRAIN_SYNC_FAILURES_DIR = dir;
|
||||
if (process.env.GBRAIN_DEBUG_PRELOAD === '1') {
|
||||
console.error(`[sync-failures-preload] GBRAIN_SYNC_FAILURES_DIR=${dir}`);
|
||||
}
|
||||
}
|
||||
@@ -23,10 +23,12 @@ import {
|
||||
resolvePriority,
|
||||
newestCommitMs,
|
||||
lagFromContentMs,
|
||||
resolveStalenessCeilingSeconds,
|
||||
_resetPriorityWarningsForTest,
|
||||
} from '../src/core/source-health.ts';
|
||||
import { isSourceUnchangedSinceSync } from '../src/core/git-head.ts';
|
||||
import { loadAllSources } from '../src/core/sources-load.ts';
|
||||
import { withEnv } from './helpers/with-env.ts';
|
||||
|
||||
const HOUR = 3600_000;
|
||||
|
||||
@@ -161,12 +163,56 @@ describe('lagFromContentMs (pure remote/column comparator)', () => {
|
||||
test('null content → wall-clock fallback', () => {
|
||||
expect(lagFromContentMs(null, now - 100 * HOUR, now)).toBe(360_000); // 100h in s
|
||||
});
|
||||
test('content at/before last sync → caught up (0)', () => {
|
||||
expect(lagFromContentMs(now - 200 * HOUR, now - 100 * HOUR, now)).toBe(0);
|
||||
test('content at/before last sync, recently checked → caught up (0)', () => {
|
||||
// The quiet-source contract: a 0-change sync heartbeats last_sync_at
|
||||
// (sync.ts, v0.42.52.0), so a source that IS being checked stays at 0.
|
||||
expect(lagFromContentMs(now - 200 * HOUR, now - HOUR, now, 72 * 3600)).toBe(0);
|
||||
});
|
||||
test('content at/before last sync, past the ceiling → ramps (was the 71-day bug)', () => {
|
||||
// This assertion previously expected 0 at 100h and that WAS the bug: a source
|
||||
// whose clone had vanished reported fresh forever, because the caught-up branch
|
||||
// discarded the wall-clock number computed one line above it. 100h - 72h ceiling
|
||||
// = 28h of real lag. Do not "restore" this to 0.
|
||||
expect(lagFromContentMs(now - 200 * HOUR, now - 100 * HOUR, now, 72 * 3600)).toBe(28 * 3600);
|
||||
});
|
||||
test('ceiling boundary: just under → 0, just over → ramped', () => {
|
||||
const ceiling = 72 * 3600;
|
||||
expect(lagFromContentMs(now - 200 * HOUR, now - 71.9 * HOUR, now, ceiling)).toBe(0);
|
||||
// Exactly at the ceiling is still 0; one second past it is one second of lag.
|
||||
expect(lagFromContentMs(now - 200 * HOUR, now - 72 * HOUR, now, ceiling)).toBe(0);
|
||||
expect(lagFromContentMs(now - 200 * HOUR, now - (72 * HOUR + 1000), now, ceiling)).toBe(1);
|
||||
});
|
||||
test('ramp is monotonic — crosses warn before fail, never steps', () => {
|
||||
const ceiling = 72 * 3600;
|
||||
const at = (h: number) => lagFromContentMs(now - 500 * HOUR, now - h * HOUR, now, ceiling)!;
|
||||
// Strictly increasing past the ceiling, so the 1h/24h/72h consumers escalate in
|
||||
// order instead of all tripping in the same instant.
|
||||
expect(at(80)).toBeLessThan(at(100));
|
||||
expect(at(100)).toBeLessThan(at(200));
|
||||
// Enters the band from 0 rather than jumping straight to the ceiling value:
|
||||
// one hour past the ceiling is one hour of lag, not 72.
|
||||
expect(at(73)).toBe(3600);
|
||||
});
|
||||
test('content after last sync → wall-clock since sync', () => {
|
||||
expect(lagFromContentMs(now - 10 * HOUR, now - 100 * HOUR, now)).toBe(360_000);
|
||||
});
|
||||
test('GBRAIN_STALENESS_CEILING_HOURS overrides GBRAIN_SYNC_FRESHNESS_FAIL_HOURS', async () => {
|
||||
await withEnv(
|
||||
{ GBRAIN_SYNC_FRESHNESS_FAIL_HOURS: '72', GBRAIN_STALENESS_CEILING_HOURS: '10' },
|
||||
async () => {
|
||||
// Override wins: 20h wall-clock against a 10h ceiling → 10h of lag.
|
||||
expect(resolveStalenessCeilingSeconds()).toBe(10 * 3600);
|
||||
expect(lagFromContentMs(now - 200 * HOUR, now - 20 * HOUR, now)).toBe(10 * 3600);
|
||||
},
|
||||
);
|
||||
await withEnv(
|
||||
{ GBRAIN_SYNC_FRESHNESS_FAIL_HOURS: '72', GBRAIN_STALENESS_CEILING_HOURS: undefined },
|
||||
async () => {
|
||||
// Without the override it tracks the fail threshold.
|
||||
expect(resolveStalenessCeilingSeconds()).toBe(72 * 3600);
|
||||
},
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('isSourceUnchangedSinceSync (ignore-untracked) — local caught-up contract', () => {
|
||||
@@ -338,9 +384,12 @@ describe('computeAllSourceMetrics', () => {
|
||||
expect(metrics.find((m) => m.source_id === 'behind')!.lag_seconds!).toBeGreaterThan(72 * 3600);
|
||||
});
|
||||
|
||||
test('REMOTE (default): reads newest_content_at column, NO git probe → quiet repo lag 0', async () => {
|
||||
test('REMOTE (default): reads newest_content_at column, NO git probe → quiet + recently synced = lag 0', async () => {
|
||||
// Criterion 2 (quiet-source non-regression) at the metrics layer: a bogus
|
||||
// local_path must NOT make a source stale so long as sync is still running
|
||||
// against it. This is the doctor 70->30 alert-storm case.
|
||||
const contentIso = new Date(Date.now() - 200 * HOUR).toISOString(); // content predates sync
|
||||
const syncIso = new Date(Date.now() - 100 * HOUR).toISOString();
|
||||
const syncIso = new Date(Date.now() - 1 * HOUR).toISOString();
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, local_path, last_commit, last_sync_at, newest_content_at, config)
|
||||
VALUES ('remote', 'remote', '/nonexistent/not-a-repo', 'x', $1, $2, '{"federated":true}'::jsonb)`,
|
||||
@@ -352,6 +401,24 @@ describe('computeAllSourceMetrics', () => {
|
||||
expect(metrics.find((m) => m.source_id === 'remote')!.lag_seconds).toBe(0);
|
||||
});
|
||||
|
||||
test('REMOTE (default): quiet but unsynced past the ceiling → ramps, not 0', async () => {
|
||||
// Criterion 1 (the 71-day bug) at the metrics layer. This previously
|
||||
// asserted 0 at 100h, which is exactly how a dead daemon stayed invisible.
|
||||
const contentIso = new Date(Date.now() - 200 * HOUR).toISOString();
|
||||
const syncIso = new Date(Date.now() - 100 * HOUR).toISOString();
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, local_path, last_commit, last_sync_at, newest_content_at, config)
|
||||
VALUES ('abandoned', 'abandoned', '/nonexistent/not-a-repo', 'x', $1, $2, '{"federated":true}'::jsonb)`,
|
||||
[syncIso, contentIso],
|
||||
);
|
||||
const sources = await loadAllSources(engine);
|
||||
const metrics = await computeAllSourceMetrics(engine, sources);
|
||||
const lag = metrics.find((m) => m.source_id === 'abandoned')!.lag_seconds!;
|
||||
// ~100h elapsed minus the 72h ceiling = ~28h of real lag.
|
||||
expect(lag).toBeGreaterThan(27 * 3600);
|
||||
expect(lag).toBeLessThan(29 * 3600);
|
||||
});
|
||||
|
||||
test('REMOTE (default): NULL column → wall-clock fallback', async () => {
|
||||
const syncIso = new Date(Date.now() - 100 * HOUR).toISOString();
|
||||
await engine.executeRaw(
|
||||
|
||||
@@ -202,14 +202,21 @@ describe('buildSyncStatusReport', () => {
|
||||
const syncIso = new Date(now - 100 * 60 * 60 * 1000).toISOString(); // synced 100h ago
|
||||
const sources = [
|
||||
{ id: 'quiet', name: 'quiet', local_path: '/tmp/quiet', config: { syncEnabled: true } },
|
||||
{ id: 'quiet_recent', name: 'quiet_recent', local_path: '/tmp/quiet_recent', config: { syncEnabled: true } },
|
||||
{ id: 'behind', name: 'behind', local_path: '/tmp/behind', config: { syncEnabled: true } },
|
||||
{ id: 'nocol', name: 'nocol', local_path: '/tmp/nocol', config: { syncEnabled: true } },
|
||||
];
|
||||
const engine = makeEngine({
|
||||
sourceRows: [
|
||||
// Newest commit 200h ago, synced 100h ago → caught up → lag 0 → fresh.
|
||||
// Newest commit 200h ago, synced 100h ago → caught up, but nobody has
|
||||
// synced in 100h, so the staleness ceiling ramps it to ~28h (100 - 72).
|
||||
// It lands in the MIDDLE band, not 'severe' — that ordering is the point.
|
||||
{ id: 'quiet', last_commit: 'a'.repeat(40), last_sync_at: syncIso,
|
||||
newest_content_at: new Date(now - 200 * 60 * 60 * 1000).toISOString() },
|
||||
// Same content shape but synced 1h ago → genuinely being checked → fresh.
|
||||
{ id: 'quiet_recent', last_commit: 'd'.repeat(40),
|
||||
last_sync_at: new Date(now - 1 * 60 * 60 * 1000).toISOString(),
|
||||
newest_content_at: new Date(now - 200 * 60 * 60 * 1000).toISOString() },
|
||||
// Newest commit 10h ago, synced 100h ago → behind → wall-clock → severe.
|
||||
{ id: 'behind', last_commit: 'b'.repeat(40), last_sync_at: syncIso,
|
||||
newest_content_at: new Date(now - 10 * 60 * 60 * 1000).toISOString() },
|
||||
@@ -218,6 +225,7 @@ describe('buildSyncStatusReport', () => {
|
||||
],
|
||||
countRows: [
|
||||
{ source_id: 'quiet', pages: 10, chunks_total: 20, chunks_unembedded: 0 },
|
||||
{ source_id: 'quiet_recent', pages: 10, chunks_total: 20, chunks_unembedded: 0 },
|
||||
{ source_id: 'behind', pages: 10, chunks_total: 20, chunks_unembedded: 0 },
|
||||
{ source_id: 'nocol', pages: 10, chunks_total: 20, chunks_unembedded: 0 },
|
||||
],
|
||||
@@ -225,10 +233,19 @@ describe('buildSyncStatusReport', () => {
|
||||
|
||||
const report = await buildSyncStatusReport(engine, sources);
|
||||
const byId = new Map(report.sources.map((s) => [s.source_id, s]));
|
||||
// Legacy wall-clock would have called 'quiet' severe (100h). Content-relative
|
||||
// correctly reports caught-up.
|
||||
expect(byId.get('quiet')!.staleness_hours).toBe(0);
|
||||
expect(byId.get('quiet')!.staleness_class).toBe('fresh');
|
||||
// Three-way distinction, which is the whole point of the ramp:
|
||||
// quiet_recent → 0 (caught up AND being checked)
|
||||
// quiet → ~28h (caught up but abandoned 100h ago — used to report 0
|
||||
// forever, which is how a dead daemon stayed invisible)
|
||||
// behind → 100h (real new content unsynced)
|
||||
// Naive wall-clock would flatten the first two into 'severe' alongside the third.
|
||||
expect(byId.get('quiet_recent')!.staleness_hours).toBe(0);
|
||||
expect(byId.get('quiet_recent')!.staleness_class).toBe('fresh');
|
||||
expect(byId.get('quiet')!.staleness_hours).toBeGreaterThan(27);
|
||||
expect(byId.get('quiet')!.staleness_hours).toBeLessThan(29);
|
||||
// Middle band, NOT severe: the ramp crosses warn before fail so consumers at
|
||||
// 24h and 72h escalate in order rather than tripping in the same instant.
|
||||
expect(byId.get('quiet')!.staleness_class).toBe('stale');
|
||||
expect(byId.get('behind')!.staleness_class).toBe('severe');
|
||||
expect(byId.get('nocol')!.staleness_hours).toBeGreaterThan(72);
|
||||
expect(byId.get('nocol')!.staleness_class).toBe('severe');
|
||||
|
||||
@@ -33,6 +33,32 @@ describe('buildSyncManifest', () => {
|
||||
expect(manifest.renamed).toEqual([{ from: 'people/old.md', to: 'people/new.md' }]);
|
||||
});
|
||||
|
||||
test('T (typechange) counts as modified — was silently dropped', () => {
|
||||
// file <-> symlink. Reachable with the flags gbrain actually passes
|
||||
// (`--name-status -M`). Dropping it meant the change never reached the
|
||||
// index until some later commit happened to touch the same path.
|
||||
const manifest = buildSyncManifest(`T\tpeople/now-a-symlink.md`);
|
||||
expect(manifest.modified).toEqual(['people/now-a-symlink.md']);
|
||||
expect(manifest.added).toEqual([]);
|
||||
expect(manifest.deleted).toEqual([]);
|
||||
});
|
||||
|
||||
test('U (unmerged) degrades to modified rather than vanishing', () => {
|
||||
// Only reachable in a conflicted worktree, which sync does not run against.
|
||||
// Defensive: re-import beats silently skipping.
|
||||
expect(buildSyncManifest(`U\tpeople/conflicted.md`).modified).toEqual(['people/conflicted.md']);
|
||||
});
|
||||
|
||||
test('C (copy) imports the destination — unreachable today, defensive', () => {
|
||||
// Requires -C/--find-copies, which gbrain does not pass. If the flags ever
|
||||
// change, the copy destination is a NEW path that must be imported, and the
|
||||
// source is untouched — so it is an add, not a rename.
|
||||
const manifest = buildSyncManifest(`C100\tpeople/src.md\tpeople/copy.md`);
|
||||
expect(manifest.added).toEqual(['people/copy.md']);
|
||||
expect(manifest.renamed).toEqual([]);
|
||||
expect(manifest.deleted).toEqual([]);
|
||||
});
|
||||
|
||||
test('handles empty diff', () => {
|
||||
const manifest = buildSyncManifest('');
|
||||
expect(manifest.added).toEqual([]);
|
||||
|
||||
Reference in New Issue
Block a user