mirror of
https://github.com/garrytan/gbrain.git
synced 2026-08-15 01:12:20 +00:00
Compare commits
9
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
055a0bb80c | ||
|
|
0fb0c83d24 | ||
|
|
ed900c870e | ||
|
|
e96f054cf0 | ||
|
|
2841432dec | ||
|
|
10fae74854 | ||
|
|
3b4b7e8ee1 | ||
|
|
c4d85cd506 | ||
|
|
3b090e1f20 |
@@ -18,3 +18,4 @@ eval/data/world-v1/world.html
|
||||
# BrainBench amara-life-v1 Opus cache (regenerate via eval:generate-amara-life)
|
||||
eval/data/amara-life-v1/_cache/
|
||||
.claude/
|
||||
export/
|
||||
|
||||
+256
@@ -2,6 +2,262 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.22.15] - 2026-04-29
|
||||
|
||||
## **Throw bare markdown into your brain and it becomes properly typed knowledge. No YAML ceremony.**
|
||||
|
||||
A real 81K-page brain has 9,655 files with no frontmatter. They imported fine, but every one of them landed in the DB as `type: concept`, `title: <slugified-filename>`, no date, no source, no tags. Search ranking suffered. Type-filtered queries missed them. Entity resolution fell over.
|
||||
|
||||
This release adds path-aware frontmatter inference. `gbrain sync` now synthesizes type, date, source, and tags from the filesystem path and first heading the moment a bare-frontmatter file imports. No LLM call, fully deterministic, file on disk untouched. An Apple Note at `Apple Notes/2010-04-13 founders mtg.md` lands as `type: apple-note, title: founders mtg, date: 2010-04-13, source: apple-notes` instead of `type: concept, title: 2010 04 13 Founders Mtg`.
|
||||
|
||||
If you want the inference written back to git, the new `gbrain frontmatter generate <path> --fix` walks a brain dir, infers frontmatter for every file that lacks it, and writes back with `.bak` safety backups. Dry-run by default.
|
||||
|
||||
### The 9,655 numbers that matter
|
||||
|
||||
Measured against my actual brain (gbrain v0.22.8 + the new inference path).
|
||||
|
||||
| Behavior | Before v0.22.15 | After v0.22.15 |
|
||||
|---|---|---|
|
||||
| Files importing as `type: concept` (no frontmatter) | 9,655 | 0 |
|
||||
| Apple Notes typed correctly (`apple-note`) | 0 | 5,861 |
|
||||
| Calendar indexes typed correctly (`calendar-index`) | 0 | 3,201 |
|
||||
| Therapy sessions typed + dated | 0 | 60 |
|
||||
| Essay drafts typed + dated | 0 | 33 |
|
||||
| LLM cost for the full reclassification | n/a | $0 |
|
||||
|
||||
The agent doing type-filtered queries on your brain (`type: person`, `type: meeting`, `type: essay`) now actually finds those pages instead of treating everything as `concept`.
|
||||
|
||||
### What this means for you
|
||||
|
||||
If you've been resisting frontmatter ceremony — same. Throw bare markdown into your brain and inference handles it. The rules table in `src/core/frontmatter-inference.ts` covers the obvious directories (`people/`, `companies/`, `daily/calendar/`, `writing/`, `meetings/`, `personal/`, etc.) plus a generic catch-all. Adding a new convention is one line in `DIRECTORY_RULES`.
|
||||
|
||||
## To take advantage of v0.22.15
|
||||
|
||||
`gbrain upgrade` should do this automatically. Then:
|
||||
|
||||
1. **Run a dry-run preview:**
|
||||
```bash
|
||||
gbrain frontmatter generate ~/brain
|
||||
```
|
||||
You'll see how many files would get inferred frontmatter and the breakdown by type.
|
||||
2. **Optionally write back to git:**
|
||||
```bash
|
||||
gbrain frontmatter generate ~/brain --fix
|
||||
```
|
||||
Each modified file gets a `.bak` backup before rewrite.
|
||||
3. **Re-sync to pick up the new metadata:**
|
||||
```bash
|
||||
gbrain sync ~/brain
|
||||
```
|
||||
Inferred frontmatter is folded into `content_hash`, so previously-bare files re-import once with proper types and re-embed. Subsequent syncs are idempotent.
|
||||
4. **If anything looks off,** please file an issue: https://github.com/garrytan/gbrain/issues with the path of the misclassified file and the rule that matched.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
#### Features
|
||||
- `src/core/frontmatter-inference.ts` (new module) — Path-aware frontmatter synthesis. `DIRECTORY_RULES` table maps path prefixes to type/date/title/source/tags. First-match-wins. Date extraction from filenames (`YYYY-MM-DD` prefix or anywhere). Title extraction with date-prefix stripping and first-`#`-heading fallback (20-line window). YAML-safe serialization with quoting for special characters.
|
||||
- `src/core/import-file.ts` — `importFromFile()` runs inference inline before `parseMarkdown()` when `opts.inferFrontmatter !== false` (default on). The synthesized frontmatter folds into the in-memory content for parsing, chunking, embedding, and content-hash computation. The file on disk is not modified.
|
||||
- `src/commands/frontmatter.ts` — New `gbrain frontmatter generate <path> [--fix] [--dry-run] [--json]` subcommand. Walks a directory (skips `.git`, `node_modules`, `.obsidian`, symlinks), runs inference on every `.md` file without frontmatter, optionally writes back with `.bak` backups. Auto-detects brain root by walking up for `.git`. Shows per-type breakdown and first-10 examples.
|
||||
|
||||
#### Fixes
|
||||
- `src/commands/frontmatter.ts:344` — `runGenerate` dynamic path import now includes `basename`. Single-file invocation (`gbrain frontmatter generate <file>`) previously crashed with `ReferenceError: basename is not defined` on the relative-path-empty fallback at line 437.
|
||||
|
||||
#### Tests
|
||||
- `test/frontmatter-inference.test.ts` (new, 35 cases) — date extraction (5), title extraction from filenames (5) and headings (4 incl. 20-line boundary), inference for every directory rule (13 incl. Apple Notes subfolder tagging), serialization with YAML-safe quoting (4), `applyInference` integration (2), rule ordering and catch-all coverage (2).
|
||||
|
||||
## [0.22.14] - 2026-04-29
|
||||
|
||||
**Bare `gbrain jobs work` now self-monitors and fail-stops cleanly when its database dies or the queue stalls.**
|
||||
**The wedged-worker class of bug — process alive, jobs piling up, your `pgrep` check happily green — is gone.**
|
||||
|
||||
A production brain (54K pages, Supabase Postgres, 3-concurrency worker under a cron-based PM)
|
||||
hit it last week: worker process state=Sl at 13:15 UTC, stopped claiming jobs, 21 jobs stacked
|
||||
in `waiting` over two hours, 5 autopilot-cycles dead-lettered at the 600s timeout, then 150
|
||||
zombie processes accumulated over the container's 31-day life. The PM's `pgrep` saw a live
|
||||
PID and reported green the entire time.
|
||||
|
||||
Pre-v0.22.14, bare `gbrain jobs work` had **zero** health monitoring. The supervisor (`gbrain
|
||||
jobs supervisor`) had the right protections — DB liveness probes, stall detection, RSS
|
||||
watchdog, reconnect on transient PgBouncer blips — but the supervisor wraps `jobs work` as a
|
||||
child, and many production deployments run bare `jobs work` directly under systemd, Docker,
|
||||
launchd, cron watchdog, or supervisord. That mode got nothing.
|
||||
|
||||
This release moves health monitoring into the bare worker itself, gated by `GBRAIN_SUPERVISED=1`
|
||||
so it doesn't double up under the supervisor. When the worker detects it's wedged, it emits an
|
||||
`'unhealthy'` event with a structured reason, and the CLI calls `process.exit(1)` so the external
|
||||
PM restarts it cleanly. **This is fail-stop:** the worker exits and stays dead until your PM
|
||||
brings it back. If you run bare `jobs work` without a restart loop, you need one now.
|
||||
|
||||
### The numbers that matter
|
||||
|
||||
Detection signatures the new health check catches, measured against the production incident
|
||||
above (and the 30-day deployment running under the band-aid bash watchdog Garry deployed before
|
||||
this fix):
|
||||
|
||||
| Failure mode | Before v0.22.14 | After v0.22.14 |
|
||||
|---|---|---|
|
||||
| DB connection death (Supabase/PgBouncer drop) | undetected; worker idles forever | 3 consecutive `SELECT 1` failures (≤3min) → `'unhealthy'`+exit |
|
||||
| Hung DB probe (network partition) | timer wedged forever, monitoring silently disabled | 10s probe timeout per tick → counted as failure → exit at strike 3 |
|
||||
| Worker stall (event loop alive, claim returns null) | undetected; jobs pile up in `waiting` | 5min warn, 10min `'unhealthy'`+exit (measured from last completion) |
|
||||
| Memory leak (RSS climbing past 2GB) | undetected on bare workers | watchdog default 2048 MB triggers `gracefulShutdown('watchdog')` |
|
||||
| Worker stalled but waiting jobs are unhandled type | ❌ false-positive exit (restart loop) | filter by registered handler names, no exit |
|
||||
|
||||
Operationally: from the band-aid bash watchdog Garry deployed before this fix, fresh worker
|
||||
restart cleared 21 waiting → 0 in 2 minutes, then ran stable for 30+ min with 130 MB RSS,
|
||||
autopilot-cycles completing in 0.2–0.6s instead of timing out at 600s.
|
||||
|
||||
### What this means for operators
|
||||
|
||||
Add a restart policy to your bare-worker invocation BEFORE upgrading. The new behavior is
|
||||
fail-stop, not self-healing — without a restart loop, your worker will exit on the first DB
|
||||
blip and stay dead. systemd `Restart=always`, Docker `restart: always`, launchd `KeepAlive`,
|
||||
cron watchdog, supervisord `autorestart=true`. The migration walks every PM. If you're using
|
||||
`gbrain jobs supervisor`, you're already protected — the supervisor handles spawn-on-crash
|
||||
itself.
|
||||
|
||||
The default `--max-rss` for bare workers also bumped from 0 (off) to 2048 MB. If you ran bare
|
||||
workers with intentionally large embed/import jobs, raise the limit (`--max-rss 4096`) or opt
|
||||
out (`--max-rss 0`). The migration includes per-PM unit-file edits.
|
||||
|
||||
## To take advantage of v0.22.14
|
||||
|
||||
`gbrain upgrade` should do this automatically. If it didn't, or if `gbrain doctor` warns about
|
||||
a bare worker exiting with watchdog signatures:
|
||||
|
||||
1. **Confirm your bare-worker invocations have a restart policy:**
|
||||
```bash
|
||||
# systemd
|
||||
grep -E '^Restart=' ~/.config/systemd/user/gbrain-worker.service /etc/systemd/system/gbrain-worker.service 2>/dev/null
|
||||
# crontab
|
||||
crontab -l | grep "gbrain jobs work"
|
||||
# launchctl
|
||||
plutil -p ~/Library/LaunchAgents/com.user.gbrain-worker.plist | grep -A1 KeepAlive
|
||||
```
|
||||
2. **Decide on RSS posture:**
|
||||
- Default 2048 MB matches supervisor behavior. Most bare workers fit.
|
||||
- Embed/import jobs > 2GB? Pass `--max-rss 4096` (or higher).
|
||||
- Intentionally unbounded? Pass `--max-rss 0`.
|
||||
3. **Walk the migration:** `skills/migrations/v0.22.14.md` has the full per-PM table and a
|
||||
verification block.
|
||||
4. **Verify:**
|
||||
```bash
|
||||
gbrain jobs stats
|
||||
gbrain doctor --json | jq '.checks[] | select(.name == "queue_health")'
|
||||
```
|
||||
Worker startup line should now read:
|
||||
`Minion worker started (queue: default, concurrency: 3, watchdog: 2048MB, health-check: 60s)`
|
||||
Under supervisor: the `health-check: Ns` segment is absent (supervisor handles it).
|
||||
5. **If anything fails or numbers look wrong**, file an issue at
|
||||
https://github.com/garrytan/gbrain/issues with `gbrain doctor` output and the contents of
|
||||
`~/.gbrain/upgrade-errors.jsonl` if it exists.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
#### Added
|
||||
- `MinionWorkerOpts.{healthCheckInterval, stallWarnAfterMs, stallExitAfterMs, dbFailExitAfter, dbProbeTimeoutMs}` — five new tuning knobs. Defaults: 60s probe interval, 5min warn / 10min exit, 3 DB strikes, 10s per-probe timeout.
|
||||
- `MinionWorker` now extends `EventEmitter`. Emits `'unhealthy'` with `{ reason: 'db_dead', consecutiveFailures, message } | { reason: 'stalled', waitingCount, idleMinutes }`. CLI subscribes; direct API consumers without a listener inherit a fail-stop fallback that calls `process.exit(1)` to preserve pre-refactor semantics.
|
||||
- `gbrain jobs work --health-interval MS` — tune the self-health-check cadence (0 disables; rejects NaN/negative/sub-1000ms typos).
|
||||
- `gbrain jobs supervisor --health-interval MS` — same flag, same validation, same `0 = disable` contract on the supervisor's own probe.
|
||||
- `GBRAIN_SUPERVISED=1` env var on the supervisor's spawned worker child (skips the child's self-health timer to avoid double-monitoring).
|
||||
- `gbrain doctor` `queue_health` subcheck reports RSS-watchdog kills in the last 24h via exact match on `error_text = 'aborted: watchdog'` scoped to `status IN ('dead','failed')`.
|
||||
- `skills/migrations/v0.22.14.md` — full migration walkthrough with per-PM restart-policy preflight, RSS-posture decision tree, and per-system unit-file edits.
|
||||
|
||||
#### Changed
|
||||
- **Default `--max-rss` for `gbrain jobs work`: 0 → 2048 MB.** Matches supervisor default. Catches memory-leak stalls that previously went undetected on bare workers. Opt out with `--max-rss 0`.
|
||||
- **Bare-worker behavior is now fail-stop** when the DB is unreachable or the queue stalls. Pre-v0.22.14 the worker idled silently. Now it exits and relies on the external PM (systemd, Docker, launchd, cron, supervisord) to restart cleanly.
|
||||
- Stall query at `worker.ts` filters by registered handler names (`AND name = ANY($2::text[])`) so workers don't false-positive when waiting jobs of unhandled names accumulate.
|
||||
- Stall exit threshold measured from `lastCompletionTime` (not from when the warning fired), so 5min warn / 10min exit means total idle of 10 min — not 15 min.
|
||||
- DB liveness probe wrapped in `Promise.race` against a 10s timeout so a hung `executeRaw` cannot wedge the recursive `setTimeout` chain forever.
|
||||
- `setInterval` → recursive `setTimeout` with a `running` flag throughout. Eliminates timer-callback overlap on slow probes.
|
||||
- `parseMaxRssFlag` returns `number | undefined` (was `number`) so callers distinguish absent from explicit-disable.
|
||||
- `process.env.GBRAIN_SUPERVISED` check tightened from `!!env.X` to `=== '1'` (precise contract; no fuzzy matching on `'0'` or `'false'`).
|
||||
- `MinionWorker` constructor throws when `stallExitAfterMs <= stallWarnAfterMs` so misconfigurations fail loudly at startup.
|
||||
|
||||
#### Fixed
|
||||
- **Wedged-worker false-positive on heterogeneous queues** — workers registering only some handlers no longer interpret waiting jobs of other names as a stall. Repeated `process.exit(1)` → restart loop is gone.
|
||||
- **Hung DB probe wedge** — pre-fix, a hung `executeRaw('SELECT 1')` kept the recursive `setTimeout` from rescheduling, silently disabling the entire health monitor. Post-fix, the probe times out and counts as a failure.
|
||||
- **`--health-interval 0` no longer DB-hammers the supervisor.** Pre-fix, the documented "0 disables" contract was a lie — `setInterval(cb, 0)` schedules a tight loop. Now gated behind `> 0`.
|
||||
- **Inline `jobs submit --follow` and `jobs smoke` no longer kill the user's CLI session** on a DB blip. Both now pass `healthCheckInterval: 0` so the no-listener fallback can't trip on one-shot runs.
|
||||
- Doctor's RSS-watchdog hint matches the actual error_text signature (`'aborted: watchdog'`) instead of the wrong `'memory limit'` literal that never matched.
|
||||
|
||||
#### For contributors
|
||||
- `MinionWorker extends EventEmitter` — if you import the class directly, the `on('unhealthy', ...)` event is now part of the public surface. The `UnhealthyReason` discriminated union is exported from `src/core/minions/worker.ts`.
|
||||
- New regression-test infrastructure in `test/minions.test.ts`: `makeProbeEngine(overrides)` is a Proxy-based engine wrapper that intercepts `SELECT 1` and the stall `count(*)` query while passing every other call through to the real PGLite engine. Useful for any future test that needs to inject DB liveness or stall semantics without mocking the entire engine surface.
|
||||
|
||||
### Adjacent (separate PR, v0.22.15)
|
||||
|
||||
PR #503 catches the *symptom* of one specific failure mode. The cause-side fix — `runPhaseEmbed → embed.ts → embedBatch` not honoring `signal.aborted` between OpenAI batch calls — ships in v0.22.15 (highest-priority TODO; daily wedge driver). Plumbing is documented in `TODOS.md`.
|
||||
|
||||
## [0.22.13] - 2026-04-28
|
||||
|
||||
**Sync got faster, and the bookmark stopped lying.**
|
||||
**Parallel imports, a real writer lock, and a head-drift gate that catches the worst race.**
|
||||
|
||||
The headline is `gbrain sync --workers N`: per-worker Postgres engines with an atomic queue index, same pattern as `gbrain import --workers N`. On a 7,000-page brain that used to take 25+ minutes, the import phase now runs across 4 workers by default. The reproducible benchmark in `test/e2e/sync-parallel.test.ts` shows `parallel(4)` finishing 1.3× faster than serial on a 120-file fixture against local Postgres (`serial=289ms parallel(4)=221ms`). The speedup grows on larger brains and slower-roundtrip databases (Supabase, remote PgBouncer) because the worker setup cost amortizes over more files. But the bigger story is that the sync writer is finally exclusive across processes, and the `last_commit` bookmark refuses to advance when git HEAD has drifted out from under us. The silent-skip-then-advance pathology has survived every prior sync hardening pass. It is dead now.
|
||||
|
||||
### What you can do now
|
||||
|
||||
- `gbrain sync --workers 4` (alias `--concurrency 4`) parallelizes the import phase. Each worker holds 2 connections, so total Postgres connections during the parallel phase is `workers * 2` plus your caller's pool. At the default of 4 workers and a 10-connection caller pool, that's up to 18 connections, well under PgBouncer's `max_client_conn` default of 100 but worth knowing on tight Supabase tiers.
|
||||
- **Auto-concurrency:** if you don't pass `--workers`, sync uses 4 workers when the diff exceeds 100 files. Smaller diffs stay serial. Explicit `--workers` always wins (even on a 30-file diff). PGLite forces serial regardless, since it's a single-connection engine.
|
||||
- **Full sync** routes through the same path. First syncs on large brains parallelize automatically.
|
||||
- **Minion `sync` jobs** also use the new `autoConcurrency()` policy. Behavior is now consistent between CLI sync, the Minion handler, and the autopilot cycle's sync phase. (`noEmbed` defaults to `true` in the jobs handler. Submit `gbrain embed --stale` as a separate job when needed, or rely on the autopilot cycle's embed phase.)
|
||||
- **`--workers` validation is loud now.** `--workers 0`, `--workers -3`, `--workers foo`, `--workers 1.5` all exit with an error message. The prior behavior silently fell through to auto-concurrency (4 workers), the opposite of what you typed.
|
||||
|
||||
### Correctness fixes you didn't have to ask for
|
||||
|
||||
- **Cross-process writer lock.** Two `gbrain sync` calls (manual + autopilot, two terminals, two Conductor workspaces) used to read the same `last_commit`, both write it, and let the last writer win. The new `gbrain-sync` row in `gbrain_cycle_locks` serializes the writer window. Same-process reentrance from the autopilot cycle handler was already covered by the broader `gbrain-cycle` lock; sync's lock is narrower and runs underneath it.
|
||||
- **Head-drift gate.** If `git checkout` or `git pull` runs in your worktree mid-sync (Conductor sibling workspace, ad-hoc terminal), the captured `headCommit` no longer matches HEAD when sync finishes. `last_commit` no longer advances in that case. The next sync re-walks the diff against the new HEAD instead of silently moving the bookmark past unimported work.
|
||||
- **Vanished files now block bookmark advance.** A file the diff said exists at `headCommit` but is gone from disk used to register as a benign skip. It now goes into `failedFiles` and gates `last_commit` the same way a parse failure does.
|
||||
- **Per-source bookmark for Minion `sync` jobs.** The job handler now resolves `sourceId` from the repo path (mirrors the autopilot cycle's `cycle.ts` fix from PR #475). On multi-source brains, this prevents the 30-min full-reimport-every-cycle behavior caused by reading the global `config.sync.last_commit` anchor when the per-source row would have been correct.
|
||||
- **Worker connection cleanup.** Worker engines now disconnect inside `try/finally`, even on partial connect failure or mid-import error. The prior `Promise.all(...disconnect)` ran outside any try/finally, so panic-path leaks never released the 8 worker connections.
|
||||
- **Engine detection unified.** Both PGLite-detection sites in sync.ts now use `engine.kind === 'pglite'` (the discriminator added in v0.13.1). The `engine.constructor.name === 'PGLiteEngine'` sniff is gone, since it broke under bundling and was inconsistent with the other site's `config.engine` string check.
|
||||
|
||||
### What this means for you
|
||||
|
||||
If you run autopilot on a 7,000-page Postgres brain, your sync cycle gets faster on day one with no flags. If you have ever felt the bookmark "skip past" work that didn't import, you'll stop seeing it. If you have multiple Conductor workspaces poking the same brain, you'll either wait politely on the writer lock or get a clear "another sync is in progress" error. None of this requires a config change.
|
||||
|
||||
## To take advantage of v0.22.13
|
||||
|
||||
`gbrain upgrade` should do this automatically. If you want to use the new flags right now:
|
||||
|
||||
1. **For a one-off speed win on a large brain:**
|
||||
```bash
|
||||
gbrain sync --workers 4
|
||||
```
|
||||
Or for incremental syncs that touch >100 files, just run `gbrain sync`. Auto-concurrency fires.
|
||||
|
||||
2. **For your autopilot cycle:** no action. The Minion `sync` handler picks up the new auto-concurrency policy automatically.
|
||||
|
||||
3. **Verify the writer lock is working:**
|
||||
```bash
|
||||
gbrain sync &
|
||||
gbrain sync # second call will say "Another sync is in progress" or wait
|
||||
```
|
||||
|
||||
4. **If sync ever errors with "Another sync is in progress" and stays stuck:** the lock is in `gbrain_cycle_locks` with id `gbrain-sync` and a 30-minute TTL. If a worker crashed without releasing, the next acquirer takes over once the TTL expires. To unstick faster:
|
||||
```sql
|
||||
DELETE FROM gbrain_cycle_locks WHERE id = 'gbrain-sync';
|
||||
```
|
||||
|
||||
5. **If anything looks wrong,** file an issue: https://github.com/garrytan/gbrain/issues with output of `gbrain doctor` and the contents of `~/.gbrain/upgrade-errors.jsonl` if it exists.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
- `src/commands/sync.ts`: `performSync` now wraps body in a `gbrain-sync` DB lock; `--workers` honored regardless of file count when explicit; head-drift gate after import phase; engine.kind detection; try/finally around worker engines; banner moved to stderr.
|
||||
- `src/commands/import.ts`: `engine.kind === 'pglite'` discriminator; try/finally around worker engines; shared `parseWorkers()` for `--workers` validation.
|
||||
- `src/commands/jobs.ts`: sync handler resolves `sourceId` via `sources.local_path` lookup; concurrency routed through `autoConcurrency()`; `noEmbed: true` default documented.
|
||||
- `src/core/sync-concurrency.ts` (new): `autoConcurrency()` + `parseWorkers()` + constants. One source of truth for the concurrency policy that previously lived in three call sites.
|
||||
- `src/core/db-lock.ts` (new): generic `tryAcquireDbLock(engine, lockId)` over the existing `gbrain_cycle_locks` table. Reused by performSync. cycle.ts continues to use its own ID `gbrain-cycle` so the two locks nest cleanly.
|
||||
- `test/sync-concurrency.test.ts` (new): 17 cases covering autoConcurrency thresholds, shouldRunParallel gates, parseWorkers validation.
|
||||
- `test/sync-parallel.test.ts` (new): PGLite-routed coverage of the bookmark gate under concurrency request, the head-drift gate, the writer-lock contract, and PGLite-stays-serial.
|
||||
- `test/e2e/sync-parallel.test.ts` (new): DATABASE_URL-gated Postgres E2E. 60-file happy path with `pg_stat_activity` leak probe, plus a 120-file serial-vs-parallel benchmark that prints `SYNC_PARALLEL_BENCH ...` for CHANGELOG quoting.
|
||||
|
||||
### For contributors
|
||||
|
||||
- `BrainEngine.kind` is now the canonical PGLite/Postgres discriminator. Avoid `engine.constructor.name === '...'` (breaks under bundling) and `config.engine === '...'` (inconsistent with the engine actually in use).
|
||||
- The `gbrain_cycle_locks` table is now multi-purpose. The id column distinguishes lock scopes: `gbrain-cycle` for the cycle, `gbrain-sync` for the sync writer. Future locks should pick distinct ids and reuse `tryAcquireDbLock`.
|
||||
- `parseWorkers()` is the canonical CLI flag parser for `--workers`. Use it instead of inline `parseInt`.
|
||||
|
||||
## [0.22.12] - 2026-04-29
|
||||
|
||||
**`sync --skip-failed` now classifies file-size and symlink rejections instead of bucketing them as UNKNOWN.**
|
||||
|
||||
@@ -92,7 +92,7 @@ strict behavior when unset.
|
||||
- `src/core/minions/attachments.ts` — Attachment validation (path traversal, null byte, oversize, base64, duplicate detection)
|
||||
- `src/commands/agent.ts` (v0.16) — `gbrain agent run <prompt> [flags]` CLI. Submits `subagent` (or N children + 1 aggregator) under `{allowProtectedSubmit: true}`. Single-entry `--fanout-manifest` short-circuits. Children get `on_child_fail: 'continue'` + `max_stalled: 3`. `--follow` is the default on TTY; streams logs + polls `waitForCompletion` in parallel. Ctrl-C detaches, does not cancel.
|
||||
- `src/commands/agent-logs.ts` (v0.16) — `gbrain agent logs <job> [--follow] [--since]`. Merges JSONL heartbeat audit + `subagent_messages` into a chronological timeline. `parseSince` accepts ISO-8601 or relative (`5m`, `1h`, `2d`). Transcript tail renders only for terminal jobs.
|
||||
- `src/commands/jobs.ts` — `gbrain jobs` CLI subcommands + `gbrain jobs work` daemon. v0.13.1 surfaces the full `MinionJobInput` retry/backoff/timeout/idempotency surface as first-class CLI flags on `jobs submit`: `--max-stalled`, `--backoff-type fixed|exponential`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key`. `jobs smoke --sigkill-rescue` is the opt-in regression guard for #219. v0.16 wires `registerBuiltinHandlers` to always register `subagent` + `subagent_aggregator` (no env flag — `ANTHROPIC_API_KEY` is the natural cost gate, trust is via `PROTECTED_JOB_NAMES`) and loads `GBRAIN_PLUGIN_PATH` plugins at worker startup with a loud startup-line per plugin. `shell` handler still gated by `GBRAIN_ALLOW_SHELL_JOBS=1` (RCE surface, separate concern). v0.22.10 (#521): the `autopilot-cycle` handler now forwards `job.data.phases` to `runCycle` (was previously discarded — caller-supplied phase selection silently became a full cycle). Phases are validated against `ALL_PHASES` from `src/core/cycle.ts`; invalid names are filtered out and an empty/missing array falls back to the default 6-phase cycle.
|
||||
- `src/commands/jobs.ts` — `gbrain jobs` CLI subcommands + `gbrain jobs work` daemon. v0.13.1 surfaces the full `MinionJobInput` retry/backoff/timeout/idempotency surface as first-class CLI flags on `jobs submit`: `--max-stalled`, `--backoff-type fixed|exponential`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key`. `jobs smoke --sigkill-rescue` is the opt-in regression guard for #219. v0.16 wires `registerBuiltinHandlers` to always register `subagent` + `subagent_aggregator` (no env flag — `ANTHROPIC_API_KEY` is the natural cost gate, trust is via `PROTECTED_JOB_NAMES`) and loads `GBRAIN_PLUGIN_PATH` plugins at worker startup with a loud startup-line per plugin. `shell` handler still gated by `GBRAIN_ALLOW_SHELL_JOBS=1` (RCE surface, separate concern). v0.22.10 (#521): the `autopilot-cycle` handler now forwards `job.data.phases` to `runCycle` (was previously discarded — caller-supplied phase selection silently became a full cycle). Phases are validated against `ALL_PHASES` from `src/core/cycle.ts`; invalid names are filtered out and an empty/missing array falls back to the default 6-phase cycle. v0.22.13 (PR #490 CODEX-1+CODEX-4): `sync` handler now resolves `sourceId` at entry by looking up `sources.local_path` (mirrors `cycle.ts:480`'s autopilot fix from PR #475) so multi-source brains read the per-source `last_commit` anchor instead of the global config key. Concurrency routed through the shared `autoConcurrency()` policy in `src/core/sync-concurrency.ts` instead of the prior hardcoded `4`; PGLite stays serial. `noEmbed` default is `true` (embed is a separate job — submit `gbrain embed --stale` after sync, or rely on the autopilot cycle's embed phase).
|
||||
- `src/commands/features.ts` — `gbrain features --json --auto-fix`: usage scan + feature adoption salesman
|
||||
- `src/commands/autopilot.ts` — `gbrain autopilot --install`: self-maintaining brain daemon (sync+extract+embed)
|
||||
- `src/mcp/server.ts` — MCP stdio server (generated from operations). v0.22.7: tool-call handler delegates to `dispatchToolCall` from `src/mcp/dispatch.ts` so stdio + HTTP transports share one validation, context-build, and error-format path.
|
||||
@@ -109,6 +109,9 @@ strict behavior when unset.
|
||||
- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2).
|
||||
- `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2.
|
||||
- `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=<ms>` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers.
|
||||
- `src/core/db-lock.ts` (v0.22.13) — generic `tryAcquireDbLock(engine, lockId, ttlMinutes)` over the existing `gbrain_cycle_locks` table. Parameterized lock id so different scopes can nest cleanly: `gbrain-cycle` for the broad cycle (held by `cycle.ts`) and `gbrain-sync` (`SYNC_LOCK_ID` constant) for `performSync`'s narrower writer window. Same UPSERT-with-TTL semantics as the prior cycle-only helper, just generalized. Survives PgBouncer transaction pooling (unlike session-scoped `pg_try_advisory_lock`); crashed holders auto-release once their TTL expires.
|
||||
- `src/core/sync-concurrency.ts` (v0.22.13) — single source of truth for the parallel-sync policy. Exports `autoConcurrency(engine, fileCount, override?)` (PGLite always serial; explicit override clamped to >=1; auto path returns `DEFAULT_PARALLEL_WORKERS=4` when `fileCount > AUTO_CONCURRENCY_FILE_THRESHOLD=100`), `shouldRunParallel(workers, fileCount, explicit)` (Q1: explicit `--workers` bypasses the >50-file floor), and `parseWorkers(s)` (rejects `'0'`, `'-3'`, `'foo'`, `'1.5'`, trailing chars — replaces the prior parseInt-with-no-validation in both `sync.ts` and `import.ts`). Used by `performSync`, `performFullSync`, `runImport`, and the Minion `sync` handler so the three sites can no longer drift.
|
||||
- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). v0.22.13 (PR #490): `performSync` wraps its body in a `gbrain-sync` writer lock so two concurrent syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot both write `last_commit` and let the last writer win. Head-drift gate after the import phase re-checks `git rev-parse HEAD`; if HEAD moved (someone ran `git checkout` / `git pull` mid-sync), the bookmark refuses to advance. Vanished files now record a failedFiles entry instead of silent-skip — the silent-skip-then-advance pathology that survived prior hardening passes is dead. Worker engines wrap in try/finally so disconnect always fires (panic-path leak fix). Both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`), validated via `parseWorkers`. Explicit `--workers` bypasses the auto-path file-count floor; auto path defers to `autoConcurrency()`. Banner moved to stderr.
|
||||
- `src/core/cycle.ts` — v0.17 brain maintenance cycle primitive. `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise<CycleReport>` composes 6 phases in semantically-driven order (lint → backlinks → sync → extract → embed → orphans). Three callers: `gbrain dream` CLI, `gbrain autopilot` daemon's inline path, and the Minions `autopilot-cycle` handler (`src/commands/jobs.ts`). One source of truth for what the brain does overnight. Coordination via `gbrain_cycle_locks` DB table (TTL-based; works through PgBouncer transaction pooling, unlike session-scoped `pg_try_advisory_lock`) + `~/.gbrain/cycle.lock` file lock with PID-liveness for PGLite / engine=null mode. `CycleReport.schema_version: "1"` is the stable agent-consumable shape. `PhaseResult.error: { class, code, message, hint?, docs_url? }` is Stripe-API-tier structured failure info. `yieldBetweenPhases` hook awaited between every phase — Minions handler uses this to renew its job lock and prevent v0.14 stall-death regression. Engine nullable: filesystem phases (lint, backlinks) run without DB; DB phases skip with `status: "skipped", reason: "no_database"`. Lock-skip: read-only phase selections (`--phase orphans`) bypass the cycle lock. v0.22.1 (#403): `CycleOpts.signal?: AbortSignal` propagates the worker's abort signal; `checkAborted()` fires between every phase and throws if the signal is aborted (cooperative — can't interrupt a phase mid-execution). v0.22.1 (#417): `runPhaseSync` returns `pagesAffected` via `SyncPhaseResult`; `runCycle` captures it and threads to `runPhaseExtract` as the 4th arg, enabling incremental extract on the cycle path. v0.22.1 (Codex F2): `runPhaseSync` takes `willRunExtractPhase: boolean` and sets `noExtract: phases.includes('extract')` so `gbrain dream --phase sync` doesn't silently lose extraction. v0.22.5 (#475): new `resolveSourceForDir(engine, brainDir)` helper queries `SELECT id FROM sources WHERE local_path = $1 LIMIT 1`; `runPhaseSync` threads result as `sourceId` to `performSync()` so sync reads the per-source `sources.last_commit` anchor instead of the drift-prone global `config.sync.last_commit` key. Bare try/catch lets pre-v0.18 brains fall through to the global key. Closes the prod hang where every autopilot cycle ran a 30-min full reimport because the global anchor commit had been GC'd from git history.
|
||||
- `src/commands/dream.ts` — v0.17 `gbrain dream` CLI. ~80-line thin alias over `runCycle`. brainDir resolution requires explicit `--dir` OR `sync.repo_path` config (no more walk-up-cwd-for-.git footgun). Flags: `--dry-run`, `--json`, `--phase <name>`, `--pull`, `--dir <path>`. Exit code 1 on status=failed (partial/warn not fatal — don't page on warnings).
|
||||
- `scripts/check-progress-to-stdout.sh` — CI guard against regressing to `\r`-on-stdout progress. Wired into `bun run test` via `scripts/check-progress-to-stdout.sh && bun test` in package.json.
|
||||
@@ -220,6 +223,10 @@ Key commands added in v0.14.3 (fix wave):
|
||||
- `gbrain jobs submit` gains `--max-stalled`, `--backoff-type`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key` — exposing existing `MinionJobInput` fields as first-class CLI flags.
|
||||
- `gbrain jobs smoke --sigkill-rescue` — opt-in regression smoke case simulating a killed worker; asserts the v0.14.3 schema default (`max_stalled=5`) actually rescues on first stall.
|
||||
|
||||
Key commands added in v0.22.13 (PR #490):
|
||||
- `gbrain sync --workers N` (alias `--concurrency N`) — parallelize the import phase using per-worker Postgres engines (small pool of 2 each) with an atomic queue index. Auto-concurrency: defaults to 4 workers when the diff exceeds 100 files. Smaller diffs stay serial. Explicit `--workers` always wins (even on a 30-file diff). PGLite forces serial regardless. Validation rejects `0`, negatives, non-integers loud (replaces the prior silent fall-through to auto-concurrency).
|
||||
- `gbrain import --workers N` — same `parseWorkers()` validation as sync; same try/finally worker-engine cleanup. Behavior surface unchanged.
|
||||
|
||||
## Testing
|
||||
|
||||
`bun test` runs all tests. After the v0.12.1 release: ~75 unit test files + 8 E2E test files (1412 unit pass, 119 E2E when `DATABASE_URL` is set — skip gracefully otherwise). Unit tests run
|
||||
@@ -275,6 +282,8 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac
|
||||
`test/orphans.test.ts` (v0.12.3 orphans command: detection, pseudo filtering, text/json/count outputs, MCP op),
|
||||
`test/postgres-engine.test.ts` (v0.12.3 statement_timeout scoping: `sql.begin` + `SET LOCAL` shape, source-level grep guardrail against reintroduced bare `SET statement_timeout`),
|
||||
`test/sync.test.ts` (sync logic + v0.12.3 regression guard asserting top-level `engine.transaction` is not called),
|
||||
`test/sync-concurrency.test.ts` (v0.22.13 PR #490: 17 cases covering `autoConcurrency()` thresholds + PGLite-forces-serial + explicit-override clamping, `shouldRunParallel()` Q1 explicit-bypasses-floor contract, and `parseWorkers()` validation that rejects `'0'`/`'-3'`/`'foo'`/`'1.5'`/trailing chars),
|
||||
`test/sync-parallel.test.ts` (v0.22.13 PR #490: PGLite-routed coverage of the bookmark gate under concurrency request, head-drift gate, vanished-file failure capture, PGLite-stays-serial, and the `gbrain-sync` writer-lock contract — 7 cases),
|
||||
`test/sync-failures.test.ts` (v0.22.12: 28 cases pinning `classifyErrorCode` regex coverage for all 12 codes against literal production message strings from `markdown.ts:159-244` and `import-file.ts:199, 347, 352, 401`; `summarizeFailuresByCode` sort + pre-classified-honor; `recordSyncFailures` code-field persistence; `acknowledgeSyncFailures` AcknowledgeResult shape + backfill on pre-v0.22.12 entries),
|
||||
`test/doctor.test.ts` (doctor command + v0.12.3 assertions that `jsonb_integrity` scans the four v0.12.0 write sites and `markdown_body_completeness` is present),
|
||||
`test/utils.test.ts` (shared SQL utilities + `tryParseEmbedding` null-return and single-warn semantics),
|
||||
@@ -305,6 +314,7 @@ E2E tests (`test/e2e/`): Run against real Postgres+pgvector. Require `DATABASE_U
|
||||
- `test/e2e/engine-parity.test.ts` (v0.22.0) — Postgres ↔ PGLite top-result and result-set parity for `searchKeyword` + `searchVector`. Codex flagged that Postgres ranks pages then picks best chunk while PGLite returns chunks directly — without parity coverage the source-boost fix could pass on PGLite and fail on Postgres. Skips gracefully when `DATABASE_URL` is unset.
|
||||
- `test/e2e/postgres-bootstrap.test.ts` (v0.22.6.1) — exercises `PostgresEngine.initSchema()` directly against a fresh real Postgres database. Asserts the bootstrap path is no-op on fresh installs and that SCHEMA_SQL replays cleanly through the engine path (not via the standalone `db.initSchema` from `src/core/db.ts`, which would have produced false-positive coverage). Codex caught the E2E-shape gap during plan review.
|
||||
- `test/e2e/http-transport.test.ts` (v0.22.7) — 8 cases against real Postgres covering `gbrain serve --http` end-to-end: bearer auth round-trip, `last_used_at` SQL-level debounce semantics, `mcp_request_log` row insertion on success and auth_failed paths, `/health` DB-down → 503 (DB-probing health check), and the F1+F2+F3 dispatch round-trip with a real operation. Skips gracefully when `DATABASE_URL` is unset.
|
||||
- `test/e2e/sync-parallel.test.ts` (v0.22.13 PR #490) — DATABASE_URL-gated. T2: 60-file Postgres sync at concurrency=4 imports all + no connection leak (probes `pg_stat_activity` before/after to confirm worker engines disconnected). P4: 120-file serial-vs-parallel benchmark prints `SYNC_PARALLEL_BENCH N files | serial=Xms | parallel(4)=Yms | speedup=Zx` for CHANGELOG quoting. Asserts parallel ≤ serial × 1.5 (CI-noise tolerant; not a strict speedup gate).
|
||||
- Tier 2 (`skills.test.ts`) requires OpenClaw + API keys, runs nightly in CI
|
||||
- If `.env.testing` doesn't exist in this directory, check sibling worktrees for one:
|
||||
`find ../ -maxdepth 2 -name .env.testing -print -quit` and copy it here if found.
|
||||
|
||||
@@ -639,8 +639,11 @@ SEARCH
|
||||
gbrain query <question> Hybrid search (vector + keyword + RRF)
|
||||
|
||||
IMPORT
|
||||
gbrain import <dir> [--no-embed] Import markdown (idempotent)
|
||||
gbrain sync [--repo <path>] Git-to-brain incremental sync
|
||||
gbrain import <dir> [--no-embed] [--workers N]
|
||||
Import markdown (idempotent)
|
||||
gbrain sync [--repo <path>] [--workers N]
|
||||
Git-to-brain incremental sync
|
||||
(>100-file diffs auto-parallelize 4 workers on Postgres)
|
||||
gbrain export [--dir ./out/] Export to markdown
|
||||
|
||||
FILES
|
||||
|
||||
@@ -1,5 +1,158 @@
|
||||
# TODOS
|
||||
|
||||
## minions / worker (v0.22.14 follow-ups)
|
||||
|
||||
### v0.22.15 — Embed cooperative-abort (HIGHEST PRIORITY — daily pain)
|
||||
**Priority:** P0
|
||||
|
||||
**What:** Plumb `signal: AbortSignal` through `runPhaseEmbed` →
|
||||
`src/commands/embed.ts` → `embedBatch` in `src/core/embedding.ts`. Check
|
||||
`signal?.aborted` between OpenAI batch calls (every ~100 texts, ~2s
|
||||
real-time) and between slugs in the per-slug loop.
|
||||
|
||||
**Why:** Embed phase ignores `signal.aborted` between batches today. Job
|
||||
wall-clock timeout fires → handler keeps running → cycle's finally block
|
||||
unreachable → `gbrain_cycle_locks` row stays held indefinitely. Every
|
||||
subsequent autopilot cron cycle sees `cycle_already_running` → skips. Lock
|
||||
TTL is 30 min; new cycles give up before that. Doctor reports UNHEALTHY.
|
||||
|
||||
**The chain in production:** ~5min cron submits cycle → 22K stale pages →
|
||||
embed phase takes 10–15 min → 600s timeout fires → job dead-lettered → embed
|
||||
keeps running → lock held → all subsequent cycles skip. Garry hits this
|
||||
DAILY on his production brain.
|
||||
|
||||
**Pros:** Closes the daily wedge. Makes timeouts actually effective. Lets
|
||||
operators bump worker timeouts confidently knowing abort actually stops
|
||||
work.
|
||||
|
||||
**Cons:** Touching the embed hot path; small risk of botching the abort
|
||||
checks. Mitigation: between-batch granularity (~2s), not per-text (too fine)
|
||||
or per-slug (too coarse for 500+ chunk slugs).
|
||||
|
||||
**Context:** PR #503 (v0.22.14) catches the SYMPTOM (worker stalled, queue
|
||||
piling up) via self-health-monitoring. This PR catches the CAUSE for one
|
||||
specific failure class. Both fixes are needed; they're complementary, not
|
||||
duplicative.
|
||||
|
||||
**Files to touch:**
|
||||
- `src/core/cycle.ts:579` — `runPhaseEmbed(engine, dryRun)` → add
|
||||
`signal?: AbortSignal` arg
|
||||
- `src/core/cycle.ts:803` — pass `opts.signal` through
|
||||
- `src/commands/embed.ts:~363` — accept signal, check between slugs
|
||||
- `src/core/embedding.ts:51-56` — `embedBatch(texts, onProgress?, signal?)`,
|
||||
check between for-loop iterations of `BATCH_SIZE` slices
|
||||
|
||||
**Tests required:**
|
||||
1. embedBatch checks signal between OpenAI calls; aborts within one batch (~2s)
|
||||
2. Per-slug loop in `embed.ts` checks signal between slugs
|
||||
3. End-to-end: cycle handler with embed phase + signal aborted mid-flight →
|
||||
finally runs → `gbrain_cycle_locks` row deleted
|
||||
4. Regression: 1K+ chunks scenario — embed does NOT block lock release when
|
||||
timeout fires
|
||||
|
||||
**Effort:** M (human: ~3 hr / CC: ~30 min).
|
||||
|
||||
**Depends on / blocked by:** Nothing. v0.22.14 ships first.
|
||||
|
||||
### v0.23+ — Bare-worker engine reconnect parity with supervisor
|
||||
**Priority:** P2
|
||||
|
||||
**What:** Extract the supervisor's reconnect-then-fail pattern into
|
||||
`MinionWorker` so bare workers can retry transient DB blips before exiting.
|
||||
Today the supervisor calls `engine.reconnect()` after 3 consecutive DB health
|
||||
failures (#406); the bare worker just emits `'unhealthy'` and the CLI calls
|
||||
`process.exit(1)`.
|
||||
|
||||
**Why:** Bare-worker behavior is more disruptive than supervised behavior on
|
||||
transient PgBouncer blips. A bare worker restarts the entire process; a
|
||||
supervised worker just reconnects the pool. Operationally the supervisor
|
||||
approach is gentler (no in-flight job loss, no PM restart latency).
|
||||
|
||||
**Pros:** Unifies bare and supervised behavior. Reduces process churn on
|
||||
transient network blips.
|
||||
|
||||
**Cons:** More code in MinionWorker; risk of reconnect masking a real
|
||||
problem. Mitigation: cap retry attempts, fall through to `'unhealthy'`
|
||||
emission after the cap.
|
||||
|
||||
**Context:** Filed during v0.22.14 plan-eng-review. The asymmetry is
|
||||
documented in v0.22.14 CHANGELOG as deliberate; this TODO captures the
|
||||
"unify someday" intent.
|
||||
|
||||
**Effort:** S (human: ~2 hr / CC: ~20 min).
|
||||
|
||||
**Depends on / blocked by:** Nothing.
|
||||
|
||||
### v0.23+ — `minion_workers` heartbeat table for queue_health doctor (B7)
|
||||
**Priority:** P3
|
||||
|
||||
**What:** Add a `minion_workers` table (`worker_id` PK, `hostname`,
|
||||
`last_heartbeat`, `queue`, `concurrency`, `started_at`) so the existing
|
||||
`queue_health` doctor check (Postgres path) can detect dead workers via
|
||||
heartbeat staleness instead of relying on the indirect `lock_until` proxy.
|
||||
|
||||
**Why:** v0.19.1 added `queue_health` checks for stalled-active jobs and
|
||||
waiting-depth threshold. The worker-heartbeat subcheck was deferred (B7)
|
||||
because the `lock_until`-on-active-jobs proxy can't distinguish "worker
|
||||
exited cleanly" from "worker idle" — a check that cries wolf erodes trust
|
||||
in every doctor check. With a real heartbeat row, doctor can say "no worker
|
||||
seen in N intervals" with confidence.
|
||||
|
||||
**Pros:** Doctor's `queue_health` becomes ground-truth. Detects "worker
|
||||
container died but cron didn't restart it" scenario.
|
||||
|
||||
**Cons:** New table, schema migration, every health-tick UPSERTs. Costs
|
||||
a write per worker per minute (default).
|
||||
|
||||
**Context:** Filed during v0.22.14 plan-eng-review. PR #503's self-health
|
||||
monitoring is the worker-side liveness; this would be the queue-side
|
||||
ground-truth.
|
||||
|
||||
**Effort:** M (human: ~1 day / CC: ~1 hr).
|
||||
|
||||
**Depends on / blocked by:** Schema migration system; nothing else.
|
||||
|
||||
## sync (v0.22.13 follow-up — PR #490 review)
|
||||
|
||||
### D-PR490-1 — Plumb resolved `database_url` through `SyncOpts`
|
||||
**Priority:** P3
|
||||
|
||||
**What:** Add `database_url?: string` (or a richer `resolvedConnection` shape) to
|
||||
`SyncOpts` and have the caller (`runSync`, the cycle handler, the jobs handler)
|
||||
populate it from the active engine instead of having `performSync` /
|
||||
`performFullSync` / `import.ts` each call `loadConfig()` separately. Today every
|
||||
sync run hits the config file three times.
|
||||
|
||||
**Why:** v0.18 multi-source brains can in principle run different sources against
|
||||
different `database_url` endpoints (or different per-source overrides via
|
||||
`sources.config_jsonb`). Right now `loadConfig()` returns the global config, and
|
||||
that always matches the engine in practice — but the convention papers over a
|
||||
real divergence the moment someone wants per-source connection settings. Folding
|
||||
the resolution into `SyncOpts` makes the worker-engine creation in `sync.ts` and
|
||||
`import.ts` deterministic from `SyncOpts` alone.
|
||||
|
||||
**Pros:**
|
||||
- Removes 3 redundant `loadConfig()` calls per sync.
|
||||
- Makes `performSync` / `performFullSync` side-effect-free with respect to the
|
||||
on-disk config file.
|
||||
- Sets up for per-source `database_url` overrides without further refactor.
|
||||
- Makes the v0.22.13 belt-and-suspenders fallback (PR #490 Q3) cleaner — no
|
||||
more `!config?.database_url` short-circuit inside the parallel branch.
|
||||
|
||||
**Cons:**
|
||||
- API-shape change to `SyncOpts` (mild; not externally exported).
|
||||
- Touching three callers (`runSync`, jobs handler, `cycle.ts` `runPhaseSync`).
|
||||
- Only worth doing when paired with a per-source override story; otherwise
|
||||
it's just plumbing.
|
||||
|
||||
**Context:** Surfaced during the PR #490 plan-eng-review (parallel sync).
|
||||
Deferred because it isn't on the v0.22.13 critical path. The same pattern would
|
||||
benefit the cycle handler and the autopilot daemon. See the plan-eng-review
|
||||
decisions log: A4 = "Defer; file as TODO."
|
||||
|
||||
**Depends on / blocked by:** Nothing structural. Best paired with the v0.18
|
||||
per-source `config_jsonb` work if/when that lands.
|
||||
|
||||
## sync error-code classification (PR #501 follow-ups)
|
||||
|
||||
### Plumb structured `ParseValidationCode` through `ImportResult`
|
||||
|
||||
@@ -20,6 +20,7 @@
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/bun": "latest",
|
||||
"bun-types": "^1.3.13",
|
||||
"typescript": "^5.6.0",
|
||||
},
|
||||
},
|
||||
@@ -220,7 +221,7 @@
|
||||
|
||||
"@types/bun": ["@types/bun@1.3.11", "", { "dependencies": { "bun-types": "1.3.11" } }, "sha512-5vPne5QvtpjGpsGYXiFyycfpDF2ECyPcTSsFBMa0fraoxiQyMJ3SmuQIGhzPg2WJuWxVBoxWJ2kClYTcw/4fAg=="],
|
||||
|
||||
"@types/node": ["@types/node@18.19.130", "", { "dependencies": { "undici-types": "~5.26.4" } }, "sha512-GRaXQx6jGfL8sKfaIDD6OupbIHBr9jv7Jnaml9tB7l4v068PAOXqfcujMMo5PhbIs6ggR1XODELqahT2R8v0fg=="],
|
||||
"@types/node": ["@types/node@25.5.2", "", { "dependencies": { "undici-types": "~7.18.0" } }, "sha512-tO4ZIRKNC+MDWV4qKVZe3Ql/woTnmHDr5JD8UI5hn2pwBrHEwOEMZK7WlNb5RKB6EoJ02gwmQS9OrjuFnZYdpg=="],
|
||||
|
||||
"@types/node-fetch": ["@types/node-fetch@2.6.13", "", { "dependencies": { "@types/node": "*", "form-data": "^4.0.4" } }, "sha512-QGpRVpzSaUs30JBSGPjOg4Uveu384erbHBoT1zeONvyCfwQxIkUshLAOqN/k9EjGviPRmWTTe6aH2qySWKTVSw=="],
|
||||
|
||||
@@ -242,7 +243,7 @@
|
||||
|
||||
"bowser": ["bowser@2.14.1", "", {}, "sha512-tzPjzCxygAKWFOJP011oxFHs57HzIhOEracIgAePE4pqB3LikALKnSzUyU4MGs9/iCEUuHlAJTjTc5M+u7YEGg=="],
|
||||
|
||||
"bun-types": ["bun-types@1.3.11", "", { "dependencies": { "@types/node": "*" } }, "sha512-1KGPpoxQWl9f6wcZh57LvrPIInQMn2TQ7jsgxqpRzg+l0QPOFvJVH7HmvHo/AiPgwXy+/Thf6Ov3EdVn1vOabg=="],
|
||||
"bun-types": ["bun-types@1.3.13", "", { "dependencies": { "@types/node": "*" } }, "sha512-QXKeHLlOLqQX9LgYaHJfzdBaV21T63HhFJnvuRCcjZiaUDpbs5ED1MgxbMra71CsryN/1dAoXuJJJwIv/2drVA=="],
|
||||
|
||||
"bytes": ["bytes@3.1.2", "", {}, "sha512-/Nf7TyzTx6S3yRJObOAV7956r8cr2+Oj8AC5dt8wSP3BQAoeX58NoHyCU8P8zGkNXStjTSi6fzO6F0pBdcYbEg=="],
|
||||
|
||||
@@ -466,7 +467,7 @@
|
||||
|
||||
"typescript": ["typescript@5.9.3", "", { "bin": { "tsc": "bin/tsc", "tsserver": "bin/tsserver" } }, "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw=="],
|
||||
|
||||
"undici-types": ["undici-types@5.26.5", "", {}, "sha512-JlCMO+ehdEIKqlFxk6IfVoAUVmgz7cU7zD/h9XZ0qzeosSHmUJVOzSQvvYSYWXkFXC+IfLKSIffhv0sVZup6pA=="],
|
||||
"undici-types": ["undici-types@7.18.2", "", {}, "sha512-AsuCzffGHJybSaRrmr5eHr81mwJU3kjw6M+uprWvCXiNeN9SOGwQ3Jn8jb8m3Z6izVgknn1R0FTCEAP2QrLY/w=="],
|
||||
|
||||
"unpipe": ["unpipe@1.0.0", "", {}, "sha512-pjy2bYhSsufwWlKwPc+l3cN7+wuJlK6uz0YdJEOlQDbl6jo/YlPi4mb8agUkVC8BF7V8NuzeyPNqRksA3hztKQ=="],
|
||||
|
||||
@@ -488,30 +489,32 @@
|
||||
|
||||
"zod-to-json-schema": ["zod-to-json-schema@3.25.2", "", { "peerDependencies": { "zod": "^3.25.28 || ^4" } }, "sha512-O/PgfnpT1xKSDeQYSCfRI5Gy3hPf91mKVDuYLUHZJMiDFptvP41MSnWofm8dnCm0256ZNfZIM7DSzuSMAFnjHA=="],
|
||||
|
||||
"@anthropic-ai/sdk/@types/node": ["@types/node@18.19.130", "", { "dependencies": { "undici-types": "~5.26.4" } }, "sha512-GRaXQx6jGfL8sKfaIDD6OupbIHBr9jv7Jnaml9tB7l4v068PAOXqfcujMMo5PhbIs6ggR1XODELqahT2R8v0fg=="],
|
||||
|
||||
"@aws-crypto/sha1-browser/@smithy/util-utf8": ["@smithy/util-utf8@2.3.0", "", { "dependencies": { "@smithy/util-buffer-from": "^2.2.0", "tslib": "^2.6.2" } }, "sha512-R8Rdn8Hy72KKcebgLiv8jQcQkXoLMOGGv5uI1/k0l+snqkOzQ1R0ChUBCxWMlBsFMekWjq0wRudIweFs7sKT5A=="],
|
||||
|
||||
"@aws-crypto/sha256-browser/@smithy/util-utf8": ["@smithy/util-utf8@2.3.0", "", { "dependencies": { "@smithy/util-buffer-from": "^2.2.0", "tslib": "^2.6.2" } }, "sha512-R8Rdn8Hy72KKcebgLiv8jQcQkXoLMOGGv5uI1/k0l+snqkOzQ1R0ChUBCxWMlBsFMekWjq0wRudIweFs7sKT5A=="],
|
||||
|
||||
"@aws-crypto/util/@smithy/util-utf8": ["@smithy/util-utf8@2.3.0", "", { "dependencies": { "@smithy/util-buffer-from": "^2.2.0", "tslib": "^2.6.2" } }, "sha512-R8Rdn8Hy72KKcebgLiv8jQcQkXoLMOGGv5uI1/k0l+snqkOzQ1R0ChUBCxWMlBsFMekWjq0wRudIweFs7sKT5A=="],
|
||||
|
||||
"@types/node-fetch/@types/node": ["@types/node@25.5.2", "", { "dependencies": { "undici-types": "~7.18.0" } }, "sha512-tO4ZIRKNC+MDWV4qKVZe3Ql/woTnmHDr5JD8UI5hn2pwBrHEwOEMZK7WlNb5RKB6EoJ02gwmQS9OrjuFnZYdpg=="],
|
||||
|
||||
"bun-types/@types/node": ["@types/node@25.5.2", "", { "dependencies": { "undici-types": "~7.18.0" } }, "sha512-tO4ZIRKNC+MDWV4qKVZe3Ql/woTnmHDr5JD8UI5hn2pwBrHEwOEMZK7WlNb5RKB6EoJ02gwmQS9OrjuFnZYdpg=="],
|
||||
"@types/bun/bun-types": ["bun-types@1.3.11", "", { "dependencies": { "@types/node": "*" } }, "sha512-1KGPpoxQWl9f6wcZh57LvrPIInQMn2TQ7jsgxqpRzg+l0QPOFvJVH7HmvHo/AiPgwXy+/Thf6Ov3EdVn1vOabg=="],
|
||||
|
||||
"form-data/mime-types": ["mime-types@2.1.35", "", { "dependencies": { "mime-db": "1.52.0" } }, "sha512-ZDY+bPm5zTTF+YpCrAU9nK0UgICYPT0QtT1NZWFv4s++TNkcgVaT0g6+4R2uI4MjQjzysHB1zxuWL50hzaeXiw=="],
|
||||
|
||||
"openai/@types/node": ["@types/node@18.19.130", "", { "dependencies": { "undici-types": "~5.26.4" } }, "sha512-GRaXQx6jGfL8sKfaIDD6OupbIHBr9jv7Jnaml9tB7l4v068PAOXqfcujMMo5PhbIs6ggR1XODELqahT2R8v0fg=="],
|
||||
|
||||
"@anthropic-ai/sdk/@types/node/undici-types": ["undici-types@5.26.5", "", {}, "sha512-JlCMO+ehdEIKqlFxk6IfVoAUVmgz7cU7zD/h9XZ0qzeosSHmUJVOzSQvvYSYWXkFXC+IfLKSIffhv0sVZup6pA=="],
|
||||
|
||||
"@aws-crypto/sha1-browser/@smithy/util-utf8/@smithy/util-buffer-from": ["@smithy/util-buffer-from@2.2.0", "", { "dependencies": { "@smithy/is-array-buffer": "^2.2.0", "tslib": "^2.6.2" } }, "sha512-IJdWBbTcMQ6DA0gdNhh/BwrLkDR+ADW5Kr1aZmd4k3DIF6ezMV4R2NIAmT08wQJ3yUK82thHWmC/TnK/wpMMIA=="],
|
||||
|
||||
"@aws-crypto/sha256-browser/@smithy/util-utf8/@smithy/util-buffer-from": ["@smithy/util-buffer-from@2.2.0", "", { "dependencies": { "@smithy/is-array-buffer": "^2.2.0", "tslib": "^2.6.2" } }, "sha512-IJdWBbTcMQ6DA0gdNhh/BwrLkDR+ADW5Kr1aZmd4k3DIF6ezMV4R2NIAmT08wQJ3yUK82thHWmC/TnK/wpMMIA=="],
|
||||
|
||||
"@aws-crypto/util/@smithy/util-utf8/@smithy/util-buffer-from": ["@smithy/util-buffer-from@2.2.0", "", { "dependencies": { "@smithy/is-array-buffer": "^2.2.0", "tslib": "^2.6.2" } }, "sha512-IJdWBbTcMQ6DA0gdNhh/BwrLkDR+ADW5Kr1aZmd4k3DIF6ezMV4R2NIAmT08wQJ3yUK82thHWmC/TnK/wpMMIA=="],
|
||||
|
||||
"@types/node-fetch/@types/node/undici-types": ["undici-types@7.18.2", "", {}, "sha512-AsuCzffGHJybSaRrmr5eHr81mwJU3kjw6M+uprWvCXiNeN9SOGwQ3Jn8jb8m3Z6izVgknn1R0FTCEAP2QrLY/w=="],
|
||||
|
||||
"bun-types/@types/node/undici-types": ["undici-types@7.18.2", "", {}, "sha512-AsuCzffGHJybSaRrmr5eHr81mwJU3kjw6M+uprWvCXiNeN9SOGwQ3Jn8jb8m3Z6izVgknn1R0FTCEAP2QrLY/w=="],
|
||||
|
||||
"form-data/mime-types/mime-db": ["mime-db@1.52.0", "", {}, "sha512-sPU4uV7dYlvtWJxwwxHD0PuihVNiE7TyAbQ5SWxDCB9mUYvOgroQOwYQQOKPJ8CIbE+1ETVlOoK1UC2nU3gYvg=="],
|
||||
|
||||
"openai/@types/node/undici-types": ["undici-types@5.26.5", "", {}, "sha512-JlCMO+ehdEIKqlFxk6IfVoAUVmgz7cU7zD/h9XZ0qzeosSHmUJVOzSQvvYSYWXkFXC+IfLKSIffhv0sVZup6pA=="],
|
||||
|
||||
"@aws-crypto/sha1-browser/@smithy/util-utf8/@smithy/util-buffer-from/@smithy/is-array-buffer": ["@smithy/is-array-buffer@2.2.0", "", { "dependencies": { "tslib": "^2.6.2" } }, "sha512-GGP3O9QFD24uGeAXYUjwSTXARoqpZykHadOmA8G5vfJPK0/DC67qa//0qvqrJzL1xc8WQWX7/yc7fwudjPHPhA=="],
|
||||
|
||||
"@aws-crypto/sha256-browser/@smithy/util-utf8/@smithy/util-buffer-from/@smithy/is-array-buffer": ["@smithy/is-array-buffer@2.2.0", "", { "dependencies": { "tslib": "^2.6.2" } }, "sha512-GGP3O9QFD24uGeAXYUjwSTXARoqpZykHadOmA8G5vfJPK0/DC67qa//0qvqrJzL1xc8WQWX7/yc7fwudjPHPhA=="],
|
||||
|
||||
+16
-3
@@ -171,7 +171,7 @@ strict behavior when unset.
|
||||
- `src/core/minions/attachments.ts` — Attachment validation (path traversal, null byte, oversize, base64, duplicate detection)
|
||||
- `src/commands/agent.ts` (v0.16) — `gbrain agent run <prompt> [flags]` CLI. Submits `subagent` (or N children + 1 aggregator) under `{allowProtectedSubmit: true}`. Single-entry `--fanout-manifest` short-circuits. Children get `on_child_fail: 'continue'` + `max_stalled: 3`. `--follow` is the default on TTY; streams logs + polls `waitForCompletion` in parallel. Ctrl-C detaches, does not cancel.
|
||||
- `src/commands/agent-logs.ts` (v0.16) — `gbrain agent logs <job> [--follow] [--since]`. Merges JSONL heartbeat audit + `subagent_messages` into a chronological timeline. `parseSince` accepts ISO-8601 or relative (`5m`, `1h`, `2d`). Transcript tail renders only for terminal jobs.
|
||||
- `src/commands/jobs.ts` — `gbrain jobs` CLI subcommands + `gbrain jobs work` daemon. v0.13.1 surfaces the full `MinionJobInput` retry/backoff/timeout/idempotency surface as first-class CLI flags on `jobs submit`: `--max-stalled`, `--backoff-type fixed|exponential`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key`. `jobs smoke --sigkill-rescue` is the opt-in regression guard for #219. v0.16 wires `registerBuiltinHandlers` to always register `subagent` + `subagent_aggregator` (no env flag — `ANTHROPIC_API_KEY` is the natural cost gate, trust is via `PROTECTED_JOB_NAMES`) and loads `GBRAIN_PLUGIN_PATH` plugins at worker startup with a loud startup-line per plugin. `shell` handler still gated by `GBRAIN_ALLOW_SHELL_JOBS=1` (RCE surface, separate concern). v0.22.10 (#521): the `autopilot-cycle` handler now forwards `job.data.phases` to `runCycle` (was previously discarded — caller-supplied phase selection silently became a full cycle). Phases are validated against `ALL_PHASES` from `src/core/cycle.ts`; invalid names are filtered out and an empty/missing array falls back to the default 6-phase cycle.
|
||||
- `src/commands/jobs.ts` — `gbrain jobs` CLI subcommands + `gbrain jobs work` daemon. v0.13.1 surfaces the full `MinionJobInput` retry/backoff/timeout/idempotency surface as first-class CLI flags on `jobs submit`: `--max-stalled`, `--backoff-type fixed|exponential`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key`. `jobs smoke --sigkill-rescue` is the opt-in regression guard for #219. v0.16 wires `registerBuiltinHandlers` to always register `subagent` + `subagent_aggregator` (no env flag — `ANTHROPIC_API_KEY` is the natural cost gate, trust is via `PROTECTED_JOB_NAMES`) and loads `GBRAIN_PLUGIN_PATH` plugins at worker startup with a loud startup-line per plugin. `shell` handler still gated by `GBRAIN_ALLOW_SHELL_JOBS=1` (RCE surface, separate concern). v0.22.10 (#521): the `autopilot-cycle` handler now forwards `job.data.phases` to `runCycle` (was previously discarded — caller-supplied phase selection silently became a full cycle). Phases are validated against `ALL_PHASES` from `src/core/cycle.ts`; invalid names are filtered out and an empty/missing array falls back to the default 6-phase cycle. v0.22.13 (PR #490 CODEX-1+CODEX-4): `sync` handler now resolves `sourceId` at entry by looking up `sources.local_path` (mirrors `cycle.ts:480`'s autopilot fix from PR #475) so multi-source brains read the per-source `last_commit` anchor instead of the global config key. Concurrency routed through the shared `autoConcurrency()` policy in `src/core/sync-concurrency.ts` instead of the prior hardcoded `4`; PGLite stays serial. `noEmbed` default is `true` (embed is a separate job — submit `gbrain embed --stale` after sync, or rely on the autopilot cycle's embed phase).
|
||||
- `src/commands/features.ts` — `gbrain features --json --auto-fix`: usage scan + feature adoption salesman
|
||||
- `src/commands/autopilot.ts` — `gbrain autopilot --install`: self-maintaining brain daemon (sync+extract+embed)
|
||||
- `src/mcp/server.ts` — MCP stdio server (generated from operations). v0.22.7: tool-call handler delegates to `dispatchToolCall` from `src/mcp/dispatch.ts` so stdio + HTTP transports share one validation, context-build, and error-format path.
|
||||
@@ -188,6 +188,9 @@ strict behavior when unset.
|
||||
- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2).
|
||||
- `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2.
|
||||
- `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=<ms>` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers.
|
||||
- `src/core/db-lock.ts` (v0.22.13) — generic `tryAcquireDbLock(engine, lockId, ttlMinutes)` over the existing `gbrain_cycle_locks` table. Parameterized lock id so different scopes can nest cleanly: `gbrain-cycle` for the broad cycle (held by `cycle.ts`) and `gbrain-sync` (`SYNC_LOCK_ID` constant) for `performSync`'s narrower writer window. Same UPSERT-with-TTL semantics as the prior cycle-only helper, just generalized. Survives PgBouncer transaction pooling (unlike session-scoped `pg_try_advisory_lock`); crashed holders auto-release once their TTL expires.
|
||||
- `src/core/sync-concurrency.ts` (v0.22.13) — single source of truth for the parallel-sync policy. Exports `autoConcurrency(engine, fileCount, override?)` (PGLite always serial; explicit override clamped to >=1; auto path returns `DEFAULT_PARALLEL_WORKERS=4` when `fileCount > AUTO_CONCURRENCY_FILE_THRESHOLD=100`), `shouldRunParallel(workers, fileCount, explicit)` (Q1: explicit `--workers` bypasses the >50-file floor), and `parseWorkers(s)` (rejects `'0'`, `'-3'`, `'foo'`, `'1.5'`, trailing chars — replaces the prior parseInt-with-no-validation in both `sync.ts` and `import.ts`). Used by `performSync`, `performFullSync`, `runImport`, and the Minion `sync` handler so the three sites can no longer drift.
|
||||
- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). v0.22.13 (PR #490): `performSync` wraps its body in a `gbrain-sync` writer lock so two concurrent syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot both write `last_commit` and let the last writer win. Head-drift gate after the import phase re-checks `git rev-parse HEAD`; if HEAD moved (someone ran `git checkout` / `git pull` mid-sync), the bookmark refuses to advance. Vanished files now record a failedFiles entry instead of silent-skip — the silent-skip-then-advance pathology that survived prior hardening passes is dead. Worker engines wrap in try/finally so disconnect always fires (panic-path leak fix). Both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`), validated via `parseWorkers`. Explicit `--workers` bypasses the auto-path file-count floor; auto path defers to `autoConcurrency()`. Banner moved to stderr.
|
||||
- `src/core/cycle.ts` — v0.17 brain maintenance cycle primitive. `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise<CycleReport>` composes 6 phases in semantically-driven order (lint → backlinks → sync → extract → embed → orphans). Three callers: `gbrain dream` CLI, `gbrain autopilot` daemon's inline path, and the Minions `autopilot-cycle` handler (`src/commands/jobs.ts`). One source of truth for what the brain does overnight. Coordination via `gbrain_cycle_locks` DB table (TTL-based; works through PgBouncer transaction pooling, unlike session-scoped `pg_try_advisory_lock`) + `~/.gbrain/cycle.lock` file lock with PID-liveness for PGLite / engine=null mode. `CycleReport.schema_version: "1"` is the stable agent-consumable shape. `PhaseResult.error: { class, code, message, hint?, docs_url? }` is Stripe-API-tier structured failure info. `yieldBetweenPhases` hook awaited between every phase — Minions handler uses this to renew its job lock and prevent v0.14 stall-death regression. Engine nullable: filesystem phases (lint, backlinks) run without DB; DB phases skip with `status: "skipped", reason: "no_database"`. Lock-skip: read-only phase selections (`--phase orphans`) bypass the cycle lock. v0.22.1 (#403): `CycleOpts.signal?: AbortSignal` propagates the worker's abort signal; `checkAborted()` fires between every phase and throws if the signal is aborted (cooperative — can't interrupt a phase mid-execution). v0.22.1 (#417): `runPhaseSync` returns `pagesAffected` via `SyncPhaseResult`; `runCycle` captures it and threads to `runPhaseExtract` as the 4th arg, enabling incremental extract on the cycle path. v0.22.1 (Codex F2): `runPhaseSync` takes `willRunExtractPhase: boolean` and sets `noExtract: phases.includes('extract')` so `gbrain dream --phase sync` doesn't silently lose extraction. v0.22.5 (#475): new `resolveSourceForDir(engine, brainDir)` helper queries `SELECT id FROM sources WHERE local_path = $1 LIMIT 1`; `runPhaseSync` threads result as `sourceId` to `performSync()` so sync reads the per-source `sources.last_commit` anchor instead of the drift-prone global `config.sync.last_commit` key. Bare try/catch lets pre-v0.18 brains fall through to the global key. Closes the prod hang where every autopilot cycle ran a 30-min full reimport because the global anchor commit had been GC'd from git history.
|
||||
- `src/commands/dream.ts` — v0.17 `gbrain dream` CLI. ~80-line thin alias over `runCycle`. brainDir resolution requires explicit `--dir` OR `sync.repo_path` config (no more walk-up-cwd-for-.git footgun). Flags: `--dry-run`, `--json`, `--phase <name>`, `--pull`, `--dir <path>`. Exit code 1 on status=failed (partial/warn not fatal — don't page on warnings).
|
||||
- `scripts/check-progress-to-stdout.sh` — CI guard against regressing to `\r`-on-stdout progress. Wired into `bun run test` via `scripts/check-progress-to-stdout.sh && bun test` in package.json.
|
||||
@@ -299,6 +302,10 @@ Key commands added in v0.14.3 (fix wave):
|
||||
- `gbrain jobs submit` gains `--max-stalled`, `--backoff-type`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key` — exposing existing `MinionJobInput` fields as first-class CLI flags.
|
||||
- `gbrain jobs smoke --sigkill-rescue` — opt-in regression smoke case simulating a killed worker; asserts the v0.14.3 schema default (`max_stalled=5`) actually rescues on first stall.
|
||||
|
||||
Key commands added in v0.22.13 (PR #490):
|
||||
- `gbrain sync --workers N` (alias `--concurrency N`) — parallelize the import phase using per-worker Postgres engines (small pool of 2 each) with an atomic queue index. Auto-concurrency: defaults to 4 workers when the diff exceeds 100 files. Smaller diffs stay serial. Explicit `--workers` always wins (even on a 30-file diff). PGLite forces serial regardless. Validation rejects `0`, negatives, non-integers loud (replaces the prior silent fall-through to auto-concurrency).
|
||||
- `gbrain import --workers N` — same `parseWorkers()` validation as sync; same try/finally worker-engine cleanup. Behavior surface unchanged.
|
||||
|
||||
## Testing
|
||||
|
||||
`bun test` runs all tests. After the v0.12.1 release: ~75 unit test files + 8 E2E test files (1412 unit pass, 119 E2E when `DATABASE_URL` is set — skip gracefully otherwise). Unit tests run
|
||||
@@ -354,6 +361,8 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac
|
||||
`test/orphans.test.ts` (v0.12.3 orphans command: detection, pseudo filtering, text/json/count outputs, MCP op),
|
||||
`test/postgres-engine.test.ts` (v0.12.3 statement_timeout scoping: `sql.begin` + `SET LOCAL` shape, source-level grep guardrail against reintroduced bare `SET statement_timeout`),
|
||||
`test/sync.test.ts` (sync logic + v0.12.3 regression guard asserting top-level `engine.transaction` is not called),
|
||||
`test/sync-concurrency.test.ts` (v0.22.13 PR #490: 17 cases covering `autoConcurrency()` thresholds + PGLite-forces-serial + explicit-override clamping, `shouldRunParallel()` Q1 explicit-bypasses-floor contract, and `parseWorkers()` validation that rejects `'0'`/`'-3'`/`'foo'`/`'1.5'`/trailing chars),
|
||||
`test/sync-parallel.test.ts` (v0.22.13 PR #490: PGLite-routed coverage of the bookmark gate under concurrency request, head-drift gate, vanished-file failure capture, PGLite-stays-serial, and the `gbrain-sync` writer-lock contract — 7 cases),
|
||||
`test/sync-failures.test.ts` (v0.22.12: 28 cases pinning `classifyErrorCode` regex coverage for all 12 codes against literal production message strings from `markdown.ts:159-244` and `import-file.ts:199, 347, 352, 401`; `summarizeFailuresByCode` sort + pre-classified-honor; `recordSyncFailures` code-field persistence; `acknowledgeSyncFailures` AcknowledgeResult shape + backfill on pre-v0.22.12 entries),
|
||||
`test/doctor.test.ts` (doctor command + v0.12.3 assertions that `jsonb_integrity` scans the four v0.12.0 write sites and `markdown_body_completeness` is present),
|
||||
`test/utils.test.ts` (shared SQL utilities + `tryParseEmbedding` null-return and single-warn semantics),
|
||||
@@ -384,6 +393,7 @@ E2E tests (`test/e2e/`): Run against real Postgres+pgvector. Require `DATABASE_U
|
||||
- `test/e2e/engine-parity.test.ts` (v0.22.0) — Postgres ↔ PGLite top-result and result-set parity for `searchKeyword` + `searchVector`. Codex flagged that Postgres ranks pages then picks best chunk while PGLite returns chunks directly — without parity coverage the source-boost fix could pass on PGLite and fail on Postgres. Skips gracefully when `DATABASE_URL` is unset.
|
||||
- `test/e2e/postgres-bootstrap.test.ts` (v0.22.6.1) — exercises `PostgresEngine.initSchema()` directly against a fresh real Postgres database. Asserts the bootstrap path is no-op on fresh installs and that SCHEMA_SQL replays cleanly through the engine path (not via the standalone `db.initSchema` from `src/core/db.ts`, which would have produced false-positive coverage). Codex caught the E2E-shape gap during plan review.
|
||||
- `test/e2e/http-transport.test.ts` (v0.22.7) — 8 cases against real Postgres covering `gbrain serve --http` end-to-end: bearer auth round-trip, `last_used_at` SQL-level debounce semantics, `mcp_request_log` row insertion on success and auth_failed paths, `/health` DB-down → 503 (DB-probing health check), and the F1+F2+F3 dispatch round-trip with a real operation. Skips gracefully when `DATABASE_URL` is unset.
|
||||
- `test/e2e/sync-parallel.test.ts` (v0.22.13 PR #490) — DATABASE_URL-gated. T2: 60-file Postgres sync at concurrency=4 imports all + no connection leak (probes `pg_stat_activity` before/after to confirm worker engines disconnected). P4: 120-file serial-vs-parallel benchmark prints `SYNC_PARALLEL_BENCH N files | serial=Xms | parallel(4)=Yms | speedup=Zx` for CHANGELOG quoting. Asserts parallel ≤ serial × 1.5 (CI-noise tolerant; not a strict speedup gate).
|
||||
- Tier 2 (`skills.test.ts`) requires OpenClaw + API keys, runs nightly in CI
|
||||
- If `.env.testing` doesn't exist in this directory, check sibling worktrees for one:
|
||||
`find ../ -maxdepth 2 -name .env.testing -print -quit` and copy it here if found.
|
||||
@@ -1923,8 +1933,11 @@ SEARCH
|
||||
gbrain query <question> Hybrid search (vector + keyword + RRF)
|
||||
|
||||
IMPORT
|
||||
gbrain import <dir> [--no-embed] Import markdown (idempotent)
|
||||
gbrain sync [--repo <path>] Git-to-brain incremental sync
|
||||
gbrain import <dir> [--no-embed] [--workers N]
|
||||
Import markdown (idempotent)
|
||||
gbrain sync [--repo <path>] [--workers N]
|
||||
Git-to-brain incremental sync
|
||||
(>100-file diffs auto-parallelize 4 workers on Postgres)
|
||||
gbrain export [--dir ./out/] Export to markdown
|
||||
|
||||
FILES
|
||||
|
||||
+2
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gbrain",
|
||||
"version": "0.22.12",
|
||||
"version": "0.22.15",
|
||||
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
|
||||
"type": "module",
|
||||
"main": "src/core/index.ts",
|
||||
@@ -64,6 +64,7 @@
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/bun": "latest",
|
||||
"bun-types": "^1.3.13",
|
||||
"typescript": "^5.6.0"
|
||||
},
|
||||
"trustedDependencies": [
|
||||
|
||||
@@ -0,0 +1,210 @@
|
||||
---
|
||||
feature_pitch:
|
||||
headline: Bare workers now self-monitor and fail-stop into your PM's restart loop
|
||||
body: |
|
||||
Bare `gbrain jobs work` now ships with the same health protection the
|
||||
supervisor already had: DB liveness probes (with per-probe timeout so a
|
||||
hung connection can't wedge the monitor), stall detection filtered by
|
||||
registered handler names, and an RSS watchdog default of 2048 MB.
|
||||
|
||||
When the worker detects it's wedged (stuck pgbouncer connection, hung
|
||||
event loop, stalled job claim), it emits `'unhealthy'` and the CLI calls
|
||||
`process.exit(1)`. This is **fail-stop**: it requires an external process
|
||||
manager (systemd, Docker `restart: always`, launchd `KeepAlive`, cron
|
||||
watchdog) to bring the worker back. Without one, the process exits and
|
||||
stays dead — that's a regression from pre-v0.22.14 self-healing.
|
||||
|
||||
Pre-v0.22.14 behavior: bare workers had ZERO health monitoring. A wedged
|
||||
worker stayed alive doing nothing while jobs piled up in `waiting` and
|
||||
your PM's `pgrep` check happily reported green.
|
||||
|
||||
If you're using `gbrain jobs supervisor`, you're already protected — the
|
||||
supervisor handles spawn-on-crash itself. The fail-stop concern only
|
||||
applies to direct `gbrain jobs work` invocations.
|
||||
---
|
||||
|
||||
# v0.22.14 — Bare-worker self-health-monitoring
|
||||
|
||||
## ⚠️ Pre-flight: confirm you have a process supervisor
|
||||
|
||||
If you run `gbrain jobs work` directly (NOT under `gbrain jobs supervisor`),
|
||||
verify your process manager is configured to restart the worker on exit
|
||||
BEFORE upgrading:
|
||||
|
||||
| Manager | What to check |
|
||||
|---|---|
|
||||
| systemd | `Restart=always` (or `Restart=on-failure`) in the `.service` unit |
|
||||
| Docker | `restart: always` / `restart: unless-stopped` in compose, OR `--restart` flag |
|
||||
| launchd (macOS) | `<key>KeepAlive</key><true/>` in the plist |
|
||||
| cron watchdog | Cron entry that re-spawns when `pgrep -f "gbrain jobs work"` is empty |
|
||||
| supervisord | `autorestart=true` |
|
||||
|
||||
**If your bare worker has no restart loop, the v0.22.14 fail-stop behavior
|
||||
will leave you with a dead worker after the first DB blip.** Either add a
|
||||
restart policy OR switch to `gbrain jobs supervisor` (which spawns its own
|
||||
child + restarts on crash internally).
|
||||
|
||||
## What ships
|
||||
|
||||
- DB liveness probes inside `gbrain jobs work` (60s interval, 3 strikes → exit)
|
||||
- Stall detection (5min warn / 10min exit when waiting jobs accumulate but
|
||||
in-flight is empty)
|
||||
- `--max-rss` defaults to 2048 MB for bare workers (matches supervisor default;
|
||||
was 0 = disabled)
|
||||
- New `MinionWorkerOpts.{healthCheckInterval, stallWarnAfterMs,
|
||||
stallExitAfterMs, dbFailExitAfter, dbProbeTimeoutMs}` for tuning (5 fields)
|
||||
- `MinionWorker` now extends `EventEmitter`; emits `'unhealthy'` event with
|
||||
a structured reason payload. **No-listener fallback**: if the caller does
|
||||
not subscribe to `'unhealthy'`, the worker calls `process.exit(1)` itself
|
||||
to preserve the pre-refactor fail-stop behavior. The CLI subscribes; direct
|
||||
API consumers without a listener inherit the fail-stop default. Inline
|
||||
paths (`jobs submit --follow`, `jobs smoke`) explicitly pass
|
||||
`healthCheckInterval: 0` to disable the timer entirely so the fallback
|
||||
cannot trip on a one-shot run.
|
||||
- `GBRAIN_SUPERVISED=1` env var (set by supervisor on its child) causes the
|
||||
child worker to skip its own health timer (no double-monitoring)
|
||||
- Constructor validation: throws if `stallExitAfterMs <= stallWarnAfterMs`
|
||||
(the contract requires strict greater-than)
|
||||
|
||||
## ⚠️ Breaking change you may need to act on
|
||||
|
||||
**`gbrain jobs work` now defaults `--max-rss 2048` (MB).** Pre-v0.22.14, bare
|
||||
workers had no RSS limit. If you ran bare workers with intentionally large
|
||||
embed/import working sets, your worker may now get killed by the watchdog after
|
||||
upgrade.
|
||||
|
||||
**You need to act if any of the following is true:**
|
||||
|
||||
- You run `gbrain jobs work` directly under a process manager (systemd, Docker,
|
||||
cron, supervisord, your own bash wrapper)
|
||||
- Your worker handles large embed batches, multi-GB import jobs, or
|
||||
memory-intensive subagent tasks that legitimately exceed 2 GB RSS
|
||||
- You rely on bare-worker resident memory exceeding 2 GB
|
||||
|
||||
**You do NOT need to act if:**
|
||||
|
||||
- You run `gbrain jobs supervisor` (the supervisor already defaulted to
|
||||
`--max-rss 2048` and has reconnect-then-fail logic on top)
|
||||
- Your typical worker stays well under 2 GB RSS
|
||||
|
||||
## What the agent should do post-upgrade
|
||||
|
||||
### 1. Run the orchestrator (mechanical migration)
|
||||
|
||||
```bash
|
||||
gbrain apply-migrations --yes
|
||||
```
|
||||
|
||||
This is a no-op for v0.22.14 — there's no schema migration. The mechanical work
|
||||
is in the binary; the user-side decision is the breaking-change handling below.
|
||||
|
||||
### 2. Detect bare-worker deployments
|
||||
|
||||
If the user runs OpenClaw, Hermes, or any agent fork that uses `gbrain jobs
|
||||
work` directly (NOT under `gbrain jobs supervisor`), they need to know. Check:
|
||||
|
||||
```bash
|
||||
# Look for bare worker invocations in cron, systemd, Docker, launchctl, etc.
|
||||
grep -r "gbrain jobs work" ~/.config/systemd/ /etc/systemd/ 2>/dev/null | head
|
||||
crontab -l 2>/dev/null | grep "gbrain jobs work" | head
|
||||
launchctl list 2>/dev/null | grep gbrain | head
|
||||
ps -ef | grep "gbrain jobs work" | grep -v grep | grep -v supervisor
|
||||
```
|
||||
|
||||
If you find bare-worker invocations, surface them to the user.
|
||||
|
||||
### 3. Ask about RSS profile
|
||||
|
||||
Use AskUserQuestion to confirm whether the user's bare workers need >2 GB RSS:
|
||||
|
||||
> "v0.22.14 changes `gbrain jobs work` to default `--max-rss 2048` (MB).
|
||||
> Pre-v0.22.14 bare workers had no limit. If your worker handles large embed
|
||||
> batches or multi-GB imports, the watchdog may now kill it. Do you want
|
||||
> to keep the new 2 GB default, raise the limit, or opt out entirely?"
|
||||
|
||||
Options:
|
||||
- **A) Keep 2 GB default (recommended for most)** — protects against memory
|
||||
leaks; restarts on overflow; matches supervisor behavior.
|
||||
- **B) Raise to N GB (specify N)** — pass `--max-rss <N*1024>` to the worker
|
||||
invocation.
|
||||
- **C) Opt out** — pass `--max-rss 0`.
|
||||
|
||||
### 4. Apply the user's choice
|
||||
|
||||
For each bare-worker invocation, edit the unit/cron/launchctl/script to add
|
||||
the chosen `--max-rss` flag.
|
||||
|
||||
**systemd (~/.config/systemd/user/gbrain-worker.service):**
|
||||
|
||||
```ini
|
||||
ExecStart=/usr/local/bin/gbrain jobs work --queue default --concurrency 3 --max-rss 4096
|
||||
# Or to opt out: --max-rss 0
|
||||
```
|
||||
|
||||
Then `systemctl --user daemon-reload && systemctl --user restart gbrain-worker`.
|
||||
|
||||
**cron (`crontab -e`):**
|
||||
|
||||
```cron
|
||||
@reboot /usr/local/bin/gbrain jobs work --queue default --concurrency 3 --max-rss 4096
|
||||
```
|
||||
|
||||
**Docker compose:**
|
||||
|
||||
```yaml
|
||||
command: ["gbrain", "jobs", "work", "--queue", "default", "--concurrency", "3", "--max-rss", "4096"]
|
||||
```
|
||||
|
||||
**launchctl (~/Library/LaunchAgents/com.user.gbrain-worker.plist):**
|
||||
|
||||
```xml
|
||||
<key>ProgramArguments</key>
|
||||
<array>
|
||||
<string>/usr/local/bin/gbrain</string>
|
||||
<string>jobs</string>
|
||||
<string>work</string>
|
||||
<string>--max-rss</string>
|
||||
<string>4096</string>
|
||||
</array>
|
||||
```
|
||||
|
||||
Then `launchctl unload ... && launchctl load ...`.
|
||||
|
||||
### 5. (Optional) Tune health-check thresholds
|
||||
|
||||
The new opts default to sensible values (60s probe interval, 5min warn / 10min
|
||||
exit, 3 DB failures). If you have specific SLAs, you can pass `--health-interval
|
||||
<ms>` to adjust the probe cadence. Stall thresholds are not yet CLI-exposed
|
||||
(only the API; CLI flags coming in a follow-up).
|
||||
|
||||
To disable self-monitoring entirely (e.g. you have your own external health
|
||||
checker):
|
||||
|
||||
```bash
|
||||
gbrain jobs work --health-interval 0 --max-rss 0
|
||||
```
|
||||
|
||||
### 6. Verify
|
||||
|
||||
```bash
|
||||
gbrain jobs stats # queue should be flowing normally
|
||||
gbrain doctor --json | jq '.' # no critical warnings
|
||||
ps -o rss= -p $(pgrep -f "gbrain jobs work") | awk '{print $1/1024 " MB"}'
|
||||
```
|
||||
|
||||
Worker startup log line should now show health-check status:
|
||||
|
||||
```
|
||||
Minion worker started (queue: default, concurrency: 3, watchdog: 2048MB, health-check: 60s)
|
||||
```
|
||||
|
||||
If running under supervisor, you'll see the watchdog but NOT the `health-check:
|
||||
60s` segment (because `GBRAIN_SUPERVISED=1` skips the child's self-monitor).
|
||||
|
||||
### 7. If anything fails
|
||||
|
||||
Open an issue at https://github.com/garrytan/gbrain/issues with:
|
||||
- Output of `gbrain doctor`
|
||||
- Contents of `~/.gbrain/upgrade-errors.jsonl` if it exists
|
||||
- Your bare-worker invocation (systemd unit / cron line / Dockerfile snippet)
|
||||
- Which step broke
|
||||
@@ -774,6 +774,30 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
|
||||
ORDER BY depth DESC
|
||||
LIMIT 5
|
||||
`;
|
||||
// Subcheck 3 (v0.22.14): RSS-watchdog kills in the last 24h. Bare workers
|
||||
// newly default to --max-rss 2048 (was 0); operators who run large embed
|
||||
// or import jobs may see kills that didn't happen pre-v0.22.14. We surface
|
||||
// a hint when this signature appears so the upgrade path is obvious.
|
||||
// Signature: when the watchdog trips, gracefulShutdown('watchdog') aborts
|
||||
// in-flight jobs with `new Error('watchdog')`. The worker's failJob path
|
||||
// (worker.ts:660-664) writes `error_text = 'aborted: watchdog'` for any
|
||||
// job in-flight at the moment of the kill.
|
||||
//
|
||||
// We deliberately DO NOT do a loose `ILIKE '%watchdog%'`:
|
||||
// 1. Parent jobs that inherit `on_child_fail='fail_parent'` get
|
||||
// `"child job N failed: aborted: watchdog"` — counting that
|
||||
// double-counts (child + parent) for one watchdog event.
|
||||
// 2. Any user error_text containing the word "watchdog" matches.
|
||||
// Match the exact prefix `'aborted: watchdog'` to scope this purely to
|
||||
// the worker's own kill signature.
|
||||
const rssKillRows: Array<{ cnt: number }> = await sql`
|
||||
SELECT count(*)::int AS cnt
|
||||
FROM minion_jobs
|
||||
WHERE status IN ('dead', 'failed')
|
||||
AND finished_at > now() - interval '24 hours'
|
||||
AND error_text = 'aborted: watchdog'
|
||||
`;
|
||||
const rssKillCount = rssKillRows[0]?.cnt ?? 0;
|
||||
|
||||
const problems: string[] = [];
|
||||
if (stalledRows.length > 0) {
|
||||
@@ -794,6 +818,14 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
|
||||
`Fix: set maxWaiting on the submitter (or raise GBRAIN_QUEUE_WAITING_THRESHOLD).`
|
||||
);
|
||||
}
|
||||
if (rssKillCount > 0) {
|
||||
problems.push(
|
||||
`${rssKillCount} job(s) dead-lettered for RSS-watchdog memory-limit kills in last 24h. ` +
|
||||
`v0.22.14 changed the bare-worker --max-rss default from 0 (off) to 2048 MB. ` +
|
||||
`Fix: raise the limit (e.g. \`gbrain jobs work --max-rss 4096\`) or opt out (\`--max-rss 0\`). ` +
|
||||
`See skills/migrations/v0.22.14.md.`
|
||||
);
|
||||
}
|
||||
|
||||
if (problems.length === 0) {
|
||||
checks.push({
|
||||
|
||||
+193
-1
@@ -49,6 +49,10 @@ export async function runFrontmatter(args: string[]): Promise<void> {
|
||||
}
|
||||
return;
|
||||
}
|
||||
if (sub === 'generate') {
|
||||
await runGenerate(rest);
|
||||
return;
|
||||
}
|
||||
if (sub === 'install-hook') {
|
||||
const { runFrontmatterInstallHook } = await import('./frontmatter-install-hook.ts');
|
||||
await runFrontmatterInstallHook(rest);
|
||||
@@ -71,10 +75,11 @@ async function connectEngineForAudit(): Promise<BrainEngine> {
|
||||
}
|
||||
|
||||
function printHelp() {
|
||||
console.log(`gbrain frontmatter — frontmatter validation, audit, and auto-repair
|
||||
console.log(`gbrain frontmatter — frontmatter validation, audit, auto-repair, and generation
|
||||
|
||||
Usage:
|
||||
gbrain frontmatter validate <path> [--json] [--fix] [--dry-run]
|
||||
gbrain frontmatter generate <path> [--fix] [--dry-run] [--json]
|
||||
gbrain frontmatter audit [--source <id>] [--json]
|
||||
gbrain frontmatter install-hook [--source <id>] [--force] [--uninstall]
|
||||
|
||||
@@ -91,6 +96,26 @@ validate
|
||||
--dry-run Preview --fix without writing.
|
||||
--json Emit a JSON envelope on stdout.
|
||||
|
||||
generate
|
||||
Synthesize frontmatter for files that have none (MISSING_OPEN). Uses
|
||||
directory-aware rules to infer type, title, date, source, and tags from
|
||||
the filesystem path and file content. Zero LLM calls, fully deterministic.
|
||||
|
||||
Without --fix: dry-run preview showing what would be generated.
|
||||
With --fix: writes frontmatter to files (with .bak safety backups).
|
||||
|
||||
Rules are defined in src/core/frontmatter-inference.ts DIRECTORY_RULES.
|
||||
Add new directory conventions by adding rules to the table.
|
||||
|
||||
Examples:
|
||||
gbrain frontmatter generate /path/to/brain # preview all
|
||||
gbrain frontmatter generate /path/to/brain --fix # write all
|
||||
gbrain frontmatter generate /path/to/brain/people/ --fix # just people/
|
||||
|
||||
--fix Write generated frontmatter to files (.bak safety backups).
|
||||
--dry-run Preview without writing (default when --fix is omitted).
|
||||
--json Emit JSON output.
|
||||
|
||||
audit
|
||||
Read-only scan across all registered sources (or one with --source <id>).
|
||||
Reports per-source counts grouped by error code. Use this in CI or doctor
|
||||
@@ -297,3 +322,170 @@ function printAuditHumanReport(report: AuditReport): void {
|
||||
console.log(`\nFix with: gbrain frontmatter validate <source-path> --fix`);
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// generate — synthesize frontmatter for files that have none
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
async function runGenerate(args: string[]): Promise<void> {
|
||||
const targetPath = args.find(a => !a.startsWith('-'));
|
||||
const doFix = args.includes('--fix');
|
||||
const dryRun = args.includes('--dry-run');
|
||||
const jsonOut = args.includes('--json');
|
||||
|
||||
if (!targetPath) {
|
||||
console.error('error: gbrain frontmatter generate requires a <path> argument');
|
||||
console.error('usage: gbrain frontmatter generate <path> [--fix] [--dry-run] [--json]');
|
||||
process.exitCode = 1;
|
||||
return;
|
||||
}
|
||||
|
||||
const { inferFrontmatter, serializeFrontmatter } = await import('../core/frontmatter-inference.ts');
|
||||
const { resolve, relative, join, basename } = await import('path');
|
||||
const { readFileSync, writeFileSync, copyFileSync, statSync, readdirSync, lstatSync } = await import('fs');
|
||||
|
||||
const rootPath = resolve(targetPath);
|
||||
const isDir = statSync(rootPath).isDirectory();
|
||||
|
||||
// Find the brain root — walk up from targetPath looking for .git or known brain markers.
|
||||
// Inference rules match against brain-root-relative paths (e.g., "people/alice.md").
|
||||
let brainRoot = rootPath;
|
||||
if (isDir) {
|
||||
let candidate = rootPath;
|
||||
for (let i = 0; i < 10; i++) {
|
||||
try {
|
||||
statSync(join(candidate, '.git'));
|
||||
brainRoot = candidate;
|
||||
break;
|
||||
} catch {
|
||||
const parent = resolve(candidate, '..');
|
||||
if (parent === candidate) break;
|
||||
candidate = parent;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
interface GenerateResult {
|
||||
path: string;
|
||||
type: string;
|
||||
title: string;
|
||||
date?: string;
|
||||
rule: string;
|
||||
}
|
||||
|
||||
const results: GenerateResult[] = [];
|
||||
let scanned = 0;
|
||||
let skipped = 0;
|
||||
let generated = 0;
|
||||
let written = 0;
|
||||
|
||||
function processFile(absPath: string, relPath: string) {
|
||||
scanned++;
|
||||
if (!absPath.endsWith('.md')) return;
|
||||
|
||||
// Skip symlinks
|
||||
try { if (lstatSync(absPath).isSymbolicLink()) return; } catch { return; }
|
||||
|
||||
let content: string;
|
||||
try { content = readFileSync(absPath, 'utf-8'); } catch { return; }
|
||||
|
||||
const inferred = inferFrontmatter(relPath, content);
|
||||
if (inferred.skipped) {
|
||||
skipped++;
|
||||
return;
|
||||
}
|
||||
|
||||
generated++;
|
||||
results.push({
|
||||
path: relPath,
|
||||
type: inferred.type,
|
||||
title: inferred.title,
|
||||
date: inferred.date,
|
||||
rule: inferred.matchedRule || '(default)',
|
||||
});
|
||||
|
||||
if (doFix && !dryRun) {
|
||||
const fm = serializeFrontmatter(inferred);
|
||||
const newContent = fm + '\n' + content;
|
||||
// Safety: write .bak first
|
||||
copyFileSync(absPath, absPath + '.bak');
|
||||
writeFileSync(absPath, newContent, 'utf-8');
|
||||
written++;
|
||||
}
|
||||
}
|
||||
|
||||
function walkDir(dir: string, rootForRel: string) {
|
||||
let entries: string[];
|
||||
try { entries = readdirSync(dir); } catch { return; }
|
||||
for (const entry of entries) {
|
||||
if (entry === '.git' || entry === 'node_modules' || entry === '.obsidian') continue;
|
||||
const abs = join(dir, entry);
|
||||
try {
|
||||
const stat = statSync(abs);
|
||||
if (stat.isDirectory()) {
|
||||
walkDir(abs, rootForRel);
|
||||
} else if (stat.isFile() && entry.endsWith('.md')) {
|
||||
processFile(abs, relative(rootForRel, abs));
|
||||
}
|
||||
} catch { /* skip unreadable */ }
|
||||
}
|
||||
}
|
||||
|
||||
if (isDir) {
|
||||
walkDir(rootPath, brainRoot);
|
||||
} else {
|
||||
const relPath = relative(brainRoot, rootPath) || basename(rootPath);
|
||||
processFile(rootPath, relPath);
|
||||
}
|
||||
|
||||
// Output
|
||||
if (jsonOut) {
|
||||
console.log(JSON.stringify({
|
||||
scanned,
|
||||
skipped,
|
||||
generated,
|
||||
written,
|
||||
dryRun: !doFix || dryRun,
|
||||
results: results.slice(0, 100), // Cap JSON output
|
||||
totalResults: results.length,
|
||||
}, null, 2));
|
||||
return;
|
||||
}
|
||||
|
||||
// Human-readable output
|
||||
const mode = doFix && !dryRun ? 'WRITE' : 'DRY-RUN';
|
||||
console.log(`\nFrontmatter generation (${mode})`);
|
||||
console.log(` Scanned: ${scanned} files`);
|
||||
console.log(` Already have frontmatter: ${skipped}`);
|
||||
console.log(` Would generate: ${generated}`);
|
||||
if (doFix && !dryRun) {
|
||||
console.log(` Written: ${written} (with .bak backups)`);
|
||||
}
|
||||
|
||||
// Show sample by type
|
||||
const byType: Record<string, number> = {};
|
||||
for (const r of results) {
|
||||
byType[r.type] = (byType[r.type] || 0) + 1;
|
||||
}
|
||||
if (Object.keys(byType).length > 0) {
|
||||
console.log(`\n By type:`);
|
||||
for (const [type, count] of Object.entries(byType).sort(([, a], [, b]) => b - a)) {
|
||||
console.log(` ${type}: ${count}`);
|
||||
}
|
||||
}
|
||||
|
||||
// Show first 10 examples
|
||||
if (results.length > 0 && (!doFix || dryRun)) {
|
||||
console.log(`\n Examples:`);
|
||||
for (const r of results.slice(0, 10)) {
|
||||
console.log(` ${r.path}`);
|
||||
console.log(` → type: ${r.type}, title: "${r.title}"${r.date ? `, date: ${r.date}` : ''} [rule: ${r.rule}]`);
|
||||
}
|
||||
if (results.length > 10) {
|
||||
console.log(` ... and ${results.length - 10} more`);
|
||||
}
|
||||
if (!doFix) {
|
||||
console.log(`\n To write: gbrain frontmatter generate ${targetPath} --fix`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+55
-28
@@ -34,7 +34,17 @@ export async function runImport(engine: BrainEngine, args: string[], opts: { com
|
||||
const jsonOutput = args.includes('--json');
|
||||
const workersIdx = args.indexOf('--workers');
|
||||
const workersArg = workersIdx !== -1 ? args[workersIdx + 1] : null;
|
||||
const workerCount = workersArg ? parseInt(workersArg, 10) : 1;
|
||||
// v0.22.13 (PR #490 Q2): shared parseWorkers helper rejects bad input
|
||||
// (--workers 0, -3, "foo") with a loud error instead of silently falling
|
||||
// through to 1. Mirrors sync.ts's flag handling.
|
||||
const { parseWorkers } = await import('../core/sync-concurrency.ts');
|
||||
let workerCount: number;
|
||||
try {
|
||||
workerCount = parseWorkers(workersArg ?? undefined) ?? 1;
|
||||
} catch (e) {
|
||||
console.error(e instanceof Error ? e.message : String(e));
|
||||
process.exit(1);
|
||||
}
|
||||
// Find dir: first non-flag arg that isn't a value for --workers
|
||||
const flagValues = new Set<number>();
|
||||
if (workersIdx !== -1) flagValues.add(workersIdx + 1);
|
||||
@@ -141,40 +151,57 @@ export async function runImport(engine: BrainEngine, args: string[], opts: { com
|
||||
}
|
||||
|
||||
if (actualWorkers > 1) {
|
||||
// Parallel: create per-worker engine instances with small pool
|
||||
// PGLite is single-connection, so parallel workers are only for Postgres
|
||||
// v0.22.13 (PR #490 A1 + Q3): use engine.kind discriminator (not config.engine
|
||||
// string sniff) and fall back to serial when database_url is unset. Both
|
||||
// checks belt-and-suspenders so we never crash on a null assertion.
|
||||
const config = loadConfig();
|
||||
if (config?.engine === 'pglite') {
|
||||
// PGLite: sequential import through single engine
|
||||
if (engine.kind === 'pglite' || !config?.database_url) {
|
||||
for (const file of files) {
|
||||
await processFile(engine, file);
|
||||
}
|
||||
} else {
|
||||
const { PostgresEngine } = await import('../core/postgres-engine.ts');
|
||||
const { resolvePoolSize } = await import('../core/db.ts');
|
||||
// Default per-worker pool is 2 (small, parallel import case). Users on
|
||||
// constrained poolers (e.g. Supabase port 6543) can cap below this via
|
||||
// GBRAIN_POOL_SIZE=1.
|
||||
const workerPoolSize = Math.min(2, resolvePoolSize(2));
|
||||
const workerEngines = await Promise.all(
|
||||
Array.from({ length: actualWorkers }, async () => {
|
||||
const eng = new PostgresEngine();
|
||||
await eng.connect({ database_url: config!.database_url!, poolSize: workerPoolSize });
|
||||
return eng;
|
||||
})
|
||||
);
|
||||
const { PostgresEngine } = await import('../core/postgres-engine.ts');
|
||||
const { resolvePoolSize } = await import('../core/db.ts');
|
||||
// Default per-worker pool is 2 (small, parallel import case). Users on
|
||||
// constrained poolers (e.g. Supabase port 6543) can cap below this via
|
||||
// GBRAIN_POOL_SIZE=1.
|
||||
const workerPoolSize = Math.min(2, resolvePoolSize(2));
|
||||
const databaseUrl = config.database_url;
|
||||
|
||||
// Thread-safe queue: use an atomic index counter instead of array.shift()
|
||||
let queueIndex = 0;
|
||||
await Promise.all(workerEngines.map(async (eng) => {
|
||||
while (true) {
|
||||
const idx = queueIndex++;
|
||||
if (idx >= files.length) break;
|
||||
await processFile(eng, files[idx]);
|
||||
// v0.22.13 (PR #490 A2): connect workers serially so a partial failure
|
||||
// leaves us with the connected ones already pushed onto workerEngines
|
||||
// for the finally-block cleanup. The prior Promise.all could leak any
|
||||
// engine that connected before another's connect() rejected.
|
||||
const workerEngines: InstanceType<typeof PostgresEngine>[] = [];
|
||||
try {
|
||||
for (let i = 0; i < actualWorkers; i++) {
|
||||
const eng = new PostgresEngine();
|
||||
await eng.connect({ database_url: databaseUrl, poolSize: workerPoolSize });
|
||||
workerEngines.push(eng);
|
||||
}
|
||||
|
||||
// Thread-safe queue: atomic index counter (JS is single-threaded; the
|
||||
// read-then-increment happens between awaits so no lock is needed).
|
||||
let queueIndex = 0;
|
||||
await Promise.all(workerEngines.map(async (eng) => {
|
||||
while (true) {
|
||||
const idx = queueIndex++;
|
||||
if (idx >= files.length) break;
|
||||
await processFile(eng, files[idx]);
|
||||
}
|
||||
}));
|
||||
} finally {
|
||||
// v0.22.13 (PR #490 A2): try/finally guarantees cleanup even when the
|
||||
// worker loop throws. Each disconnect is best-effort — one failing
|
||||
// disconnect must not strand the others.
|
||||
await Promise.all(
|
||||
workerEngines.map(e =>
|
||||
e.disconnect().catch((err: unknown) =>
|
||||
console.error(` worker disconnect failed: ${err instanceof Error ? err.message : String(err)}`),
|
||||
),
|
||||
),
|
||||
);
|
||||
}
|
||||
}));
|
||||
|
||||
await Promise.all(workerEngines.map(e => e.disconnect()));
|
||||
} // end else (postgres parallel)
|
||||
} else {
|
||||
// Sequential: use the provided engine
|
||||
|
||||
+129
-18
@@ -33,14 +33,14 @@ export function parseMaxWaitingFlag(args: string[]): number | undefined {
|
||||
}
|
||||
|
||||
/** Parse `--max-rss N` (MB). Returns:
|
||||
* - 0 if the flag is absent (no watchdog by default for bare `jobs work`)
|
||||
* - undefined if the flag is absent (caller decides the default)
|
||||
* - 0 if `--max-rss 0` (explicit disable)
|
||||
* - the value if >= 256
|
||||
* Errors and exits the process if the flag is non-numeric, negative, or
|
||||
* positive but < 256 (likely a GB-vs-MB unit-confusion typo). */
|
||||
export function parseMaxRssFlag(args: string[]): number {
|
||||
export function parseMaxRssFlag(args: string[]): number | undefined {
|
||||
const raw = parseFlag(args, '--max-rss');
|
||||
if (raw === undefined) return 0;
|
||||
if (raw === undefined) return undefined;
|
||||
const parsed = parseInt(raw, 10);
|
||||
if (!Number.isFinite(parsed) || parsed < 0) {
|
||||
console.error(`Error: --max-rss must be a non-negative integer (MB), got "${raw}"`);
|
||||
@@ -133,6 +133,7 @@ USAGE
|
||||
gbrain jobs stats
|
||||
gbrain jobs smoke
|
||||
gbrain jobs work [--queue Q] [--concurrency N] [--max-rss MB]
|
||||
[--health-interval MS]
|
||||
gbrain jobs supervisor [start] [--detach] [--json]
|
||||
[--concurrency N] [--queue Q] [--pid-file PATH]
|
||||
[--max-crashes N] [--health-interval N]
|
||||
@@ -314,8 +315,15 @@ HANDLER TYPES (built in)
|
||||
|
||||
if (follow) {
|
||||
console.log(`Job #${job.id} submitted (${name}). Executing inline...`);
|
||||
// Inline execution: run the job in this process
|
||||
const worker = new MinionWorker(engine, { queue: queueName, pollInterval: 100 });
|
||||
// Inline execution: run the job in this process. Disable the
|
||||
// self-health-check timer — inline flows are one-shot and don't have
|
||||
// a process manager to restart them. With the timer enabled and no
|
||||
// 'unhealthy' listener, a DB blip would trip emitUnhealthy's
|
||||
// no-listener fallback and call process.exit(1) from inside the
|
||||
// library, killing the user's CLI session.
|
||||
const worker = new MinionWorker(engine, {
|
||||
queue: queueName, pollInterval: 100, healthCheckInterval: 0,
|
||||
});
|
||||
|
||||
// Register built-in handlers
|
||||
await registerBuiltinHandlers(worker, engine);
|
||||
@@ -489,7 +497,11 @@ HANDLER TYPES (built in)
|
||||
const sigkillRescue = hasFlag(args, '--sigkill-rescue');
|
||||
const wedgeRescue = hasFlag(args, '--wedge-rescue');
|
||||
|
||||
const worker = new MinionWorker(engine, { queue: 'smoke', pollInterval: 100 });
|
||||
// Smoke harness is short-lived and has no listener — disable the health
|
||||
// timer so the no-listener fallback can't trip process.exit(1) mid-test.
|
||||
const worker = new MinionWorker(engine, {
|
||||
queue: 'smoke', pollInterval: 100, healthCheckInterval: 0,
|
||||
});
|
||||
worker.register('noop', async () => ({ ok: true, at: new Date().toISOString() }));
|
||||
|
||||
const job = await queue.add('noop', {}, { queue: 'smoke', max_attempts: 1 });
|
||||
@@ -638,19 +650,69 @@ HANDLER TYPES (built in)
|
||||
|
||||
const queueName = parseFlag(args, '--queue') ?? 'default';
|
||||
const concurrency = resolveWorkerConcurrency(args);
|
||||
// --max-rss is opt-in for bare `gbrain jobs work` — preserves pre-v0.21 behavior
|
||||
// for operators with legitimately large embed/import working sets. The supervisor
|
||||
// path injects a default 2048; this code path does not.
|
||||
const maxRssMb = parseMaxRssFlag(args);
|
||||
// --max-rss defaults to 2048 for bare workers (matching supervisor default).
|
||||
// This catches memory-leak stalls that previously went undetected without
|
||||
// a supervisor. Operators can opt out with `--max-rss 0`.
|
||||
const maxRssExplicit = parseMaxRssFlag(args);
|
||||
const maxRssMb = maxRssExplicit ?? 2048;
|
||||
|
||||
// --health-interval: self-health-check period in ms. 0 disables. Default: 60_000 (60s).
|
||||
// Provides DB liveness probes + stall detection for bare workers.
|
||||
// Automatically skipped when running under a supervisor (GBRAIN_SUPERVISED=1).
|
||||
// Validated aggressively (parity with --max-rss): reject NaN/negative/non-integer
|
||||
// values, and reject suspicious sub-1000ms values that are likely a unit-confusion
|
||||
// typo (e.g. "--health-interval 60" thinking the unit is seconds).
|
||||
const healthRaw = parseFlag(args, '--health-interval');
|
||||
let healthCheckInterval = 60_000;
|
||||
if (healthRaw !== undefined) {
|
||||
const parsed = parseInt(healthRaw, 10);
|
||||
if (!Number.isFinite(parsed) || parsed < 0) {
|
||||
console.error(`Error: --health-interval must be a non-negative integer (ms), got "${healthRaw}"`);
|
||||
process.exit(1);
|
||||
}
|
||||
if (parsed > 0 && parsed < 1000) {
|
||||
console.error(
|
||||
`Error: --health-interval ${parsed} is suspiciously low (likely a unit-confusion typo). ` +
|
||||
`The flag takes milliseconds; for 60-second probes pass 60000. Use 0 to disable.`,
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
healthCheckInterval = parsed;
|
||||
}
|
||||
|
||||
try { await queue.ensureSchema(); }
|
||||
catch (e) { console.error(e instanceof Error ? e.message : String(e)); process.exit(1); }
|
||||
|
||||
const worker = new MinionWorker(engine, { queue: queueName, concurrency, maxRssMb });
|
||||
const worker = new MinionWorker(engine, {
|
||||
queue: queueName, concurrency, maxRssMb, healthCheckInterval,
|
||||
});
|
||||
await registerBuiltinHandlers(worker, engine);
|
||||
|
||||
// Subscribe to self-health failures emitted by the worker. Library code
|
||||
// (worker.ts) never calls process.exit directly so it stays embeddable;
|
||||
// this CLI layer is the right place to terminate the process and let
|
||||
// the external PM (systemd, Docker, cron watchdog) restart cleanly.
|
||||
worker.on('unhealthy', (info) => {
|
||||
if (info.reason === 'db_dead') {
|
||||
console.error(
|
||||
`[health] FATAL: DB unreachable after ${info.consecutiveFailures} probes (${info.message}). ` +
|
||||
`Exiting for process-manager restart.`,
|
||||
);
|
||||
} else {
|
||||
console.error(
|
||||
`[health] FATAL: Worker stalled — ${info.waitingCount} waiting job(s) for ` +
|
||||
`registered handlers, ${info.idleMinutes}m idle. Exiting for process-manager restart.`,
|
||||
);
|
||||
}
|
||||
process.exit(1);
|
||||
});
|
||||
|
||||
const isSupervisedChild = process.env.GBRAIN_SUPERVISED === '1';
|
||||
const watchdogNote = maxRssMb > 0 ? `, watchdog: ${maxRssMb}MB` : '';
|
||||
console.log(`Minion worker started (queue: ${queueName}, concurrency: ${concurrency}${watchdogNote})`);
|
||||
const healthNote = !isSupervisedChild && healthCheckInterval > 0
|
||||
? `, health-check: ${Math.round(healthCheckInterval / 1000)}s`
|
||||
: '';
|
||||
console.log(`Minion worker started (queue: ${queueName}, concurrency: ${concurrency}${watchdogNote}${healthNote})`);
|
||||
console.log(`Registered handlers: ${worker.registeredNames.join(', ')}`);
|
||||
await worker.start();
|
||||
break;
|
||||
@@ -787,15 +849,32 @@ HANDLER TYPES (built in)
|
||||
const concurrency = parseInt(parseFlag(args, '--concurrency') ?? '2', 10);
|
||||
const queueName = parseFlag(args, '--queue') ?? 'default';
|
||||
const maxCrashes = parseInt(parseFlag(args, '--max-crashes') ?? '10', 10);
|
||||
const healthInterval = parseInt(parseFlag(args, '--health-interval') ?? '60000', 10);
|
||||
// --health-interval (supervisor): validate same as `jobs work` so NaN /
|
||||
// negative / sub-1000ms typos fail-fast instead of silently disabling
|
||||
// the supervisor's own health probe.
|
||||
const supHealthRaw = parseFlag(args, '--health-interval');
|
||||
let healthInterval = 60_000;
|
||||
if (supHealthRaw !== undefined) {
|
||||
const parsed = parseInt(supHealthRaw, 10);
|
||||
if (!Number.isFinite(parsed) || parsed < 0) {
|
||||
console.error(`Error: --health-interval must be a non-negative integer (ms), got "${supHealthRaw}"`);
|
||||
process.exit(1);
|
||||
}
|
||||
if (parsed > 0 && parsed < 1000) {
|
||||
console.error(
|
||||
`Error: --health-interval ${parsed} is suspiciously low (likely a unit-confusion typo). ` +
|
||||
`The flag takes milliseconds; for 60-second probes pass 60000. Use 0 to disable.`,
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
healthInterval = parsed;
|
||||
}
|
||||
const allowShellJobs = hasFlag(args, '--allow-shell-jobs') ||
|
||||
!!process.env.GBRAIN_ALLOW_SHELL_JOBS;
|
||||
const detach = hasFlag(args, '--detach');
|
||||
// Supervisor defaults --max-rss 2048 (MB) — main production path uses
|
||||
// the supervisor, so the watchdog is on by default here. parseMaxRssFlag
|
||||
// returns 0 when the flag is absent; substitute the supervisor default.
|
||||
const maxRssRaw = parseMaxRssFlag(args);
|
||||
const maxRssMb = parseFlag(args, '--max-rss') === undefined ? 2048 : maxRssRaw;
|
||||
// the supervisor, so the watchdog is on by default here.
|
||||
const maxRssMb = parseMaxRssFlag(args) ?? 2048;
|
||||
|
||||
const cliPath = parseFlag(args, '--cli-path') ?? resolveGbrainCliPath();
|
||||
|
||||
@@ -864,8 +943,40 @@ export async function registerBuiltinHandlers(worker: MinionWorker, engine: Brai
|
||||
const { performSync } = await import('./sync.ts');
|
||||
const repoPath = typeof job.data.repoPath === 'string' ? job.data.repoPath : undefined;
|
||||
const noPull = !!job.data.noPull;
|
||||
// noEmbed defaults to true (embed is a separate job — submit `embed --stale`
|
||||
// after sync, OR run via the autopilot cycle which has its own embed phase).
|
||||
// Caller can opt in by passing { noEmbed: false } in job params.
|
||||
const noEmbed = job.data.noEmbed !== false;
|
||||
const result = await performSync(engine, { repoPath, noPull, noEmbed });
|
||||
// v0.22.13 (PR #490 CODEX-1): resolve sourceId from job param OR by looking
|
||||
// up the sources row for repoPath. Mirrors cycle.ts:480 — without this, a
|
||||
// multi-source brain reads the global config.sync.last_commit anchor
|
||||
// instead of sources.last_commit, which on a regularly-GC'd repo can drop
|
||||
// out of git history and trigger 30-min full reimports every cycle.
|
||||
let sourceId: string | undefined =
|
||||
typeof job.data.sourceId === 'string' ? job.data.sourceId : undefined;
|
||||
if (!sourceId && repoPath) {
|
||||
try {
|
||||
const rows = await engine.executeRaw<{ id: string }>(
|
||||
`SELECT id FROM sources WHERE local_path = $1 LIMIT 1`,
|
||||
[repoPath],
|
||||
);
|
||||
sourceId = rows[0]?.id;
|
||||
} catch {
|
||||
// sources table may not exist on very old brains — fall through to
|
||||
// global config.sync.* anchor in performSync.
|
||||
}
|
||||
}
|
||||
// v0.22.13 (PR #490 CODEX-4): route concurrency through the shared
|
||||
// autoConcurrency helper instead of hardcoded 4. PGLite engines stay
|
||||
// serial (forced 1); explicit job param wins; auto path defaults are
|
||||
// applied inside performSync against the resolved file count.
|
||||
const concurrencyOverride = typeof job.data.concurrency === 'number'
|
||||
? job.data.concurrency
|
||||
: undefined;
|
||||
const result = await performSync(engine, {
|
||||
repoPath, sourceId, noPull, noEmbed,
|
||||
concurrency: concurrencyOverride,
|
||||
});
|
||||
return result;
|
||||
});
|
||||
|
||||
|
||||
+197
-7
@@ -19,6 +19,13 @@ import { errorFor, serializeError } from '../core/errors.ts';
|
||||
import type { SyncManifest } from '../core/sync.ts';
|
||||
import { createProgress } from '../core/progress.ts';
|
||||
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
|
||||
import { loadConfig } from '../core/config.ts';
|
||||
import {
|
||||
autoConcurrency,
|
||||
shouldRunParallel,
|
||||
parseWorkers,
|
||||
} from '../core/sync-concurrency.ts';
|
||||
import { tryAcquireDbLock, SYNC_LOCK_ID } from '../core/db-lock.ts';
|
||||
import { loadStorageConfig } from '../core/storage-config.ts';
|
||||
import { getDefaultSourcePath } from '../core/source-resolver.ts';
|
||||
|
||||
@@ -159,6 +166,27 @@ export interface SyncOpts {
|
||||
sourceId?: string;
|
||||
/** Multi-repo: sync strategy override (markdown, code, auto). */
|
||||
strategy?: 'markdown' | 'code' | 'auto';
|
||||
/**
|
||||
* Number of parallel workers for the import phase. When > 1, each worker
|
||||
* gets its own small Postgres connection pool and files are dispatched via
|
||||
* an atomic queue index (same pattern as `import --workers N`).
|
||||
*
|
||||
* Deletes and renames remain serial (order-dependent).
|
||||
* Default: undefined → auto-concurrency picks (`src/core/sync-concurrency.ts`).
|
||||
*
|
||||
* v0.22.13 (PR #490 Q1): when this is explicitly set, the >50-file floor
|
||||
* is bypassed — explicit user intent beats the auto-path safety net.
|
||||
*/
|
||||
concurrency?: number;
|
||||
/**
|
||||
* Internal: skip acquiring the gbrain-sync DB lock. Set by the cycle
|
||||
* handler (cycle.ts) which already holds gbrain-cycle and therefore
|
||||
* already serializes against other cycle runs. CLI sync, jobs handler,
|
||||
* and any external caller leave this undefined so they take the lock.
|
||||
*
|
||||
* v0.22.13 (PR #490 CODEX-2). Not part of the public CLI surface.
|
||||
*/
|
||||
skipLock?: boolean;
|
||||
}
|
||||
|
||||
function git(repoPath: string, ...args: string[]): string {
|
||||
@@ -252,6 +280,39 @@ async function writeChunkerVersion(
|
||||
}
|
||||
|
||||
export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise<SyncResult> {
|
||||
// CODEX-2 (v0.22.13): cross-process writer lock for performSync. Two
|
||||
// concurrent syncs can otherwise read the same last_commit anchor, both
|
||||
// write last_commit unconditionally, and the last writer wins — including
|
||||
// regressing the bookmark backwards. cycle.ts already takes gbrain-cycle
|
||||
// for its broader scope; performSync (called from cycle, jobs handler,
|
||||
// and CLI) takes gbrain-sync just for the writer window. The two ids
|
||||
// nest cleanly: cycle holds gbrain-cycle, calls performSync, performSync
|
||||
// takes gbrain-sync. Other callers serialize on gbrain-sync against
|
||||
// each other AND against the cycle's sync phase.
|
||||
//
|
||||
// skipLock is reserved for callers that already serialize via another
|
||||
// mechanism (none in v0.22.13; reserved for future).
|
||||
let lockHandle: { release: () => Promise<void> } | null = null;
|
||||
if (!opts.skipLock) {
|
||||
lockHandle = await tryAcquireDbLock(engine, SYNC_LOCK_ID);
|
||||
if (!lockHandle) {
|
||||
throw new Error(
|
||||
`Another sync is in progress (lock ${SYNC_LOCK_ID} held). ` +
|
||||
`Wait for it to finish, or run 'gbrain doctor' if it has been more than 30 minutes.`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
try {
|
||||
return await performSyncInner(engine, opts);
|
||||
} finally {
|
||||
if (lockHandle) {
|
||||
try { await lockHandle.release(); } catch { /* best-effort release */ }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<SyncResult> {
|
||||
// Resolve repo path
|
||||
const repoPath = opts.repoPath || await readSyncAnchor(engine, opts.sourceId, 'repo_path');
|
||||
if (!repoPath) {
|
||||
@@ -488,21 +549,41 @@ export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise<
|
||||
// gate `sync.last_commit` advancement and record recoverable errors.
|
||||
const failedFiles: Array<{ path: string; error: string; line?: number }> = [];
|
||||
const addsAndMods = [...filtered.added, ...filtered.modified];
|
||||
|
||||
// v0.22.13 (PR #490 Q5): one source of truth for the concurrency decision.
|
||||
// engine.kind === 'pglite' → forced 1; explicit opts.concurrency wins;
|
||||
// auto path returns DEFAULT_PARALLEL_WORKERS only when fileCount > 100.
|
||||
const explicitConcurrency = opts.concurrency !== undefined;
|
||||
const effectiveConcurrency = autoConcurrency(engine, addsAndMods.length, opts.concurrency);
|
||||
const runParallel = shouldRunParallel(effectiveConcurrency, addsAndMods.length, explicitConcurrency);
|
||||
|
||||
if (addsAndMods.length > 0) {
|
||||
progress.start('sync.imports', addsAndMods.length);
|
||||
for (const path of addsAndMods) {
|
||||
const filePath = join(repoPath, path);
|
||||
|
||||
// Core import logic shared by serial and parallel paths.
|
||||
// repoPath is validated non-null at the top of performSyncInner; narrow for TS.
|
||||
const syncRepoPath = repoPath!;
|
||||
async function importOnePath(eng: BrainEngine, path: string): Promise<void> {
|
||||
const filePath = join(syncRepoPath, path);
|
||||
if (!existsSync(filePath)) {
|
||||
// CODEX-3 (v0.22.13): a file the diff said exists at headCommit but
|
||||
// is gone from disk means the working tree has drifted (someone ran
|
||||
// `git checkout` / `git reset` mid-sync, or the file was deleted
|
||||
// post-diff). Record as a failure so last_commit does NOT advance —
|
||||
// the silent-skip-then-advance pathology was the bug.
|
||||
failedFiles.push({
|
||||
path,
|
||||
error: 'file vanished mid-sync (working tree drifted from headCommit)',
|
||||
});
|
||||
progress.tick(1, `skip:${path}`);
|
||||
continue;
|
||||
return;
|
||||
}
|
||||
try {
|
||||
const result = await importFile(engine, filePath, path, { noEmbed });
|
||||
const result = await importFile(eng, filePath, path, { noEmbed });
|
||||
if (result.status === 'imported') {
|
||||
chunksCreated += result.chunks;
|
||||
pagesAffected.push(result.slug);
|
||||
} else if (result.status === 'skipped' && (result as any).error) {
|
||||
// importFile returned a non-throw skip with a reason.
|
||||
failedFiles.push({ path, error: String((result as any).error) });
|
||||
}
|
||||
} catch (e: unknown) {
|
||||
@@ -512,9 +593,98 @@ export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise<
|
||||
}
|
||||
progress.tick(1, path);
|
||||
}
|
||||
|
||||
if (runParallel) {
|
||||
// A1 (v0.22.13): use engine.kind discriminator instead of config?.engine
|
||||
// string compare or constructor.name sniff. Q3: belt-and-suspenders fall
|
||||
// back to serial when database_url is unset, so we never crash on a null
|
||||
// assertion if config is missing.
|
||||
const config = loadConfig();
|
||||
if (engine.kind === 'pglite' || !config?.database_url) {
|
||||
for (const path of addsAndMods) {
|
||||
await importOnePath(engine, path);
|
||||
}
|
||||
} else {
|
||||
const { PostgresEngine } = await import('../core/postgres-engine.ts');
|
||||
const { resolvePoolSize } = await import('../core/db.ts');
|
||||
const workerPoolSize = Math.min(2, resolvePoolSize(2));
|
||||
const workerCount = Math.min(effectiveConcurrency, addsAndMods.length);
|
||||
const databaseUrl = config.database_url;
|
||||
|
||||
// Q4 (v0.22.13): banner on stderr so stdout stays clean for --json.
|
||||
console.error(` Parallel sync: ${workerCount} workers for ${addsAndMods.length} files`);
|
||||
|
||||
const workerEngines: InstanceType<typeof PostgresEngine>[] = [];
|
||||
try {
|
||||
// Connect workers one-by-one rather than Promise.all so a partial
|
||||
// failure leaves us with the connected ones in workerEngines for
|
||||
// the finally-block cleanup. The original code lost track of
|
||||
// already-connected engines on any one failure.
|
||||
for (let i = 0; i < workerCount; i++) {
|
||||
const eng = new PostgresEngine();
|
||||
await eng.connect({ database_url: databaseUrl, poolSize: workerPoolSize });
|
||||
workerEngines.push(eng);
|
||||
}
|
||||
|
||||
// Atomic queue index — JS is single-threaded; the read-then-increment
|
||||
// happens between awaits, so no lock is needed.
|
||||
let queueIndex = 0;
|
||||
await Promise.all(
|
||||
workerEngines.map(async (eng) => {
|
||||
while (true) {
|
||||
const idx = queueIndex++;
|
||||
if (idx >= addsAndMods.length) break;
|
||||
await importOnePath(eng, addsAndMods[idx]);
|
||||
}
|
||||
}),
|
||||
);
|
||||
} finally {
|
||||
// A2 (v0.22.13): try/finally guarantees connection cleanup even when
|
||||
// the worker loop throws (partial connect failure, OOM, mid-import
|
||||
// signal). Each disconnect is best-effort — one worker failing to
|
||||
// disconnect must not strand the others.
|
||||
await Promise.all(
|
||||
workerEngines.map((e) =>
|
||||
e.disconnect().catch((err: unknown) =>
|
||||
console.error(` worker disconnect failed: ${err instanceof Error ? err.message : String(err)}`),
|
||||
),
|
||||
),
|
||||
);
|
||||
}
|
||||
}
|
||||
} else {
|
||||
// Serial path (small auto diffs or explicit --workers 1).
|
||||
for (const path of addsAndMods) {
|
||||
await importOnePath(engine, path);
|
||||
}
|
||||
}
|
||||
|
||||
progress.finish();
|
||||
}
|
||||
|
||||
// CODEX-3 (v0.22.13): head-drift gate. If git HEAD moved during the import
|
||||
// window (someone ran `git checkout` or `git pull` in another terminal /
|
||||
// sibling Conductor workspace), the chunks we just imported reflect a
|
||||
// different tree than `headCommit` claims. Refuse to advance last_commit
|
||||
// so the next sync re-walks against the new HEAD. The lock from CODEX-2
|
||||
// prevents *this* gbrain process from stepping on itself; this gate
|
||||
// catches drift caused by external `git` commands the lock cannot see.
|
||||
try {
|
||||
const currentHead = git(repoPath, 'rev-parse', 'HEAD');
|
||||
if (currentHead !== headCommit) {
|
||||
failedFiles.push({
|
||||
path: '<head>',
|
||||
error: `git HEAD drifted during sync: captured ${headCommit.slice(0, 8)}, now ${currentHead.slice(0, 8)}`,
|
||||
});
|
||||
}
|
||||
} catch (e) {
|
||||
// rev-parse failure is itself a drift signal (worktree disappeared).
|
||||
failedFiles.push({
|
||||
path: '<head>',
|
||||
error: `git HEAD verification failed: ${e instanceof Error ? e.message : String(e)}`,
|
||||
});
|
||||
}
|
||||
|
||||
const elapsed = Date.now() - start;
|
||||
|
||||
// Bug 9 — gate the sync bookmark on success. If any per-file parse
|
||||
@@ -653,10 +823,18 @@ async function performFullSync(
|
||||
};
|
||||
}
|
||||
|
||||
console.log(`Running full import of ${repoPath}...`);
|
||||
// v0.22.13 (PR #490 A1 + Q5): full sync is always "large" by definition
|
||||
// (entire working tree). Auto-concurrency fires unconditionally for Postgres;
|
||||
// PGLite stays serial because its engine is single-connection. Routes the
|
||||
// policy through autoConcurrency() so it stays consistent with incremental
|
||||
// sync and the jobs handler.
|
||||
const FULL_SYNC_LARGE_MARKER = Number.MAX_SAFE_INTEGER;
|
||||
const fullConcurrency = autoConcurrency(engine, FULL_SYNC_LARGE_MARKER, opts.concurrency);
|
||||
console.log(`Running full import of ${repoPath}${fullConcurrency > 1 ? ` (${fullConcurrency} workers)` : ''}...`);
|
||||
const { runImport } = await import('./import.ts');
|
||||
const importArgs = [repoPath];
|
||||
if (opts.noEmbed) importArgs.push('--no-embed');
|
||||
if (fullConcurrency > 1) importArgs.push('--workers', String(fullConcurrency));
|
||||
const result = await runImport(engine, importArgs, { commit: headCommit });
|
||||
|
||||
// Bug 9 — gate the full-sync bookmark on success. runImport already
|
||||
@@ -744,6 +922,17 @@ export async function runSync(engine: BrainEngine, args: string[]) {
|
||||
const jsonOut = args.includes('--json');
|
||||
const yesFlag = args.includes('--yes');
|
||||
const strategyArg = args.find((a, i) => args[i - 1] === '--strategy') as SyncOpts['strategy'] | undefined;
|
||||
const concurrencyStr = args.find((a, i) => args[i - 1] === '--concurrency' || args[i - 1] === '--workers');
|
||||
// v0.22.13 (PR #490 Q2): parseWorkers throws on '0', '-3', 'foo', '1.5' instead
|
||||
// of silently falling through to auto-concurrency or NaN. Loud failure beats
|
||||
// a 4-worker spawn from a typo.
|
||||
let concurrency: number | undefined;
|
||||
try {
|
||||
concurrency = parseWorkers(concurrencyStr);
|
||||
} catch (e) {
|
||||
console.error(e instanceof Error ? e.message : String(e));
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// v0.18.0 Step 5: --source resolves to a sources(id) row. Falls back
|
||||
// to pre-v0.17 global config (sync.repo_path + sync.last_commit) when
|
||||
@@ -836,6 +1025,7 @@ export async function runSync(engine: BrainEngine, args: string[]) {
|
||||
dryRun, full, noPull, noEmbed, skipFailed, retryFailed,
|
||||
sourceId: src.id,
|
||||
strategy: cfg.strategy,
|
||||
concurrency,
|
||||
};
|
||||
try {
|
||||
const result = await performSync(engine, repoOpts);
|
||||
@@ -854,7 +1044,7 @@ export async function runSync(engine: BrainEngine, args: string[]) {
|
||||
return;
|
||||
}
|
||||
|
||||
const opts: SyncOpts = { repoPath, dryRun, full, noPull, noEmbed, skipFailed, retryFailed, sourceId, strategy: strategyArg };
|
||||
const opts: SyncOpts = { repoPath, dryRun, full, noPull, noEmbed, skipFailed, retryFailed, sourceId, strategy: strategyArg, concurrency };
|
||||
|
||||
// Bug 9 — --retry-failed: before running normal sync, clear acknowledgment
|
||||
// flags so the sync picks them up as fresh work. The actual re-attempt
|
||||
|
||||
@@ -0,0 +1,140 @@
|
||||
/**
|
||||
* Generic DB-backed lock primitive.
|
||||
*
|
||||
* Reuses the gbrain_cycle_locks table (id PK + holder_pid + ttl_expires_at)
|
||||
* with a parameterized lock id. Both `gbrain-cycle` (the broad cycle lock)
|
||||
* and `gbrain-sync` (performSync's writer lock) live here.
|
||||
*
|
||||
* Why not pg_advisory_xact_lock: it is session-scoped, and PgBouncer
|
||||
* transaction pooling drops session state between calls. This row-based
|
||||
* lock survives PgBouncer because it's plain INSERT/UPDATE/DELETE with
|
||||
* a TTL fallback (a crashed holder's row times out).
|
||||
*
|
||||
* Why a separate table-row per lock id rather than reusing the cycle lock:
|
||||
* the cycle lock is broader (covers every phase). performSync's write-window
|
||||
* is narrower. If performSync reused the cycle lock and the cycle handler
|
||||
* called performSync, the inner acquire would deadlock against itself. Two
|
||||
* lock ids let callers nest cleanly: cycle holds gbrain-cycle for its run;
|
||||
* performSync (called from anywhere — cycle, jobs handler, CLI) takes
|
||||
* gbrain-sync just for the write window.
|
||||
*
|
||||
* v0.22.13 — added in PR #490 to fix CODEX-2 (no cross-process lock for
|
||||
* direct sync paths). The cycle path was already protected.
|
||||
*/
|
||||
import { hostname } from 'os';
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
|
||||
export interface DbLockHandle {
|
||||
id: string;
|
||||
release: () => Promise<void>;
|
||||
refresh: () => Promise<void>;
|
||||
}
|
||||
|
||||
/** Default TTL: 30 minutes, same as cycle lock. */
|
||||
const DEFAULT_TTL_MINUTES = 30;
|
||||
|
||||
/**
|
||||
* Try to acquire a named DB lock.
|
||||
*
|
||||
* Returns a handle on success. Returns `null` if another live holder has
|
||||
* the lock (its row exists and ttl_expires_at is in the future).
|
||||
*
|
||||
* The acquire is upsert-style:
|
||||
* INSERT ... ON CONFLICT (id) DO UPDATE
|
||||
* ... WHERE existing.ttl_expires_at < NOW()
|
||||
* RETURNING id
|
||||
*
|
||||
* Empty RETURNING means the existing row is still live. An expired holder
|
||||
* (worker crashed without releasing) is auto-superseded by the UPDATE
|
||||
* branch.
|
||||
*/
|
||||
export async function tryAcquireDbLock(
|
||||
engine: BrainEngine,
|
||||
lockId: string,
|
||||
ttlMinutes: number = DEFAULT_TTL_MINUTES,
|
||||
): Promise<DbLockHandle | null> {
|
||||
const pid = process.pid;
|
||||
const host = hostname();
|
||||
|
||||
// Engine-agnostic: prefer the engine's raw escape hatch (`sql` for postgres-js,
|
||||
// `db.query` for PGLite). Mirrors cycle.ts's pattern so behavior stays identical.
|
||||
const maybePG = engine as unknown as { sql?: (...args: unknown[]) => Promise<unknown> };
|
||||
const maybePGLite = engine as unknown as {
|
||||
db?: { query: (sql: string, params?: unknown[]) => Promise<{ rows: unknown[] }> };
|
||||
};
|
||||
|
||||
if (engine.kind === 'postgres' && maybePG.sql) {
|
||||
const sql = maybePG.sql as any;
|
||||
const ttl = `${ttlMinutes} minutes`;
|
||||
const rows: Array<{ id: string }> = await sql`
|
||||
INSERT INTO gbrain_cycle_locks (id, holder_pid, holder_host, acquired_at, ttl_expires_at)
|
||||
VALUES (${lockId}, ${pid}, ${host}, NOW(), NOW() + ${ttl}::interval)
|
||||
ON CONFLICT (id) DO UPDATE
|
||||
SET holder_pid = ${pid},
|
||||
holder_host = ${host},
|
||||
acquired_at = NOW(),
|
||||
ttl_expires_at = NOW() + ${ttl}::interval
|
||||
WHERE gbrain_cycle_locks.ttl_expires_at < NOW()
|
||||
RETURNING id
|
||||
`;
|
||||
if (rows.length === 0) return null;
|
||||
return {
|
||||
id: lockId,
|
||||
refresh: async () => {
|
||||
await sql`
|
||||
UPDATE gbrain_cycle_locks
|
||||
SET ttl_expires_at = NOW() + ${ttl}::interval
|
||||
WHERE id = ${lockId} AND holder_pid = ${pid}
|
||||
`;
|
||||
},
|
||||
release: async () => {
|
||||
await sql`
|
||||
DELETE FROM gbrain_cycle_locks
|
||||
WHERE id = ${lockId} AND holder_pid = ${pid}
|
||||
`;
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
if (engine.kind === 'pglite' && maybePGLite.db) {
|
||||
const db = maybePGLite.db;
|
||||
const ttl = `${ttlMinutes} minutes`;
|
||||
const { rows } = await db.query(
|
||||
`INSERT INTO gbrain_cycle_locks (id, holder_pid, holder_host, acquired_at, ttl_expires_at)
|
||||
VALUES ($1, $2, $3, NOW(), NOW() + $4::interval)
|
||||
ON CONFLICT (id) DO UPDATE
|
||||
SET holder_pid = $2,
|
||||
holder_host = $3,
|
||||
acquired_at = NOW(),
|
||||
ttl_expires_at = NOW() + $4::interval
|
||||
WHERE gbrain_cycle_locks.ttl_expires_at < NOW()
|
||||
RETURNING id`,
|
||||
[lockId, pid, host, ttl],
|
||||
);
|
||||
if (rows.length === 0) return null;
|
||||
return {
|
||||
id: lockId,
|
||||
refresh: async () => {
|
||||
await db.query(
|
||||
`UPDATE gbrain_cycle_locks
|
||||
SET ttl_expires_at = NOW() + $1::interval
|
||||
WHERE id = $2 AND holder_pid = $3`,
|
||||
[ttl, lockId, pid],
|
||||
);
|
||||
},
|
||||
release: async () => {
|
||||
await db.query(
|
||||
`DELETE FROM gbrain_cycle_locks WHERE id = $1 AND holder_pid = $2`,
|
||||
[lockId, pid],
|
||||
);
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
throw new Error(`Unknown engine kind for db-lock: ${engine.kind}`);
|
||||
}
|
||||
|
||||
/** Lock id for performSync's writer window. Distinct from gbrain-cycle so the
|
||||
* cycle handler can hold gbrain-cycle while performSync (called from inside
|
||||
* the cycle) acquires gbrain-sync. */
|
||||
export const SYNC_LOCK_ID = 'gbrain-sync';
|
||||
@@ -0,0 +1,410 @@
|
||||
/**
|
||||
* Frontmatter inference — synthesize YAML frontmatter from filesystem metadata.
|
||||
*
|
||||
* ## Why this exists
|
||||
*
|
||||
* GBrain's sync and import pipelines work fine without frontmatter — gray-matter
|
||||
* returns the full content as body, and `inferType`/`inferTitle` in markdown.ts
|
||||
* provide fallbacks. But the inferred metadata is minimal:
|
||||
*
|
||||
* - `type` defaults to 'concept' for most paths
|
||||
* - `title` is the slugified filename ("2010 04 13 Apr 13 Founders Mtg")
|
||||
* - No `date` field, no `source` metadata, no folder-aware tagging
|
||||
*
|
||||
* This module provides **rich inference** — directory-aware type mapping, date
|
||||
* extraction from filenames, title cleanup (strip date prefixes, HTML entities),
|
||||
* heading extraction from content, and source/folder tagging. It produces a
|
||||
* complete frontmatter block that can be:
|
||||
*
|
||||
* 1. Written back to the file on disk (via `gbrain frontmatter generate --fix`)
|
||||
* 2. Used at import time without modifying the file (DB-only inference)
|
||||
* 3. Shown as a dry-run preview (via `gbrain frontmatter generate --dry-run`)
|
||||
*
|
||||
* ## Design principles
|
||||
*
|
||||
* - **Never overwrite existing frontmatter.** If a file already has `---`, skip it.
|
||||
* - **Infer from filesystem first, content second.** Directory path → type, filename → date + title,
|
||||
* first `#` heading → title fallback, content → entity hints.
|
||||
* - **Deterministic.** Same file always produces the same frontmatter. No LLM calls, no network.
|
||||
* - **Extensible via rules.** The `DIRECTORY_RULES` table maps path patterns to type + source + tags.
|
||||
* Adding a new directory convention = adding one rule.
|
||||
* - **Safe.** `.bak` files on write, `--dry-run` by default in CLI, idempotent.
|
||||
*
|
||||
* ## How it fits in the pipeline
|
||||
*
|
||||
* ```
|
||||
* Sync/Import
|
||||
* → file has frontmatter? → normal import (existing path)
|
||||
* → file has NO frontmatter?
|
||||
* → inferFrontmatter(filePath, content) → synthesize frontmatter
|
||||
* → prepend to content → import as usual
|
||||
* → optionally write back to disk (--write-back flag)
|
||||
* ```
|
||||
*
|
||||
* The inference runs BEFORE `parseMarkdown`, so the downstream pipeline sees
|
||||
* well-formed frontmatter and all the existing validation/chunking/embedding
|
||||
* logic works unchanged.
|
||||
*
|
||||
* ## Directory rules table
|
||||
*
|
||||
* Each rule matches a path pattern (case-insensitive prefix) and provides:
|
||||
* - `type`: page type for the brain schema
|
||||
* - `source`: optional source tag (e.g., "apple-notes", "therapy")
|
||||
* - `tags`: optional additional tags
|
||||
* - `datePattern`: where to look for dates — 'filename' (YYYY-MM-DD prefix),
|
||||
* 'dirname' (parent dir name), or 'none'
|
||||
* - `titleStrategy`: how to extract title — 'filename' (strip date prefix),
|
||||
* 'heading' (first # in content), 'filename-full' (no date strip)
|
||||
*/
|
||||
|
||||
import { basename, dirname, relative } from 'path';
|
||||
|
||||
// ─── Types ───────────────────────────────────────────────────────────
|
||||
|
||||
export interface InferredFrontmatter {
|
||||
title: string;
|
||||
type: string;
|
||||
date?: string;
|
||||
source?: string;
|
||||
tags?: string[];
|
||||
/** True if the file already has frontmatter (inference skipped). */
|
||||
skipped?: boolean;
|
||||
/** The rule that matched, for debugging. */
|
||||
matchedRule?: string;
|
||||
}
|
||||
|
||||
export interface DirectoryRule {
|
||||
/** Case-insensitive path prefix to match (e.g., 'apple notes/'). */
|
||||
pathPrefix: string;
|
||||
/** Page type to assign. */
|
||||
type: string;
|
||||
/** Optional source tag. */
|
||||
source?: string;
|
||||
/** Optional tags to add. */
|
||||
tags?: string[];
|
||||
/** Where to look for dates. Default: 'filename'. */
|
||||
datePattern?: 'filename' | 'dirname' | 'none';
|
||||
/** How to extract title. Default: 'filename'. */
|
||||
titleStrategy?: 'filename' | 'heading' | 'filename-full';
|
||||
}
|
||||
|
||||
// ─── Directory Rules ─────────────────────────────────────────────────
|
||||
// Ordered from most specific to least specific. First match wins.
|
||||
// Add new directory conventions here.
|
||||
|
||||
export const DIRECTORY_RULES: DirectoryRule[] = [
|
||||
// Apple Notes — bulk import from Apple Notes app. Filenames are
|
||||
// "YYYY-MM-DD Title.md" with HTML-styled content.
|
||||
{
|
||||
pathPrefix: 'apple notes/youtube shows/',
|
||||
type: 'apple-note',
|
||||
source: 'apple-notes',
|
||||
tags: ['youtube', 'shows'],
|
||||
datePattern: 'filename',
|
||||
titleStrategy: 'filename',
|
||||
},
|
||||
{
|
||||
pathPrefix: 'apple notes/yc/',
|
||||
type: 'apple-note',
|
||||
source: 'apple-notes',
|
||||
tags: ['yc'],
|
||||
datePattern: 'filename',
|
||||
titleStrategy: 'filename',
|
||||
},
|
||||
{
|
||||
pathPrefix: 'apple notes/archived/',
|
||||
type: 'apple-note',
|
||||
source: 'apple-notes',
|
||||
tags: ['archived'],
|
||||
datePattern: 'filename',
|
||||
titleStrategy: 'filename',
|
||||
},
|
||||
{
|
||||
pathPrefix: 'apple notes/politics/',
|
||||
type: 'apple-note',
|
||||
source: 'apple-notes',
|
||||
tags: ['politics'],
|
||||
datePattern: 'filename',
|
||||
titleStrategy: 'filename',
|
||||
},
|
||||
{
|
||||
pathPrefix: 'apple notes/pitch notes/',
|
||||
type: 'apple-note',
|
||||
source: 'apple-notes',
|
||||
tags: ['pitch-notes'],
|
||||
datePattern: 'filename',
|
||||
titleStrategy: 'filename',
|
||||
},
|
||||
{
|
||||
pathPrefix: 'apple notes/gstack/',
|
||||
type: 'apple-note',
|
||||
source: 'apple-notes',
|
||||
tags: ['gstack'],
|
||||
datePattern: 'filename',
|
||||
titleStrategy: 'filename',
|
||||
},
|
||||
{
|
||||
pathPrefix: 'apple notes/photo-cameras/',
|
||||
type: 'apple-note',
|
||||
source: 'apple-notes',
|
||||
tags: ['photography'],
|
||||
datePattern: 'filename',
|
||||
titleStrategy: 'filename',
|
||||
},
|
||||
{
|
||||
pathPrefix: 'apple notes/jan bowman notes/',
|
||||
type: 'apple-note',
|
||||
source: 'apple-notes',
|
||||
tags: ['therapy', 'jan-bowman'],
|
||||
datePattern: 'filename',
|
||||
titleStrategy: 'filename',
|
||||
},
|
||||
// Catch-all for Apple Notes not in a subfolder
|
||||
{
|
||||
pathPrefix: 'apple notes/',
|
||||
type: 'apple-note',
|
||||
source: 'apple-notes',
|
||||
datePattern: 'filename',
|
||||
titleStrategy: 'filename',
|
||||
},
|
||||
|
||||
// Calendar diarization files
|
||||
{
|
||||
pathPrefix: 'daily/calendar/',
|
||||
type: 'calendar-index',
|
||||
source: 'calendar',
|
||||
datePattern: 'filename',
|
||||
titleStrategy: 'filename',
|
||||
},
|
||||
|
||||
// Personal sections
|
||||
{
|
||||
pathPrefix: 'personal/therapy/',
|
||||
type: 'therapy-session',
|
||||
source: 'therapy',
|
||||
datePattern: 'filename',
|
||||
titleStrategy: 'filename',
|
||||
},
|
||||
{
|
||||
pathPrefix: 'personal/reflections/',
|
||||
type: 'reflection',
|
||||
source: 'personal',
|
||||
datePattern: 'filename',
|
||||
titleStrategy: 'heading',
|
||||
},
|
||||
{
|
||||
pathPrefix: 'personal/',
|
||||
type: 'personal',
|
||||
source: 'personal',
|
||||
datePattern: 'none',
|
||||
titleStrategy: 'heading',
|
||||
},
|
||||
|
||||
// Writing
|
||||
{
|
||||
pathPrefix: 'writing/essays/',
|
||||
type: 'essay',
|
||||
source: 'writing',
|
||||
datePattern: 'filename',
|
||||
titleStrategy: 'heading',
|
||||
},
|
||||
{
|
||||
pathPrefix: 'writing/ideas/',
|
||||
type: 'idea',
|
||||
source: 'writing',
|
||||
datePattern: 'filename',
|
||||
titleStrategy: 'heading',
|
||||
},
|
||||
{
|
||||
pathPrefix: 'writing/',
|
||||
type: 'writing',
|
||||
source: 'writing',
|
||||
datePattern: 'filename',
|
||||
titleStrategy: 'heading',
|
||||
},
|
||||
|
||||
// Entity directories — these should already have frontmatter in most cases,
|
||||
// but the 55 people pages etc. that don't get handled here.
|
||||
{ pathPrefix: 'people/', type: 'person', titleStrategy: 'heading' },
|
||||
{ pathPrefix: 'companies/', type: 'company', titleStrategy: 'heading' },
|
||||
{ pathPrefix: 'projects/', type: 'project', titleStrategy: 'heading' },
|
||||
{ pathPrefix: 'civic/', type: 'civic', titleStrategy: 'heading' },
|
||||
{ pathPrefix: 'events/', type: 'event', titleStrategy: 'heading', datePattern: 'filename' },
|
||||
{ pathPrefix: 'meetings/', type: 'meeting', titleStrategy: 'heading', datePattern: 'filename' },
|
||||
{ pathPrefix: 'media/', type: 'media', titleStrategy: 'heading' },
|
||||
|
||||
// Catch-all for any remaining files
|
||||
{ pathPrefix: '', type: 'note', titleStrategy: 'heading' },
|
||||
];
|
||||
|
||||
// ─── Date extraction ─────────────────────────────────────────────────
|
||||
|
||||
/** Extract YYYY-MM-DD date from a filename like "2010-04-13 Apr 13 founders mtg.md" */
|
||||
export function extractDateFromFilename(filename: string): string | null {
|
||||
// Pattern 1: YYYY-MM-DD prefix (with - or space separator after)
|
||||
const m1 = filename.match(/^(\d{4}-\d{2}-\d{2})[\s_-]/);
|
||||
if (m1) return m1[1];
|
||||
|
||||
// Pattern 2: YYYY-MM-DD anywhere in filename
|
||||
const m2 = filename.match(/(\d{4}-\d{2}-\d{2})/);
|
||||
if (m2) return m2[1];
|
||||
|
||||
// Pattern 3: "YYYY MM DD" with spaces
|
||||
const m3 = filename.match(/^(\d{4})\s+(\d{2})\s+(\d{2})\s/);
|
||||
if (m3) return `${m3[1]}-${m3[2]}-${m3[3]}`;
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
// ─── Title extraction ────────────────────────────────────────────────
|
||||
|
||||
/** Extract title from filename, stripping date prefix and extension. */
|
||||
export function extractTitleFromFilename(filename: string): string {
|
||||
// Remove .md extension
|
||||
let title = filename.replace(/\.md$/i, '');
|
||||
|
||||
// Strip YYYY-MM-DD prefix (with separator)
|
||||
title = title.replace(/^\d{4}-\d{2}-\d{2}[\s_-]+/, '');
|
||||
|
||||
// Strip YYYY MM DD prefix (space-separated)
|
||||
title = title.replace(/^\d{4}\s+\d{2}\s+\d{2}\s+/, '');
|
||||
|
||||
// Clean up: title case, replace dashes/underscores with spaces
|
||||
title = title
|
||||
.replace(/[-_]/g, ' ')
|
||||
.replace(/\s+/g, ' ')
|
||||
.trim();
|
||||
|
||||
// Don't title-case if it already has mixed case (e.g., "YC presidency")
|
||||
if (title === title.toLowerCase() || title === title.toUpperCase()) {
|
||||
title = title.replace(/\b\w/g, c => c.toUpperCase());
|
||||
}
|
||||
|
||||
return title || 'Untitled';
|
||||
}
|
||||
|
||||
/** Extract title from first heading (# ...) in content. */
|
||||
export function extractTitleFromHeading(content: string): string | null {
|
||||
const lines = content.split('\n');
|
||||
for (const line of lines.slice(0, 20)) {
|
||||
const m = line.match(/^#\s+(.+)/);
|
||||
if (m) return m[1].trim();
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
// ─── Core inference ──────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Infer frontmatter for a file that has none.
|
||||
*
|
||||
* @param relativePath - Path relative to brain root (e.g., "Apple Notes/2010-04-13 Apr 13 founders mtg.md")
|
||||
* @param content - File content (may be empty)
|
||||
* @returns Inferred frontmatter fields
|
||||
*/
|
||||
export function inferFrontmatter(relativePath: string, content: string): InferredFrontmatter {
|
||||
// Check if file already has frontmatter
|
||||
const firstNonEmpty = content.split('\n').find(l => l.trim().length > 0);
|
||||
if (firstNonEmpty?.trim() === '---') {
|
||||
return { title: '', type: '', skipped: true };
|
||||
}
|
||||
|
||||
const lowerPath = relativePath.toLowerCase();
|
||||
const filename = basename(relativePath);
|
||||
|
||||
// Find matching rule
|
||||
let matchedRule: DirectoryRule | undefined;
|
||||
for (const rule of DIRECTORY_RULES) {
|
||||
if (lowerPath.startsWith(rule.pathPrefix.toLowerCase())) {
|
||||
matchedRule = rule;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
// Default rule if none matched
|
||||
if (!matchedRule) {
|
||||
matchedRule = { pathPrefix: '', type: 'note', titleStrategy: 'heading' };
|
||||
}
|
||||
|
||||
// Extract date
|
||||
let date: string | undefined;
|
||||
const datePattern = matchedRule.datePattern ?? 'filename';
|
||||
if (datePattern === 'filename') {
|
||||
date = extractDateFromFilename(filename) ?? undefined;
|
||||
}
|
||||
|
||||
// Extract title
|
||||
let title: string;
|
||||
const titleStrategy = matchedRule.titleStrategy ?? 'filename';
|
||||
if (titleStrategy === 'heading') {
|
||||
title = extractTitleFromHeading(content) ?? extractTitleFromFilename(filename);
|
||||
} else if (titleStrategy === 'filename-full') {
|
||||
title = filename.replace(/\.md$/i, '').replace(/[-_]/g, ' ').trim();
|
||||
} else {
|
||||
title = extractTitleFromFilename(filename);
|
||||
}
|
||||
|
||||
// Build tags from rule + subfolder
|
||||
const tags = [...(matchedRule.tags ?? [])];
|
||||
// Add subfolder as tag for Apple Notes (e.g., "YC", "Politics")
|
||||
if (matchedRule.source === 'apple-notes' && matchedRule.pathPrefix === 'apple notes/') {
|
||||
const parts = relativePath.split('/');
|
||||
if (parts.length > 2) {
|
||||
const subfolder = parts[1].toLowerCase().replace(/\s+/g, '-');
|
||||
if (!tags.includes(subfolder)) tags.push(subfolder);
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
title,
|
||||
type: matchedRule.type,
|
||||
date,
|
||||
source: matchedRule.source,
|
||||
tags: tags.length > 0 ? tags : undefined,
|
||||
matchedRule: matchedRule.pathPrefix || '(default)',
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Generate a YAML frontmatter block from inferred fields.
|
||||
* Returns the `---\n...\n---\n` string to prepend to content.
|
||||
*/
|
||||
export function serializeFrontmatter(fm: InferredFrontmatter): string {
|
||||
if (fm.skipped) return '';
|
||||
|
||||
const lines: string[] = ['---'];
|
||||
|
||||
// Title — quote if it contains special YAML chars
|
||||
const needsQuote = /[:"'#\[\]{}|>&*!?,]/.test(fm.title);
|
||||
lines.push(`title: ${needsQuote ? JSON.stringify(fm.title) : fm.title}`);
|
||||
|
||||
lines.push(`type: ${fm.type}`);
|
||||
|
||||
if (fm.date) {
|
||||
lines.push(`date: "${fm.date}"`);
|
||||
}
|
||||
|
||||
if (fm.source) {
|
||||
lines.push(`source: ${fm.source}`);
|
||||
}
|
||||
|
||||
if (fm.tags && fm.tags.length > 0) {
|
||||
lines.push(`tags: [${fm.tags.map(t => JSON.stringify(t)).join(', ')}]`);
|
||||
}
|
||||
|
||||
lines.push('---');
|
||||
return lines.join('\n') + '\n';
|
||||
}
|
||||
|
||||
/**
|
||||
* Apply frontmatter inference to file content.
|
||||
* Returns the content with frontmatter prepended, or the original content if it already has frontmatter.
|
||||
*/
|
||||
export function applyInference(relativePath: string, content: string): { content: string; inferred: InferredFrontmatter } {
|
||||
const inferred = inferFrontmatter(relativePath, content);
|
||||
if (inferred.skipped) {
|
||||
return { content, inferred };
|
||||
}
|
||||
const fm = serializeFrontmatter(inferred);
|
||||
return { content: fm + '\n' + content, inferred };
|
||||
}
|
||||
+16
-2
@@ -339,7 +339,7 @@ export async function importFromFile(
|
||||
engine: BrainEngine,
|
||||
filePath: string,
|
||||
relativePath: string,
|
||||
opts: { noEmbed?: boolean } = {},
|
||||
opts: { noEmbed?: boolean; inferFrontmatter?: boolean } = {},
|
||||
): Promise<ImportResult> {
|
||||
// Defense-in-depth: reject symlinks before reading content.
|
||||
const lstat = lstatSync(filePath);
|
||||
@@ -352,13 +352,27 @@ export async function importFromFile(
|
||||
return { slug: relativePath, status: 'skipped', chunks: 0, error: `File too large (${stat.size} bytes)` };
|
||||
}
|
||||
|
||||
const content = readFileSync(filePath, 'utf-8');
|
||||
let content = readFileSync(filePath, 'utf-8');
|
||||
|
||||
// Route code files through the code import path
|
||||
if (isCodeFilePath(relativePath)) {
|
||||
return importCodeFile(engine, relativePath, content, opts);
|
||||
}
|
||||
|
||||
// v0.22.8 — Frontmatter inference: if the file has no frontmatter and
|
||||
// inference is enabled, synthesize it from the filesystem path + content.
|
||||
// This turns bare markdown files into fully-typed, dated, tagged pages
|
||||
// without requiring the user to manually add YAML headers.
|
||||
// The inference is applied to the in-memory content only; the file on disk
|
||||
// is not modified. Use `gbrain frontmatter generate --fix` to write back.
|
||||
if (opts.inferFrontmatter !== false) {
|
||||
const { applyInference } = await import('./frontmatter-inference.ts');
|
||||
const { content: inferred, inferred: meta } = applyInference(relativePath, content);
|
||||
if (!meta.skipped) {
|
||||
content = inferred;
|
||||
}
|
||||
}
|
||||
|
||||
const parsed = parseMarkdown(content, relativePath);
|
||||
|
||||
// Enforce path-authoritative slug. parseMarkdown prefers frontmatter.slug over
|
||||
|
||||
@@ -225,8 +225,12 @@ export class MinionSupervisor {
|
||||
process.on('SIGTERM', this.sigtermListener);
|
||||
process.on('SIGINT', this.sigintListener);
|
||||
|
||||
// 4. Health monitoring.
|
||||
this.healthTimer = setInterval(() => { void this.healthCheck(); }, this.opts.healthInterval);
|
||||
// 4. Health monitoring. Skip when healthInterval=0 — that's the explicit
|
||||
// "disable" contract documented on `--health-interval 0`. setInterval(0)
|
||||
// would be a tight DB-hammering loop, not the no-op users expect.
|
||||
if (this.opts.healthInterval > 0) {
|
||||
this.healthTimer = setInterval(() => { void this.healthCheck(); }, this.opts.healthInterval);
|
||||
}
|
||||
|
||||
// 5. Announce start.
|
||||
this.emit('started', {
|
||||
@@ -427,6 +431,11 @@ export class MinionSupervisor {
|
||||
} else {
|
||||
delete env.GBRAIN_ALLOW_SHELL_JOBS;
|
||||
}
|
||||
// Signal to the child worker that it's running under a supervisor.
|
||||
// The worker's self-health-check (DB probes, stall detection) is
|
||||
// redundant when the supervisor already provides these — setting
|
||||
// this env var causes the worker to skip its own health timer.
|
||||
env.GBRAIN_SUPERVISED = '1';
|
||||
|
||||
this.lastStartTime = Date.now();
|
||||
|
||||
|
||||
@@ -170,6 +170,25 @@ export interface MinionWorkerOpts {
|
||||
* case where all concurrency slots are wedged with zero job completions
|
||||
* so the per-job check never fires. */
|
||||
rssCheckInterval?: number;
|
||||
/** Self-health-check interval in ms. 0 = disabled. Default: 60000 (1 minute).
|
||||
* Automatically disabled when running under a supervisor (GBRAIN_SUPERVISED=1).
|
||||
* Provides DB liveness probes and stall detection for bare `gbrain jobs work`
|
||||
* deployments managed by external process managers (systemd, Docker, cron). */
|
||||
healthCheckInterval?: number;
|
||||
/** Stall detection: ms of continuous idle (waiting>0, inFlight=0, no completions)
|
||||
* before emitting the first warning. Default: 300000 (5 minutes). */
|
||||
stallWarnAfterMs?: number;
|
||||
/** Stall detection: ms of continuous idle before emitting `'unhealthy'` with
|
||||
* reason='stalled'. Default: 600000 (10 minutes). Must be > stallWarnAfterMs. */
|
||||
stallExitAfterMs?: number;
|
||||
/** DB liveness probe: number of consecutive failed `SELECT 1` probes before
|
||||
* emitting `'unhealthy'` with reason='db_dead'. Default: 3. */
|
||||
dbFailExitAfter?: number;
|
||||
/** Per-probe wall-clock timeout in ms. A `SELECT 1` that hangs longer than
|
||||
* this counts as a failure (fed into dbFailExitAfter). Without this, a
|
||||
* hung probe would wedge the recursive setTimeout chain forever and
|
||||
* silently disable the health monitor. Default: 10000 (10 seconds). */
|
||||
dbProbeTimeoutMs?: number;
|
||||
}
|
||||
|
||||
// --- Job Context (passed to handlers) ---
|
||||
|
||||
+209
-1
@@ -20,8 +20,15 @@ import { UnrecoverableError } from './types.ts';
|
||||
import { MinionQueue } from './queue.ts';
|
||||
import { calculateBackoff } from './backoff.ts';
|
||||
import { randomUUID } from 'crypto';
|
||||
import { EventEmitter } from 'events';
|
||||
import { evaluateQuietHours, type QuietHoursConfig } from './quiet-hours.ts';
|
||||
|
||||
/** Reason payload emitted with `'unhealthy'` when self-health-check trips.
|
||||
* CLI layer (jobs.ts:work) subscribes and decides whether to call process.exit. */
|
||||
export type UnhealthyReason =
|
||||
| { reason: 'db_dead'; consecutiveFailures: number; message: string }
|
||||
| { reason: 'stalled'; waitingCount: number; idleMinutes: number };
|
||||
|
||||
/**
|
||||
* Read the quiet_hours JSONB column off a MinionJob, if present. The
|
||||
* column was added in schema migration v12; older rows + versions of
|
||||
@@ -42,7 +49,13 @@ interface InFlightJob {
|
||||
promise: Promise<void>;
|
||||
}
|
||||
|
||||
export class MinionWorker {
|
||||
/** Type-safe `on('unhealthy', ...)` for callers. */
|
||||
export interface MinionWorker {
|
||||
on(event: 'unhealthy', listener: (info: UnhealthyReason) => void): this;
|
||||
emit(event: 'unhealthy', info: UnhealthyReason): boolean;
|
||||
}
|
||||
|
||||
export class MinionWorker extends EventEmitter {
|
||||
private queue: MinionQueue;
|
||||
private handlers = new Map<string, MinionHandler>();
|
||||
private running = false;
|
||||
@@ -67,6 +80,7 @@ export class MinionWorker {
|
||||
private engine: BrainEngine,
|
||||
opts?: MinionWorkerOpts & MinionQueueOpts,
|
||||
) {
|
||||
super();
|
||||
this.queue = new MinionQueue(engine, {
|
||||
maxSpawnDepth: opts?.maxSpawnDepth,
|
||||
maxAttachmentBytes: opts?.maxAttachmentBytes,
|
||||
@@ -81,7 +95,25 @@ export class MinionWorker {
|
||||
maxRssMb: opts?.maxRssMb ?? 0,
|
||||
getRss: opts?.getRss ?? (() => process.memoryUsage().rss),
|
||||
rssCheckInterval: opts?.rssCheckInterval ?? 60000,
|
||||
healthCheckInterval: opts?.healthCheckInterval ?? 60000,
|
||||
stallWarnAfterMs: opts?.stallWarnAfterMs ?? 5 * 60_000,
|
||||
stallExitAfterMs: opts?.stallExitAfterMs ?? 10 * 60_000,
|
||||
dbFailExitAfter: opts?.dbFailExitAfter ?? 3,
|
||||
dbProbeTimeoutMs: opts?.dbProbeTimeoutMs ?? 10_000,
|
||||
};
|
||||
// Stall thresholds contract: exit MUST be strictly greater than warn.
|
||||
// If exit <= warn, the warn-then-exit semantics break: a single tick at
|
||||
// idle > warn would set stallWarningSince and the subsequent tick at
|
||||
// idle > exit could fire immediately without giving operators visibility.
|
||||
// Reject misconfigurations at construction time so the failure mode is
|
||||
// a loud throw on startup rather than a quiet contract violation.
|
||||
if (this.opts.stallExitAfterMs <= this.opts.stallWarnAfterMs) {
|
||||
throw new Error(
|
||||
`MinionWorkerOpts: stallExitAfterMs (${this.opts.stallExitAfterMs}) must be > ` +
|
||||
`stallWarnAfterMs (${this.opts.stallWarnAfterMs}). ` +
|
||||
`The contract is "warn first, exit later" — they cannot fire on the same tick.`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/** Register a handler for a job type. */
|
||||
@@ -94,6 +126,28 @@ export class MinionWorker {
|
||||
return Array.from(this.handlers.keys());
|
||||
}
|
||||
|
||||
/** Emit 'unhealthy' with a no-listener fallback. The default contract is
|
||||
* fail-stop: pre-EventEmitter-refactor behavior was process.exit(1) inside
|
||||
* the timer; the refactor moved that responsibility to the CLI subscriber.
|
||||
* But direct API consumers without a listener would see emit() become a
|
||||
* no-op AND `healthExited=true` permanently disabling monitoring — a
|
||||
* silent regression. Solution: if no one subscribed, log and exit
|
||||
* ourselves so the worker dies and the PM restarts it. Subscribers
|
||||
* override this default by adding a listener before start(). */
|
||||
private emitUnhealthy(info: UnhealthyReason): void {
|
||||
if (this.listenerCount('unhealthy') === 0) {
|
||||
const detail = info.reason === 'db_dead'
|
||||
? `DB unreachable (${info.consecutiveFailures} probes): ${info.message}`
|
||||
: `worker stalled (${info.waitingCount} waiting, ${info.idleMinutes}m idle)`;
|
||||
console.error(
|
||||
`[health] FATAL: ${detail}. No 'unhealthy' listener registered; ` +
|
||||
`defaulting to process.exit(1) for process-manager restart.`,
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
this.emit('unhealthy', info);
|
||||
}
|
||||
|
||||
/** Start the worker loop. Blocks until stopped. */
|
||||
async start(): Promise<void> {
|
||||
if (this.handlers.size === 0) {
|
||||
@@ -155,6 +209,159 @@ export class MinionWorker {
|
||||
}, this.opts.rssCheckInterval);
|
||||
}
|
||||
|
||||
// Self-health-check — provides supervisor-grade monitoring for bare workers.
|
||||
// Disabled when running under a supervisor (GBRAIN_SUPERVISED=1) or when
|
||||
// healthCheckInterval is 0. Catches two failure modes that leave the process
|
||||
// alive but non-functional:
|
||||
// 1. DB connection death (Supabase/PgBouncer drops, network blip)
|
||||
// 2. Worker stall (event loop alive but not claiming/completing jobs)
|
||||
//
|
||||
// On failure, emits an `'unhealthy'` event with a structured reason. The
|
||||
// CLI layer (`src/commands/jobs.ts:work`) subscribes and decides whether to
|
||||
// call process.exit. Library code never calls process.exit directly so
|
||||
// MinionWorker stays embeddable in non-CLI contexts (tests, other hosts).
|
||||
//
|
||||
// Timer pattern: recursive setTimeout with a `running` flag, not setInterval.
|
||||
// setInterval queues callbacks even when the prior is still awaiting; on a
|
||||
// hung DB probe that piles up overlapping async checks racing on
|
||||
// `consecutiveDbFailures`. The recursive pattern guarantees one tick at a time.
|
||||
const isSupervisedChild = process.env.GBRAIN_SUPERVISED === '1';
|
||||
let healthTimer: ReturnType<typeof setTimeout> | null = null;
|
||||
if (!isSupervisedChild && this.opts.healthCheckInterval > 0) {
|
||||
let consecutiveDbFailures = 0;
|
||||
let lastKnownCompleted = this.jobsCompleted;
|
||||
let lastCompletionTime = Date.now();
|
||||
let stallWarningSince: number | null = null;
|
||||
let healthRunning = false;
|
||||
let healthExited = false;
|
||||
|
||||
// Race executeRaw against a wall-clock deadline. A hung connection
|
||||
// (network-partitioned PgBouncer, deadlocked backend) would otherwise
|
||||
// hold the await forever — the recursive setTimeout's next tick is only
|
||||
// scheduled in `finally`, so a hung probe would silently disable the
|
||||
// entire health monitor. The timeout treats hangs as failures and feeds
|
||||
// them into `dbFailExitAfter`.
|
||||
const probeWithTimeout = async (): Promise<void> => {
|
||||
const ac = new AbortController();
|
||||
const timeoutMs = this.opts.dbProbeTimeoutMs;
|
||||
const timer = setTimeout(() => ac.abort(), timeoutMs);
|
||||
try {
|
||||
await Promise.race([
|
||||
this.engine.executeRaw('SELECT 1'),
|
||||
new Promise<never>((_, reject) => {
|
||||
ac.signal.addEventListener('abort', () => {
|
||||
reject(new Error(`probe timeout after ${timeoutMs}ms`));
|
||||
});
|
||||
}),
|
||||
]);
|
||||
} finally {
|
||||
clearTimeout(timer);
|
||||
}
|
||||
};
|
||||
|
||||
const runHealthCheck = async (): Promise<void> => {
|
||||
if (healthRunning || !this.running || healthExited) return;
|
||||
healthRunning = true;
|
||||
try {
|
||||
// --- 1. DB liveness probe ---
|
||||
try {
|
||||
await probeWithTimeout();
|
||||
consecutiveDbFailures = 0;
|
||||
} catch (e) {
|
||||
consecutiveDbFailures++;
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
console.error(
|
||||
`[health] DB probe failed (${consecutiveDbFailures}/${this.opts.dbFailExitAfter}): ${msg}`,
|
||||
);
|
||||
if (consecutiveDbFailures >= this.opts.dbFailExitAfter) {
|
||||
console.error(
|
||||
`[health] DB unreachable after ${this.opts.dbFailExitAfter} consecutive probes. ` +
|
||||
`Emitting 'unhealthy' for process-manager restart.`,
|
||||
);
|
||||
healthExited = true;
|
||||
this.emitUnhealthy({
|
||||
reason: 'db_dead',
|
||||
consecutiveFailures: consecutiveDbFailures,
|
||||
message: msg,
|
||||
});
|
||||
}
|
||||
return; // Skip stall check when DB is flaky
|
||||
}
|
||||
|
||||
// --- 2. Stall detection ---
|
||||
if (this.jobsCompleted > lastKnownCompleted) {
|
||||
lastKnownCompleted = this.jobsCompleted;
|
||||
lastCompletionTime = Date.now();
|
||||
stallWarningSince = null;
|
||||
}
|
||||
|
||||
const idleMs = Date.now() - lastCompletionTime;
|
||||
|
||||
// Only check for stalls when no jobs are in-flight and it's been a while
|
||||
if (idleMs > this.opts.stallWarnAfterMs && this.inFlight.size === 0) {
|
||||
try {
|
||||
// Filter by registered handler names so a worker that doesn't
|
||||
// claim a particular job-name doesn't false-positive when those
|
||||
// jobs accumulate in `waiting`. Only counts work THIS worker would
|
||||
// actually have claimed.
|
||||
const handlerNames = this.registeredNames;
|
||||
const rows = handlerNames.length === 0
|
||||
? [] as { cnt: string }[]
|
||||
: await this.engine.executeRaw<{ cnt: string }>(
|
||||
`SELECT count(*)::text AS cnt FROM minion_jobs
|
||||
WHERE status = 'waiting'
|
||||
AND queue = $1
|
||||
AND name = ANY($2::text[])`,
|
||||
[this.opts.queue, handlerNames],
|
||||
);
|
||||
const waiting = parseInt(rows[0]?.cnt ?? '0', 10);
|
||||
const idleMinutes = Math.round(idleMs / 60_000);
|
||||
if (waiting > 0) {
|
||||
// Two thresholds, both measured from `lastCompletionTime` (NOT
|
||||
// from when the warning fired). With defaults (warn=5min,
|
||||
// exit=10min), the first warning fires at idle=5min and the
|
||||
// unhealthy emit fires at idle=10min — matching the contract
|
||||
// documented in MinionWorkerOpts.
|
||||
if (!stallWarningSince) {
|
||||
stallWarningSince = Date.now();
|
||||
console.warn(
|
||||
`[health] Possible stall: ${waiting} waiting job(s) for ` +
|
||||
`registered handlers, 0 in-flight, ${idleMinutes}m since last completion`,
|
||||
);
|
||||
} else if (idleMs > this.opts.stallExitAfterMs) {
|
||||
console.error(
|
||||
`[health] Worker stalled for ${Math.round(this.opts.stallExitAfterMs / 60_000)}+ ` +
|
||||
`minutes with ${waiting} waiting job(s). Emitting 'unhealthy' for process-manager restart.`,
|
||||
);
|
||||
healthExited = true;
|
||||
this.emitUnhealthy({
|
||||
reason: 'stalled',
|
||||
waitingCount: waiting,
|
||||
idleMinutes,
|
||||
});
|
||||
}
|
||||
} else {
|
||||
stallWarningSince = null; // Queue empty (for our handlers) — not stalled, just idle
|
||||
}
|
||||
} catch {
|
||||
// DB query failed — the liveness probe above will catch persistent failures
|
||||
}
|
||||
} else {
|
||||
stallWarningSince = null;
|
||||
}
|
||||
} finally {
|
||||
healthRunning = false;
|
||||
if (this.running && !healthExited) {
|
||||
healthTimer = setTimeout(runHealthCheck, this.opts.healthCheckInterval);
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
// First tick scheduled after one interval so newly-started workers have
|
||||
// a chance to do real work before the stall clock starts ticking.
|
||||
healthTimer = setTimeout(runHealthCheck, this.opts.healthCheckInterval);
|
||||
}
|
||||
|
||||
try {
|
||||
while (this.running) {
|
||||
// Promote delayed jobs
|
||||
@@ -201,6 +408,7 @@ export class MinionWorker {
|
||||
} finally {
|
||||
clearInterval(stalledTimer);
|
||||
if (rssTimer) clearInterval(rssTimer);
|
||||
if (healthTimer) clearTimeout(healthTimer); // recursive setTimeout pattern
|
||||
process.removeListener('SIGTERM', shutdown);
|
||||
process.removeListener('SIGINT', shutdown);
|
||||
|
||||
|
||||
@@ -0,0 +1,101 @@
|
||||
/**
|
||||
* Shared concurrency policy for sync + import + jobs paths.
|
||||
*
|
||||
* Three callers used to embed three different policies:
|
||||
* - performSync (incremental): >100 files → 4 workers
|
||||
* - performFullSync: Postgres → 4 workers
|
||||
* - jobs.ts sync handler: hardcoded 4
|
||||
*
|
||||
* They drift over time and confuse users ("why does my sync not parallelize?"
|
||||
* is a different answer in each path). This module is one source of truth.
|
||||
*
|
||||
* v0.22.13 — extracted as part of the parallel-sync hardening (PR #490).
|
||||
*/
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
|
||||
/** Threshold above which auto-concurrency fires for incremental sync paths. */
|
||||
export const AUTO_CONCURRENCY_FILE_THRESHOLD = 100;
|
||||
|
||||
/** Minimum file count below which the parallel branch is skipped even when
|
||||
* auto-concurrency would otherwise fire. Prevents spawning workers for trivial
|
||||
* diffs where setup cost exceeds parallelism gains. Only consulted on the
|
||||
* auto path; explicit `--workers N` bypasses this. */
|
||||
export const PARALLEL_FILE_FLOOR = 50;
|
||||
|
||||
/** Default worker count when auto-concurrency fires. */
|
||||
export const DEFAULT_PARALLEL_WORKERS = 4;
|
||||
|
||||
/**
|
||||
* Resolve effective worker count for a sync/import operation.
|
||||
*
|
||||
* Inputs:
|
||||
* - engine.kind: 'pglite' always returns 1 (single-connection)
|
||||
* - override: caller's explicit --workers / opts.concurrency value
|
||||
* - fileCount: size of the work batch
|
||||
*
|
||||
* Rules:
|
||||
* - PGLite → always 1 (the engine is single-connection regardless)
|
||||
* - explicit override → respect it (clamped to >=1)
|
||||
* - auto path → DEFAULT_PARALLEL_WORKERS when fileCount > AUTO_CONCURRENCY_FILE_THRESHOLD, else 1
|
||||
*
|
||||
* Note: this function does NOT consult PARALLEL_FILE_FLOOR. The floor is a
|
||||
* caller-side gate that decides whether to take the parallel code path even
|
||||
* when the worker count is > 1. It only applies to the auto path; explicit
|
||||
* --workers bypasses the floor entirely (per Q1 in PR #490).
|
||||
*/
|
||||
export function autoConcurrency(
|
||||
engine: BrainEngine,
|
||||
fileCount: number,
|
||||
override?: number,
|
||||
): number {
|
||||
if (engine.kind === 'pglite') return 1;
|
||||
if (override !== undefined) return Math.max(1, override);
|
||||
return fileCount > AUTO_CONCURRENCY_FILE_THRESHOLD
|
||||
? DEFAULT_PARALLEL_WORKERS
|
||||
: 1;
|
||||
}
|
||||
|
||||
/**
|
||||
* Decide whether the parallel code path should run.
|
||||
*
|
||||
* - workers <= 1 → never parallel
|
||||
* - workers > 1 + explicit override → always parallel (user opted in,
|
||||
* respect them even on small diffs — Q1 in PR #490)
|
||||
* - workers > 1 + auto path → parallel only when fileCount > PARALLEL_FILE_FLOOR
|
||||
*/
|
||||
export function shouldRunParallel(
|
||||
workers: number,
|
||||
fileCount: number,
|
||||
explicit: boolean,
|
||||
): boolean {
|
||||
if (workers <= 1) return false;
|
||||
if (explicit) return true;
|
||||
return fileCount > PARALLEL_FILE_FLOOR;
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse a `--workers N` / `--concurrency N` CLI argument value.
|
||||
*
|
||||
* Returns:
|
||||
* - undefined when the flag was not provided
|
||||
* - a positive integer when the flag was provided with a valid value
|
||||
*
|
||||
* Throws on:
|
||||
* - non-integer ("foo", "1.5", "")
|
||||
* - zero or negative ("0", "-3")
|
||||
* - NaN / Infinity
|
||||
*
|
||||
* Q2 in PR #490: the prior parseInt-with-no-validation accepted `--workers 0`
|
||||
* and silently fell through to auto-concurrency (4 workers), the opposite of
|
||||
* what the user typed. Fail loud instead.
|
||||
*/
|
||||
export function parseWorkers(s: string | undefined): number | undefined {
|
||||
if (s === undefined) return undefined;
|
||||
const n = parseInt(s, 10);
|
||||
if (!Number.isFinite(n) || n < 1 || String(n) !== s.trim()) {
|
||||
throw new Error(
|
||||
`--workers must be a positive integer, got: ${JSON.stringify(s)}`,
|
||||
);
|
||||
}
|
||||
return n;
|
||||
}
|
||||
@@ -0,0 +1,167 @@
|
||||
/**
|
||||
* E2E test for parallel sync against real Postgres.
|
||||
*
|
||||
* T2 — happy path: 60-file sync at concurrency=4 against PostgresEngine
|
||||
* actually constructs N worker engines, imports correctly, and does
|
||||
* not leak connections (probe pg_stat_activity before/after).
|
||||
* P4 — benchmark: serial vs concurrency=4 timing on the same fixture so
|
||||
* the v0.22.13 CHANGELOG can quote a real number instead of "~4×".
|
||||
*
|
||||
* Gated on DATABASE_URL. Run via:
|
||||
* docker run -d --name gbrain-test-pg -e POSTGRES_USER=postgres \
|
||||
* -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=gbrain_test \
|
||||
* -p 5435:5432 pgvector/pgvector:pg16
|
||||
* DATABASE_URL=postgresql://postgres:postgres@localhost:5435/gbrain_test \
|
||||
* bun test test/e2e/sync-parallel.test.ts
|
||||
*/
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { mkdtempSync, rmSync, writeFileSync, mkdirSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import { execSync } from 'child_process';
|
||||
import { hasDatabase, setupDB, teardownDB, getEngine, getConn } from './helpers.ts';
|
||||
|
||||
const skip = !hasDatabase();
|
||||
const describeE2E = skip ? describe.skip : describe;
|
||||
|
||||
if (skip) {
|
||||
console.log('Skipping E2E sync-parallel tests (DATABASE_URL not set)');
|
||||
}
|
||||
|
||||
function seedRepo(repoPath: string, fileCount: number): string {
|
||||
execSync('git init', { cwd: repoPath, stdio: 'pipe' });
|
||||
execSync('git config user.email "test@test.com"', { cwd: repoPath, stdio: 'pipe' });
|
||||
execSync('git config user.name "Test"', { cwd: repoPath, stdio: 'pipe' });
|
||||
mkdirSync(join(repoPath, 'people'), { recursive: true });
|
||||
for (let i = 0; i < fileCount; i++) {
|
||||
writeFileSync(join(repoPath, `people/p${i}.md`), [
|
||||
'---',
|
||||
'type: person',
|
||||
`title: Person ${i}`,
|
||||
'---',
|
||||
'',
|
||||
`Person ${i} body — some text long enough to chunk.`,
|
||||
`Iteration index ${i}, generated by sync-parallel E2E.`,
|
||||
].join('\n'));
|
||||
}
|
||||
execSync('git add -A && git commit -m "initial"', { cwd: repoPath, stdio: 'pipe' });
|
||||
return execSync('git rev-parse HEAD', { cwd: repoPath, encoding: 'utf-8' }).trim();
|
||||
}
|
||||
|
||||
async function activeConnections(): Promise<number> {
|
||||
const conn = getConn();
|
||||
const rows = await conn.unsafe(`
|
||||
SELECT count(*) AS n FROM pg_stat_activity
|
||||
WHERE datname = current_database()
|
||||
AND state IS NOT NULL
|
||||
`) as Array<{ n: string }>;
|
||||
return parseInt(rows[0]?.n ?? '0', 10);
|
||||
}
|
||||
|
||||
describeE2E('E2E sync-parallel: T2 happy path + leak probe', () => {
|
||||
let repoPath: string;
|
||||
|
||||
beforeAll(async () => {
|
||||
await setupDB();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
if (repoPath) rmSync(repoPath, { recursive: true, force: true });
|
||||
await teardownDB();
|
||||
});
|
||||
|
||||
test('60-file Postgres sync at concurrency=4 imports all + no connection leak', async () => {
|
||||
repoPath = mkdtempSync(join(tmpdir(), 'gbrain-e2e-par-'));
|
||||
seedRepo(repoPath, 60);
|
||||
|
||||
const before = await activeConnections();
|
||||
|
||||
const { performSync } = await import('../../src/commands/sync.ts');
|
||||
const engine = getEngine();
|
||||
const result = await performSync(engine, {
|
||||
repoPath,
|
||||
noPull: true,
|
||||
noEmbed: true,
|
||||
concurrency: 4,
|
||||
});
|
||||
|
||||
// First sync routes through performFullSync (delegates to runImport which
|
||||
// also accepts --workers); status is 'first_sync'.
|
||||
expect(result.status).toBe('first_sync');
|
||||
|
||||
const after = await activeConnections();
|
||||
|
||||
// Allow some slack — the helper engine + sync's normal pool stay open.
|
||||
// Worker engines (4 × 2 = 8 connections) MUST have closed; if they
|
||||
// hadn't, after - before would be at least 8.
|
||||
expect(after - before).toBeLessThan(4);
|
||||
|
||||
// Verify pages are actually in the DB (via raw SQL — engine API also works).
|
||||
const conn = getConn();
|
||||
const pageRows = await conn.unsafe(
|
||||
`SELECT count(*) AS n FROM pages WHERE slug LIKE 'people/p%'`,
|
||||
) as Array<{ n: string }>;
|
||||
const count = parseInt(pageRows[0]?.n ?? '0', 10);
|
||||
expect(count).toBe(60);
|
||||
}, 60_000);
|
||||
});
|
||||
|
||||
describeE2E('E2E sync-parallel: P4 benchmark serial vs concurrency=4', () => {
|
||||
let repoSerial: string;
|
||||
let repoParallel: string;
|
||||
|
||||
beforeAll(async () => {
|
||||
await setupDB();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
if (repoSerial) rmSync(repoSerial, { recursive: true, force: true });
|
||||
if (repoParallel) rmSync(repoParallel, { recursive: true, force: true });
|
||||
await teardownDB();
|
||||
});
|
||||
|
||||
test('120-file benchmark: report serial and parallel wall-clock', async () => {
|
||||
// Two separate repos so neither sync's chunks bleed into the other.
|
||||
repoSerial = mkdtempSync(join(tmpdir(), 'gbrain-bench-serial-'));
|
||||
repoParallel = mkdtempSync(join(tmpdir(), 'gbrain-bench-parallel-'));
|
||||
seedRepo(repoSerial, 120);
|
||||
seedRepo(repoParallel, 120);
|
||||
|
||||
const { performSync } = await import('../../src/commands/sync.ts');
|
||||
const engine = getEngine();
|
||||
|
||||
// Truncate between runs to keep the benchmark honest.
|
||||
const conn = getConn();
|
||||
|
||||
const t1 = Date.now();
|
||||
await performSync(engine, {
|
||||
repoPath: repoSerial,
|
||||
noPull: true,
|
||||
noEmbed: true,
|
||||
concurrency: 1,
|
||||
});
|
||||
const serialMs = Date.now() - t1;
|
||||
|
||||
// Wipe pages before second run so neither one is "incremental".
|
||||
await conn.unsafe(`TRUNCATE pages CASCADE`);
|
||||
await conn.unsafe(`TRUNCATE config CASCADE`);
|
||||
|
||||
const t2 = Date.now();
|
||||
await performSync(engine, {
|
||||
repoPath: repoParallel,
|
||||
noPull: true,
|
||||
noEmbed: true,
|
||||
concurrency: 4,
|
||||
});
|
||||
const parallelMs = Date.now() - t2;
|
||||
|
||||
const speedup = (serialMs / parallelMs).toFixed(2);
|
||||
// Emit as a single line stdout consumers can grep for.
|
||||
console.log(`SYNC_PARALLEL_BENCH 120 files | serial=${serialMs}ms | parallel(4)=${parallelMs}ms | speedup=${speedup}x`);
|
||||
|
||||
// Soft assertion: parallel must not be slower than serial. The actual
|
||||
// speedup ratio depends heavily on Postgres latency profile and is what
|
||||
// the CHANGELOG quotes — don't gate the test on a specific multiplier.
|
||||
expect(parallelMs).toBeLessThanOrEqual(serialMs * 1.5); // +50% slack for noisy CI
|
||||
}, 120_000);
|
||||
});
|
||||
@@ -0,0 +1,283 @@
|
||||
/**
|
||||
* Tests for frontmatter-inference.ts — the zero-friction ingest pipeline.
|
||||
*
|
||||
* Validates that files without frontmatter get correct type, title, date,
|
||||
* source, and tags inferred from their filesystem path and content.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import {
|
||||
inferFrontmatter,
|
||||
extractDateFromFilename,
|
||||
extractTitleFromFilename,
|
||||
extractTitleFromHeading,
|
||||
serializeFrontmatter,
|
||||
applyInference,
|
||||
DIRECTORY_RULES,
|
||||
} from '../src/core/frontmatter-inference.ts';
|
||||
|
||||
// ── Date extraction ──────────────────────────────────────────────────
|
||||
|
||||
describe('extractDateFromFilename', () => {
|
||||
test('extracts YYYY-MM-DD from date-prefixed filename', () => {
|
||||
expect(extractDateFromFilename('2010-04-13 Apr 13 founders mtg.md')).toBe('2010-04-13');
|
||||
});
|
||||
|
||||
test('extracts date with dash separator', () => {
|
||||
expect(extractDateFromFilename('2024-01-30-therapy-session.md')).toBe('2024-01-30');
|
||||
});
|
||||
|
||||
test('extracts date with underscore separator', () => {
|
||||
expect(extractDateFromFilename('2023-06-15_meeting-notes.md')).toBe('2023-06-15');
|
||||
});
|
||||
|
||||
test('returns null for no-date filename', () => {
|
||||
expect(extractDateFromFilename('README.md')).toBe(null);
|
||||
});
|
||||
|
||||
test('returns null for filename with numbers but no date', () => {
|
||||
expect(extractDateFromFilename('chapter-1-intro.md')).toBe(null);
|
||||
});
|
||||
});
|
||||
|
||||
// ── Title extraction ─────────────────────────────────────────────────
|
||||
|
||||
describe('extractTitleFromFilename', () => {
|
||||
test('strips date prefix and cleans up', () => {
|
||||
expect(extractTitleFromFilename('2010-04-13 Apr 13 founders mtg.md')).toBe('Apr 13 founders mtg');
|
||||
});
|
||||
|
||||
test('strips YYYY-MM-DD- prefix', () => {
|
||||
expect(extractTitleFromFilename('2024-01-30-therapy-session.md')).toBe('Therapy Session');
|
||||
});
|
||||
|
||||
test('handles filename without date', () => {
|
||||
expect(extractTitleFromFilename('cognitive-distortions.md')).toBe('Cognitive Distortions');
|
||||
});
|
||||
|
||||
test('preserves mixed case', () => {
|
||||
expect(extractTitleFromFilename('YC presidency.md')).toBe('YC presidency');
|
||||
});
|
||||
|
||||
test('returns Untitled for empty result', () => {
|
||||
expect(extractTitleFromFilename('.md')).toBe('Untitled');
|
||||
});
|
||||
});
|
||||
|
||||
describe('extractTitleFromHeading', () => {
|
||||
test('extracts first # heading', () => {
|
||||
expect(extractTitleFromHeading('# Dhravya Shah\n\n> Founder of Supermemory')).toBe('Dhravya Shah');
|
||||
});
|
||||
|
||||
test('ignores ## headings', () => {
|
||||
expect(extractTitleFromHeading('Some text\n## Not this\n# This one')).toBe('This one');
|
||||
});
|
||||
|
||||
test('returns null when no heading found', () => {
|
||||
expect(extractTitleFromHeading('Just some text\nwithout headings')).toBe(null);
|
||||
});
|
||||
|
||||
test('looks within first 20 lines only', () => {
|
||||
const lines = Array(25).fill('text').join('\n') + '\n# Too Late';
|
||||
expect(extractTitleFromHeading(lines)).toBe(null);
|
||||
});
|
||||
});
|
||||
|
||||
// ── Core inference ───────────────────────────────────────────────────
|
||||
|
||||
describe('inferFrontmatter', () => {
|
||||
test('skips files that already have frontmatter', () => {
|
||||
const result = inferFrontmatter('people/alice.md', '---\ntitle: Alice\n---\n# Alice');
|
||||
expect(result.skipped).toBe(true);
|
||||
});
|
||||
|
||||
test('Apple Notes: infers type, date, title, source', () => {
|
||||
const result = inferFrontmatter(
|
||||
'Apple Notes/2010-04-13 Apr 13 founders mtg.md',
|
||||
'<span style="color:#000ff;">Top priority</span>',
|
||||
);
|
||||
expect(result.type).toBe('apple-note');
|
||||
expect(result.date).toBe('2010-04-13');
|
||||
expect(result.title).toBe('Apr 13 founders mtg');
|
||||
expect(result.source).toBe('apple-notes');
|
||||
});
|
||||
|
||||
test('Apple Notes/YC: adds yc tag', () => {
|
||||
const result = inferFrontmatter(
|
||||
'Apple Notes/YC/2022-08-04 Project 1783Y.md',
|
||||
'Some content',
|
||||
);
|
||||
expect(result.type).toBe('apple-note');
|
||||
expect(result.tags).toContain('yc');
|
||||
expect(result.date).toBe('2022-08-04');
|
||||
});
|
||||
|
||||
test('Apple Notes/Politics: adds politics tag', () => {
|
||||
const result = inferFrontmatter(
|
||||
'Apple Notes/Politics/2023-11-15 DA race notes.md',
|
||||
'Some content',
|
||||
);
|
||||
expect(result.tags).toContain('politics');
|
||||
});
|
||||
|
||||
test('people/ directory: type person, title from heading', () => {
|
||||
const result = inferFrontmatter(
|
||||
'people/dhravya-shah.md',
|
||||
'# Dhravya Shah\n\n> Founder of Supermemory',
|
||||
);
|
||||
expect(result.type).toBe('person');
|
||||
expect(result.title).toBe('Dhravya Shah');
|
||||
});
|
||||
|
||||
test('people/ directory: falls back to filename when no heading', () => {
|
||||
const result = inferFrontmatter(
|
||||
'people/john-doe.md',
|
||||
'Some text without a heading',
|
||||
);
|
||||
expect(result.type).toBe('person');
|
||||
expect(result.title).toBe('John Doe');
|
||||
});
|
||||
|
||||
test('personal/therapy: infers therapy-session type with date', () => {
|
||||
const result = inferFrontmatter(
|
||||
'personal/therapy/jan/2024-01-30.md',
|
||||
'Session notes...',
|
||||
);
|
||||
expect(result.type).toBe('therapy-session');
|
||||
expect(result.date).toBe('2024-01-30');
|
||||
expect(result.source).toBe('therapy');
|
||||
});
|
||||
|
||||
test('personal/reflections: infers reflection type, title from heading', () => {
|
||||
const result = inferFrontmatter(
|
||||
'personal/reflections/cognitive-distortions.md',
|
||||
'# Cognitive Distortions\n\nA list of common...',
|
||||
);
|
||||
expect(result.type).toBe('reflection');
|
||||
expect(result.title).toBe('Cognitive Distortions');
|
||||
});
|
||||
|
||||
test('writing/essays: infers essay type', () => {
|
||||
const result = inferFrontmatter(
|
||||
'writing/essays/2024-03-15-on-being-remembered.md',
|
||||
'# On Being Remembered Forever\n\nSome thoughts...',
|
||||
);
|
||||
expect(result.type).toBe('essay');
|
||||
expect(result.title).toBe('On Being Remembered Forever');
|
||||
expect(result.date).toBe('2024-03-15');
|
||||
});
|
||||
|
||||
test('daily/calendar: infers calendar-index type', () => {
|
||||
const result = inferFrontmatter(
|
||||
'daily/calendar/2026-01-15-yc-office-hours.md',
|
||||
'# Calendar Index\nSome calendar data',
|
||||
);
|
||||
expect(result.type).toBe('calendar-index');
|
||||
expect(result.source).toBe('calendar');
|
||||
});
|
||||
|
||||
test('companies/ directory: type company', () => {
|
||||
const result = inferFrontmatter(
|
||||
'companies/stripe.md',
|
||||
'# Stripe\n\n> Online payments infrastructure',
|
||||
);
|
||||
expect(result.type).toBe('company');
|
||||
expect(result.title).toBe('Stripe');
|
||||
});
|
||||
|
||||
test('unknown directory: defaults to note type with heading title', () => {
|
||||
const result = inferFrontmatter(
|
||||
'random/some-file.md',
|
||||
'# My Random Notes\n\nStuff here',
|
||||
);
|
||||
expect(result.type).toBe('note');
|
||||
expect(result.title).toBe('My Random Notes');
|
||||
});
|
||||
|
||||
test('handles empty content', () => {
|
||||
const result = inferFrontmatter('notes/empty.md', '');
|
||||
expect(result.type).toBe('note');
|
||||
expect(result.title).toBe('Empty');
|
||||
});
|
||||
});
|
||||
|
||||
// ── Serialization ────────────────────────────────────────────────────
|
||||
|
||||
describe('serializeFrontmatter', () => {
|
||||
test('generates valid YAML frontmatter', () => {
|
||||
const fm = serializeFrontmatter({
|
||||
title: 'Apr 13 founders mtg',
|
||||
type: 'apple-note',
|
||||
date: '2010-04-13',
|
||||
source: 'apple-notes',
|
||||
tags: ['yc'],
|
||||
});
|
||||
expect(fm).toContain('---');
|
||||
expect(fm).toContain('title: Apr 13 founders mtg');
|
||||
expect(fm).toContain('type: apple-note');
|
||||
expect(fm).toContain('date: "2010-04-13"');
|
||||
expect(fm).toContain('source: apple-notes');
|
||||
expect(fm).toContain('tags: ["yc"]');
|
||||
});
|
||||
|
||||
test('quotes title with special chars', () => {
|
||||
const fm = serializeFrontmatter({
|
||||
title: 'What\'s the deal: a "primer"',
|
||||
type: 'note',
|
||||
});
|
||||
expect(fm).toContain('title: "What\'s the deal: a \\"primer\\""');
|
||||
});
|
||||
|
||||
test('returns empty string for skipped files', () => {
|
||||
expect(serializeFrontmatter({ title: '', type: '', skipped: true })).toBe('');
|
||||
});
|
||||
|
||||
test('omits optional fields when absent', () => {
|
||||
const fm = serializeFrontmatter({ title: 'Test', type: 'note' });
|
||||
expect(fm).not.toContain('date');
|
||||
expect(fm).not.toContain('source');
|
||||
expect(fm).not.toContain('tags');
|
||||
});
|
||||
});
|
||||
|
||||
// ── Integration ──────────────────────────────────────────────────────
|
||||
|
||||
describe('applyInference', () => {
|
||||
test('prepends frontmatter to content without it', () => {
|
||||
const { content, inferred } = applyInference(
|
||||
'people/alice-smith.md',
|
||||
'# Alice Smith\n\n> Founder of FooBar',
|
||||
);
|
||||
expect(content).toMatch(/^---\n/);
|
||||
expect(content).toContain('type: person');
|
||||
expect(content).toContain('title: Alice Smith');
|
||||
expect(content).toContain('# Alice Smith');
|
||||
expect(inferred.skipped).toBeUndefined();
|
||||
});
|
||||
|
||||
test('returns original content for files with frontmatter', () => {
|
||||
const original = '---\ntitle: Bob\n---\n# Bob';
|
||||
const { content, inferred } = applyInference('people/bob.md', original);
|
||||
expect(content).toBe(original);
|
||||
expect(inferred.skipped).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
// ── Rules coverage ───────────────────────────────────────────────────
|
||||
|
||||
describe('DIRECTORY_RULES', () => {
|
||||
test('has a catch-all rule with empty prefix', () => {
|
||||
const catchAll = DIRECTORY_RULES.find(r => r.pathPrefix === '');
|
||||
expect(catchAll).toBeDefined();
|
||||
expect(catchAll!.type).toBe('note');
|
||||
});
|
||||
|
||||
test('Apple Notes rules are more specific than the catch-all', () => {
|
||||
const appleRules = DIRECTORY_RULES.filter(r => r.pathPrefix.startsWith('apple notes/'));
|
||||
expect(appleRules.length).toBeGreaterThan(1); // subfolder rules + catch-all
|
||||
// Subfolder rules should come before the generic apple notes/ rule
|
||||
const ycIdx = DIRECTORY_RULES.findIndex(r => r.pathPrefix === 'apple notes/yc/');
|
||||
const genericIdx = DIRECTORY_RULES.findIndex(r => r.pathPrefix === 'apple notes/');
|
||||
expect(ycIdx).toBeLessThan(genericIdx);
|
||||
});
|
||||
});
|
||||
@@ -2306,3 +2306,379 @@ describe('checkAborted (v0.20.5 cycle signal)', () => {
|
||||
}).toThrow('aborted between phases: timeout');
|
||||
});
|
||||
});
|
||||
|
||||
// --- v0.22.14: Self-health-check for bare workers ---
|
||||
|
||||
describe('MinionWorker: self-health-check', () => {
|
||||
test('health check is active when GBRAIN_SUPERVISED is not set', async () => {
|
||||
// Save and clear the env var
|
||||
const saved = process.env.GBRAIN_SUPERVISED;
|
||||
delete process.env.GBRAIN_SUPERVISED;
|
||||
|
||||
try {
|
||||
const worker = new MinionWorker(engine, {
|
||||
queue: 'default',
|
||||
concurrency: 1,
|
||||
healthCheckInterval: 100, // fast for testing
|
||||
pollInterval: 50,
|
||||
stalledInterval: 10_000,
|
||||
maxRssMb: 0,
|
||||
});
|
||||
|
||||
worker.register('noop', async () => {});
|
||||
await queue.add('noop', {});
|
||||
|
||||
const startPromise = worker.start();
|
||||
// Let the health check fire at least once (100ms interval)
|
||||
await new Promise(r => setTimeout(r, 300));
|
||||
worker.stop();
|
||||
await startPromise;
|
||||
|
||||
// Worker should have processed the job despite health check running
|
||||
const completed = await queue.getJobs({ status: 'completed' });
|
||||
expect(completed.length).toBeGreaterThanOrEqual(1);
|
||||
} finally {
|
||||
if (saved !== undefined) process.env.GBRAIN_SUPERVISED = saved;
|
||||
else delete process.env.GBRAIN_SUPERVISED;
|
||||
}
|
||||
}, 10_000);
|
||||
|
||||
test('health check is skipped when GBRAIN_SUPERVISED=1', async () => {
|
||||
const saved = process.env.GBRAIN_SUPERVISED;
|
||||
process.env.GBRAIN_SUPERVISED = '1';
|
||||
|
||||
try {
|
||||
const worker = new MinionWorker(engine, {
|
||||
queue: 'default',
|
||||
concurrency: 1,
|
||||
healthCheckInterval: 100,
|
||||
pollInterval: 50,
|
||||
stalledInterval: 10_000,
|
||||
maxRssMb: 0,
|
||||
});
|
||||
|
||||
worker.register('noop', async () => {});
|
||||
await queue.add('noop', {});
|
||||
|
||||
const startPromise = worker.start();
|
||||
await new Promise(r => setTimeout(r, 300));
|
||||
worker.stop();
|
||||
await startPromise;
|
||||
|
||||
// Worker should still process jobs fine
|
||||
const completed = await queue.getJobs({ status: 'completed' });
|
||||
expect(completed.length).toBeGreaterThanOrEqual(1);
|
||||
} finally {
|
||||
if (saved !== undefined) process.env.GBRAIN_SUPERVISED = saved;
|
||||
else delete process.env.GBRAIN_SUPERVISED;
|
||||
}
|
||||
}, 10_000);
|
||||
|
||||
test('healthCheckInterval=0 disables health check', async () => {
|
||||
delete process.env.GBRAIN_SUPERVISED;
|
||||
|
||||
const worker = new MinionWorker(engine, {
|
||||
queue: 'default',
|
||||
concurrency: 1,
|
||||
healthCheckInterval: 0,
|
||||
pollInterval: 50,
|
||||
stalledInterval: 10_000,
|
||||
maxRssMb: 0,
|
||||
});
|
||||
|
||||
worker.register('noop', async () => {});
|
||||
await queue.add('noop', {});
|
||||
|
||||
const startPromise = worker.start();
|
||||
await new Promise(r => setTimeout(r, 300));
|
||||
worker.stop();
|
||||
await startPromise;
|
||||
|
||||
const completed = await queue.getJobs({ status: 'completed' });
|
||||
expect(completed.length).toBeGreaterThanOrEqual(1);
|
||||
}, 10_000);
|
||||
});
|
||||
|
||||
// --- v0.22.14: Self-health-check behavior tests (D7) ---
|
||||
// These tests use a Proxy around the real engine so executeRaw can be
|
||||
// intercepted by SQL pattern. SELECT 1 = liveness probe; the count(*) query
|
||||
// = stall detection. Anything else passes through to the underlying engine.
|
||||
|
||||
interface ProbeOverrides {
|
||||
/** When set, executeRaw('SELECT 1') uses this function instead of pass-through.
|
||||
* Returning a thrown error simulates DB death; returning [{}] simulates success. */
|
||||
selectOne?: () => Promise<unknown>;
|
||||
/** When set, executeRaw of the stall-detection count(*) query returns this. */
|
||||
countWaiting?: (handlers: string[]) => number;
|
||||
/** Captures the last SQL string that matched the stall-count regex. Tests
|
||||
* use this to assert the production SQL still contains `name = ANY(...)`
|
||||
* so a future refactor that drops the predicate is caught. */
|
||||
capturedStallSql?: { sql: string | null };
|
||||
}
|
||||
|
||||
function makeProbeEngine(overrides: ProbeOverrides) {
|
||||
return new Proxy(engine, {
|
||||
get(target, prop, receiver) {
|
||||
if (prop === 'executeRaw') {
|
||||
return async (sql: string, params?: unknown[]): Promise<unknown[]> => {
|
||||
if (overrides.selectOne && /^\s*SELECT\s+1\s*$/i.test(sql)) {
|
||||
const r = await overrides.selectOne();
|
||||
return Array.isArray(r) ? r : [r];
|
||||
}
|
||||
if (overrides.countWaiting && /count\(\*\).*minion_jobs.*WHERE\s+status\s*=\s*'waiting'/is.test(sql)) {
|
||||
if (overrides.capturedStallSql) overrides.capturedStallSql.sql = sql;
|
||||
const handlers = (params?.[1] as string[]) ?? [];
|
||||
return [{ cnt: String(overrides.countWaiting(handlers)) }];
|
||||
}
|
||||
// Pass through to real engine for anything else (claim queries etc.)
|
||||
return (target as unknown as { executeRaw: (s: string, p?: unknown[]) => Promise<unknown[]> })
|
||||
.executeRaw(sql, params);
|
||||
};
|
||||
}
|
||||
return Reflect.get(target, prop, receiver);
|
||||
},
|
||||
}) as unknown as PGLiteEngine;
|
||||
}
|
||||
|
||||
describe('MinionWorker: self-health-check behavior (v0.22.14)', () => {
|
||||
test('emits unhealthy{db_dead} after dbFailExitAfter consecutive DB probe failures', async () => {
|
||||
delete process.env.GBRAIN_SUPERVISED;
|
||||
|
||||
let probeCount = 0;
|
||||
const probeEngine = makeProbeEngine({
|
||||
selectOne: async () => {
|
||||
probeCount++;
|
||||
throw new Error('connection terminated unexpectedly');
|
||||
},
|
||||
});
|
||||
|
||||
const worker = new MinionWorker(probeEngine, {
|
||||
queue: 'default',
|
||||
concurrency: 1,
|
||||
healthCheckInterval: 30,
|
||||
dbFailExitAfter: 3,
|
||||
pollInterval: 50,
|
||||
stalledInterval: 10_000,
|
||||
maxRssMb: 0,
|
||||
});
|
||||
|
||||
worker.register('noop', async () => {});
|
||||
|
||||
const events: Array<{ reason: string }> = [];
|
||||
worker.on('unhealthy', (info) => { events.push(info); });
|
||||
|
||||
const startPromise = worker.start();
|
||||
// 3 ticks at 30ms = 90ms; give extra slack.
|
||||
await new Promise(r => setTimeout(r, 250));
|
||||
worker.stop();
|
||||
await startPromise;
|
||||
|
||||
expect(probeCount).toBeGreaterThanOrEqual(3);
|
||||
expect(events.length).toBeGreaterThanOrEqual(1);
|
||||
expect(events[0].reason).toBe('db_dead');
|
||||
}, 10_000);
|
||||
|
||||
test('DB recovery resets the failure counter (no exit after intermittent failures)', async () => {
|
||||
delete process.env.GBRAIN_SUPERVISED;
|
||||
|
||||
let probeCount = 0;
|
||||
// Pattern: fail, fail, succeed (resets), fail, fail, then permanently succeed.
|
||||
// No 3 consecutive failures, so dbFailExitAfter=3 must NOT trip.
|
||||
const probeEngine = makeProbeEngine({
|
||||
selectOne: async () => {
|
||||
const idx = probeCount++;
|
||||
if (idx === 0 || idx === 1 || idx === 3 || idx === 4) {
|
||||
throw new Error('transient blip');
|
||||
}
|
||||
return [{ ok: 1 }];
|
||||
},
|
||||
});
|
||||
|
||||
const worker = new MinionWorker(probeEngine, {
|
||||
queue: 'default',
|
||||
concurrency: 1,
|
||||
healthCheckInterval: 30,
|
||||
dbFailExitAfter: 3,
|
||||
pollInterval: 50,
|
||||
stalledInterval: 10_000,
|
||||
maxRssMb: 0,
|
||||
});
|
||||
|
||||
worker.register('noop', async () => {});
|
||||
|
||||
const events: Array<{ reason: string }> = [];
|
||||
worker.on('unhealthy', (info) => { events.push(info); });
|
||||
|
||||
const startPromise = worker.start();
|
||||
await new Promise(r => setTimeout(r, 250));
|
||||
worker.stop();
|
||||
await startPromise;
|
||||
|
||||
// Counter should never have hit 3 consecutive — success at index 2 resets it.
|
||||
const dbDeadEvents = events.filter(e => e.reason === 'db_dead');
|
||||
expect(dbDeadEvents.length).toBe(0);
|
||||
}, 10_000);
|
||||
|
||||
test('emits unhealthy{stalled} after stallExitAfterMs of continuous idle with waiting jobs', async () => {
|
||||
delete process.env.GBRAIN_SUPERVISED;
|
||||
|
||||
const probeEngine = makeProbeEngine({
|
||||
selectOne: async () => [{ ok: 1 }],
|
||||
countWaiting: () => 5, // pretend 5 jobs are waiting for our handler names
|
||||
});
|
||||
|
||||
const worker = new MinionWorker(probeEngine, {
|
||||
queue: 'default',
|
||||
concurrency: 1,
|
||||
healthCheckInterval: 30,
|
||||
stallWarnAfterMs: 50,
|
||||
stallExitAfterMs: 100,
|
||||
pollInterval: 50,
|
||||
stalledInterval: 10_000,
|
||||
maxRssMb: 0,
|
||||
});
|
||||
|
||||
worker.register('noop', async () => {});
|
||||
// Don't queue any real jobs — claim returns null, inFlight stays 0,
|
||||
// jobsCompleted stays 0, idle clock advances.
|
||||
|
||||
const events: Array<{ reason: string; waitingCount?: number }> = [];
|
||||
worker.on('unhealthy', (info) => { events.push(info); });
|
||||
|
||||
const startPromise = worker.start();
|
||||
// Both thresholds measured from lastCompletionTime (corrected per codex r2):
|
||||
// - tick @ +30ms: idle=30ms, < stallWarnAfterMs(50), no warn
|
||||
// - tick @ +60ms: idle=60ms, > 50, warn fires (stallWarningSince set)
|
||||
// - tick @ +90ms: idle=90ms, < stallExitAfterMs(100), no exit yet
|
||||
// - tick @ +120ms: idle=120ms, > 100 → exit fires (unhealthy event)
|
||||
// Wait 350ms which leaves comfortable slack for setTimeout drift.
|
||||
await new Promise(r => setTimeout(r, 350));
|
||||
worker.stop();
|
||||
await startPromise;
|
||||
|
||||
const stalledEvents = events.filter(e => e.reason === 'stalled');
|
||||
expect(stalledEvents.length).toBeGreaterThanOrEqual(1);
|
||||
expect(stalledEvents[0].waitingCount).toBe(5);
|
||||
// The idleMinutes payload should reflect total idle, not warn-since.
|
||||
// With idle ~120ms at exit time, idleMinutes rounds to 0 — that's
|
||||
// expected; the value is informative, not load-bearing.
|
||||
}, 10_000);
|
||||
|
||||
test('inFlight > 0 blocks stall detection (long-running legitimate job)', async () => {
|
||||
delete process.env.GBRAIN_SUPERVISED;
|
||||
|
||||
const probeEngine = makeProbeEngine({
|
||||
selectOne: async () => [{ ok: 1 }],
|
||||
countWaiting: () => 5,
|
||||
});
|
||||
|
||||
const worker = new MinionWorker(probeEngine, {
|
||||
queue: 'default',
|
||||
concurrency: 1,
|
||||
healthCheckInterval: 30,
|
||||
stallWarnAfterMs: 50,
|
||||
stallExitAfterMs: 100,
|
||||
pollInterval: 50,
|
||||
stalledInterval: 10_000,
|
||||
maxRssMb: 0,
|
||||
});
|
||||
|
||||
worker.register('noop', async () => {});
|
||||
|
||||
const events: Array<{ reason: string }> = [];
|
||||
worker.on('unhealthy', (info) => { events.push(info); });
|
||||
|
||||
// Inject a fake in-flight entry directly. This bypasses the claim path
|
||||
// (which goes through the proxy and complicates the cleanup race) and
|
||||
// tests exactly what we want: the stall check's `inFlight.size === 0`
|
||||
// gate when there's legitimate ongoing work.
|
||||
const fakeInFlight = (worker as unknown as {
|
||||
inFlight: Map<number, { lockTimer: NodeJS.Timeout; abort: AbortController; promise: Promise<void> }>
|
||||
}).inFlight;
|
||||
const fakeAbort = new AbortController();
|
||||
const fakePromise = new Promise<void>(() => { /* never resolves */ });
|
||||
const fakeTimer = setInterval(() => {}, 60_000); // dummy lock timer
|
||||
fakeInFlight.set(99999, { lockTimer: fakeTimer, abort: fakeAbort, promise: fakePromise });
|
||||
|
||||
const startPromise = worker.start();
|
||||
await new Promise(r => setTimeout(r, 350));
|
||||
// Remove our fake entry before stop so the worker doesn't wait 30s for it.
|
||||
clearInterval(fakeTimer);
|
||||
fakeInFlight.delete(99999);
|
||||
worker.stop();
|
||||
await startPromise;
|
||||
|
||||
// No stall event should fire — inFlight.size > 0 gates the stall check.
|
||||
const stalledEvents = events.filter(e => e.reason === 'stalled');
|
||||
expect(stalledEvents.length).toBe(0);
|
||||
}, 10_000);
|
||||
|
||||
test('regression (D1): waiting jobs of unregistered handler names do NOT trigger stall exit', async () => {
|
||||
delete process.env.GBRAIN_SUPERVISED;
|
||||
|
||||
// The count(*) query is filtered by registered handler names. If handlers=['noop']
|
||||
// and the queue has 5 'widget-fn' jobs, the SQL `name = ANY($2)` filter returns 0.
|
||||
// The probe engine simulates this by checking handlers before returning a count;
|
||||
// we ALSO capture the SQL to assert the predicate text is actually present (so a
|
||||
// future refactor that silently drops `AND name = ANY(...)` is caught).
|
||||
const capturedStallSql = { sql: null as string | null };
|
||||
const probeEngine = makeProbeEngine({
|
||||
selectOne: async () => [{ ok: 1 }],
|
||||
countWaiting: (handlers) => handlers.includes('widget-fn') ? 5 : 0,
|
||||
capturedStallSql,
|
||||
});
|
||||
|
||||
const worker = new MinionWorker(probeEngine, {
|
||||
queue: 'default',
|
||||
concurrency: 1,
|
||||
healthCheckInterval: 50,
|
||||
stallWarnAfterMs: 100,
|
||||
stallExitAfterMs: 200,
|
||||
pollInterval: 50,
|
||||
stalledInterval: 10_000,
|
||||
maxRssMb: 0,
|
||||
});
|
||||
|
||||
// Register 'noop' but pretend the queue is full of 'widget-fn' (unhandled).
|
||||
worker.register('noop', async () => {});
|
||||
|
||||
const events: Array<{ reason: string }> = [];
|
||||
worker.on('unhealthy', (info) => { events.push(info); });
|
||||
|
||||
const startPromise = worker.start();
|
||||
// Window > stallExitAfterMs; if D1 fix wasn't applied, stall would fire.
|
||||
await new Promise(r => setTimeout(r, 500));
|
||||
worker.stop();
|
||||
await startPromise;
|
||||
|
||||
// No stall event — the count for 'noop' handlers is 0, so worker is correctly idle.
|
||||
const stalledEvents = events.filter(e => e.reason === 'stalled');
|
||||
expect(stalledEvents.length).toBe(0);
|
||||
// SQL shape assertion: the production query MUST filter by handler names.
|
||||
// Without this assertion, a future change that drops the predicate would
|
||||
// pass the no-event check above (the handler array would be irrelevant
|
||||
// to the underlying DB but our probe just needs to return 0).
|
||||
expect(capturedStallSql.sql).not.toBeNull();
|
||||
expect(capturedStallSql.sql).toMatch(/name\s*=\s*ANY/i);
|
||||
}, 10_000);
|
||||
|
||||
test('regression (R3): constructor throws when stallExitAfterMs <= stallWarnAfterMs', () => {
|
||||
// The contract on MinionWorkerOpts.stallExitAfterMs says "Must be >
|
||||
// stallWarnAfterMs". Without validation, an exit threshold equal to or
|
||||
// less than the warn threshold made the configured exit time a lie
|
||||
// (warn fires first, exit can't preempt). The constructor now throws
|
||||
// loudly so misconfigurations fail at startup, not at idle-time.
|
||||
expect(() => new MinionWorker(engine, {
|
||||
stallWarnAfterMs: 200,
|
||||
stallExitAfterMs: 100, // less than warn — invalid
|
||||
})).toThrow(/stallExitAfterMs.*must be > stallWarnAfterMs/i);
|
||||
|
||||
expect(() => new MinionWorker(engine, {
|
||||
stallWarnAfterMs: 100,
|
||||
stallExitAfterMs: 100, // equal to warn — also invalid (must be strictly >)
|
||||
})).toThrow(/stallExitAfterMs.*must be > stallWarnAfterMs/i);
|
||||
|
||||
// Sanity: defaults (5min warn / 10min exit) construct without throwing.
|
||||
expect(() => new MinionWorker(engine, {})).not.toThrow();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -328,6 +328,79 @@ describe('MinionSupervisor', () => {
|
||||
}, 15_000);
|
||||
});
|
||||
|
||||
describe('integration: GBRAIN_SUPERVISED env var (v0.22.14)', () => {
|
||||
it('sets GBRAIN_SUPERVISED=1 on spawned worker child', async () => {
|
||||
const outFile = join(tmpdir(), `gbrain-sup-supervised-${process.pid}-${Date.now()}.txt`);
|
||||
try { unlinkSync(outFile); } catch { /* may not exist */ }
|
||||
|
||||
const h = makeHarness('supervised-env', `printf '%s\n' "\${GBRAIN_SUPERVISED-UNSET}" > "$OUT_FILE" ; exit 0`);
|
||||
|
||||
try {
|
||||
const sup = spawnSupervisor(h, {
|
||||
OUT_FILE: outFile,
|
||||
SUP_MAX_CRASHES: '1',
|
||||
});
|
||||
|
||||
await sup.exited;
|
||||
|
||||
expect(existsSync(outFile)).toBe(true);
|
||||
const childSawEnv = readFileSync(outFile, 'utf8').trim();
|
||||
expect(childSawEnv).toBe('1');
|
||||
} finally {
|
||||
try { unlinkSync(outFile); } catch { /* noop */ }
|
||||
h.cleanup();
|
||||
}
|
||||
}, 15_000);
|
||||
});
|
||||
|
||||
describe('regression (R3): healthInterval=0 disables timer (v0.22.14)', () => {
|
||||
// Pre-fix: supervisor unconditionally called setInterval(callback, 0),
|
||||
// which schedules a tight loop on the next event-loop tick. The
|
||||
// operator-facing CLI claim "Use 0 to disable" was a lie — passing 0
|
||||
// produced a DB-probe loop that hammered Postgres.
|
||||
//
|
||||
// Post-fix: setInterval is gated on healthInterval > 0. With 0, the
|
||||
// supervisor runs its supervise loop normally with the health timer
|
||||
// entirely absent.
|
||||
//
|
||||
// Assertion strategy: spawn the supervisor with SUP_HEALTH_INTERVAL_MS=0,
|
||||
// a fast worker that exits cleanly, and SUP_MAX_CRASHES=1. A working fix
|
||||
// should produce a single worker spawn → exit → supervisor shutdown
|
||||
// sequence. If the tight-loop bug returned, the supervisor would still
|
||||
// exit (max-crashes path) but the audit trail would show the tell-tale
|
||||
// signature of an extremely high health-check call rate during the brief
|
||||
// window before max-crashes fires. We assert the basic completion path
|
||||
// and let CI's wall-clock detect any pathological CPU spike.
|
||||
it('completes a normal supervise lifecycle with healthInterval=0', async () => {
|
||||
const h = makeHarness('health-interval-zero', 'exit 0');
|
||||
|
||||
try {
|
||||
const sup = spawnSupervisor(h, {
|
||||
SUP_HEALTH_INTERVAL_MS: '0',
|
||||
SUP_MAX_CRASHES: '1',
|
||||
});
|
||||
|
||||
const start = Date.now();
|
||||
const { code } = await sup.exited;
|
||||
const elapsedMs = Date.now() - start;
|
||||
|
||||
// Clean exit (max-crashes path returns 1; this is fine — we just
|
||||
// want to confirm the supervisor reached its terminal state without
|
||||
// hanging or runaway looping).
|
||||
expect(code).toBe(1);
|
||||
|
||||
// Sanity: a tight loop on setInterval(0) plus the spawn-respawn
|
||||
// loop would still terminate at max-crashes, but it would be
|
||||
// measurably slower than a clean run because the event loop is
|
||||
// saturated with health-check callbacks. Cap the upper bound at
|
||||
// 10s — clean runs typically finish in 1–2s.
|
||||
expect(elapsedMs).toBeLessThan(10_000);
|
||||
} finally {
|
||||
h.cleanup();
|
||||
}
|
||||
}, 15_000);
|
||||
});
|
||||
|
||||
describe('integration: --max-rss spawn args (v0.21)', () => {
|
||||
it('passes --max-rss 2048 to spawned worker by default', async () => {
|
||||
const outFile = join(tmpdir(), `gbrain-sup-maxrss-${process.pid}-${Date.now()}.txt`);
|
||||
|
||||
@@ -0,0 +1,123 @@
|
||||
/**
|
||||
* Unit tests for the shared concurrency-policy helper. Covers:
|
||||
*
|
||||
* - Q5: autoConcurrency() returns correct counts for PGLite, explicit
|
||||
* override, auto path above/below threshold.
|
||||
* - Q1: shouldRunParallel() respects explicit opt-in even on small diffs.
|
||||
* - Q2/T3: parseWorkers() throws on bad CLI input (0, -3, "foo", "1.5").
|
||||
*
|
||||
* These exist because the prior policy was duplicated across three call sites
|
||||
* (performSync, performFullSync, jobs handler) with subtle differences.
|
||||
* Centralized helper + tests prevents the next drift.
|
||||
*/
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import {
|
||||
autoConcurrency,
|
||||
shouldRunParallel,
|
||||
parseWorkers,
|
||||
AUTO_CONCURRENCY_FILE_THRESHOLD,
|
||||
PARALLEL_FILE_FLOOR,
|
||||
DEFAULT_PARALLEL_WORKERS,
|
||||
} from '../src/core/sync-concurrency.ts';
|
||||
import type { BrainEngine } from '../src/core/engine.ts';
|
||||
|
||||
// Minimal engine stub — autoConcurrency only reads .kind.
|
||||
function engineOfKind(kind: 'postgres' | 'pglite'): BrainEngine {
|
||||
return { kind } as unknown as BrainEngine;
|
||||
}
|
||||
|
||||
describe('autoConcurrency', () => {
|
||||
test('PGLite always serial (single connection)', () => {
|
||||
expect(autoConcurrency(engineOfKind('pglite'), 1000)).toBe(1);
|
||||
expect(autoConcurrency(engineOfKind('pglite'), 1000, 8)).toBe(1);
|
||||
expect(autoConcurrency(engineOfKind('pglite'), 0)).toBe(1);
|
||||
});
|
||||
|
||||
test('Postgres + explicit override wins', () => {
|
||||
expect(autoConcurrency(engineOfKind('postgres'), 5, 4)).toBe(4);
|
||||
expect(autoConcurrency(engineOfKind('postgres'), 5, 1)).toBe(1);
|
||||
expect(autoConcurrency(engineOfKind('postgres'), 5, 16)).toBe(16);
|
||||
});
|
||||
|
||||
test('Postgres explicit 0 clamped to 1 (paranoia — parseWorkers should reject first)', () => {
|
||||
expect(autoConcurrency(engineOfKind('postgres'), 100, 0)).toBe(1);
|
||||
expect(autoConcurrency(engineOfKind('postgres'), 100, -5)).toBe(1);
|
||||
});
|
||||
|
||||
test('Postgres + auto path: under threshold serial', () => {
|
||||
expect(autoConcurrency(engineOfKind('postgres'), 50)).toBe(1);
|
||||
expect(autoConcurrency(engineOfKind('postgres'), AUTO_CONCURRENCY_FILE_THRESHOLD)).toBe(1);
|
||||
});
|
||||
|
||||
test('Postgres + auto path: above threshold parallel', () => {
|
||||
expect(autoConcurrency(engineOfKind('postgres'), AUTO_CONCURRENCY_FILE_THRESHOLD + 1)).toBe(DEFAULT_PARALLEL_WORKERS);
|
||||
expect(autoConcurrency(engineOfKind('postgres'), 7000)).toBe(DEFAULT_PARALLEL_WORKERS);
|
||||
});
|
||||
|
||||
test('full-sync large marker fires parallel for Postgres', () => {
|
||||
expect(autoConcurrency(engineOfKind('postgres'), Number.MAX_SAFE_INTEGER)).toBe(DEFAULT_PARALLEL_WORKERS);
|
||||
});
|
||||
});
|
||||
|
||||
describe('shouldRunParallel', () => {
|
||||
test('serial when worker count <= 1', () => {
|
||||
expect(shouldRunParallel(1, 1000, false)).toBe(false);
|
||||
expect(shouldRunParallel(1, 1000, true)).toBe(false);
|
||||
expect(shouldRunParallel(0, 1000, true)).toBe(false);
|
||||
});
|
||||
|
||||
test('Q1: explicit opt-in beats the file-count floor', () => {
|
||||
// User typed --workers 4 with 30 files. Prior behavior: silently serial.
|
||||
// New behavior: respect the user.
|
||||
expect(shouldRunParallel(4, 30, /*explicit*/ true)).toBe(true);
|
||||
expect(shouldRunParallel(2, 1, true)).toBe(true);
|
||||
});
|
||||
|
||||
test('auto path honors PARALLEL_FILE_FLOOR', () => {
|
||||
// No explicit opt-in: use the floor as the gate.
|
||||
expect(shouldRunParallel(4, PARALLEL_FILE_FLOOR, false)).toBe(false);
|
||||
expect(shouldRunParallel(4, PARALLEL_FILE_FLOOR + 1, false)).toBe(true);
|
||||
expect(shouldRunParallel(4, 0, false)).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('parseWorkers (Q2)', () => {
|
||||
test('undefined input → undefined output', () => {
|
||||
expect(parseWorkers(undefined)).toBeUndefined();
|
||||
});
|
||||
|
||||
test('positive integer accepted', () => {
|
||||
expect(parseWorkers('1')).toBe(1);
|
||||
expect(parseWorkers('4')).toBe(4);
|
||||
expect(parseWorkers('128')).toBe(128);
|
||||
});
|
||||
|
||||
test('zero rejected (the original silent footgun)', () => {
|
||||
expect(() => parseWorkers('0')).toThrow(/positive integer/);
|
||||
});
|
||||
|
||||
test('negative rejected', () => {
|
||||
expect(() => parseWorkers('-3')).toThrow(/positive integer/);
|
||||
expect(() => parseWorkers('-1')).toThrow(/positive integer/);
|
||||
});
|
||||
|
||||
test('non-numeric rejected', () => {
|
||||
expect(() => parseWorkers('foo')).toThrow(/positive integer/);
|
||||
expect(() => parseWorkers('')).toThrow(/positive integer/);
|
||||
});
|
||||
|
||||
test('non-integer (decimal) rejected', () => {
|
||||
// parseInt("1.5") returns 1, but "1.5" !== "1" so we reject.
|
||||
expect(() => parseWorkers('1.5')).toThrow(/positive integer/);
|
||||
});
|
||||
|
||||
test('integer with trailing chars rejected', () => {
|
||||
// parseInt("4abc") returns 4 silently; we want loud failure.
|
||||
expect(() => parseWorkers('4abc')).toThrow(/positive integer/);
|
||||
});
|
||||
|
||||
test('whitespace tolerated (since CLI parsers may pass the literal)', () => {
|
||||
// " 4 " trims to "4" which equals String(4). Accepted.
|
||||
expect(parseWorkers(' 4 ')).toBe(4);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,257 @@
|
||||
/**
|
||||
* Parallel-sync regression tests (PGLite, in-memory).
|
||||
*
|
||||
* T1 — sync.last_commit failure-gate under concurrency=4 request.
|
||||
* T4 — PGLite + concurrency=4 stays serial (no crash, no PostgresEngine
|
||||
* construction). Tightens the engine.kind guard introduced in
|
||||
* v0.22.13 (PR #490 A1).
|
||||
* CODEX-3 — head-drift gate: when git HEAD moves between performSync's
|
||||
* capture and its post-import re-check, last_commit must NOT advance.
|
||||
*
|
||||
* PGLite forces concurrency=1 internally regardless of the requested value,
|
||||
* which is the *whole point* of T4 — but the bookmark-gate logic
|
||||
* (failedFiles → don't advance) is engine-agnostic, so PGLite is fine for
|
||||
* the T1 + CODEX-3 contracts. A separate Postgres E2E covers worker-engine
|
||||
* construction directly.
|
||||
*/
|
||||
import { describe, expect, test, beforeEach, afterEach } from 'bun:test';
|
||||
import { mkdtempSync, rmSync, writeFileSync, mkdirSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import { execSync } from 'child_process';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
|
||||
function git(repo: string, ...args: string[]): string {
|
||||
return execSync(`git ${args.join(' ')}`, { cwd: repo, encoding: 'utf-8' }).trim();
|
||||
}
|
||||
|
||||
function seedRepoWithMarkdown(repoPath: string, fileCount: number): string {
|
||||
execSync('git init', { cwd: repoPath, stdio: 'pipe' });
|
||||
execSync('git config user.email "test@test.com"', { cwd: repoPath, stdio: 'pipe' });
|
||||
execSync('git config user.name "Test"', { cwd: repoPath, stdio: 'pipe' });
|
||||
mkdirSync(join(repoPath, 'people'), { recursive: true });
|
||||
for (let i = 0; i < fileCount; i++) {
|
||||
writeFileSync(join(repoPath, `people/p${i}.md`), [
|
||||
'---',
|
||||
'type: person',
|
||||
`title: Person ${i}`,
|
||||
'---',
|
||||
'',
|
||||
`This is person ${i}.`,
|
||||
].join('\n'));
|
||||
}
|
||||
execSync('git add -A && git commit -m "initial"', { cwd: repoPath, stdio: 'pipe' });
|
||||
return git(repoPath, 'rev-parse', 'HEAD');
|
||||
}
|
||||
|
||||
describe('sync-parallel: PGLite + concurrency=4 (T4)', () => {
|
||||
let engine: PGLiteEngine;
|
||||
let repoPath: string;
|
||||
|
||||
beforeEach(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
repoPath = mkdtempSync(join(tmpdir(), 'gbrain-sync-par-'));
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
await engine.disconnect();
|
||||
if (repoPath) rmSync(repoPath, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
test('PGLite + concurrency=4 + 60 files: imports all without crashing', async () => {
|
||||
seedRepoWithMarkdown(repoPath, 60);
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
const result = await performSync(engine, {
|
||||
repoPath,
|
||||
noPull: true,
|
||||
noEmbed: true,
|
||||
concurrency: 4,
|
||||
});
|
||||
// First sync routes through performFullSync, returning 'first_sync'.
|
||||
expect(result.status).toBe('first_sync');
|
||||
// PGLite stayed single-connection; if the parallel branch had tried to
|
||||
// construct PostgresEngine without database_url, this test would crash.
|
||||
});
|
||||
|
||||
test('PGLite + explicit concurrency=4 + 30 files (below floor): still safe', async () => {
|
||||
// Q1 path: explicit opt-in beats the >50 floor. PGLite forces serial
|
||||
// anyway (engine.kind), so the test is that nothing crashes and the
|
||||
// sync advances correctly.
|
||||
seedRepoWithMarkdown(repoPath, 30);
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
const result = await performSync(engine, {
|
||||
repoPath,
|
||||
noPull: true,
|
||||
noEmbed: true,
|
||||
concurrency: 4,
|
||||
});
|
||||
expect(result.status).toBe('first_sync');
|
||||
});
|
||||
});
|
||||
|
||||
describe('sync-parallel: bookmark gate under concurrency request (T1)', () => {
|
||||
let engine: PGLiteEngine;
|
||||
let repoPath: string;
|
||||
|
||||
beforeEach(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
repoPath = mkdtempSync(join(tmpdir(), 'gbrain-sync-gate-'));
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
await engine.disconnect();
|
||||
if (repoPath) rmSync(repoPath, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
test('clean parallel sync advances last_commit', async () => {
|
||||
const initialHead = seedRepoWithMarkdown(repoPath, 5);
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
await performSync(engine, {
|
||||
repoPath,
|
||||
noPull: true,
|
||||
noEmbed: true,
|
||||
concurrency: 4,
|
||||
});
|
||||
const lastCommit = await engine.getConfig('sync.last_commit');
|
||||
expect(lastCommit).toBe(initialHead);
|
||||
});
|
||||
|
||||
test('failure-injection blocks last_commit advance', async () => {
|
||||
// First sync: clean state.
|
||||
const firstHead = seedRepoWithMarkdown(repoPath, 5);
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
await performSync(engine, {
|
||||
repoPath, noPull: true, noEmbed: true,
|
||||
});
|
||||
const lastAfterFirst = await engine.getConfig('sync.last_commit');
|
||||
expect(lastAfterFirst).toBe(firstHead);
|
||||
|
||||
// Now add a malformed file (broken YAML frontmatter — closing --- missing
|
||||
// means the parser hits a real failure that importFile reports).
|
||||
writeFileSync(join(repoPath, 'people/broken.md'), [
|
||||
'---',
|
||||
'type: person',
|
||||
'title: Broken', // intentionally no closing ---
|
||||
'this line is body but parser thinks it is YAML',
|
||||
].join('\n'));
|
||||
execSync('git add -A && git commit -m "add broken"', { cwd: repoPath, stdio: 'pipe' });
|
||||
const secondHead = git(repoPath, 'rev-parse', 'HEAD');
|
||||
expect(secondHead).not.toBe(firstHead);
|
||||
|
||||
// Second sync: should record failure and NOT advance the bookmark.
|
||||
const result = await performSync(engine, {
|
||||
repoPath, noPull: true, noEmbed: true, concurrency: 4,
|
||||
});
|
||||
|
||||
// Only fail the test when the parser actually rejected the broken file.
|
||||
// Some YAML parsers are permissive; if so this test exercises the
|
||||
// happy path AND the assertion below (lastCommit advanced) holds.
|
||||
if (result.status === 'blocked_by_failures') {
|
||||
const lastAfterBroken = await engine.getConfig('sync.last_commit');
|
||||
expect(lastAfterBroken).toBe(firstHead); // unchanged — gate held
|
||||
expect(result.failedFiles ?? 0).toBeGreaterThan(0);
|
||||
} else {
|
||||
// If the parser was permissive, at least confirm the bookmark moved.
|
||||
const lastAfterBroken = await engine.getConfig('sync.last_commit');
|
||||
expect(lastAfterBroken).toBe(secondHead);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('sync-parallel: head-drift gate (CODEX-3)', () => {
|
||||
let engine: PGLiteEngine;
|
||||
let repoPath: string;
|
||||
|
||||
beforeEach(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
repoPath = mkdtempSync(join(tmpdir(), 'gbrain-sync-drift-'));
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
await engine.disconnect();
|
||||
if (repoPath) rmSync(repoPath, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
test('static-HEAD sync advances last_commit (control)', async () => {
|
||||
const head = seedRepoWithMarkdown(repoPath, 3);
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
await performSync(engine, { repoPath, noPull: true, noEmbed: true });
|
||||
expect(await engine.getConfig('sync.last_commit')).toBe(head);
|
||||
});
|
||||
|
||||
test('vanished-mid-sync file produces a failedFiles entry', async () => {
|
||||
// First sync: clean state for incremental.
|
||||
seedRepoWithMarkdown(repoPath, 3);
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
await performSync(engine, { repoPath, noPull: true, noEmbed: true });
|
||||
|
||||
// Add a file, commit, then delete the file from disk WITHOUT amending the
|
||||
// commit — diff says it exists at HEAD, but the file is gone. This is the
|
||||
// "checkout/race deleted my file mid-sync" simulation.
|
||||
writeFileSync(join(repoPath, 'people/will-vanish.md'), [
|
||||
'---', 'type: person', 'title: Vanish', '---', '', 'body',
|
||||
].join('\n'));
|
||||
execSync('git add -A && git commit -m "add vanish"', { cwd: repoPath, stdio: 'pipe' });
|
||||
rmSync(join(repoPath, 'people/will-vanish.md'));
|
||||
|
||||
const result = await performSync(engine, {
|
||||
repoPath, noPull: true, noEmbed: true,
|
||||
});
|
||||
// Per CODEX-3 (v0.22.13): vanished files now go into failedFiles
|
||||
// (prior behavior was a benign skip, which let last_commit advance).
|
||||
expect(result.status).toBe('blocked_by_failures');
|
||||
expect(result.failedFiles ?? 0).toBeGreaterThan(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('sync-parallel: writer lock prevents reentrance (CODEX-2)', () => {
|
||||
let engine: PGLiteEngine;
|
||||
let repoPath: string;
|
||||
|
||||
beforeEach(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
repoPath = mkdtempSync(join(tmpdir(), 'gbrain-sync-lock-'));
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
await engine.disconnect();
|
||||
if (repoPath) rmSync(repoPath, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
test('two parallel performSync calls in same process: second waits or fails fast', async () => {
|
||||
seedRepoWithMarkdown(repoPath, 5);
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
|
||||
// Same-process concurrent calls: PGLite serializes engine ops via its
|
||||
// exclusive transaction mutex, but the writer-lock is the right barrier.
|
||||
// We verify that one call completes (the lock holder) and any concurrent
|
||||
// call either completes after (lock released) or surfaces the
|
||||
// "Another sync is in progress" error.
|
||||
const promise1 = performSync(engine, { repoPath, noPull: true, noEmbed: true });
|
||||
|
||||
let secondError: unknown = null;
|
||||
try {
|
||||
// Tiny delay so promise1 captures the lock first.
|
||||
await new Promise((r) => setTimeout(r, 10));
|
||||
await performSync(engine, { repoPath, noPull: true, noEmbed: true });
|
||||
} catch (e) {
|
||||
secondError = e;
|
||||
}
|
||||
await promise1;
|
||||
|
||||
// Either: (a) second call completed after first released, both succeeded
|
||||
// OR (b) second call hit the lock-busy error path. Either is correct.
|
||||
if (secondError) {
|
||||
const msg = secondError instanceof Error ? secondError.message : String(secondError);
|
||||
expect(msg).toMatch(/Another sync is in progress|lock|gbrain-sync/i);
|
||||
}
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user