mirror of
https://github.com/garrytan/gbrain.git
synced 2026-08-15 01:12:20 +00:00
Compare commits
35
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Garry TanandClaude Opus 4.7
|
9fc81beac2 | ||
|
Garry TanandClaude Opus 4.7
|
dbb00495df | ||
|
Garry TanandClaude Opus 4.7
|
0a46d98954 | ||
|
Garry TanandClaude Opus 4.7
|
cd0432a19e | ||
|
Garry TanandClaude Opus 4.7
|
9870646f14 | ||
|
Garry TanandClaude Opus 4.7
|
b78ff8b616 | ||
|
Garry TanandClaude Opus 4.7
|
c4286a2f02 | ||
|
Garry TanandClaude Opus 4.7
|
5640e416e6 | ||
|
Garry TanandClaude Opus 4.7
|
54a18ea9b3 | ||
|
Garry TanandClaude Opus 4.7
|
05429e3196 | ||
|
Garry TanandClaude Opus 4.7
|
4009477b87 | ||
|
Garry TanandClaude Opus 4.7
|
a2b79a9dbc | ||
|
Garry TanandClaude Opus 4.7
|
c773a0fc1d | ||
|
Garry TanandClaude Opus 4.7
|
1323ad538f | ||
|
Garry TanandClaude Opus 4.7
|
9257d5abd2 | ||
|
Garry TanandClaude Opus 4.7
|
9f24d42c61 | ||
|
Garry Tan
|
3aee858870 | ||
|
Garry Tan
|
a046245fe8 | ||
|
Garry Tan
|
42efae6e2d | ||
|
Garry Tan
|
003ce54ec2 | ||
|
Garry Tan
|
8b60c81f45 | ||
|
Garry Tan
|
3b0accaa03 | ||
|
Garry Tan
|
af85191fdc | ||
|
Garry Tan
|
130cefbc8a | ||
|
Garry Tan
|
e6fd4edf2c | ||
|
Garry Tan
|
84f42e0593 | ||
|
Garry Tan
|
6819e21339 | ||
|
Garry Tan
|
65bffb9751 | ||
|
Garry Tan
|
70e179121f | ||
|
Garry Tan
|
f1d7307d2c | ||
|
Garry Tan
|
b9a273b408 | ||
|
Garry Tan
|
d7b7041416 | ||
|
Garry Tan
|
52bab8f2f0 | ||
|
Garry Tan
|
4ba817114f | ||
|
Garry Tan
|
9cbfe6805d |
@@ -21,22 +21,11 @@ jobs:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
test:
|
||||
# ubuntu-latest is free 2-core/7GB. Larger runners (16-cores, etc.) require
|
||||
# a provisioned runner pool in repo settings. Falling back to default keeps
|
||||
# the matrix shard speedup (~5-6x via parallelism) at zero cost.
|
||||
runs-on: ubuntu-latest
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
shard: [1, 2, 3, 4]
|
||||
steps:
|
||||
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
|
||||
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
|
||||
with:
|
||||
bun-version: latest
|
||||
- run: bun install
|
||||
- name: Pre-test gates (shard 1 only — they're not test files)
|
||||
if: matrix.shard == 1
|
||||
run: scripts/check-jsonb-pattern.sh && scripts/check-progress-to-stdout.sh && scripts/check-wasm-embedded.sh && bun run typecheck
|
||||
- name: Run test shard ${{ matrix.shard }}/4
|
||||
run: scripts/test-shard.sh ${{ matrix.shard }} 4
|
||||
- run: bun run test
|
||||
|
||||
@@ -17,4 +17,3 @@ eval/data/world-v1/world.html
|
||||
|
||||
# BrainBench amara-life-v1 Opus cache (regenerate via eval:generate-amara-life)
|
||||
eval/data/amara-life-v1/_cache/
|
||||
.claude/
|
||||
|
||||
-953
@@ -2,959 +2,6 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.22.13] - 2026-04-28
|
||||
|
||||
**Sync got faster, and the bookmark stopped lying.**
|
||||
**Parallel imports, a real writer lock, and a head-drift gate that catches the worst race.**
|
||||
|
||||
The headline is `gbrain sync --workers N`: per-worker Postgres engines with an atomic queue index, same pattern as `gbrain import --workers N`. On a 7,000-page brain that used to take 25+ minutes, the import phase now runs across 4 workers by default. The reproducible benchmark in `test/e2e/sync-parallel.test.ts` shows `parallel(4)` finishing 1.3× faster than serial on a 120-file fixture against local Postgres (`serial=289ms parallel(4)=221ms`). The speedup grows on larger brains and slower-roundtrip databases (Supabase, remote PgBouncer) because the worker setup cost amortizes over more files. But the bigger story is that the sync writer is finally exclusive across processes, and the `last_commit` bookmark refuses to advance when git HEAD has drifted out from under us. The silent-skip-then-advance pathology has survived every prior sync hardening pass. It is dead now.
|
||||
|
||||
### What you can do now
|
||||
|
||||
- `gbrain sync --workers 4` (alias `--concurrency 4`) parallelizes the import phase. Each worker holds 2 connections, so total Postgres connections during the parallel phase is `workers * 2` plus your caller's pool. At the default of 4 workers and a 10-connection caller pool, that's up to 18 connections, well under PgBouncer's `max_client_conn` default of 100 but worth knowing on tight Supabase tiers.
|
||||
- **Auto-concurrency:** if you don't pass `--workers`, sync uses 4 workers when the diff exceeds 100 files. Smaller diffs stay serial. Explicit `--workers` always wins (even on a 30-file diff). PGLite forces serial regardless, since it's a single-connection engine.
|
||||
- **Full sync** routes through the same path. First syncs on large brains parallelize automatically.
|
||||
- **Minion `sync` jobs** also use the new `autoConcurrency()` policy. Behavior is now consistent between CLI sync, the Minion handler, and the autopilot cycle's sync phase. (`noEmbed` defaults to `true` in the jobs handler. Submit `gbrain embed --stale` as a separate job when needed, or rely on the autopilot cycle's embed phase.)
|
||||
- **`--workers` validation is loud now.** `--workers 0`, `--workers -3`, `--workers foo`, `--workers 1.5` all exit with an error message. The prior behavior silently fell through to auto-concurrency (4 workers), the opposite of what you typed.
|
||||
|
||||
### Correctness fixes you didn't have to ask for
|
||||
|
||||
- **Cross-process writer lock.** Two `gbrain sync` calls (manual + autopilot, two terminals, two Conductor workspaces) used to read the same `last_commit`, both write it, and let the last writer win. The new `gbrain-sync` row in `gbrain_cycle_locks` serializes the writer window. Same-process reentrance from the autopilot cycle handler was already covered by the broader `gbrain-cycle` lock; sync's lock is narrower and runs underneath it.
|
||||
- **Head-drift gate.** If `git checkout` or `git pull` runs in your worktree mid-sync (Conductor sibling workspace, ad-hoc terminal), the captured `headCommit` no longer matches HEAD when sync finishes. `last_commit` no longer advances in that case. The next sync re-walks the diff against the new HEAD instead of silently moving the bookmark past unimported work.
|
||||
- **Vanished files now block bookmark advance.** A file the diff said exists at `headCommit` but is gone from disk used to register as a benign skip. It now goes into `failedFiles` and gates `last_commit` the same way a parse failure does.
|
||||
- **Per-source bookmark for Minion `sync` jobs.** The job handler now resolves `sourceId` from the repo path (mirrors the autopilot cycle's `cycle.ts` fix from PR #475). On multi-source brains, this prevents the 30-min full-reimport-every-cycle behavior caused by reading the global `config.sync.last_commit` anchor when the per-source row would have been correct.
|
||||
- **Worker connection cleanup.** Worker engines now disconnect inside `try/finally`, even on partial connect failure or mid-import error. The prior `Promise.all(...disconnect)` ran outside any try/finally, so panic-path leaks never released the 8 worker connections.
|
||||
- **Engine detection unified.** Both PGLite-detection sites in sync.ts now use `engine.kind === 'pglite'` (the discriminator added in v0.13.1). The `engine.constructor.name === 'PGLiteEngine'` sniff is gone, since it broke under bundling and was inconsistent with the other site's `config.engine` string check.
|
||||
|
||||
### What this means for you
|
||||
|
||||
If you run autopilot on a 7,000-page Postgres brain, your sync cycle gets faster on day one with no flags. If you have ever felt the bookmark "skip past" work that didn't import, you'll stop seeing it. If you have multiple Conductor workspaces poking the same brain, you'll either wait politely on the writer lock or get a clear "another sync is in progress" error. None of this requires a config change.
|
||||
|
||||
## To take advantage of v0.22.13
|
||||
|
||||
`gbrain upgrade` should do this automatically. If you want to use the new flags right now:
|
||||
|
||||
1. **For a one-off speed win on a large brain:**
|
||||
```bash
|
||||
gbrain sync --workers 4
|
||||
```
|
||||
Or for incremental syncs that touch >100 files, just run `gbrain sync`. Auto-concurrency fires.
|
||||
|
||||
2. **For your autopilot cycle:** no action. The Minion `sync` handler picks up the new auto-concurrency policy automatically.
|
||||
|
||||
3. **Verify the writer lock is working:**
|
||||
```bash
|
||||
gbrain sync &
|
||||
gbrain sync # second call will say "Another sync is in progress" or wait
|
||||
```
|
||||
|
||||
4. **If sync ever errors with "Another sync is in progress" and stays stuck:** the lock is in `gbrain_cycle_locks` with id `gbrain-sync` and a 30-minute TTL. If a worker crashed without releasing, the next acquirer takes over once the TTL expires. To unstick faster:
|
||||
```sql
|
||||
DELETE FROM gbrain_cycle_locks WHERE id = 'gbrain-sync';
|
||||
```
|
||||
|
||||
5. **If anything looks wrong,** file an issue: https://github.com/garrytan/gbrain/issues with output of `gbrain doctor` and the contents of `~/.gbrain/upgrade-errors.jsonl` if it exists.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
- `src/commands/sync.ts`: `performSync` now wraps body in a `gbrain-sync` DB lock; `--workers` honored regardless of file count when explicit; head-drift gate after import phase; engine.kind detection; try/finally around worker engines; banner moved to stderr.
|
||||
- `src/commands/import.ts`: `engine.kind === 'pglite'` discriminator; try/finally around worker engines; shared `parseWorkers()` for `--workers` validation.
|
||||
- `src/commands/jobs.ts`: sync handler resolves `sourceId` via `sources.local_path` lookup; concurrency routed through `autoConcurrency()`; `noEmbed: true` default documented.
|
||||
- `src/core/sync-concurrency.ts` (new): `autoConcurrency()` + `parseWorkers()` + constants. One source of truth for the concurrency policy that previously lived in three call sites.
|
||||
- `src/core/db-lock.ts` (new): generic `tryAcquireDbLock(engine, lockId)` over the existing `gbrain_cycle_locks` table. Reused by performSync. cycle.ts continues to use its own ID `gbrain-cycle` so the two locks nest cleanly.
|
||||
- `test/sync-concurrency.test.ts` (new): 17 cases covering autoConcurrency thresholds, shouldRunParallel gates, parseWorkers validation.
|
||||
- `test/sync-parallel.test.ts` (new): PGLite-routed coverage of the bookmark gate under concurrency request, the head-drift gate, the writer-lock contract, and PGLite-stays-serial.
|
||||
- `test/e2e/sync-parallel.test.ts` (new): DATABASE_URL-gated Postgres E2E. 60-file happy path with `pg_stat_activity` leak probe, plus a 120-file serial-vs-parallel benchmark that prints `SYNC_PARALLEL_BENCH ...` for CHANGELOG quoting.
|
||||
|
||||
### For contributors
|
||||
|
||||
- `BrainEngine.kind` is now the canonical PGLite/Postgres discriminator. Avoid `engine.constructor.name === '...'` (breaks under bundling) and `config.engine === '...'` (inconsistent with the engine actually in use).
|
||||
- The `gbrain_cycle_locks` table is now multi-purpose. The id column distinguishes lock scopes: `gbrain-cycle` for the cycle, `gbrain-sync` for the sync writer. Future locks should pick distinct ids and reuse `tryAcquireDbLock`.
|
||||
- `parseWorkers()` is the canonical CLI flag parser for `--workers`. Use it instead of inline `parseInt`.
|
||||
|
||||
## [0.22.12] - 2026-04-29
|
||||
|
||||
**`sync --skip-failed` now classifies file-size and symlink rejections instead of bucketing them as UNKNOWN.**
|
||||
**Plus a full end-to-end test for the failure loop.**
|
||||
|
||||
v0.22.9 shipped the headline classifier work: code-grouped breakdowns at sync time,
|
||||
DB-vs-YAML disambiguation, doctor surfaces both unacked and historical entries with
|
||||
`[CODE=N]` lines. v0.22.12 closes the last two coverage gaps that v0.22.9 left on
|
||||
the table:
|
||||
|
||||
- **FILE_TOO_LARGE** now covers the three real production sites in
|
||||
`src/core/import-file.ts:199, 352, 401` ("Content too large", "File too large",
|
||||
"Code file too large"). On v0.22.9 these all bucketed as UNKNOWN — the same
|
||||
silent-systemic-failure pattern that motivated the original issue.
|
||||
- **SYMLINK_NOT_ALLOWED** covers `src/core/import-file.ts:347` ("Skipping symlink").
|
||||
Security-relevant rejection that operators should see.
|
||||
- **End-to-end failure-loop test** in `test/e2e/sync.test.ts` exercises the full
|
||||
chain: broken file → sync blocks with grouped breakdown → `--skip-failed`
|
||||
advances bookmark with grouped acknowledgement → second broken file → second
|
||||
cycle. PostgreSQL-backed; verifies bookmark gating, JSONL state, dedup, and
|
||||
summary aggregation. v0.22.9's coverage was unit-tests-only.
|
||||
|
||||
Twelve total error codes ship in the classifier:
|
||||
`SLUG_MISMATCH`, `YAML_PARSE`, `YAML_DUPLICATE_KEY`, `DB_DUPLICATE_KEY`,
|
||||
`MISSING_OPEN`, `MISSING_CLOSE`, `NESTED_QUOTES`, `EMPTY_FRONTMATTER`,
|
||||
`NULL_BYTES`, `INVALID_UTF8`, `STATEMENT_TIMEOUT`, `FILE_TOO_LARGE`,
|
||||
`SYMLINK_NOT_ALLOWED`. Anything the regex set doesn't recognize falls through
|
||||
as `UNKNOWN`.
|
||||
|
||||
### What this means for you
|
||||
|
||||
If your brain rejects oversized files or symlinks, you now see those rejections
|
||||
in the doctor breakdown and at sync time grouped by code, instead of as
|
||||
`UNKNOWN`. Run `gbrain upgrade`. No manual action required.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
#### Added
|
||||
- `FILE_TOO_LARGE` classifier code covering `src/core/import-file.ts:199, 352, 401`.
|
||||
- `SYMLINK_NOT_ALLOWED` classifier code covering `src/core/import-file.ts:347`.
|
||||
- Two new unit tests in `test/sync-failures.test.ts` pinning the new codes against
|
||||
literal production message strings (`File too large (N bytes)`, `Skipping symlink: ...`).
|
||||
- `test/e2e/sync.test.ts` — new failure-loop test exercising broken-file → block →
|
||||
`--skip-failed` → second cycle. Hermetic on developer machines (saves+restores
|
||||
the user's real `~/.gbrain/sync-failures.jsonl`).
|
||||
|
||||
## To take advantage of v0.22.12
|
||||
|
||||
No manual action required. Run `gbrain upgrade`. The new `FILE_TOO_LARGE` and
|
||||
`SYMLINK_NOT_ALLOWED` classifier codes apply on the next `gbrain sync`.
|
||||
|
||||
## [0.22.11] - 2026-04-27
|
||||
|
||||
**Storage tiering, finally working. Brains scaling past 100K files stop bloating git.**
|
||||
|
||||
The original storage-tiering branch shipped two silent bugs (gray-matter on YAML returned empty data; `manageGitignore` was defined and never invoked) so the feature was a no-op for every user who tried it. v0.22.11 rewrites the broken bits, hardens the surface, and adds proper test coverage. If you have a brain repo north of 100K files where bulk machine-generated content (tweets, articles, transcripts) is the size driver, this is the release that pulls it out of git without losing any data.
|
||||
|
||||
Configure tiering in `gbrain.yml` at the brain repo root:
|
||||
|
||||
```yaml
|
||||
storage:
|
||||
db_tracked:
|
||||
- people/
|
||||
- companies/
|
||||
- deals/
|
||||
db_only:
|
||||
- media/x/
|
||||
- media/articles/
|
||||
- meetings/transcripts/
|
||||
```
|
||||
|
||||
`gbrain sync` then auto-manages your `.gitignore` for `db_only` directories so bulk content stops landing in commits. `gbrain export --restore-only` repopulates missing `db_only` files from the database (container restart, fresh clone, accidental rm). `gbrain storage status` shows the breakdown — counts, disk usage, missing files.
|
||||
|
||||
### The numbers that matter
|
||||
|
||||
200K-page brain, half tweets and articles. Before v0.22.11:
|
||||
|
||||
| Metric | Before | After | Δ |
|
||||
|--------|--------|-------|---|
|
||||
| `gbrain.yml` actually loads | no (silent null) | yes | feature works |
|
||||
| `.gitignore` auto-manages | no (function never called) | yes | docs match reality |
|
||||
| `--restore-only` without `--repo` | silent full export | hard error | no data-loss footgun |
|
||||
| `media/xerox` matched against `media/x` | yes (collision) | no | path-segment matching |
|
||||
| Per-page disk syscalls during status | ~400K (existsSync + statSync) | ~one per dir + one stat per .md | single-walk scan |
|
||||
| Validation surfaces overlap | warning only | throws StorageConfigError | semantic error caught |
|
||||
|
||||
### What this means for your brain
|
||||
|
||||
If you've been reading the storage-tiering docs and waiting for the feature to actually do something: it does now. If you're already over 50K files: configure `gbrain.yml`, run `gbrain sync`, watch `.gitignore` update itself, watch your next clone get faster.
|
||||
|
||||
## To take advantage of v0.22.11
|
||||
|
||||
1. Add a `storage:` section to `gbrain.yml` at your brain repo root with `db_tracked` and `db_only` arrays. The directory paths must end with `/` (the validator auto-normalizes if you forget, with a one-time info note).
|
||||
2. Run `gbrain sync`. It updates `.gitignore` automatically on success.
|
||||
3. Run `gbrain storage status` to see the tier breakdown and any missing `db_only` files.
|
||||
4. If files are missing on disk (e.g., after a container restart): `gbrain export --restore-only --repo /path/to/brain`.
|
||||
5. If you previously had `git_tracked` / `supabase_only` keys: they still load, with a once-per-process deprecation warning. Rename to `db_tracked` / `db_only` at your convenience.
|
||||
6. On PGLite: tiering has limited effect (the "DB" is your local file). The `.gitignore` housekeeping still helps. A one-time soft-warn explains.
|
||||
|
||||
If anything looks off, file an issue at <https://github.com/garrytan/gbrain/issues> with `gbrain doctor` output and the contents of your `gbrain.yml`.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
#### Critical fixes
|
||||
|
||||
- **YAML parser swap**: replaced `gray-matter` with a dedicated YAML reader for the `gbrain.yml` shape. The original code called `matter()` on a delimiter-less file, which always returned `{data: {}}` — `loadStorageConfig` returned null on every install. The dedicated parser handles top-level `storage:` plus nested array-valued keys, with comment + blank-line tolerance. Once-per-process sanity warning when `gbrain.yml` exists but has no `storage:` section.
|
||||
- **`manageGitignore` actually runs now**: wired into `runSync` after every successful sync (skipped on dry-run, blocked-by-failures, and unhandled errors). Idempotent. Detects git submodule context (`.git` is a file, not a directory) and skips with an actionable warning. Honors `GBRAIN_NO_GITIGNORE=1` for shared-repo setups.
|
||||
- **No more silent `--restore-only` footgun**: `gbrain export --restore-only` without `--repo` now resolves through a typed `getDefaultSourcePath()` accessor (sources table → null → hard error). Never falls through to the current directory. Never silently re-exports your entire database into the wrong place.
|
||||
|
||||
#### New + renamed surface
|
||||
|
||||
- **Canonical key names**: `db_tracked` / `db_only` replace the vendor-baked `git_tracked` / `supabase_only`. The deprecated keys still load, with a once-per-process warning suggesting `gbrain doctor --fix` for an automated rename. Canonical wins when both shapes coexist.
|
||||
- **Engine-side `slugPrefix` filter**: `PageFilters.slugPrefix` lands on both engines as `WHERE slug LIKE prefix || '%'` with literal-escape of LIKE metacharacters. Uses the existing `(source_id, slug)` UNIQUE btree index for range scans. Powers `gbrain export --restore-only` per-tier queries and `gbrain export --slug-prefix`.
|
||||
- **Single-walk filesystem scan**: `src/core/disk-walk.ts` exposes `walkBrainRepo(repoPath)` that returns `Map<slug, {size, mtimeMs}>` from one recursive `readdirSync`. Replaces the per-page `existsSync + statSync` loop in `gbrain storage status` (~400K syscalls on a 200K-page brain → tens).
|
||||
- **Path-segment matching**: tier directory matcher requires trailing `/` and treats the slash as a path separator. `media/x/` does not match `media/xerox/foo`. Validator (`normalizeAndValidateStorageConfig`) auto-fixes missing trailing `/`, throws `StorageConfigError` on tier overlap.
|
||||
|
||||
#### Architecture cleanup
|
||||
|
||||
- `src/commands/storage.ts` split into pure data + JSON formatter + human formatter + thin dispatcher, matching the `orphans.ts` precedent. `getStorageStatus` is exported for `gbrain doctor` integration. ASCII-only output (no unicode box-drawing) for cross-platform terminal compatibility.
|
||||
- Distinct nominal types `PageCountsByTier` and `DiskUsageByTier` so accidental swaps between page counts and byte totals are compile-time errors.
|
||||
- PGLite soft-warn on storage tiering (D4): the feature is partial on PGLite (the "DB" is your local file), but `.gitignore` housekeeping still helps. Once-per-process warning explains and proceeds.
|
||||
|
||||
#### Tests + CI guards
|
||||
|
||||
- New unit tests across `test/storage-config.test.ts`, `test/storage-sync.test.ts`, `test/storage-status.test.ts`, `test/storage-export.test.ts`, `test/storage-pglite.test.ts`, `test/disk-walk.test.ts`. Plus extensions to `test/source-resolver.test.ts` and `test/pglite-engine.test.ts`. The single-line test that would have caught the original gray-matter P0 (write a real `gbrain.yml`, call `loadStorageConfig`, assert non-null) now exists.
|
||||
- New CI guard `scripts/check-trailing-newline.sh` (sibling to the existing jsonb-pattern + progress-to-stdout guards). Wired into `bun run test`. Fixed pre-existing missing newline in `docs/storage-tiering.md`.
|
||||
|
||||
### For contributors
|
||||
|
||||
- The eng-review path forward is documented in `~/.claude/plans/lets-take-a-look-ticklish-pizza.md` (15 numbered defects + D1-D8 abstraction calls). Every commit on this branch maps to one numbered step in the plan.
|
||||
|
||||
## [0.22.10] - 2026-04-30
|
||||
|
||||
**`gbrain jobs submit autopilot-cycle --params '{"phases":["lint","backlinks"]}'` now actually runs only those phases.**
|
||||
|
||||
If you ever submitted an `autopilot-cycle` job with a `phases:` array hoping to skip embed for a fast cycle, you got the full 6-phase cycle anyway. The handler in `src/commands/jobs.ts` was calling `runCycle(...)` without forwarding `job.data.phases`, so per-cycle phase selection was silently ignored.
|
||||
|
||||
This release wires the array through. The handler imports `ALL_PHASES` from `src/core/cycle.ts`, builds a `Set` for O(1) validation, and filters the caller's `phases` array against it before forwarding to `runCycle`. Invalid phase names get dropped (no injection surface — `ALL_PHASES` is the authoritative list). Empty arrays and non-array values fall back to the default (run all phases), preserving the prior behavior for callers who didn't ask for selective phases.
|
||||
|
||||
### What this means for you
|
||||
|
||||
If you've been using `gbrain jobs submit autopilot-cycle --params '{"phases":[...]}'` for triage cycles (e.g. `["lint","backlinks"]` for a fast structural sweep, skipping the slow embed phase), you'll now see those cycles take seconds instead of minutes. The CLI surface didn't change — only the worker's handler now respects the `phases` it was already accepting.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
#### Fixed
|
||||
|
||||
- `autopilot-cycle` minion handler in `src/commands/jobs.ts` now forwards `job.data.phases` to `runCycle()`. Previously the handler accepted the array via `MinionJobInput.params` but discarded it before dispatch.
|
||||
- Phase names validated against `ALL_PHASES` from `src/core/cycle.ts`. Filter is exhaustive: array → filtered, non-array → undefined (default), filtered-to-empty → no `phases` key in opts (also default).
|
||||
|
||||
#### Tests
|
||||
|
||||
- 4 new test cases in `test/handlers.test.ts` under `autopilot-cycle handler — phase passthrough`: valid phases forwarded, invalid names filtered, empty array falls back to all-phases, non-array `phases` value ignored. Pin both the contract and the fallback semantics.
|
||||
- `test/cycle-abort.test.ts` regression-guard window widened from 500 → 2000 chars so the source-level `signal: job.signal` check finds the line after the new validation block was added between `worker.register('autopilot-cycle', ...)` and the `runCycle(...)` call. Pure test fix; the handler still propagates the abort signal correctly.
|
||||
|
||||
## [0.22.9] - 2026-04-29
|
||||
|
||||
**Sync failures now tell you why, not just how many.**
|
||||
**`gbrain sync --skip-failed` and `gbrain doctor` group failures by error code, so 2,685 silent SLUG_MISMATCH files don't hide behind a single count.**
|
||||
|
||||
Before this release, when sync hit per-file parse errors the only signal was a number:
|
||||
|
||||
```
|
||||
Sync blocked: 2688 file(s) failed to parse. Fix the YAML frontmatter...
|
||||
```
|
||||
|
||||
That count is useless when you're staring at 2,688 files and don't know what's wrong. On a real 81K-page brain, 2,685 of those turned out to be `SLUG_MISMATCH` from a posterous import — a single root cause hiding behind a giant number. It took manual `cat ~/.gbrain/sync-failures.jsonl | jq` to figure that out.
|
||||
|
||||
After:
|
||||
|
||||
```
|
||||
Sync blocked: 2688 file(s) failed to parse:
|
||||
SLUG_MISMATCH: 2685
|
||||
YAML_DUPLICATE_KEY: 3
|
||||
|
||||
Fix the YAML frontmatter in the files above and re-run, or use 'gbrain sync --skip-failed' to acknowledge and move on.
|
||||
|
||||
# gbrain sync --skip-failed
|
||||
Acknowledged 2688 failure(s) and advancing past them:
|
||||
SLUG_MISMATCH: 2685
|
||||
YAML_DUPLICATE_KEY: 3
|
||||
```
|
||||
|
||||
`gbrain doctor` shows the same breakdown for unacknowledged AND historical entries:
|
||||
|
||||
```
|
||||
[WARN] sync_failures: 2688 unacknowledged sync failure(s) [SLUG_MISMATCH=2685, YAML_DUPLICATE_KEY=3].
|
||||
[OK] sync_failures: 500544 historical sync failure(s), all acknowledged [SLUG_MISMATCH=2685, ...].
|
||||
```
|
||||
|
||||
The classifier knows the canonical messages from `collectValidationErrors()` in `src/core/markdown.ts` (8 frontmatter codes), Postgres unique-constraint violations (`DB_DUPLICATE_KEY`), statement-timeout errors (`STATEMENT_TIMEOUT`), invalid UTF-8, and YAML duplicates. DB-layer errors check before YAML-layer ones — so a Postgres `duplicate key value violates unique constraint` no longer mislabels as a YAML duplicate. Unrecognized errors fall through to `UNKNOWN`.
|
||||
|
||||
### What this means for you
|
||||
|
||||
If `gbrain sync` blocks with parse failures, the breakdown tells you what to fix first. SLUG_MISMATCH is one fix-pattern (frontmatter says one slug, path says another); YAML_PARSE is a different one (malformed YAML); STATEMENT_TIMEOUT means a DB timeout, not a parse problem. You stop staring at counts and start fixing root causes.
|
||||
|
||||
### For contributors
|
||||
|
||||
`acknowledgeSyncFailures()` in `src/core/sync.ts` now returns `{count, summary}` instead of `number`. If you import this directly from `gbrain/sync`, replace `n` with `result.count` and use `result.summary` (an `Array<{code, count}>`) for the new code-grouped breakdown. The function is reachable via the package exports map; this is a deliberate, non-shimmed breaking change. There is a new `formatCodeBreakdown()` helper in the same module that accepts either raw failures or pre-summarized input — use it instead of building breakdown strings inline.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
#### Added
|
||||
|
||||
- `classifyErrorCode(errorMsg)` in `src/core/sync.ts` — best-effort error-code extraction from sync failure messages. Codes: `SLUG_MISMATCH`, `YAML_PARSE`, `YAML_DUPLICATE_KEY`, `MISSING_OPEN`, `MISSING_CLOSE`, `EMPTY_FRONTMATTER`, `NULL_BYTES`, `NESTED_QUOTES`, `DB_DUPLICATE_KEY`, `STATEMENT_TIMEOUT`, `INVALID_UTF8`, `UNKNOWN`.
|
||||
- `summarizeFailuresByCode(failures)` — groups failures by code and returns a sorted `Array<{code, count}>`.
|
||||
- `formatCodeBreakdown(input)` — renders a multi-line `code: count` string from either raw failures or a pre-computed summary. Single helper, two input shapes.
|
||||
- `code?: string` field on the `SyncFailure` JSONL row in `~/.gbrain/sync-failures.jsonl`. Populated at write-time so the classifier runs once per failure, not on every load.
|
||||
- `AcknowledgeResult` interface as the new return shape of `acknowledgeSyncFailures()`.
|
||||
- 15 new test cases in `test/sync-failures.test.ts`: DB-vs-YAML duplicate-key disambiguation, canonical-message coverage for all 7 frontmatter codes, `acknowledgeSyncFailures()` legacy-entry backfill branch, `formatCodeBreakdown()` dual-input shape.
|
||||
|
||||
#### Changed
|
||||
|
||||
- `gbrain sync` blocked-message: now lists code breakdown above the fix instructions (both incremental and full-sync paths).
|
||||
- `gbrain sync --skip-failed` ack message: now lists what was skipped, grouped by code.
|
||||
- `gbrain doctor` `sync_failures` check: warn-and-ok messages both include `[code=count, ...]` breakdown.
|
||||
- `recordSyncFailures()` now stores `code` alongside `error` so downstream readers don't re-classify.
|
||||
- `acknowledgeSyncFailures()` backfills `code` on legacy rows that predate the field — upgrade-safe for users with existing `~/.gbrain/sync-failures.jsonl`.
|
||||
- DB-layer error patterns (`DB_DUPLICATE_KEY`, `STATEMENT_TIMEOUT`) check BEFORE YAML patterns in the classifier, so Postgres errors don't get YAML-labeled.
|
||||
- Frontmatter regex patterns rewritten to match canonical messages from `collectValidationErrors()` (`File is empty...`, `No closing --- delimiter found`, `Frontmatter block is empty`) instead of aspirational code-token strings (`missing.*open`) that never appeared in practice.
|
||||
|
||||
Closes #500.
|
||||
|
||||
## [0.22.8] - 2026-04-28
|
||||
|
||||
## **Doctor stops timing out on Supabase. Integrity scan finishes in ~6s, multi-source brains get correct counts.**
|
||||
|
||||
If you've been hitting the 60-second `gbrain doctor` timeout on Supabase or any pooled-connection deployment, this fixes it. The integrity check used to call `getPage()` 500 times sequentially through PgBouncer transaction-mode pooling. Each call required a full connection acquire/release cycle, which doctor couldn't finish before CI killed it. The new path batch-loads all 500 pages in a single SQL query, finishing in ~6s.
|
||||
|
||||
While shipping the perf fix, codex review caught a correctness regression for multi-source brains: the batch SQL was scanning raw `(source_id, slug)` rows while the sequential path scanned unique slugs. Multi-source brains were getting inflated counts. `SELECT DISTINCT ON (slug)` mirrors the sequential path's `Set<string>` semantics; parity tests against real Postgres pin both paths to the same output.
|
||||
|
||||
Plus a Linux CI fix: `gbrain skillpack` lockfile checks were intermittently failing on ext4's sub-millisecond `mtimeMs` timestamps when `Date.now()` returned an integer ms behind the file's recorded mtime. Lock age now clamps to zero.
|
||||
|
||||
### The numbers that matter
|
||||
|
||||
Measured against the real failure mode on a Supabase PgBouncer deployment that hit the 60s CI timeout pre-fix.
|
||||
|
||||
| Behavior | Before v0.22.8 | After v0.22.8 |
|
||||
|---|---|---|
|
||||
| `gbrain doctor` wall-clock (Postgres + PgBouncer) | 60s+ timeout (killed) | ~6s |
|
||||
| `integrity_sample` query round-trips | ~500 (sequential `getPage`) | 1 (`SELECT DISTINCT ON`) |
|
||||
| Multi-source brain scan accuracy | Overcounted by `source_id` | Exact per unique slug |
|
||||
|
||||
### What this means for Supabase deployments
|
||||
|
||||
If you've been avoiding `gbrain doctor` because it timed out, run it again. If you maintain a multi-source brain (imported pages from another gbrain deployment under a non-default `source_id`), the scan now treats each slug once instead of once-per-source — your output is exact, not inflated. Single-source users see no behavior change; PGLite users were never affected (the batch path is Postgres-only).
|
||||
|
||||
## To take advantage of v0.22.8
|
||||
|
||||
`gbrain upgrade` should do this automatically. If it didn't, or if `gbrain doctor` warns about anything afterwards:
|
||||
|
||||
1. **Run the upgrade:**
|
||||
```bash
|
||||
gbrain upgrade
|
||||
```
|
||||
2. **Verify doctor finishes cleanly (especially relevant if you hit timeouts before):**
|
||||
```bash
|
||||
gbrain doctor
|
||||
```
|
||||
On Postgres + PgBouncer deployments, you should see `integrity_sample` finish in ~6s instead of timing out at 60s.
|
||||
3. **If `doctor` still times out or output looks wrong,** please file an issue:
|
||||
https://github.com/garrytan/gbrain/issues with:
|
||||
- output of `gbrain doctor` (full)
|
||||
- which engine (Postgres vs PGLite)
|
||||
- whether you use multi-source brains
|
||||
|
||||
### Itemized changes
|
||||
|
||||
#### Performance
|
||||
- `gbrain doctor` integrity sample now batch-loads via a single SQL query on Postgres deployments (60s+ timeout → ~6s wall-clock, 500 round-trips → 1).
|
||||
- Batch path explicitly gated to Postgres via `engine.kind` so PGLite never attempts it (clean fallback signal).
|
||||
|
||||
#### Correctness
|
||||
- `scanIntegrity` batch path uses `SELECT DISTINCT ON (slug)` to scope by unique slug, matching `engine.getAllSlugs()`'s `Set<string>` semantics. Multi-source brains (UNIQUE(source_id, slug) since v0.18.0) now get correct counts instead of one-scan-per-source-row.
|
||||
- `IntegrityScanResult.pagesScanned` now reflects unique slugs scanned, not raw row count. Single-source brains: unchanged. Multi-source brains: counts now match expected distinct-page semantics.
|
||||
- Batch-path fallback narrowed: real Postgres errors (deadlock, connection drop, SQL bug) surface via `GBRAIN_DEBUG=1` instead of being silently swallowed.
|
||||
|
||||
#### Tests
|
||||
- New `test/e2e/integrity-batch.test.ts` — four parity cases (dedup, hits, validate, topPages) asserting batch ≡ sequential against real Postgres. Pinning the multi-source dedup case requires a raw-SQL fixture for the alt-source row since `engine.putPage` doesn't take a `source_id`.
|
||||
|
||||
#### Infrastructure
|
||||
- `src/core/skillpack/installer.ts` — clamp negative lock-age to 0, fixing intermittent Linux ext4 CI flakes from sub-millisecond `mtimeMs` precision (Date.now is integer ms; mtime can be ~0.3ms ahead). New regression test in `test/skillpack-install.test.ts` deterministically reproduces via `utimesSync`.
|
||||
- `CLAUDE.md` test inventory updated for the new test files.
|
||||
|
||||
## [0.22.7] - 2026-04-28
|
||||
|
||||
## **Built-in HTTP transport with bearer auth for remote MCP.**
|
||||
## **Postgres-backed tokens, default-deny CORS, two-bucket rate limit, body cap, per-request audit.**
|
||||
|
||||
v0.22.7 ships `gbrain serve --http`: a built-in HTTP transport for remote MCP, authenticating via the existing `access_tokens` table that `gbrain auth create/list/revoke` already manages. Bearer-only, no OAuth surface, no registration endpoint, no self-service tokens. SECURITY.md is the canonical reference for the hardening posture and recommended deployment.
|
||||
|
||||
The hardening lives inside the transport, not in the doc:
|
||||
|
||||
| Layer | Default | Configurable via |
|
||||
|---|---|---|
|
||||
| CORS | default-deny (no `Access-Control-Allow-Origin`) | `GBRAIN_HTTP_CORS_ORIGIN=a.com,b.com` |
|
||||
| Pre-auth IP rate limit | 30 req / 60s | `GBRAIN_HTTP_RATE_LIMIT_IP` |
|
||||
| Post-auth token rate limit | 60 req / 60s | `GBRAIN_HTTP_RATE_LIMIT_TOKEN` |
|
||||
| Body cap | 1 MiB, stream-counted | `GBRAIN_HTTP_MAX_BODY_BYTES` |
|
||||
| `last_used_at` debounce | once per token per 60s | (SQL-level WHERE clause, race-tolerant) |
|
||||
| Per-request audit | `mcp_request_log` row per `/mcp` | (existing schema, since v4) |
|
||||
| Reverse-proxy trust | off | `GBRAIN_HTTP_TRUST_PROXY=1` to honor X-Forwarded-For |
|
||||
|
||||
The IP rate-limit fires **before** the auth lookup so the limit caps load on the auth path itself, not just response codes. The token-id rate limit fires after auth so a runaway authenticated client gets throttled at the right principal. Both buckets live in a bounded LRU map (default 10K keys, TTL prune at 2× window) so unique-key growth can't drift into memory pressure.
|
||||
|
||||
### What changed for users
|
||||
|
||||
You can now expose GBrain remotely with the built-in transport:
|
||||
|
||||
```bash
|
||||
gbrain auth create my-laptop # tokens managed via the existing CLI
|
||||
gbrain serve --http --port 8787 # Postgres-only; PGLite users see a clear fail-fast
|
||||
ngrok http 8787 --url your-brain.ngrok.app # any tunnel works
|
||||
```
|
||||
|
||||
Then point Claude Desktop, claude.ai/code, or any MCP client at `http://your-tunnel/mcp` with `Authorization: Bearer <token>`. CORS, rate limits, and body caps are on by default. `gbrain auth` is now wired into the main CLI, so it works from the compiled binary the same as `gbrain doctor` or `gbrain serve`.
|
||||
|
||||
### For contributors
|
||||
|
||||
- `src/mcp/dispatch.ts` (new) — shared `dispatchToolCall(engine, name, params, opts)` consumed by both stdio (`server.ts`) and HTTP (`http-transport.ts`). One source of truth for `validateParams`, `OperationContext` construction, and handler invocation, so the two transports can't drift apart.
|
||||
- `src/mcp/rate-limit.ts` (new) — bounded-LRU token-bucket. Tracks `lastTouchedMs` separately from `lastRefillMs` so an exhausted key can't be reset by hammering past the TTL.
|
||||
- `src/mcp/http-transport.ts` — built on the new dispatch + rate-limit modules. `application/json` response shape (gbrain MCP tools are synchronous; the Streamable HTTP transport spec allows JSON for non-streaming responses).
|
||||
- `src/cli.ts` + `src/commands/auth.ts` — `auth` is now a wired CLI subcommand. Direct-script usage (`bun run src/commands/auth.ts ...`) still works for environments without a compiled binary.
|
||||
- 23 unit cases in `test/http-transport.test.ts`, 8 E2E cases in `test/e2e/http-transport.test.ts`. Unit covers the full dispatch round-trip with a real operation; E2E covers `last_used_at` debounce against real Postgres semantics.
|
||||
|
||||
### Known limits
|
||||
|
||||
- `gbrain serve --http` is **Postgres-only**. PGLite has no `access_tokens` or `mcp_request_log` table by design (`src/core/pglite-schema.ts:5-6`). Local agents continue to use stdio (`gbrain serve`).
|
||||
- Behind a tunnel (ngrok, Tailscale Funnel, Cloudflare Tunnel), all requests share one egress IP. The pre-auth IP bucket becomes effectively shared by all clients on that tunnel; the token-id bucket is the load-bearing limiter for tunnel deployments. Documented in SECURITY.md.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
- New: `gbrain serve --http [--port N]` ships the built-in HTTP transport
|
||||
- New: `gbrain auth create/list/revoke/test` wired into the main CLI (was a standalone script)
|
||||
- New: SECURITY.md documents the disclosure path, the recommended remote-MCP setup, and the full hardening reference
|
||||
- New: `src/mcp/dispatch.ts` — shared dispatch path for stdio + HTTP
|
||||
- New: `src/mcp/rate-limit.ts` — bounded-LRU token-bucket limiter
|
||||
- Hardening: CORS default-deny, two-bucket rate limit (per-IP pre-auth + per-token post-auth), 1 MiB body cap with stream-counted enforcement, `mcp_request_log` per-request audit, `last_used_at` SQL-level debounce
|
||||
- Tests: 23 unit + 8 E2E covering auth, dispatch, CORS, body cap, rate limit, and audit
|
||||
- Docs: SECURITY.md, DEPLOY.md, and per-client setup guides updated to recommend `--http` and document the env vars
|
||||
|
||||
## To take advantage of v0.22.7
|
||||
|
||||
`gbrain upgrade` should do this automatically. If it didn't, or if you want to expose your brain over HTTP:
|
||||
|
||||
1. **Confirm migrations are at v4 or higher** (the `access_tokens` + `mcp_request_log` tables were added in migration v4):
|
||||
```bash
|
||||
gbrain doctor # schema_version check should pass
|
||||
gbrain apply-migrations --yes # if not, run this
|
||||
```
|
||||
2. **Create a token for each remote client:**
|
||||
```bash
|
||||
gbrain auth create my-laptop # prints the token once — copy it
|
||||
```
|
||||
3. **Start the HTTP server:**
|
||||
```bash
|
||||
gbrain serve --http --port 8787
|
||||
```
|
||||
4. **(Optional) configure CORS allowlist if a browser client will hit it:**
|
||||
```bash
|
||||
GBRAIN_HTTP_CORS_ORIGIN=https://claude.ai gbrain serve --http --port 8787
|
||||
```
|
||||
5. **(Optional) audit who's hitting your brain:**
|
||||
```bash
|
||||
psql $DATABASE_URL -c "SELECT created_at, token_name, operation, status, latency_ms
|
||||
FROM mcp_request_log ORDER BY created_at DESC LIMIT 50"
|
||||
```
|
||||
6. **If `gbrain serve --http` exits with "Postgres engine required":** PGLite is local-only by design. Either keep using stdio (`gbrain serve`) for local agents, or migrate to Postgres (`gbrain migrate --to supabase`).
|
||||
|
||||
If anything breaks: `gbrain doctor`, `~/.gbrain/upgrade-errors.jsonl` (if present), and please file an issue at https://github.com/garrytan/gbrain/issues with both.
|
||||
|
||||
## [0.22.6.1] - 2026-04-26
|
||||
|
||||
**Old brains can upgrade again.**
|
||||
**Two-year, ten-issue wedge cycle ends. Pre-v0.13/v0.18/v0.19 brains all upgrade clean.**
|
||||
|
||||
If you've been pinned to an older gbrain because `gbrain upgrade` wedges your brain
|
||||
with `column "source_id" does not exist` or `column "link_source" does not exist`,
|
||||
v0.22.6.1 unblocks you. The fix lives in `initSchema()` itself, where it should
|
||||
have lived all along.
|
||||
|
||||
The bug class is structural: gbrain ships an "embedded latest schema" SQL blob
|
||||
that runs before numbered migrations on every connect. The blob references
|
||||
columns that newer migrations introduce. On any brain older than the migration
|
||||
that adds those columns, the blob crashes before the migration can run. This
|
||||
incident family hit users 10+ times across 6 schema versions over 2 years
|
||||
(issues #239, #243, #266, #357, #366, #374, #375, #378, #395, #396).
|
||||
|
||||
The fix is a narrow pre-schema bootstrap. `initSchema()` now probes for the
|
||||
specific forward-referenced state the schema blob needs (`pages.source_id`,
|
||||
`links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`,
|
||||
`content_chunks.language`, plus the `sources` FK target table) and adds only
|
||||
that state if missing. Then SCHEMA_SQL replays cleanly. Then the normal
|
||||
migration chain runs as usual. Fresh installs and modern brains both no-op.
|
||||
|
||||
A test guard prevents this incident family from recurring. Every future
|
||||
migration that adds a column-with-index to PGLITE_SCHEMA_SQL must extend the
|
||||
bootstrap; the CI guard fails loudly if not. The pattern that broke gbrain ten
|
||||
times in two years is now structurally prevented.
|
||||
|
||||
Also includes the v24 PGLite RLS fix from #395 (community PR by @jdcastro2):
|
||||
`rls_backfill_missing_tables` now no-ops on PGLite via `sqlFor.pglite: ''`,
|
||||
since PGLite has no RLS engine and is single-tenant by definition.
|
||||
|
||||
### The numbers that matter
|
||||
|
||||
| Metric | v0.22.0 | v0.22.6.1 | Δ |
|
||||
|---|---|---|---|
|
||||
| Pre-v0.13 brain upgrades cleanly | wedges on `link_source` | passes | ✓ |
|
||||
| Pre-v0.18 brain upgrades cleanly | wedges on `source_id` | passes | ✓ |
|
||||
| Pre-v0.21 brain upgrades cleanly | wedges on `symbol_name` | passes | ✓ |
|
||||
| v24 RLS migration on PGLite | wedges (table doesn't exist) | no-op | ✓ |
|
||||
| Issues closed | — | #366, #375, #378, #395, #396 | 5 |
|
||||
| Issue families resolved | — | wedge-cycle | the whole class |
|
||||
|
||||
### What this means for you
|
||||
|
||||
If you've been on v0.13.x, v0.14.x, v0.17.x, v0.18.x, v0.19.x, v0.20.x, or v0.22.0 and
|
||||
your `gbrain upgrade` failed, run it again. It should walk to v0.22.6.1 cleanly.
|
||||
If you wedged on the v24 RLS migration on a PGLite brain, the same thing.
|
||||
|
||||
If you're on a fresh install or already on v0.22.0, this patch is invisible.
|
||||
The bootstrap probe runs once per connect, sees nothing to do, and returns.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
#### Fixed
|
||||
- `gbrain upgrade` no longer wedges on pre-v0.18 brains that lack `pages.source_id`. The schema blob's `CREATE INDEX idx_pages_source_id` previously crashed before migration v21 could add the column. Closes #366, #375, #378, #396.
|
||||
- `gbrain upgrade` no longer wedges on pre-v0.13 brains that lack `links.link_source` or `links.origin_page_id`. The schema blob's `CREATE INDEX idx_links_source/origin` previously crashed before migration v11 could add the columns. Closes #266, #357.
|
||||
- `gbrain upgrade` no longer wedges on pre-v0.19 brains that lack `content_chunks.symbol_name` or `content_chunks.language`. The schema blob's partial indexes previously crashed before migration v26 could add the columns.
|
||||
- Migration v24 (`rls_backfill_missing_tables`) no-ops on PGLite via `sqlFor.pglite: ''`. PGLite has no RLS engine and is single-tenant. The migration previously tried to ALTER subagent tables that don't exist in pglite-schema.ts. Closes #395. Contributed by @jdcastro2.
|
||||
|
||||
#### Changed
|
||||
- `PGLiteEngine.initSchema()` and `PostgresEngine.initSchema()` now call a new private `applyForwardReferenceBootstrap()` before running the embedded schema blob. The bootstrap probes for missing forward-referenced state and adds only what's needed. No-op on fresh installs and modern brains.
|
||||
|
||||
#### For contributors
|
||||
- New CI guard `test/schema-bootstrap-coverage.test.ts` enforces that `applyForwardReferenceBootstrap` covers every forward reference in PGLITE_SCHEMA_SQL. When you add a new column-with-index in the schema blob, extend `REQUIRED_BOOTSTRAP_COVERAGE` and the bootstrap function. The test fails loudly if you skip step one.
|
||||
- New `test/bootstrap.test.ts` covers the bootstrap contract: no-op on fresh install, idempotent, no-op on modern brain, full path pre-v0.18, fresh-install regression, pre-v0.13 links shape.
|
||||
- New `test/e2e/postgres-bootstrap.test.ts` exercises `PostgresEngine.initSchema()` directly (not the standalone `db.initSchema` from `src/core/db.ts`, which only runs SCHEMA_SQL and would have produced false-positive coverage). Codex caught this E2E shape gap during plan review.
|
||||
- Wave PRs incorporated with attribution: @vinsew (#398), @jdcastro2 (#399), @schnubb-web (#402). The narrow-bootstrap shape supersedes #402's broader "run all migrations early" approach, which would have crashed on v24 trying to alter tables that the schema blob hadn't created yet (codex finding during plan review).
|
||||
|
||||
## To take advantage of v0.22.6.1
|
||||
|
||||
`gbrain upgrade` should do this automatically. If you're currently wedged on a
|
||||
prior version's upgrade attempt:
|
||||
|
||||
1. **Run the upgrade:**
|
||||
```bash
|
||||
gbrain upgrade
|
||||
```
|
||||
2. **Verify the outcome:**
|
||||
```bash
|
||||
gbrain doctor
|
||||
```
|
||||
Expected: `schema_version: Version 29 (latest: 29)` clean, no
|
||||
`column "..." does not exist` errors, no wedged migration ledger.
|
||||
|
||||
3. **If wedged after upgrade,** run the migration runner directly:
|
||||
```bash
|
||||
gbrain apply-migrations --yes
|
||||
```
|
||||
|
||||
4. **If any step still fails,** please file an issue:
|
||||
https://github.com/garrytan/gbrain/issues with:
|
||||
- output of `gbrain doctor`
|
||||
- your prior gbrain version (`gbrain --version`)
|
||||
- which step broke
|
||||
|
||||
## [0.22.6] - 2026-04-28
|
||||
|
||||
### Schema verification after migrations
|
||||
|
||||
- Post-migration schema verification catches columns that were defined in migrations but silently failed to create (common with PgBouncer transaction-mode poolers).
|
||||
- Self-healing: automatically adds missing columns via ALTER TABLE when detected.
|
||||
- Prevents the "column X does not exist" embed failures that occur when schema version is ahead of actual table state.
|
||||
|
||||
## [0.22.5] - 2026-04-27
|
||||
|
||||
## **Autopilot stops re-importing your whole brain when a commit gets garbage-collected.**
|
||||
## **Cycle reads the per-source `sources.last_commit` anchor instead of the drift-prone global key.**
|
||||
|
||||
`gbrain dream` and the `autopilot-cycle` worker were calling `performSync()` without `sourceId`, so sync read the global `config.sync.last_commit` key. When that commit gets GC'd from git history (a force push, a squash, an `--amend` chain), `git cat-file -t <anchor>` fails, sync concludes "force push happened," and triggers a full reimport of every page. On a 78K-page brain that's ~30 minutes per cycle, the autopilot job hits its timeout, dead-letters, and the next cron tick does it again. Production OpenClaw deployment hit exactly this pattern: every cycle ran the full reimport while the per-source `sources.last_commit` (`00a62e50`) was a valid HEAD ancestor the entire time.
|
||||
|
||||
v0.22.5 threads `sourceId` through the cycle. `runPhaseSync()` now resolves the brain directory against the `sources` table (`SELECT id FROM sources WHERE local_path = $1`) and passes the result to `performSync()`. When a source row matches, sync reads `sources.last_commit` (per-source, always written back on every successful sync). When no row matches (pre-v0.18 brain or never-registered path), it falls through to the global key ... fully backward compatible. Six new regression tests pin the resolver behavior, including the table-missing fallback for old brains and the empty-string-id defensive case.
|
||||
|
||||
### The numbers that matter
|
||||
|
||||
Production behavior on a 78,797-page brain:
|
||||
|
||||
| Metric | Pre-v0.22.5 (master) | v0.22.5 | Δ |
|
||||
|---|---|---|---|
|
||||
| Autopilot cycle wall time (steady state) | 30+ min (then timeout) | <1 sec | -1800x |
|
||||
| Files re-imported per cycle (steady state) | 78,797 | 0 | -78,797 |
|
||||
| `autopilot-cycle` jobs hitting `max_stalled` | every cycle | 0 | -100% |
|
||||
| Cycle phases that consult per-source anchor | 0 | 1 (sync) | +1 |
|
||||
| New regression tests in `test/core/cycle.test.ts` | n/a | 6 | +6 |
|
||||
|
||||
Resolver behavior matrix (every row covered by a test):
|
||||
|
||||
| Scenario | sourceId passed | Anchor read from | Backward compatible |
|
||||
|---|---|---|---|
|
||||
| Sources row matches `brainDir` (current install) | `"default"` | `sources.last_commit` ✅ | Yes |
|
||||
| No sources row (pre-v0.18 brain) | `undefined` | `config.sync.last_commit` | Yes |
|
||||
| `sources` table doesn't exist (very old brain) | `undefined` (catch) | `config.sync.last_commit` | Yes |
|
||||
| Multiple rows share a `local_path` (no UNIQUE) | one of the matching ids (non-deterministic) | the matched row's anchor | Yes |
|
||||
| Empty-string id row | `""` (defensive ... won't happen in practice) | empty-string source row | Yes |
|
||||
|
||||
### What this means for builders
|
||||
|
||||
If your brain has been silently doing a full reimport every autopilot cycle, `gbrain upgrade` plus your next cycle will fix it ... no manual action needed. The fix is mechanical and idempotent. If you've been running with the operational band-aid that copied the per-source anchor to the global key every 5 minutes (the pre-PR workaround), you can take it out after upgrading. Two follow-ups are filed for v0.23: a `UNIQUE` index on `sources.local_path` so duplicate-path resolution is deterministic, and narrowing the resolver's bare `catch` to PostgreSQL's `42P01` (undefined_table) so real DB errors don't get silently swallowed into the global-fallback path.
|
||||
|
||||
## To take advantage of v0.22.5
|
||||
|
||||
`gbrain upgrade` runs `gbrain post-upgrade` which runs `gbrain apply-migrations`. v0.22.5 has no schema migration ... the fix is pure code, no data backfill ... so the upgrade itself is the entire action.
|
||||
|
||||
1. **Upgrade:**
|
||||
```bash
|
||||
gbrain upgrade
|
||||
```
|
||||
|
||||
2. **Verify the next autopilot cycle is fast.** Either let `gbrain autopilot` tick naturally, or run one cycle directly:
|
||||
```bash
|
||||
gbrain dream --phase sync --json | jq '.phases[] | select(.phase == "sync")'
|
||||
```
|
||||
On a brain with a registered source, the sync phase should report incremental status (`up_to_date` or a small added/modified count) and complete in seconds. If it reports thousands of files added/modified on a brain you haven't actually changed, file an issue ... the resolver isn't matching your `brainDir` to a `sources.local_path` (likely a path-normalization mismatch ... see TODO 1 below).
|
||||
|
||||
3. **Optional ... confirm the resolver matched.** The `sources` row used by `gbrain dream` should match your brain directory exactly:
|
||||
```bash
|
||||
gbrain query 'SELECT id, local_path FROM sources' --json
|
||||
```
|
||||
If the path stored in `sources.local_path` differs from the directory `gbrain dream --dir <path>` is invoked with (trailing slash, symlink resolution), v0.22.5 will fall back to the legacy global-key path silently for that source. A future v0.23 fix will normalize both sides; for now you can re-register the source with the canonical absolute path.
|
||||
|
||||
4. **If any step fails or the numbers look wrong,** file an issue: https://github.com/garrytan/gbrain/issues with:
|
||||
- output of `gbrain doctor`
|
||||
- contents of `~/.gbrain/upgrade-errors.jsonl` if it exists
|
||||
- which step broke
|
||||
|
||||
This feedback loop is how the gbrain maintainers find fragile upgrade paths. Thank you.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
**Hotfix.** `src/core/cycle.ts` ... new `resolveSourceForDir(engine, brainDir)` helper queries `SELECT id FROM sources WHERE local_path = $1 LIMIT 1`. `runPhaseSync()` calls it before `performSync()` and threads the result as `sourceId`. Bare `try/catch` swallows missing-table errors so pre-v0.18 brains keep working unchanged. 26 new lines, one file. The fix funnels into the existing `readSyncAnchor()` branching at `src/commands/sync.ts:174-188`, which already chose between per-source and global anchors when given a `sourceId`; the cycle just wasn't passing one.
|
||||
|
||||
**Tests.** 6 new test cases in `test/core/cycle.test.ts` covering every branch of the resolver:
|
||||
- **Test 1** ... seeded `sources` row → `performSync` receives matching `sourceId`.
|
||||
- **Test 2** ... no row → `sourceId=undefined`, falls through to global key.
|
||||
- **Test 3** ... different `brainDir` than registered source → undefined (no cross-match).
|
||||
- **Test 4** ... `sources` table missing (very old brain) → catch returns undefined, sync still runs. Uses a fresh `PGLiteEngine` (not the shared one) because `initSchema()` only re-runs PENDING migrations; `DROP TABLE` on the shared engine would have left it permanently degraded for every subsequent test in the file. Codex review caught this landmine.
|
||||
- **Test 5** ... duplicate `local_path` rows → resolver returns one of the matching ids (non-deterministic; the SQL has no `ORDER BY`). Documents the contract for the v0.23 UNIQUE-constraint follow-up.
|
||||
- **Test 6** ... empty-string id row → resolver propagates `""` (defensive case Codex flagged ... PK prevents NULL but `''` can be inserted).
|
||||
|
||||
The `performSync` mock in `test/core/cycle.test.ts:50-65` was extended to capture `sourceId` alongside the existing `dryRun / noPull / noExtract` opts. The new `describe` block runs after the existing 22 tests; the shared PGLite engine cleanup pattern (`DELETE FROM sources` in `beforeEach`) keeps state from leaking between tests.
|
||||
|
||||
### For contributors
|
||||
|
||||
When threading new options through `runCycle → runPhaseSync → performSync`, extend the `syncCalls` capture shape in `test/core/cycle.test.ts:20` and add per-option assertions to the existing `describe('runCycle — dryRun propagates...')` and `describe('runCycle — phase selection')` blocks. The `cycle.test.ts` shared-engine pattern is fast (~1.4s for 28 tests on PGLite in-memory) but `initSchema()` only runs PENDING migrations ... if your test needs to mutate the schema mid-suite (DROP TABLE, ALTER, etc.), spin up a fresh `PGLiteEngine` and dispose in `finally` instead of touching the shared engine. The v0.22.5 test 4 is the canonical example.
|
||||
|
||||
The bare `catch` in `resolveSourceForDir` is intentional for v0.22.5 because narrowing to a PG-specific error code (`error.code === '42P01'`) requires engine-aware error introspection that the existing PGLite engine doesn't expose uniformly with postgres-engine. v0.23 will add a small `isMissingRelationError(error, engine.kind)` helper to `src/core/utils.ts` and the resolver will rethrow everything else.
|
||||
|
||||
## [0.22.4] - 2026-04-26
|
||||
|
||||
## **Frontmatter-guard ships. Broken brain pages can't hide.**
|
||||
## **Seven validation classes, source-aware audit, doctor subcheck, pre-commit hook, zero resolver warnings.**
|
||||
|
||||
v0.22.4 fixes the seven `gbrain check-resolvable` warnings that lived on master and ships frontmatter-guard as a real feature: a TypeScript validator inside `parseMarkdown(..., {validate:true})`, a top-level `gbrain frontmatter` CLI (`validate` / `audit` / `install-hook`), a new `frontmatter_integrity` subcheck under `gbrain doctor`, and an audit-only migration that surveys every registered source and queues per-source TODOs without mutating brain content. PR #392's aspirational `lib/brain-writer.mjs` is finally written, in TypeScript, on top of the tools gbrain already ships.
|
||||
|
||||
The migration is **audit-only**. It writes a JSON report to `~/.gbrain/migrations/v0.22.4-audit.json` and emits per-source entries to `pending-host-work.jsonl` with the exact fix command. It never silently rewrites your brain pages. The agent reads `skills/migrations/v0.22.4.md` after upgrade, surfaces the counts to you, and runs `gbrain frontmatter validate <source-path> --fix` only with explicit consent. `--fix` writes `.bak` backups for every modified file (the safety contract for non-git brain repos, which `getWorkingTreeStatus` rejects).
|
||||
|
||||
`gbrain frontmatter` is source-aware throughout. `audit [--source <id>]` walks every registered source via `source-resolver.ts` (gbrain has been multi-source since v0.18.0; the single-`brainRoot` model would have shipped a half-broken feature). The CLI, doctor subcheck, and migration phase all call into one shared `scanBrainSources()` ... single source of truth for what counts as malformed.
|
||||
|
||||
### The numbers that matter
|
||||
|
||||
Counted against gbrain's own checked-in `skills/` tree:
|
||||
|
||||
| Metric | Pre-v0.22.4 (master) | v0.22.4 | Δ |
|
||||
|---|---|---|---|
|
||||
| `gbrain check-resolvable` warnings | 7 | 0 | -7 |
|
||||
| Frontmatter validation classes | 3 (in `lint`) | 7 (in `parseMarkdown`) | +4 |
|
||||
| Auto-fixable error codes | 0 | 4 (NULL_BYTES, MISSING_CLOSE, NESTED_QUOTES, SLUG_MISMATCH) | +4 |
|
||||
| Doctor subchecks | 17 | 18 (+frontmatter_integrity) | +1 |
|
||||
| `gbrain frontmatter` subcommands | 0 | 3 (validate, audit, install-hook) | +3 |
|
||||
| Skills in `skills/` | 29 | 30 (+frontmatter-guard) | +1 |
|
||||
| Pre-commit hook helper | none | `gbrain frontmatter install-hook` | ✓ |
|
||||
| Source-aware audit | n/a | walks every registered source | ✓ |
|
||||
|
||||
Frontmatter validation surface (the 7 codes shipped):
|
||||
|
||||
| Code | What it catches | Auto-fix |
|
||||
|---|---|---|
|
||||
| `MISSING_OPEN` | File doesn't start with `---` | No (human review) |
|
||||
| `MISSING_CLOSE` | No closing `---` before first heading | Yes ... inserts `---` |
|
||||
| `YAML_PARSE` | YAML failed to parse | Sometimes |
|
||||
| `SLUG_MISMATCH` | Frontmatter `slug:` differs from path-derived slug | Yes ... removes field |
|
||||
| `NULL_BYTES` | Binary corruption (`\x00`) | Yes ... strips bytes |
|
||||
| `NESTED_QUOTES` | `title: "outer "inner" outer"` shape | Yes ... switches outer to single quotes |
|
||||
| `EMPTY_FRONTMATTER` | Open + close present, nothing meaningful between | No (human review) |
|
||||
|
||||
### What this means for builders
|
||||
|
||||
If you've been ignoring `gbrain check-resolvable` warnings because the messages were misleading (the action message said "Add disambiguation rule in RESOLVER.md OR narrow triggers" ... but only the second branch actually silenced the MECE warning, since the checker doesn't parse RESOLVER.md disambiguation rules), v0.22.4 closes the loop. Trigger overlap is fixed at the frontmatter layer. `enrich/SKILL.md` delegates citation rules to `conventions/quality.md` instead of inlining them. Routing-eval fixtures embed actual trigger keywords. `frontmatter-guard` is registered. `gbrain check-resolvable --json` returns `ok: true, issues: []`.
|
||||
|
||||
If your agent writes brain pages, plumb its writes through `parseMarkdown(content, path, { validate: true, expectedSlug })` (the export is in `gbrain/markdown`) and check the returned `errors` array. The 7-error envelope is stable from v0.22.4 onward. Or call `gbrain frontmatter validate <path> --json` from your script and parse the envelope. For brain repos that ARE git repos, install the pre-commit hook with `gbrain frontmatter install-hook` and stop bad frontmatter at the commit boundary.
|
||||
|
||||
If you maintain a downstream OpenClaw fork, see `docs/UPGRADING_DOWNSTREAM_AGENTS.md` for the v0.22.4 diff pattern. The short version: drop any references to the never-existed `lib/brain-writer.mjs` and replace with `gbrain frontmatter validate` calls.
|
||||
|
||||
## To take advantage of v0.22.4
|
||||
|
||||
`gbrain upgrade` runs `gbrain post-upgrade` which runs `gbrain apply-migrations`. If that chain was interrupted or if `gbrain doctor` reports `frontmatter_integrity` issues:
|
||||
|
||||
1. **Run the orchestrator manually:**
|
||||
```bash
|
||||
gbrain apply-migrations --yes
|
||||
```
|
||||
The `v0.22.4` orchestrator (v0_22_4.ts) runs schema (no-op) → audit → emit-todo. The audit phase writes a per-source JSON report to `~/.gbrain/migrations/v0.22.4-audit.json` and queues one entry per source with issues to `~/.gbrain/migrations/pending-host-work.jsonl`. **It never modifies brain content.**
|
||||
|
||||
2. **Read the audit report:**
|
||||
```bash
|
||||
cat ~/.gbrain/migrations/v0.22.4-audit.json | jq '.errors_by_code, .per_source[].source_id'
|
||||
```
|
||||
|
||||
3. **Fix mechanical issues with explicit consent.** For each source with errors > 0, run:
|
||||
```bash
|
||||
gbrain frontmatter validate <source-path> --fix
|
||||
```
|
||||
This writes `.bak` backups for every modified file. SLUG_MISMATCH errors are surfaced for manual review (gbrain derives slug from path; a mismatch usually means the file was renamed deliberately or the slug field is stale).
|
||||
|
||||
4. **Verify the outcome:**
|
||||
```bash
|
||||
gbrain doctor --json | jq '.checks[] | select(.name == "frontmatter_integrity")'
|
||||
gbrain frontmatter audit --json | jq '.total'
|
||||
gbrain check-resolvable --json | jq '.report.issues | map(select(.severity=="warning" or .severity=="error")) | length'
|
||||
```
|
||||
All three should report 0 issues.
|
||||
|
||||
5. **If any step fails or the numbers look wrong,** file an issue at https://github.com/garrytan/gbrain/issues with:
|
||||
- output of `gbrain doctor`
|
||||
- contents of `~/.gbrain/migrations/v0.22.4-audit.json`
|
||||
- contents of `~/.gbrain/upgrade-errors.jsonl` if it exists
|
||||
- which step broke
|
||||
|
||||
This feedback loop is how the gbrain maintainers find fragile upgrade paths. Thank you.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
**Part A ... `gbrain check-resolvable` reaches 0 warnings.** Drop `"citation audit"` from `skills/maintain/SKILL.md` frontmatter; the trigger lives only on `citation-fixer` now. RESOLVER.md gains a citation-audit disambiguation row pointing both skills so agents still pick the right one. RESOLVER.md broadens query triggers (`"who is"`, `"background on"`, `"notes on"`) and `query/SKILL.md` mirrors them in its frontmatter. `skills/enrich/SKILL.md` replaces the inlined citation rules block with `> **Convention:** see \`skills/conventions/quality.md\`` (the format `extractDelegationTargets` recognizes). Routing-eval fixtures for `citation-fixer` rewritten to embed `"fix citations"` so substring matching passes.
|
||||
|
||||
**Part B ... frontmatter-guard library + CLI + doctor + migration + skill + pre-commit hook.**
|
||||
|
||||
- **`src/core/markdown.ts`** ... `parseMarkdown(content, filePath?, opts?)` gains an opt-in `opts.validate` flag. When true, returns `errors[]` with the seven canonical codes. Existing callers unaffected. Validation logic for all seven codes lives here as the single source of truth.
|
||||
- **`src/commands/lint.ts`** ... frontmatter-rule lint cases delegate to `parseMarkdown(..., {validate:true})`. New rule names: `frontmatter-missing-close`, `frontmatter-yaml-parse`, `frontmatter-null-bytes`, `frontmatter-nested-quotes`, `frontmatter-slug-mismatch`, `frontmatter-empty`. Suppresses MISSING_OPEN to avoid double-reporting with the legacy `no-frontmatter` rule.
|
||||
- **`src/core/brain-writer.ts`** (NEW) ... thin orchestrator (~280 lines). Exports `autoFixFrontmatter`, `writeBrainPage`, `scanBrainSources`. `writeBrainPage` is path-guarded (refuses writes outside `sourcePath`), always writes `<file>.bak` before any in-place mutation. `scanBrainSources` walks every registered source via direct SQL against `sources.local_path`, uses `isSyncable()` from sync.ts as the canonical brain-page filter, blocks symlinks (matches sync's no-symlink policy), and respects `AbortSignal`.
|
||||
- **`src/commands/frontmatter.ts`** (NEW) ... `gbrain frontmatter validate <path> [--json] [--fix] [--dry-run]` and `gbrain frontmatter audit [--source <id>] [--json]`. The `audit` subcommand is read-only; `--fix` only exists on `validate`. CLI handles `--help` without a DB connection.
|
||||
- **`src/commands/frontmatter-install-hook.ts`** (NEW) ... `gbrain frontmatter install-hook [--source <id>] [--force] [--uninstall]`. Writes `.githooks/pre-commit` per source (skips non-git sources with a one-line note), runs `git config core.hooksPath .githooks` if unset, refuses to clobber existing hooks without `--force` (writes `.bak`). The hook script gracefully degrades when `gbrain` is missing on PATH (prints a warning, exits 0 ... doesn't break commits).
|
||||
- **`src/commands/doctor.ts`** ... new `frontmatter_integrity` subcheck calls `scanBrainSources()` and reports per-source counts plus the fix hint. Wraps in a doctor progress phase with heartbeat.
|
||||
- **`src/commands/migrations/v0_22_4.ts`** (NEW) ... audit-only orchestrator with three phases (schema no-op, audit, emit-todo). Idempotent + resumable. Skips cleanly when no sources are registered. Per-source TODO entries reference the dotted-filename migration doc (`skills/migrations/v0.22.4.md`) per the existing `pending-host-work.jsonl` convention.
|
||||
- **`skills/frontmatter-guard/SKILL.md`** (NEW) ... agent-agnostic; routes to `gbrain frontmatter` CLI invocations, drops OpenClaw-specific paths from PR #392's spec. Registered in `skills/manifest.json` and `skills/RESOLVER.md` with substring-matchable triggers.
|
||||
- **`docs/integrations/pre-commit.md`** (NEW) ... recipe doc covering install / bypass / uninstall and downstream-fork notes.
|
||||
- **`docs/UPGRADING_DOWNSTREAM_AGENTS.md`** ... v0.22.4 section with the diff pattern for forks that had inline frontmatter validators.
|
||||
|
||||
**Tests.** 9 new test files / 4 updated test files. Unit coverage on every new module:
|
||||
- `test/markdown-validation.test.ts` (NEW) ... all 7 codes exercised against hand-crafted fixtures.
|
||||
- `test/lint-frontmatter.test.ts` (NEW) ... lint emits findings for each fixable code; double-report suppression verified.
|
||||
- `test/brain-writer.test.ts` (NEW) ... `autoFixFrontmatter` idempotency, `writeBrainPage` path-guard + `.bak` backup, `scanBrainSources` per-source rollup, AbortSignal mid-scan, single-source filter, missing-source-path graceful skip, symlink no-loop.
|
||||
- `test/frontmatter-cli.test.ts` (NEW) ... subprocess `validate / --fix --dry-run / --fix / --json` + recursive directory scan with `isSyncable` filter parity.
|
||||
- `test/frontmatter-install-hook.test.ts` (NEW) ... hook install / overwrite-protection / `--force` / `--uninstall` / silent-refresh on already-installed.
|
||||
- `test/migrations-v0_22_4.test.ts` (NEW) ... orchestrator phase coverage including dotted-filename JSONL contract and idempotent re-emit.
|
||||
- `test/check-resolvable.test.ts` (UPDATE) ... regression guard asserting the actual checked-in `skills/` tree has 0 warnings + 0 errors.
|
||||
- `test/doctor.test.ts` (UPDATE) ... assertion that `frontmatter_integrity` subcheck calls `scanBrainSources` and the fix hint references the right CLI command.
|
||||
- `test/apply-migrations.test.ts` (UPDATE) ... `skippedFuture` arrays extended to include v0.22.4.
|
||||
- `test/migration-orchestrator-v0_21_0.test.ts` (UPDATE) ... relaxed "is the latest" assertion to "is registered with v0.22.4 after it."
|
||||
|
||||
### For contributors
|
||||
|
||||
`brain-writer.ts` is the canonical place to add new frontmatter validation rules. Add the code to `parseMarkdown`'s `collectValidationErrors`, surface the lint rule name in `lint.ts`'s `FRONTMATTER_RULE_NAMES`, decide if it's auto-fixable (add to `FRONTMATTER_FIXABLE`), and write the auto-fix logic in `brain-writer.ts:autoFixFrontmatter`. Tests in `test/markdown-validation.test.ts` + `test/brain-writer.test.ts`. The lint output uses the `frontmatter-<code>` naming convention; CI consumers can target specific rule names in their lint configs.
|
||||
|
||||
`gbrain frontmatter` is wired through `src/cli.ts:handleCliOnly` so `--help` works without a DB connection. The `audit` subcommand instantiates an engine internally via `loadConfig() + createEngine()`. New subcommands of `frontmatter` should follow this pattern: parse flags first, only connect to the engine when the subcommand actually needs DB access.
|
||||
|
||||
The v0.22.4 orchestrator is intentionally audit-only because brain content is too important to silently mutate during `apply-migrations`. Future migrations that need to rewrite brain pages should follow this two-step pattern: write the audit report + queue the fix command, let the agent run the fix with explicit user consent.
|
||||
|
||||
## [0.22.2] - 2026-04-26
|
||||
|
||||
**Worker no longer freezes silently. Restart-on-RSS, cold-start retry, autopilot backpressure.**
|
||||
|
||||
The minions worker has been freezing every few hours in production. RSS climbs from 68 MB at boot to ~15 GB over ~7 hours, the process stops claiming jobs but never crashes (no OOM, no SIGSEGV), the cron keeps enqueuing autopilot-cycle jobs every 5 minutes into a queue nobody is draining, and within 2-3 hours the queue piles up to 28+ waiting jobs. Shell jobs in flight when the worker froze hit `max_stalled` and dead-letter, producing an 18% shell-job failure rate over 24h. The brainstorm caught the root chain ... memory leak, wedged worker, supervisor cold-start race, no backpressure ... and v0.22.2 ships the three in-repo defenses that close the cascade end-to-end while the underlying memory leak gets investigated separately.
|
||||
|
||||
The watchdog is the keystone. The worker now self-terminates when RSS crosses a threshold (default 2048 MB under the supervisor) and the supervisor's exponential-backoff respawn picks up a fresh process. Both per-job AND a 60-second periodic timer check, so the watchdog still fires when every concurrency slot is wedged and zero jobs are completing ... the actual production freeze pattern. On trip, the worker fires `shutdownAbort` (so the shell handler runs its SIGTERM→5s→SIGKILL cleanup on child processes) and aborts every per-job signal (so cooperative handlers bail instead of waiting out the 30s drain). Closes the zombie-shell-children gap a Codex review surfaced.
|
||||
|
||||
Cold-start auth races on container boot are gone. Every CLI command's `connectEngine()` bootstrap retries transient errors (3 attempts, 1s/2s/4s backoff) by default. PgBouncer rejecting the first connect on a freshly-pinged Supabase pooler is the production failure mode that killed autopilot on cold start; the retry handles it transparently. Operators who genuinely want fail-fast on a misconfigured `DATABASE_URL` pass `--no-retry-connect` or set `GBRAIN_NO_RETRY_CONNECT=1`.
|
||||
|
||||
Autopilot stops piling jobs into a dead queue. `autopilot-cycle` submissions now use `maxWaiting: 1` so the v0.19.1 `pg_advisory_xact_lock` coalesce path caps the queue at 1 active + 1 waiting instead of letting it grow unbounded. The 3rd+ submission coalesces and writes a backpressure-audit JSONL line. Combined with the existing per-slot `idempotency_key`, cross-slot pile-ups are bounded.
|
||||
|
||||
### The numbers that matter
|
||||
|
||||
Production data from the 2026-04-25 incident, plus the watchdog defaults:
|
||||
|
||||
| Metric | Before | After (supervised path) |
|
||||
|-----------------------------------------|-----------------|-------------------------|
|
||||
| Waiting-jobs pileup at freeze | 28+ | 2 (capped at 1+1) |
|
||||
| Worker RSS at freeze | 14.8 GB | ~2 GB self-terminate |
|
||||
| Time to detect freeze | hours (manual) | ≤60s (periodic timer) |
|
||||
| Cold-start auth-fail recovery | manual restart | 3 attempts in ~7s |
|
||||
|
||||
Bare `gbrain jobs work` (operators not using the supervisor) keeps current unbounded behavior to preserve workloads with legitimately large embed/import working sets ... pass `--max-rss N` explicitly to enable the watchdog there.
|
||||
|
||||
### What this means for operators
|
||||
|
||||
If you run `gbrain jobs supervisor` (the production-recommended path), `gbrain upgrade` is the only step. The supervisor injects `--max-rss 2048` to its spawned worker by default; hourly watchdog exits look like clean shutdowns to the supervisor's stable-run reset, not crashes. If you run `gbrain autopilot --install`, the autopilot's worker spawn loop now has the same stable-run reset pattern, so a watchdog-driven exit every hour does NOT trip the give-up-after-5-crashes threshold. If your container hits zombie process accumulation, add `--init` to `docker run` or `tini` as PID 1 ... that's a host-side concern, not a gbrain change.
|
||||
|
||||
## To take advantage of v0.22.2
|
||||
|
||||
`gbrain upgrade` should do this automatically. If it didn't, or if `gbrain doctor` warns about a partial migration:
|
||||
|
||||
1. **Run the orchestrator manually:**
|
||||
```bash
|
||||
gbrain apply-migrations --yes
|
||||
```
|
||||
2. **No manual SKILL.md or AGENTS.md edits required.** This release is code-only ... no schema changes, no new skills.
|
||||
3. **Verify the watchdog is wired (Postgres + supervisor path):**
|
||||
```bash
|
||||
gbrain jobs supervisor --json &
|
||||
ps -ef | grep "gbrain jobs work" | grep -- "--max-rss 2048"
|
||||
```
|
||||
You should see the spawned worker child carrying `--max-rss 2048` in its argv.
|
||||
4. **If you supervise via `gbrain autopilot --install`,** the watchdog gets injected automatically. Existing crontab/launchd/systemd installs do not need to be reinstalled ... the autopilot binary picks up the new spawn args on next restart.
|
||||
5. **For hosts hitting zombie process accumulation** (PID-table fills up over weeks): add `--init` to `docker run`, or set `tini` as PID 1 in your Dockerfile. Not a gbrain code change ... operational note.
|
||||
6. **If any step fails or behavior looks off,** please file an issue at https://github.com/garrytan/gbrain/issues with the output of `gbrain doctor` and the contents of `~/.gbrain/upgrade-errors.jsonl` if it exists.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
#### Added
|
||||
|
||||
- `MinionWorkerOpts` gains `maxRssMb`, `getRss`, and `rssCheckInterval` ... watchdog plumbing with a deterministic-test seam for the RSS readback.
|
||||
- `MinionWorker.gracefulShutdown(reason)` ... unified-style shutdown that fires `shutdownAbort` + per-job aborts + `running=false`. Reused by the per-job and periodic-timer check sites.
|
||||
- 60-second periodic RSS check (`rssCheckInterval` default 60_000) running alongside the existing stalled-jobs timer in `start()`. Closes the freeze-with-zero-completions production scenario.
|
||||
- `--max-rss MB` flag on `gbrain jobs work` (no default, opt-in for bare workers) and `gbrain jobs supervisor` (default 2048). `--max-rss 0` disables; `< 256` errors out as a likely GB-vs-MB unit-confusion typo.
|
||||
- `connectWithRetry()` + `isRetryableDbConnectError()` in `src/core/db.ts`. 5-pattern transient-error matcher (auth-failed, connection-refused, db-starting, terminated-unexpectedly, ECONNRESET). Permanent errors (extension-missing, schema conflicts) do NOT retry.
|
||||
- `--no-retry-connect` flag and `GBRAIN_NO_RETRY_CONNECT=1` env var ... operator escape hatch for fail-fast on misconfigured DATABASE_URL.
|
||||
- Autopilot worker spawn now carries `--max-rss 2048` and a stable-run reset window (5 minutes uptime → reset crash counter to 1). Mirrors the supervisor pattern at `supervisor.ts:471-476` so hourly watchdog exits don't kill autopilot after ~5 hours.
|
||||
- `autopilot-cycle` submission passes `maxWaiting: 1` to `queue.add()`. Combined with the existing per-slot `idempotency_key`, this caps cross-slot queue depth at 1 active + 1 waiting.
|
||||
- 11 new tests in `test/minions.test.ts` covering the watchdog (5 cases including the production-freeze-regression case where zero jobs ever complete) and `connectWithRetry` (6 cases including the noRetry opt-out, transient/permanent error distinction, and successful retry).
|
||||
- New supervisor integration test asserting `--max-rss 2048` lands in the spawned worker's argv by default.
|
||||
|
||||
#### Changed
|
||||
|
||||
- `MinionSupervisor` `SupervisorOpts` gains `maxRssMb` (default 2048). The spawn-args builder appends `--max-rss N` when `maxRssMb > 0`.
|
||||
- `connectEngine()` in `src/cli.ts` now wraps `engine.connect()` in `connectWithRetry` by default. Behavior change for cold-start auth races; preserve original fail-fast with `--no-retry-connect` per call site.
|
||||
|
||||
#### Out of scope (follow-ups)
|
||||
|
||||
- The 40 MB/job memory leak itself ... separate investigation needs heap snapshots and a real reproducer. The watchdog removes urgency.
|
||||
- Zombie process reaping via `tini` or `--init` ... Render/Docker host-side configuration, documented above.
|
||||
- Refactoring SIGTERM/SIGINT/watchdog into one `unifiedShutdown(reason)` helper ... right shape long-term, premature for this PR.
|
||||
|
||||
### For contributors
|
||||
|
||||
- The watchdog cleanup path (`gracefulShutdown`) is intentionally co-located with `MinionWorker.stop()`. When a third caller appears (e.g., a future `pause()` method), extracting `unifiedShutdown(reason)` becomes worth the refactor. Until then, three lines is not a DRY emergency.
|
||||
- `isRetryableDbConnectError()` lives in `src/core/db.ts` and owns its own 5-pattern matcher. PR #406 (when it merges) introduces a 13-pattern matcher in `src/core/minions/supervisor.ts`; the right move at that merge is to delete the supervisor's local copy and import from `db.ts` (correct dependency direction, low → high). A follow-up TODO captures this.
|
||||
## [0.22.1] - 2026-04-26
|
||||
|
||||
**Autopilot stops being a noisy neighbor.**
|
||||
|
||||
Five hotfixes shipping together: incremental extract, cooperative cycle abort, supervisor watchdog reconnect, session-level connection timeouts, and server-side embed-stale filtering. The wave's theme is unified: gbrain's overnight maintenance loop was reading too much, ignoring abort signals, and quietly poisoning shared infrastructure when things went wrong. After this release the loop only reads pages that changed, bails cleanly when timeouts fire, and recovers from connection-pool poisoning without manual intervention.
|
||||
|
||||
### For everyone
|
||||
|
||||
These two fixes apply to both PGLite (default install) and Postgres / Supabase users:
|
||||
|
||||
- **#417 incremental extract** — `gbrain dream` cycles no longer re-read every markdown file when only a handful changed. The cycle still walks the directory tree to build the link-resolution set (a fast `readdir` pass), but `readFileSync` runs only on pages sync flagged as added or modified. On a 54,461-page production brain this turned a 10-minute extract phase into a sub-second pass; on a 500-page brain you get the same proportional win.
|
||||
- **#403 cycle abort** — when a cycle phase hits a per-job timeout, `runCycle` now bails at the next phase boundary instead of grinding through extract → embed → orphans while the worker thinks the job is done. A 30-second grace-then-evict safety net in `MinionWorker` frees the slot even if a future handler ignores the abort signal entirely. Cooperative — can't interrupt a phase mid-execution — but prevents the cascade that was wedging workers.
|
||||
|
||||
### For Postgres / Supabase users
|
||||
|
||||
Three fixes that no-op on PGLite (no network, no pooler, no per-connection state):
|
||||
|
||||
- **#406 supervisor watchdog reconnect** — when the connection pool gets poisoned (PgBouncer rotation, Supabase pool bounce), the supervisor's watchdog now detects three consecutive health-check failures and calls `engine.reconnect()` to swap in a fresh pool. Workers crash cleanly on poisoned connections; supervisor catches it within ~3 health-check intervals (~3 minutes) instead of staying degraded until manual restart. Recovery is structural, not per-call magic.
|
||||
- **#363 session timeouts** *(Contributed by @orendi84)* — every Postgres connection now sets `statement_timeout` and `idle_in_transaction_session_timeout` as connection-time startup parameters. An orphaned pgbouncer backend can no longer hold a `RowExclusiveLock` for hours and block schema migrations. Defaults: 5 minutes each. Override per-GUC via `GBRAIN_STATEMENT_TIMEOUT` / `GBRAIN_IDLE_TX_TIMEOUT` / `GBRAIN_CLIENT_CHECK_INTERVAL`. Closes #361.
|
||||
- **#409 embed egress** *(Contributed by @atrevino47)* — `embed --stale` now filters server-side on `embedding IS NULL` instead of pulling every chunk's `vector(1536)` over the wire and discarding the unwanted ones client-side. On a fully-embedded 1.5K-page brain that's the difference between ~76 MB per call and a single `count()` round-trip. With autopilot firing every 5–10 minutes plus a 2-hour cron, one production user blew past Supabase's 5 GB free-tier ceiling at 102 GB used — that pattern is gone now. Two new `BrainEngine` methods (`countStaleChunks`, `listStaleChunks`) plus a consistency fix in `upsertChunks` so when `chunk_text` changes without a new embedding, both `embedding` and `embedded_at` reset to NULL together (no more "embedded_at says yes, embedding says NULL").
|
||||
|
||||
### Production proof point
|
||||
|
||||
The wave was driven by a 54,461-page OpenClaw production deployment where extract took 600+ seconds and the queue stalled at 20–36 waiting jobs (all returning `skipped: cycle_already_running`). All five fixes ran as hotfixes there for 12+ hours stable before this release. The numbers are extreme; the underlying bugs are not.
|
||||
|
||||
### Eng-review tightening
|
||||
|
||||
The original #406 wrapped `executeRaw` in a per-call retry that auto-recovered from connection errors. Eng-review dropped that wrapper as unsound — a SQL-prefix regex isn't a safe idempotence boundary (writable CTEs, side-effecting SELECTs). What ships from #406 is the structural reconnect path, not the per-call retry. Recovery moves up one layer to the supervisor watchdog. See `TODOS.md` for the planned caller-opt-in retry follow-up.
|
||||
|
||||
### Test coverage
|
||||
|
||||
15 new test cases across `test/extract-incremental.test.ts` (new), `test/core/cycle.test.ts`, and `test/connection-resilience.test.ts`:
|
||||
- 8 cases for `#417`: empty/undefined slugs, [a,b]-only reads, deleted-file handling, mode filter, dry-run, BATCH_SIZE flush, full-slug-set resolution.
|
||||
- 4 cases for `#417` + Codex F2: cycle threads `pagesAffected` into extract, full-walk fallback, F2 noExtract gating (full cycle vs sync-only).
|
||||
- 3 cases for D3: `executeRaw` has no per-call retry wrapper, `reconnect()` still exists, supervisor still has 3-strikes path.
|
||||
|
||||
### To take advantage of v0.22.1
|
||||
|
||||
No manual step. PGLite users get the universal fixes automatically on next cycle. Postgres users additionally get session timeouts on the next pool reconnect, server-side stale filtering on the next `embed --stale`, and supervisor reconnect on the next pool poisoning event.
|
||||
|
||||
```bash
|
||||
gbrain upgrade
|
||||
gbrain doctor # verify (optional)
|
||||
```
|
||||
|
||||
If anything looks wrong post-upgrade, file an issue: https://github.com/garrytan/gbrain/issues with `gbrain doctor` output.
|
||||
|
||||
## [0.22.0] - 2026-04-25
|
||||
|
||||
**Search stops getting swamped by chat logs. Curated pages win by default.**
|
||||
|
||||
For the last few releases, multi-word topic queries against a real brain returned chat-log pages at #1 and #2 because chat pages are 50KB and contain mentions of every topic. The actual article you wrote about the topic ranked #5. v0.22.0 fixes that at the SQL layer ... ranking is now source-aware, curated directories outrank bulk content, and bookkeeping directories like `test/` and `archive/` never enter the candidate set.
|
||||
|
||||
The fix layers on top of v0.21.0's Cathedral II chunk-grain FTS and two-pass retrieval. Different mechanism, additive effect. Chat pages get dampened at the chunk-rank stage; curated content gets boosted; the two-pass walk and source-boost both run in the same pipeline. Temporal queries (`when`, `last week`, `YYYY-MM`) bypass the gate entirely so date-framed chat lookups still work. Two new env vars (`GBRAIN_SOURCE_BOOST`, `GBRAIN_SEARCH_EXCLUDE`) tune per-deployment. `unset` them to revert to v0.21.0 ranking exactly.
|
||||
|
||||
Two SearchOpts additions plumb hard-exclude through the API: `exclude_slug_prefixes` (additive over defaults + env) and `include_slug_prefixes` (subtractive opt-back-in). The four default hard-excludes (`test/`, `archive/`, `attachments/`, `.raw/`) were silently polluting search results before.
|
||||
|
||||
### The numbers that matter
|
||||
|
||||
A new BrainBench category — **Cat 13b: Source Swamp Resistance** — ships in the sibling [gbrain-evals](https://github.com/garrytan/gbrain-evals) repo. The corpus is 20 pages: 10 short opinionated `originals/` pages and 10 long `wintermute/chat/` dumps that mention the same multi-word phrases at higher per-byte density. 30 hand-curated queries assert the curated page wins.
|
||||
|
||||
| gbrain version | Top-1 hit | Top-3 hit | Swamp@top |
|
||||
|--------------------------------------|-----------|-----------|-----------|
|
||||
| v0.20.4 (pre-Cathedral II) | 90.0% | 100.0% | 10.0% |
|
||||
| v0.21.0 (Cathedral II — two-pass) | 90.0% | 100.0% | 10.0% |
|
||||
| **v0.22.0 (this release)** | **93.3%** | **100.0%** | **6.7%** |
|
||||
|
||||
v0.21.0's two-pass retrieval is orthogonal to source-swamp resistance — it's about call-graph edges and parent-scope chunking, which doesn't reach the directory-level ranking signal that source-boost provides. v0.22.0 adds +3.3pts top-1 and -3.3pts swamp on top of v0.21.0.
|
||||
|
||||
The world-v1 corpus (BrainBench Cats 1+2 retrieval, 145 relational queries) is unchanged at P@5 49.1% / R@5 97.9% — every existing benchmark axis stays put within ±2pp tolerance.
|
||||
|
||||
### What this means for you
|
||||
|
||||
If your brain's biggest directories are chat dumps, daily logs, or X archives, search just got dramatically better for the topic queries you actually run. If you depend on chat surfacing for date-framed questions ("what did we discuss last week"), nothing changed ... the intent classifier routes those to `detail=high` which bypasses source-boost. If you want a different boost map, set `GBRAIN_SOURCE_BOOST=originals/:1.8,wintermute/chat/:0.3` and ship.
|
||||
|
||||
## To take advantage of v0.22.0
|
||||
|
||||
`gbrain upgrade` should do this automatically. No DB migration is needed ... the change is purely a SQL ranking refactor on existing tables.
|
||||
|
||||
1. **No manual migration step required.** The new ranking is on by default. Defaults are tuned for a brain with the canonical `originals/`, `concepts/`, `writing/`, `meetings/`, `daily/`, `media/x/`, `wintermute/chat/` shape.
|
||||
2. **Tune for your brain (optional):**
|
||||
```bash
|
||||
# Stronger originals boost, harder chat dampening
|
||||
export GBRAIN_SOURCE_BOOST="originals/:1.8,wintermute/chat/:0.3"
|
||||
# Add a directory to the hard-exclude list
|
||||
export GBRAIN_SEARCH_EXCLUDE="scratch/,private/"
|
||||
```
|
||||
3. **Verify the outcome:**
|
||||
```bash
|
||||
gbrain search "<a multi-word topic phrase from your brain>"
|
||||
# Expect: curated content (originals/, concepts/, writing/) at the top.
|
||||
gbrain search "<phrase>" --detail high
|
||||
# Expect: source-boost bypassed; chat pages allowed back.
|
||||
```
|
||||
4. **Rollback one-liner** if something looks off:
|
||||
```bash
|
||||
unset GBRAIN_SOURCE_BOOST GBRAIN_SEARCH_EXCLUDE
|
||||
```
|
||||
Reverts ranking to v0.21.0 behavior exactly.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
#### Source-aware retrieval
|
||||
|
||||
- New module `src/core/search/source-boost.ts` ships the default boost map (`originals/` 1.5, `concepts/` 1.3, `writing/` 1.4, `people/companies/deals/` 1.2, `daily/` 0.8, `media/x/` 0.7, `wintermute/chat/` 0.5) and the four default hard-exclude prefixes (`test/`, `archive/`, `attachments/`, `.raw/`). Both knobs override via env (`GBRAIN_SOURCE_BOOST`, `GBRAIN_SEARCH_EXCLUDE`) or per-call SearchOpts.
|
||||
- New module `src/core/search/sql-ranking.ts` is a pair of pure SQL-fragment builders shared between Postgres and PGLite engines. `buildSourceFactorCase` emits a longest-prefix-match CASE expression and returns literal `'1.0'` when `detail === 'high'` so temporal queries bypass source-boost. `buildHardExcludeClause` emits `NOT (col LIKE 'p1%' OR col LIKE 'p2%')` ... OR-chain wrapped in NOT, never `NOT LIKE ALL/ANY` (those don't express set-exclusion). LIKE meta-character escape covers `%`, `_`, AND `\` (backslash matters because it's Postgres LIKE's default escape char). Single-quote doubling renders SQL-injection-style inputs inert.
|
||||
- `src/core/postgres-engine.ts` and `src/core/pglite-engine.ts` ... three methods wired: `searchKeyword` (chunk-grain CTE → DISTINCT ON page dedup, multiplies ts_rank by source-factor), `searchKeywordChunks` (the chunk-grain anchor primitive used by Cathedral II two-pass retrieval, also gets source-boost so the anchor pool is dampened on chat dirs), and `searchVector` (becomes a two-stage CTE: pure-distance HNSW inner ORDER BY, source-boost re-rank in outer SELECT, innerLimit scales with offset to preserve pagination).
|
||||
- `src/core/types.ts` ... SearchOpts gains two fields: `exclude_slug_prefixes?: string[]` (additive over defaults + env) and `include_slug_prefixes?: string[]` (subtractive opt-back-in).
|
||||
|
||||
#### Tests
|
||||
|
||||
- `test/sql-ranking.test.ts` ... 39 unit cases covering longest-prefix-match, detail=high temporal-bypass, three-meta-char LIKE escape, single-quote SQL-literal doubling, env-var parsing, resolver merge semantics.
|
||||
- `test/e2e/search-swamp.test.ts` ... reproduces the headline case in PGLite. Curated article competes with two chat pages stuffed with the same multi-word phrase. Asserts article wins both keyword and vector ranking, detail=high lets chat re-surface, source_id passes through two-stage CTE.
|
||||
- `test/e2e/search-exclude.test.ts` ... verifies test/ + archive/ pages hidden by default, include_slug_prefixes opts back in, exclude_slug_prefixes adds to defaults.
|
||||
- `test/e2e/engine-parity.test.ts` ... Postgres ↔ PGLite top-result + result-set parity for both search methods plus a hard-exclude parity case. Skips gracefully when DATABASE_URL is unset.
|
||||
|
||||
#### Won't break what was already working
|
||||
|
||||
The change is additive at the SQL layer; no `hybrid.ts`, `intent.ts`, `dedup.ts`, `expansion.ts`, `two-pass.ts`, or operations-layer changes. RRF fusion, compiled-truth boost, backlink boost, multi-query expansion, source-aware dedup, and v0.21.0's Cathedral II two-pass retrieval all run unchanged downstream of the new ranking. The `sql.begin` + `SET LOCAL statement_timeout` v0.19 wrap is preserved (transaction-scoped GUC; bare SET would leak onto pooled connections, documented DoS vector). RLS-enabled brains still work because both inner and outer CTE SELECTs are subject to row-level policies.
|
||||
|
||||
### For contributors
|
||||
|
||||
- The two new helpers are pure functions with explicit params and zero engine dependencies. Both engines call them to build identical SQL. Useful pattern for any future SQL-side ranking signal that needs to land in both Postgres and PGLite.
|
||||
- The two-stage CTE pattern (HNSW-safe pure-distance inner ORDER BY, re-rank in outer SELECT) is the right shape for any future per-prefix or per-page boost in vector search. Folding extra factors into the outer ORDER BY keeps the index usable.
|
||||
- BrainBench Cat 13b lives in [gbrain-evals](https://github.com/garrytan/gbrain-evals) on `feat/cat13b-source-swamp` ... 20-page corpus + 30 hand-curated queries. Companion PR.
|
||||
|
||||
## [0.21.0] - 2026-04-25
|
||||
|
||||
## **Your brain walks the code graph now.**
|
||||
|
||||
@@ -25,30 +25,24 @@ strict behavior when unset.
|
||||
- `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`. `OperationContext.remote` flags untrusted callers.
|
||||
- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports.
|
||||
- `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`)
|
||||
- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for the specific forward-referenced state the embedded schema blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target table) and adds only what's missing. Closes the upgrade-wedge bug class that bit users 10+ times across 6 schema versions over 2 years (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396). No-op on fresh installs and modern brains.
|
||||
- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly.
|
||||
- `src/core/pglite-schema.ts` — PGLite-specific DDL (pgvector, pg_trgm, triggers)
|
||||
- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. v0.22.0: `searchKeyword`, `searchKeywordChunks`, and `searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude clause from `src/core/search/sql-ranking.ts`. `searchVector` switches to a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in the outer SELECT) and carries `p.source_id` through inner→outer for v0.18 multi-source callers. v0.22.1 (#406): `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures). `executeRaw` is a single-statement passthrough — no per-call retry (D3 dropped that as unsound for non-idempotent statements; recovery is supervisor-driven). v0.22.1 (#363, contributed by @orendi84): `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. v0.22.1 (#409, contributed by @atrevino47): `countStaleChunks()` + `listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale`, eliminating ~76 MB/call client-side pull on a fully-embedded brain; `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding (consistency). As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL on the same forward-reference probe set as the PGLite engine, so old Postgres brains pinned at v0.13/v0.18/v0.19 walk forward cleanly instead of wedging on `column "..." does not exist`.
|
||||
- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query.
|
||||
- `src/core/utils.ts` — Shared SQL utilities extracted from postgres-engine.ts. Exports `parseEmbedding(value)` (throws on unknown input, used by migration + ingest paths where data integrity matters) and as of v0.12.3 `tryParseEmbedding(value)` (returns `null` + warns once per process, used by search/rescore paths where availability matters more than strictness).
|
||||
- `src/core/db.ts` — Connection management, schema initialization. v0.22.1 (#363, contributed by @orendi84): `resolveSessionTimeouts()` returns `statement_timeout` + `idle_in_transaction_session_timeout` (defaults: 5min each, env-overridable via `GBRAIN_STATEMENT_TIMEOUT` / `GBRAIN_IDLE_TX_TIMEOUT` / `GBRAIN_CLIENT_CHECK_INTERVAL`). Both `connect()` (module singleton) and `PostgresEngine.connect()` (worker pool) consume the result via postgres.js's `connection` option, sending GUCs as startup parameters that survive PgBouncer transaction mode (unlike the prior `setSessionDefaults` post-pool SET, kept as a back-compat no-op shim).
|
||||
- `src/core/db.ts` — Connection management, schema initialization
|
||||
- `src/commands/migrate-engine.ts` — Bidirectional engine migration (`gbrain migrate --to supabase/pglite`)
|
||||
- `src/core/import-file.ts` — importFromFile + importFromContent (chunk + embed + tags)
|
||||
- `src/core/sync.ts` — Pure sync functions (manifest parsing, filtering, slug conversion). v0.22.12 (#500, foundation by @wintermute via #501): `classifyErrorCode(errorMsg)` regex-based classifier with 12 codes (`SLUG_MISMATCH`, `YAML_PARSE`, `YAML_DUPLICATE_KEY`, `MISSING_OPEN`, `MISSING_CLOSE`, `NESTED_QUOTES`, `EMPTY_FRONTMATTER`, `NULL_BYTES`, `INVALID_UTF8`, `STATEMENT_TIMEOUT`, `FILE_TOO_LARGE`, `SYMLINK_NOT_ALLOWED`) plus `UNKNOWN` fallback. `summarizeFailuresByCode(failures)` returns sorted `[{code, count}]`. `code?` optional field on `SyncFailure`; backfilled at ack time on pre-v0.22.12 entries. `acknowledgeSyncFailures()` returns `AcknowledgeResult { count, summary }`. Three regexes (`MISSING_OPEN`, `MISSING_CLOSE`, `EMPTY_FRONTMATTER`) broadened to match actual `markdown.ts:159-244` validator message strings, not just the literal code-name prefix. `FILE_TOO_LARGE` covers all three production size sites in `import-file.ts:199, 352, 401`; `SYMLINK_NOT_ALLOWED` covers the rejection at `:347`. Closes the silent-skip pattern that motivated #500.
|
||||
- `src/core/sync.ts` — Pure sync functions (manifest parsing, filtering, slug conversion)
|
||||
- `src/core/storage.ts` — Pluggable storage interface (S3, Supabase Storage, local)
|
||||
- `src/core/storage-config.ts` (v0.22.11) — Storage tiering: `loadStorageConfig` reads `gbrain.yml`, normalizes deprecated keys (`git_tracked` / `supabase_only`) to canonical (`db_tracked` / `db_only`) with once-per-process deprecation warning, and runs `normalizeAndValidateStorageConfig` (auto-fixes missing trailing `/`, throws `StorageConfigError` on tier overlap). Path-segment matcher: `media/x/` does NOT match `media/xerox/foo`. Replaces gray-matter (broken on delimiter-less YAML) with a dedicated parser for the `gbrain.yml` shape.
|
||||
- `src/core/disk-walk.ts` (v0.22.11) — `walkBrainRepo(repoPath)` returns `Map<slug, {size, mtimeMs}>` from one recursive `readdirSync`. Skips dot-dirs, `node_modules`, non-`.md` files. Used by `gbrain storage status` to replace per-page `existsSync + statSync` (~400K syscalls on 200K-page brains → tens).
|
||||
- `src/commands/storage.ts` (v0.22.11) — `gbrain storage status [--repo P] [--json]`. Split into pure data (`getStorageStatus`) + JSON formatter + human formatter (ASCII-only per D10) matching the `orphans.ts` pattern. `PageCountsByTier` and `DiskUsageByTier` are distinct nominal types so swaps fail at compile time.
|
||||
- `gbrain.yml` (brain repo root, v0.22.11) — Optional storage tiering config. Top-level `storage:` section with `db_tracked:` and `db_only:` array-valued keys. `gbrain sync` auto-manages `.gitignore` for `db_only` paths on successful sync (skips on dry-run, blocked-by-failures, submodule context, or `GBRAIN_NO_GITIGNORE=1`). `gbrain export --restore-only [--repo P] [--type T] [--slug-prefix S]` repopulates missing `db_only` files from the database.
|
||||
- `src/core/supabase-admin.ts` — Supabase admin API (project discovery, pgvector check)
|
||||
- `src/core/file-resolver.ts` — File resolution with fallback chain (local -> .redirect.yaml -> .redirect -> .supabase)
|
||||
- `src/core/chunkers/` — 3-tier chunking (recursive, semantic, LLM-guided). v0.19.0 adds `code.ts` — tree-sitter-based semantic chunker for 29 languages with embedded-asset WASMs (`src/assets/wasm/`), `@dqbd/tiktoken` cl100k_base tokenizer, small-sibling merging. `CHUNKER_VERSION` constant folded into `importCodeFile`'s `content_hash` so chunker shape changes force clean re-chunks across releases.
|
||||
- `src/core/errors.ts` (v0.19.0) — `StructuredAgentError` + `buildError` + `serializeError`. Every new v0.19.0 agent-facing surface (code-def, code-refs, usage errors) uses this envelope; matches v0.17.0 `CycleReport.PhaseResult.error` shape.
|
||||
- `src/assets/wasm/` (v0.19.0) — 36 tree-sitter grammar WASMs + tree-sitter runtime. Committed to the repo so `bun --compile` embeds them deterministically via `import path from ... with { type: 'file' }`. The CI guard `scripts/check-wasm-embedded.sh` fails the build if the compiled binary ever silently falls through to recursive chunks.
|
||||
- `src/commands/code-def.ts` + `src/commands/code-refs.ts` (v0.19.0) — symbol definition + references lookup. Query `content_chunks.symbol_name` or chunk_text ILIKE with `page_kind='code'` filter. Auto-JSON when stdout is not a TTY (gh-CLI convention). Bypass the standard `searchKeyword` `DISTINCT ON (slug)` collapse so multiple call-sites from the same file surface.
|
||||
- `src/core/search/` — Hybrid search: vector + keyword + RRF + multi-query expansion + dedup. As of v0.22.0, `searchKeyword` / `searchKeywordChunks` / `searchVector` apply source-aware ranking at the SQL layer (curated content like `originals/`, `concepts/`, `writing/` outranks bulk content like `wintermute/chat/`, `daily/`, `media/x/`). `searchVector` uses a two-stage CTE so source-boost re-ranking doesn't kill the HNSW index. Hard-exclude prefixes (`test/`, `archive/`, `attachments/`, `.raw/` by default) filter at retrieval, not post-rank. Both gates honor `detail !== 'high'` so temporal queries surface chat pages normally.
|
||||
- `src/core/search/` — Hybrid search: vector + keyword + RRF + multi-query expansion + dedup
|
||||
- `src/core/search/intent.ts` — Query intent classifier (entity/temporal/event/general → auto-selects detail level)
|
||||
- `src/core/search/eval.ts` — Retrieval eval harness: P@k, R@k, MRR, nDCG@k metrics + runEval() orchestrator
|
||||
- `src/core/search/source-boost.ts` (v0.22.0) — Source-type boost map keyed by slug prefix. `DEFAULT_SOURCE_BOOSTS` (originals/ 1.5, concepts/ 1.3, writing/ 1.4, people/companies/deals/ 1.2, daily/ 0.8, media/x/ 0.7, wintermute/chat/ 0.5) and `DEFAULT_HARD_EXCLUDES` (test/, archive/, attachments/, .raw/). `parseSourceBoostEnv` / `parseHardExcludesEnv` parse comma-separated `prefix:factor` pairs from `GBRAIN_SOURCE_BOOST` / `GBRAIN_SEARCH_EXCLUDE` env vars. `resolveBoostMap` and `resolveHardExcludes` merge defaults + env + caller `SearchOpts.exclude_slug_prefixes`/`include_slug_prefixes`.
|
||||
- `src/core/search/sql-ranking.ts` (v0.22.0) — Pure SQL string builders. `buildSourceFactorCase(slugColumn, boostMap, detail)` emits a CASE expression with longest-prefix-match wins (returns literal `'1.0'` when `detail === 'high'` for temporal-bypass parity with COMPILED_TRUTH_BOOST). `buildHardExcludeClause(slugColumn, prefixes)` emits `NOT (col LIKE 'p1%' OR col LIKE 'p2%')` — OR-chain wrapped in NOT, NOT `NOT LIKE ALL/ANY` (those quantifiers don't express set-exclusion). LIKE meta-character escape covers all three of `%`, `_`, AND `\` (backslash matters because it's Postgres LIKE's default escape char). Single-quote doubling on SQL string literals so injection-style inputs are inert text.
|
||||
- `src/commands/eval.ts` — `gbrain eval` command: single-run table + A/B config comparison
|
||||
- `src/core/embedding.ts` — OpenAI text-embedding-3-large, batch, retry, backoff
|
||||
- `src/core/check-resolvable.ts` — Resolver validation: reachability, MECE overlap, DRY checks, structured fix objects. v0.14.1: `CROSS_CUTTING_PATTERNS.conventions` is an array (notability gate accepts both `conventions/quality.md` and `_brain-filing-rules.md`). New `extractDelegationTargets()` parses `> **Convention:**`, `> **Filing rule:**`, and inline backtick references. DRY suppression is proximity-based via `DRY_PROXIMITY_LINES = 40`.
|
||||
@@ -67,14 +61,12 @@ strict behavior when unset.
|
||||
- `src/core/transcription.ts` — Audio transcription: Groq Whisper (default), OpenAI fallback, ffmpeg segmentation for >25MB
|
||||
- `src/core/enrichment-service.ts` — Global enrichment service: entity slug generation, tier auto-escalation, batch throttling
|
||||
- `src/core/data-research.ts` — Recipe validation, field extraction (MRR/ARR regex), dedup, tracker parsing, HTML stripping
|
||||
- `src/commands/embed.ts` — `gbrain embed [--stale|--all] [--slugs ...]`. v0.22.1 (#409, contributed by @atrevino47): `--stale` path now starts with `engine.countStaleChunks()` (single SELECT count(*) WHERE embedding IS NULL, ~50 bytes wire). On a fully-embedded brain that's a 1-line short-circuit — no further reads. When stale chunks exist, `engine.listStaleChunks()` returns just the chunks needing embeddings (slug + chunk_index + chunk_text + metadata, no `vector(1536)` payload). Caller groups by slug, embeds via OpenAI, re-upserts via `upsertChunks`. Replaces the prior page-walk that pulled every chunk's embedding column over the wire and discarded most.
|
||||
- `src/commands/extract.ts` — `gbrain extract links|timeline|all [--source fs|db]`: batch link/timeline extraction. fs walks markdown files, db walks pages from the engine (mutation-immune snapshot iteration; use this for live brains with no local checkout). As of v0.12.1 there is no in-memory dedup pre-load — candidates are buffered 100 at a time and flushed via `addLinksBatch` / `addTimelineEntriesBatch`; `ON CONFLICT DO NOTHING` enforces uniqueness at the DB layer, and the `created` counter returns real rows inserted (truthful on re-runs). v0.22.1 (#417): `ExtractOpts.slugs?: string[]` enables incremental extract — when set, `extractForSlugs()` reads ONLY those slugs' files (single combined links+timeline pass) instead of the full directory walk. CLI `gbrain extract` keeps full-walk behavior; the cycle path threads sync's `pagesAffected` through. `walkMarkdownFiles(brainDir)` still runs at line 455 to build `allSlugs` for link resolution — see `TODOS.md` for replacing it with `engine.getAllSlugs()`.
|
||||
- `src/commands/extract.ts` — `gbrain extract links|timeline|all [--source fs|db]`: batch link/timeline extraction. fs walks markdown files, db walks pages from the engine (mutation-immune snapshot iteration; use this for live brains with no local checkout). As of v0.12.1 there is no in-memory dedup pre-load — candidates are buffered 100 at a time and flushed via `addLinksBatch` / `addTimelineEntriesBatch`; `ON CONFLICT DO NOTHING` enforces uniqueness at the DB layer, and the `created` counter returns real rows inserted (truthful on re-runs).
|
||||
- `src/commands/graph-query.ts` — `gbrain graph-query <slug> [--type T] [--depth N] [--direction in|out|both]`: typed-edge relationship traversal (renders indented tree)
|
||||
- `src/core/link-extraction.ts` — shared library for the v0.12.0 graph layer. extractEntityRefs (canonical, replaces backlinks.ts duplicate) matches both `[Name](people/slug)` markdown links and Obsidian `[[people/slug|Name]]` wikilinks as of v0.12.3. extractPageLinks, inferLinkType heuristics (attended/works_at/invested_in/founded/advises/source/mentions), parseTimelineEntries, isAutoLinkEnabled config helper. `DIR_PATTERN` covers `people`, `companies`, `deals`, `topics`, `concepts`, `projects`, `entities`, `tech`, `finance`, `personal`, `openclaw`. Used by extract.ts, operations.ts auto-link post-hook, and backlinks.ts.
|
||||
- `src/core/minions/` — Minions job queue: BullMQ-inspired, Postgres-native (queue, worker, backoff, types, protected-names, quiet-hours, stagger, handlers/shell).
|
||||
- `src/core/minions/queue.ts` — MinionQueue class (submit, claim, complete, fail, stall detection, parent-child, depth/child-cap, per-job timeouts, cascade-kill, attachments, idempotency keys, child_done inbox, removeOnComplete/Fail). `add()` takes a 4th `trusted` arg (separate from `opts` to prevent spread leakage); protected names in `PROTECTED_JOB_NAMES` require `{allowProtectedSubmit: true}` and the check runs trim-normalized (whitespace-bypass safe). v0.14.1 #219: `add()` plumbs `max_stalled` through with a `[1, 100]` clamp; omitted values let the schema DEFAULT (5) kick in. v0.19.0: `handleWallClockTimeouts(lockDurationMs)` is Layer 3 kill shot for jobs where `FOR UPDATE SKIP LOCKED` stall detection and the timeout sweep both fail to evict (wedged worker holding a row lock via a pending transaction). v0.19.1: `maxWaiting` coalesce path now uses `pg_advisory_xact_lock` keyed on `(name, queue)` to serialize concurrent submits for the same key, and filters on `queue` in addition to `name` so cross-queue same-name jobs don't suppress each other.
|
||||
- `src/core/minions/worker.ts` — MinionWorker class (handler registry, lock renewal, graceful shutdown, timeout safety net). v0.14.0 abort-path fix: aborted jobs now call `failJob` with reason (`timeout`/`cancel`/`lock-lost`/`shutdown`) instead of returning silently. `shutdownAbort` (instance field) fires on process SIGTERM/SIGINT and propagates to `ctx.shutdownSignal` — shell handler listens to it; non-shell handlers don't. v0.22.1 (#403): per-job timeout fires `abort.abort(new Error('timeout'))` then a 30-second grace-then-evict safety net force-evicts the job from `inFlight` and marks it dead in DB if the handler ignores the abort signal — frees the slot even when a handler wedges (the 98-waiting-0-active prod incident driver).
|
||||
- `src/core/minions/supervisor.ts` — MinionSupervisor process manager. Spawns `gbrain jobs work` as a child, restarts on crash with exponential backoff, periodic health check. v0.22.1 (#406): `consecutiveHealthFailures` counter; on 3 consecutive failures emits `health_warn` with `reason: 'db_connection_degraded'` and calls `engine.reconnect()` to swap in a fresh pool, then resets the counter. Worker exit classifier emits `likely_cause` field on `worker_exited` events: `oom_or_external_kill` (SIGKILL), `graceful_shutdown` (SIGTERM), `runtime_error` (code 1), `clean_exit` (code 0), `unknown`.
|
||||
- `src/core/minions/worker.ts` — MinionWorker class (handler registry, lock renewal, graceful shutdown, timeout safety net). v0.14.0 abort-path fix: aborted jobs now call `failJob` with reason (`timeout`/`cancel`/`lock-lost`/`shutdown`) instead of returning silently. `shutdownAbort` (instance field) fires on process SIGTERM/SIGINT and propagates to `ctx.shutdownSignal` — shell handler listens to it; non-shell handlers don't.
|
||||
- `src/core/minions/types.ts` — `MinionJobInput` + `MinionJobStatus` + handler context types. `MinionJobInput.max_stalled` (new in v0.14.1) is optional; omitted values let the schema DEFAULT (5) kick in, provided values are clamped to `[1, 100]`.
|
||||
- `src/core/minions/protected-names.ts` — side-effect-free constant module exporting `PROTECTED_JOB_NAMES` + `isProtectedJobName()`. Kept pure so queue core can import without loading handler modules.
|
||||
- `src/core/minions/handlers/shell.ts` — `shell` job handler. Spawns `/bin/sh -c cmd` (absolute path, PATH-override-safe) or `argv[0] argv[1..]` (no shell). Env allowlist: `PATH, HOME, USER, LANG, TZ, NODE_ENV` + caller `env:` overrides. UTF-8-safe stdout/stderr tail via `string_decoder.StringDecoder`. Abort (either `ctx.signal` or `ctx.shutdownSignal`) fires SIGTERM → 5s grace → SIGKILL on child. Requires `GBRAIN_ALLOW_SHELL_JOBS=1` on worker (gated by `registerBuiltinHandlers`).
|
||||
@@ -92,27 +84,20 @@ strict behavior when unset.
|
||||
- `src/core/minions/attachments.ts` — Attachment validation (path traversal, null byte, oversize, base64, duplicate detection)
|
||||
- `src/commands/agent.ts` (v0.16) — `gbrain agent run <prompt> [flags]` CLI. Submits `subagent` (or N children + 1 aggregator) under `{allowProtectedSubmit: true}`. Single-entry `--fanout-manifest` short-circuits. Children get `on_child_fail: 'continue'` + `max_stalled: 3`. `--follow` is the default on TTY; streams logs + polls `waitForCompletion` in parallel. Ctrl-C detaches, does not cancel.
|
||||
- `src/commands/agent-logs.ts` (v0.16) — `gbrain agent logs <job> [--follow] [--since]`. Merges JSONL heartbeat audit + `subagent_messages` into a chronological timeline. `parseSince` accepts ISO-8601 or relative (`5m`, `1h`, `2d`). Transcript tail renders only for terminal jobs.
|
||||
- `src/commands/jobs.ts` — `gbrain jobs` CLI subcommands + `gbrain jobs work` daemon. v0.13.1 surfaces the full `MinionJobInput` retry/backoff/timeout/idempotency surface as first-class CLI flags on `jobs submit`: `--max-stalled`, `--backoff-type fixed|exponential`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key`. `jobs smoke --sigkill-rescue` is the opt-in regression guard for #219. v0.16 wires `registerBuiltinHandlers` to always register `subagent` + `subagent_aggregator` (no env flag — `ANTHROPIC_API_KEY` is the natural cost gate, trust is via `PROTECTED_JOB_NAMES`) and loads `GBRAIN_PLUGIN_PATH` plugins at worker startup with a loud startup-line per plugin. `shell` handler still gated by `GBRAIN_ALLOW_SHELL_JOBS=1` (RCE surface, separate concern). v0.22.10 (#521): the `autopilot-cycle` handler now forwards `job.data.phases` to `runCycle` (was previously discarded — caller-supplied phase selection silently became a full cycle). Phases are validated against `ALL_PHASES` from `src/core/cycle.ts`; invalid names are filtered out and an empty/missing array falls back to the default 6-phase cycle. v0.22.13 (PR #490 CODEX-1+CODEX-4): `sync` handler now resolves `sourceId` at entry by looking up `sources.local_path` (mirrors `cycle.ts:480`'s autopilot fix from PR #475) so multi-source brains read the per-source `last_commit` anchor instead of the global config key. Concurrency routed through the shared `autoConcurrency()` policy in `src/core/sync-concurrency.ts` instead of the prior hardcoded `4`; PGLite stays serial. `noEmbed` default is `true` (embed is a separate job — submit `gbrain embed --stale` after sync, or rely on the autopilot cycle's embed phase).
|
||||
- `src/commands/jobs.ts` — `gbrain jobs` CLI subcommands + `gbrain jobs work` daemon. v0.13.1 surfaces the full `MinionJobInput` retry/backoff/timeout/idempotency surface as first-class CLI flags on `jobs submit`: `--max-stalled`, `--backoff-type fixed|exponential`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key`. `jobs smoke --sigkill-rescue` is the opt-in regression guard for #219. v0.16 wires `registerBuiltinHandlers` to always register `subagent` + `subagent_aggregator` (no env flag — `ANTHROPIC_API_KEY` is the natural cost gate, trust is via `PROTECTED_JOB_NAMES`) and loads `GBRAIN_PLUGIN_PATH` plugins at worker startup with a loud startup-line per plugin. `shell` handler still gated by `GBRAIN_ALLOW_SHELL_JOBS=1` (RCE surface, separate concern).
|
||||
- `src/commands/features.ts` — `gbrain features --json --auto-fix`: usage scan + feature adoption salesman
|
||||
- `src/commands/autopilot.ts` — `gbrain autopilot --install`: self-maintaining brain daemon (sync+extract+embed)
|
||||
- `src/mcp/server.ts` — MCP stdio server (generated from operations). v0.22.7: tool-call handler delegates to `dispatchToolCall` from `src/mcp/dispatch.ts` so stdio + HTTP transports share one validation, context-build, and error-format path.
|
||||
- `src/mcp/dispatch.ts` (v0.22.7) — Shared tool-call dispatch consumed by both stdio (`server.ts`) and HTTP (`http-transport.ts`). Exports `dispatchToolCall(engine, name, params, opts)`, `buildOperationContext(engine, params, opts)`, and `validateParams(op, params)`. Single source of truth for `(ctx, params)` handler arg order and the 5-field `OperationContext` shape (engine + config + logger + dryRun + remote). Defaults to `remote: true` (untrusted); local CLI callers pass `remote: false`. Closed F1 (reversed handler args) + F2 (incomplete OperationContext) + F3 (no param validation) drift bugs in the original v0.22.5 HTTP transport.
|
||||
- `src/mcp/rate-limit.ts` (v0.22.7) — Bounded-LRU token-bucket limiter for `gbrain serve --http`. `buildDefaultLimiters()` returns the two-bucket pipeline used by http-transport: pre-auth IP (default 30/60s, fires BEFORE the DB lookup so brute-force load against `access_tokens` is actually capped) + post-auth token-id (default 60/60s). Tracks `lastTouchedMs` separately from `lastRefillMs` so an exhausted key can't be reset by hammering past the TTL. LRU cap (default 10K keys) bounds memory under attacker-controlled key growth; TTL prune at 2× window evicts abandoned buckets.
|
||||
- `src/mcp/http-transport.ts` (v0.22.7, rewrite) — `gbrain serve --http` HTTP transport. Postgres-only — fails fast at startup on PGLite (the `access_tokens` table only exists on Postgres). Bearer auth against SHA-256 hashes in `access_tokens`. CORS default-deny via `GBRAIN_HTTP_CORS_ORIGIN` allowlist. Body cap stream-counted (1 MiB default via `GBRAIN_HTTP_MAX_BODY_BYTES`) so chunked transfers without Content-Length still hit the cap. `last_used_at` SQL-level debounce (one UPDATE per token per 60s). Per-request audit row in `mcp_request_log` with token_name + operation + status + latency. Optional `GBRAIN_HTTP_TRUST_PROXY=1` honors `X-Forwarded-For` — only safe when bound to a private interface AND the proxy strips client-supplied XFF (otherwise enables IP spoofing past the pre-auth rate limit). `/health` does `SELECT 1` against Postgres and returns 503 + `status:unhealthy` when the DB is unreachable so orchestration doesn't see green pods while clients get misleading 401s. Replaces the standalone OAuth wrapper that was vulnerable to unauthenticated client registration.
|
||||
- `src/commands/auth.ts` — Token management for the HTTP transport. `gbrain auth create/list/revoke/test`. As of v0.22.7 wired into the main CLI (`src/cli.ts`); also runs standalone via `bun run src/commands/auth.ts ...` for environments without a compiled binary. Tokens stored as SHA-256 hashes in `access_tokens` (Postgres-only).
|
||||
- `src/mcp/server.ts` — MCP stdio server (generated from operations)
|
||||
- `src/commands/auth.ts` — Standalone token management (create/list/revoke/test)
|
||||
- `src/commands/upgrade.ts` — Self-update CLI. `runPostUpgrade()` enumerates migrations from the TS registry (src/commands/migrations/index.ts) and tail-calls `runApplyMigrations(['--yes', '--non-interactive'])` so the mechanical side of every outstanding migration runs unconditionally.
|
||||
- `src/commands/migrations/` — TS migration registry (compiled into the binary; no filesystem walk of `skills/migrations/*.md` needed at runtime). `index.ts` lists migrations in semver order. `v0_11_0.ts` = Minions adoption orchestrator (8 phases). `v0_12_0.ts` = Knowledge Graph auto-wire orchestrator (5 phases: schema → config check → backfill links → backfill timeline → verify). `phaseASchema` has a 600s timeout (bumped from 60s in v0.12.1 for duplicate-heavy brains). `v0_12_2.ts` = JSONB double-encode repair orchestrator (4 phases: schema → repair-jsonb → verify → record). `v0_14_0.ts` = shell-jobs + autopilot cooperative (2 phases: schema ALTER minion_jobs.max_stalled SET DEFAULT 3 — superseded by v0.14.3's schema-level DEFAULT 5 + UPDATE backfill; pending-host-work ping for skills/migrations/v0.14.0.md). All orchestrators are idempotent and resumable from `partial` status. As of v0.14.2 (Bug 3), the RUNNER owns all ledger writes — orchestrators return `OrchestratorResult` and `apply-migrations.ts` persists a canonical `{version, status, phases}` shape after return. Orchestrators no longer call `appendCompletedMigration` directly. `statusForVersion` prefers `complete` over `partial` (never regresses). 3 consecutive partials → wedged → `--force-retry <version>` writes a `'retry'` reset marker. v0.14.3 (fix wave) ships schema-only migrations v14 (`pages_updated_at_index`) + v15 (`minion_jobs_max_stalled_default_5` with UPDATE backfill) via the `MIGRATIONS` array in `src/core/migrate.ts` — no orchestrator phases needed.
|
||||
- `src/commands/repair-jsonb.ts` — `gbrain repair-jsonb [--dry-run] [--json]`: rewrites `jsonb_typeof='string'` rows in place across 5 affected columns (pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata, page_versions.frontmatter). Fixes v0.12.0 double-encode bug on Postgres; PGLite no-ops. Idempotent.
|
||||
- `src/commands/orphans.ts` — `gbrain orphans [--json] [--count] [--include-pseudo]`: surfaces pages with zero inbound wikilinks, grouped by domain. Auto-generated/raw/pseudo pages filtered by default. Also exposed as `find_orphans` MCP operation. Shipped in v0.12.3 (contributed by @knee5).
|
||||
- `src/commands/integrity.ts` — `gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). v0.22.8: batch-load fast path on Postgres uses `SELECT DISTINCT ON (slug)` in a single SQL query to fix the PgBouncer round-trip timeout (60s → ~6s) while preserving `engine.getAllSlugs()`'s `Set<string>` semantics on multi-source brains. Gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1` so real Postgres errors are diagnosable.
|
||||
- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel <id>`. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count.
|
||||
- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2).
|
||||
- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel <id>`.
|
||||
- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows.
|
||||
- `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2.
|
||||
- `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=<ms>` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers.
|
||||
- `src/core/db-lock.ts` (v0.22.13) — generic `tryAcquireDbLock(engine, lockId, ttlMinutes)` over the existing `gbrain_cycle_locks` table. Parameterized lock id so different scopes can nest cleanly: `gbrain-cycle` for the broad cycle (held by `cycle.ts`) and `gbrain-sync` (`SYNC_LOCK_ID` constant) for `performSync`'s narrower writer window. Same UPSERT-with-TTL semantics as the prior cycle-only helper, just generalized. Survives PgBouncer transaction pooling (unlike session-scoped `pg_try_advisory_lock`); crashed holders auto-release once their TTL expires.
|
||||
- `src/core/sync-concurrency.ts` (v0.22.13) — single source of truth for the parallel-sync policy. Exports `autoConcurrency(engine, fileCount, override?)` (PGLite always serial; explicit override clamped to >=1; auto path returns `DEFAULT_PARALLEL_WORKERS=4` when `fileCount > AUTO_CONCURRENCY_FILE_THRESHOLD=100`), `shouldRunParallel(workers, fileCount, explicit)` (Q1: explicit `--workers` bypasses the >50-file floor), and `parseWorkers(s)` (rejects `'0'`, `'-3'`, `'foo'`, `'1.5'`, trailing chars — replaces the prior parseInt-with-no-validation in both `sync.ts` and `import.ts`). Used by `performSync`, `performFullSync`, `runImport`, and the Minion `sync` handler so the three sites can no longer drift.
|
||||
- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). v0.22.13 (PR #490): `performSync` wraps its body in a `gbrain-sync` writer lock so two concurrent syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot both write `last_commit` and let the last writer win. Head-drift gate after the import phase re-checks `git rev-parse HEAD`; if HEAD moved (someone ran `git checkout` / `git pull` mid-sync), the bookmark refuses to advance. Vanished files now record a failedFiles entry instead of silent-skip — the silent-skip-then-advance pathology that survived prior hardening passes is dead. Worker engines wrap in try/finally so disconnect always fires (panic-path leak fix). Both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`), validated via `parseWorkers`. Explicit `--workers` bypasses the auto-path file-count floor; auto path defers to `autoConcurrency()`. Banner moved to stderr.
|
||||
- `src/core/cycle.ts` — v0.17 brain maintenance cycle primitive. `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise<CycleReport>` composes 6 phases in semantically-driven order (lint → backlinks → sync → extract → embed → orphans). Three callers: `gbrain dream` CLI, `gbrain autopilot` daemon's inline path, and the Minions `autopilot-cycle` handler (`src/commands/jobs.ts`). One source of truth for what the brain does overnight. Coordination via `gbrain_cycle_locks` DB table (TTL-based; works through PgBouncer transaction pooling, unlike session-scoped `pg_try_advisory_lock`) + `~/.gbrain/cycle.lock` file lock with PID-liveness for PGLite / engine=null mode. `CycleReport.schema_version: "1"` is the stable agent-consumable shape. `PhaseResult.error: { class, code, message, hint?, docs_url? }` is Stripe-API-tier structured failure info. `yieldBetweenPhases` hook awaited between every phase — Minions handler uses this to renew its job lock and prevent v0.14 stall-death regression. Engine nullable: filesystem phases (lint, backlinks) run without DB; DB phases skip with `status: "skipped", reason: "no_database"`. Lock-skip: read-only phase selections (`--phase orphans`) bypass the cycle lock. v0.22.1 (#403): `CycleOpts.signal?: AbortSignal` propagates the worker's abort signal; `checkAborted()` fires between every phase and throws if the signal is aborted (cooperative — can't interrupt a phase mid-execution). v0.22.1 (#417): `runPhaseSync` returns `pagesAffected` via `SyncPhaseResult`; `runCycle` captures it and threads to `runPhaseExtract` as the 4th arg, enabling incremental extract on the cycle path. v0.22.1 (Codex F2): `runPhaseSync` takes `willRunExtractPhase: boolean` and sets `noExtract: phases.includes('extract')` so `gbrain dream --phase sync` doesn't silently lose extraction. v0.22.5 (#475): new `resolveSourceForDir(engine, brainDir)` helper queries `SELECT id FROM sources WHERE local_path = $1 LIMIT 1`; `runPhaseSync` threads result as `sourceId` to `performSync()` so sync reads the per-source `sources.last_commit` anchor instead of the drift-prone global `config.sync.last_commit` key. Bare try/catch lets pre-v0.18 brains fall through to the global key. Closes the prod hang where every autopilot cycle ran a 30-min full reimport because the global anchor commit had been GC'd from git history.
|
||||
- `src/core/cycle.ts` — v0.17 brain maintenance cycle primitive. `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise<CycleReport>` composes 6 phases in semantically-driven order (lint → backlinks → sync → extract → embed → orphans). Three callers: `gbrain dream` CLI, `gbrain autopilot` daemon's inline path, and the Minions `autopilot-cycle` handler (`src/commands/jobs.ts`). One source of truth for what the brain does overnight. Coordination via `gbrain_cycle_locks` DB table (TTL-based; works through PgBouncer transaction pooling, unlike session-scoped `pg_try_advisory_lock`) + `~/.gbrain/cycle.lock` file lock with PID-liveness for PGLite / engine=null mode. `CycleReport.schema_version: "1"` is the stable agent-consumable shape. `PhaseResult.error: { class, code, message, hint?, docs_url? }` is Stripe-API-tier structured failure info. `yieldBetweenPhases` hook awaited between every phase — Minions handler uses this to renew its job lock and prevent v0.14 stall-death regression. Engine nullable: filesystem phases (lint, backlinks) run without DB; DB phases skip with `status: "skipped", reason: "no_database"`. Lock-skip: read-only phase selections (`--phase orphans`) bypass the cycle lock.
|
||||
- `src/commands/dream.ts` — v0.17 `gbrain dream` CLI. ~80-line thin alias over `runCycle`. brainDir resolution requires explicit `--dir` OR `sync.repo_path` config (no more walk-up-cwd-for-.git footgun). Flags: `--dry-run`, `--json`, `--phase <name>`, `--pull`, `--dir <path>`. Exit code 1 on status=failed (partial/warn not fatal — don't page on warnings).
|
||||
- `scripts/check-progress-to-stdout.sh` — CI guard against regressing to `\r`-on-stdout progress. Wired into `bun run test` via `scripts/check-progress-to-stdout.sh && bun test` in package.json.
|
||||
- `docs/progress-events.md` — Canonical JSON event schema reference. Stable from v0.15.2, additive only.
|
||||
@@ -223,10 +208,6 @@ Key commands added in v0.14.3 (fix wave):
|
||||
- `gbrain jobs submit` gains `--max-stalled`, `--backoff-type`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key` — exposing existing `MinionJobInput` fields as first-class CLI flags.
|
||||
- `gbrain jobs smoke --sigkill-rescue` — opt-in regression smoke case simulating a killed worker; asserts the v0.14.3 schema default (`max_stalled=5`) actually rescues on first stall.
|
||||
|
||||
Key commands added in v0.22.13 (PR #490):
|
||||
- `gbrain sync --workers N` (alias `--concurrency N`) — parallelize the import phase using per-worker Postgres engines (small pool of 2 each) with an atomic queue index. Auto-concurrency: defaults to 4 workers when the diff exceeds 100 files. Smaller diffs stay serial. Explicit `--workers` always wins (even on a 30-file diff). PGLite forces serial regardless. Validation rejects `0`, negatives, non-integers loud (replaces the prior silent fall-through to auto-concurrency).
|
||||
- `gbrain import --workers N` — same `parseWorkers()` validation as sync; same try/finally worker-engine cleanup. Behavior surface unchanged.
|
||||
|
||||
## Testing
|
||||
|
||||
`bun test` runs all tests. After the v0.12.1 release: ~75 unit test files + 8 E2E test files (1412 unit pass, 119 E2E when `DATABASE_URL` is set — skip gracefully otherwise). Unit tests run
|
||||
@@ -238,9 +219,7 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac
|
||||
`test/files.test.ts` (MIME/hash), `test/import-file.test.ts` (import pipeline),
|
||||
`test/upgrade.test.ts` (schema migrations),
|
||||
`test/file-migration.test.ts` (file migration), `test/file-resolver.test.ts` (file resolution),
|
||||
`test/import-resume.test.ts` (import checkpoints), `test/migrate.test.ts` (migration; v8/v9 helper-btree-index SQL structural assertions + 1000-row wall-clock fixtures that guard the O(n²)→O(n log n) fix + v0.13.1 assertions on v12/v13 SQL shape, `sqlFor` + `transaction:false` runner semantics, the `max_stalled DEFAULT 1` regression guard, and v0.22.6.1 v24 `sqlFor.pglite: ''` no-op assertion),
|
||||
`test/bootstrap.test.ts` (v0.22.6.1 — bootstrap contract: no-op on fresh install, idempotent across two `initSchema()` calls, no-op on modern brain that already has every probed column, full bootstrap path on simulated pre-v0.18 brain, fresh-install regression guard, pre-v0.13 `links` shape coverage),
|
||||
`test/schema-bootstrap-coverage.test.ts` (v0.22.6.1 CI guard — `REQUIRED_BOOTSTRAP_COVERAGE` lists every forward reference in PGLITE_SCHEMA_SQL; the test fails loudly if `applyForwardReferenceBootstrap` skips one. When you add a column-with-index to the embedded schema blob, you extend both arrays or this guard fails. The pattern that broke gbrain ten times in two years is now structurally prevented.),
|
||||
`test/import-resume.test.ts` (import checkpoints), `test/migrate.test.ts` (migration; v8/v9 helper-btree-index SQL structural assertions + 1000-row wall-clock fixtures that guard the O(n²)→O(n log n) fix + v0.13.1 assertions on v12/v13 SQL shape, `sqlFor` + `transaction:false` runner semantics, and the `max_stalled DEFAULT 1` regression guard),
|
||||
`test/setup-branching.test.ts` (setup flow), `test/slug-validation.test.ts` (slug validation),
|
||||
`test/storage.test.ts` (storage backends), `test/supabase-admin.test.ts` (Supabase admin),
|
||||
`test/yaml-lite.test.ts` (YAML parsing), `test/check-update.test.ts` (version check + update CLI),
|
||||
@@ -254,7 +233,6 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac
|
||||
`test/skills-conformance.test.ts` (skill frontmatter + required sections validation),
|
||||
`test/resolver.test.ts` (RESOLVER.md coverage, routing validation + v0.20.4 round-trip: every quoted RESOLVER.md trigger must match a frontmatter `triggers:` entry in the target skill, and every `name="<word>"` reference in any SKILL.md must resolve to a declared op in `src/core/operations.ts` or a Minions handler in `PROTECTED_JOB_NAMES`),
|
||||
`test/search.test.ts` (RRF normalization, compiled truth boost, cosine similarity, dedup key),
|
||||
`test/sql-ranking.test.ts` (v0.22.0 source-boost helpers: 39 cases covering longest-prefix-match in SQL CASE, detail=high temporal-bypass, three-meta-char LIKE escape (%, _, \\), single-quote SQL-literal doubling, env override parsing for GBRAIN_SOURCE_BOOST + GBRAIN_SEARCH_EXCLUDE, resolveBoostMap / resolveHardExcludes merge semantics),
|
||||
`test/dedup.test.ts` (source-aware dedup, compiled truth guarantee, layer interactions),
|
||||
`test/intent.test.ts` (query intent classification: entity/temporal/event/general),
|
||||
`test/eval.test.ts` (retrieval metrics: precisionAtK, recallAtK, mrr, ndcgAtK, parseQrels),
|
||||
@@ -282,9 +260,6 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac
|
||||
`test/orphans.test.ts` (v0.12.3 orphans command: detection, pseudo filtering, text/json/count outputs, MCP op),
|
||||
`test/postgres-engine.test.ts` (v0.12.3 statement_timeout scoping: `sql.begin` + `SET LOCAL` shape, source-level grep guardrail against reintroduced bare `SET statement_timeout`),
|
||||
`test/sync.test.ts` (sync logic + v0.12.3 regression guard asserting top-level `engine.transaction` is not called),
|
||||
`test/sync-concurrency.test.ts` (v0.22.13 PR #490: 17 cases covering `autoConcurrency()` thresholds + PGLite-forces-serial + explicit-override clamping, `shouldRunParallel()` Q1 explicit-bypasses-floor contract, and `parseWorkers()` validation that rejects `'0'`/`'-3'`/`'foo'`/`'1.5'`/trailing chars),
|
||||
`test/sync-parallel.test.ts` (v0.22.13 PR #490: PGLite-routed coverage of the bookmark gate under concurrency request, head-drift gate, vanished-file failure capture, PGLite-stays-serial, and the `gbrain-sync` writer-lock contract — 7 cases),
|
||||
`test/sync-failures.test.ts` (v0.22.12: 28 cases pinning `classifyErrorCode` regex coverage for all 12 codes against literal production message strings from `markdown.ts:159-244` and `import-file.ts:199, 347, 352, 401`; `summarizeFailuresByCode` sort + pre-classified-honor; `recordSyncFailures` code-field persistence; `acknowledgeSyncFailures` AcknowledgeResult shape + backfill on pre-v0.22.12 entries),
|
||||
`test/doctor.test.ts` (doctor command + v0.12.3 assertions that `jsonb_integrity` scans the four v0.12.0 write sites and `markdown_body_completeness` is present),
|
||||
`test/utils.test.ts` (shared SQL utilities + `tryParseEmbedding` null-return and single-warn semantics),
|
||||
`test/build-llms.test.ts` (llms.txt/llms-full.txt generator: path resolution, idempotence, spec shape, regen-drift guard, content contract, AGENTS.md install-path mirror, size-budget enforcement — 7 cases),
|
||||
@@ -295,26 +270,17 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac
|
||||
`test/skill-manifest.test.ts` (v0.19 skill manifest parser: drift detection, managed-block markers),
|
||||
`test/skillify-scaffold.test.ts` (v0.19 `gbrain skillify scaffold` stubs: SKILL.md, script, tests, routing-eval fixtures),
|
||||
`test/skillpack-install.test.ts` (v0.19 `gbrain skillpack install` managed-block install / update / no-clobber semantics),
|
||||
`test/skillpack-sync-guard.test.ts` (v0.19 sync-guard: bundled skills stay byte-identical to `skills/` source),
|
||||
`test/http-transport.test.ts` (v0.22.7 HTTP transport: 23 unit cases covering bearer auth + missing/no-Bearer/unknown/revoked + `/health` bypass, F1+F2 round-trip via dispatch.ts, F3 invalid_params, application/json response shape (not SSE), CORS default-deny + allowlist, body cap on Content-Length AND chunked, two-bucket rate limit (refill, exhaust+Retry-After, LRU eviction, TTL prune, pre-auth IP fires before DB), and `mcp_request_log` audit on success + auth_failed).
|
||||
`test/skillpack-sync-guard.test.ts` (v0.19 sync-guard: bundled skills stay byte-identical to `skills/` source).
|
||||
|
||||
E2E tests (`test/e2e/`): Run against real Postgres+pgvector. Require `DATABASE_URL`.
|
||||
- `bun run test:e2e` runs Tier 1 (mechanical, all operations, no API keys). Includes 9 dedicated cases for the postgres-engine `addLinksBatch` / `addTimelineEntriesBatch` bind path — postgres-js's `unnest()` binding is structurally different from PGLite's and gets its own coverage.
|
||||
- `test/e2e/search-quality.test.ts` runs search quality E2E against PGLite (no API keys, in-memory)
|
||||
- `test/e2e/graph-quality.test.ts` runs the v0.10.3 knowledge graph pipeline (auto-link via put_page, reconciliation, traversePaths) against PGLite in-memory
|
||||
- `test/e2e/postgres-jsonb.test.ts` — v0.12.2 regression test. Round-trips all 5 JSONB write sites (pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata, page_versions.frontmatter) against real Postgres and asserts `jsonb_typeof='object'` plus `->>'key'` returns the expected scalar. The test that should have caught the original double-encode bug.
|
||||
- `test/e2e/integrity-batch.test.ts` (v0.22.8) — parity tests for `scanIntegrity`'s batch-load fast path vs sequential. Four cases (dedup, hits, validate, topPages) seed a fixture and assert both paths return identical results. Dedup case uses raw SQL via `getConn().unsafe()` to seed a `(test-source-2, people/alice)` row alongside the default-source row, since `engine.putPage` doesn't take a `source_id`. Pins the codex-caught multi-source overcounting regression.
|
||||
- `test/e2e/jsonb-roundtrip.test.ts` — v0.12.3 companion regression against the 4 doctor-scanned JSONB sites. Assertion-level overlap with `postgres-jsonb.test.ts` is intentional defense-in-depth: if doctor's scan surface ever drifts from the actual write surface, one of these tests catches it.
|
||||
- `test/e2e/sync.test.ts` (v0.22.12 — `--skip-failed` failure-loop test, alongside the existing 13 happy-path tests): exercises the full chain — broken file → `performSync` returns `blocked_by_failures` with grouped breakdown → `performSync({skipFailed: true})` advances bookmark and returns `AcknowledgeResult` with code summary → second broken file → second cycle. Saves and restores the user's real `~/.gbrain/sync-failures.jsonl` so the test is hermetic on a developer machine. Asserts bookmark gating, JSONL state, dedup across paths, summary aggregation, and the literal doctor-rendering string format. This is the integration test that proves the v0.22.12 chain holds together — unit tests cover the pure functions in isolation, this covers the integration.
|
||||
- `test/e2e/upgrade.test.ts` runs check-update E2E against real GitHub API (network required)
|
||||
- `test/e2e/minions-shell-pglite.test.ts` (v0.20.4) exercises the PGLite `--follow` inline shell-job path (in-memory, no `DATABASE_URL` required) — the path the consolidated minion-orchestrator skill documents for dev use
|
||||
- `test/e2e/openclaw-reference-compat.test.ts` (v0.19) — exercises `check-resolvable` + `skillpack install` against a minimal AGENTS.md workspace fixture (`test/fixtures/openclaw-reference-minimal/`), regression guard for the 107-skill OpenClaw deployment shape
|
||||
- `test/e2e/search-swamp.test.ts` (v0.22.0) — reproduces the headline source-swamp case. Seeds a curated `originals/talks/article-outline-fat-code` page against two `wintermute/chat/` pages stuffed with the same multi-word phrase. Asserts the article wins keyword AND vector ranking, that `detail=high` lets the chat swamp re-surface (temporal-query workflow preserved), and that `source_id` passes through the two-stage CTE intact. PGLite in-memory.
|
||||
- `test/e2e/search-exclude.test.ts` (v0.22.0) — verifies `test/` + `archive/` pages are hidden by default, that `include_slug_prefixes` opts back in, and that caller-supplied `exclude_slug_prefixes` adds to defaults. Both keyword and vector search paths covered.
|
||||
- `test/e2e/engine-parity.test.ts` (v0.22.0) — Postgres ↔ PGLite top-result and result-set parity for `searchKeyword` + `searchVector`. Codex flagged that Postgres ranks pages then picks best chunk while PGLite returns chunks directly — without parity coverage the source-boost fix could pass on PGLite and fail on Postgres. Skips gracefully when `DATABASE_URL` is unset.
|
||||
- `test/e2e/postgres-bootstrap.test.ts` (v0.22.6.1) — exercises `PostgresEngine.initSchema()` directly against a fresh real Postgres database. Asserts the bootstrap path is no-op on fresh installs and that SCHEMA_SQL replays cleanly through the engine path (not via the standalone `db.initSchema` from `src/core/db.ts`, which would have produced false-positive coverage). Codex caught the E2E-shape gap during plan review.
|
||||
- `test/e2e/http-transport.test.ts` (v0.22.7) — 8 cases against real Postgres covering `gbrain serve --http` end-to-end: bearer auth round-trip, `last_used_at` SQL-level debounce semantics, `mcp_request_log` row insertion on success and auth_failed paths, `/health` DB-down → 503 (DB-probing health check), and the F1+F2+F3 dispatch round-trip with a real operation. Skips gracefully when `DATABASE_URL` is unset.
|
||||
- `test/e2e/sync-parallel.test.ts` (v0.22.13 PR #490) — DATABASE_URL-gated. T2: 60-file Postgres sync at concurrency=4 imports all + no connection leak (probes `pg_stat_activity` before/after to confirm worker engines disconnected). P4: 120-file serial-vs-parallel benchmark prints `SYNC_PARALLEL_BENCH N files | serial=Xms | parallel(4)=Yms | speedup=Zx` for CHANGELOG quoting. Asserts parallel ≤ serial × 1.5 (CI-noise tolerant; not a strict speedup gate).
|
||||
- Tier 2 (`skills.test.ts`) requires OpenClaw + API keys, runs nightly in CI
|
||||
- If `.env.testing` doesn't exist in this directory, check sibling worktrees for one:
|
||||
`find ../ -maxdepth 2 -name .env.testing -print -quit` and copy it here if found.
|
||||
@@ -429,59 +395,6 @@ in bulk paths, the CI guard will fail the build.
|
||||
|
||||
`bun build --compile --outfile bin/gbrain src/cli.ts`
|
||||
|
||||
## Version locations (single source of truth: `VERSION` file)
|
||||
|
||||
Every release advances the version in **five files at once**. Keep these in
|
||||
sync. `/ship` enforces this via Step 12's idempotency check (VERSION vs
|
||||
package.json drift), but the canonical list lives here so future runs and
|
||||
the auto-update agent know where to look.
|
||||
|
||||
**Required (every release must update all five):**
|
||||
|
||||
| File | What lives there | Format |
|
||||
|---|---|---|
|
||||
| `VERSION` | The single source of truth. Read first by `/ship`, the binary, and CI version-gate. | Bare 4-digit string `MAJOR.MINOR.PATCH.MICRO` (e.g. `0.22.1`), no leading `v`, no trailing newline-sensitivity issues. |
|
||||
| `package.json` | Bun/npm package version. `gbrain --version` reads it via the compiled binary's bundled package metadata. CI version-gate cross-checks this against `VERSION` and fails if they drift. | `"version": "0.22.1"` |
|
||||
| `CHANGELOG.md` | Top entry header `## [0.22.1] - YYYY-MM-DD` plus the "To take advantage of v0.22.1" block. | Standard Keep-a-Changelog header. |
|
||||
| `TODOS.md` | Any TODO entries that mention "follow-up from vX.Y.Z" use the version of the release that filed them. Update only when filing NEW follow-up TODOs. | Inline `vX.Y.Z` references in TODO bodies. |
|
||||
| `CLAUDE.md` | The Key Files section's per-file annotations carry `vX.Y.Z (#NNN)` tags noting which release introduced a behavior. Update whenever a wave's annotations get folded in. | Inline `vX.Y.Z (#NNN, contributed by @user)` references. |
|
||||
|
||||
**Auto-derived (no manual edit; refreshed by their own commands):**
|
||||
|
||||
- `bun.lock` — root-package version is auto-pinned from `package.json`. After
|
||||
bumping `package.json`, run `bun install` to refresh the lockfile.
|
||||
- `llms-full.txt` / `llms.txt` — auto-generated documentation bundles. After
|
||||
any release ship that touches the Key Files annotations in `CLAUDE.md`,
|
||||
run `bun run build:llms` to regenerate. The bundles do not contain a
|
||||
version pin per se; they reflect the current state of the docs they index.
|
||||
|
||||
**Historical (DO NOT bump on release):**
|
||||
|
||||
- `skills/migrations/v0.21.0.md` — migration files use the version they
|
||||
shipped FROM as their filename. v0.21.0's migration always says v0.21.0.
|
||||
- `src/commands/migrations/v0_21_0.ts` — same: migration code references
|
||||
the schema version it migrates to.
|
||||
- `test/migrations-v0_21_0.test.ts`, `test/migration-orchestrator-v0_21_0.test.ts`,
|
||||
`test/migrate.test.ts` — migration tests reference historical migration
|
||||
versions; these are correct as-is and should not move.
|
||||
- `src/core/db.ts`, `src/core/migrate.ts`, `src/core/import-file.ts`,
|
||||
`src/commands/reindex-code.ts` — code comments cite the release that
|
||||
introduced a feature. Once written, these are historical record.
|
||||
- `README.md` — references the latest published feature names by version
|
||||
(e.g. "v0.21.0 Code Cathedral"); update only when the README's marketing
|
||||
copy is intentionally being refreshed, NOT on every micro/patch bump.
|
||||
|
||||
**The /ship workflow's version idempotency check:** Step 12 reads
|
||||
`VERSION` and `package.json`, classifies as FRESH / ALREADY_BUMPED /
|
||||
DRIFT_STALE_PKG / DRIFT_UNEXPECTED, and refuses to proceed on
|
||||
DRIFT_UNEXPECTED. This is why the two must move together.
|
||||
|
||||
**The CI version-gate** rejects pushes where `VERSION` and
|
||||
`package.json` disagree, OR where `VERSION` is not strictly greater
|
||||
than master's VERSION. If a queue collision claims your version on
|
||||
master before yours lands, /ship's queue-aware allocator (Step 12)
|
||||
will detect drift and re-bump on the next run.
|
||||
|
||||
## Pre-ship requirements
|
||||
|
||||
Before shipping (/ship) or reviewing (/review), always run the full test suite:
|
||||
|
||||
@@ -80,13 +80,12 @@ Add to `~/.claude/server.json` (Claude Code), Settings > MCP Servers (Cursor), o
|
||||
### Remote MCP (Claude Desktop, Cowork, Perplexity)
|
||||
|
||||
```bash
|
||||
gbrain auth create "claude-desktop" # tokens via the existing CLI
|
||||
gbrain serve --http --port 8787 # built-in HTTP transport (Postgres-only)
|
||||
ngrok http 8787 --url your-brain.ngrok.app # any tunnel works
|
||||
ngrok http 8787 --url your-brain.ngrok.app
|
||||
bun run src/commands/auth.ts create "claude-desktop"
|
||||
claude mcp add gbrain -t http https://your-brain.ngrok.app/mcp -H "Authorization: Bearer TOKEN"
|
||||
```
|
||||
|
||||
Per-client guides: [`docs/mcp/`](docs/mcp/DEPLOY.md). Hardening defaults, env vars, and threat model: [SECURITY.md](SECURITY.md). ChatGPT requires OAuth 2.1 (not yet implemented).
|
||||
Per-client guides: [`docs/mcp/`](docs/mcp/DEPLOY.md). ChatGPT requires OAuth 2.1 (not yet implemented).
|
||||
|
||||
### Using gbrain with GStack
|
||||
|
||||
@@ -360,30 +359,6 @@ accumulate rows across separate single-skill installs instead of overwriting eac
|
||||
Read [`skills/skillify/SKILL.md`](skills/skillify/SKILL.md) for the full 10-item checklist
|
||||
and the anti-patterns it catches.
|
||||
|
||||
## Storage tiering: keep bulk content out of git (v0.22.11)
|
||||
|
||||
When your brain crosses 100K files and bulk machine-generated content (tweets, articles, transcripts)
|
||||
becomes the size driver, declare which directories belong in git and which live in the database only.
|
||||
|
||||
```yaml
|
||||
# gbrain.yml at the brain repo root
|
||||
storage:
|
||||
db_tracked:
|
||||
- people/
|
||||
- companies/
|
||||
- deals/
|
||||
db_only:
|
||||
- media/x/
|
||||
- media/articles/
|
||||
- meetings/transcripts/
|
||||
```
|
||||
|
||||
`gbrain sync` auto-manages your `.gitignore` for `db_only` paths. `gbrain export --restore-only --repo .`
|
||||
repopulates missing files from the database (container restart, fresh clone, accidental rm).
|
||||
`gbrain storage status` shows the tier breakdown.
|
||||
|
||||
Full guide: [docs/storage-tiering.md](docs/storage-tiering.md).
|
||||
|
||||
## Getting Data In
|
||||
|
||||
GBrain ships integration recipes that your agent sets up for you. Each recipe tells the agent what credentials to ask for, how to validate, and what cron to register.
|
||||
@@ -530,8 +505,6 @@ Question
|
||||
│ ├─ Multi-query expansion (Haiku rephrases the question 3 ways)
|
||||
│ ├─ Vector search (HNSW cosine over OpenAI embeddings)
|
||||
│ ├─ Keyword search (Postgres tsvector + websearch_to_tsquery)
|
||||
│ ├─ Source-aware ranking (curated dirs outrank chat/daily swamp at SQL layer)
|
||||
│ ├─ Hard-exclude (test/ archive/ attachments/ .raw/ filtered before retrieval)
|
||||
│ ├─ Reciprocal Rank Fusion (score = sum 1/(60+rank) across both)
|
||||
│ ├─ Cosine re-scoring (re-rank chunks against actual query embedding)
|
||||
│ ├─ Compiled-truth boost (assessments outrank timeline noise)
|
||||
@@ -639,11 +612,8 @@ SEARCH
|
||||
gbrain query <question> Hybrid search (vector + keyword + RRF)
|
||||
|
||||
IMPORT
|
||||
gbrain import <dir> [--no-embed] [--workers N]
|
||||
Import markdown (idempotent)
|
||||
gbrain sync [--repo <path>] [--workers N]
|
||||
Git-to-brain incremental sync
|
||||
(>100-file diffs auto-parallelize 4 workers on Postgres)
|
||||
gbrain import <dir> [--no-embed] Import markdown (idempotent)
|
||||
gbrain sync [--repo <path>] Git-to-brain incremental sync
|
||||
gbrain export [--dir ./out/] Export to markdown
|
||||
|
||||
FILES
|
||||
@@ -685,8 +655,6 @@ ADMIN
|
||||
gbrain doctor --locks List idle-in-tx backends (57014 diagnostic, Postgres only)
|
||||
gbrain stats Brain statistics
|
||||
gbrain serve MCP server (stdio)
|
||||
gbrain serve --http --port 8787 MCP server (HTTP, Postgres-only, bearer auth)
|
||||
gbrain auth create|list|revoke|test Token management for the HTTP transport
|
||||
gbrain integrations Integration recipe dashboard
|
||||
gbrain sources list|add|remove|... Multi-source brain management (v0.18)
|
||||
gbrain dream [--dry-run] [--phase N] One maintenance cycle then exit (cron-friendly)
|
||||
|
||||
-168
@@ -1,168 +0,0 @@
|
||||
# Security
|
||||
|
||||
## Reporting Vulnerabilities
|
||||
|
||||
If you discover a security issue in GBrain, please report it privately by opening
|
||||
a [private security advisory](https://github.com/garrytan/gbrain/security/advisories/new)
|
||||
on GitHub.
|
||||
|
||||
Do not open a public issue for security vulnerabilities.
|
||||
|
||||
## Remote MCP Security
|
||||
|
||||
### ⚠️ Do NOT use open OAuth client registration for remote MCP
|
||||
|
||||
If you deploy GBrain's MCP server behind an HTTP wrapper with OAuth 2.1
|
||||
support, **never allow unauthenticated client registration**. An attacker
|
||||
who discovers your server URL can:
|
||||
|
||||
1. Register a new OAuth client via `POST /register`
|
||||
2. Use `client_credentials` grant to obtain a bearer token
|
||||
3. Access all brain data via the MCP tools
|
||||
|
||||
### Recommended: `gbrain serve --http`
|
||||
|
||||
As of v0.22.7, GBrain ships a built-in HTTP transport that uses the
|
||||
existing `access_tokens` table for authentication:
|
||||
|
||||
```bash
|
||||
# Create a token
|
||||
gbrain auth create "my-client"
|
||||
|
||||
# Start the HTTP server
|
||||
gbrain serve --http --port 8787
|
||||
|
||||
# Connect via ngrok, Tailscale, or any tunnel
|
||||
ngrok http 8787 --url your-brain.ngrok.app
|
||||
```
|
||||
|
||||
This is the recommended way to expose GBrain remotely. No OAuth, no
|
||||
registration endpoint, no self-service tokens. Tokens are managed
|
||||
exclusively via `gbrain auth create/list/revoke`.
|
||||
|
||||
### If you must use a custom HTTP wrapper
|
||||
|
||||
1. **Require a secret for client registration** — check a header or body
|
||||
parameter before creating new OAuth clients
|
||||
2. **Disable `client_credentials` grant** — only allow `authorization_code`
|
||||
with browser-based approval
|
||||
3. **Restrict scopes** — never issue tokens with unlimited scope
|
||||
4. **Log all token issuance** — alert on unexpected registrations
|
||||
5. **Rate-limit registration and token endpoints**
|
||||
|
||||
### Token Management
|
||||
|
||||
```bash
|
||||
gbrain auth create "claude-desktop" # Create a new token
|
||||
gbrain auth list # List all tokens
|
||||
gbrain auth revoke "claude-desktop" # Revoke a token
|
||||
gbrain auth test <url> --token <tok> # Smoke-test a remote server
|
||||
```
|
||||
|
||||
Tokens are stored as SHA-256 hashes in the `access_tokens` table. The
|
||||
plaintext token is shown once at creation and never stored.
|
||||
|
||||
## `gbrain serve --http` hardening (v0.22.7+)
|
||||
|
||||
The built-in HTTP transport ships with several layers of hardening on by
|
||||
default. All env vars below are optional; the defaults are intentionally
|
||||
conservative.
|
||||
|
||||
### Postgres-only
|
||||
|
||||
`gbrain serve --http` requires a Postgres engine. PGLite is local-only by
|
||||
design and the `access_tokens` / `mcp_request_log` tables don't exist in
|
||||
the PGLite schema. Local agents continue to use stdio (`gbrain serve`).
|
||||
Running `--http` against a PGLite-backed install fails fast with a clear
|
||||
error message at startup.
|
||||
|
||||
### CORS
|
||||
|
||||
Default-deny: no `Access-Control-Allow-Origin` header is sent unless an
|
||||
allowlist is configured. To allow browser-based MCP clients:
|
||||
|
||||
```bash
|
||||
GBRAIN_HTTP_CORS_ORIGIN=https://claude.ai gbrain serve --http --port 8787
|
||||
# Multiple origins: comma-separated
|
||||
GBRAIN_HTTP_CORS_ORIGIN=https://claude.ai,https://your.app gbrain serve --http
|
||||
```
|
||||
|
||||
When the request `Origin` matches the allowlist, the server echoes it
|
||||
back in `Access-Control-Allow-Origin` (with `Vary: Origin`). Otherwise no
|
||||
CORS header is sent and the browser blocks the request.
|
||||
|
||||
### Rate limiting
|
||||
|
||||
Two buckets, both stored in a bounded LRU map (default 10K keys, evicts
|
||||
least-recently-used on overflow, prunes entries older than 2× the
|
||||
window):
|
||||
|
||||
| Bucket | When it fires | Default | Env var |
|
||||
|---|---|---|---|
|
||||
| Pre-auth IP | Before the DB lookup, on every `/mcp` request | 30 req / 60s | `GBRAIN_HTTP_RATE_LIMIT_IP` |
|
||||
| Post-auth token | After a valid token is resolved | 60 req / 60s | `GBRAIN_HTTP_RATE_LIMIT_TOKEN` |
|
||||
| LRU cap | Maximum distinct keys across both buckets | 10000 | `GBRAIN_HTTP_RATE_LIMIT_LRU` |
|
||||
|
||||
On exhaustion the server returns `429 Too Many Requests` with a
|
||||
`Retry-After` header.
|
||||
|
||||
**Caveat for tunneled deployments (ngrok, Tailscale Funnel, Cloudflare
|
||||
Tunnel):** all requests share one egress IP, so the pre-auth IP bucket
|
||||
becomes effectively shared by all clients on that tunnel. The
|
||||
post-auth token-id bucket is the load-bearing limiter for tunnel-fronted
|
||||
deployments.
|
||||
|
||||
### Reverse-proxy trust
|
||||
|
||||
Disabled by default. To honor `X-Forwarded-For` (or `X-Real-IP`) when
|
||||
gbrain runs behind a trusted reverse proxy:
|
||||
|
||||
```bash
|
||||
GBRAIN_HTTP_TRUST_PROXY=1 gbrain serve --http --port 8787
|
||||
```
|
||||
|
||||
**Critical safety contract:** only set `GBRAIN_HTTP_TRUST_PROXY=1` when
|
||||
**both** of these are true:
|
||||
|
||||
1. gbrain is reachable only via a trusted reverse proxy (not directly
|
||||
exposed to the internet on the configured port). The simplest
|
||||
guarantee is to bind gbrain to `127.0.0.1` or a private interface
|
||||
and have the proxy forward to it.
|
||||
2. The proxy strips any client-supplied `X-Forwarded-For` and `X-Real-IP`
|
||||
headers, then sets them itself. (nginx with `proxy_set_header
|
||||
X-Forwarded-For $remote_addr` does this; Cloudflare and most cloud
|
||||
load balancers handle it automatically.)
|
||||
|
||||
If gbrain is reachable directly AND `GBRAIN_HTTP_TRUST_PROXY=1` is set,
|
||||
clients can spoof their IP by sending arbitrary `X-Forwarded-For`
|
||||
headers, defeating the pre-auth IP rate limit. Without the flag, gbrain
|
||||
ignores all forwarded-for headers and uses the socket peer address,
|
||||
which is the safe default for direct-exposure deployments.
|
||||
|
||||
### Body size cap
|
||||
|
||||
Default 1 MiB, stream-counted (chunked transfers without
|
||||
`Content-Length` are still capped). Override:
|
||||
|
||||
```bash
|
||||
GBRAIN_HTTP_MAX_BODY_BYTES=2097152 gbrain serve --http # 2 MiB
|
||||
```
|
||||
|
||||
Over-cap requests get `413 Payload Too Large` immediately, before any
|
||||
body is materialized in memory.
|
||||
|
||||
### Audit log
|
||||
|
||||
Every `/mcp` request writes one row to `mcp_request_log`:
|
||||
|
||||
```bash
|
||||
psql "$DATABASE_URL" -c \
|
||||
"SELECT created_at, token_name, operation, status, latency_ms
|
||||
FROM mcp_request_log
|
||||
ORDER BY created_at DESC LIMIT 100"
|
||||
```
|
||||
|
||||
`status` is one of: `success`, `error`, `auth_failed`, `rate_limited`,
|
||||
`body_too_large`, `parse_error`, `unknown_method`. Failed-auth rows have
|
||||
`token_name = NULL`. Inserts are fire-and-forget so audit failures
|
||||
never block requests.
|
||||
@@ -1,201 +1,5 @@
|
||||
# TODOS
|
||||
|
||||
## sync (v0.22.13 follow-up — PR #490 review)
|
||||
|
||||
### D-PR490-1 — Plumb resolved `database_url` through `SyncOpts`
|
||||
**Priority:** P3
|
||||
|
||||
**What:** Add `database_url?: string` (or a richer `resolvedConnection` shape) to
|
||||
`SyncOpts` and have the caller (`runSync`, the cycle handler, the jobs handler)
|
||||
populate it from the active engine instead of having `performSync` /
|
||||
`performFullSync` / `import.ts` each call `loadConfig()` separately. Today every
|
||||
sync run hits the config file three times.
|
||||
|
||||
**Why:** v0.18 multi-source brains can in principle run different sources against
|
||||
different `database_url` endpoints (or different per-source overrides via
|
||||
`sources.config_jsonb`). Right now `loadConfig()` returns the global config, and
|
||||
that always matches the engine in practice — but the convention papers over a
|
||||
real divergence the moment someone wants per-source connection settings. Folding
|
||||
the resolution into `SyncOpts` makes the worker-engine creation in `sync.ts` and
|
||||
`import.ts` deterministic from `SyncOpts` alone.
|
||||
|
||||
**Pros:**
|
||||
- Removes 3 redundant `loadConfig()` calls per sync.
|
||||
- Makes `performSync` / `performFullSync` side-effect-free with respect to the
|
||||
on-disk config file.
|
||||
- Sets up for per-source `database_url` overrides without further refactor.
|
||||
- Makes the v0.22.13 belt-and-suspenders fallback (PR #490 Q3) cleaner — no
|
||||
more `!config?.database_url` short-circuit inside the parallel branch.
|
||||
|
||||
**Cons:**
|
||||
- API-shape change to `SyncOpts` (mild; not externally exported).
|
||||
- Touching three callers (`runSync`, jobs handler, `cycle.ts` `runPhaseSync`).
|
||||
- Only worth doing when paired with a per-source override story; otherwise
|
||||
it's just plumbing.
|
||||
|
||||
**Context:** Surfaced during the PR #490 plan-eng-review (parallel sync).
|
||||
Deferred because it isn't on the v0.22.13 critical path. The same pattern would
|
||||
benefit the cycle handler and the autopilot daemon. See the plan-eng-review
|
||||
decisions log: A4 = "Defer; file as TODO."
|
||||
|
||||
**Depends on / blocked by:** Nothing structural. Best paired with the v0.18
|
||||
per-source `config_jsonb` work if/when that lands.
|
||||
|
||||
## sync error-code classification (PR #501 follow-ups)
|
||||
|
||||
### Plumb structured `ParseValidationCode` through `ImportResult`
|
||||
**Priority:** P2
|
||||
|
||||
**What:** Replace the regex-on-error-message path in `src/core/sync.ts:classifyErrorCode`
|
||||
with a structured `code` field threaded through `ImportResult` from the parse layer.
|
||||
|
||||
Three changes:
|
||||
1. `src/core/import-file.ts:362` — call `parseMarkdown(content, relativePath, { validate: true, expectedSlug })`
|
||||
so `parsed.errors[0].code` is populated.
|
||||
2. `src/core/import-file.ts` — add `code?: string` to `ImportResult`. Promote the
|
||||
structured code (or `'SLUG_MISMATCH'` when the existing expectedSlug check trips)
|
||||
into the result envelope alongside `error`.
|
||||
3. `src/commands/sync.ts:488` — extend `failedFiles` shape with `code?: string`.
|
||||
`recordSyncFailures` already accepts the field; the only thing missing is the
|
||||
capture site populating it.
|
||||
4. `src/core/sync.ts:classifyErrorCode` — keep as a fallback for un-coded errors
|
||||
(DB exceptions, generic catches). Primary path reads the structured code.
|
||||
|
||||
**Why:** The repo already has `ParseValidationCode` + `ParseValidationError` in
|
||||
`src/core/markdown.ts:5-18`, and three other consumers (`src/commands/lint.ts:72`,
|
||||
`src/commands/frontmatter.ts:148`, `src/core/brain-writer.ts:314`) read structured
|
||||
errors directly. Sync is the outlier — it calls `parseMarkdown` without validation
|
||||
and reverse-engineers codes via regex. PR #501 shipped that regex out of pragmatism;
|
||||
this TODO removes ~50% of `classifyErrorCode` and eliminates a class of false-positives.
|
||||
|
||||
**Pros:**
|
||||
- One source of truth for parse codes (the enum in `markdown.ts`).
|
||||
- Eliminates regex fragility — adding a new validation code in `markdown.ts`
|
||||
automatically flows to sync without a new regex.
|
||||
- Closes the case where canonical messages (`File is empty...`, `No closing ---...`)
|
||||
don't match aspirational regex patterns.
|
||||
|
||||
**Cons:** Touches `ImportResult` interface, which ripples through `src/commands/import.ts:105`,
|
||||
`src/commands/sync.ts:498-510`, `src/core/cycle.ts`, brain-writer reconciler.
|
||||
|
||||
**Context:** PR #501 documented this as P3 in the eng review at
|
||||
`~/.claude/plans/then-codex-synchronous-toucan.md`. Codex's outside-voice review
|
||||
agreed independently. The fix is small — ~50 lines including tests + downstream
|
||||
call sites — and it's the correct architectural endpoint.
|
||||
|
||||
**Effort:** M (human: ~2 hr / CC: ~20 min).
|
||||
|
||||
**Depends on / blocked by:** Nothing.
|
||||
|
||||
### CHANGELOG migration note for `acknowledgeSyncFailures()` shape change
|
||||
**Priority:** P0 — required at /ship time
|
||||
|
||||
**What:** When PR #501 ships, the release CHANGELOG entry MUST include this
|
||||
`### For contributors` block:
|
||||
|
||||
```markdown
|
||||
### For contributors
|
||||
|
||||
`acknowledgeSyncFailures()` now returns `{count, summary}` instead of `number`.
|
||||
If you import this directly from `gbrain/sync`, replace `n` with `result.count`
|
||||
and use `result.summary` for the new code-grouped breakdown.
|
||||
```
|
||||
|
||||
**Why:** The function is exported from `src/core/sync.ts:433` and reachable via
|
||||
the package exports map. External TS consumers (gbrain-evals, host agent forks)
|
||||
that imported it got `number` and now get an object — silent type break.
|
||||
|
||||
**Effort:** XS (human: ~1 min). Just don't forget.
|
||||
|
||||
**Depends on / blocked by:** PR #501 ship.
|
||||
|
||||
### Concurrent-safe ack of `~/.gbrain/sync-failures.jsonl`
|
||||
**Priority:** P3
|
||||
|
||||
**What:** Two concurrent `gbrain sync` runs hitting `acknowledgeSyncFailures()`
|
||||
can clobber each other. The function does a whole-file `writeFileSync` rewrite
|
||||
(`src/core/sync.ts:433-455`); `recordSyncFailures()` does independent
|
||||
`appendFileSync` (`src/core/sync.ts:395-416`). Concurrent ack + append can lose rows.
|
||||
|
||||
**Why:** Pre-existing — predates PR #501. Real risk only on autopilot setups where
|
||||
multiple sync invocations might overlap (rare today, more likely as multi-source
|
||||
sync matures).
|
||||
|
||||
**Fix sketch:** Atomic rename pattern (write to `sync-failures.jsonl.tmp`, then
|
||||
`renameSync`) plus a file lock for the read-modify-write cycle. Or move the
|
||||
acknowledged-set to the DB.
|
||||
|
||||
**Effort:** S (human: ~1 hr / CC: ~10 min).
|
||||
|
||||
**Depends on / blocked by:** Nothing.
|
||||
|
||||
## test-infra
|
||||
|
||||
### Parallel-load timeout flake on v0.21 PGLite-heavy tests
|
||||
**Priority:** P0
|
||||
|
||||
**What:** 22 tests added in v0.21.0 (Code Cathedral II) consistently fail in the full `bun test` run with timeout-pattern elapsed times of 7-10s, but pass in isolation. Every failing test calls `engine.initSchema()` in `beforeAll` without a timeout extension. Under parallel load (168 test files now run concurrently after v0.21 added ~24 new files), `initSchema` exceeds bun's default 5s `beforeAll` timeout.
|
||||
|
||||
Affected files include (non-exhaustive): `test/sync-strategy.test.ts`, `test/cathedral-ii-brainbench.test.ts`, `test/code-edges.test.ts`, `test/reindex-code.test.ts`, `test/reconcile-links.test.ts`, `test/two-pass.test.ts`, `test/parent-symbol-path.test.ts`, `test/pglite-v0_19.test.ts`.
|
||||
|
||||
**Why:** Currently triaged as "skip pre-existing, ship anyway" but that's not a real fix. Blocks /ship for anyone whose CHANGELOG-time test run sees them.
|
||||
|
||||
**Pros:** Fixing it lets /ship run cleanly without manual triage every release.
|
||||
|
||||
**Cons:** ~22 file edits adding `beforeAll(async () => {...}, 30000)` is mechanical but dull.
|
||||
|
||||
**Context:** Same pattern fixed in v0.20.5 wave for `test/e2e/minions-shell-pglite.test.ts`. Single-file repro: each fails in `bun test`, passes in `bun test <file>`. Reproduces with my changes stashed, so it's on master.
|
||||
|
||||
**Effort:** S (human: ~30 min / CC: ~5 min). Mechanical: grep for `beforeAll(async () => {` in affected files, add `, 30000)` argument.
|
||||
|
||||
**Depends on / blocked by:** Nothing.
|
||||
|
||||
## resolver / check-resolvable (v0.22.4 follow-ups)
|
||||
|
||||
### D10 — Extend `check-resolvable` to parse RESOLVER.md disambiguation rules
|
||||
**Priority:** P2
|
||||
|
||||
**What:** Extend `src/core/check-resolvable.ts:357-390` to parse a structured
|
||||
disambiguation block in `RESOLVER.md` (e.g. a `## Disambiguation rules`
|
||||
numbered list with parseable `<trigger>` → `<winning-skill>` shape) and treat
|
||||
resolved overlaps as non-issues. Then the action message at
|
||||
`src/core/check-resolvable.ts:388` ("Add disambiguation rule in RESOLVER.md OR
|
||||
narrow triggers") stops lying about the OR — currently only the second branch
|
||||
silences the warning.
|
||||
|
||||
**Why:** The current MECE-overlap fix path forces authors to delete user-facing
|
||||
triggers from skill frontmatter. That's wrong for cases where two skills
|
||||
legitimately respond to the same phrase under different contexts (e.g.
|
||||
"citation audit" → focused fix vs broader brain health). A real
|
||||
disambiguation parser would let `RESOLVER.md` carry the resolution while
|
||||
keeping both skills' triggers intact for chaining.
|
||||
|
||||
**Pros:**
|
||||
- The action message stops misleading users.
|
||||
- v0.22.4 D2 used the "narrow triggers" path because the disambiguation
|
||||
parser doesn't exist yet; landing this would let v0.23+ keep dual triggers
|
||||
for genuinely-overlapping skills.
|
||||
- Aligns RESOLVER.md's stated role (the dispatcher) with what the checker
|
||||
actually reads.
|
||||
|
||||
**Cons:**
|
||||
- Introduces a new `RESOLVER.md` syntactic contract that other tooling now
|
||||
has to respect (parser, lint, downstream forks reading the same file).
|
||||
- Risk of false-positive resolution if the parser is loose.
|
||||
- ~80 lines of parser + tests; not blocking anything in v0.22.4.
|
||||
|
||||
**Context:**
|
||||
- The "OR" in the action message is misleading today. Confirmed at
|
||||
`src/core/check-resolvable.ts:388`.
|
||||
- The MECE detector loop is at `src/core/check-resolvable.ts:357-390`.
|
||||
- The disambiguation rules already exist as prose in
|
||||
`skills/RESOLVER.md` (the citation-audit row added in v0.22.4 is the
|
||||
pattern). They're agent-facing routing hints today, not parsed structure.
|
||||
|
||||
**Effort:** S (human: ~4-6 hours / CC: ~30 min for parser + 12-16 test cases).
|
||||
|
||||
**Depends on / blocked by:** Nothing.
|
||||
|
||||
## code-indexing (v0.21.0 Cathedral II follow-ups)
|
||||
|
||||
### B2 — Magika auto-detect for extension-less files (Layer 9 deferred)
|
||||
@@ -669,135 +473,3 @@ iteration's residuals.
|
||||
|
||||
### Implement AWS Signature V4 for S3 storage backend
|
||||
**Completed:** v0.6.0 (2026-04-10) — replaced with @aws-sdk/client-s3 for proper SigV4 signing.
|
||||
|
||||
### Caller-opt-in retry for `executeRaw` (D3 follow-up from v0.22.1)
|
||||
**What:** Add `PostgresEngine.executeRawIdempotent(sql, params)` (or a `{retry: true}` parameter flag on `executeRaw`) so callers explicitly opt into auto-retry for statements they know are idempotent. Audit existing call sites and migrate the read-only ones (search, page fetches, etc.) to the new method.
|
||||
|
||||
**Why:** Closes the gap left by D3's drop-the-wrapper decision in v0.22.1. The original #406 wrapped `executeRaw` in a regex-gated retry that was unsound for writable CTEs and side-effecting SELECTs. Recovery moved up to the supervisor watchdog, but per-call recovery for reads (the bulk of `executeRaw` traffic from MCP, search, page fetches) is gone. A caller-opt-in flag puts the idempotency decision where it belongs (at the call site, with full statement context).
|
||||
|
||||
**Pros:** Restores per-call auto-recovery for reads without the phantom-write risk on mutations. Explicit > clever: each call site declares its own idempotency posture. Future caller-added mutations get safe-by-default behavior.
|
||||
|
||||
**Cons:** Touches every existing `executeRaw` call site (~25). Requires careful audit — accidentally tagging a mutation as idempotent re-introduces the phantom-write bug.
|
||||
|
||||
**Context:** Codex F3 demonstrated that `READ_ONLY_PREFIX = /^(\s|--.*\n)*(SELECT|WITH)\b/i` is unsound — `WITH x AS (UPDATE … RETURNING …) SELECT …` matches the prefix but updates a row; `SELECT pg_advisory_xact_lock(...)` is a SELECT with side effects. The plan-eng-review wrap-up in `~/.claude/plans/system-instruction-you-are-working-tender-horizon.md` has the full discussion.
|
||||
|
||||
**Effort estimate:** M (human: ~1 day / CC: ~30 min including call-site audit).
|
||||
**Priority:** P2 — current behavior (no retry, supervisor recovers within ~3 min) is acceptable but per-call recovery is a real ergonomic win.
|
||||
**Depends on:** Nothing.
|
||||
|
||||
### Replace `walkMarkdownFiles` with `engine.getAllSlugs()` in `extractForSlugs` (F1 follow-up from v0.22.1)
|
||||
**What:** The cycle path's `extractForSlugs()` at `src/commands/extract.ts:455` still does a `walkMarkdownFiles(brainDir)` to build the `allSlugs` set for link resolution. On a 54K-page brain that's a single `readdir` traversal (~hundreds of ms — acceptable, dominated by the file-content-read elimination from #417). But `engine.getAllSlugs()` exists at `extract.ts:728` and produces the same set via a single SQL query (~tens of ms).
|
||||
|
||||
**Why:** Eliminates the residual directory walk on every cycle. Codex F1 noted that the v0.22.1 plan's "cycle never re-walks the whole tree again" claim was overstated — it stops READING file contents but still walks the directory. This TODO closes that gap honestly.
|
||||
|
||||
**Pros:** Cycle becomes O(slugs sync touched), not O(total brain size). No more readdir on a growing brain. ~5 LOC change.
|
||||
|
||||
**Cons:** Crosses an FS-vs-DB consistency boundary in the FS-source extract path. Edge case: a file deleted from disk but still in DB. Currently `extractForSlugs` skips with `if (!existsSync(fullPath)) continue` — unchanged. But if a markdown file references a slug whose page exists in DB but file was deleted, the link would resolve via DB but the original extractor caught it. Needs a careful test for this case.
|
||||
|
||||
**Context:** Codex plan-review during v0.22.1 wrap, verified at `extract.ts:455-456`. The plan-eng-review session captured the rationale.
|
||||
|
||||
**Effort estimate:** S (human: ~2 hr / CC: ~10 min including the consistency-edge-case test).
|
||||
**Priority:** P3 — pure perf, no correctness gap.
|
||||
**Depends on:** Nothing.
|
||||
|
||||
### `err.code`-based connection-error matching in `postgres-engine.ts` (B1 follow-up from v0.22.1)
|
||||
**What:** The CONNECTION_ERROR_PATTERNS array (~12 strings: `ECONNREFUSED`, `connection terminated`, `password authentication failed`, etc.) matched against `err.message` and `err.code`. Replace with structured matching against `err.code` only, using postgres.js's typed error classes (`PostgresError` with structured codes).
|
||||
|
||||
**Why:** String matching against error messages breaks on library upgrades (postgres.js could change its error message phrasing without bumping major). Code matching is durable. The Layer 1 cleanup follows: gbrain itself doesn't define connection-error codes; it should defer to postgres.js's classification.
|
||||
|
||||
**Pros:** More durable across library updates. Less code (drop the 12-string array). Follows the typed-errors pattern v0.21.0 introduced (`src/core/errors.ts`).
|
||||
|
||||
**Cons:** Requires verifying which `err.code` values postgres.js actually exposes for each connection-failure mode. May need fallback to message-substring matching for codes that postgres.js doesn't surface.
|
||||
|
||||
**Context:** Section 2/B1 from the v0.22.1 plan-eng-review. After D3 dropped the per-call retry, `isConnectionError` is no longer in the hot path — only the supervisor watchdog cares about classifying connection errors, and it currently catches *anything*. This TODO is a cleanup pass when someone next touches that surface.
|
||||
|
||||
**Effort estimate:** S (human: ~2 hr / CC: ~10 min).
|
||||
**Priority:** P3.
|
||||
**Depends on:** The above caller-opt-in retry (#1) is the natural co-lander since both touch the same error-classification surface.
|
||||
|
||||
## remote MCP / HTTP transport (v0.22.7 follow-ups)
|
||||
|
||||
### Audit-log write amplification on rejected `/mcp` traffic
|
||||
**What:** `src/mcp/http-transport.ts` writes a row to `mcp_request_log` for every
|
||||
incoming `/mcp` request, including rate-limited (429), oversized (413), and
|
||||
auth-failed (401) traffic. Under sustained attack the IP rate limit caps audit
|
||||
writes per IP at 30/min, but at scale (10K distinct IPs) that's still 300K
|
||||
inserts/min. Two follow-ups: (1) instrument the audit-write rate so we can see
|
||||
the actual production volume; (2) consider a separate "rejected" table or
|
||||
sampling for failed-auth rows so the success-path audit table doesn't get
|
||||
swamped.
|
||||
|
||||
**Why:** Codex flagged this during the v0.22.7 ship adversarial review. We kept
|
||||
the full audit on purpose — forensic data of an attack is valuable — but want
|
||||
to revisit once we have real volume numbers.
|
||||
|
||||
**Pros:** Bounds DB write volume under attack. Keeps the success-path audit
|
||||
table small enough for fast queries.
|
||||
|
||||
**Cons:** Adds a second table or a sampling rule. Not free complexity. Probably
|
||||
not worth it until production hits a real attack pattern.
|
||||
|
||||
**Context:** `src/mcp/http-transport.ts:222,235,245` (the three audit-on-reject
|
||||
call sites) + `src/schema.sql:342` (the unbounded table).
|
||||
|
||||
**Effort estimate:** M (human: ~half day / CC: ~30 min once we have volume data).
|
||||
**Priority:** P3 — wait for evidence.
|
||||
**Depends on:** Production telemetry on `mcp_request_log` insert rate.
|
||||
|
||||
### `validateParams` doesn't check enum values or array item types
|
||||
**What:** `src/mcp/dispatch.ts:27` (extracted from `src/mcp/server.ts` in
|
||||
v0.22.7) only checks top-level JS types. Operations declare `enum` constraints
|
||||
(e.g. `direction: 'in' | 'out' | 'both'`) and array `items: { type: ... }`
|
||||
schemas in `src/core/operations.ts`, but `validateParams` ignores both. Bad
|
||||
inputs still reach handlers — concretely, an invalid `direction` falls through
|
||||
the engine's else branch at `src/core/postgres-engine.ts:954`, widening
|
||||
traversal unexpectedly; malformed `pages_updated` arrays could be written as
|
||||
garbage JSONB.
|
||||
|
||||
**Why:** Codex flagged this during the v0.22.7 ship adversarial review. The
|
||||
validator was lifted verbatim from the pre-existing stdio path during the
|
||||
dispatch.ts extraction — same gap exists on the stdio MCP server today, so
|
||||
this isn't a v0.22.7 regression. Still worth tightening, since "shared
|
||||
validation" is now the architectural guarantee both transports rely on.
|
||||
|
||||
**Pros:** Better defense-in-depth at the MCP boundary. Catches malformed agent
|
||||
inputs before the engine layer has to.
|
||||
|
||||
**Cons:** Need to walk every operation's param schema and decide which enum
|
||||
violations are user-facing errors vs internal bugs. May need a typed Zod-style
|
||||
schema layer to do this cleanly.
|
||||
|
||||
**Context:** `src/mcp/dispatch.ts:27` + `src/core/operations.ts` (param defs).
|
||||
Same gap pre-existed on stdio MCP path.
|
||||
|
||||
**Effort estimate:** M (human: ~half day / CC: ~30 min if we use the existing
|
||||
ParamDef shape; XL if a Zod migration is the chosen direction).
|
||||
**Priority:** P2.
|
||||
**Depends on:** Whether we want to keep the lightweight ParamDef shape or
|
||||
migrate to typed schemas.
|
||||
|
||||
### Streaming MCP tool support (re-add SSE based on Accept header)
|
||||
**What:** v0.22.7 dropped SSE entirely from `gbrain serve --http` because no
|
||||
current MCP tool streams. When the first streaming tool ships (long-running
|
||||
agent delegation as an MCP tool, `resources/subscribe`, `sampling/createMessage`),
|
||||
re-add SSE in `/mcp` based on the `Accept` header per the Streamable HTTP
|
||||
transport spec. ~30 lines + spec compliance test.
|
||||
|
||||
**Why:** Removing SSE simplified the v0.22.7 transport (one response path,
|
||||
fewer test cases). Adding it back when actually needed is cheap and keeps the
|
||||
code lean in the meantime.
|
||||
|
||||
**Effort estimate:** S (human: ~2 hr / CC: ~15 min).
|
||||
**Priority:** P3 — wait for the first streaming tool.
|
||||
**Depends on:** A streaming MCP tool actually existing.
|
||||
|
||||
### `access_tokens.scopes` enforcement
|
||||
**What:** The `access_tokens` schema has had a `scopes TEXT[]` column since
|
||||
migration v4 (`src/core/migrate.ts:84`), but nothing enforces it. v0.22.7's
|
||||
`gbrain auth create` doesn't accept a `--scopes` flag, and `dispatchToolCall`
|
||||
doesn't gate on scopes. Adding per-tool scope enforcement would let
|
||||
"claude-desktop-readonly" and "ingest-only" tokens exist.
|
||||
|
||||
**Effort estimate:** M (human: ~1 day / CC: ~30 min for the schema-aware gate).
|
||||
**Priority:** P3.
|
||||
**Depends on:** Nothing.
|
||||
|
||||
@@ -20,7 +20,6 @@
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/bun": "latest",
|
||||
"bun-types": "^1.3.13",
|
||||
"typescript": "^5.6.0",
|
||||
},
|
||||
},
|
||||
@@ -221,7 +220,7 @@
|
||||
|
||||
"@types/bun": ["@types/bun@1.3.11", "", { "dependencies": { "bun-types": "1.3.11" } }, "sha512-5vPne5QvtpjGpsGYXiFyycfpDF2ECyPcTSsFBMa0fraoxiQyMJ3SmuQIGhzPg2WJuWxVBoxWJ2kClYTcw/4fAg=="],
|
||||
|
||||
"@types/node": ["@types/node@25.5.2", "", { "dependencies": { "undici-types": "~7.18.0" } }, "sha512-tO4ZIRKNC+MDWV4qKVZe3Ql/woTnmHDr5JD8UI5hn2pwBrHEwOEMZK7WlNb5RKB6EoJ02gwmQS9OrjuFnZYdpg=="],
|
||||
"@types/node": ["@types/node@18.19.130", "", { "dependencies": { "undici-types": "~5.26.4" } }, "sha512-GRaXQx6jGfL8sKfaIDD6OupbIHBr9jv7Jnaml9tB7l4v068PAOXqfcujMMo5PhbIs6ggR1XODELqahT2R8v0fg=="],
|
||||
|
||||
"@types/node-fetch": ["@types/node-fetch@2.6.13", "", { "dependencies": { "@types/node": "*", "form-data": "^4.0.4" } }, "sha512-QGpRVpzSaUs30JBSGPjOg4Uveu384erbHBoT1zeONvyCfwQxIkUshLAOqN/k9EjGviPRmWTTe6aH2qySWKTVSw=="],
|
||||
|
||||
@@ -243,7 +242,7 @@
|
||||
|
||||
"bowser": ["bowser@2.14.1", "", {}, "sha512-tzPjzCxygAKWFOJP011oxFHs57HzIhOEracIgAePE4pqB3LikALKnSzUyU4MGs9/iCEUuHlAJTjTc5M+u7YEGg=="],
|
||||
|
||||
"bun-types": ["bun-types@1.3.13", "", { "dependencies": { "@types/node": "*" } }, "sha512-QXKeHLlOLqQX9LgYaHJfzdBaV21T63HhFJnvuRCcjZiaUDpbs5ED1MgxbMra71CsryN/1dAoXuJJJwIv/2drVA=="],
|
||||
"bun-types": ["bun-types@1.3.11", "", { "dependencies": { "@types/node": "*" } }, "sha512-1KGPpoxQWl9f6wcZh57LvrPIInQMn2TQ7jsgxqpRzg+l0QPOFvJVH7HmvHo/AiPgwXy+/Thf6Ov3EdVn1vOabg=="],
|
||||
|
||||
"bytes": ["bytes@3.1.2", "", {}, "sha512-/Nf7TyzTx6S3yRJObOAV7956r8cr2+Oj8AC5dt8wSP3BQAoeX58NoHyCU8P8zGkNXStjTSi6fzO6F0pBdcYbEg=="],
|
||||
|
||||
@@ -467,7 +466,7 @@
|
||||
|
||||
"typescript": ["typescript@5.9.3", "", { "bin": { "tsc": "bin/tsc", "tsserver": "bin/tsserver" } }, "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw=="],
|
||||
|
||||
"undici-types": ["undici-types@7.18.2", "", {}, "sha512-AsuCzffGHJybSaRrmr5eHr81mwJU3kjw6M+uprWvCXiNeN9SOGwQ3Jn8jb8m3Z6izVgknn1R0FTCEAP2QrLY/w=="],
|
||||
"undici-types": ["undici-types@5.26.5", "", {}, "sha512-JlCMO+ehdEIKqlFxk6IfVoAUVmgz7cU7zD/h9XZ0qzeosSHmUJVOzSQvvYSYWXkFXC+IfLKSIffhv0sVZup6pA=="],
|
||||
|
||||
"unpipe": ["unpipe@1.0.0", "", {}, "sha512-pjy2bYhSsufwWlKwPc+l3cN7+wuJlK6uz0YdJEOlQDbl6jo/YlPi4mb8agUkVC8BF7V8NuzeyPNqRksA3hztKQ=="],
|
||||
|
||||
@@ -489,31 +488,29 @@
|
||||
|
||||
"zod-to-json-schema": ["zod-to-json-schema@3.25.2", "", { "peerDependencies": { "zod": "^3.25.28 || ^4" } }, "sha512-O/PgfnpT1xKSDeQYSCfRI5Gy3hPf91mKVDuYLUHZJMiDFptvP41MSnWofm8dnCm0256ZNfZIM7DSzuSMAFnjHA=="],
|
||||
|
||||
"@anthropic-ai/sdk/@types/node": ["@types/node@18.19.130", "", { "dependencies": { "undici-types": "~5.26.4" } }, "sha512-GRaXQx6jGfL8sKfaIDD6OupbIHBr9jv7Jnaml9tB7l4v068PAOXqfcujMMo5PhbIs6ggR1XODELqahT2R8v0fg=="],
|
||||
|
||||
"@aws-crypto/sha1-browser/@smithy/util-utf8": ["@smithy/util-utf8@2.3.0", "", { "dependencies": { "@smithy/util-buffer-from": "^2.2.0", "tslib": "^2.6.2" } }, "sha512-R8Rdn8Hy72KKcebgLiv8jQcQkXoLMOGGv5uI1/k0l+snqkOzQ1R0ChUBCxWMlBsFMekWjq0wRudIweFs7sKT5A=="],
|
||||
|
||||
"@aws-crypto/sha256-browser/@smithy/util-utf8": ["@smithy/util-utf8@2.3.0", "", { "dependencies": { "@smithy/util-buffer-from": "^2.2.0", "tslib": "^2.6.2" } }, "sha512-R8Rdn8Hy72KKcebgLiv8jQcQkXoLMOGGv5uI1/k0l+snqkOzQ1R0ChUBCxWMlBsFMekWjq0wRudIweFs7sKT5A=="],
|
||||
|
||||
"@aws-crypto/util/@smithy/util-utf8": ["@smithy/util-utf8@2.3.0", "", { "dependencies": { "@smithy/util-buffer-from": "^2.2.0", "tslib": "^2.6.2" } }, "sha512-R8Rdn8Hy72KKcebgLiv8jQcQkXoLMOGGv5uI1/k0l+snqkOzQ1R0ChUBCxWMlBsFMekWjq0wRudIweFs7sKT5A=="],
|
||||
|
||||
"@types/bun/bun-types": ["bun-types@1.3.11", "", { "dependencies": { "@types/node": "*" } }, "sha512-1KGPpoxQWl9f6wcZh57LvrPIInQMn2TQ7jsgxqpRzg+l0QPOFvJVH7HmvHo/AiPgwXy+/Thf6Ov3EdVn1vOabg=="],
|
||||
"@types/node-fetch/@types/node": ["@types/node@25.5.2", "", { "dependencies": { "undici-types": "~7.18.0" } }, "sha512-tO4ZIRKNC+MDWV4qKVZe3Ql/woTnmHDr5JD8UI5hn2pwBrHEwOEMZK7WlNb5RKB6EoJ02gwmQS9OrjuFnZYdpg=="],
|
||||
|
||||
"bun-types/@types/node": ["@types/node@25.5.2", "", { "dependencies": { "undici-types": "~7.18.0" } }, "sha512-tO4ZIRKNC+MDWV4qKVZe3Ql/woTnmHDr5JD8UI5hn2pwBrHEwOEMZK7WlNb5RKB6EoJ02gwmQS9OrjuFnZYdpg=="],
|
||||
|
||||
"form-data/mime-types": ["mime-types@2.1.35", "", { "dependencies": { "mime-db": "1.52.0" } }, "sha512-ZDY+bPm5zTTF+YpCrAU9nK0UgICYPT0QtT1NZWFv4s++TNkcgVaT0g6+4R2uI4MjQjzysHB1zxuWL50hzaeXiw=="],
|
||||
|
||||
"openai/@types/node": ["@types/node@18.19.130", "", { "dependencies": { "undici-types": "~5.26.4" } }, "sha512-GRaXQx6jGfL8sKfaIDD6OupbIHBr9jv7Jnaml9tB7l4v068PAOXqfcujMMo5PhbIs6ggR1XODELqahT2R8v0fg=="],
|
||||
|
||||
"@anthropic-ai/sdk/@types/node/undici-types": ["undici-types@5.26.5", "", {}, "sha512-JlCMO+ehdEIKqlFxk6IfVoAUVmgz7cU7zD/h9XZ0qzeosSHmUJVOzSQvvYSYWXkFXC+IfLKSIffhv0sVZup6pA=="],
|
||||
|
||||
"@aws-crypto/sha1-browser/@smithy/util-utf8/@smithy/util-buffer-from": ["@smithy/util-buffer-from@2.2.0", "", { "dependencies": { "@smithy/is-array-buffer": "^2.2.0", "tslib": "^2.6.2" } }, "sha512-IJdWBbTcMQ6DA0gdNhh/BwrLkDR+ADW5Kr1aZmd4k3DIF6ezMV4R2NIAmT08wQJ3yUK82thHWmC/TnK/wpMMIA=="],
|
||||
|
||||
"@aws-crypto/sha256-browser/@smithy/util-utf8/@smithy/util-buffer-from": ["@smithy/util-buffer-from@2.2.0", "", { "dependencies": { "@smithy/is-array-buffer": "^2.2.0", "tslib": "^2.6.2" } }, "sha512-IJdWBbTcMQ6DA0gdNhh/BwrLkDR+ADW5Kr1aZmd4k3DIF6ezMV4R2NIAmT08wQJ3yUK82thHWmC/TnK/wpMMIA=="],
|
||||
|
||||
"@aws-crypto/util/@smithy/util-utf8/@smithy/util-buffer-from": ["@smithy/util-buffer-from@2.2.0", "", { "dependencies": { "@smithy/is-array-buffer": "^2.2.0", "tslib": "^2.6.2" } }, "sha512-IJdWBbTcMQ6DA0gdNhh/BwrLkDR+ADW5Kr1aZmd4k3DIF6ezMV4R2NIAmT08wQJ3yUK82thHWmC/TnK/wpMMIA=="],
|
||||
|
||||
"form-data/mime-types/mime-db": ["mime-db@1.52.0", "", {}, "sha512-sPU4uV7dYlvtWJxwwxHD0PuihVNiE7TyAbQ5SWxDCB9mUYvOgroQOwYQQOKPJ8CIbE+1ETVlOoK1UC2nU3gYvg=="],
|
||||
"@types/node-fetch/@types/node/undici-types": ["undici-types@7.18.2", "", {}, "sha512-AsuCzffGHJybSaRrmr5eHr81mwJU3kjw6M+uprWvCXiNeN9SOGwQ3Jn8jb8m3Z6izVgknn1R0FTCEAP2QrLY/w=="],
|
||||
|
||||
"openai/@types/node/undici-types": ["undici-types@5.26.5", "", {}, "sha512-JlCMO+ehdEIKqlFxk6IfVoAUVmgz7cU7zD/h9XZ0qzeosSHmUJVOzSQvvYSYWXkFXC+IfLKSIffhv0sVZup6pA=="],
|
||||
"bun-types/@types/node/undici-types": ["undici-types@7.18.2", "", {}, "sha512-AsuCzffGHJybSaRrmr5eHr81mwJU3kjw6M+uprWvCXiNeN9SOGwQ3Jn8jb8m3Z6izVgknn1R0FTCEAP2QrLY/w=="],
|
||||
|
||||
"form-data/mime-types/mime-db": ["mime-db@1.52.0", "", {}, "sha512-sPU4uV7dYlvtWJxwwxHD0PuihVNiE7TyAbQ5SWxDCB9mUYvOgroQOwYQQOKPJ8CIbE+1ETVlOoK1UC2nU3gYvg=="],
|
||||
|
||||
"@aws-crypto/sha1-browser/@smithy/util-utf8/@smithy/util-buffer-from/@smithy/is-array-buffer": ["@smithy/is-array-buffer@2.2.0", "", { "dependencies": { "tslib": "^2.6.2" } }, "sha512-GGP3O9QFD24uGeAXYUjwSTXARoqpZykHadOmA8G5vfJPK0/DC67qa//0qvqrJzL1xc8WQWX7/yc7fwudjPHPhA=="],
|
||||
|
||||
|
||||
@@ -458,75 +458,6 @@ in depth, not the primary boundary.
|
||||
|
||||
---
|
||||
|
||||
## v0.22.4 — frontmatter-guard adoption
|
||||
|
||||
### 1. Stop hand-rolling frontmatter validators
|
||||
|
||||
If your fork has scripts that call `js-yaml` directly to validate brain page
|
||||
frontmatter, replace them with `gbrain frontmatter validate` calls. The CLI
|
||||
covers the seven canonical error classes and ships a `--json` envelope that's
|
||||
stable across releases.
|
||||
|
||||
```diff
|
||||
- # Custom validator script
|
||||
- node scripts/validate-frontmatter.mjs <path>
|
||||
+ gbrain frontmatter validate <path> --json
|
||||
```
|
||||
|
||||
For consumers that need the validator inside another script, import from
|
||||
gbrain's `markdown` export instead of duplicating logic:
|
||||
|
||||
```ts
|
||||
import { parseMarkdown } from 'gbrain/markdown';
|
||||
|
||||
const parsed = parseMarkdown(content, filePath, { validate: true, expectedSlug });
|
||||
for (const err of parsed.errors ?? []) {
|
||||
// err.code: MISSING_OPEN | MISSING_CLOSE | YAML_PARSE | SLUG_MISMATCH |
|
||||
// NULL_BYTES | NESTED_QUOTES | EMPTY_FRONTMATTER
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Drop any references to `lib/brain-writer.mjs`
|
||||
|
||||
If your fork's skills or scripts referenced an aspirational
|
||||
`lib/brain-writer.mjs` (it never shipped — the spec was in PR #392 and never
|
||||
landed), replace those references with the gbrain CLI. The `frontmatter-guard`
|
||||
skill lives at `skills/frontmatter-guard/SKILL.md` and points at
|
||||
`gbrain frontmatter validate` / `audit` / `install-hook`.
|
||||
|
||||
### 3. Wire the doctor subcheck into your health pipeline
|
||||
|
||||
`gbrain doctor` now reports `frontmatter_integrity` automatically. If your
|
||||
fork has a custom health pipeline (e.g. a daily Slack post about brain
|
||||
health), pull from `gbrain doctor --json` and surface the
|
||||
`frontmatter_integrity` row counts.
|
||||
|
||||
### 4. (Optional) Install the pre-commit hook on brain repos
|
||||
|
||||
For sources backed by git, the v0.22.4 install-hook helper drops a
|
||||
pre-commit script that blocks commits with malformed frontmatter:
|
||||
|
||||
```bash
|
||||
gbrain frontmatter install-hook
|
||||
```
|
||||
|
||||
Skip this if your brain isn't a git repo or if your downstream agent already
|
||||
enforces validation at write time. See `docs/integrations/pre-commit.md` for
|
||||
the full recipe.
|
||||
|
||||
### 5. Migration ergonomics — read pending-host-work.jsonl
|
||||
|
||||
After `gbrain apply-migrations --yes` runs the v0.22.4 audit, your agent
|
||||
should read `~/.gbrain/migrations/pending-host-work.jsonl` (filter to
|
||||
`migration === "0.22.4"`) and walk each entry's `command` field. Each entry
|
||||
points to a per-source `gbrain frontmatter validate <source_path> --fix`
|
||||
command — surface counts to the user, get explicit consent, then run.
|
||||
|
||||
The migration is **audit-only**. It never mutates brain content during
|
||||
`apply-migrations`. Your agent runs the fix command with user consent.
|
||||
|
||||
---
|
||||
|
||||
## Future versions
|
||||
|
||||
When gbrain ships a new version, this doc will be updated with the diffs for that
|
||||
|
||||
@@ -1,105 +0,0 @@
|
||||
# Pre-commit hook for brain repos (v0.22.4+)
|
||||
|
||||
`gbrain frontmatter install-hook` installs a git pre-commit hook in your
|
||||
brain source's repo that runs `gbrain frontmatter validate` against staged
|
||||
`.md` and `.mdx` files. Malformed frontmatter blocks the commit. Bypass with
|
||||
`git commit --no-verify`.
|
||||
|
||||
## What the hook catches
|
||||
|
||||
The same seven validation classes the `frontmatter-guard` skill and
|
||||
`gbrain doctor`'s `frontmatter_integrity` subcheck report:
|
||||
|
||||
| Code | What it catches |
|
||||
|-------------------|---------------------------------------------------------------------|
|
||||
| `MISSING_OPEN` | File doesn't start with `---` |
|
||||
| `MISSING_CLOSE` | No closing `---` before first heading |
|
||||
| `YAML_PARSE` | YAML failed to parse (syntax or structure) |
|
||||
| `SLUG_MISMATCH` | `slug:` in frontmatter doesn't match path-derived slug |
|
||||
| `NULL_BYTES` | Binary corruption (`\x00`) anywhere in the content |
|
||||
| `NESTED_QUOTES` | `title: "outer "inner" outer"` shape that breaks YAML |
|
||||
| `EMPTY_FRONTMATTER` | `---` ... `---` with nothing meaningful between |
|
||||
|
||||
## Install
|
||||
|
||||
For all registered sources that are git repos:
|
||||
|
||||
```bash
|
||||
gbrain frontmatter install-hook
|
||||
```
|
||||
|
||||
For one source:
|
||||
|
||||
```bash
|
||||
gbrain frontmatter install-hook --source <id>
|
||||
```
|
||||
|
||||
For force-overwrite of an existing pre-commit hook (writes a `.bak`):
|
||||
|
||||
```bash
|
||||
gbrain frontmatter install-hook --force
|
||||
```
|
||||
|
||||
The hook lands at `<source>/.githooks/pre-commit`. If `core.hooksPath` is
|
||||
unset, the install also runs `git config core.hooksPath .githooks` so the
|
||||
hook is picked up without manual git config.
|
||||
|
||||
## Bypass
|
||||
|
||||
Standard git escape hatch:
|
||||
|
||||
```bash
|
||||
git commit --no-verify
|
||||
```
|
||||
|
||||
This skips ALL pre-commit hooks. Use sparingly — the next time the user
|
||||
runs `gbrain doctor`, the issues will surface.
|
||||
|
||||
## Uninstall
|
||||
|
||||
```bash
|
||||
gbrain frontmatter install-hook --uninstall
|
||||
```
|
||||
|
||||
If a `.bak` was saved during install, it's restored as the active hook.
|
||||
Otherwise the hook is removed cleanly.
|
||||
|
||||
## Behavior on machines without gbrain installed
|
||||
|
||||
The hook script checks for `gbrain` on `$PATH`. When missing, it prints a
|
||||
one-line warning to stderr and exits 0 — commits aren't blocked just because
|
||||
a developer hasn't installed gbrain locally. Once gbrain is installed, the
|
||||
hook resumes blocking malformed pages.
|
||||
|
||||
## For downstream agent forks
|
||||
|
||||
If your fork (Wintermute, Hermes, OpenClaw) wraps gbrain in a host repo
|
||||
that's not the brain repo itself, you may want a separate hook strategy:
|
||||
|
||||
- **Brain repo IS the host repo** (gbrain skills + brain pages in one repo):
|
||||
install via `gbrain frontmatter install-hook` as above.
|
||||
- **Brain repo is a separate registered source** (e.g. `~/brain` registered
|
||||
as a source, host repo is `~/agent-fork`): install in the brain repo only;
|
||||
agent-fork code doesn't need this hook.
|
||||
- **Brain repo is auto-generated** (e.g. by a sync daemon writing to a
|
||||
bucket): skip the hook entirely; gate at the writer instead via
|
||||
`import { writeBrainPage } from 'gbrain/brain-writer'` (planned in a
|
||||
later release; currently the CLI is the surface).
|
||||
|
||||
## How it fits into the broader frontmatter pipeline
|
||||
|
||||
```
|
||||
agent writes a page git commit doctor scan
|
||||
↓ ↓ ↓
|
||||
[source content] → [pre-commit hook validates] → [frontmatter_integrity check]
|
||||
↓ ↓ ↓
|
||||
raw file on disk blocks malformed commits surfaces existing issues
|
||||
↓
|
||||
`gbrain frontmatter validate
|
||||
<source-path> --fix`
|
||||
(writes .bak backups)
|
||||
```
|
||||
|
||||
The hook is the write-time gate; doctor is the audit gate; the CLI is the
|
||||
fix tool. They share `parseMarkdown(..., {validate:true})` as the single
|
||||
source of truth for what counts as malformed.
|
||||
@@ -1,9 +1,8 @@
|
||||
# Remote MCP Deployment Options
|
||||
|
||||
GBrain's MCP server runs via `gbrain serve` (stdio transport). To make it
|
||||
accessible from other devices and AI clients, run `gbrain serve --http`
|
||||
(built-in HTTP transport with bearer auth, Postgres-only ... see
|
||||
[DEPLOY.md](DEPLOY.md)) behind a public tunnel. Here are your tunnel options.
|
||||
accessible from other devices and AI clients, you need an HTTP wrapper and
|
||||
a public tunnel. Here are your options.
|
||||
|
||||
## ngrok (recommended)
|
||||
|
||||
@@ -14,9 +13,8 @@ accessible from other devices and AI clients, run `gbrain serve --http`
|
||||
# 1. Install ngrok
|
||||
brew install ngrok
|
||||
|
||||
# 2. Start the built-in HTTP transport
|
||||
gbrain serve --http --port 8787
|
||||
# See docs/mcp/DEPLOY.md for token setup
|
||||
# 2. Start your MCP server (behind an HTTP wrapper)
|
||||
# See docs/mcp/DEPLOY.md for the server setup
|
||||
|
||||
# 3. Expose via ngrok
|
||||
ngrok http 8787 --url your-brain.ngrok.app
|
||||
@@ -61,7 +59,6 @@ Both run Bun natively. No bundling, no Deno, no cold start, no timeout limits.
|
||||
| All 30 operations | Yes | Yes | Yes |
|
||||
| Setup time | 5 min | 10 min | 15 min |
|
||||
|
||||
**Note:** `gbrain serve --http` is the built-in HTTP transport (v0.22.7+). Bearer auth
|
||||
against the `access_tokens` table, default-deny CORS, two-bucket rate limit, body cap,
|
||||
per-request audit log. Postgres-only by design (PGLite is local-only). See
|
||||
[DEPLOY.md](DEPLOY.md) and [SECURITY.md](../../SECURITY.md) for env vars and tunables.
|
||||
**Note:** `gbrain serve --http` (built-in HTTP transport) is planned but not yet
|
||||
implemented. Currently, remote MCP requires a custom HTTP wrapper around `gbrain serve`.
|
||||
See [DEPLOY.md](DEPLOY.md) for details.
|
||||
|
||||
@@ -21,7 +21,7 @@ claude mcp add gbrain -t http \
|
||||
```
|
||||
|
||||
Replace `YOUR-DOMAIN` with your ngrok domain and `YOUR_TOKEN` with a token
|
||||
from `gbrain auth create "claude-code"`.
|
||||
from `bun run src/commands/auth.ts create "claude-code"`.
|
||||
|
||||
## Verify
|
||||
|
||||
|
||||
@@ -12,7 +12,7 @@ For Team/Enterprise plans, an org Owner adds the connector:
|
||||
https://YOUR-DOMAIN.ngrok.app/mcp
|
||||
```
|
||||
3. Add Bearer token authentication in Advanced Settings
|
||||
(create one with `gbrain auth create "cowork"`)
|
||||
(create one with `bun run src/commands/auth.ts create "cowork"`)
|
||||
4. Save
|
||||
|
||||
Note: Cowork connects from Anthropic's cloud, not your device. Your server
|
||||
|
||||
@@ -16,7 +16,7 @@ Remote HTTP servers must be added through the GUI.
|
||||
Replace `YOUR-DOMAIN` with your ngrok domain (see
|
||||
[ngrok-tunnel recipe](../../recipes/ngrok-tunnel.md) for setup).
|
||||
5. Set authentication to **Bearer Token** and paste your token
|
||||
(create one with `gbrain auth create "claude-desktop"`)
|
||||
(create one with `bun run src/commands/auth.ts create "claude-desktop"`)
|
||||
6. Save
|
||||
|
||||
## Verify
|
||||
|
||||
+14
-21
@@ -1,13 +1,8 @@
|
||||
# Deploy GBrain Remote MCP Server
|
||||
|
||||
> **v0.22.7+:** Use `gbrain serve --http` for remote access. It includes built-in
|
||||
> bearer token auth, default-deny CORS, two-bucket rate limiting, body cap, and
|
||||
> per-request audit log. **Postgres-only** (PGLite is local-only by design).
|
||||
> See [SECURITY.md](../../SECURITY.md) for env vars and tunable defaults.
|
||||
|
||||
Access your brain from any device, any AI client. GBrain's MCP server runs locally
|
||||
via `gbrain serve` (stdio). For remote access, expose it via the built-in HTTP
|
||||
transport behind a public tunnel.
|
||||
via `gbrain serve` (stdio). For remote access, wrap it in an HTTP server behind a
|
||||
public tunnel.
|
||||
|
||||
## Two Paths
|
||||
|
||||
@@ -18,23 +13,21 @@ gbrain serve
|
||||
```
|
||||
|
||||
Works with Claude Code, Cursor, Windsurf, and any MCP client that supports stdio.
|
||||
No server, no tunnel, no token needed. Works on both PGLite and Postgres engines.
|
||||
No server, no tunnel, no token needed.
|
||||
|
||||
### Remote (any device, any AI client) — Postgres only
|
||||
### Remote (any device, any AI client)
|
||||
|
||||
```
|
||||
Your AI client (Claude Desktop, Perplexity, etc.)
|
||||
→ ngrok tunnel (https://YOUR-DOMAIN.ngrok.app)
|
||||
→ gbrain serve --http (built-in transport with bearer auth)
|
||||
→ Postgres (pooler connection or self-hosted)
|
||||
→ Your HTTP server (wraps gbrain serve)
|
||||
→ Supabase Postgres (via pooler connection string)
|
||||
```
|
||||
|
||||
This requires:
|
||||
1. A Postgres-backed brain (the `access_tokens` table only exists on Postgres;
|
||||
running `gbrain serve --http` against a PGLite install fails fast at startup)
|
||||
2. A machine running `gbrain serve --http`
|
||||
3. A public tunnel (ngrok, Tailscale, or cloud host)
|
||||
4. A bearer token created via `gbrain auth create <name>`
|
||||
1. A machine running `gbrain serve` behind an HTTP wrapper
|
||||
2. A public tunnel (ngrok, Tailscale, or cloud host)
|
||||
3. Bearer token auth for security
|
||||
|
||||
## Remote Setup
|
||||
|
||||
@@ -53,13 +46,13 @@ ngrok http 8787 --url your-brain.ngrok.app # Hobby tier for fixed domain
|
||||
|
||||
```bash
|
||||
# Create a token for each client
|
||||
gbrain auth create "claude-desktop"
|
||||
bun run src/commands/auth.ts create "claude-desktop"
|
||||
|
||||
# List all tokens
|
||||
gbrain auth list
|
||||
bun run src/commands/auth.ts list
|
||||
|
||||
# Revoke a token
|
||||
gbrain auth revoke "claude-desktop"
|
||||
bun run src/commands/auth.ts revoke "claude-desktop"
|
||||
```
|
||||
|
||||
Tokens are per-client. Create one for each device/app. Revoke individually
|
||||
@@ -75,7 +68,7 @@ if compromised. Tokens are stored SHA-256 hashed in your database.
|
||||
### 4. Verify
|
||||
|
||||
```bash
|
||||
gbrain auth test \
|
||||
bun run src/commands/auth.ts test \
|
||||
https://YOUR-DOMAIN.ngrok.app/mcp \
|
||||
--token YOUR_TOKEN
|
||||
```
|
||||
@@ -103,7 +96,7 @@ Funnel, and cloud hosts (Fly.io, Railway).
|
||||
Include the Authorization header: `Authorization: Bearer YOUR_TOKEN`
|
||||
|
||||
**"invalid_token" error**
|
||||
Run `gbrain auth list` to see active tokens.
|
||||
Run `bun run src/commands/auth.ts list` to see active tokens.
|
||||
|
||||
**"service_unavailable" error**
|
||||
Database connection failed. Check your Supabase dashboard for outages.
|
||||
|
||||
@@ -10,7 +10,7 @@ Perplexity Computer supports remote MCP servers with bearer token authentication
|
||||
- **URL:** `https://YOUR-DOMAIN.ngrok.app/mcp`
|
||||
- **Authentication:** API Key / Bearer Token
|
||||
- **Token:** your GBrain access token
|
||||
(create one with `gbrain auth create "perplexity"`)
|
||||
(create one with `bun run src/commands/auth.ts create "perplexity"`)
|
||||
4. Save
|
||||
|
||||
Replace `YOUR-DOMAIN` with your ngrok domain (see
|
||||
|
||||
@@ -1,210 +0,0 @@
|
||||
# Storage Tiering: db-tracked vs db-only directories
|
||||
|
||||
## Overview
|
||||
|
||||
GBrain supports storage tiering to separate version-controlled content from bulk machine-generated data. This prevents git repositories from becoming bloated with large amounts of automatically generated content while still preserving it in the database.
|
||||
|
||||
> Note on naming: prior to v0.22.11 the keys were `git_tracked` / `supabase_only`. The canonical names are now `db_tracked` / `db_only` (engine-agnostic — works on both PGLite and Postgres). The deprecated keys still load with a once-per-process warning. Run `gbrain doctor --fix` for an automated rename when that path lands.
|
||||
|
||||
## Configuration
|
||||
|
||||
Add a `storage` section to your `gbrain.yml` file in the brain repository root:
|
||||
|
||||
```yaml
|
||||
storage:
|
||||
# Directories that are version-controlled (human-edited, committed to git).
|
||||
db_tracked:
|
||||
- people/
|
||||
- companies/
|
||||
- deals/
|
||||
- concepts/
|
||||
- yc/
|
||||
- ideas/
|
||||
- projects/
|
||||
|
||||
# Directories persisted via the brain database only (bulk machine-generated
|
||||
# content). Written to disk as a local cache but not committed to git;
|
||||
# `gbrain sync` auto-manages .gitignore for these paths. `gbrain export
|
||||
# --restore-only` repopulates missing files from the database.
|
||||
db_only:
|
||||
- media/x/
|
||||
- media/articles/
|
||||
- meetings/transcripts/
|
||||
```
|
||||
|
||||
Path requirements:
|
||||
|
||||
- Each directory must end with `/` for canonical form. The validator auto-normalizes missing trailing slashes (one-time info note shows what changed).
|
||||
- A directory cannot appear in both tiers — that's a tier-overlap error and `loadStorageConfig` throws `StorageConfigError`. Edit `gbrain.yml` to remove the overlap and try again.
|
||||
|
||||
## Behavior Changes
|
||||
|
||||
### 1. `gbrain sync` — automatic .gitignore management
|
||||
|
||||
When storage configuration is present, `gbrain sync` automatically manages `.gitignore` entries on every successful sync:
|
||||
|
||||
- Adds missing `db_only` directory patterns to `.gitignore`.
|
||||
- Idempotent — re-running adds no duplicate entries.
|
||||
- Stable comment header so the managed block is grep-able.
|
||||
- Skipped on `--dry-run` (don't mutate disk in preview mode).
|
||||
- Skipped on `blocked_by_failures` status (sync state is inconsistent).
|
||||
- Skipped when the repo is a git submodule (`.git` is a file, not a directory) — submodule .gitignore changes don't survive parent updates. A warning explains.
|
||||
- Skipped entirely when `GBRAIN_NO_GITIGNORE=1` is set (escape hatch for shared-repo setups where a maintainer wants gbrain to leave .gitignore alone).
|
||||
- Failures (write permission denied, etc.) are caught and logged, never crash sync.
|
||||
|
||||
Example `.gitignore` addition:
|
||||
|
||||
```gitignore
|
||||
# Auto-managed by gbrain (db_only directories)
|
||||
media/x/
|
||||
media/articles/
|
||||
meetings/transcripts/
|
||||
```
|
||||
|
||||
### 2. `gbrain export --restore-only` — repopulate missing db_only files
|
||||
|
||||
```bash
|
||||
# Restore only missing db_only files from the database.
|
||||
gbrain export --restore-only --repo /path/to/brain
|
||||
|
||||
# Filter by page type.
|
||||
gbrain export --restore-only --type media --repo /path/to/brain
|
||||
|
||||
# Filter by slug prefix.
|
||||
gbrain export --restore-only --slug-prefix media/x/ --repo /path/to/brain
|
||||
|
||||
# Combine filters.
|
||||
gbrain export --restore-only --type media --slug-prefix media/x/ --repo /path/to/brain
|
||||
```
|
||||
|
||||
The `--restore-only` flag:
|
||||
|
||||
- Resolves repoPath via the chain `--repo` → typed `sources.getDefault()` → hard error.
|
||||
Never falls through to the current directory.
|
||||
- Only exports pages that match `db_only` patterns AND are missing from disk.
|
||||
- Ideal for container restart recovery and fresh clones.
|
||||
|
||||
### 3. `gbrain storage status` — storage-tier health dashboard
|
||||
|
||||
```bash
|
||||
# Human-readable status.
|
||||
gbrain storage status --repo /path/to/brain
|
||||
|
||||
# JSON output for scripts and orchestrators.
|
||||
gbrain storage status --repo /path/to/brain --json
|
||||
```
|
||||
|
||||
Output includes:
|
||||
|
||||
- Total page counts by storage tier.
|
||||
- Disk usage breakdown by tier.
|
||||
- Missing files that need restoration (top 10 shown; full list in `--json`).
|
||||
- Configuration validation warnings.
|
||||
- Current tier directory listing.
|
||||
|
||||
Example output:
|
||||
|
||||
```
|
||||
Storage Status
|
||||
==============
|
||||
|
||||
Repository: /data/brain
|
||||
Total pages: 15,243
|
||||
|
||||
Storage Tiers:
|
||||
-------------
|
||||
DB tracked: 2,156 pages
|
||||
DB only: 12,887 pages
|
||||
Unspecified: 200 pages
|
||||
|
||||
Disk Usage:
|
||||
-----------
|
||||
DB tracked: 45.2 MB
|
||||
DB only: 2.1 GB
|
||||
|
||||
Missing Files (need restore):
|
||||
-----------------------------
|
||||
media/x/tweet-1234567890
|
||||
media/x/tweet-0987654321
|
||||
... and 47 more
|
||||
|
||||
Use: gbrain export --restore-only --repo "/data/brain"
|
||||
|
||||
Configuration:
|
||||
--------------
|
||||
DB tracked directories:
|
||||
- people/
|
||||
- companies/
|
||||
- deals/
|
||||
|
||||
DB-only directories:
|
||||
- media/x/
|
||||
- media/articles/
|
||||
- meetings/transcripts/
|
||||
```
|
||||
|
||||
## Validation
|
||||
|
||||
`loadStorageConfig` runs `normalizeAndValidateStorageConfig` after parsing:
|
||||
|
||||
- Auto-fixes (silent, with one-time info note showing what changed):
|
||||
- Missing trailing `/` is added: `'media/x'` → `'media/x/'`.
|
||||
- Throws `StorageConfigError` (caller sees a clean exit-1 with actionable message):
|
||||
- Same directory in both `db_tracked` and `db_only` (ambiguous routing).
|
||||
|
||||
## Use cases
|
||||
|
||||
### Brain repository scaling
|
||||
|
||||
Perfect for brain repositories crossing 50K-200K+ files where:
|
||||
|
||||
- Core knowledge (people, companies, deals) remains git-tracked.
|
||||
- Bulk data (tweets, articles, transcripts) moves to db_only.
|
||||
- Development stays fast with smaller git repos.
|
||||
- Full data remains available via the database.
|
||||
|
||||
### Container-based deployments
|
||||
|
||||
Essential for ephemeral container environments:
|
||||
|
||||
- Git repo contains only essential files.
|
||||
- Container restarts don't lose db_only data.
|
||||
- `gbrain export --restore-only` quickly restores bulk files when needed.
|
||||
- Local disk acts as a cache layer.
|
||||
|
||||
### Multi-environment consistency
|
||||
|
||||
Enables consistent data access across environments:
|
||||
|
||||
- Development: small git clone, restore bulk data on demand.
|
||||
- Production: full dataset via the database, selective local caching.
|
||||
- CI/CD: fast tests with git-tracked data only.
|
||||
|
||||
## Migration strategy
|
||||
|
||||
1. **Assess current repository**: use `gbrain storage status` to understand current distribution.
|
||||
2. **Plan directory structure**: identify which directories should be db_tracked vs db_only.
|
||||
3. **Create `gbrain.yml`**: add storage configuration to the repository root.
|
||||
4. **Test with dry-run**: `gbrain sync --dry-run` to verify behavior; `.gitignore` is NOT touched on dry-run.
|
||||
5. **Run a real sync**: `gbrain sync` updates `.gitignore` automatically on success.
|
||||
6. **Verify restore**: test `gbrain export --restore-only --repo .` against a small db_only directory.
|
||||
|
||||
## Best practices
|
||||
|
||||
- **Directory naming**: end storage paths with `/` (canonical form). The validator normalizes if you forget.
|
||||
- **Start small**: begin with clearly machine-generated directories in `db_only`.
|
||||
- **Address validation errors**: tier overlap is an error, not a warning. Fix it before sync.
|
||||
- **Test restore**: regularly test `--restore-only` in staging environments.
|
||||
- **Document decisions**: comment your `gbrain.yml` to explain tier choices.
|
||||
|
||||
## PGLite engine note
|
||||
|
||||
On the PGLite engine (gbrain's local-only embedded Postgres), the "DB" your db_only pages live in IS the local file gbrain uses for everything else. The `.gitignore` housekeeping still helps (keeps bulk content out of git history), but the offload-to-DB promise is technically vacuous. A once-per-process soft-warn explains when the engine is detected. To get full tiering, migrate to Postgres with `gbrain migrate --to supabase`.
|
||||
|
||||
## Compatibility
|
||||
|
||||
- **Backward compatible**: systems without `gbrain.yml` work unchanged.
|
||||
- **Progressive enhancement**: add configuration when needed.
|
||||
- **Database unchanged**: all data remains in Postgres regardless of tier.
|
||||
- **Existing workflows**: all existing `sync` and `export` behavior preserved.
|
||||
- **Deprecated keys**: `git_tracked` / `supabase_only` still load with a once-per-process warning.
|
||||
-18
@@ -1,18 +0,0 @@
|
||||
storage:
|
||||
# Directories that are version-controlled — human-curated, edited by hand.
|
||||
db_tracked:
|
||||
- people/
|
||||
- companies/
|
||||
- deals/
|
||||
- concepts/
|
||||
- yc/
|
||||
- ideas/
|
||||
- projects/
|
||||
|
||||
# Directories persisted via the brain database only — bulk machine-generated
|
||||
# content. .gitignored automatically by `gbrain sync`. Restorable from the DB
|
||||
# via `gbrain export --restore-only`.
|
||||
db_only:
|
||||
- media/x/
|
||||
- media/articles/
|
||||
- meetings/transcripts/
|
||||
+35
-232
@@ -104,30 +104,24 @@ strict behavior when unset.
|
||||
- `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`. `OperationContext.remote` flags untrusted callers.
|
||||
- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports.
|
||||
- `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`)
|
||||
- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for the specific forward-referenced state the embedded schema blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target table) and adds only what's missing. Closes the upgrade-wedge bug class that bit users 10+ times across 6 schema versions over 2 years (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396). No-op on fresh installs and modern brains.
|
||||
- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly.
|
||||
- `src/core/pglite-schema.ts` — PGLite-specific DDL (pgvector, pg_trgm, triggers)
|
||||
- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. v0.22.0: `searchKeyword`, `searchKeywordChunks`, and `searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude clause from `src/core/search/sql-ranking.ts`. `searchVector` switches to a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in the outer SELECT) and carries `p.source_id` through inner→outer for v0.18 multi-source callers. v0.22.1 (#406): `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures). `executeRaw` is a single-statement passthrough — no per-call retry (D3 dropped that as unsound for non-idempotent statements; recovery is supervisor-driven). v0.22.1 (#363, contributed by @orendi84): `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. v0.22.1 (#409, contributed by @atrevino47): `countStaleChunks()` + `listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale`, eliminating ~76 MB/call client-side pull on a fully-embedded brain; `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding (consistency). As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL on the same forward-reference probe set as the PGLite engine, so old Postgres brains pinned at v0.13/v0.18/v0.19 walk forward cleanly instead of wedging on `column "..." does not exist`.
|
||||
- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query.
|
||||
- `src/core/utils.ts` — Shared SQL utilities extracted from postgres-engine.ts. Exports `parseEmbedding(value)` (throws on unknown input, used by migration + ingest paths where data integrity matters) and as of v0.12.3 `tryParseEmbedding(value)` (returns `null` + warns once per process, used by search/rescore paths where availability matters more than strictness).
|
||||
- `src/core/db.ts` — Connection management, schema initialization. v0.22.1 (#363, contributed by @orendi84): `resolveSessionTimeouts()` returns `statement_timeout` + `idle_in_transaction_session_timeout` (defaults: 5min each, env-overridable via `GBRAIN_STATEMENT_TIMEOUT` / `GBRAIN_IDLE_TX_TIMEOUT` / `GBRAIN_CLIENT_CHECK_INTERVAL`). Both `connect()` (module singleton) and `PostgresEngine.connect()` (worker pool) consume the result via postgres.js's `connection` option, sending GUCs as startup parameters that survive PgBouncer transaction mode (unlike the prior `setSessionDefaults` post-pool SET, kept as a back-compat no-op shim).
|
||||
- `src/core/db.ts` — Connection management, schema initialization
|
||||
- `src/commands/migrate-engine.ts` — Bidirectional engine migration (`gbrain migrate --to supabase/pglite`)
|
||||
- `src/core/import-file.ts` — importFromFile + importFromContent (chunk + embed + tags)
|
||||
- `src/core/sync.ts` — Pure sync functions (manifest parsing, filtering, slug conversion). v0.22.12 (#500, foundation by @wintermute via #501): `classifyErrorCode(errorMsg)` regex-based classifier with 12 codes (`SLUG_MISMATCH`, `YAML_PARSE`, `YAML_DUPLICATE_KEY`, `MISSING_OPEN`, `MISSING_CLOSE`, `NESTED_QUOTES`, `EMPTY_FRONTMATTER`, `NULL_BYTES`, `INVALID_UTF8`, `STATEMENT_TIMEOUT`, `FILE_TOO_LARGE`, `SYMLINK_NOT_ALLOWED`) plus `UNKNOWN` fallback. `summarizeFailuresByCode(failures)` returns sorted `[{code, count}]`. `code?` optional field on `SyncFailure`; backfilled at ack time on pre-v0.22.12 entries. `acknowledgeSyncFailures()` returns `AcknowledgeResult { count, summary }`. Three regexes (`MISSING_OPEN`, `MISSING_CLOSE`, `EMPTY_FRONTMATTER`) broadened to match actual `markdown.ts:159-244` validator message strings, not just the literal code-name prefix. `FILE_TOO_LARGE` covers all three production size sites in `import-file.ts:199, 352, 401`; `SYMLINK_NOT_ALLOWED` covers the rejection at `:347`. Closes the silent-skip pattern that motivated #500.
|
||||
- `src/core/sync.ts` — Pure sync functions (manifest parsing, filtering, slug conversion)
|
||||
- `src/core/storage.ts` — Pluggable storage interface (S3, Supabase Storage, local)
|
||||
- `src/core/storage-config.ts` (v0.22.11) — Storage tiering: `loadStorageConfig` reads `gbrain.yml`, normalizes deprecated keys (`git_tracked` / `supabase_only`) to canonical (`db_tracked` / `db_only`) with once-per-process deprecation warning, and runs `normalizeAndValidateStorageConfig` (auto-fixes missing trailing `/`, throws `StorageConfigError` on tier overlap). Path-segment matcher: `media/x/` does NOT match `media/xerox/foo`. Replaces gray-matter (broken on delimiter-less YAML) with a dedicated parser for the `gbrain.yml` shape.
|
||||
- `src/core/disk-walk.ts` (v0.22.11) — `walkBrainRepo(repoPath)` returns `Map<slug, {size, mtimeMs}>` from one recursive `readdirSync`. Skips dot-dirs, `node_modules`, non-`.md` files. Used by `gbrain storage status` to replace per-page `existsSync + statSync` (~400K syscalls on 200K-page brains → tens).
|
||||
- `src/commands/storage.ts` (v0.22.11) — `gbrain storage status [--repo P] [--json]`. Split into pure data (`getStorageStatus`) + JSON formatter + human formatter (ASCII-only per D10) matching the `orphans.ts` pattern. `PageCountsByTier` and `DiskUsageByTier` are distinct nominal types so swaps fail at compile time.
|
||||
- `gbrain.yml` (brain repo root, v0.22.11) — Optional storage tiering config. Top-level `storage:` section with `db_tracked:` and `db_only:` array-valued keys. `gbrain sync` auto-manages `.gitignore` for `db_only` paths on successful sync (skips on dry-run, blocked-by-failures, submodule context, or `GBRAIN_NO_GITIGNORE=1`). `gbrain export --restore-only [--repo P] [--type T] [--slug-prefix S]` repopulates missing `db_only` files from the database.
|
||||
- `src/core/supabase-admin.ts` — Supabase admin API (project discovery, pgvector check)
|
||||
- `src/core/file-resolver.ts` — File resolution with fallback chain (local -> .redirect.yaml -> .redirect -> .supabase)
|
||||
- `src/core/chunkers/` — 3-tier chunking (recursive, semantic, LLM-guided). v0.19.0 adds `code.ts` — tree-sitter-based semantic chunker for 29 languages with embedded-asset WASMs (`src/assets/wasm/`), `@dqbd/tiktoken` cl100k_base tokenizer, small-sibling merging. `CHUNKER_VERSION` constant folded into `importCodeFile`'s `content_hash` so chunker shape changes force clean re-chunks across releases.
|
||||
- `src/core/errors.ts` (v0.19.0) — `StructuredAgentError` + `buildError` + `serializeError`. Every new v0.19.0 agent-facing surface (code-def, code-refs, usage errors) uses this envelope; matches v0.17.0 `CycleReport.PhaseResult.error` shape.
|
||||
- `src/assets/wasm/` (v0.19.0) — 36 tree-sitter grammar WASMs + tree-sitter runtime. Committed to the repo so `bun --compile` embeds them deterministically via `import path from ... with { type: 'file' }`. The CI guard `scripts/check-wasm-embedded.sh` fails the build if the compiled binary ever silently falls through to recursive chunks.
|
||||
- `src/commands/code-def.ts` + `src/commands/code-refs.ts` (v0.19.0) — symbol definition + references lookup. Query `content_chunks.symbol_name` or chunk_text ILIKE with `page_kind='code'` filter. Auto-JSON when stdout is not a TTY (gh-CLI convention). Bypass the standard `searchKeyword` `DISTINCT ON (slug)` collapse so multiple call-sites from the same file surface.
|
||||
- `src/core/search/` — Hybrid search: vector + keyword + RRF + multi-query expansion + dedup. As of v0.22.0, `searchKeyword` / `searchKeywordChunks` / `searchVector` apply source-aware ranking at the SQL layer (curated content like `originals/`, `concepts/`, `writing/` outranks bulk content like `wintermute/chat/`, `daily/`, `media/x/`). `searchVector` uses a two-stage CTE so source-boost re-ranking doesn't kill the HNSW index. Hard-exclude prefixes (`test/`, `archive/`, `attachments/`, `.raw/` by default) filter at retrieval, not post-rank. Both gates honor `detail !== 'high'` so temporal queries surface chat pages normally.
|
||||
- `src/core/search/` — Hybrid search: vector + keyword + RRF + multi-query expansion + dedup
|
||||
- `src/core/search/intent.ts` — Query intent classifier (entity/temporal/event/general → auto-selects detail level)
|
||||
- `src/core/search/eval.ts` — Retrieval eval harness: P@k, R@k, MRR, nDCG@k metrics + runEval() orchestrator
|
||||
- `src/core/search/source-boost.ts` (v0.22.0) — Source-type boost map keyed by slug prefix. `DEFAULT_SOURCE_BOOSTS` (originals/ 1.5, concepts/ 1.3, writing/ 1.4, people/companies/deals/ 1.2, daily/ 0.8, media/x/ 0.7, wintermute/chat/ 0.5) and `DEFAULT_HARD_EXCLUDES` (test/, archive/, attachments/, .raw/). `parseSourceBoostEnv` / `parseHardExcludesEnv` parse comma-separated `prefix:factor` pairs from `GBRAIN_SOURCE_BOOST` / `GBRAIN_SEARCH_EXCLUDE` env vars. `resolveBoostMap` and `resolveHardExcludes` merge defaults + env + caller `SearchOpts.exclude_slug_prefixes`/`include_slug_prefixes`.
|
||||
- `src/core/search/sql-ranking.ts` (v0.22.0) — Pure SQL string builders. `buildSourceFactorCase(slugColumn, boostMap, detail)` emits a CASE expression with longest-prefix-match wins (returns literal `'1.0'` when `detail === 'high'` for temporal-bypass parity with COMPILED_TRUTH_BOOST). `buildHardExcludeClause(slugColumn, prefixes)` emits `NOT (col LIKE 'p1%' OR col LIKE 'p2%')` — OR-chain wrapped in NOT, NOT `NOT LIKE ALL/ANY` (those quantifiers don't express set-exclusion). LIKE meta-character escape covers all three of `%`, `_`, AND `\` (backslash matters because it's Postgres LIKE's default escape char). Single-quote doubling on SQL string literals so injection-style inputs are inert text.
|
||||
- `src/commands/eval.ts` — `gbrain eval` command: single-run table + A/B config comparison
|
||||
- `src/core/embedding.ts` — OpenAI text-embedding-3-large, batch, retry, backoff
|
||||
- `src/core/check-resolvable.ts` — Resolver validation: reachability, MECE overlap, DRY checks, structured fix objects. v0.14.1: `CROSS_CUTTING_PATTERNS.conventions` is an array (notability gate accepts both `conventions/quality.md` and `_brain-filing-rules.md`). New `extractDelegationTargets()` parses `> **Convention:**`, `> **Filing rule:**`, and inline backtick references. DRY suppression is proximity-based via `DRY_PROXIMITY_LINES = 40`.
|
||||
@@ -146,14 +140,12 @@ strict behavior when unset.
|
||||
- `src/core/transcription.ts` — Audio transcription: Groq Whisper (default), OpenAI fallback, ffmpeg segmentation for >25MB
|
||||
- `src/core/enrichment-service.ts` — Global enrichment service: entity slug generation, tier auto-escalation, batch throttling
|
||||
- `src/core/data-research.ts` — Recipe validation, field extraction (MRR/ARR regex), dedup, tracker parsing, HTML stripping
|
||||
- `src/commands/embed.ts` — `gbrain embed [--stale|--all] [--slugs ...]`. v0.22.1 (#409, contributed by @atrevino47): `--stale` path now starts with `engine.countStaleChunks()` (single SELECT count(*) WHERE embedding IS NULL, ~50 bytes wire). On a fully-embedded brain that's a 1-line short-circuit — no further reads. When stale chunks exist, `engine.listStaleChunks()` returns just the chunks needing embeddings (slug + chunk_index + chunk_text + metadata, no `vector(1536)` payload). Caller groups by slug, embeds via OpenAI, re-upserts via `upsertChunks`. Replaces the prior page-walk that pulled every chunk's embedding column over the wire and discarded most.
|
||||
- `src/commands/extract.ts` — `gbrain extract links|timeline|all [--source fs|db]`: batch link/timeline extraction. fs walks markdown files, db walks pages from the engine (mutation-immune snapshot iteration; use this for live brains with no local checkout). As of v0.12.1 there is no in-memory dedup pre-load — candidates are buffered 100 at a time and flushed via `addLinksBatch` / `addTimelineEntriesBatch`; `ON CONFLICT DO NOTHING` enforces uniqueness at the DB layer, and the `created` counter returns real rows inserted (truthful on re-runs). v0.22.1 (#417): `ExtractOpts.slugs?: string[]` enables incremental extract — when set, `extractForSlugs()` reads ONLY those slugs' files (single combined links+timeline pass) instead of the full directory walk. CLI `gbrain extract` keeps full-walk behavior; the cycle path threads sync's `pagesAffected` through. `walkMarkdownFiles(brainDir)` still runs at line 455 to build `allSlugs` for link resolution — see `TODOS.md` for replacing it with `engine.getAllSlugs()`.
|
||||
- `src/commands/extract.ts` — `gbrain extract links|timeline|all [--source fs|db]`: batch link/timeline extraction. fs walks markdown files, db walks pages from the engine (mutation-immune snapshot iteration; use this for live brains with no local checkout). As of v0.12.1 there is no in-memory dedup pre-load — candidates are buffered 100 at a time and flushed via `addLinksBatch` / `addTimelineEntriesBatch`; `ON CONFLICT DO NOTHING` enforces uniqueness at the DB layer, and the `created` counter returns real rows inserted (truthful on re-runs).
|
||||
- `src/commands/graph-query.ts` — `gbrain graph-query <slug> [--type T] [--depth N] [--direction in|out|both]`: typed-edge relationship traversal (renders indented tree)
|
||||
- `src/core/link-extraction.ts` — shared library for the v0.12.0 graph layer. extractEntityRefs (canonical, replaces backlinks.ts duplicate) matches both `[Name](people/slug)` markdown links and Obsidian `[[people/slug|Name]]` wikilinks as of v0.12.3. extractPageLinks, inferLinkType heuristics (attended/works_at/invested_in/founded/advises/source/mentions), parseTimelineEntries, isAutoLinkEnabled config helper. `DIR_PATTERN` covers `people`, `companies`, `deals`, `topics`, `concepts`, `projects`, `entities`, `tech`, `finance`, `personal`, `openclaw`. Used by extract.ts, operations.ts auto-link post-hook, and backlinks.ts.
|
||||
- `src/core/minions/` — Minions job queue: BullMQ-inspired, Postgres-native (queue, worker, backoff, types, protected-names, quiet-hours, stagger, handlers/shell).
|
||||
- `src/core/minions/queue.ts` — MinionQueue class (submit, claim, complete, fail, stall detection, parent-child, depth/child-cap, per-job timeouts, cascade-kill, attachments, idempotency keys, child_done inbox, removeOnComplete/Fail). `add()` takes a 4th `trusted` arg (separate from `opts` to prevent spread leakage); protected names in `PROTECTED_JOB_NAMES` require `{allowProtectedSubmit: true}` and the check runs trim-normalized (whitespace-bypass safe). v0.14.1 #219: `add()` plumbs `max_stalled` through with a `[1, 100]` clamp; omitted values let the schema DEFAULT (5) kick in. v0.19.0: `handleWallClockTimeouts(lockDurationMs)` is Layer 3 kill shot for jobs where `FOR UPDATE SKIP LOCKED` stall detection and the timeout sweep both fail to evict (wedged worker holding a row lock via a pending transaction). v0.19.1: `maxWaiting` coalesce path now uses `pg_advisory_xact_lock` keyed on `(name, queue)` to serialize concurrent submits for the same key, and filters on `queue` in addition to `name` so cross-queue same-name jobs don't suppress each other.
|
||||
- `src/core/minions/worker.ts` — MinionWorker class (handler registry, lock renewal, graceful shutdown, timeout safety net). v0.14.0 abort-path fix: aborted jobs now call `failJob` with reason (`timeout`/`cancel`/`lock-lost`/`shutdown`) instead of returning silently. `shutdownAbort` (instance field) fires on process SIGTERM/SIGINT and propagates to `ctx.shutdownSignal` — shell handler listens to it; non-shell handlers don't. v0.22.1 (#403): per-job timeout fires `abort.abort(new Error('timeout'))` then a 30-second grace-then-evict safety net force-evicts the job from `inFlight` and marks it dead in DB if the handler ignores the abort signal — frees the slot even when a handler wedges (the 98-waiting-0-active prod incident driver).
|
||||
- `src/core/minions/supervisor.ts` — MinionSupervisor process manager. Spawns `gbrain jobs work` as a child, restarts on crash with exponential backoff, periodic health check. v0.22.1 (#406): `consecutiveHealthFailures` counter; on 3 consecutive failures emits `health_warn` with `reason: 'db_connection_degraded'` and calls `engine.reconnect()` to swap in a fresh pool, then resets the counter. Worker exit classifier emits `likely_cause` field on `worker_exited` events: `oom_or_external_kill` (SIGKILL), `graceful_shutdown` (SIGTERM), `runtime_error` (code 1), `clean_exit` (code 0), `unknown`.
|
||||
- `src/core/minions/worker.ts` — MinionWorker class (handler registry, lock renewal, graceful shutdown, timeout safety net). v0.14.0 abort-path fix: aborted jobs now call `failJob` with reason (`timeout`/`cancel`/`lock-lost`/`shutdown`) instead of returning silently. `shutdownAbort` (instance field) fires on process SIGTERM/SIGINT and propagates to `ctx.shutdownSignal` — shell handler listens to it; non-shell handlers don't.
|
||||
- `src/core/minions/types.ts` — `MinionJobInput` + `MinionJobStatus` + handler context types. `MinionJobInput.max_stalled` (new in v0.14.1) is optional; omitted values let the schema DEFAULT (5) kick in, provided values are clamped to `[1, 100]`.
|
||||
- `src/core/minions/protected-names.ts` — side-effect-free constant module exporting `PROTECTED_JOB_NAMES` + `isProtectedJobName()`. Kept pure so queue core can import without loading handler modules.
|
||||
- `src/core/minions/handlers/shell.ts` — `shell` job handler. Spawns `/bin/sh -c cmd` (absolute path, PATH-override-safe) or `argv[0] argv[1..]` (no shell). Env allowlist: `PATH, HOME, USER, LANG, TZ, NODE_ENV` + caller `env:` overrides. UTF-8-safe stdout/stderr tail via `string_decoder.StringDecoder`. Abort (either `ctx.signal` or `ctx.shutdownSignal`) fires SIGTERM → 5s grace → SIGKILL on child. Requires `GBRAIN_ALLOW_SHELL_JOBS=1` on worker (gated by `registerBuiltinHandlers`).
|
||||
@@ -171,27 +163,20 @@ strict behavior when unset.
|
||||
- `src/core/minions/attachments.ts` — Attachment validation (path traversal, null byte, oversize, base64, duplicate detection)
|
||||
- `src/commands/agent.ts` (v0.16) — `gbrain agent run <prompt> [flags]` CLI. Submits `subagent` (or N children + 1 aggregator) under `{allowProtectedSubmit: true}`. Single-entry `--fanout-manifest` short-circuits. Children get `on_child_fail: 'continue'` + `max_stalled: 3`. `--follow` is the default on TTY; streams logs + polls `waitForCompletion` in parallel. Ctrl-C detaches, does not cancel.
|
||||
- `src/commands/agent-logs.ts` (v0.16) — `gbrain agent logs <job> [--follow] [--since]`. Merges JSONL heartbeat audit + `subagent_messages` into a chronological timeline. `parseSince` accepts ISO-8601 or relative (`5m`, `1h`, `2d`). Transcript tail renders only for terminal jobs.
|
||||
- `src/commands/jobs.ts` — `gbrain jobs` CLI subcommands + `gbrain jobs work` daemon. v0.13.1 surfaces the full `MinionJobInput` retry/backoff/timeout/idempotency surface as first-class CLI flags on `jobs submit`: `--max-stalled`, `--backoff-type fixed|exponential`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key`. `jobs smoke --sigkill-rescue` is the opt-in regression guard for #219. v0.16 wires `registerBuiltinHandlers` to always register `subagent` + `subagent_aggregator` (no env flag — `ANTHROPIC_API_KEY` is the natural cost gate, trust is via `PROTECTED_JOB_NAMES`) and loads `GBRAIN_PLUGIN_PATH` plugins at worker startup with a loud startup-line per plugin. `shell` handler still gated by `GBRAIN_ALLOW_SHELL_JOBS=1` (RCE surface, separate concern). v0.22.10 (#521): the `autopilot-cycle` handler now forwards `job.data.phases` to `runCycle` (was previously discarded — caller-supplied phase selection silently became a full cycle). Phases are validated against `ALL_PHASES` from `src/core/cycle.ts`; invalid names are filtered out and an empty/missing array falls back to the default 6-phase cycle. v0.22.13 (PR #490 CODEX-1+CODEX-4): `sync` handler now resolves `sourceId` at entry by looking up `sources.local_path` (mirrors `cycle.ts:480`'s autopilot fix from PR #475) so multi-source brains read the per-source `last_commit` anchor instead of the global config key. Concurrency routed through the shared `autoConcurrency()` policy in `src/core/sync-concurrency.ts` instead of the prior hardcoded `4`; PGLite stays serial. `noEmbed` default is `true` (embed is a separate job — submit `gbrain embed --stale` after sync, or rely on the autopilot cycle's embed phase).
|
||||
- `src/commands/jobs.ts` — `gbrain jobs` CLI subcommands + `gbrain jobs work` daemon. v0.13.1 surfaces the full `MinionJobInput` retry/backoff/timeout/idempotency surface as first-class CLI flags on `jobs submit`: `--max-stalled`, `--backoff-type fixed|exponential`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key`. `jobs smoke --sigkill-rescue` is the opt-in regression guard for #219. v0.16 wires `registerBuiltinHandlers` to always register `subagent` + `subagent_aggregator` (no env flag — `ANTHROPIC_API_KEY` is the natural cost gate, trust is via `PROTECTED_JOB_NAMES`) and loads `GBRAIN_PLUGIN_PATH` plugins at worker startup with a loud startup-line per plugin. `shell` handler still gated by `GBRAIN_ALLOW_SHELL_JOBS=1` (RCE surface, separate concern).
|
||||
- `src/commands/features.ts` — `gbrain features --json --auto-fix`: usage scan + feature adoption salesman
|
||||
- `src/commands/autopilot.ts` — `gbrain autopilot --install`: self-maintaining brain daemon (sync+extract+embed)
|
||||
- `src/mcp/server.ts` — MCP stdio server (generated from operations). v0.22.7: tool-call handler delegates to `dispatchToolCall` from `src/mcp/dispatch.ts` so stdio + HTTP transports share one validation, context-build, and error-format path.
|
||||
- `src/mcp/dispatch.ts` (v0.22.7) — Shared tool-call dispatch consumed by both stdio (`server.ts`) and HTTP (`http-transport.ts`). Exports `dispatchToolCall(engine, name, params, opts)`, `buildOperationContext(engine, params, opts)`, and `validateParams(op, params)`. Single source of truth for `(ctx, params)` handler arg order and the 5-field `OperationContext` shape (engine + config + logger + dryRun + remote). Defaults to `remote: true` (untrusted); local CLI callers pass `remote: false`. Closed F1 (reversed handler args) + F2 (incomplete OperationContext) + F3 (no param validation) drift bugs in the original v0.22.5 HTTP transport.
|
||||
- `src/mcp/rate-limit.ts` (v0.22.7) — Bounded-LRU token-bucket limiter for `gbrain serve --http`. `buildDefaultLimiters()` returns the two-bucket pipeline used by http-transport: pre-auth IP (default 30/60s, fires BEFORE the DB lookup so brute-force load against `access_tokens` is actually capped) + post-auth token-id (default 60/60s). Tracks `lastTouchedMs` separately from `lastRefillMs` so an exhausted key can't be reset by hammering past the TTL. LRU cap (default 10K keys) bounds memory under attacker-controlled key growth; TTL prune at 2× window evicts abandoned buckets.
|
||||
- `src/mcp/http-transport.ts` (v0.22.7, rewrite) — `gbrain serve --http` HTTP transport. Postgres-only — fails fast at startup on PGLite (the `access_tokens` table only exists on Postgres). Bearer auth against SHA-256 hashes in `access_tokens`. CORS default-deny via `GBRAIN_HTTP_CORS_ORIGIN` allowlist. Body cap stream-counted (1 MiB default via `GBRAIN_HTTP_MAX_BODY_BYTES`) so chunked transfers without Content-Length still hit the cap. `last_used_at` SQL-level debounce (one UPDATE per token per 60s). Per-request audit row in `mcp_request_log` with token_name + operation + status + latency. Optional `GBRAIN_HTTP_TRUST_PROXY=1` honors `X-Forwarded-For` — only safe when bound to a private interface AND the proxy strips client-supplied XFF (otherwise enables IP spoofing past the pre-auth rate limit). `/health` does `SELECT 1` against Postgres and returns 503 + `status:unhealthy` when the DB is unreachable so orchestration doesn't see green pods while clients get misleading 401s. Replaces the standalone OAuth wrapper that was vulnerable to unauthenticated client registration.
|
||||
- `src/commands/auth.ts` — Token management for the HTTP transport. `gbrain auth create/list/revoke/test`. As of v0.22.7 wired into the main CLI (`src/cli.ts`); also runs standalone via `bun run src/commands/auth.ts ...` for environments without a compiled binary. Tokens stored as SHA-256 hashes in `access_tokens` (Postgres-only).
|
||||
- `src/mcp/server.ts` — MCP stdio server (generated from operations)
|
||||
- `src/commands/auth.ts` — Standalone token management (create/list/revoke/test)
|
||||
- `src/commands/upgrade.ts` — Self-update CLI. `runPostUpgrade()` enumerates migrations from the TS registry (src/commands/migrations/index.ts) and tail-calls `runApplyMigrations(['--yes', '--non-interactive'])` so the mechanical side of every outstanding migration runs unconditionally.
|
||||
- `src/commands/migrations/` — TS migration registry (compiled into the binary; no filesystem walk of `skills/migrations/*.md` needed at runtime). `index.ts` lists migrations in semver order. `v0_11_0.ts` = Minions adoption orchestrator (8 phases). `v0_12_0.ts` = Knowledge Graph auto-wire orchestrator (5 phases: schema → config check → backfill links → backfill timeline → verify). `phaseASchema` has a 600s timeout (bumped from 60s in v0.12.1 for duplicate-heavy brains). `v0_12_2.ts` = JSONB double-encode repair orchestrator (4 phases: schema → repair-jsonb → verify → record). `v0_14_0.ts` = shell-jobs + autopilot cooperative (2 phases: schema ALTER minion_jobs.max_stalled SET DEFAULT 3 — superseded by v0.14.3's schema-level DEFAULT 5 + UPDATE backfill; pending-host-work ping for skills/migrations/v0.14.0.md). All orchestrators are idempotent and resumable from `partial` status. As of v0.14.2 (Bug 3), the RUNNER owns all ledger writes — orchestrators return `OrchestratorResult` and `apply-migrations.ts` persists a canonical `{version, status, phases}` shape after return. Orchestrators no longer call `appendCompletedMigration` directly. `statusForVersion` prefers `complete` over `partial` (never regresses). 3 consecutive partials → wedged → `--force-retry <version>` writes a `'retry'` reset marker. v0.14.3 (fix wave) ships schema-only migrations v14 (`pages_updated_at_index`) + v15 (`minion_jobs_max_stalled_default_5` with UPDATE backfill) via the `MIGRATIONS` array in `src/core/migrate.ts` — no orchestrator phases needed.
|
||||
- `src/commands/repair-jsonb.ts` — `gbrain repair-jsonb [--dry-run] [--json]`: rewrites `jsonb_typeof='string'` rows in place across 5 affected columns (pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata, page_versions.frontmatter). Fixes v0.12.0 double-encode bug on Postgres; PGLite no-ops. Idempotent.
|
||||
- `src/commands/orphans.ts` — `gbrain orphans [--json] [--count] [--include-pseudo]`: surfaces pages with zero inbound wikilinks, grouped by domain. Auto-generated/raw/pseudo pages filtered by default. Also exposed as `find_orphans` MCP operation. Shipped in v0.12.3 (contributed by @knee5).
|
||||
- `src/commands/integrity.ts` — `gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). v0.22.8: batch-load fast path on Postgres uses `SELECT DISTINCT ON (slug)` in a single SQL query to fix the PgBouncer round-trip timeout (60s → ~6s) while preserving `engine.getAllSlugs()`'s `Set<string>` semantics on multi-source brains. Gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1` so real Postgres errors are diagnosable.
|
||||
- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel <id>`. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count.
|
||||
- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2).
|
||||
- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel <id>`.
|
||||
- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows.
|
||||
- `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2.
|
||||
- `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=<ms>` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers.
|
||||
- `src/core/db-lock.ts` (v0.22.13) — generic `tryAcquireDbLock(engine, lockId, ttlMinutes)` over the existing `gbrain_cycle_locks` table. Parameterized lock id so different scopes can nest cleanly: `gbrain-cycle` for the broad cycle (held by `cycle.ts`) and `gbrain-sync` (`SYNC_LOCK_ID` constant) for `performSync`'s narrower writer window. Same UPSERT-with-TTL semantics as the prior cycle-only helper, just generalized. Survives PgBouncer transaction pooling (unlike session-scoped `pg_try_advisory_lock`); crashed holders auto-release once their TTL expires.
|
||||
- `src/core/sync-concurrency.ts` (v0.22.13) — single source of truth for the parallel-sync policy. Exports `autoConcurrency(engine, fileCount, override?)` (PGLite always serial; explicit override clamped to >=1; auto path returns `DEFAULT_PARALLEL_WORKERS=4` when `fileCount > AUTO_CONCURRENCY_FILE_THRESHOLD=100`), `shouldRunParallel(workers, fileCount, explicit)` (Q1: explicit `--workers` bypasses the >50-file floor), and `parseWorkers(s)` (rejects `'0'`, `'-3'`, `'foo'`, `'1.5'`, trailing chars — replaces the prior parseInt-with-no-validation in both `sync.ts` and `import.ts`). Used by `performSync`, `performFullSync`, `runImport`, and the Minion `sync` handler so the three sites can no longer drift.
|
||||
- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). v0.22.13 (PR #490): `performSync` wraps its body in a `gbrain-sync` writer lock so two concurrent syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot both write `last_commit` and let the last writer win. Head-drift gate after the import phase re-checks `git rev-parse HEAD`; if HEAD moved (someone ran `git checkout` / `git pull` mid-sync), the bookmark refuses to advance. Vanished files now record a failedFiles entry instead of silent-skip — the silent-skip-then-advance pathology that survived prior hardening passes is dead. Worker engines wrap in try/finally so disconnect always fires (panic-path leak fix). Both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`), validated via `parseWorkers`. Explicit `--workers` bypasses the auto-path file-count floor; auto path defers to `autoConcurrency()`. Banner moved to stderr.
|
||||
- `src/core/cycle.ts` — v0.17 brain maintenance cycle primitive. `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise<CycleReport>` composes 6 phases in semantically-driven order (lint → backlinks → sync → extract → embed → orphans). Three callers: `gbrain dream` CLI, `gbrain autopilot` daemon's inline path, and the Minions `autopilot-cycle` handler (`src/commands/jobs.ts`). One source of truth for what the brain does overnight. Coordination via `gbrain_cycle_locks` DB table (TTL-based; works through PgBouncer transaction pooling, unlike session-scoped `pg_try_advisory_lock`) + `~/.gbrain/cycle.lock` file lock with PID-liveness for PGLite / engine=null mode. `CycleReport.schema_version: "1"` is the stable agent-consumable shape. `PhaseResult.error: { class, code, message, hint?, docs_url? }` is Stripe-API-tier structured failure info. `yieldBetweenPhases` hook awaited between every phase — Minions handler uses this to renew its job lock and prevent v0.14 stall-death regression. Engine nullable: filesystem phases (lint, backlinks) run without DB; DB phases skip with `status: "skipped", reason: "no_database"`. Lock-skip: read-only phase selections (`--phase orphans`) bypass the cycle lock. v0.22.1 (#403): `CycleOpts.signal?: AbortSignal` propagates the worker's abort signal; `checkAborted()` fires between every phase and throws if the signal is aborted (cooperative — can't interrupt a phase mid-execution). v0.22.1 (#417): `runPhaseSync` returns `pagesAffected` via `SyncPhaseResult`; `runCycle` captures it and threads to `runPhaseExtract` as the 4th arg, enabling incremental extract on the cycle path. v0.22.1 (Codex F2): `runPhaseSync` takes `willRunExtractPhase: boolean` and sets `noExtract: phases.includes('extract')` so `gbrain dream --phase sync` doesn't silently lose extraction. v0.22.5 (#475): new `resolveSourceForDir(engine, brainDir)` helper queries `SELECT id FROM sources WHERE local_path = $1 LIMIT 1`; `runPhaseSync` threads result as `sourceId` to `performSync()` so sync reads the per-source `sources.last_commit` anchor instead of the drift-prone global `config.sync.last_commit` key. Bare try/catch lets pre-v0.18 brains fall through to the global key. Closes the prod hang where every autopilot cycle ran a 30-min full reimport because the global anchor commit had been GC'd from git history.
|
||||
- `src/core/cycle.ts` — v0.17 brain maintenance cycle primitive. `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise<CycleReport>` composes 6 phases in semantically-driven order (lint → backlinks → sync → extract → embed → orphans). Three callers: `gbrain dream` CLI, `gbrain autopilot` daemon's inline path, and the Minions `autopilot-cycle` handler (`src/commands/jobs.ts`). One source of truth for what the brain does overnight. Coordination via `gbrain_cycle_locks` DB table (TTL-based; works through PgBouncer transaction pooling, unlike session-scoped `pg_try_advisory_lock`) + `~/.gbrain/cycle.lock` file lock with PID-liveness for PGLite / engine=null mode. `CycleReport.schema_version: "1"` is the stable agent-consumable shape. `PhaseResult.error: { class, code, message, hint?, docs_url? }` is Stripe-API-tier structured failure info. `yieldBetweenPhases` hook awaited between every phase — Minions handler uses this to renew its job lock and prevent v0.14 stall-death regression. Engine nullable: filesystem phases (lint, backlinks) run without DB; DB phases skip with `status: "skipped", reason: "no_database"`. Lock-skip: read-only phase selections (`--phase orphans`) bypass the cycle lock.
|
||||
- `src/commands/dream.ts` — v0.17 `gbrain dream` CLI. ~80-line thin alias over `runCycle`. brainDir resolution requires explicit `--dir` OR `sync.repo_path` config (no more walk-up-cwd-for-.git footgun). Flags: `--dry-run`, `--json`, `--phase <name>`, `--pull`, `--dir <path>`. Exit code 1 on status=failed (partial/warn not fatal — don't page on warnings).
|
||||
- `scripts/check-progress-to-stdout.sh` — CI guard against regressing to `\r`-on-stdout progress. Wired into `bun run test` via `scripts/check-progress-to-stdout.sh && bun test` in package.json.
|
||||
- `docs/progress-events.md` — Canonical JSON event schema reference. Stable from v0.15.2, additive only.
|
||||
@@ -302,10 +287,6 @@ Key commands added in v0.14.3 (fix wave):
|
||||
- `gbrain jobs submit` gains `--max-stalled`, `--backoff-type`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key` — exposing existing `MinionJobInput` fields as first-class CLI flags.
|
||||
- `gbrain jobs smoke --sigkill-rescue` — opt-in regression smoke case simulating a killed worker; asserts the v0.14.3 schema default (`max_stalled=5`) actually rescues on first stall.
|
||||
|
||||
Key commands added in v0.22.13 (PR #490):
|
||||
- `gbrain sync --workers N` (alias `--concurrency N`) — parallelize the import phase using per-worker Postgres engines (small pool of 2 each) with an atomic queue index. Auto-concurrency: defaults to 4 workers when the diff exceeds 100 files. Smaller diffs stay serial. Explicit `--workers` always wins (even on a 30-file diff). PGLite forces serial regardless. Validation rejects `0`, negatives, non-integers loud (replaces the prior silent fall-through to auto-concurrency).
|
||||
- `gbrain import --workers N` — same `parseWorkers()` validation as sync; same try/finally worker-engine cleanup. Behavior surface unchanged.
|
||||
|
||||
## Testing
|
||||
|
||||
`bun test` runs all tests. After the v0.12.1 release: ~75 unit test files + 8 E2E test files (1412 unit pass, 119 E2E when `DATABASE_URL` is set — skip gracefully otherwise). Unit tests run
|
||||
@@ -317,9 +298,7 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac
|
||||
`test/files.test.ts` (MIME/hash), `test/import-file.test.ts` (import pipeline),
|
||||
`test/upgrade.test.ts` (schema migrations),
|
||||
`test/file-migration.test.ts` (file migration), `test/file-resolver.test.ts` (file resolution),
|
||||
`test/import-resume.test.ts` (import checkpoints), `test/migrate.test.ts` (migration; v8/v9 helper-btree-index SQL structural assertions + 1000-row wall-clock fixtures that guard the O(n²)→O(n log n) fix + v0.13.1 assertions on v12/v13 SQL shape, `sqlFor` + `transaction:false` runner semantics, the `max_stalled DEFAULT 1` regression guard, and v0.22.6.1 v24 `sqlFor.pglite: ''` no-op assertion),
|
||||
`test/bootstrap.test.ts` (v0.22.6.1 — bootstrap contract: no-op on fresh install, idempotent across two `initSchema()` calls, no-op on modern brain that already has every probed column, full bootstrap path on simulated pre-v0.18 brain, fresh-install regression guard, pre-v0.13 `links` shape coverage),
|
||||
`test/schema-bootstrap-coverage.test.ts` (v0.22.6.1 CI guard — `REQUIRED_BOOTSTRAP_COVERAGE` lists every forward reference in PGLITE_SCHEMA_SQL; the test fails loudly if `applyForwardReferenceBootstrap` skips one. When you add a column-with-index to the embedded schema blob, you extend both arrays or this guard fails. The pattern that broke gbrain ten times in two years is now structurally prevented.),
|
||||
`test/import-resume.test.ts` (import checkpoints), `test/migrate.test.ts` (migration; v8/v9 helper-btree-index SQL structural assertions + 1000-row wall-clock fixtures that guard the O(n²)→O(n log n) fix + v0.13.1 assertions on v12/v13 SQL shape, `sqlFor` + `transaction:false` runner semantics, and the `max_stalled DEFAULT 1` regression guard),
|
||||
`test/setup-branching.test.ts` (setup flow), `test/slug-validation.test.ts` (slug validation),
|
||||
`test/storage.test.ts` (storage backends), `test/supabase-admin.test.ts` (Supabase admin),
|
||||
`test/yaml-lite.test.ts` (YAML parsing), `test/check-update.test.ts` (version check + update CLI),
|
||||
@@ -333,7 +312,6 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac
|
||||
`test/skills-conformance.test.ts` (skill frontmatter + required sections validation),
|
||||
`test/resolver.test.ts` (RESOLVER.md coverage, routing validation + v0.20.4 round-trip: every quoted RESOLVER.md trigger must match a frontmatter `triggers:` entry in the target skill, and every `name="<word>"` reference in any SKILL.md must resolve to a declared op in `src/core/operations.ts` or a Minions handler in `PROTECTED_JOB_NAMES`),
|
||||
`test/search.test.ts` (RRF normalization, compiled truth boost, cosine similarity, dedup key),
|
||||
`test/sql-ranking.test.ts` (v0.22.0 source-boost helpers: 39 cases covering longest-prefix-match in SQL CASE, detail=high temporal-bypass, three-meta-char LIKE escape (%, _, \\), single-quote SQL-literal doubling, env override parsing for GBRAIN_SOURCE_BOOST + GBRAIN_SEARCH_EXCLUDE, resolveBoostMap / resolveHardExcludes merge semantics),
|
||||
`test/dedup.test.ts` (source-aware dedup, compiled truth guarantee, layer interactions),
|
||||
`test/intent.test.ts` (query intent classification: entity/temporal/event/general),
|
||||
`test/eval.test.ts` (retrieval metrics: precisionAtK, recallAtK, mrr, ndcgAtK, parseQrels),
|
||||
@@ -361,9 +339,6 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac
|
||||
`test/orphans.test.ts` (v0.12.3 orphans command: detection, pseudo filtering, text/json/count outputs, MCP op),
|
||||
`test/postgres-engine.test.ts` (v0.12.3 statement_timeout scoping: `sql.begin` + `SET LOCAL` shape, source-level grep guardrail against reintroduced bare `SET statement_timeout`),
|
||||
`test/sync.test.ts` (sync logic + v0.12.3 regression guard asserting top-level `engine.transaction` is not called),
|
||||
`test/sync-concurrency.test.ts` (v0.22.13 PR #490: 17 cases covering `autoConcurrency()` thresholds + PGLite-forces-serial + explicit-override clamping, `shouldRunParallel()` Q1 explicit-bypasses-floor contract, and `parseWorkers()` validation that rejects `'0'`/`'-3'`/`'foo'`/`'1.5'`/trailing chars),
|
||||
`test/sync-parallel.test.ts` (v0.22.13 PR #490: PGLite-routed coverage of the bookmark gate under concurrency request, head-drift gate, vanished-file failure capture, PGLite-stays-serial, and the `gbrain-sync` writer-lock contract — 7 cases),
|
||||
`test/sync-failures.test.ts` (v0.22.12: 28 cases pinning `classifyErrorCode` regex coverage for all 12 codes against literal production message strings from `markdown.ts:159-244` and `import-file.ts:199, 347, 352, 401`; `summarizeFailuresByCode` sort + pre-classified-honor; `recordSyncFailures` code-field persistence; `acknowledgeSyncFailures` AcknowledgeResult shape + backfill on pre-v0.22.12 entries),
|
||||
`test/doctor.test.ts` (doctor command + v0.12.3 assertions that `jsonb_integrity` scans the four v0.12.0 write sites and `markdown_body_completeness` is present),
|
||||
`test/utils.test.ts` (shared SQL utilities + `tryParseEmbedding` null-return and single-warn semantics),
|
||||
`test/build-llms.test.ts` (llms.txt/llms-full.txt generator: path resolution, idempotence, spec shape, regen-drift guard, content contract, AGENTS.md install-path mirror, size-budget enforcement — 7 cases),
|
||||
@@ -374,26 +349,17 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac
|
||||
`test/skill-manifest.test.ts` (v0.19 skill manifest parser: drift detection, managed-block markers),
|
||||
`test/skillify-scaffold.test.ts` (v0.19 `gbrain skillify scaffold` stubs: SKILL.md, script, tests, routing-eval fixtures),
|
||||
`test/skillpack-install.test.ts` (v0.19 `gbrain skillpack install` managed-block install / update / no-clobber semantics),
|
||||
`test/skillpack-sync-guard.test.ts` (v0.19 sync-guard: bundled skills stay byte-identical to `skills/` source),
|
||||
`test/http-transport.test.ts` (v0.22.7 HTTP transport: 23 unit cases covering bearer auth + missing/no-Bearer/unknown/revoked + `/health` bypass, F1+F2 round-trip via dispatch.ts, F3 invalid_params, application/json response shape (not SSE), CORS default-deny + allowlist, body cap on Content-Length AND chunked, two-bucket rate limit (refill, exhaust+Retry-After, LRU eviction, TTL prune, pre-auth IP fires before DB), and `mcp_request_log` audit on success + auth_failed).
|
||||
`test/skillpack-sync-guard.test.ts` (v0.19 sync-guard: bundled skills stay byte-identical to `skills/` source).
|
||||
|
||||
E2E tests (`test/e2e/`): Run against real Postgres+pgvector. Require `DATABASE_URL`.
|
||||
- `bun run test:e2e` runs Tier 1 (mechanical, all operations, no API keys). Includes 9 dedicated cases for the postgres-engine `addLinksBatch` / `addTimelineEntriesBatch` bind path — postgres-js's `unnest()` binding is structurally different from PGLite's and gets its own coverage.
|
||||
- `test/e2e/search-quality.test.ts` runs search quality E2E against PGLite (no API keys, in-memory)
|
||||
- `test/e2e/graph-quality.test.ts` runs the v0.10.3 knowledge graph pipeline (auto-link via put_page, reconciliation, traversePaths) against PGLite in-memory
|
||||
- `test/e2e/postgres-jsonb.test.ts` — v0.12.2 regression test. Round-trips all 5 JSONB write sites (pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata, page_versions.frontmatter) against real Postgres and asserts `jsonb_typeof='object'` plus `->>'key'` returns the expected scalar. The test that should have caught the original double-encode bug.
|
||||
- `test/e2e/integrity-batch.test.ts` (v0.22.8) — parity tests for `scanIntegrity`'s batch-load fast path vs sequential. Four cases (dedup, hits, validate, topPages) seed a fixture and assert both paths return identical results. Dedup case uses raw SQL via `getConn().unsafe()` to seed a `(test-source-2, people/alice)` row alongside the default-source row, since `engine.putPage` doesn't take a `source_id`. Pins the codex-caught multi-source overcounting regression.
|
||||
- `test/e2e/jsonb-roundtrip.test.ts` — v0.12.3 companion regression against the 4 doctor-scanned JSONB sites. Assertion-level overlap with `postgres-jsonb.test.ts` is intentional defense-in-depth: if doctor's scan surface ever drifts from the actual write surface, one of these tests catches it.
|
||||
- `test/e2e/sync.test.ts` (v0.22.12 — `--skip-failed` failure-loop test, alongside the existing 13 happy-path tests): exercises the full chain — broken file → `performSync` returns `blocked_by_failures` with grouped breakdown → `performSync({skipFailed: true})` advances bookmark and returns `AcknowledgeResult` with code summary → second broken file → second cycle. Saves and restores the user's real `~/.gbrain/sync-failures.jsonl` so the test is hermetic on a developer machine. Asserts bookmark gating, JSONL state, dedup across paths, summary aggregation, and the literal doctor-rendering string format. This is the integration test that proves the v0.22.12 chain holds together — unit tests cover the pure functions in isolation, this covers the integration.
|
||||
- `test/e2e/upgrade.test.ts` runs check-update E2E against real GitHub API (network required)
|
||||
- `test/e2e/minions-shell-pglite.test.ts` (v0.20.4) exercises the PGLite `--follow` inline shell-job path (in-memory, no `DATABASE_URL` required) — the path the consolidated minion-orchestrator skill documents for dev use
|
||||
- `test/e2e/openclaw-reference-compat.test.ts` (v0.19) — exercises `check-resolvable` + `skillpack install` against a minimal AGENTS.md workspace fixture (`test/fixtures/openclaw-reference-minimal/`), regression guard for the 107-skill OpenClaw deployment shape
|
||||
- `test/e2e/search-swamp.test.ts` (v0.22.0) — reproduces the headline source-swamp case. Seeds a curated `originals/talks/article-outline-fat-code` page against two `wintermute/chat/` pages stuffed with the same multi-word phrase. Asserts the article wins keyword AND vector ranking, that `detail=high` lets the chat swamp re-surface (temporal-query workflow preserved), and that `source_id` passes through the two-stage CTE intact. PGLite in-memory.
|
||||
- `test/e2e/search-exclude.test.ts` (v0.22.0) — verifies `test/` + `archive/` pages are hidden by default, that `include_slug_prefixes` opts back in, and that caller-supplied `exclude_slug_prefixes` adds to defaults. Both keyword and vector search paths covered.
|
||||
- `test/e2e/engine-parity.test.ts` (v0.22.0) — Postgres ↔ PGLite top-result and result-set parity for `searchKeyword` + `searchVector`. Codex flagged that Postgres ranks pages then picks best chunk while PGLite returns chunks directly — without parity coverage the source-boost fix could pass on PGLite and fail on Postgres. Skips gracefully when `DATABASE_URL` is unset.
|
||||
- `test/e2e/postgres-bootstrap.test.ts` (v0.22.6.1) — exercises `PostgresEngine.initSchema()` directly against a fresh real Postgres database. Asserts the bootstrap path is no-op on fresh installs and that SCHEMA_SQL replays cleanly through the engine path (not via the standalone `db.initSchema` from `src/core/db.ts`, which would have produced false-positive coverage). Codex caught the E2E-shape gap during plan review.
|
||||
- `test/e2e/http-transport.test.ts` (v0.22.7) — 8 cases against real Postgres covering `gbrain serve --http` end-to-end: bearer auth round-trip, `last_used_at` SQL-level debounce semantics, `mcp_request_log` row insertion on success and auth_failed paths, `/health` DB-down → 503 (DB-probing health check), and the F1+F2+F3 dispatch round-trip with a real operation. Skips gracefully when `DATABASE_URL` is unset.
|
||||
- `test/e2e/sync-parallel.test.ts` (v0.22.13 PR #490) — DATABASE_URL-gated. T2: 60-file Postgres sync at concurrency=4 imports all + no connection leak (probes `pg_stat_activity` before/after to confirm worker engines disconnected). P4: 120-file serial-vs-parallel benchmark prints `SYNC_PARALLEL_BENCH N files | serial=Xms | parallel(4)=Yms | speedup=Zx` for CHANGELOG quoting. Asserts parallel ≤ serial × 1.5 (CI-noise tolerant; not a strict speedup gate).
|
||||
- Tier 2 (`skills.test.ts`) requires OpenClaw + API keys, runs nightly in CI
|
||||
- If `.env.testing` doesn't exist in this directory, check sibling worktrees for one:
|
||||
`find ../ -maxdepth 2 -name .env.testing -print -quit` and copy it here if found.
|
||||
@@ -508,59 +474,6 @@ in bulk paths, the CI guard will fail the build.
|
||||
|
||||
`bun build --compile --outfile bin/gbrain src/cli.ts`
|
||||
|
||||
## Version locations (single source of truth: `VERSION` file)
|
||||
|
||||
Every release advances the version in **five files at once**. Keep these in
|
||||
sync. `/ship` enforces this via Step 12's idempotency check (VERSION vs
|
||||
package.json drift), but the canonical list lives here so future runs and
|
||||
the auto-update agent know where to look.
|
||||
|
||||
**Required (every release must update all five):**
|
||||
|
||||
| File | What lives there | Format |
|
||||
|---|---|---|
|
||||
| `VERSION` | The single source of truth. Read first by `/ship`, the binary, and CI version-gate. | Bare 4-digit string `MAJOR.MINOR.PATCH.MICRO` (e.g. `0.22.1`), no leading `v`, no trailing newline-sensitivity issues. |
|
||||
| `package.json` | Bun/npm package version. `gbrain --version` reads it via the compiled binary's bundled package metadata. CI version-gate cross-checks this against `VERSION` and fails if they drift. | `"version": "0.22.1"` |
|
||||
| `CHANGELOG.md` | Top entry header `## [0.22.1] - YYYY-MM-DD` plus the "To take advantage of v0.22.1" block. | Standard Keep-a-Changelog header. |
|
||||
| `TODOS.md` | Any TODO entries that mention "follow-up from vX.Y.Z" use the version of the release that filed them. Update only when filing NEW follow-up TODOs. | Inline `vX.Y.Z` references in TODO bodies. |
|
||||
| `CLAUDE.md` | The Key Files section's per-file annotations carry `vX.Y.Z (#NNN)` tags noting which release introduced a behavior. Update whenever a wave's annotations get folded in. | Inline `vX.Y.Z (#NNN, contributed by @user)` references. |
|
||||
|
||||
**Auto-derived (no manual edit; refreshed by their own commands):**
|
||||
|
||||
- `bun.lock` — root-package version is auto-pinned from `package.json`. After
|
||||
bumping `package.json`, run `bun install` to refresh the lockfile.
|
||||
- `llms-full.txt` / `llms.txt` — auto-generated documentation bundles. After
|
||||
any release ship that touches the Key Files annotations in `CLAUDE.md`,
|
||||
run `bun run build:llms` to regenerate. The bundles do not contain a
|
||||
version pin per se; they reflect the current state of the docs they index.
|
||||
|
||||
**Historical (DO NOT bump on release):**
|
||||
|
||||
- `skills/migrations/v0.21.0.md` — migration files use the version they
|
||||
shipped FROM as their filename. v0.21.0's migration always says v0.21.0.
|
||||
- `src/commands/migrations/v0_21_0.ts` — same: migration code references
|
||||
the schema version it migrates to.
|
||||
- `test/migrations-v0_21_0.test.ts`, `test/migration-orchestrator-v0_21_0.test.ts`,
|
||||
`test/migrate.test.ts` — migration tests reference historical migration
|
||||
versions; these are correct as-is and should not move.
|
||||
- `src/core/db.ts`, `src/core/migrate.ts`, `src/core/import-file.ts`,
|
||||
`src/commands/reindex-code.ts` — code comments cite the release that
|
||||
introduced a feature. Once written, these are historical record.
|
||||
- `README.md` — references the latest published feature names by version
|
||||
(e.g. "v0.21.0 Code Cathedral"); update only when the README's marketing
|
||||
copy is intentionally being refreshed, NOT on every micro/patch bump.
|
||||
|
||||
**The /ship workflow's version idempotency check:** Step 12 reads
|
||||
`VERSION` and `package.json`, classifies as FRESH / ALREADY_BUMPED /
|
||||
DRIFT_STALE_PKG / DRIFT_UNEXPECTED, and refuses to proceed on
|
||||
DRIFT_UNEXPECTED. This is why the two must move together.
|
||||
|
||||
**The CI version-gate** rejects pushes where `VERSION` and
|
||||
`package.json` disagree, OR where `VERSION` is not strictly greater
|
||||
than master's VERSION. If a queue collision claims your version on
|
||||
master before yours lands, /ship's queue-aware allocator (Step 12)
|
||||
will detect drift and re-bump on the next run.
|
||||
|
||||
## Pre-ship requirements
|
||||
|
||||
Before shipping (/ship) or reviewing (/review), always run the full test suite:
|
||||
@@ -1196,15 +1109,13 @@ This is the dispatcher. Skills are the implementation. **Read the skill file bef
|
||||
|
||||
| Trigger | Skill |
|
||||
|---------|-------|
|
||||
| "What do we know about", "tell me about", "search for", "who is", "background on", "notes on" | `skills/query/SKILL.md` |
|
||||
| "What do we know about", "tell me about", "search for" | `skills/query/SKILL.md` |
|
||||
| "Who knows who", "relationship between", "connections", "graph query" | `skills/query/SKILL.md` (use graph-query) |
|
||||
| Creating/enriching a person or company page | `skills/enrich/SKILL.md` |
|
||||
| Where does a new file go? Filing rules | `skills/repo-architecture/SKILL.md` |
|
||||
| Fix broken citations in brain pages | `skills/citation-fixer/SKILL.md` |
|
||||
| "citation audit", "check citations", "fix citations" | `skills/citation-fixer/SKILL.md` (focused fix). For broader brain health, chain into `skills/maintain/SKILL.md` |
|
||||
| "Research", "track", "extract from email", "investor updates", "donations" | `skills/data-research/SKILL.md` |
|
||||
| Share a brain page as a link | `skills/publish/SKILL.md` |
|
||||
| "validate frontmatter", "check frontmatter", "fix frontmatter", "frontmatter audit", "brain lint" | `skills/frontmatter-guard/SKILL.md` |
|
||||
|
||||
## Content & media ingestion
|
||||
|
||||
@@ -1374,13 +1285,12 @@ Add to `~/.claude/server.json` (Claude Code), Settings > MCP Servers (Cursor), o
|
||||
### Remote MCP (Claude Desktop, Cowork, Perplexity)
|
||||
|
||||
```bash
|
||||
gbrain auth create "claude-desktop" # tokens via the existing CLI
|
||||
gbrain serve --http --port 8787 # built-in HTTP transport (Postgres-only)
|
||||
ngrok http 8787 --url your-brain.ngrok.app # any tunnel works
|
||||
ngrok http 8787 --url your-brain.ngrok.app
|
||||
bun run src/commands/auth.ts create "claude-desktop"
|
||||
claude mcp add gbrain -t http https://your-brain.ngrok.app/mcp -H "Authorization: Bearer TOKEN"
|
||||
```
|
||||
|
||||
Per-client guides: [`docs/mcp/`](docs/mcp/DEPLOY.md). Hardening defaults, env vars, and threat model: [SECURITY.md](SECURITY.md). ChatGPT requires OAuth 2.1 (not yet implemented).
|
||||
Per-client guides: [`docs/mcp/`](docs/mcp/DEPLOY.md). ChatGPT requires OAuth 2.1 (not yet implemented).
|
||||
|
||||
### Using gbrain with GStack
|
||||
|
||||
@@ -1654,30 +1564,6 @@ accumulate rows across separate single-skill installs instead of overwriting eac
|
||||
Read [`skills/skillify/SKILL.md`](skills/skillify/SKILL.md) for the full 10-item checklist
|
||||
and the anti-patterns it catches.
|
||||
|
||||
## Storage tiering: keep bulk content out of git (v0.22.11)
|
||||
|
||||
When your brain crosses 100K files and bulk machine-generated content (tweets, articles, transcripts)
|
||||
becomes the size driver, declare which directories belong in git and which live in the database only.
|
||||
|
||||
```yaml
|
||||
# gbrain.yml at the brain repo root
|
||||
storage:
|
||||
db_tracked:
|
||||
- people/
|
||||
- companies/
|
||||
- deals/
|
||||
db_only:
|
||||
- media/x/
|
||||
- media/articles/
|
||||
- meetings/transcripts/
|
||||
```
|
||||
|
||||
`gbrain sync` auto-manages your `.gitignore` for `db_only` paths. `gbrain export --restore-only --repo .`
|
||||
repopulates missing files from the database (container restart, fresh clone, accidental rm).
|
||||
`gbrain storage status` shows the tier breakdown.
|
||||
|
||||
Full guide: [docs/storage-tiering.md](docs/storage-tiering.md).
|
||||
|
||||
## Getting Data In
|
||||
|
||||
GBrain ships integration recipes that your agent sets up for you. Each recipe tells the agent what credentials to ask for, how to validate, and what cron to register.
|
||||
@@ -1824,8 +1710,6 @@ Question
|
||||
│ ├─ Multi-query expansion (Haiku rephrases the question 3 ways)
|
||||
│ ├─ Vector search (HNSW cosine over OpenAI embeddings)
|
||||
│ ├─ Keyword search (Postgres tsvector + websearch_to_tsquery)
|
||||
│ ├─ Source-aware ranking (curated dirs outrank chat/daily swamp at SQL layer)
|
||||
│ ├─ Hard-exclude (test/ archive/ attachments/ .raw/ filtered before retrieval)
|
||||
│ ├─ Reciprocal Rank Fusion (score = sum 1/(60+rank) across both)
|
||||
│ ├─ Cosine re-scoring (re-rank chunks against actual query embedding)
|
||||
│ ├─ Compiled-truth boost (assessments outrank timeline noise)
|
||||
@@ -1933,11 +1817,8 @@ SEARCH
|
||||
gbrain query <question> Hybrid search (vector + keyword + RRF)
|
||||
|
||||
IMPORT
|
||||
gbrain import <dir> [--no-embed] [--workers N]
|
||||
Import markdown (idempotent)
|
||||
gbrain sync [--repo <path>] [--workers N]
|
||||
Git-to-brain incremental sync
|
||||
(>100-file diffs auto-parallelize 4 workers on Postgres)
|
||||
gbrain import <dir> [--no-embed] Import markdown (idempotent)
|
||||
gbrain sync [--repo <path>] Git-to-brain incremental sync
|
||||
gbrain export [--dir ./out/] Export to markdown
|
||||
|
||||
FILES
|
||||
@@ -1979,8 +1860,6 @@ ADMIN
|
||||
gbrain doctor --locks List idle-in-tx backends (57014 diagnostic, Postgres only)
|
||||
gbrain stats Brain statistics
|
||||
gbrain serve MCP server (stdio)
|
||||
gbrain serve --http --port 8787 MCP server (HTTP, Postgres-only, bearer auth)
|
||||
gbrain auth create|list|revoke|test Token management for the HTTP transport
|
||||
gbrain integrations Integration recipe dashboard
|
||||
gbrain sources list|add|remove|... Multi-source brain management (v0.18)
|
||||
gbrain dream [--dry-run] [--phase N] One maintenance cycle then exit (cron-friendly)
|
||||
@@ -4161,14 +4040,9 @@ Source: https://raw.githubusercontent.com/garrytan/gbrain/master/docs/mcp/DEPLOY
|
||||
|
||||
# Deploy GBrain Remote MCP Server
|
||||
|
||||
> **v0.22.7+:** Use `gbrain serve --http` for remote access. It includes built-in
|
||||
> bearer token auth, default-deny CORS, two-bucket rate limiting, body cap, and
|
||||
> per-request audit log. **Postgres-only** (PGLite is local-only by design).
|
||||
> See [SECURITY.md](../../SECURITY.md) for env vars and tunable defaults.
|
||||
|
||||
Access your brain from any device, any AI client. GBrain's MCP server runs locally
|
||||
via `gbrain serve` (stdio). For remote access, expose it via the built-in HTTP
|
||||
transport behind a public tunnel.
|
||||
via `gbrain serve` (stdio). For remote access, wrap it in an HTTP server behind a
|
||||
public tunnel.
|
||||
|
||||
## Two Paths
|
||||
|
||||
@@ -4179,23 +4053,21 @@ gbrain serve
|
||||
```
|
||||
|
||||
Works with Claude Code, Cursor, Windsurf, and any MCP client that supports stdio.
|
||||
No server, no tunnel, no token needed. Works on both PGLite and Postgres engines.
|
||||
No server, no tunnel, no token needed.
|
||||
|
||||
### Remote (any device, any AI client) — Postgres only
|
||||
### Remote (any device, any AI client)
|
||||
|
||||
```
|
||||
Your AI client (Claude Desktop, Perplexity, etc.)
|
||||
→ ngrok tunnel (https://YOUR-DOMAIN.ngrok.app)
|
||||
→ gbrain serve --http (built-in transport with bearer auth)
|
||||
→ Postgres (pooler connection or self-hosted)
|
||||
→ Your HTTP server (wraps gbrain serve)
|
||||
→ Supabase Postgres (via pooler connection string)
|
||||
```
|
||||
|
||||
This requires:
|
||||
1. A Postgres-backed brain (the `access_tokens` table only exists on Postgres;
|
||||
running `gbrain serve --http` against a PGLite install fails fast at startup)
|
||||
2. A machine running `gbrain serve --http`
|
||||
3. A public tunnel (ngrok, Tailscale, or cloud host)
|
||||
4. A bearer token created via `gbrain auth create <name>`
|
||||
1. A machine running `gbrain serve` behind an HTTP wrapper
|
||||
2. A public tunnel (ngrok, Tailscale, or cloud host)
|
||||
3. Bearer token auth for security
|
||||
|
||||
## Remote Setup
|
||||
|
||||
@@ -4214,13 +4086,13 @@ ngrok http 8787 --url your-brain.ngrok.app # Hobby tier for fixed domain
|
||||
|
||||
```bash
|
||||
# Create a token for each client
|
||||
gbrain auth create "claude-desktop"
|
||||
bun run src/commands/auth.ts create "claude-desktop"
|
||||
|
||||
# List all tokens
|
||||
gbrain auth list
|
||||
bun run src/commands/auth.ts list
|
||||
|
||||
# Revoke a token
|
||||
gbrain auth revoke "claude-desktop"
|
||||
bun run src/commands/auth.ts revoke "claude-desktop"
|
||||
```
|
||||
|
||||
Tokens are per-client. Create one for each device/app. Revoke individually
|
||||
@@ -4236,7 +4108,7 @@ if compromised. Tokens are stored SHA-256 hashed in your database.
|
||||
### 4. Verify
|
||||
|
||||
```bash
|
||||
gbrain auth test \
|
||||
bun run src/commands/auth.ts test \
|
||||
https://YOUR-DOMAIN.ngrok.app/mcp \
|
||||
--token YOUR_TOKEN
|
||||
```
|
||||
@@ -4264,7 +4136,7 @@ Funnel, and cloud hosts (Fly.io, Railway).
|
||||
Include the Authorization header: `Authorization: Bearer YOUR_TOKEN`
|
||||
|
||||
**"invalid_token" error**
|
||||
Run `gbrain auth list` to see active tokens.
|
||||
Run `bun run src/commands/auth.ts list` to see active tokens.
|
||||
|
||||
**"service_unavailable" error**
|
||||
Database connection failed. Check your Supabase dashboard for outages.
|
||||
@@ -5298,75 +5170,6 @@ in depth, not the primary boundary.
|
||||
|
||||
---
|
||||
|
||||
## v0.22.4 — frontmatter-guard adoption
|
||||
|
||||
### 1. Stop hand-rolling frontmatter validators
|
||||
|
||||
If your fork has scripts that call `js-yaml` directly to validate brain page
|
||||
frontmatter, replace them with `gbrain frontmatter validate` calls. The CLI
|
||||
covers the seven canonical error classes and ships a `--json` envelope that's
|
||||
stable across releases.
|
||||
|
||||
```diff
|
||||
- # Custom validator script
|
||||
- node scripts/validate-frontmatter.mjs <path>
|
||||
+ gbrain frontmatter validate <path> --json
|
||||
```
|
||||
|
||||
For consumers that need the validator inside another script, import from
|
||||
gbrain's `markdown` export instead of duplicating logic:
|
||||
|
||||
```ts
|
||||
import { parseMarkdown } from 'gbrain/markdown';
|
||||
|
||||
const parsed = parseMarkdown(content, filePath, { validate: true, expectedSlug });
|
||||
for (const err of parsed.errors ?? []) {
|
||||
// err.code: MISSING_OPEN | MISSING_CLOSE | YAML_PARSE | SLUG_MISMATCH |
|
||||
// NULL_BYTES | NESTED_QUOTES | EMPTY_FRONTMATTER
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Drop any references to `lib/brain-writer.mjs`
|
||||
|
||||
If your fork's skills or scripts referenced an aspirational
|
||||
`lib/brain-writer.mjs` (it never shipped — the spec was in PR #392 and never
|
||||
landed), replace those references with the gbrain CLI. The `frontmatter-guard`
|
||||
skill lives at `skills/frontmatter-guard/SKILL.md` and points at
|
||||
`gbrain frontmatter validate` / `audit` / `install-hook`.
|
||||
|
||||
### 3. Wire the doctor subcheck into your health pipeline
|
||||
|
||||
`gbrain doctor` now reports `frontmatter_integrity` automatically. If your
|
||||
fork has a custom health pipeline (e.g. a daily Slack post about brain
|
||||
health), pull from `gbrain doctor --json` and surface the
|
||||
`frontmatter_integrity` row counts.
|
||||
|
||||
### 4. (Optional) Install the pre-commit hook on brain repos
|
||||
|
||||
For sources backed by git, the v0.22.4 install-hook helper drops a
|
||||
pre-commit script that blocks commits with malformed frontmatter:
|
||||
|
||||
```bash
|
||||
gbrain frontmatter install-hook
|
||||
```
|
||||
|
||||
Skip this if your brain isn't a git repo or if your downstream agent already
|
||||
enforces validation at write time. See `docs/integrations/pre-commit.md` for
|
||||
the full recipe.
|
||||
|
||||
### 5. Migration ergonomics — read pending-host-work.jsonl
|
||||
|
||||
After `gbrain apply-migrations --yes` runs the v0.22.4 audit, your agent
|
||||
should read `~/.gbrain/migrations/pending-host-work.jsonl` (filter to
|
||||
`migration === "0.22.4"`) and walk each entry's `command` field. Each entry
|
||||
points to a per-source `gbrain frontmatter validate <source_path> --fix`
|
||||
command — surface counts to the user, get explicit consent, then run.
|
||||
|
||||
The migration is **audit-only**. It never mutates brain content during
|
||||
`apply-migrations`. Your agent runs the fix command with user consent.
|
||||
|
||||
---
|
||||
|
||||
## Future versions
|
||||
|
||||
When gbrain ships a new version, this doc will be updated with the diffs for that
|
||||
|
||||
+2
-4
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gbrain",
|
||||
"version": "0.22.13",
|
||||
"version": "0.21.0",
|
||||
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
|
||||
"type": "module",
|
||||
"main": "src/core/index.ts",
|
||||
@@ -32,9 +32,8 @@
|
||||
"build:all": "bun build --compile --target=bun-darwin-arm64 --outfile bin/gbrain-darwin-arm64 src/cli.ts && bun build --compile --target=bun-linux-x64 --outfile bin/gbrain-linux-x64 src/cli.ts",
|
||||
"build:schema": "bash scripts/build-schema.sh",
|
||||
"build:llms": "bun run scripts/build-llms.ts",
|
||||
"test": "scripts/check-jsonb-pattern.sh && scripts/check-progress-to-stdout.sh && scripts/check-trailing-newline.sh && scripts/check-wasm-embedded.sh && bun run typecheck && bun test --timeout=60000",
|
||||
"test": "scripts/check-jsonb-pattern.sh && scripts/check-progress-to-stdout.sh && scripts/check-wasm-embedded.sh && bun run typecheck && bun test --timeout=60000",
|
||||
"check:wasm": "scripts/check-wasm-embedded.sh",
|
||||
"check:newlines": "scripts/check-trailing-newline.sh",
|
||||
"test:e2e": "bash scripts/run-e2e.sh",
|
||||
"typecheck": "tsc --noEmit",
|
||||
"check:jsonb": "scripts/check-jsonb-pattern.sh",
|
||||
@@ -64,7 +63,6 @@
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/bun": "latest",
|
||||
"bun-types": "^1.3.13",
|
||||
"typescript": "^5.6.0"
|
||||
},
|
||||
"trustedDependencies": [
|
||||
|
||||
@@ -1,44 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# CI guard: every text file under src/, test/, and the repo root .yml/.md
|
||||
# files must end with a newline. POSIX-noncompliant trailing data shows up
|
||||
# as a phantom diff on every future edit and trips most linters.
|
||||
#
|
||||
# Sibling to scripts/check-progress-to-stdout.sh and
|
||||
# scripts/check-jsonb-pattern.sh per CLAUDE.md's CI guard pattern.
|
||||
# Wired into `bun run test` via package.json's `test` script.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Files to check: anything tracked under src/ + test/ that's a code/text file.
|
||||
# Also the top-level *.yml + *.md the repo controls. Portable to bash 3.2
|
||||
# (macOS default) — no mapfile, no associative arrays.
|
||||
files=$(
|
||||
git ls-files \
|
||||
'src/**/*.ts' 'src/**/*.js' 'src/**/*.json' 'src/**/*.sql' 'src/**/*.md' \
|
||||
'test/**/*.ts' 'test/**/*.js' 'test/**/*.json' 'test/**/*.md' \
|
||||
'gbrain.yml' '*.md' \
|
||||
2>/dev/null | sort -u
|
||||
)
|
||||
|
||||
missing=""
|
||||
total=0
|
||||
while IFS= read -r f; do
|
||||
[ -n "$f" ] || continue
|
||||
[ -f "$f" ] || continue
|
||||
[ -s "$f" ] || continue
|
||||
total=$((total + 1))
|
||||
if [ -n "$(tail -c 1 "$f")" ]; then
|
||||
missing="${missing} $f"$'\n'
|
||||
fi
|
||||
done <<< "$files"
|
||||
|
||||
if [ -n "$missing" ]; then
|
||||
echo "ERROR: the following files are missing a trailing newline:" >&2
|
||||
printf '%s' "$missing" >&2
|
||||
echo >&2
|
||||
echo "Fix: append a newline. e.g. \`printf '\\n' >> <file>\` or your editor's" >&2
|
||||
echo "'final newline' setting (most editors do this automatically)." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "trailing-newline check: ok ($total files)"
|
||||
+1
-6
@@ -15,11 +15,6 @@
|
||||
# the natural per-file test time of 5-10s.
|
||||
#
|
||||
# Exits non-zero on the first failing file so CI fails fast.
|
||||
#
|
||||
# `--timeout=60000` matches the unit test suite. Bun's default is 5s,
|
||||
# which is too tight for setupDB's TRUNCATE CASCADE on ~30 tables on
|
||||
# CI runners under load (one CI flake observed on PR #475 hitting
|
||||
# exactly 5000.09ms in the Tags beforeAll).
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
@@ -35,7 +30,7 @@ for f in test/e2e/*.test.ts; do
|
||||
name=$(basename "$f")
|
||||
echo ""
|
||||
echo "=== $name ==="
|
||||
if output=$(bun test --timeout=60000 "$f" 2>&1); then
|
||||
if output=$(bun test "$f" 2>&1); then
|
||||
pass_files=$((pass_files + 1))
|
||||
# Extract pass/fail counts from bun's summary (e.g., "123 pass")
|
||||
p=$(echo "$output" | grep -oE '[0-9]+ pass' | tail -1 | grep -oE '[0-9]+' || echo 0)
|
||||
|
||||
@@ -1,76 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Partition unit test files into N shards by stable hash and run one shard.
|
||||
#
|
||||
# Usage: scripts/test-shard.sh <shard-index> <total-shards>
|
||||
# shard-index: 1-based (1..N)
|
||||
# total-shards: positive integer
|
||||
#
|
||||
# E2E tests under test/e2e/ are excluded — they need DATABASE_URL and run via
|
||||
# bun run test:e2e separately.
|
||||
#
|
||||
# Stable partitioning: a file's shard is `(hash(path) % N) + 1`. Same file
|
||||
# lands in the same shard on every run, regardless of how many other files
|
||||
# exist, so retries are reproducible. Hash is FNV-1a — pure shell, no jq.
|
||||
set -euo pipefail
|
||||
|
||||
if [ "$#" -ne 2 ]; then
|
||||
echo "usage: scripts/test-shard.sh <shard-index> <total-shards>" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
SHARD_INDEX="$1"
|
||||
TOTAL_SHARDS="$2"
|
||||
|
||||
if ! [[ "$SHARD_INDEX" =~ ^[0-9]+$ ]] || ! [[ "$TOTAL_SHARDS" =~ ^[0-9]+$ ]]; then
|
||||
echo "error: shard index and total must be positive integers" >&2
|
||||
exit 1
|
||||
fi
|
||||
if [ "$SHARD_INDEX" -lt 1 ] || [ "$SHARD_INDEX" -gt "$TOTAL_SHARDS" ]; then
|
||||
echo "error: shard index $SHARD_INDEX out of range 1..$TOTAL_SHARDS" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
# Find all unit test files, deterministic order. Excludes test/e2e/.
|
||||
# Portable: avoid `mapfile` (bash 4+) so this runs on macOS bash 3.2 too.
|
||||
FILES=()
|
||||
while IFS= read -r line; do
|
||||
FILES+=("$line")
|
||||
done < <(find test -name '*.test.ts' -not -path 'test/e2e/*' | sort)
|
||||
|
||||
if [ "${#FILES[@]}" -eq 0 ]; then
|
||||
echo "no test files found under test/" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# FNV-1a 32-bit hash of a string — implemented in pure bash so we don't depend
|
||||
# on python/openssl/etc on the runner. Output is decimal.
|
||||
fnv1a() {
|
||||
local str="$1"
|
||||
local h=2166136261 # FNV offset basis
|
||||
local i ord
|
||||
for (( i=0; i<${#str}; i++ )); do
|
||||
ord=$(printf '%d' "'${str:$i:1}")
|
||||
h=$(( (h ^ ord) & 0xFFFFFFFF ))
|
||||
h=$(( (h * 16777619) & 0xFFFFFFFF ))
|
||||
done
|
||||
echo "$h"
|
||||
}
|
||||
|
||||
SHARD_FILES=()
|
||||
for f in "${FILES[@]}"; do
|
||||
hash=$(fnv1a "$f")
|
||||
bucket=$(( hash % TOTAL_SHARDS + 1 ))
|
||||
if [ "$bucket" -eq "$SHARD_INDEX" ]; then
|
||||
SHARD_FILES+=("$f")
|
||||
fi
|
||||
done
|
||||
|
||||
echo "shard $SHARD_INDEX/$TOTAL_SHARDS: ${#SHARD_FILES[@]}/${#FILES[@]} files"
|
||||
if [ "${#SHARD_FILES[@]}" -eq 0 ]; then
|
||||
echo "warning: shard $SHARD_INDEX has no files (rehash or reduce shard count)" >&2
|
||||
exit 0
|
||||
fi
|
||||
|
||||
exec bun test --timeout=60000 "${SHARD_FILES[@]}"
|
||||
+1
-3
@@ -13,15 +13,13 @@ This is the dispatcher. Skills are the implementation. **Read the skill file bef
|
||||
|
||||
| Trigger | Skill |
|
||||
|---------|-------|
|
||||
| "What do we know about", "tell me about", "search for", "who is", "background on", "notes on" | `skills/query/SKILL.md` |
|
||||
| "What do we know about", "tell me about", "search for" | `skills/query/SKILL.md` |
|
||||
| "Who knows who", "relationship between", "connections", "graph query" | `skills/query/SKILL.md` (use graph-query) |
|
||||
| Creating/enriching a person or company page | `skills/enrich/SKILL.md` |
|
||||
| Where does a new file go? Filing rules | `skills/repo-architecture/SKILL.md` |
|
||||
| Fix broken citations in brain pages | `skills/citation-fixer/SKILL.md` |
|
||||
| "citation audit", "check citations", "fix citations" | `skills/citation-fixer/SKILL.md` (focused fix). For broader brain health, chain into `skills/maintain/SKILL.md` |
|
||||
| "Research", "track", "extract from email", "investor updates", "donations" | `skills/data-research/SKILL.md` |
|
||||
| Share a brain page as a link | `skills/publish/SKILL.md` |
|
||||
| "validate frontmatter", "check frontmatter", "fix frontmatter", "frontmatter audit", "brain lint" | `skills/frontmatter-guard/SKILL.md` |
|
||||
|
||||
## Content & media ingestion
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
// Routing eval fixtures for skills/citation-fixer. Check 5 (W2, v0.17).
|
||||
// Layer A (structural) requires intents to contain trigger words from
|
||||
// the resolver. Paraphrase the trigger framing, not its meaning.
|
||||
{"intent": "please fix citations in the latest batch of brain pages", "expected_skill": "citation-fixer"}
|
||||
{"intent": "I need to fix citations across these pages", "expected_skill": "citation-fixer"}
|
||||
{"intent": "please fix broken citations across the latest batch of pages", "expected_skill": "citation-fixer"}
|
||||
{"intent": "I think we need to fix broken citations in these brain pages", "expected_skill": "citation-fixer"}
|
||||
// Negative case: something that sounds similar but should NOT route here.
|
||||
{"intent": "What does this book say about mentorship", "expected_skill": null, "ambiguous_with": []}
|
||||
|
||||
+12
-1
@@ -55,7 +55,18 @@ they building, what makes them tick, where are they headed.
|
||||
|
||||
## Citation Requirements (MANDATORY)
|
||||
|
||||
> **Convention:** see `skills/conventions/quality.md` for citation formats and source precedence.
|
||||
Every fact must carry an inline `[Source: ...]` citation.
|
||||
|
||||
Three formats:
|
||||
- **Direct attribution:** `[Source: User, {context}, YYYY-MM-DD]`
|
||||
- **API/external:** `[Source: {provider} enrichment, YYYY-MM-DD]`
|
||||
- **Synthesis:** `[Source: compiled from {list of sources}]`
|
||||
|
||||
Source precedence (highest to lowest):
|
||||
1. User's direct statements
|
||||
2. Compiled truth (pre-existing brain synthesis)
|
||||
3. Timeline entries (raw evidence)
|
||||
4. External sources (API enrichment, web search)
|
||||
|
||||
When sources conflict, note the contradiction with both citations.
|
||||
|
||||
|
||||
@@ -1,180 +0,0 @@
|
||||
---
|
||||
name: frontmatter-guard
|
||||
version: 1.0.0
|
||||
description: |
|
||||
Validate and auto-repair YAML frontmatter on brain pages. Catches malformed
|
||||
pages before they enter the brain (missing closing ---, nested quotes, slug
|
||||
mismatches, null bytes, empty frontmatter, YAML parse failures). Wraps the
|
||||
`gbrain frontmatter` CLI for agent-driven workflows.
|
||||
triggers:
|
||||
- "validate frontmatter"
|
||||
- "check frontmatter"
|
||||
- "fix frontmatter"
|
||||
- "frontmatter audit"
|
||||
- "brain lint"
|
||||
tools:
|
||||
- exec
|
||||
mutating: true
|
||||
---
|
||||
|
||||
# Frontmatter Guard Skill
|
||||
|
||||
> **Convention:** see `skills/conventions/quality.md` for citation rules; this skill is structural validation, not citation auditing.
|
||||
|
||||
## Contract
|
||||
|
||||
This skill guarantees:
|
||||
- Every brain page is scanned against the seven canonical frontmatter validation classes
|
||||
- Mechanical errors (nested quotes, missing closing `---`, null bytes, slug mismatch) are auto-repairable on demand with `.bak` backups
|
||||
- Validation logic is shared with `gbrain doctor`'s `frontmatter_integrity` subcheck — single source of truth
|
||||
- Reports per source (gbrain is multi-source since v0.18.0); never silently audits the wrong root
|
||||
|
||||
## Why This Exists
|
||||
|
||||
Brain pages pile up over months. Agents write them with malformed frontmatter:
|
||||
- Missing closing `---` (entity detector bugs)
|
||||
- Unstructured YAML in meeting pages (ingestion bugs)
|
||||
- Slug mismatches (path renames not propagated)
|
||||
- Null bytes (binary corruption from copy-paste accidents)
|
||||
- Nested double quotes in titles (`title: "Phil "Nick" Last"`)
|
||||
|
||||
Without a guard, these accumulate silently until `gbrain sync` chokes or search returns garbage. The guard makes the failure visible at audit time and trivially fixable.
|
||||
|
||||
## Validation classes
|
||||
|
||||
| Code | Meaning | Auto-fixable? |
|
||||
|------|---------|---------------|
|
||||
| `MISSING_OPEN` | File doesn't start with `---` | No (needs human) |
|
||||
| `MISSING_CLOSE` | No closing `---` before first heading | Yes |
|
||||
| `YAML_PARSE` | YAML failed to parse | Sometimes (depends on cause) |
|
||||
| `SLUG_MISMATCH` | Frontmatter `slug:` differs from path-derived slug | Yes (removes the field) |
|
||||
| `NULL_BYTES` | Binary corruption (`\x00`) | Yes |
|
||||
| `NESTED_QUOTES` | `title: "outer "inner" outer"` shape | Yes |
|
||||
| `EMPTY_FRONTMATTER` | Open + close present but nothing between | No (needs human) |
|
||||
|
||||
## Phases
|
||||
|
||||
### Phase 1: Audit
|
||||
|
||||
Run a read-only scan across all registered sources (or one with `--source <id>`).
|
||||
|
||||
```bash
|
||||
gbrain frontmatter audit --json
|
||||
```
|
||||
|
||||
Reports:
|
||||
- Per-source counts grouped by error code
|
||||
- Sample of up to 20 affected pages per source
|
||||
- Total count
|
||||
- Scan timestamp
|
||||
|
||||
Output is JSON; agents parse `errors_by_code` and `per_source` to decide next steps.
|
||||
|
||||
### Phase 2: Validate one path
|
||||
|
||||
Validate a single file or directory (does not require source registration):
|
||||
|
||||
```bash
|
||||
gbrain frontmatter validate <path> --json
|
||||
```
|
||||
|
||||
Exit code 0 = clean; 1 = errors found. Use this in CI pipelines or pre-commit hooks.
|
||||
|
||||
### Phase 3: Fix
|
||||
|
||||
When issues are found:
|
||||
|
||||
```bash
|
||||
gbrain frontmatter validate <path> --fix
|
||||
```
|
||||
|
||||
`--fix` writes `<file>.bak` for every modified file before mutating. The backup is the safety contract — works whether the brain is a git repo or a plain directory.
|
||||
|
||||
`--dry-run` previews without writing. Use this before applying fixes in batch.
|
||||
|
||||
### Phase 4: Pre-commit hook (optional)
|
||||
|
||||
For brain repos that ARE git repos, install the pre-commit hook to block malformed pages from being committed in the first place:
|
||||
|
||||
```bash
|
||||
gbrain frontmatter install-hook [--source <id>]
|
||||
```
|
||||
|
||||
The hook runs `gbrain frontmatter validate` against staged `.md`/`.mdx` files. Bypass with `git commit --no-verify`.
|
||||
|
||||
## Trigger words
|
||||
|
||||
When the user says any of these, route here:
|
||||
- "validate frontmatter"
|
||||
- "check frontmatter"
|
||||
- "fix frontmatter"
|
||||
- "frontmatter audit"
|
||||
- "brain lint"
|
||||
|
||||
## Output rules
|
||||
|
||||
- Always run `gbrain frontmatter audit --json` first; never assume a brain is clean.
|
||||
- Surface counts to the user in plain language; do not dump raw JSON.
|
||||
- For `--fix` operations: state how many files will be modified BEFORE running, then confirm.
|
||||
- `SLUG_MISMATCH` fixes remove the frontmatter `slug:` field — gbrain derives slug from path. Mention this when the user's title is intentionally renamed.
|
||||
- Never auto-fix `MISSING_OPEN` or `EMPTY_FRONTMATTER` without explicit user input — these usually mean a human author started a page and didn't finish.
|
||||
|
||||
## Chains with
|
||||
|
||||
- `gbrain doctor` — the `frontmatter_integrity` subcheck reports the same counts as `audit`.
|
||||
- `skills/maintain/SKILL.md` — broader brain health audit; chain after this skill if other classes of issue are suspected.
|
||||
- `skills/lint/SKILL.md` (via `gbrain lint`) — overlapping rules for skill-file lint; the `frontmatter-*` rule names in lint output come from this skill's validation surface.
|
||||
|
||||
## Output Format
|
||||
|
||||
Audit summary (terse, agent-friendly):
|
||||
|
||||
```
|
||||
Frontmatter audit — 17 issue(s) across 1 source(s)
|
||||
|
||||
[default] /Users/me/brain
|
||||
17 issue(s)
|
||||
MISSING_CLOSE: 8
|
||||
NESTED_QUOTES: 5
|
||||
NULL_BYTES: 4
|
||||
sample:
|
||||
people/jane.md — MISSING_CLOSE
|
||||
companies/acme.md — NESTED_QUOTES
|
||||
(+ 12 more)
|
||||
|
||||
Fix with: gbrain frontmatter validate /Users/me/brain --fix
|
||||
```
|
||||
|
||||
JSON envelope (when `--json` is passed):
|
||||
|
||||
```json
|
||||
{
|
||||
"ok": false,
|
||||
"total": 17,
|
||||
"errors_by_code": { "MISSING_CLOSE": 8, "NESTED_QUOTES": 5, "NULL_BYTES": 4 },
|
||||
"per_source": [
|
||||
{
|
||||
"source_id": "default",
|
||||
"source_path": "/Users/me/brain",
|
||||
"total": 17,
|
||||
"errors_by_code": { "MISSING_CLOSE": 8, "NESTED_QUOTES": 5, "NULL_BYTES": 4 },
|
||||
"sample": [{ "path": "people/jane.md", "codes": ["MISSING_CLOSE"] }]
|
||||
}
|
||||
],
|
||||
"scanned_at": "2026-04-25T22:30:00.000Z"
|
||||
}
|
||||
```
|
||||
|
||||
`gbrain frontmatter validate <path> --json` returns a similar envelope keyed on per-file results instead of per-source.
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
**Don't auto-fix `MISSING_OPEN` or `EMPTY_FRONTMATTER` without user input.** These usually mean a human author started a page and didn't finish — silently inserting `---` markers around an unfinished draft is wrong.
|
||||
|
||||
**Don't use `--fix` to "make doctor green" without reading the audit first.** SLUG_MISMATCH cases are surfaced for manual review specifically because gbrain derives the slug from path. A mismatch usually means the user renamed a file intentionally; auto-removing the slug field is the right outcome only when you've confirmed the rename was deliberate.
|
||||
|
||||
**Don't skip the `.bak` backups.** The `.bak` is the safety contract for non-git brain repos. If `.bak` files accumulate after a fix run, that's a feature, not a bug — the user can review the diffs and delete the backups when satisfied.
|
||||
|
||||
**Don't run `audit` on a brain where sources aren't registered.** The CLI returns "no registered sources to audit" gracefully, but the migration emits a `skipped: no_sources` phase result. Don't paper over this with a manual path-walk; the right fix is to register the source via `gbrain sources add`.
|
||||
|
||||
**Don't install the pre-commit hook on non-git brain dirs.** The install-hook command skips them automatically with a one-line note. If you see "skipped — not a git repo" and want validation at write time anyway, use the `audit` command on a cron schedule.
|
||||
@@ -1,8 +0,0 @@
|
||||
// Routing eval fixtures for skills/frontmatter-guard. Check 5 (W2, v0.17).
|
||||
// Layer A (structural) requires intents to contain trigger words from
|
||||
// the resolver. Paraphrase the trigger framing, not its meaning.
|
||||
{"intent": "please validate frontmatter on the latest batch of brain pages", "expected_skill": "frontmatter-guard"}
|
||||
{"intent": "fix frontmatter on these pages", "expected_skill": "frontmatter-guard"}
|
||||
{"intent": "I want to run a frontmatter audit across the brain", "expected_skill": "frontmatter-guard"}
|
||||
// Negative case: something that sounds similar but should NOT route here.
|
||||
{"intent": "what's for breakfast", "expected_skill": null, "ambiguous_with": []}
|
||||
@@ -8,6 +8,7 @@ description: |
|
||||
triggers:
|
||||
- "brain health"
|
||||
- "check backlinks"
|
||||
- "citation audit"
|
||||
- "maintenance"
|
||||
- "orphan pages"
|
||||
- "stale pages"
|
||||
|
||||
@@ -44,11 +44,6 @@
|
||||
"path": "publish/SKILL.md",
|
||||
"description": "Share brain pages as beautiful password-protected HTML (code + skill pair, zero LLM calls)"
|
||||
},
|
||||
{
|
||||
"name": "frontmatter-guard",
|
||||
"path": "frontmatter-guard/SKILL.md",
|
||||
"description": "Validate and auto-repair YAML frontmatter on brain pages; gates against malformed YAML, missing closing ---, nested quotes, slug mismatches, null bytes"
|
||||
},
|
||||
{
|
||||
"name": "signal-detector",
|
||||
"path": "signal-detector/SKILL.md",
|
||||
|
||||
@@ -1,113 +0,0 @@
|
||||
# v0.22.4 — Frontmatter Guard
|
||||
|
||||
## What ships
|
||||
|
||||
- `gbrain frontmatter` CLI (validate / audit / install-hook)
|
||||
- `frontmatter_integrity` subcheck under `gbrain doctor`
|
||||
- New `frontmatter-guard` skill (`skills/frontmatter-guard/SKILL.md`)
|
||||
- Pre-commit hook helper for git-backed brain repos
|
||||
- Audit-only migration that scans every registered source, writes
|
||||
`~/.gbrain/migrations/v0.22.4-audit.json`, and queues per-source TODO
|
||||
entries to `~/.gbrain/migrations/pending-host-work.jsonl`
|
||||
- 0 warnings on `gbrain check-resolvable` (down from 7 on master)
|
||||
|
||||
## What the agent should do post-upgrade
|
||||
|
||||
The orchestrator handles the mechanical side. Your job is to surface the audit
|
||||
to the user and apply fixes with their consent.
|
||||
|
||||
### 1. Run the orchestrator
|
||||
|
||||
```bash
|
||||
gbrain apply-migrations --yes
|
||||
```
|
||||
|
||||
This runs three idempotent phases:
|
||||
|
||||
- `schema` (no-op in v0.22.4)
|
||||
- `audit` — `scanBrainSources()` over every registered source; writes
|
||||
`~/.gbrain/migrations/v0.22.4-audit.json`
|
||||
- `emit-todo` — appends one entry per source-with-issues to
|
||||
`~/.gbrain/migrations/pending-host-work.jsonl`. Each entry contains the
|
||||
exact `gbrain frontmatter validate <source-path> --fix` command.
|
||||
|
||||
The migration **never mutates brain pages**. Auto-fixes only run when the user
|
||||
explicitly invokes the validate-with-fix command below.
|
||||
|
||||
### 2. Read the audit report
|
||||
|
||||
```bash
|
||||
cat ~/.gbrain/migrations/v0.22.4-audit.json
|
||||
```
|
||||
|
||||
The shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"ok": false,
|
||||
"total": 17,
|
||||
"errors_by_code": { "MISSING_CLOSE": 8, "NESTED_QUOTES": 5, "NULL_BYTES": 4 },
|
||||
"per_source": [
|
||||
{
|
||||
"source_id": "default",
|
||||
"source_path": "/Users/me/brain",
|
||||
"total": 17,
|
||||
"errors_by_code": { "MISSING_CLOSE": 8, "NESTED_QUOTES": 5, "NULL_BYTES": 4 },
|
||||
"sample": [
|
||||
{ "path": "people/jane.md", "codes": ["MISSING_CLOSE"] }
|
||||
]
|
||||
}
|
||||
],
|
||||
"scanned_at": "2026-04-25T22:30:00.000Z"
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Surface the report to the user
|
||||
|
||||
State the per-source counts in plain language. Example:
|
||||
|
||||
> "v0.22.4 ships frontmatter-guard. I ran an audit and found 17 issues across
|
||||
> 1 source (default: 8 MISSING_CLOSE, 5 NESTED_QUOTES, 4 NULL_BYTES). The
|
||||
> mechanical errors are auto-fixable; SLUG_MISMATCH cases (if any) need your
|
||||
> review. Want me to fix the auto-fixable ones now?"
|
||||
|
||||
### 4. Run the fix (with consent)
|
||||
|
||||
Per source with issues, the queued command is:
|
||||
|
||||
```bash
|
||||
gbrain frontmatter validate <source_path> --fix
|
||||
```
|
||||
|
||||
`--fix` writes a `.bak` backup for every modified file. SLUG_MISMATCH errors
|
||||
are surfaced for manual review (not auto-fixed) — gbrain derives slugs from
|
||||
path, so a mismatched slug usually means the user renamed the file
|
||||
intentionally or the slug field is stale.
|
||||
|
||||
### 5. (Optional) Install the pre-commit hook
|
||||
|
||||
For git-backed sources only:
|
||||
|
||||
```bash
|
||||
gbrain frontmatter install-hook [--source <id>]
|
||||
```
|
||||
|
||||
This blocks future malformed-frontmatter commits at the git layer. Bypass with
|
||||
`git commit --no-verify`. Skip this step for non-git brains.
|
||||
|
||||
### 6. Verify
|
||||
|
||||
```bash
|
||||
gbrain doctor --json | jq '.checks[] | select(.name == "frontmatter_integrity")'
|
||||
gbrain frontmatter audit --json | jq '.total'
|
||||
```
|
||||
|
||||
Both should report 0 issues after fixes are applied.
|
||||
|
||||
### 7. If anything fails
|
||||
|
||||
Open an issue at https://github.com/garrytan/gbrain/issues with:
|
||||
- output of `gbrain doctor`
|
||||
- contents of `~/.gbrain/migrations/v0.22.4-audit.json`
|
||||
- contents of `~/.gbrain/upgrade-errors.jsonl` if it exists
|
||||
- which step broke
|
||||
@@ -12,8 +12,6 @@ triggers:
|
||||
- "what happened"
|
||||
- "search for"
|
||||
- "look up"
|
||||
- "background on"
|
||||
- "notes on"
|
||||
- "who knows who"
|
||||
- "relationship between"
|
||||
- "connections"
|
||||
|
||||
+3
-25
@@ -19,7 +19,7 @@ for (const op of operations) {
|
||||
}
|
||||
|
||||
// CLI-only commands that bypass the operation layer
|
||||
const CLI_ONLY = new Set(['init', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'storage', 'repos', 'code-def', 'code-refs', 'reindex-code', 'code-callers', 'code-callees', 'frontmatter', 'auth']);
|
||||
const CLI_ONLY = new Set(['init', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'repos', 'code-def', 'code-refs', 'reindex-code', 'code-callers', 'code-callees']);
|
||||
|
||||
async function main() {
|
||||
// Parse global flags (--quiet / --progress-json / --progress-interval)
|
||||
@@ -285,11 +285,6 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
await runIntegrations(args);
|
||||
return;
|
||||
}
|
||||
if (command === 'auth') {
|
||||
const { runAuth } = await import('./commands/auth.ts');
|
||||
await runAuth(args);
|
||||
return;
|
||||
}
|
||||
if (command === 'resolvers') {
|
||||
const { runResolvers } = await import('./commands/resolvers.ts');
|
||||
await runResolvers(args);
|
||||
@@ -310,11 +305,6 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
await runBacklinks(args);
|
||||
return;
|
||||
}
|
||||
if (command === 'frontmatter') {
|
||||
const { runFrontmatter } = await import('./commands/frontmatter.ts');
|
||||
await runFrontmatter(args);
|
||||
return;
|
||||
}
|
||||
if (command === 'lint') {
|
||||
const { runLint } = await import('./commands/lint.ts');
|
||||
await runLint(args);
|
||||
@@ -452,7 +442,7 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
}
|
||||
case 'serve': {
|
||||
const { runServe } = await import('./commands/serve.ts');
|
||||
await runServe(engine, args);
|
||||
await runServe(engine);
|
||||
return; // serve doesn't disconnect
|
||||
}
|
||||
case 'call': {
|
||||
@@ -530,11 +520,6 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
await runSources(engine, args);
|
||||
break;
|
||||
}
|
||||
case 'storage': {
|
||||
const { runStorage } = await import('./commands/storage.ts');
|
||||
await runStorage(engine, args);
|
||||
break;
|
||||
}
|
||||
case 'code-def': {
|
||||
const { runCodeDef } = await import('./commands/code-def.ts');
|
||||
await runCodeDef(engine, args);
|
||||
@@ -591,10 +576,7 @@ async function connectEngine(): Promise<BrainEngine> {
|
||||
}
|
||||
const { createEngine } = await import('./core/engine-factory.ts');
|
||||
const engine = await createEngine(toEngineConfig(config));
|
||||
const noRetry = process.argv.includes('--no-retry-connect') ||
|
||||
process.env.GBRAIN_NO_RETRY_CONNECT === '1';
|
||||
const { connectWithRetry } = await import('./core/db.ts');
|
||||
await connectWithRetry(engine, toEngineConfig(config), { noRetry });
|
||||
await engine.connect(toEngineConfig(config));
|
||||
return engine;
|
||||
}
|
||||
|
||||
@@ -650,8 +632,6 @@ IMPORT/EXPORT
|
||||
sync --watch [--interval N] Continuous sync (loops until stopped)
|
||||
sync --install-cron Install persistent sync daemon
|
||||
export [--dir ./out/] Export to markdown
|
||||
export --restore-only [--repo <p>] Restore missing supabase-only files
|
||||
[--type T] [--slug-prefix S] With optional filters
|
||||
|
||||
FILES
|
||||
files list [slug] List stored files
|
||||
@@ -733,8 +713,6 @@ ADMIN
|
||||
features [--json] [--auto-fix] Scan usage + recommend unused features
|
||||
autopilot [--repo] [--interval N] Self-maintaining brain daemon
|
||||
config [show|get|set] <key> [val] Brain config
|
||||
storage status [--repo <path>] Storage tier status and health
|
||||
[--json] (git-tracked vs supabase-only)
|
||||
serve MCP server (stdio)
|
||||
call <tool> '<json>' Raw tool invocation
|
||||
version Version info
|
||||
|
||||
+31
-52
@@ -1,29 +1,20 @@
|
||||
#!/usr/bin/env bun
|
||||
/**
|
||||
* GBrain token management.
|
||||
* GBrain token management — standalone script, no gbrain CLI dependency.
|
||||
*
|
||||
* Wired into the CLI as of v0.22.5:
|
||||
* gbrain auth create "claude-desktop"
|
||||
* gbrain auth list
|
||||
* gbrain auth revoke "claude-desktop"
|
||||
* gbrain auth test <url> --token <token>
|
||||
*
|
||||
* Also runs standalone (no compiled binary required):
|
||||
* Usage:
|
||||
* DATABASE_URL=... bun run src/commands/auth.ts create "claude-desktop"
|
||||
*
|
||||
* Both paths require DATABASE_URL or GBRAIN_DATABASE_URL (except `test`,
|
||||
* which only hits the remote URL and doesn't need a local DB).
|
||||
* DATABASE_URL=... bun run src/commands/auth.ts list
|
||||
* DATABASE_URL=... bun run src/commands/auth.ts revoke "claude-desktop"
|
||||
* DATABASE_URL=... bun run src/commands/auth.ts test <url> --token <token>
|
||||
*/
|
||||
import postgres from 'postgres';
|
||||
import { createHash, randomBytes } from 'crypto';
|
||||
|
||||
function getDatabaseUrl(requireDb: boolean): string | undefined {
|
||||
const url = process.env.DATABASE_URL || process.env.GBRAIN_DATABASE_URL;
|
||||
if (!url && requireDb) {
|
||||
console.error('Set DATABASE_URL or GBRAIN_DATABASE_URL environment variable.');
|
||||
process.exit(1);
|
||||
}
|
||||
return url;
|
||||
const DATABASE_URL = process.env.DATABASE_URL || process.env.GBRAIN_DATABASE_URL;
|
||||
if (!DATABASE_URL && process.argv[2] !== 'test') {
|
||||
console.error('Set DATABASE_URL or GBRAIN_DATABASE_URL environment variable.');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
function hashToken(token: string): string {
|
||||
@@ -36,7 +27,7 @@ function generateToken(): string {
|
||||
|
||||
async function create(name: string) {
|
||||
if (!name) { console.error('Usage: auth create <name>'); process.exit(1); }
|
||||
const sql = postgres(getDatabaseUrl(true)!);
|
||||
const sql = postgres(DATABASE_URL!);
|
||||
const token = generateToken();
|
||||
const hash = hashToken(token);
|
||||
|
||||
@@ -62,7 +53,7 @@ async function create(name: string) {
|
||||
}
|
||||
|
||||
async function list() {
|
||||
const sql = postgres(getDatabaseUrl(true)!);
|
||||
const sql = postgres(DATABASE_URL!);
|
||||
try {
|
||||
const rows = await sql`
|
||||
SELECT name, created_at, last_used_at, revoked_at
|
||||
@@ -89,7 +80,7 @@ async function list() {
|
||||
|
||||
async function revoke(name: string) {
|
||||
if (!name) { console.error('Usage: auth revoke <name>'); process.exit(1); }
|
||||
const sql = postgres(getDatabaseUrl(true)!);
|
||||
const sql = postgres(DATABASE_URL!);
|
||||
try {
|
||||
const result = await sql`
|
||||
UPDATE access_tokens SET revoked_at = now()
|
||||
@@ -225,38 +216,26 @@ async function test(url: string, token: string) {
|
||||
console.log(`\n🧠 Your brain is live! (${elapsed}s)`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Entry point for the `gbrain auth` CLI subcommand. Also reused by the
|
||||
* direct-script path (see bottom of file) so `bun run src/commands/auth.ts`
|
||||
* still works.
|
||||
*/
|
||||
export async function runAuth(args: string[]): Promise<void> {
|
||||
const [cmd, ...rest] = args;
|
||||
switch (cmd) {
|
||||
case 'create': await create(rest[0]); return;
|
||||
case 'list': await list(); return;
|
||||
case 'revoke': await revoke(rest[0]); return;
|
||||
case 'test': {
|
||||
const tokenIdx = rest.indexOf('--token');
|
||||
const url = rest.find(a => !a.startsWith('--') && a !== rest[tokenIdx + 1]);
|
||||
const token = tokenIdx >= 0 ? rest[tokenIdx + 1] : '';
|
||||
await test(url || '', token || '');
|
||||
return;
|
||||
}
|
||||
default:
|
||||
console.log(`GBrain Token Management
|
||||
// CLI dispatch
|
||||
const [cmd, ...args] = process.argv.slice(2);
|
||||
switch (cmd) {
|
||||
case 'create': await create(args[0]); break;
|
||||
case 'list': await list(); break;
|
||||
case 'revoke': await revoke(args[0]); break;
|
||||
case 'test': {
|
||||
const tokenIdx = args.indexOf('--token');
|
||||
const url = args.find(a => !a.startsWith('--') && a !== args[tokenIdx + 1]);
|
||||
const token = tokenIdx >= 0 ? args[tokenIdx + 1] : '';
|
||||
await test(url || '', token || '');
|
||||
break;
|
||||
}
|
||||
default:
|
||||
console.log(`GBrain Token Management
|
||||
|
||||
Usage:
|
||||
gbrain auth create <name> Create a new access token
|
||||
gbrain auth list List all tokens
|
||||
gbrain auth revoke <name> Revoke a token
|
||||
gbrain auth test <url> --token <t> Smoke-test a remote MCP server
|
||||
bun run src/commands/auth.ts create <name> Create a new access token
|
||||
bun run src/commands/auth.ts list List all tokens
|
||||
bun run src/commands/auth.ts revoke <name> Revoke a token
|
||||
bun run src/commands/auth.ts test <url> --token <token> Smoke test a remote MCP server
|
||||
`);
|
||||
}
|
||||
}
|
||||
|
||||
// Direct-script entry point — only runs when this file is invoked as the main module
|
||||
// (e.g. `bun run src/commands/auth.ts ...`). When imported by cli.ts, this block is skipped.
|
||||
if (import.meta.main) {
|
||||
await runAuth(process.argv.slice(2));
|
||||
}
|
||||
|
||||
@@ -147,41 +147,22 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
|
||||
let stopping = false;
|
||||
let workerProc: ChildProcess | null = null;
|
||||
let crashCount = 0;
|
||||
let lastWorkerStartTime = 0;
|
||||
|
||||
// Stable-run reset window (matches MinionSupervisor.ts:471-476 pattern). If the
|
||||
// worker ran > 5min before exit, treat as a fresh cycle (crashCount=1) so the
|
||||
// RSS watchdog firing hourly does NOT trip autopilot's give-up threshold after
|
||||
// ~5 hours of healthy uptime.
|
||||
const STABLE_RUN_RESET_MS = 5 * 60 * 1000;
|
||||
|
||||
if (spawnManagedWorker) {
|
||||
const cliPath = resolveGbrainCliPath();
|
||||
const startWorker = () => {
|
||||
// Inject the RSS watchdog default (2048 MB) for the autopilot-supervised
|
||||
// worker. Bare `gbrain jobs work` has no default; the supervisor and
|
||||
// autopilot are the production paths that opt in.
|
||||
const args = ['jobs', 'work', '--max-rss', '2048'];
|
||||
const child = spawn(cliPath, args, { stdio: 'inherit', env: process.env });
|
||||
const child = spawn(cliPath, ['jobs', 'work'], { stdio: 'inherit', env: process.env });
|
||||
workerProc = child;
|
||||
lastWorkerStartTime = Date.now();
|
||||
console.log(`[autopilot] Minions worker spawned (pid: ${child.pid}, watchdog: 2048MB)`);
|
||||
console.log(`[autopilot] Minions worker spawned (pid: ${child.pid})`);
|
||||
child.on('exit', (code) => {
|
||||
workerProc = null;
|
||||
if (stopping) return;
|
||||
const runDuration = Date.now() - lastWorkerStartTime;
|
||||
if (runDuration > STABLE_RUN_RESET_MS) {
|
||||
// Stable run — forgive prior crash history. A watchdog-driven hourly
|
||||
// exit (the production path post-fix) lands here every time.
|
||||
crashCount = 1;
|
||||
} else {
|
||||
crashCount++;
|
||||
}
|
||||
if (crashCount >= 5) {
|
||||
console.error(`[autopilot] 5 consecutive worker crashes (run ${runDuration}ms), giving up.`);
|
||||
console.error('[autopilot] 5 consecutive worker crashes, giving up.');
|
||||
process.exit(1);
|
||||
}
|
||||
console.error(`[autopilot] worker exited code=${code} after ${runDuration}ms, restart #${crashCount} in 10s`);
|
||||
crashCount++;
|
||||
console.error(`[autopilot] worker exited code=${code}, restart #${crashCount} in 10s`);
|
||||
setTimeout(startWorker, 10_000);
|
||||
});
|
||||
};
|
||||
@@ -309,12 +290,6 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
|
||||
idempotency_key: `autopilot-cycle:${slot}`,
|
||||
max_attempts: 2,
|
||||
timeout_ms: timeoutMs,
|
||||
// Submission backpressure: when the worker is dead or wedged,
|
||||
// idempotency_key only dedupes within a slot; cross-slot pile-up
|
||||
// is what produced the 28+ waiting-jobs production incident.
|
||||
// maxWaiting: 1 caps at 1 active + 1 waiting; queue.add coalesces
|
||||
// the 3rd+ submission and writes a backpressure-audit JSONL line.
|
||||
maxWaiting: 1,
|
||||
},
|
||||
);
|
||||
if (jsonMode) {
|
||||
|
||||
+5
-75
@@ -5,7 +5,6 @@ import { checkResolvable } from '../core/check-resolvable.ts';
|
||||
import { autoFixDryViolations, type AutoFixReport, type FixOutcome } from '../core/dry-fix.ts';
|
||||
import { findRepoRoot } from '../core/repo-root.ts';
|
||||
import { loadCompletedMigrations } from '../core/preferences.ts';
|
||||
import { compareVersions } from './migrations/index.ts';
|
||||
import { createProgress, startHeartbeat, type ProgressReporter } from '../core/progress.ts';
|
||||
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
|
||||
import type { DbUrlSource } from '../core/config.ts';
|
||||
@@ -111,15 +110,6 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
|
||||
// Typical cause: v0.11.0 stopgap wrote a partial record but nobody ran
|
||||
// `gbrain apply-migrations --yes` afterward. This check fires on every
|
||||
// `gbrain doctor` invocation so your OpenClaw's health skill catches it.
|
||||
//
|
||||
// Forward-progress override: a partial entry for vX.Y.Z is treated as
|
||||
// stale (not stuck) if there is a `complete` entry for any vA.B.C >= vX.Y.Z
|
||||
// anywhere in the file. The reasoning: if a newer migration successfully
|
||||
// landed, the install moved past the older partial — the old record is
|
||||
// historical noise from a stopgap that never finished cleanly, but the
|
||||
// schema clearly advanced. Without this, every install that went through
|
||||
// a v0.11.0 stopgap and then upgraded carries the "MINIONS HALF-INSTALLED"
|
||||
// flag forever, even on installs that have been at v0.22+ for months.
|
||||
try {
|
||||
const completed = loadCompletedMigrations();
|
||||
const byVersion = new Map<string, { complete: boolean; partial: boolean }>();
|
||||
@@ -129,17 +119,8 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
|
||||
if (entry.status === 'partial') seen.partial = true;
|
||||
byVersion.set(entry.version, seen);
|
||||
}
|
||||
const completedVersions = Array.from(byVersion.entries())
|
||||
.filter(([, s]) => s.complete)
|
||||
.map(([v]) => v);
|
||||
const stuck = Array.from(byVersion.entries())
|
||||
.filter(([v, s]) => {
|
||||
if (!s.partial || s.complete) return false;
|
||||
// Forward-progress override: if any version >= v has completed, the
|
||||
// partial is stale. compareVersions returns 1 when first arg is newer.
|
||||
const supersededBy = completedVersions.find(cv => compareVersions(cv, v) >= 0);
|
||||
return supersededBy === undefined;
|
||||
})
|
||||
.filter(([, s]) => s.partial && !s.complete)
|
||||
.map(([v]) => v);
|
||||
if (stuck.length > 0) {
|
||||
checks.push({
|
||||
@@ -249,29 +230,25 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
|
||||
// Without this doctor check, users see "sync blocked" and have no
|
||||
// surface showing which files to fix.
|
||||
try {
|
||||
const { unacknowledgedSyncFailures, loadSyncFailures, summarizeFailuresByCode } = await import('../core/sync.ts');
|
||||
const { unacknowledgedSyncFailures, loadSyncFailures } = await import('../core/sync.ts');
|
||||
const unacked = unacknowledgedSyncFailures();
|
||||
const all = loadSyncFailures();
|
||||
if (unacked.length > 0) {
|
||||
const codeSummary = summarizeFailuresByCode(unacked);
|
||||
const codeBreakdown = codeSummary.map(s => `${s.code}=${s.count}`).join(', ');
|
||||
const preview = unacked.slice(0, 3).map(f => `${f.path} (${f.error.slice(0, 60)})`).join('; ');
|
||||
checks.push({
|
||||
name: 'sync_failures',
|
||||
status: 'warn',
|
||||
message:
|
||||
`${unacked.length} unacknowledged sync failure(s) [${codeBreakdown}]. ${preview}` +
|
||||
`${unacked.length} unacknowledged sync failure(s). ${preview}` +
|
||||
`${unacked.length > 3 ? `, and ${unacked.length - 3} more` : ''}. ` +
|
||||
`Fix the file(s) and re-run 'gbrain sync', or use 'gbrain sync --skip-failed' to acknowledge.`,
|
||||
});
|
||||
} else if (all.length > 0) {
|
||||
// Acknowledged-only: show code breakdown for visibility.
|
||||
const ackedSummary = summarizeFailuresByCode(all);
|
||||
const ackedBreakdown = ackedSummary.map(s => `${s.code}=${s.count}`).join(', ');
|
||||
// Acknowledged-only: informational, not a warning.
|
||||
checks.push({
|
||||
name: 'sync_failures',
|
||||
status: 'ok',
|
||||
message: `${all.length} historical sync failure(s), all acknowledged [${ackedBreakdown}].`,
|
||||
message: `${all.length} historical sync failure(s), all acknowledged.`,
|
||||
});
|
||||
}
|
||||
} catch {
|
||||
@@ -672,53 +649,6 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
|
||||
mbcHb();
|
||||
}
|
||||
|
||||
// 11a. Frontmatter integrity (v0.22.4).
|
||||
// scanBrainSources walks every registered source's local_path on disk
|
||||
// (not from the DB), invoking parseMarkdown(..., {validate:true}) per
|
||||
// file. Reports per-source counts grouped by error code. The fix path is
|
||||
// `gbrain frontmatter validate <source-path> --fix`, which writes .bak
|
||||
// backups so it works for both git and non-git brain repos.
|
||||
progress.heartbeat('frontmatter_integrity');
|
||||
const fmHb = startHeartbeat(progress, 'scanning frontmatter…');
|
||||
try {
|
||||
const { scanBrainSources } = await import('../core/brain-writer.ts');
|
||||
const report = await scanBrainSources(engine);
|
||||
if (report.total === 0) {
|
||||
const sources = report.per_source.length;
|
||||
checks.push({
|
||||
name: 'frontmatter_integrity',
|
||||
status: 'ok',
|
||||
message: sources === 0
|
||||
? 'No registered sources to scan'
|
||||
: `${sources} source(s) clean — no frontmatter issues`,
|
||||
});
|
||||
} else {
|
||||
const sourceMessages: string[] = [];
|
||||
for (const src of report.per_source) {
|
||||
if (src.total === 0) continue;
|
||||
const codes = Object.entries(src.errors_by_code)
|
||||
.map(([k, v]) => `${k}=${v}`)
|
||||
.join(', ');
|
||||
sourceMessages.push(`${src.source_id}: ${src.total} (${codes})`);
|
||||
}
|
||||
checks.push({
|
||||
name: 'frontmatter_integrity',
|
||||
status: 'warn',
|
||||
message:
|
||||
`${report.total} frontmatter issue(s) across ${sourceMessages.length} source(s). ` +
|
||||
`${sourceMessages.join('; ')}. Fix: gbrain frontmatter validate <source-path> --fix`,
|
||||
});
|
||||
}
|
||||
} catch (e) {
|
||||
checks.push({
|
||||
name: 'frontmatter_integrity',
|
||||
status: 'warn',
|
||||
message: `Could not scan frontmatter: ${e instanceof Error ? e.message : String(e)}`,
|
||||
});
|
||||
} finally {
|
||||
fmHb();
|
||||
}
|
||||
|
||||
// 11b. Queue health (v0.19.1 queue-resilience wave).
|
||||
// Postgres-only because PGLite has no multi-process worker surface. Two
|
||||
// subchecks, both cheap (single SELECT each, status-index-covered):
|
||||
|
||||
+3
-135
@@ -220,23 +220,6 @@ async function embedAll(
|
||||
result: EmbedResult,
|
||||
onProgress?: (done: number, total: number, embedded: number) => void,
|
||||
) {
|
||||
// ─────────────────────────────────────────────────────────────
|
||||
// Stale-only fast path: avoid the listPages + per-page getChunks
|
||||
// bomb that pulled every page row + every chunk's embedding column
|
||||
// (~76 MB on a 1.5K-page brain) only to client-side-filter for
|
||||
// chunks where embedding IS NULL. The new path issues one SQL
|
||||
// pre-check + at most one slug-grouped SELECT excluding the
|
||||
// (always-null on stale rows) embedding column. On a 100%-embedded
|
||||
// brain (the autopilot common case) we exit after ~50 bytes wire.
|
||||
//
|
||||
// For --all (staleOnly=false) we keep the original behavior — the
|
||||
// user is explicitly asking to re-embed everything, including
|
||||
// chunks that already have embeddings.
|
||||
// ─────────────────────────────────────────────────────────────
|
||||
if (staleOnly) {
|
||||
return await embedAllStale(engine, dryRun, result, onProgress);
|
||||
}
|
||||
|
||||
const pages = await engine.listPages({ limit: 100000 });
|
||||
let processed = 0;
|
||||
|
||||
@@ -252,7 +235,9 @@ async function embedAll(
|
||||
|
||||
async function embedOnePage(page: typeof pages[number]) {
|
||||
const chunks = await engine.getChunks(page.slug);
|
||||
const toEmbed = chunks; // staleOnly path handled above via embedAllStale
|
||||
const toEmbed = staleOnly
|
||||
? chunks.filter(c => !c.embedded_at)
|
||||
: chunks;
|
||||
|
||||
result.total_chunks += chunks.length;
|
||||
result.skipped += chunks.length - toEmbed.length;
|
||||
@@ -321,120 +306,3 @@ async function embedAll(
|
||||
console.log(`Embedded ${result.embedded} chunks across ${pages.length} pages`);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* SQL-side stale path: replaces the listPages + per-page getChunks
|
||||
* walk with a count + slug-grouped SELECT. Preserves the existing
|
||||
* functional contract (every chunk where embedding IS NULL gets
|
||||
* embedded; nothing else is touched) without paying egress on
|
||||
* already-embedded chunks.
|
||||
*
|
||||
* Why a separate function: the staleOnly path doesn't need
|
||||
* listPages at all and groups by slug differently. Forking the
|
||||
* function makes the read-bytes path explicit and keeps the --all
|
||||
* path verbatim from prior behavior.
|
||||
*
|
||||
* Staleness predicate: `embedding IS NULL`. We deliberately do NOT
|
||||
* use `embedded_at IS NULL` here — the bulk-import path can leave
|
||||
* embedded_at populated while embedding is NULL (see upsertChunks
|
||||
* consistency notes), and `embedding IS NULL` is the truth source
|
||||
* for "this chunk needs an embedding".
|
||||
*/
|
||||
async function embedAllStale(
|
||||
engine: BrainEngine,
|
||||
dryRun: boolean,
|
||||
result: EmbedResult,
|
||||
onProgress?: (done: number, total: number, embedded: number) => void,
|
||||
) {
|
||||
// Pre-flight: 0 stale chunks → nothing to do, no further DB reads.
|
||||
// Cheapest possible exit on the autopilot common case.
|
||||
const staleCount = await engine.countStaleChunks();
|
||||
if (staleCount === 0) {
|
||||
if (dryRun) {
|
||||
console.log('[dry-run] Would embed 0 chunks (0 stale found)');
|
||||
} else {
|
||||
console.log('Embedded 0 chunks (0 stale found)');
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
// Pull only the stale chunks (no embedding column).
|
||||
const staleRows = await engine.listStaleChunks();
|
||||
// Group by slug so each slug → array of stale chunks for batched embedding.
|
||||
const bySlug = new Map<string, typeof staleRows>();
|
||||
for (const row of staleRows) {
|
||||
const list = bySlug.get(row.slug);
|
||||
if (list) list.push(row);
|
||||
else bySlug.set(row.slug, [row]);
|
||||
}
|
||||
|
||||
const slugs = Array.from(bySlug.keys());
|
||||
const totalStaleChunks = staleRows.length;
|
||||
result.total_chunks += totalStaleChunks;
|
||||
// skipped is "chunks we considered and skipped due to having an embedding".
|
||||
// We never considered the non-stale chunks here, so leave skipped at 0.
|
||||
// Callers reading EmbedResult who care about coverage should call
|
||||
// engine.getStats() / engine.getHealth() afterward.
|
||||
|
||||
if (dryRun) {
|
||||
result.would_embed += totalStaleChunks;
|
||||
result.pages_processed += slugs.length;
|
||||
if (onProgress) {
|
||||
// Emit a single tick to satisfy the contract (CLI progress reporters
|
||||
// expect at least one start/finish pair).
|
||||
onProgress(slugs.length, slugs.length, 0);
|
||||
}
|
||||
console.log(`[dry-run] Would embed ${totalStaleChunks} chunks across ${slugs.length} pages`);
|
||||
return;
|
||||
}
|
||||
|
||||
const CONCURRENCY = parseInt(process.env.GBRAIN_EMBED_CONCURRENCY || '20', 10);
|
||||
let processed = 0;
|
||||
|
||||
async function embedOneSlug(slug: string) {
|
||||
const stale = bySlug.get(slug)!;
|
||||
try {
|
||||
const embeddings = await embedBatch(stale.map(c => c.chunk_text));
|
||||
// CRITICAL: passing ONLY the stale indices to upsertChunks would
|
||||
// delete every non-stale chunk on the same page (the != ALL filter
|
||||
// wipes any chunk_index NOT in the input). To preserve them, we
|
||||
// re-fetch existing chunks for this page and merge. Bounded by the
|
||||
// stale slug count, not by total slugs — autopilot common case
|
||||
// is 0 stale (pre-flight short-circuit, never reaches this path).
|
||||
const existing = await engine.getChunks(slug);
|
||||
const staleIdxToEmbedding = new Map<number, Float32Array>();
|
||||
for (let j = 0; j < stale.length; j++) {
|
||||
staleIdxToEmbedding.set(stale[j].chunk_index, embeddings[j]);
|
||||
}
|
||||
const merged: ChunkInput[] = existing.map(c => ({
|
||||
chunk_index: c.chunk_index,
|
||||
chunk_text: c.chunk_text,
|
||||
chunk_source: c.chunk_source,
|
||||
// For stale chunks: pass the new embedding.
|
||||
// For non-stale chunks: pass undefined → COALESCE preserves existing embedding.
|
||||
embedding: staleIdxToEmbedding.get(c.chunk_index) ?? undefined,
|
||||
token_count: c.token_count || Math.ceil(c.chunk_text.length / 4),
|
||||
}));
|
||||
await engine.upsertChunks(slug, merged);
|
||||
result.embedded += stale.length;
|
||||
} catch (e: unknown) {
|
||||
console.error(`\n Error embedding ${slug}: ${e instanceof Error ? e.message : e}`);
|
||||
}
|
||||
processed++;
|
||||
result.pages_processed++;
|
||||
onProgress?.(processed, slugs.length, result.embedded);
|
||||
}
|
||||
|
||||
let nextIdx = 0;
|
||||
async function worker() {
|
||||
while (nextIdx < slugs.length) {
|
||||
const idx = nextIdx++;
|
||||
await embedOneSlug(slugs[idx]);
|
||||
}
|
||||
}
|
||||
|
||||
const numWorkers = Math.min(CONCURRENCY, slugs.length);
|
||||
await Promise.all(Array.from({ length: numWorkers }, () => worker()));
|
||||
|
||||
console.log(`Embedded ${result.embedded} chunks across ${slugs.length} pages`);
|
||||
}
|
||||
|
||||
+4
-97
@@ -1,105 +1,16 @@
|
||||
import { writeFileSync, mkdirSync, existsSync } from 'fs';
|
||||
import { writeFileSync, mkdirSync } from 'fs';
|
||||
import { join, dirname } from 'path';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { serializeMarkdown } from '../core/markdown.ts';
|
||||
import { createProgress } from '../core/progress.ts';
|
||||
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
|
||||
import { loadStorageConfig, isDbOnly } from '../core/storage-config.ts';
|
||||
import { getDefaultSourcePath } from '../core/source-resolver.ts';
|
||||
import type { PageType } from '../core/types.ts';
|
||||
|
||||
export async function runExport(engine: BrainEngine, args: string[]) {
|
||||
const dirIdx = args.indexOf('--dir');
|
||||
const outDir = dirIdx !== -1 ? args[dirIdx + 1] : './export';
|
||||
|
||||
const repoIdx = args.indexOf('--repo');
|
||||
const explicitRepoPath = repoIdx !== -1 ? args[repoIdx + 1] : null;
|
||||
|
||||
const typeIdx = args.indexOf('--type');
|
||||
const typeFilter = typeIdx !== -1 ? (args[typeIdx + 1] as PageType) : undefined;
|
||||
|
||||
const slugPrefixIdx = args.indexOf('--slug-prefix');
|
||||
const slugPrefix = slugPrefixIdx !== -1 ? args[slugPrefixIdx + 1] : undefined;
|
||||
|
||||
const restoreOnly = args.includes('--restore-only');
|
||||
|
||||
// Resolution chain (D5): explicit --repo → typed sources.getDefault() →
|
||||
// hard-error for restore-only paths (never fall through to cwd).
|
||||
// For non-restore exports, repoPath stays null because regular export
|
||||
// doesn't need a brain repo to run (D26 — exports include everything).
|
||||
let repoPath: string | null = explicitRepoPath;
|
||||
if (restoreOnly && !repoPath) {
|
||||
repoPath = await getDefaultSourcePath(engine);
|
||||
if (!repoPath) {
|
||||
console.error(
|
||||
`Error: gbrain export --restore-only requires --repo <path> or a configured\n` +
|
||||
`default source with a local_path. Run \`gbrain sources list\` to inspect\n` +
|
||||
`sources, or pass --repo explicitly.`,
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
// Load storage configuration if repo path is provided
|
||||
const storageConfig = repoPath ? loadStorageConfig(repoPath) : null;
|
||||
|
||||
// D5 + Codex P0: refuse --restore-only when there's no storage config to
|
||||
// scope the restore. Without storageConfig, the selective filter (db_only
|
||||
// pages missing on disk) can't run, and falling through to the full
|
||||
// listPages export silently dumps the entire DB. Catch this before any
|
||||
// page query fires.
|
||||
if (restoreOnly && !storageConfig) {
|
||||
console.error(
|
||||
`Error: gbrain export --restore-only requires a storage tiering config\n` +
|
||||
`(gbrain.yml with a "storage:" section) at ${repoPath}/gbrain.yml.\n` +
|
||||
`Without it, there's nothing to scope the restore to.\n` +
|
||||
`Run \`gbrain storage status\` to inspect the current configuration.`,
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Build filters. slugPrefix is engine-side (Issue #13) — no in-memory
|
||||
// post-filter, no full-table load.
|
||||
const filters: import('../core/types.ts').PageFilters = { limit: 100000 };
|
||||
if (typeFilter) filters.type = typeFilter;
|
||||
if (slugPrefix) filters.slugPrefix = slugPrefix;
|
||||
|
||||
let pages: import('../core/types.ts').Page[];
|
||||
|
||||
// Restore-only path: query each db_only directory with slugPrefix instead
|
||||
// of loading every page in the brain. On a 200K-page brain where 95% is
|
||||
// db_only, this is roughly the same load — but on brains where only 5K
|
||||
// out of 200K are db_only, this is a ~40x reduction.
|
||||
if (restoreOnly && repoPath && storageConfig) {
|
||||
const seen = new Set<string>();
|
||||
pages = [];
|
||||
for (const dir of storageConfig.db_only) {
|
||||
const tierFilters: import('../core/types.ts').PageFilters = {
|
||||
...filters,
|
||||
slugPrefix: filters.slugPrefix
|
||||
? // If user passed --slug-prefix, only include tier dirs that start with it.
|
||||
(dir.startsWith(filters.slugPrefix) ? dir : undefined)
|
||||
: dir,
|
||||
};
|
||||
if (!tierFilters.slugPrefix) continue;
|
||||
const tierPages = await engine.listPages(tierFilters);
|
||||
for (const p of tierPages) {
|
||||
if (seen.has(p.slug)) continue;
|
||||
seen.add(p.slug);
|
||||
if (!isDbOnly(p.slug, storageConfig)) continue; // belt-and-suspenders
|
||||
const filePath = join(repoPath, p.slug + '.md');
|
||||
if (existsSync(filePath)) continue;
|
||||
pages.push(p);
|
||||
}
|
||||
}
|
||||
} else {
|
||||
pages = await engine.listPages(filters);
|
||||
}
|
||||
if (restoreOnly) {
|
||||
console.log(`Restoring ${pages.length} db_only pages to ${outDir}/`);
|
||||
} else {
|
||||
console.log(`Exporting ${pages.length} pages to ${outDir}/`);
|
||||
}
|
||||
const pages = await engine.listPages({ limit: 100000 });
|
||||
console.log(`Exporting ${pages.length} pages to ${outDir}/`);
|
||||
|
||||
// Progress on stderr so stdout stays clean for scripts parsing counts.
|
||||
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
|
||||
@@ -141,9 +52,5 @@ export async function runExport(engine: BrainEngine, args: string[]) {
|
||||
|
||||
progress.finish();
|
||||
// Stdout summary preserved so scripts that grep for "Exported N pages" keep working.
|
||||
if (restoreOnly) {
|
||||
console.log(`Restored ${exported} pages to ${outDir}/`);
|
||||
} else {
|
||||
console.log(`Exported ${exported} pages to ${outDir}/`);
|
||||
}
|
||||
console.log(`Exported ${exported} pages to ${outDir}/`);
|
||||
}
|
||||
|
||||
@@ -295,13 +295,6 @@ export interface ExtractOpts {
|
||||
dryRun?: boolean;
|
||||
/** Emit JSON (progress to stderr, result to stdout) instead of human text. */
|
||||
jsonMode?: boolean;
|
||||
/**
|
||||
* Incremental mode: only extract from these specific slugs.
|
||||
* When provided, skips the full directory walk and reads only the
|
||||
* files corresponding to these slugs. Massive perf win on large brains.
|
||||
* Pass undefined or omit for a full walk (CLI / first-run path).
|
||||
*/
|
||||
slugs?: string[];
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -322,21 +315,6 @@ export async function runExtractCore(engine: BrainEngine, opts: ExtractOpts): Pr
|
||||
const jsonMode = !!opts.jsonMode;
|
||||
const result: ExtractResult = { links_created: 0, timeline_entries_created: 0, pages_processed: 0 };
|
||||
|
||||
// Incremental path: if specific slugs provided, only extract from those files.
|
||||
// This is the cycle path — sync tells us what changed, we only re-extract those.
|
||||
if (opts.slugs !== undefined) {
|
||||
if (opts.slugs.length === 0) {
|
||||
// Nothing changed — skip entirely.
|
||||
return result;
|
||||
}
|
||||
const r = await extractForSlugs(engine, opts.dir, opts.slugs, opts.mode, dryRun, jsonMode);
|
||||
result.links_created = r.links_created;
|
||||
result.timeline_entries_created = r.timeline_created;
|
||||
result.pages_processed = r.pages;
|
||||
return result;
|
||||
}
|
||||
|
||||
// Full walk path: CLI `gbrain extract` or first-run.
|
||||
if (opts.mode === 'links' || opts.mode === 'all') {
|
||||
const r = await extractLinksFromDir(engine, opts.dir, dryRun, jsonMode);
|
||||
result.links_created = r.created;
|
||||
@@ -433,118 +411,6 @@ export async function runExtract(engine: BrainEngine, args: string[]) {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Incremental extract: process only the specified slugs.
|
||||
*
|
||||
* Instead of walking 54K+ files, reads only the files that sync says changed.
|
||||
* Still needs the full slug set for link resolution (resolveSlug needs to know
|
||||
* all valid targets), but that's a single readdir, not 54K readFileSync calls.
|
||||
*
|
||||
* Combines links + timeline extraction in a single pass over each file —
|
||||
* the full-walk path reads every file TWICE (once for links, once for timeline).
|
||||
*/
|
||||
async function extractForSlugs(
|
||||
engine: BrainEngine,
|
||||
brainDir: string,
|
||||
slugs: string[],
|
||||
mode: 'links' | 'timeline' | 'all',
|
||||
dryRun: boolean,
|
||||
jsonMode: boolean,
|
||||
): Promise<{ links_created: number; timeline_created: number; pages: number }> {
|
||||
// Build the full slug set for link resolution (fast: just readdir, no file reads)
|
||||
const allFiles = walkMarkdownFiles(brainDir);
|
||||
const allSlugs = new Set(allFiles.map(f => f.relPath.replace('.md', '')));
|
||||
|
||||
const doLinks = mode === 'links' || mode === 'all';
|
||||
const doTimeline = mode === 'timeline' || mode === 'all';
|
||||
|
||||
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
|
||||
progress.start('extract.incremental', slugs.length);
|
||||
|
||||
let linksCreated = 0;
|
||||
let timelineCreated = 0;
|
||||
let pagesProcessed = 0;
|
||||
|
||||
const linkBatch: LinkBatchInput[] = [];
|
||||
const timelineBatch: TimelineBatchInput[] = [];
|
||||
|
||||
async function flushLinks() {
|
||||
if (linkBatch.length === 0) return;
|
||||
try {
|
||||
linksCreated += await engine.addLinksBatch(linkBatch);
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
if (!jsonMode) console.error(` link batch error (${linkBatch.length} rows lost): ${msg}`);
|
||||
} finally {
|
||||
linkBatch.length = 0;
|
||||
}
|
||||
}
|
||||
|
||||
async function flushTimeline() {
|
||||
if (timelineBatch.length === 0) return;
|
||||
try {
|
||||
timelineCreated += await engine.addTimelineEntriesBatch(timelineBatch);
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
if (!jsonMode) console.error(` timeline batch error (${timelineBatch.length} rows lost): ${msg}`);
|
||||
} finally {
|
||||
timelineBatch.length = 0;
|
||||
}
|
||||
}
|
||||
|
||||
for (const slug of slugs) {
|
||||
const relPath = slug + '.md';
|
||||
const fullPath = join(brainDir, relPath);
|
||||
|
||||
try {
|
||||
if (!existsSync(fullPath)) continue; // deleted file — sync already handled removal
|
||||
const content = readFileSync(fullPath, 'utf-8');
|
||||
|
||||
// Links
|
||||
if (doLinks) {
|
||||
const links = await extractLinksFromFile(content, relPath, allSlugs);
|
||||
for (const link of links) {
|
||||
if (dryRun) {
|
||||
if (!jsonMode) console.log(` ${link.from_slug} → ${link.to_slug} (${link.link_type})`);
|
||||
linksCreated++;
|
||||
} else {
|
||||
linkBatch.push(link);
|
||||
if (linkBatch.length >= BATCH_SIZE) await flushLinks();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Timeline
|
||||
if (doTimeline) {
|
||||
const entries = extractTimelineFromContent(content, slug);
|
||||
for (const entry of entries) {
|
||||
if (dryRun) {
|
||||
if (!jsonMode) console.log(` ${entry.slug}: ${entry.date} — ${entry.summary}`);
|
||||
timelineCreated++;
|
||||
} else {
|
||||
timelineBatch.push({ slug: entry.slug, date: entry.date, source: entry.source, summary: entry.summary, detail: entry.detail });
|
||||
if (timelineBatch.length >= BATCH_SIZE) await flushTimeline();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pagesProcessed++;
|
||||
} catch { /* skip unreadable */ }
|
||||
progress.tick(1);
|
||||
}
|
||||
|
||||
await flushLinks();
|
||||
await flushTimeline();
|
||||
progress.finish();
|
||||
|
||||
if (!jsonMode) {
|
||||
const label = dryRun ? '(dry run) would create' : 'created';
|
||||
console.log(`Incremental extract: ${label} ${linksCreated} link(s), ${timelineCreated} timeline entries from ${pagesProcessed}/${slugs.length} page(s)`);
|
||||
}
|
||||
|
||||
return { links_created: linksCreated, timeline_created: timelineCreated, pages: pagesProcessed };
|
||||
}
|
||||
|
||||
async function extractLinksFromDir(
|
||||
engine: BrainEngine, brainDir: string, dryRun: boolean, jsonMode: boolean,
|
||||
): Promise<{ created: number; pages: number }> {
|
||||
|
||||
@@ -1,216 +0,0 @@
|
||||
/**
|
||||
* gbrain frontmatter install-hook — Install a pre-commit hook in a brain
|
||||
* source's git repo that runs `gbrain frontmatter validate` against staged
|
||||
* .md/.mdx files. Skips non-git sources with a one-line note.
|
||||
*
|
||||
* Usage:
|
||||
* gbrain frontmatter install-hook [--source <id>] [--force] [--uninstall]
|
||||
*
|
||||
* --source <id> Limit to one registered source. Default: all sources.
|
||||
* --force Overwrite an existing pre-commit hook (writes <hook>.bak).
|
||||
* --uninstall Remove the hook; restore <hook>.bak if present.
|
||||
*
|
||||
* Hook contract:
|
||||
* - Located at <source>/.githooks/pre-commit. We `git config core.hooksPath
|
||||
* .githooks` if no other hooksPath is set.
|
||||
* - When the gbrain binary is missing, the hook prints a one-line warning
|
||||
* and exits 0 (don't break commits if a developer uninstalls gbrain).
|
||||
* - Bypass via `git commit --no-verify`.
|
||||
*/
|
||||
|
||||
import { existsSync, readFileSync, writeFileSync, mkdirSync, chmodSync, rmSync, copyFileSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { execFileSync } from 'child_process';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { loadConfig, toEngineConfig } from '../core/config.ts';
|
||||
import { createEngine } from '../core/engine-factory.ts';
|
||||
|
||||
const HOOK_BANNER = '# gbrain frontmatter pre-commit hook (v0.22.4+)';
|
||||
|
||||
const HOOK_SCRIPT = `#!/bin/sh
|
||||
${HOOK_BANNER}
|
||||
# Validates YAML frontmatter on staged .md / .mdx files. Bypass with
|
||||
# 'git commit --no-verify'. Uninstall with 'gbrain frontmatter install-hook --uninstall'.
|
||||
|
||||
set -e
|
||||
|
||||
if ! command -v gbrain >/dev/null 2>&1; then
|
||||
echo "gbrain not on PATH; skipping frontmatter pre-commit (install gbrain to re-enable)." >&2
|
||||
exit 0
|
||||
fi
|
||||
|
||||
staged=$(git diff --cached --name-only --diff-filter=ACM | grep -E '\\\\.mdx?$' || true)
|
||||
[ -z "$staged" ] && exit 0
|
||||
|
||||
failed=0
|
||||
for f in $staged; do
|
||||
[ -f "$f" ] || continue
|
||||
if ! gbrain frontmatter validate "$f" >/dev/null 2>&1; then
|
||||
gbrain frontmatter validate "$f" >&2
|
||||
failed=1
|
||||
fi
|
||||
done
|
||||
|
||||
if [ $failed -ne 0 ]; then
|
||||
echo "" >&2
|
||||
echo "Frontmatter validation failed. Run 'gbrain frontmatter validate <file> --fix' to repair, or 'git commit --no-verify' to bypass." >&2
|
||||
exit 1
|
||||
fi
|
||||
`;
|
||||
|
||||
interface SourceRow {
|
||||
id: string;
|
||||
local_path: string | null;
|
||||
}
|
||||
|
||||
export async function runFrontmatterInstallHook(args: string[]): Promise<void> {
|
||||
let force = false;
|
||||
let uninstall = false;
|
||||
let sourceId: string | undefined;
|
||||
let help = false;
|
||||
for (let i = 0; i < args.length; i++) {
|
||||
const a = args[i];
|
||||
if (a === '--help' || a === '-h') help = true;
|
||||
else if (a === '--force') force = true;
|
||||
else if (a === '--uninstall') uninstall = true;
|
||||
else if (a === '--source') sourceId = args[++i];
|
||||
else if (a.startsWith('--source=')) sourceId = a.slice('--source='.length);
|
||||
}
|
||||
|
||||
if (help) {
|
||||
printHelp();
|
||||
return;
|
||||
}
|
||||
|
||||
const config = loadConfig();
|
||||
if (!config) {
|
||||
throw new Error('No brain configured. Run: gbrain init');
|
||||
}
|
||||
const engineConfig = toEngineConfig(config);
|
||||
const engine = await createEngine(engineConfig);
|
||||
await engine.connect(engineConfig);
|
||||
try {
|
||||
const sources = await listSources(engine, sourceId);
|
||||
if (sources.length === 0) {
|
||||
console.log(sourceId
|
||||
? `Source "${sourceId}" not found.`
|
||||
: 'No registered sources. Run `gbrain sources list` to inspect.');
|
||||
return;
|
||||
}
|
||||
|
||||
let installed = 0;
|
||||
let skipped = 0;
|
||||
for (const src of sources) {
|
||||
if (!src.local_path || !existsSync(src.local_path)) {
|
||||
console.log(`[${src.id}] skipped — local_path missing on disk`);
|
||||
skipped++;
|
||||
continue;
|
||||
}
|
||||
if (!isGitRepo(src.local_path)) {
|
||||
console.log(`[${src.id}] ${src.local_path} — skipped, not a git repo`);
|
||||
skipped++;
|
||||
continue;
|
||||
}
|
||||
if (uninstall) {
|
||||
if (uninstallHook(src.local_path)) {
|
||||
console.log(`[${src.id}] hook removed`);
|
||||
installed++;
|
||||
} else {
|
||||
console.log(`[${src.id}] no gbrain pre-commit hook found; nothing to uninstall`);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
const result = installHook(src.local_path, force);
|
||||
if (result === 'installed') {
|
||||
console.log(`[${src.id}] hook installed at .githooks/pre-commit`);
|
||||
installed++;
|
||||
} else if (result === 'skipped_existing') {
|
||||
console.log(`[${src.id}] existing pre-commit hook found; pass --force to overwrite (.bak created)`);
|
||||
skipped++;
|
||||
} else {
|
||||
console.log(`[${src.id}] hook already up to date`);
|
||||
}
|
||||
}
|
||||
|
||||
console.log(`\nDone. ${installed} ${uninstall ? 'removed' : 'installed/updated'}, ${skipped} skipped.`);
|
||||
} finally {
|
||||
await engine.disconnect();
|
||||
}
|
||||
}
|
||||
|
||||
function printHelp() {
|
||||
console.log(`gbrain frontmatter install-hook — install pre-commit hook in source git repos
|
||||
|
||||
Usage:
|
||||
gbrain frontmatter install-hook [--source <id>] [--force] [--uninstall]
|
||||
|
||||
The hook runs \`gbrain frontmatter validate\` against staged .md/.mdx files,
|
||||
blocking commits with malformed frontmatter. Bypass with 'git commit --no-verify'.
|
||||
|
||||
Options:
|
||||
--source <id> Limit to one registered source. Default: all sources.
|
||||
--force Overwrite an existing pre-commit hook (writes <hook>.bak).
|
||||
--uninstall Remove the hook; restore <hook>.bak if present.
|
||||
`);
|
||||
}
|
||||
|
||||
async function listSources(engine: BrainEngine, sourceId?: string): Promise<SourceRow[]> {
|
||||
if (sourceId) {
|
||||
return engine.executeRaw<SourceRow>(`SELECT id, local_path FROM sources WHERE id = $1`, [sourceId]);
|
||||
}
|
||||
return engine.executeRaw<SourceRow>(`SELECT id, local_path FROM sources WHERE local_path IS NOT NULL ORDER BY id`);
|
||||
}
|
||||
|
||||
function isGitRepo(dir: string): boolean {
|
||||
return existsSync(join(dir, '.git'));
|
||||
}
|
||||
|
||||
type InstallResult = 'installed' | 'skipped_existing' | 'unchanged';
|
||||
|
||||
export function installHook(repoPath: string, force: boolean): InstallResult {
|
||||
const hooksDir = join(repoPath, '.githooks');
|
||||
const hookPath = join(hooksDir, 'pre-commit');
|
||||
mkdirSync(hooksDir, { recursive: true });
|
||||
|
||||
if (existsSync(hookPath)) {
|
||||
const existing = readFileSync(hookPath, 'utf8');
|
||||
if (existing.includes(HOOK_BANNER)) {
|
||||
// Already a gbrain hook — refresh the script content silently.
|
||||
writeFileSync(hookPath, HOOK_SCRIPT);
|
||||
chmodSync(hookPath, 0o755);
|
||||
return 'unchanged';
|
||||
}
|
||||
if (!force) return 'skipped_existing';
|
||||
copyFileSync(hookPath, hookPath + '.bak');
|
||||
}
|
||||
|
||||
writeFileSync(hookPath, HOOK_SCRIPT);
|
||||
chmodSync(hookPath, 0o755);
|
||||
|
||||
// Set core.hooksPath unless the user has set it to something else already.
|
||||
try {
|
||||
const current = execFileSync('git', ['-C', repoPath, 'config', '--get', 'core.hooksPath'], { encoding: 'utf8' }).trim();
|
||||
if (current && current !== '.githooks') return 'installed';
|
||||
} catch {
|
||||
// git config returns non-zero when the key is unset; that's the normal case.
|
||||
}
|
||||
try {
|
||||
execFileSync('git', ['-C', repoPath, 'config', 'core.hooksPath', '.githooks']);
|
||||
} catch {
|
||||
// Best-effort. Hook still exists; user can configure manually.
|
||||
}
|
||||
return 'installed';
|
||||
}
|
||||
|
||||
export function uninstallHook(repoPath: string): boolean {
|
||||
const hookPath = join(repoPath, '.githooks', 'pre-commit');
|
||||
if (!existsSync(hookPath)) return false;
|
||||
const content = readFileSync(hookPath, 'utf8');
|
||||
if (!content.includes(HOOK_BANNER)) return false;
|
||||
rmSync(hookPath);
|
||||
if (existsSync(hookPath + '.bak')) {
|
||||
copyFileSync(hookPath + '.bak', hookPath);
|
||||
rmSync(hookPath + '.bak');
|
||||
}
|
||||
return true;
|
||||
}
|
||||
@@ -1,299 +0,0 @@
|
||||
/**
|
||||
* gbrain frontmatter — Frontmatter validation, audit, and auto-repair.
|
||||
*
|
||||
* Subcommands:
|
||||
* gbrain frontmatter validate <path> [--json] [--fix] [--dry-run]
|
||||
* Validate one file or recursively a directory. --fix writes .bak then
|
||||
* rewrites in place. --dry-run previews without writing.
|
||||
*
|
||||
* gbrain frontmatter audit [--source <id>] [--json]
|
||||
* Read-only scan across all registered sources (or one with --source).
|
||||
* Returns AuditReport-shaped JSON with --json.
|
||||
*
|
||||
* The audit subcommand is intentionally read-only; --fix only exists on
|
||||
* validate. Pass an explicit path to validate a non-source-registered tree.
|
||||
*/
|
||||
|
||||
import { readFileSync, writeFileSync, existsSync, lstatSync, readdirSync, copyFileSync } from 'fs';
|
||||
import { join, relative, resolve } from 'path';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { loadConfig, toEngineConfig } from '../core/config.ts';
|
||||
import { createEngine } from '../core/engine-factory.ts';
|
||||
import { parseMarkdown, type ParseValidationCode } from '../core/markdown.ts';
|
||||
import {
|
||||
autoFixFrontmatter,
|
||||
scanBrainSources,
|
||||
type AuditReport,
|
||||
type AuditFix,
|
||||
} from '../core/brain-writer.ts';
|
||||
import { isSyncable, slugifyPath } from '../core/sync.ts';
|
||||
|
||||
export async function runFrontmatter(args: string[]): Promise<void> {
|
||||
const sub = args[0];
|
||||
if (!sub || sub === '--help' || sub === '-h') {
|
||||
printHelp();
|
||||
return;
|
||||
}
|
||||
const rest = args.slice(1);
|
||||
|
||||
if (sub === 'validate') {
|
||||
await runValidate(rest);
|
||||
return;
|
||||
}
|
||||
if (sub === 'audit') {
|
||||
const engine = await connectEngineForAudit();
|
||||
try {
|
||||
await runAudit(engine, rest);
|
||||
} finally {
|
||||
await engine.disconnect();
|
||||
}
|
||||
return;
|
||||
}
|
||||
if (sub === 'install-hook') {
|
||||
const { runFrontmatterInstallHook } = await import('./frontmatter-install-hook.ts');
|
||||
await runFrontmatterInstallHook(rest);
|
||||
return;
|
||||
}
|
||||
console.error(`Unknown frontmatter subcommand: ${sub}\n`);
|
||||
printHelp();
|
||||
process.exitCode = 1;
|
||||
}
|
||||
|
||||
async function connectEngineForAudit(): Promise<BrainEngine> {
|
||||
const config = loadConfig();
|
||||
if (!config) {
|
||||
throw new Error('No brain configured. Run: gbrain init');
|
||||
}
|
||||
const engineConfig = toEngineConfig(config);
|
||||
const engine = await createEngine(engineConfig);
|
||||
await engine.connect(engineConfig);
|
||||
return engine;
|
||||
}
|
||||
|
||||
function printHelp() {
|
||||
console.log(`gbrain frontmatter — frontmatter validation, audit, and auto-repair
|
||||
|
||||
Usage:
|
||||
gbrain frontmatter validate <path> [--json] [--fix] [--dry-run]
|
||||
gbrain frontmatter audit [--source <id>] [--json]
|
||||
gbrain frontmatter install-hook [--source <id>] [--force] [--uninstall]
|
||||
|
||||
validate
|
||||
Validate one .md file or recursively a directory. Each file is parsed via
|
||||
parseMarkdown(..., {validate:true}); errors are reported by code:
|
||||
MISSING_OPEN, MISSING_CLOSE, YAML_PARSE, SLUG_MISMATCH,
|
||||
NULL_BYTES, NESTED_QUOTES, EMPTY_FRONTMATTER
|
||||
|
||||
--fix Auto-repair the fixable subset (NULL_BYTES, MISSING_CLOSE,
|
||||
NESTED_QUOTES, SLUG_MISMATCH). Writes <file>.bak before any
|
||||
in-place rewrite. .bak is the safety contract; works for both
|
||||
git and non-git brain repos.
|
||||
--dry-run Preview --fix without writing.
|
||||
--json Emit a JSON envelope on stdout.
|
||||
|
||||
audit
|
||||
Read-only scan across all registered sources (or one with --source <id>).
|
||||
Reports per-source counts grouped by error code. Use this in CI or doctor
|
||||
pipelines. Exits 0 even when issues are found — the count is the signal.
|
||||
|
||||
--source <id> Limit scan to one registered source.
|
||||
--json Emit AuditReport-shaped JSON on stdout.
|
||||
`);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// validate
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
interface ValidateFlags {
|
||||
json: boolean;
|
||||
fix: boolean;
|
||||
dryRun: boolean;
|
||||
}
|
||||
|
||||
interface FileValidation {
|
||||
path: string;
|
||||
errors: { code: ParseValidationCode; message: string; line?: number }[];
|
||||
fixesApplied?: AuditFix[];
|
||||
}
|
||||
|
||||
async function runValidate(rest: string[]): Promise<void> {
|
||||
const flags: ValidateFlags = { json: false, fix: false, dryRun: false };
|
||||
let target: string | null = null;
|
||||
for (const a of rest) {
|
||||
if (a === '--json') flags.json = true;
|
||||
else if (a === '--fix') flags.fix = true;
|
||||
else if (a === '--dry-run') flags.dryRun = true;
|
||||
else if (!a.startsWith('--')) target = a;
|
||||
}
|
||||
if (!target) {
|
||||
console.error('error: gbrain frontmatter validate requires a <path> argument');
|
||||
process.exitCode = 1;
|
||||
return;
|
||||
}
|
||||
|
||||
const resolved = resolve(target);
|
||||
if (!existsSync(resolved)) {
|
||||
console.error(`error: path not found: ${target}`);
|
||||
process.exitCode = 1;
|
||||
return;
|
||||
}
|
||||
|
||||
const files = collectFiles(resolved);
|
||||
const results: FileValidation[] = [];
|
||||
|
||||
for (const file of files) {
|
||||
const content = readFileSync(file, 'utf8');
|
||||
const expectedSlug = slugifyPath(relative(resolve(target), file) || file);
|
||||
const parsed = parseMarkdown(content, file, { validate: true, expectedSlug });
|
||||
const errs = parsed.errors ?? [];
|
||||
const result: FileValidation = {
|
||||
path: file,
|
||||
errors: errs.map(e => ({ code: e.code, message: e.message, line: e.line })),
|
||||
};
|
||||
|
||||
if (flags.fix && errs.length > 0) {
|
||||
const { content: fixed, fixes } = autoFixFrontmatter(content, { filePath: file });
|
||||
result.fixesApplied = fixes;
|
||||
if (fixes.length > 0 && !flags.dryRun) {
|
||||
copyFileSync(file, file + '.bak');
|
||||
writeFileSync(file, fixed, 'utf8');
|
||||
}
|
||||
}
|
||||
|
||||
results.push(result);
|
||||
}
|
||||
|
||||
const totalErrors = results.reduce((n, r) => n + r.errors.length, 0);
|
||||
const filesWithErrors = results.filter(r => r.errors.length > 0).length;
|
||||
const filesFixed = results.filter(r => (r.fixesApplied?.length ?? 0) > 0).length;
|
||||
|
||||
if (flags.json) {
|
||||
const envelope = {
|
||||
ok: totalErrors === 0,
|
||||
target: resolved,
|
||||
total_files: files.length,
|
||||
files_with_errors: filesWithErrors,
|
||||
total_errors: totalErrors,
|
||||
files_fixed: flags.fix ? filesFixed : undefined,
|
||||
dry_run: flags.dryRun || undefined,
|
||||
results,
|
||||
};
|
||||
console.log(JSON.stringify(envelope, null, 2));
|
||||
} else {
|
||||
if (totalErrors === 0) {
|
||||
console.log(`OK — ${files.length} file(s) scanned, no frontmatter issues`);
|
||||
} else {
|
||||
console.log(`Found ${totalErrors} issue(s) across ${filesWithErrors} file(s) (scanned ${files.length})`);
|
||||
for (const r of results) {
|
||||
if (r.errors.length === 0) continue;
|
||||
console.log(`\n${r.path}`);
|
||||
for (const e of r.errors) {
|
||||
const lineHint = e.line !== undefined ? `:${e.line}` : '';
|
||||
console.log(` [${e.code}]${lineHint} ${e.message}`);
|
||||
}
|
||||
if (r.fixesApplied && r.fixesApplied.length > 0) {
|
||||
const verb = flags.dryRun ? 'would fix' : 'fixed';
|
||||
for (const f of r.fixesApplied) {
|
||||
console.log(` ${verb}: ${f.description}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
if (flags.fix && !flags.dryRun) {
|
||||
console.log(`\nWrote .bak backups for ${filesFixed} file(s).`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
process.exitCode = totalErrors > 0 && !flags.fix ? 1 : 0;
|
||||
}
|
||||
|
||||
function collectFiles(target: string): string[] {
|
||||
const st = lstatSync(target);
|
||||
if (st.isFile()) {
|
||||
return [target];
|
||||
}
|
||||
const out: string[] = [];
|
||||
const stack = [target];
|
||||
while (stack.length > 0) {
|
||||
const dir = stack.pop()!;
|
||||
let entries: string[];
|
||||
try {
|
||||
entries = readdirSync(dir);
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
for (const name of entries) {
|
||||
const full = join(dir, name);
|
||||
let entryStat: ReturnType<typeof lstatSync>;
|
||||
try {
|
||||
entryStat = lstatSync(full);
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
if (entryStat.isSymbolicLink()) continue;
|
||||
if (entryStat.isDirectory()) {
|
||||
stack.push(full);
|
||||
} else if (entryStat.isFile()) {
|
||||
const rel = relative(target, full);
|
||||
if (isSyncable(rel, { strategy: 'markdown' })) {
|
||||
out.push(full);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// audit
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
async function runAudit(engine: BrainEngine, rest: string[]): Promise<void> {
|
||||
let json = false;
|
||||
let sourceId: string | undefined;
|
||||
for (let i = 0; i < rest.length; i++) {
|
||||
const a = rest[i];
|
||||
if (a === '--json') json = true;
|
||||
else if (a === '--source') sourceId = rest[++i];
|
||||
else if (a.startsWith('--source=')) sourceId = a.slice('--source='.length);
|
||||
}
|
||||
|
||||
const report = await scanBrainSources(engine, { sourceId });
|
||||
|
||||
if (json) {
|
||||
console.log(JSON.stringify(report, null, 2));
|
||||
return;
|
||||
}
|
||||
|
||||
printAuditHumanReport(report);
|
||||
}
|
||||
|
||||
function printAuditHumanReport(report: AuditReport): void {
|
||||
if (report.per_source.length === 0) {
|
||||
console.log('No registered sources to audit. Run `gbrain sources list` to inspect.');
|
||||
return;
|
||||
}
|
||||
console.log(`Frontmatter audit — ${report.total} issue(s) across ${report.per_source.length} source(s) (scanned at ${report.scanned_at})`);
|
||||
for (const src of report.per_source) {
|
||||
console.log(`\n[${src.source_id}] ${src.source_path}`);
|
||||
if (src.total === 0) {
|
||||
console.log(' clean');
|
||||
continue;
|
||||
}
|
||||
console.log(` ${src.total} issue(s)`);
|
||||
for (const [code, n] of Object.entries(src.errors_by_code)) {
|
||||
console.log(` ${code}: ${n}`);
|
||||
}
|
||||
if (src.sample.length > 0) {
|
||||
console.log(` sample:`);
|
||||
for (const s of src.sample.slice(0, 5)) {
|
||||
console.log(` ${s.path} — ${s.codes.join(', ')}`);
|
||||
}
|
||||
if (src.sample.length > 5) console.log(` (+ ${src.sample.length - 5} more)`);
|
||||
}
|
||||
}
|
||||
if (report.total > 0) {
|
||||
console.log(`\nFix with: gbrain frontmatter validate <source-path> --fix`);
|
||||
}
|
||||
}
|
||||
+28
-55
@@ -34,17 +34,7 @@ export async function runImport(engine: BrainEngine, args: string[], opts: { com
|
||||
const jsonOutput = args.includes('--json');
|
||||
const workersIdx = args.indexOf('--workers');
|
||||
const workersArg = workersIdx !== -1 ? args[workersIdx + 1] : null;
|
||||
// v0.22.13 (PR #490 Q2): shared parseWorkers helper rejects bad input
|
||||
// (--workers 0, -3, "foo") with a loud error instead of silently falling
|
||||
// through to 1. Mirrors sync.ts's flag handling.
|
||||
const { parseWorkers } = await import('../core/sync-concurrency.ts');
|
||||
let workerCount: number;
|
||||
try {
|
||||
workerCount = parseWorkers(workersArg ?? undefined) ?? 1;
|
||||
} catch (e) {
|
||||
console.error(e instanceof Error ? e.message : String(e));
|
||||
process.exit(1);
|
||||
}
|
||||
const workerCount = workersArg ? parseInt(workersArg, 10) : 1;
|
||||
// Find dir: first non-flag arg that isn't a value for --workers
|
||||
const flagValues = new Set<number>();
|
||||
if (workersIdx !== -1) flagValues.add(workersIdx + 1);
|
||||
@@ -151,57 +141,40 @@ export async function runImport(engine: BrainEngine, args: string[], opts: { com
|
||||
}
|
||||
|
||||
if (actualWorkers > 1) {
|
||||
// v0.22.13 (PR #490 A1 + Q3): use engine.kind discriminator (not config.engine
|
||||
// string sniff) and fall back to serial when database_url is unset. Both
|
||||
// checks belt-and-suspenders so we never crash on a null assertion.
|
||||
// Parallel: create per-worker engine instances with small pool
|
||||
// PGLite is single-connection, so parallel workers are only for Postgres
|
||||
const config = loadConfig();
|
||||
if (engine.kind === 'pglite' || !config?.database_url) {
|
||||
if (config?.engine === 'pglite') {
|
||||
// PGLite: sequential import through single engine
|
||||
for (const file of files) {
|
||||
await processFile(engine, file);
|
||||
}
|
||||
} else {
|
||||
const { PostgresEngine } = await import('../core/postgres-engine.ts');
|
||||
const { resolvePoolSize } = await import('../core/db.ts');
|
||||
// Default per-worker pool is 2 (small, parallel import case). Users on
|
||||
// constrained poolers (e.g. Supabase port 6543) can cap below this via
|
||||
// GBRAIN_POOL_SIZE=1.
|
||||
const workerPoolSize = Math.min(2, resolvePoolSize(2));
|
||||
const databaseUrl = config.database_url;
|
||||
const { PostgresEngine } = await import('../core/postgres-engine.ts');
|
||||
const { resolvePoolSize } = await import('../core/db.ts');
|
||||
// Default per-worker pool is 2 (small, parallel import case). Users on
|
||||
// constrained poolers (e.g. Supabase port 6543) can cap below this via
|
||||
// GBRAIN_POOL_SIZE=1.
|
||||
const workerPoolSize = Math.min(2, resolvePoolSize(2));
|
||||
const workerEngines = await Promise.all(
|
||||
Array.from({ length: actualWorkers }, async () => {
|
||||
const eng = new PostgresEngine();
|
||||
await eng.connect({ database_url: config!.database_url!, poolSize: workerPoolSize });
|
||||
return eng;
|
||||
})
|
||||
);
|
||||
|
||||
// v0.22.13 (PR #490 A2): connect workers serially so a partial failure
|
||||
// leaves us with the connected ones already pushed onto workerEngines
|
||||
// for the finally-block cleanup. The prior Promise.all could leak any
|
||||
// engine that connected before another's connect() rejected.
|
||||
const workerEngines: InstanceType<typeof PostgresEngine>[] = [];
|
||||
try {
|
||||
for (let i = 0; i < actualWorkers; i++) {
|
||||
const eng = new PostgresEngine();
|
||||
await eng.connect({ database_url: databaseUrl, poolSize: workerPoolSize });
|
||||
workerEngines.push(eng);
|
||||
}
|
||||
|
||||
// Thread-safe queue: atomic index counter (JS is single-threaded; the
|
||||
// read-then-increment happens between awaits so no lock is needed).
|
||||
let queueIndex = 0;
|
||||
await Promise.all(workerEngines.map(async (eng) => {
|
||||
while (true) {
|
||||
const idx = queueIndex++;
|
||||
if (idx >= files.length) break;
|
||||
await processFile(eng, files[idx]);
|
||||
}
|
||||
}));
|
||||
} finally {
|
||||
// v0.22.13 (PR #490 A2): try/finally guarantees cleanup even when the
|
||||
// worker loop throws. Each disconnect is best-effort — one failing
|
||||
// disconnect must not strand the others.
|
||||
await Promise.all(
|
||||
workerEngines.map(e =>
|
||||
e.disconnect().catch((err: unknown) =>
|
||||
console.error(` worker disconnect failed: ${err instanceof Error ? err.message : String(err)}`),
|
||||
),
|
||||
),
|
||||
);
|
||||
// Thread-safe queue: use an atomic index counter instead of array.shift()
|
||||
let queueIndex = 0;
|
||||
await Promise.all(workerEngines.map(async (eng) => {
|
||||
while (true) {
|
||||
const idx = queueIndex++;
|
||||
if (idx >= files.length) break;
|
||||
await processFile(eng, files[idx]);
|
||||
}
|
||||
}));
|
||||
|
||||
await Promise.all(workerEngines.map(e => e.disconnect()));
|
||||
} // end else (postgres parallel)
|
||||
} else {
|
||||
// Sequential: use the provided engine
|
||||
|
||||
@@ -31,7 +31,6 @@ import { join, dirname } from 'path';
|
||||
import { loadConfig, toEngineConfig } from '../core/config.ts';
|
||||
import { createEngine } from '../core/engine-factory.ts';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import * as db from '../core/db.ts';
|
||||
import { BrainWriter } from '../core/output/writer.ts';
|
||||
import {
|
||||
getDefaultRegistry,
|
||||
@@ -267,12 +266,6 @@ export interface IntegrityScanOptions {
|
||||
limit?: number;
|
||||
/** Slug prefix filter (e.g. "people") — matches slugs starting with `${typeFilter}/`. */
|
||||
typeFilter?: string;
|
||||
/**
|
||||
* When true (default), batch-load pages via a single SQL query instead of
|
||||
* sequential getPage() calls. Falls back to sequential on error (e.g. PGLite).
|
||||
* Eliminates 500 round-trips through PgBouncer that caused doctor timeouts.
|
||||
*/
|
||||
batchLoad?: boolean;
|
||||
}
|
||||
|
||||
export interface IntegrityScanResult {
|
||||
@@ -294,29 +287,7 @@ export async function scanIntegrity(
|
||||
engine: BrainEngine,
|
||||
opts: IntegrityScanOptions = {},
|
||||
): Promise<IntegrityScanResult> {
|
||||
const { limit = Infinity, typeFilter, batchLoad = true } = opts;
|
||||
|
||||
// Fast path: single SQL query instead of N sequential getPage() calls.
|
||||
// Eliminates ~500 round-trips through PgBouncer that caused doctor to
|
||||
// timeout on transaction-mode pooling. Postgres-only: PGLite has no
|
||||
// postgres.js connection, so the gate keeps the GBRAIN_DEBUG fallback
|
||||
// log clean for real Postgres errors instead of expected PGLite skips.
|
||||
if (batchLoad && limit !== Infinity && engine.kind === 'postgres') {
|
||||
try {
|
||||
return await scanIntegrityBatch(limit, typeFilter);
|
||||
} catch (err) {
|
||||
// GBRAIN_DEBUG=1 surfaces real Postgres errors (deadlock, connection
|
||||
// drop, SQL bug) that would otherwise vanish into the sequential
|
||||
// fallback. Quiet by default since the fallback is harmless.
|
||||
if (process.env.GBRAIN_DEBUG) {
|
||||
console.error(
|
||||
'[integrity] batch path failed, falling back to sequential:',
|
||||
err instanceof Error ? err.message : err,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const { limit = Infinity, typeFilter } = opts;
|
||||
const allSlugs = [...(await engine.getAllSlugs())].sort();
|
||||
|
||||
const bareHits: BareTweetHit[] = [];
|
||||
@@ -345,52 +316,6 @@ export async function scanIntegrity(
|
||||
return { pagesScanned, bareHits, externalHits, topPages };
|
||||
}
|
||||
|
||||
/**
|
||||
* Batch-load integrity scan: fetches all candidate pages in a single SQL
|
||||
* query, then scans in-memory. Reduces PgBouncer round-trips from ~500 to 1.
|
||||
*/
|
||||
async function scanIntegrityBatch(
|
||||
limit: number,
|
||||
typeFilter?: string,
|
||||
): Promise<IntegrityScanResult> {
|
||||
const sql = db.getConnection();
|
||||
const typeCondition = typeFilter ? sql`AND slug LIKE ${typeFilter + '/%'}` : sql``;
|
||||
// Boolean validate is the documented contract; stringly-typed 'false' (quoted
|
||||
// YAML) diverges from the sequential path's strict === false check. Intentional
|
||||
// — gbrain lint should reject stringly-typed validate at write time.
|
||||
const validateCondition = sql`AND (frontmatter->>'validate' IS NULL OR frontmatter->>'validate' != 'false')`;
|
||||
|
||||
// DISTINCT ON (slug) mirrors getAllSlugs()'s Set<string> semantics: multi-source
|
||||
// brains can have the same slug under multiple source_ids (UNIQUE(source_id, slug)
|
||||
// since v0.18.0); we want one scan per slug, not one per row.
|
||||
const rows = await sql`
|
||||
SELECT DISTINCT ON (slug) slug, compiled_truth, frontmatter
|
||||
FROM pages
|
||||
WHERE 1=1 ${typeCondition} ${validateCondition}
|
||||
ORDER BY slug
|
||||
LIMIT ${limit}
|
||||
`;
|
||||
|
||||
const bareHits: BareTweetHit[] = [];
|
||||
const externalHits: ExternalLinkHit[] = [];
|
||||
|
||||
for (const row of rows) {
|
||||
const slug = row.slug as string;
|
||||
const compiledTruth = row.compiled_truth as string;
|
||||
bareHits.push(...findBareTweetHits(compiledTruth, slug));
|
||||
externalHits.push(...findExternalLinks(compiledTruth, slug));
|
||||
}
|
||||
|
||||
const byPage = new Map<string, number>();
|
||||
for (const h of bareHits) byPage.set(h.slug, (byPage.get(h.slug) ?? 0) + 1);
|
||||
const topPages = [...byPage.entries()]
|
||||
.sort((a, b) => b[1] - a[1])
|
||||
.slice(0, 10)
|
||||
.map(([slug, count]) => ({ slug, count }));
|
||||
|
||||
return { pagesScanned: rows.length, bareHits, externalHits, topPages };
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// auto — three-bucket repair
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
+4
-84
@@ -32,32 +32,6 @@ export function parseMaxWaitingFlag(args: string[]): number | undefined {
|
||||
return Math.max(1, Math.min(100, parsed));
|
||||
}
|
||||
|
||||
/** Parse `--max-rss N` (MB). Returns:
|
||||
* - 0 if the flag is absent (no watchdog by default for bare `jobs work`)
|
||||
* - 0 if `--max-rss 0` (explicit disable)
|
||||
* - the value if >= 256
|
||||
* Errors and exits the process if the flag is non-numeric, negative, or
|
||||
* positive but < 256 (likely a GB-vs-MB unit-confusion typo). */
|
||||
export function parseMaxRssFlag(args: string[]): number {
|
||||
const raw = parseFlag(args, '--max-rss');
|
||||
if (raw === undefined) return 0;
|
||||
const parsed = parseInt(raw, 10);
|
||||
if (!Number.isFinite(parsed) || parsed < 0) {
|
||||
console.error(`Error: --max-rss must be a non-negative integer (MB), got "${raw}"`);
|
||||
process.exit(1);
|
||||
}
|
||||
if (parsed === 0) return 0;
|
||||
if (parsed < 256) {
|
||||
console.error(
|
||||
`Error: --max-rss ${parsed} is too low for production (likely a unit confusion: ` +
|
||||
`--max-rss takes megabytes, not gigabytes). Use --max-rss 0 to disable, ` +
|
||||
`or set a value >= 256.`
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
return parsed;
|
||||
}
|
||||
|
||||
export function resolveWorkerConcurrency(args: string[], env: NodeJS.ProcessEnv = process.env): number {
|
||||
const raw = parseFlag(args, '--concurrency') ?? env.GBRAIN_WORKER_CONCURRENCY ?? '1';
|
||||
const parsed = parseInt(raw, 10);
|
||||
@@ -132,12 +106,11 @@ USAGE
|
||||
gbrain jobs delete <id>
|
||||
gbrain jobs stats
|
||||
gbrain jobs smoke
|
||||
gbrain jobs work [--queue Q] [--concurrency N] [--max-rss MB]
|
||||
gbrain jobs work [--queue Q] [--concurrency N]
|
||||
gbrain jobs supervisor [start] [--detach] [--json]
|
||||
[--concurrency N] [--queue Q] [--pid-file PATH]
|
||||
[--max-crashes N] [--health-interval N]
|
||||
[--allow-shell-jobs] [--cli-path PATH]
|
||||
[--max-rss MB]
|
||||
gbrain jobs supervisor status [--json] [--pid-file PATH]
|
||||
gbrain jobs supervisor stop [--json] [--pid-file PATH]
|
||||
|
||||
@@ -638,19 +611,14 @@ HANDLER TYPES (built in)
|
||||
|
||||
const queueName = parseFlag(args, '--queue') ?? 'default';
|
||||
const concurrency = resolveWorkerConcurrency(args);
|
||||
// --max-rss is opt-in for bare `gbrain jobs work` — preserves pre-v0.21 behavior
|
||||
// for operators with legitimately large embed/import working sets. The supervisor
|
||||
// path injects a default 2048; this code path does not.
|
||||
const maxRssMb = parseMaxRssFlag(args);
|
||||
|
||||
try { await queue.ensureSchema(); }
|
||||
catch (e) { console.error(e instanceof Error ? e.message : String(e)); process.exit(1); }
|
||||
|
||||
const worker = new MinionWorker(engine, { queue: queueName, concurrency, maxRssMb });
|
||||
const worker = new MinionWorker(engine, { queue: queueName, concurrency });
|
||||
await registerBuiltinHandlers(worker, engine);
|
||||
|
||||
const watchdogNote = maxRssMb > 0 ? `, watchdog: ${maxRssMb}MB` : '';
|
||||
console.log(`Minion worker started (queue: ${queueName}, concurrency: ${concurrency}${watchdogNote})`);
|
||||
console.log(`Minion worker started (queue: ${queueName}, concurrency: ${concurrency})`);
|
||||
console.log(`Registered handlers: ${worker.registeredNames.join(', ')}`);
|
||||
await worker.start();
|
||||
break;
|
||||
@@ -791,11 +759,6 @@ HANDLER TYPES (built in)
|
||||
const allowShellJobs = hasFlag(args, '--allow-shell-jobs') ||
|
||||
!!process.env.GBRAIN_ALLOW_SHELL_JOBS;
|
||||
const detach = hasFlag(args, '--detach');
|
||||
// Supervisor defaults --max-rss 2048 (MB) — main production path uses
|
||||
// the supervisor, so the watchdog is on by default here. parseMaxRssFlag
|
||||
// returns 0 when the flag is absent; substitute the supervisor default.
|
||||
const maxRssRaw = parseMaxRssFlag(args);
|
||||
const maxRssMb = parseFlag(args, '--max-rss') === undefined ? 2048 : maxRssRaw;
|
||||
|
||||
const cliPath = parseFlag(args, '--cli-path') ?? resolveGbrainCliPath();
|
||||
|
||||
@@ -833,7 +796,6 @@ HANDLER TYPES (built in)
|
||||
cliPath,
|
||||
allowShellJobs,
|
||||
json: jsonMode,
|
||||
maxRssMb,
|
||||
onEvent: (emission) => writeSupervisorEvent(emission, supervisorPid),
|
||||
});
|
||||
|
||||
@@ -864,40 +826,8 @@ export async function registerBuiltinHandlers(worker: MinionWorker, engine: Brai
|
||||
const { performSync } = await import('./sync.ts');
|
||||
const repoPath = typeof job.data.repoPath === 'string' ? job.data.repoPath : undefined;
|
||||
const noPull = !!job.data.noPull;
|
||||
// noEmbed defaults to true (embed is a separate job — submit `embed --stale`
|
||||
// after sync, OR run via the autopilot cycle which has its own embed phase).
|
||||
// Caller can opt in by passing { noEmbed: false } in job params.
|
||||
const noEmbed = job.data.noEmbed !== false;
|
||||
// v0.22.13 (PR #490 CODEX-1): resolve sourceId from job param OR by looking
|
||||
// up the sources row for repoPath. Mirrors cycle.ts:480 — without this, a
|
||||
// multi-source brain reads the global config.sync.last_commit anchor
|
||||
// instead of sources.last_commit, which on a regularly-GC'd repo can drop
|
||||
// out of git history and trigger 30-min full reimports every cycle.
|
||||
let sourceId: string | undefined =
|
||||
typeof job.data.sourceId === 'string' ? job.data.sourceId : undefined;
|
||||
if (!sourceId && repoPath) {
|
||||
try {
|
||||
const rows = await engine.executeRaw<{ id: string }>(
|
||||
`SELECT id FROM sources WHERE local_path = $1 LIMIT 1`,
|
||||
[repoPath],
|
||||
);
|
||||
sourceId = rows[0]?.id;
|
||||
} catch {
|
||||
// sources table may not exist on very old brains — fall through to
|
||||
// global config.sync.* anchor in performSync.
|
||||
}
|
||||
}
|
||||
// v0.22.13 (PR #490 CODEX-4): route concurrency through the shared
|
||||
// autoConcurrency helper instead of hardcoded 4. PGLite engines stay
|
||||
// serial (forced 1); explicit job param wins; auto path defaults are
|
||||
// applied inside performSync against the resolved file count.
|
||||
const concurrencyOverride = typeof job.data.concurrency === 'number'
|
||||
? job.data.concurrency
|
||||
: undefined;
|
||||
const result = await performSync(engine, {
|
||||
repoPath, sourceId, noPull, noEmbed,
|
||||
concurrency: concurrencyOverride,
|
||||
});
|
||||
const result = await performSync(engine, { repoPath, noPull, noEmbed });
|
||||
return result;
|
||||
});
|
||||
|
||||
@@ -980,19 +910,9 @@ export async function registerBuiltinHandlers(worker: MinionWorker, engine: Brai
|
||||
? job.data.repoPath
|
||||
: (await engine.getConfig('sync.repo_path')) ?? '.';
|
||||
|
||||
// Allow callers to select phases via job data (e.g. skip embed for
|
||||
// fast cycles). Validates against ALL_PHASES to prevent injection.
|
||||
const { ALL_PHASES } = await import('../core/cycle.ts');
|
||||
const validPhases = new Set(ALL_PHASES);
|
||||
const requestedPhases = Array.isArray(job.data.phases)
|
||||
? (job.data.phases as string[]).filter(p => validPhases.has(p as any))
|
||||
: undefined;
|
||||
|
||||
const report = await runCycle(engine, {
|
||||
brainDir: repoPath,
|
||||
pull: true, // autopilot daemon opts into git pull
|
||||
signal: job.signal, // propagate abort so cycle bails on timeout/cancel
|
||||
...(requestedPhases && requestedPhases.length > 0 ? { phases: requestedPhases as any } : {}),
|
||||
yieldBetweenPhases: async () => {
|
||||
// Yield to the event loop so worker lock-renewal can fire.
|
||||
await new Promise<void>(r => setImmediate(r));
|
||||
|
||||
@@ -18,7 +18,6 @@
|
||||
|
||||
import { readFileSync, writeFileSync, readdirSync, statSync, lstatSync, existsSync } from 'fs';
|
||||
import { join, relative } from 'path';
|
||||
import { parseMarkdown, type ParseValidationCode } from '../core/markdown.ts';
|
||||
|
||||
export interface LintIssue {
|
||||
file: string;
|
||||
@@ -28,25 +27,6 @@ export interface LintIssue {
|
||||
fixable: boolean;
|
||||
}
|
||||
|
||||
/** Map of frontmatter validation codes to lint rule names. Stable across
|
||||
* releases — agents and CI consumers can target specific rule names. */
|
||||
const FRONTMATTER_RULE_NAMES: Record<ParseValidationCode, string> = {
|
||||
MISSING_OPEN: 'frontmatter-missing-open',
|
||||
MISSING_CLOSE: 'frontmatter-missing-close',
|
||||
YAML_PARSE: 'frontmatter-yaml-parse',
|
||||
SLUG_MISMATCH: 'frontmatter-slug-mismatch',
|
||||
NULL_BYTES: 'frontmatter-null-bytes',
|
||||
NESTED_QUOTES: 'frontmatter-nested-quotes',
|
||||
EMPTY_FRONTMATTER: 'frontmatter-empty',
|
||||
};
|
||||
|
||||
/** Codes whose lint findings are fixable by `gbrain frontmatter validate --fix`. */
|
||||
const FRONTMATTER_FIXABLE: ReadonlySet<ParseValidationCode> = new Set<ParseValidationCode>([
|
||||
'MISSING_CLOSE',
|
||||
'NULL_BYTES',
|
||||
'NESTED_QUOTES',
|
||||
]);
|
||||
|
||||
// ── LLM artifact patterns ──────────────────────────────────────────
|
||||
|
||||
const LLM_PREAMBLES = [
|
||||
@@ -64,25 +44,6 @@ export function lintContent(content: string, filePath: string): LintIssue[] {
|
||||
const issues: LintIssue[] = [];
|
||||
const lines = content.split('\n');
|
||||
|
||||
// ── Frontmatter validation (delegates to parseMarkdown(validate:true)) ──
|
||||
// This is the single source of truth for frontmatter shape rules. Each
|
||||
// ParseValidationCode maps to a stable lint rule name in
|
||||
// FRONTMATTER_RULE_NAMES. Keeps brain-page lint, doctor's
|
||||
// frontmatter_integrity subcheck, and the frontmatter CLI in lockstep.
|
||||
const parsed = parseMarkdown(content, filePath, { validate: true });
|
||||
for (const err of parsed.errors ?? []) {
|
||||
// Skip MISSING_OPEN — the legacy `no-frontmatter` rule below covers this
|
||||
// exact case with a stable rule name. Emitting both is double-reporting.
|
||||
if (err.code === 'MISSING_OPEN') continue;
|
||||
issues.push({
|
||||
file: filePath,
|
||||
line: err.line ?? 1,
|
||||
rule: FRONTMATTER_RULE_NAMES[err.code],
|
||||
message: err.message,
|
||||
fixable: FRONTMATTER_FIXABLE.has(err.code),
|
||||
});
|
||||
}
|
||||
|
||||
// Rule: LLM preamble artifacts
|
||||
for (const pattern of LLM_PREAMBLES) {
|
||||
pattern.lastIndex = 0;
|
||||
|
||||
@@ -21,7 +21,6 @@ import { v0_16_0 } from './v0_16_0.ts';
|
||||
import { v0_18_0 } from './v0_18_0.ts';
|
||||
import { v0_18_1 } from './v0_18_1.ts';
|
||||
import { v0_21_0 } from './v0_21_0.ts';
|
||||
import { v0_22_4 } from './v0_22_4.ts';
|
||||
|
||||
export const migrations: Migration[] = [
|
||||
v0_11_0,
|
||||
@@ -34,7 +33,6 @@ export const migrations: Migration[] = [
|
||||
v0_18_0,
|
||||
v0_18_1,
|
||||
v0_21_0,
|
||||
v0_22_4,
|
||||
];
|
||||
|
||||
/** Look up a migration by exact version string. */
|
||||
|
||||
@@ -1,228 +0,0 @@
|
||||
/**
|
||||
* v0.22.4 migration orchestrator — frontmatter-guard adoption.
|
||||
*
|
||||
* v0.22.4 ships a shared frontmatter validator (parseMarkdown(..., {validate:true})),
|
||||
* a doctor subcheck (frontmatter_integrity), a top-level `gbrain frontmatter`
|
||||
* CLI (validate / audit / install-hook), and a new `frontmatter-guard` skill.
|
||||
*
|
||||
* This migration is AUDIT-ONLY (per D5): it reads the user's brain pages,
|
||||
* writes a JSON report to ~/.gbrain/migrations/v0.22.4-audit.json, and emits
|
||||
* one entry per source-with-issues to ~/.gbrain/migrations/pending-host-work.jsonl.
|
||||
* It NEVER mutates brain content. The agent reads skills/migrations/v0.22.4.md
|
||||
* after upgrade and runs `gbrain frontmatter validate <source-path> --fix` with
|
||||
* explicit user consent.
|
||||
*
|
||||
* Phases (all idempotent):
|
||||
* A. Schema — no-op (no DB changes in v0.22.4).
|
||||
* B. Audit — scanBrainSources → write JSON report.
|
||||
* C. Emit-todo — append pending-host-work.jsonl entry per source with errors.
|
||||
* D. Record — runner-owned ledger write.
|
||||
*/
|
||||
|
||||
import { existsSync, mkdirSync, writeFileSync, readFileSync, appendFileSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
|
||||
import type { BrainEngine } from '../../core/engine.ts';
|
||||
import { loadConfig, toEngineConfig } from '../../core/config.ts';
|
||||
import { createEngine } from '../../core/engine-factory.ts';
|
||||
import { scanBrainSources, type AuditReport } from '../../core/brain-writer.ts';
|
||||
|
||||
/** Test-only injection point for the audit phase. When set, phaseBAudit uses
|
||||
* this engine instead of loading config + creating a fresh one. Mirrors the
|
||||
* repair-jsonb pattern. Reset to null in afterAll. */
|
||||
let testEngineOverride: BrainEngine | null = null;
|
||||
export function __setTestEngineOverride(engine: BrainEngine | null): void {
|
||||
testEngineOverride = engine;
|
||||
}
|
||||
|
||||
function gbrainDir(): string {
|
||||
return join(process.env.HOME || '', '.gbrain');
|
||||
}
|
||||
function migrationsDir(): string { return join(gbrainDir(), 'migrations'); }
|
||||
function auditReportPath(): string { return join(migrationsDir(), 'v0.22.4-audit.json'); }
|
||||
function pendingHostWorkPath(): string { return join(migrationsDir(), 'pending-host-work.jsonl'); }
|
||||
|
||||
interface PendingHostWorkEntry {
|
||||
migration: string;
|
||||
ts: string;
|
||||
skill: string;
|
||||
reason: string;
|
||||
source_id: string;
|
||||
source_path: string;
|
||||
command: string;
|
||||
}
|
||||
|
||||
// ── Phase A — Schema (no-op) ───────────────────────────────
|
||||
|
||||
function phaseASchema(opts: OrchestratorOpts): OrchestratorPhaseResult {
|
||||
if (opts.dryRun) return { name: 'schema', status: 'skipped', detail: 'dry-run' };
|
||||
return { name: 'schema', status: 'complete', detail: 'no schema changes in v0.22.4' };
|
||||
}
|
||||
|
||||
// ── Phase B — Audit ────────────────────────────────────────
|
||||
|
||||
async function phaseBAudit(opts: OrchestratorOpts): Promise<{ phase: OrchestratorPhaseResult; report: AuditReport | null }> {
|
||||
if (opts.dryRun) return { phase: { name: 'audit', status: 'skipped', detail: 'dry-run' }, report: null };
|
||||
try {
|
||||
let report: AuditReport;
|
||||
if (testEngineOverride) {
|
||||
// Test injection path: caller manages engine lifecycle.
|
||||
report = await scanBrainSources(testEngineOverride);
|
||||
} else {
|
||||
const config = loadConfig();
|
||||
if (!config) {
|
||||
// No brain configured (fresh dev install or test environment). The
|
||||
// migration audit needs a real brain to walk; treat this as a clean
|
||||
// skip rather than a failure so apply-migrations doesn't break.
|
||||
return {
|
||||
phase: { name: 'audit', status: 'skipped', detail: 'no_brain_configured' },
|
||||
report: null,
|
||||
};
|
||||
}
|
||||
const engineConfig = toEngineConfig(config);
|
||||
const engine = await createEngine(engineConfig);
|
||||
await engine.connect(engineConfig);
|
||||
try {
|
||||
report = await scanBrainSources(engine);
|
||||
} finally {
|
||||
await engine.disconnect();
|
||||
}
|
||||
}
|
||||
if (report.per_source.length === 0) {
|
||||
// No sources registered — fresh install or dev-only install. Skip
|
||||
// cleanly; the orchestrator should report success.
|
||||
return {
|
||||
phase: { name: 'audit', status: 'skipped', detail: 'no_sources_registered' },
|
||||
report,
|
||||
};
|
||||
}
|
||||
mkdirSync(migrationsDir(), { recursive: true });
|
||||
writeFileSync(auditReportPath(), JSON.stringify(report, null, 2));
|
||||
return {
|
||||
phase: {
|
||||
name: 'audit',
|
||||
status: 'complete',
|
||||
detail: `${report.total} issue(s) across ${report.per_source.length} source(s); report at ${auditReportPath()}`,
|
||||
},
|
||||
report,
|
||||
};
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
return { phase: { name: 'audit', status: 'failed', detail: msg }, report: null };
|
||||
}
|
||||
}
|
||||
|
||||
// ── Phase C — Emit pending-host-work entries ──────────────
|
||||
|
||||
function existingEntriesForVersion(version: string): Set<string> {
|
||||
const out = new Set<string>();
|
||||
const p = pendingHostWorkPath();
|
||||
if (!existsSync(p)) return out;
|
||||
try {
|
||||
const raw = readFileSync(p, 'utf8');
|
||||
for (const line of raw.split('\n')) {
|
||||
const trimmed = line.trim();
|
||||
if (!trimmed) continue;
|
||||
try {
|
||||
const obj = JSON.parse(trimmed) as PendingHostWorkEntry;
|
||||
if (obj.migration === version && obj.source_id) {
|
||||
out.add(obj.source_id);
|
||||
}
|
||||
} catch { /* skip malformed */ }
|
||||
}
|
||||
} catch { /* read error */ }
|
||||
return out;
|
||||
}
|
||||
|
||||
function phaseCEmitTodo(opts: OrchestratorOpts, report: AuditReport | null): OrchestratorPhaseResult {
|
||||
if (opts.dryRun) return { name: 'emit-todo', status: 'skipped', detail: 'dry-run' };
|
||||
if (!report) return { name: 'emit-todo', status: 'skipped', detail: 'no report' };
|
||||
|
||||
const sourcesWithIssues = report.per_source.filter(s => s.total > 0);
|
||||
if (sourcesWithIssues.length === 0) {
|
||||
return { name: 'emit-todo', status: 'complete', detail: 'no issues; nothing to queue' };
|
||||
}
|
||||
|
||||
try {
|
||||
mkdirSync(migrationsDir(), { recursive: true });
|
||||
const already = existingEntriesForVersion('0.22.4');
|
||||
let added = 0;
|
||||
for (const src of sourcesWithIssues) {
|
||||
if (already.has(src.source_id)) continue;
|
||||
const entry: PendingHostWorkEntry = {
|
||||
migration: '0.22.4',
|
||||
ts: new Date().toISOString(),
|
||||
skill: 'skills/migrations/v0.22.4.md',
|
||||
reason: `${src.total} frontmatter issue(s) in source ${src.source_id}`,
|
||||
source_id: src.source_id,
|
||||
source_path: src.source_path,
|
||||
command: `gbrain frontmatter validate ${src.source_path} --fix`,
|
||||
};
|
||||
appendFileSync(pendingHostWorkPath(), JSON.stringify(entry) + '\n');
|
||||
added++;
|
||||
}
|
||||
return {
|
||||
name: 'emit-todo',
|
||||
status: 'complete',
|
||||
detail: `appended ${added} entr${added === 1 ? 'y' : 'ies'} to ${pendingHostWorkPath()}`,
|
||||
};
|
||||
} catch (e) {
|
||||
return { name: 'emit-todo', status: 'failed', detail: e instanceof Error ? e.message : String(e) };
|
||||
}
|
||||
}
|
||||
|
||||
// ── Orchestrator ────────────────────────────────────────────
|
||||
|
||||
async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult> {
|
||||
console.log('');
|
||||
console.log('=== v0.22.4 — frontmatter-guard adoption ===');
|
||||
if (opts.dryRun) console.log(' (dry-run; no side effects)');
|
||||
console.log('');
|
||||
|
||||
const phases: OrchestratorPhaseResult[] = [];
|
||||
|
||||
phases.push(phaseASchema(opts));
|
||||
|
||||
const { phase: bPhase, report } = await phaseBAudit(opts);
|
||||
phases.push(bPhase);
|
||||
if (bPhase.status === 'failed') {
|
||||
return { version: '0.22.4', status: 'partial', phases };
|
||||
}
|
||||
|
||||
phases.push(phaseCEmitTodo(opts, report));
|
||||
|
||||
const overallStatus: 'complete' | 'partial' | 'failed' =
|
||||
phases.some(p => p.status === 'failed') ? 'partial' : 'complete';
|
||||
|
||||
return {
|
||||
version: '0.22.4',
|
||||
status: overallStatus,
|
||||
phases,
|
||||
pending_host_work: report?.per_source.filter(s => s.total > 0).length ?? 0,
|
||||
};
|
||||
}
|
||||
|
||||
export const v0_22_4: Migration = {
|
||||
version: '0.22.4',
|
||||
featurePitch: {
|
||||
headline: 'Frontmatter-guard ships — broken brain pages can\'t hide',
|
||||
description:
|
||||
'gbrain v0.22.4 adds end-to-end frontmatter validation: a `gbrain frontmatter` CLI ' +
|
||||
'(validate / audit / install-hook), a `frontmatter_integrity` doctor subcheck, a ' +
|
||||
'pre-commit hook helper, and a new frontmatter-guard skill. The migration is audit-only ' +
|
||||
'(it never mutates your brain) — it scans every registered source, writes a per-source ' +
|
||||
'report to ~/.gbrain/migrations/v0.22.4-audit.json, and queues a TODO with the exact fix ' +
|
||||
'command. Run `gbrain frontmatter validate <source-path> --fix` to repair (creates .bak ' +
|
||||
'backups). Resolves all 7 check-resolvable warnings on master; ships frontmatter-guard.',
|
||||
},
|
||||
orchestrator,
|
||||
};
|
||||
|
||||
/** Exported for unit tests. */
|
||||
export const __testing = {
|
||||
phaseASchema,
|
||||
phaseBAudit,
|
||||
phaseCEmitTodo,
|
||||
auditReportPath,
|
||||
pendingHostWorkPath,
|
||||
};
|
||||
+3
-15
@@ -1,19 +1,7 @@
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { startMcpServer } from '../mcp/server.ts';
|
||||
import { startHttpTransport } from '../mcp/http-transport.ts';
|
||||
|
||||
export async function runServe(engine: BrainEngine, args: string[] = []) {
|
||||
const useHttp = args.includes('--http');
|
||||
const portIdx = args.indexOf('--port');
|
||||
const port = portIdx >= 0 ? parseInt(args[portIdx + 1]) || 8787 : 8787;
|
||||
|
||||
if (useHttp) {
|
||||
console.error(`Starting GBrain MCP server (HTTP on port ${port})...`);
|
||||
await startHttpTransport({ port, engine });
|
||||
// Keep alive
|
||||
await new Promise(() => {});
|
||||
} else {
|
||||
console.error('Starting GBrain MCP server (stdio)...');
|
||||
await startMcpServer(engine);
|
||||
}
|
||||
export async function runServe(engine: BrainEngine) {
|
||||
console.error('Starting GBrain MCP server (stdio)...');
|
||||
await startMcpServer(engine);
|
||||
}
|
||||
|
||||
@@ -1,245 +0,0 @@
|
||||
import { join } from 'path';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { loadStorageConfig, validateStorageConfig, getStorageTier } from '../core/storage-config.ts';
|
||||
import type { StorageConfig, StorageTier } from '../core/storage-config.ts';
|
||||
import { walkBrainRepo, type DiskFileEntry } from '../core/disk-walk.ts';
|
||||
import { getDefaultSourcePath } from '../core/source-resolver.ts';
|
||||
|
||||
/**
|
||||
* Distinct nominal types for the two tier-keyed numeric maps. Both shapes
|
||||
* are `Record<StorageTier, number>` structurally — but they carry
|
||||
* semantically different units (page COUNT vs disk BYTES). Distinct types
|
||||
* make accidental swaps a compile-time error rather than a silent display
|
||||
* bug. Issue #11 of the eng review.
|
||||
*/
|
||||
export type PageCountsByTier = Record<StorageTier, number> & { __brand?: 'page-counts' };
|
||||
export type DiskUsageByTier = Record<StorageTier, number> & { __brand?: 'disk-bytes' };
|
||||
|
||||
/**
|
||||
* Pure-data result of a storage-status query. No side effects, no I/O
|
||||
* beyond the engine call and one filesystem walk. Consumed by both the
|
||||
* JSON formatter and the human formatter; kept narrow so it's a stable
|
||||
* MCP/scripting contract (D14: storage_status is read-only MCP-exposed).
|
||||
*/
|
||||
export interface StorageStatusResult {
|
||||
config: StorageConfig | null;
|
||||
repoPath: string | null;
|
||||
totalPages: number;
|
||||
pagesByTier: PageCountsByTier;
|
||||
missingFiles: Array<{ slug: string; expectedPath: string }>;
|
||||
diskUsageByTier: DiskUsageByTier;
|
||||
warnings: string[];
|
||||
}
|
||||
|
||||
// ── Dispatcher ────────────────────────────────────────────
|
||||
|
||||
export async function runStorage(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const subcommand = args[0];
|
||||
if (!subcommand || subcommand === 'status') {
|
||||
await runStorageStatus(engine, args.slice(1));
|
||||
return;
|
||||
}
|
||||
console.error(`Unknown storage subcommand: ${subcommand}`);
|
||||
console.error('Available subcommands: status');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
async function runStorageStatus(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
warnIfPGLite(engine);
|
||||
|
||||
// Resolution chain (D5, Issue #3): explicit --repo → typed accessor → null.
|
||||
// No cwd fallback. The original silent footgun is dead.
|
||||
let repoPath: string | null = null;
|
||||
const repoIdx = args.indexOf('--repo');
|
||||
if (repoIdx !== -1 && args[repoIdx + 1]) {
|
||||
repoPath = args[repoIdx + 1];
|
||||
} else {
|
||||
repoPath = await getDefaultSourcePath(engine);
|
||||
}
|
||||
|
||||
const result = await getStorageStatus(engine, repoPath);
|
||||
|
||||
if (args.includes('--json')) {
|
||||
console.log(formatStorageStatusJson(result));
|
||||
return;
|
||||
}
|
||||
console.log(formatStorageStatusHuman(result));
|
||||
}
|
||||
|
||||
/**
|
||||
* D4: storage tiering on PGLite is a partial feature. The "DB" the pages
|
||||
* live in IS the local file gbrain uses for everything else, so "db_only"
|
||||
* has no real offload effect. The .gitignore management still helps
|
||||
* (keeps bulk content out of git history), so we warn but proceed.
|
||||
*
|
||||
* Once-per-process via a module-local flag — sub-commands invoked from a
|
||||
* single CLI run share the same warning.
|
||||
*/
|
||||
let _pgliteWarned = false;
|
||||
function warnIfPGLite(engine: BrainEngine): void {
|
||||
if (_pgliteWarned) return;
|
||||
if (engine.kind !== 'pglite') return;
|
||||
_pgliteWarned = true;
|
||||
console.warn(
|
||||
`Note: storage tiering has limited effect on PGLite — pages live in your ` +
|
||||
`local database file regardless of tier. The .gitignore management still ` +
|
||||
`keeps bulk content out of git history. To get full tiering, migrate to ` +
|
||||
`Postgres with \`gbrain migrate --to supabase\`.`,
|
||||
);
|
||||
}
|
||||
|
||||
/** Reset for tests. */
|
||||
export function __resetPGLiteWarn(): void {
|
||||
_pgliteWarned = false;
|
||||
}
|
||||
|
||||
// ── Pure data ─────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Compute the storage status against the given engine + brain repo path.
|
||||
*
|
||||
* Side-effect-free apart from the engine.listPages call and one recursive
|
||||
* filesystem walk. Pure for testability — formatters are tested separately.
|
||||
*
|
||||
* Returns null `config` when no gbrain.yml is present at repoPath. In that
|
||||
* case pagesByTier is all zeros for db_tracked/db_only and totals roll up
|
||||
* into unspecified.
|
||||
*/
|
||||
export async function getStorageStatus(
|
||||
engine: BrainEngine,
|
||||
repoPath: string | null,
|
||||
): Promise<StorageStatusResult> {
|
||||
const config = repoPath ? loadStorageConfig(repoPath) : null;
|
||||
const warnings = config ? validateStorageConfig(config) : [];
|
||||
|
||||
const pagesByTier: PageCountsByTier = { db_tracked: 0, db_only: 0, unspecified: 0 };
|
||||
const diskUsageByTier: DiskUsageByTier = { db_tracked: 0, db_only: 0, unspecified: 0 };
|
||||
const missingFiles: Array<{ slug: string; expectedPath: string }> = [];
|
||||
|
||||
// Single recursive walk of the brain repo (Issue #14). Replaces per-page
|
||||
// existsSync+statSync — was ~400K syscalls on 200K-page brains, now ~one
|
||||
// per directory + one stat per .md file, plus O(1) lookups below.
|
||||
const fileMap: Map<string, DiskFileEntry> = repoPath ? walkBrainRepo(repoPath) : new Map();
|
||||
|
||||
const pages = await engine.listPages({ limit: 1_000_000 });
|
||||
|
||||
for (const page of pages) {
|
||||
const tier = config ? getStorageTier(page.slug, config) : 'unspecified';
|
||||
pagesByTier[tier]++;
|
||||
if (!repoPath) continue;
|
||||
const entry = fileMap.get(page.slug);
|
||||
if (entry) {
|
||||
diskUsageByTier[tier] += entry.size;
|
||||
} else if (config && tier === 'db_only') {
|
||||
missingFiles.push({ slug: page.slug, expectedPath: join(repoPath, page.slug + '.md') });
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
config,
|
||||
repoPath,
|
||||
totalPages: pages.length,
|
||||
pagesByTier,
|
||||
missingFiles,
|
||||
diskUsageByTier,
|
||||
warnings,
|
||||
};
|
||||
}
|
||||
|
||||
// ── JSON formatter ────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Serialize StorageStatusResult to a stable JSON contract. Indented for
|
||||
* human readability; agents/orchestrators can parse with a standard
|
||||
* JSON.parse. Schema is the StorageStatusResult interface above.
|
||||
*/
|
||||
export function formatStorageStatusJson(result: StorageStatusResult): string {
|
||||
return JSON.stringify(result, null, 2);
|
||||
}
|
||||
|
||||
// ── Human formatter ───────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Render StorageStatusResult to ASCII text suitable for terminal output.
|
||||
* D10 lock: ASCII separators only — universally portable. No unicode
|
||||
* box-drawing.
|
||||
*/
|
||||
export function formatStorageStatusHuman(result: StorageStatusResult): string {
|
||||
const lines: string[] = [];
|
||||
lines.push('Storage Status');
|
||||
lines.push('==============');
|
||||
lines.push('');
|
||||
|
||||
if (!result.config) {
|
||||
lines.push('No gbrain.yml configuration found.');
|
||||
if (result.repoPath) lines.push(`Checked: ${result.repoPath}/gbrain.yml`);
|
||||
lines.push('');
|
||||
lines.push('All pages are stored in git by default.');
|
||||
lines.push(`Total pages: ${result.totalPages}`);
|
||||
return lines.join('\n');
|
||||
}
|
||||
|
||||
lines.push(`Repository: ${result.repoPath}`);
|
||||
lines.push(`Total pages: ${result.totalPages}`);
|
||||
lines.push('');
|
||||
lines.push('Storage Tiers:');
|
||||
lines.push('-------------');
|
||||
lines.push(`DB tracked: ${result.pagesByTier.db_tracked.toLocaleString()} pages`);
|
||||
lines.push(`DB only: ${result.pagesByTier.db_only.toLocaleString()} pages`);
|
||||
lines.push(`Unspecified: ${result.pagesByTier.unspecified.toLocaleString()} pages`);
|
||||
|
||||
if (result.diskUsageByTier.db_tracked > 0 || result.diskUsageByTier.db_only > 0) {
|
||||
lines.push('');
|
||||
lines.push('Disk Usage:');
|
||||
lines.push('-----------');
|
||||
if (result.diskUsageByTier.db_tracked > 0) {
|
||||
lines.push(`DB tracked: ${formatBytes(result.diskUsageByTier.db_tracked)}`);
|
||||
}
|
||||
if (result.diskUsageByTier.db_only > 0) {
|
||||
lines.push(`DB only: ${formatBytes(result.diskUsageByTier.db_only)}`);
|
||||
}
|
||||
if (result.diskUsageByTier.unspecified > 0) {
|
||||
lines.push(`Unspecified: ${formatBytes(result.diskUsageByTier.unspecified)}`);
|
||||
}
|
||||
}
|
||||
|
||||
if (result.missingFiles.length > 0) {
|
||||
lines.push('');
|
||||
lines.push('Missing Files (need restore):');
|
||||
lines.push('-----------------------------');
|
||||
for (const missing of result.missingFiles.slice(0, 10)) {
|
||||
lines.push(` ${missing.slug}`);
|
||||
}
|
||||
if (result.missingFiles.length > 10) {
|
||||
lines.push(` ... and ${result.missingFiles.length - 10} more`);
|
||||
}
|
||||
lines.push('');
|
||||
lines.push(`Use: gbrain export --restore-only --repo "${result.repoPath}"`);
|
||||
}
|
||||
|
||||
if (result.warnings.length > 0) {
|
||||
lines.push('');
|
||||
lines.push('Warnings:');
|
||||
lines.push('---------');
|
||||
for (const warning of result.warnings) lines.push(` ! ${warning}`);
|
||||
}
|
||||
|
||||
lines.push('');
|
||||
lines.push('Configuration:');
|
||||
lines.push('--------------');
|
||||
lines.push('DB tracked directories:');
|
||||
for (const dir of result.config.db_tracked) lines.push(` - ${dir}`);
|
||||
lines.push('');
|
||||
lines.push('DB-only directories:');
|
||||
for (const dir of result.config.db_only) lines.push(` - ${dir}`);
|
||||
|
||||
return lines.join('\n');
|
||||
}
|
||||
|
||||
function formatBytes(bytes: number): string {
|
||||
if (bytes === 0) return '0 B';
|
||||
const k = 1024;
|
||||
const sizes = ['B', 'KB', 'MB', 'GB', 'TB'];
|
||||
const i = Math.floor(Math.log(bytes) / Math.log(k));
|
||||
return parseFloat((bytes / Math.pow(k, i)).toFixed(1)) + ' ' + sizes[i];
|
||||
}
|
||||
+14
-365
@@ -1,8 +1,9 @@
|
||||
import { existsSync, readFileSync, writeFileSync, statSync, readdirSync } from 'fs';
|
||||
import { existsSync } from 'fs';
|
||||
import { execFileSync } from 'child_process';
|
||||
import { join, relative } from 'path';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { importFile } from '../core/import-file.ts';
|
||||
import { readFileSync, statSync, readdirSync } from 'fs';
|
||||
import { createInterface } from 'readline';
|
||||
import {
|
||||
buildSyncManifest,
|
||||
@@ -11,7 +12,6 @@ import {
|
||||
recordSyncFailures,
|
||||
unacknowledgedSyncFailures,
|
||||
acknowledgeSyncFailures,
|
||||
formatCodeBreakdown,
|
||||
} from '../core/sync.ts';
|
||||
import { estimateTokens, CHUNKER_VERSION } from '../core/chunkers/code.ts';
|
||||
import { EMBEDDING_MODEL, estimateEmbeddingCostUsd } from '../core/embedding.ts';
|
||||
@@ -19,15 +19,6 @@ import { errorFor, serializeError } from '../core/errors.ts';
|
||||
import type { SyncManifest } from '../core/sync.ts';
|
||||
import { createProgress } from '../core/progress.ts';
|
||||
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
|
||||
import { loadConfig } from '../core/config.ts';
|
||||
import {
|
||||
autoConcurrency,
|
||||
shouldRunParallel,
|
||||
parseWorkers,
|
||||
} from '../core/sync-concurrency.ts';
|
||||
import { tryAcquireDbLock, SYNC_LOCK_ID } from '../core/db-lock.ts';
|
||||
import { loadStorageConfig } from '../core/storage-config.ts';
|
||||
import { getDefaultSourcePath } from '../core/source-resolver.ts';
|
||||
|
||||
export interface SyncResult {
|
||||
status: 'up_to_date' | 'synced' | 'first_sync' | 'dry_run' | 'blocked_by_failures';
|
||||
@@ -166,27 +157,6 @@ export interface SyncOpts {
|
||||
sourceId?: string;
|
||||
/** Multi-repo: sync strategy override (markdown, code, auto). */
|
||||
strategy?: 'markdown' | 'code' | 'auto';
|
||||
/**
|
||||
* Number of parallel workers for the import phase. When > 1, each worker
|
||||
* gets its own small Postgres connection pool and files are dispatched via
|
||||
* an atomic queue index (same pattern as `import --workers N`).
|
||||
*
|
||||
* Deletes and renames remain serial (order-dependent).
|
||||
* Default: undefined → auto-concurrency picks (`src/core/sync-concurrency.ts`).
|
||||
*
|
||||
* v0.22.13 (PR #490 Q1): when this is explicitly set, the >50-file floor
|
||||
* is bypassed — explicit user intent beats the auto-path safety net.
|
||||
*/
|
||||
concurrency?: number;
|
||||
/**
|
||||
* Internal: skip acquiring the gbrain-sync DB lock. Set by the cycle
|
||||
* handler (cycle.ts) which already holds gbrain-cycle and therefore
|
||||
* already serializes against other cycle runs. CLI sync, jobs handler,
|
||||
* and any external caller leave this undefined so they take the lock.
|
||||
*
|
||||
* v0.22.13 (PR #490 CODEX-2). Not part of the public CLI surface.
|
||||
*/
|
||||
skipLock?: boolean;
|
||||
}
|
||||
|
||||
function git(repoPath: string, ...args: string[]): string {
|
||||
@@ -280,39 +250,6 @@ async function writeChunkerVersion(
|
||||
}
|
||||
|
||||
export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise<SyncResult> {
|
||||
// CODEX-2 (v0.22.13): cross-process writer lock for performSync. Two
|
||||
// concurrent syncs can otherwise read the same last_commit anchor, both
|
||||
// write last_commit unconditionally, and the last writer wins — including
|
||||
// regressing the bookmark backwards. cycle.ts already takes gbrain-cycle
|
||||
// for its broader scope; performSync (called from cycle, jobs handler,
|
||||
// and CLI) takes gbrain-sync just for the writer window. The two ids
|
||||
// nest cleanly: cycle holds gbrain-cycle, calls performSync, performSync
|
||||
// takes gbrain-sync. Other callers serialize on gbrain-sync against
|
||||
// each other AND against the cycle's sync phase.
|
||||
//
|
||||
// skipLock is reserved for callers that already serialize via another
|
||||
// mechanism (none in v0.22.13; reserved for future).
|
||||
let lockHandle: { release: () => Promise<void> } | null = null;
|
||||
if (!opts.skipLock) {
|
||||
lockHandle = await tryAcquireDbLock(engine, SYNC_LOCK_ID);
|
||||
if (!lockHandle) {
|
||||
throw new Error(
|
||||
`Another sync is in progress (lock ${SYNC_LOCK_ID} held). ` +
|
||||
`Wait for it to finish, or run 'gbrain doctor' if it has been more than 30 minutes.`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
try {
|
||||
return await performSyncInner(engine, opts);
|
||||
} finally {
|
||||
if (lockHandle) {
|
||||
try { await lockHandle.release(); } catch { /* best-effort release */ }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<SyncResult> {
|
||||
// Resolve repo path
|
||||
const repoPath = opts.repoPath || await readSyncAnchor(engine, opts.sourceId, 'repo_path');
|
||||
if (!repoPath) {
|
||||
@@ -549,41 +486,21 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// gate `sync.last_commit` advancement and record recoverable errors.
|
||||
const failedFiles: Array<{ path: string; error: string; line?: number }> = [];
|
||||
const addsAndMods = [...filtered.added, ...filtered.modified];
|
||||
|
||||
// v0.22.13 (PR #490 Q5): one source of truth for the concurrency decision.
|
||||
// engine.kind === 'pglite' → forced 1; explicit opts.concurrency wins;
|
||||
// auto path returns DEFAULT_PARALLEL_WORKERS only when fileCount > 100.
|
||||
const explicitConcurrency = opts.concurrency !== undefined;
|
||||
const effectiveConcurrency = autoConcurrency(engine, addsAndMods.length, opts.concurrency);
|
||||
const runParallel = shouldRunParallel(effectiveConcurrency, addsAndMods.length, explicitConcurrency);
|
||||
|
||||
if (addsAndMods.length > 0) {
|
||||
progress.start('sync.imports', addsAndMods.length);
|
||||
|
||||
// Core import logic shared by serial and parallel paths.
|
||||
// repoPath is validated non-null at the top of performSyncInner; narrow for TS.
|
||||
const syncRepoPath = repoPath!;
|
||||
async function importOnePath(eng: BrainEngine, path: string): Promise<void> {
|
||||
const filePath = join(syncRepoPath, path);
|
||||
for (const path of addsAndMods) {
|
||||
const filePath = join(repoPath, path);
|
||||
if (!existsSync(filePath)) {
|
||||
// CODEX-3 (v0.22.13): a file the diff said exists at headCommit but
|
||||
// is gone from disk means the working tree has drifted (someone ran
|
||||
// `git checkout` / `git reset` mid-sync, or the file was deleted
|
||||
// post-diff). Record as a failure so last_commit does NOT advance —
|
||||
// the silent-skip-then-advance pathology was the bug.
|
||||
failedFiles.push({
|
||||
path,
|
||||
error: 'file vanished mid-sync (working tree drifted from headCommit)',
|
||||
});
|
||||
progress.tick(1, `skip:${path}`);
|
||||
return;
|
||||
continue;
|
||||
}
|
||||
try {
|
||||
const result = await importFile(eng, filePath, path, { noEmbed });
|
||||
const result = await importFile(engine, filePath, path, { noEmbed });
|
||||
if (result.status === 'imported') {
|
||||
chunksCreated += result.chunks;
|
||||
pagesAffected.push(result.slug);
|
||||
} else if (result.status === 'skipped' && (result as any).error) {
|
||||
// importFile returned a non-throw skip with a reason.
|
||||
failedFiles.push({ path, error: String((result as any).error) });
|
||||
}
|
||||
} catch (e: unknown) {
|
||||
@@ -593,98 +510,9 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
}
|
||||
progress.tick(1, path);
|
||||
}
|
||||
|
||||
if (runParallel) {
|
||||
// A1 (v0.22.13): use engine.kind discriminator instead of config?.engine
|
||||
// string compare or constructor.name sniff. Q3: belt-and-suspenders fall
|
||||
// back to serial when database_url is unset, so we never crash on a null
|
||||
// assertion if config is missing.
|
||||
const config = loadConfig();
|
||||
if (engine.kind === 'pglite' || !config?.database_url) {
|
||||
for (const path of addsAndMods) {
|
||||
await importOnePath(engine, path);
|
||||
}
|
||||
} else {
|
||||
const { PostgresEngine } = await import('../core/postgres-engine.ts');
|
||||
const { resolvePoolSize } = await import('../core/db.ts');
|
||||
const workerPoolSize = Math.min(2, resolvePoolSize(2));
|
||||
const workerCount = Math.min(effectiveConcurrency, addsAndMods.length);
|
||||
const databaseUrl = config.database_url;
|
||||
|
||||
// Q4 (v0.22.13): banner on stderr so stdout stays clean for --json.
|
||||
console.error(` Parallel sync: ${workerCount} workers for ${addsAndMods.length} files`);
|
||||
|
||||
const workerEngines: InstanceType<typeof PostgresEngine>[] = [];
|
||||
try {
|
||||
// Connect workers one-by-one rather than Promise.all so a partial
|
||||
// failure leaves us with the connected ones in workerEngines for
|
||||
// the finally-block cleanup. The original code lost track of
|
||||
// already-connected engines on any one failure.
|
||||
for (let i = 0; i < workerCount; i++) {
|
||||
const eng = new PostgresEngine();
|
||||
await eng.connect({ database_url: databaseUrl, poolSize: workerPoolSize });
|
||||
workerEngines.push(eng);
|
||||
}
|
||||
|
||||
// Atomic queue index — JS is single-threaded; the read-then-increment
|
||||
// happens between awaits, so no lock is needed.
|
||||
let queueIndex = 0;
|
||||
await Promise.all(
|
||||
workerEngines.map(async (eng) => {
|
||||
while (true) {
|
||||
const idx = queueIndex++;
|
||||
if (idx >= addsAndMods.length) break;
|
||||
await importOnePath(eng, addsAndMods[idx]);
|
||||
}
|
||||
}),
|
||||
);
|
||||
} finally {
|
||||
// A2 (v0.22.13): try/finally guarantees connection cleanup even when
|
||||
// the worker loop throws (partial connect failure, OOM, mid-import
|
||||
// signal). Each disconnect is best-effort — one worker failing to
|
||||
// disconnect must not strand the others.
|
||||
await Promise.all(
|
||||
workerEngines.map((e) =>
|
||||
e.disconnect().catch((err: unknown) =>
|
||||
console.error(` worker disconnect failed: ${err instanceof Error ? err.message : String(err)}`),
|
||||
),
|
||||
),
|
||||
);
|
||||
}
|
||||
}
|
||||
} else {
|
||||
// Serial path (small auto diffs or explicit --workers 1).
|
||||
for (const path of addsAndMods) {
|
||||
await importOnePath(engine, path);
|
||||
}
|
||||
}
|
||||
|
||||
progress.finish();
|
||||
}
|
||||
|
||||
// CODEX-3 (v0.22.13): head-drift gate. If git HEAD moved during the import
|
||||
// window (someone ran `git checkout` or `git pull` in another terminal /
|
||||
// sibling Conductor workspace), the chunks we just imported reflect a
|
||||
// different tree than `headCommit` claims. Refuse to advance last_commit
|
||||
// so the next sync re-walks against the new HEAD. The lock from CODEX-2
|
||||
// prevents *this* gbrain process from stepping on itself; this gate
|
||||
// catches drift caused by external `git` commands the lock cannot see.
|
||||
try {
|
||||
const currentHead = git(repoPath, 'rev-parse', 'HEAD');
|
||||
if (currentHead !== headCommit) {
|
||||
failedFiles.push({
|
||||
path: '<head>',
|
||||
error: `git HEAD drifted during sync: captured ${headCommit.slice(0, 8)}, now ${currentHead.slice(0, 8)}`,
|
||||
});
|
||||
}
|
||||
} catch (e) {
|
||||
// rev-parse failure is itself a drift signal (worktree disappeared).
|
||||
failedFiles.push({
|
||||
path: '<head>',
|
||||
error: `git HEAD verification failed: ${e instanceof Error ? e.message : String(e)}`,
|
||||
});
|
||||
}
|
||||
|
||||
const elapsed = Date.now() - start;
|
||||
|
||||
// Bug 9 — gate the sync bookmark on success. If any per-file parse
|
||||
@@ -694,13 +522,9 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// current set, --retry-failed re-parses before running the normal sync.
|
||||
if (failedFiles.length > 0) {
|
||||
recordSyncFailures(failedFiles, headCommit);
|
||||
// Emit structured summary grouped by error code so the operator
|
||||
// can see *why* files failed, not just how many.
|
||||
const codeBreakdown = formatCodeBreakdown(failedFiles);
|
||||
if (!opts.skipFailed) {
|
||||
console.error(
|
||||
`\nSync blocked: ${failedFiles.length} file(s) failed to parse:\n` +
|
||||
`${codeBreakdown}\n\n` +
|
||||
`\nSync blocked: ${failedFiles.length} file(s) failed to parse. ` +
|
||||
`Fix the YAML frontmatter in the files above and re-run, or use ` +
|
||||
`'gbrain sync --skip-failed' to acknowledge and move on.`,
|
||||
);
|
||||
@@ -723,11 +547,8 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
}
|
||||
// --skip-failed: acknowledge the now-recorded set and proceed.
|
||||
const acked = acknowledgeSyncFailures();
|
||||
if (acked.count > 0) {
|
||||
console.error(
|
||||
` Acknowledged ${acked.count} failure(s) and advancing past them:\n` +
|
||||
`${formatCodeBreakdown(acked.summary)}`,
|
||||
);
|
||||
if (acked > 0) {
|
||||
console.error(` Acknowledged ${acked} failure(s) and advancing past them.`);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -823,18 +644,10 @@ async function performFullSync(
|
||||
};
|
||||
}
|
||||
|
||||
// v0.22.13 (PR #490 A1 + Q5): full sync is always "large" by definition
|
||||
// (entire working tree). Auto-concurrency fires unconditionally for Postgres;
|
||||
// PGLite stays serial because its engine is single-connection. Routes the
|
||||
// policy through autoConcurrency() so it stays consistent with incremental
|
||||
// sync and the jobs handler.
|
||||
const FULL_SYNC_LARGE_MARKER = Number.MAX_SAFE_INTEGER;
|
||||
const fullConcurrency = autoConcurrency(engine, FULL_SYNC_LARGE_MARKER, opts.concurrency);
|
||||
console.log(`Running full import of ${repoPath}${fullConcurrency > 1 ? ` (${fullConcurrency} workers)` : ''}...`);
|
||||
console.log(`Running full import of ${repoPath}...`);
|
||||
const { runImport } = await import('./import.ts');
|
||||
const importArgs = [repoPath];
|
||||
if (opts.noEmbed) importArgs.push('--no-embed');
|
||||
if (fullConcurrency > 1) importArgs.push('--workers', String(fullConcurrency));
|
||||
const result = await runImport(engine, importArgs, { commit: headCommit });
|
||||
|
||||
// Bug 9 — gate the full-sync bookmark on success. runImport already
|
||||
@@ -843,11 +656,9 @@ async function performFullSync(
|
||||
// the sync module owns the last_commit write. Respect the same gate.
|
||||
if (result.failures.length > 0) {
|
||||
recordSyncFailures(result.failures, headCommit);
|
||||
const codeBreakdown = formatCodeBreakdown(result.failures);
|
||||
if (!opts.skipFailed) {
|
||||
console.error(
|
||||
`\nFull sync blocked: ${result.failures.length} file(s) failed:\n` +
|
||||
`${codeBreakdown}\n\n` +
|
||||
`\nFull sync blocked: ${result.failures.length} file(s) failed. ` +
|
||||
`Fix the YAML in those files and re-run, or use '--skip-failed'.`,
|
||||
);
|
||||
await engine.setConfig('sync.last_run', new Date().toISOString());
|
||||
@@ -864,12 +675,7 @@ async function performFullSync(
|
||||
};
|
||||
}
|
||||
const acked = acknowledgeSyncFailures();
|
||||
if (acked.count > 0) {
|
||||
console.error(
|
||||
` Acknowledged ${acked.count} failure(s) and advancing past them:\n` +
|
||||
`${formatCodeBreakdown(acked.summary)}`,
|
||||
);
|
||||
}
|
||||
if (acked > 0) console.error(` Acknowledged ${acked} failure(s) and advancing past them.`);
|
||||
}
|
||||
|
||||
// Persist sync state so next sync is incremental (C1 fix: was missing).
|
||||
@@ -922,17 +728,6 @@ export async function runSync(engine: BrainEngine, args: string[]) {
|
||||
const jsonOut = args.includes('--json');
|
||||
const yesFlag = args.includes('--yes');
|
||||
const strategyArg = args.find((a, i) => args[i - 1] === '--strategy') as SyncOpts['strategy'] | undefined;
|
||||
const concurrencyStr = args.find((a, i) => args[i - 1] === '--concurrency' || args[i - 1] === '--workers');
|
||||
// v0.22.13 (PR #490 Q2): parseWorkers throws on '0', '-3', 'foo', '1.5' instead
|
||||
// of silently falling through to auto-concurrency or NaN. Loud failure beats
|
||||
// a 4-worker spawn from a typo.
|
||||
let concurrency: number | undefined;
|
||||
try {
|
||||
concurrency = parseWorkers(concurrencyStr);
|
||||
} catch (e) {
|
||||
console.error(e instanceof Error ? e.message : String(e));
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// v0.18.0 Step 5: --source resolves to a sources(id) row. Falls back
|
||||
// to pre-v0.17 global config (sync.repo_path + sync.last_commit) when
|
||||
@@ -1025,18 +820,10 @@ export async function runSync(engine: BrainEngine, args: string[]) {
|
||||
dryRun, full, noPull, noEmbed, skipFailed, retryFailed,
|
||||
sourceId: src.id,
|
||||
strategy: cfg.strategy,
|
||||
concurrency,
|
||||
};
|
||||
try {
|
||||
const result = await performSync(engine, repoOpts);
|
||||
printSyncResult(result);
|
||||
// Codex P2: --all loop must also manage .gitignore per-source. Without
|
||||
// this, multi-source users who rely on `gbrain sync --all` never get
|
||||
// the advertised db_only ignore rules unless they sync each repo
|
||||
// individually.
|
||||
if (result.status !== 'dry_run' && result.status !== 'blocked_by_failures') {
|
||||
manageGitignore(src.local_path!, engine.kind);
|
||||
}
|
||||
} catch (e: unknown) {
|
||||
console.error(`Error syncing ${src.name}: ${e instanceof Error ? e.message : String(e)}`);
|
||||
}
|
||||
@@ -1044,7 +831,7 @@ export async function runSync(engine: BrainEngine, args: string[]) {
|
||||
return;
|
||||
}
|
||||
|
||||
const opts: SyncOpts = { repoPath, dryRun, full, noPull, noEmbed, skipFailed, retryFailed, sourceId, strategy: strategyArg, concurrency };
|
||||
const opts: SyncOpts = { repoPath, dryRun, full, noPull, noEmbed, skipFailed, retryFailed, sourceId, strategy: strategyArg };
|
||||
|
||||
// Bug 9 — --retry-failed: before running normal sync, clear acknowledgment
|
||||
// flags so the sync picks them up as fresh work. The actual re-attempt
|
||||
@@ -1063,18 +850,6 @@ export async function runSync(engine: BrainEngine, args: string[]) {
|
||||
if (!watch) {
|
||||
const result = await performSync(engine, opts);
|
||||
printSyncResult(result);
|
||||
// Issue #2 + eng-review pass-2 finding #1 + Codex P1: manage .gitignore ONLY
|
||||
// on successful sync. Skip on dry-run (don't mutate disk in preview mode)
|
||||
// and blocked_by_failures (sync state is inconsistent — defer .gitignore
|
||||
// until next clean run). Resolve the effective repo path so the wire-up
|
||||
// fires in the common case where the user runs `gbrain sync` without
|
||||
// passing --repo every time.
|
||||
if (result.status !== 'dry_run' && result.status !== 'blocked_by_failures') {
|
||||
const effectiveRepoPath = opts.repoPath ?? (await getDefaultSourcePath(engine));
|
||||
if (effectiveRepoPath) {
|
||||
manageGitignore(effectiveRepoPath, engine.kind);
|
||||
}
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -1090,14 +865,6 @@ export async function runSync(engine: BrainEngine, args: string[]) {
|
||||
const ts = new Date().toISOString().slice(11, 19);
|
||||
console.log(`[${ts}] Synced: +${result.added} ~${result.modified} -${result.deleted} R${result.renamed}`);
|
||||
}
|
||||
// Same gate as non-watch: only manage .gitignore on successful sync.
|
||||
// Same repo-resolution path so watch mode catches the implicit-resolved case.
|
||||
if (result.status !== 'dry_run' && result.status !== 'blocked_by_failures') {
|
||||
const effectiveRepoPath = opts.repoPath ?? (await getDefaultSourcePath(engine));
|
||||
if (effectiveRepoPath) {
|
||||
manageGitignore(effectiveRepoPath, engine.kind);
|
||||
}
|
||||
}
|
||||
} catch (e: unknown) {
|
||||
consecutiveErrors++;
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
@@ -1111,124 +878,6 @@ export async function runSync(engine: BrainEngine, args: string[]) {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Auto-manage .gitignore entries for db_only directories.
|
||||
*
|
||||
* Caller invokes ONLY on successful sync — this function trusts that the
|
||||
* sync's data state is consistent. See `runSync` for the gating logic.
|
||||
*
|
||||
* Idempotent: re-running adds no duplicate entries. The managed block has
|
||||
* a stable comment header so it's grep-able and editable.
|
||||
*
|
||||
* Skipped (with actionable warning) when:
|
||||
* - GBRAIN_NO_GITIGNORE=1 — D23 escape hatch for shared-repo setups
|
||||
* - The repo is a git submodule (`.git` is a file not a directory) —
|
||||
* D49 lock; submodule .gitignore changes don't survive parent updates
|
||||
*
|
||||
* On PGLite (D4): emits a once-per-process soft-warn explaining that
|
||||
* tiering has limited effect — but still manages the .gitignore so the
|
||||
* config-present user gets the gitignore housekeeping.
|
||||
*
|
||||
* Failures (write permission denied, EROFS, etc.) are caught, warned, and
|
||||
* swallowed (D9 lock). Sync's primary job is moving data; .gitignore
|
||||
* management is a side effect — don't kill the main job for the side effect.
|
||||
*/
|
||||
let _pgliteTierWarned = false;
|
||||
export function __resetPGLiteTierWarn(): void {
|
||||
_pgliteTierWarned = false;
|
||||
}
|
||||
|
||||
export function manageGitignore(
|
||||
repoPath: string,
|
||||
engineKind?: 'pglite' | 'postgres',
|
||||
): void {
|
||||
if (process.env.GBRAIN_NO_GITIGNORE === '1') {
|
||||
return;
|
||||
}
|
||||
|
||||
// D49: submodule detection. In a submodule, `.git` is a regular file
|
||||
// (containing `gitdir: ../path/to/parent.git/modules/x`), not a directory.
|
||||
const dotGit = join(repoPath, '.git');
|
||||
if (existsSync(dotGit)) {
|
||||
try {
|
||||
if (statSync(dotGit).isFile()) {
|
||||
console.warn(
|
||||
`Note: skipping .gitignore management — ${repoPath} is a git submodule. ` +
|
||||
`Add db_only directories to your parent repo's .gitignore manually.`,
|
||||
);
|
||||
return;
|
||||
}
|
||||
} catch {
|
||||
// proceed; can't tell, default to managing
|
||||
}
|
||||
}
|
||||
|
||||
let storageConfig;
|
||||
try {
|
||||
storageConfig = loadStorageConfig(repoPath);
|
||||
} catch (error) {
|
||||
// StorageConfigError (overlap) or read error — surface, don't manage.
|
||||
console.warn(
|
||||
`Skipped .gitignore update: ${error instanceof Error ? error.message : String(error)}`,
|
||||
);
|
||||
return;
|
||||
}
|
||||
if (!storageConfig || storageConfig.db_only.length === 0) {
|
||||
return;
|
||||
}
|
||||
|
||||
// D4 soft-warn: storage tiering has limited effect on PGLite, but the
|
||||
// .gitignore housekeeping still helps. Warn once per process; proceed.
|
||||
if (engineKind === 'pglite' && !_pgliteTierWarned) {
|
||||
_pgliteTierWarned = true;
|
||||
console.warn(
|
||||
`Note: storage tiering has limited effect on PGLite — pages live in your ` +
|
||||
`local database file regardless of tier. Managing .gitignore anyway.`,
|
||||
);
|
||||
}
|
||||
|
||||
const gitignorePath = join(repoPath, '.gitignore');
|
||||
let gitignoreContent = '';
|
||||
|
||||
if (existsSync(gitignorePath)) {
|
||||
try {
|
||||
gitignoreContent = readFileSync(gitignorePath, 'utf-8');
|
||||
} catch (error) {
|
||||
console.warn(
|
||||
`Could not read ${gitignorePath} (${error instanceof Error ? error.message : String(error)}) — ` +
|
||||
`skipping .gitignore update. Add db_only directories manually.`,
|
||||
);
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
const existingLines = new Set(gitignoreContent.split('\n').map((line) => line.trim()));
|
||||
const linesToAdd: string[] = [];
|
||||
|
||||
for (const dir of storageConfig.db_only) {
|
||||
if (!existingLines.has(dir) && !existingLines.has(`/${dir}`)) {
|
||||
linesToAdd.push(dir);
|
||||
}
|
||||
}
|
||||
|
||||
if (linesToAdd.length === 0) return;
|
||||
|
||||
if (gitignoreContent && !gitignoreContent.endsWith('\n')) {
|
||||
gitignoreContent += '\n';
|
||||
}
|
||||
gitignoreContent += '\n# Auto-managed by gbrain (db_only directories)\n';
|
||||
gitignoreContent += linesToAdd.join('\n') + '\n';
|
||||
|
||||
try {
|
||||
writeFileSync(gitignorePath, gitignoreContent);
|
||||
} catch (error) {
|
||||
console.warn(
|
||||
`Could not update ${gitignorePath} (${error instanceof Error ? error.message : String(error)}) — ` +
|
||||
`please add db_only directories manually:\n ${linesToAdd.join('\n ')}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
function printSyncResult(result: SyncResult) {
|
||||
switch (result.status) {
|
||||
case 'up_to_date':
|
||||
|
||||
@@ -1,394 +0,0 @@
|
||||
/**
|
||||
* brain-writer — frontmatter validation/audit/auto-fix orchestrator.
|
||||
*
|
||||
* Thin layer on top of `parseMarkdown(..., {validate:true})` (the canonical
|
||||
* source of frontmatter validation rules) and `isSyncable()` (the canonical
|
||||
* brain-page filter). Three consumers call into this module: the
|
||||
* `gbrain frontmatter` CLI, the `frontmatter_integrity` doctor subcheck, and
|
||||
* the v0.22.4 migration audit phase. Single source of truth — no parallel
|
||||
* validation stack.
|
||||
*
|
||||
* Path-guard contract: writeBrainPage refuses to write outside the source
|
||||
* path. .bak backups are the safety contract (works for both git and non-git
|
||||
* brain repos; the existing src/core/dry-fix.ts:getWorkingTreeStatus rejects
|
||||
* non-git repos as unsafe, which is the wrong shape for brain rewrites).
|
||||
*/
|
||||
|
||||
import { existsSync, readFileSync, readdirSync, statSync, copyFileSync, writeFileSync, mkdirSync, lstatSync } from 'fs';
|
||||
import { join, relative, resolve, dirname } from 'path';
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
import type { ProgressReporter } from './progress.ts';
|
||||
import {
|
||||
parseMarkdown,
|
||||
type ParseValidationCode,
|
||||
type ParseValidationError,
|
||||
} from './markdown.ts';
|
||||
import { isSyncable, slugifyPath } from './sync.ts';
|
||||
|
||||
export type { ParseValidationCode };
|
||||
|
||||
export interface AuditFix {
|
||||
code: ParseValidationCode;
|
||||
description: string;
|
||||
}
|
||||
|
||||
export interface PerSourceReport {
|
||||
source_id: string;
|
||||
source_path: string;
|
||||
total: number;
|
||||
errors_by_code: Partial<Record<ParseValidationCode, number>>;
|
||||
sample: { path: string; codes: ParseValidationCode[] }[];
|
||||
}
|
||||
|
||||
export interface AuditReport {
|
||||
ok: boolean;
|
||||
total: number;
|
||||
errors_by_code: Partial<Record<ParseValidationCode, number>>;
|
||||
per_source: PerSourceReport[];
|
||||
scanned_at: string;
|
||||
}
|
||||
|
||||
const SAMPLE_PER_SOURCE = 20;
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// autoFixFrontmatter
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Mechanical auto-repair for the fixable subset of validation codes:
|
||||
* - NULL_BYTES — strip \x00 characters
|
||||
* - NESTED_QUOTES — rewrite `"... "inner" ..."` to single-quoted outer
|
||||
* - MISSING_CLOSE — insert `---` before the first heading found inside
|
||||
* the YAML zone
|
||||
* - SLUG_MISMATCH — remove `slug:` line (gbrain derives slug from path)
|
||||
*
|
||||
* Idempotent: running twice is a no-op on already-clean input. Any error class
|
||||
* not in the list above is left untouched (e.g. EMPTY_FRONTMATTER, YAML_PARSE,
|
||||
* MISSING_OPEN — those need human review).
|
||||
*/
|
||||
export function autoFixFrontmatter(
|
||||
content: string,
|
||||
opts?: { filePath?: string },
|
||||
): { content: string; fixes: AuditFix[] } {
|
||||
const fixes: AuditFix[] = [];
|
||||
let working = content;
|
||||
|
||||
// 1. NULL_BYTES — strip them. Cheap, byte-level. Run first so subsequent
|
||||
// line-based passes don't trip on stray nulls.
|
||||
if (working.indexOf('\x00') >= 0) {
|
||||
working = working.replace(/\x00/g, '');
|
||||
fixes.push({ code: 'NULL_BYTES', description: 'Stripped null bytes' });
|
||||
}
|
||||
|
||||
// 2. MISSING_CLOSE — if there's an opener but no closer before a heading,
|
||||
// insert `---` immediately before the heading. Walk lines once.
|
||||
{
|
||||
const lines = working.split('\n');
|
||||
let firstNonEmpty = -1;
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
if (lines[i].trim().length > 0) { firstNonEmpty = i; break; }
|
||||
}
|
||||
if (firstNonEmpty >= 0 && lines[firstNonEmpty].trim() === '---') {
|
||||
let closeIdx = -1;
|
||||
let headingIdx = -1;
|
||||
for (let i = firstNonEmpty + 1; i < lines.length; i++) {
|
||||
const t = lines[i].trim();
|
||||
if (t === '---') { closeIdx = i; break; }
|
||||
if (/^#{1,6}\s/.test(t)) { headingIdx = i; break; }
|
||||
}
|
||||
if (closeIdx === -1 && headingIdx >= 0) {
|
||||
const fixed = [
|
||||
...lines.slice(0, headingIdx),
|
||||
'---',
|
||||
'',
|
||||
...lines.slice(headingIdx),
|
||||
];
|
||||
working = fixed.join('\n');
|
||||
fixes.push({
|
||||
code: 'MISSING_CLOSE',
|
||||
description: `Inserted closing --- before heading at line ${headingIdx + 1}`,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 3. NESTED_QUOTES — rewrite `key: "...inner..."` lines that have 3+ unescaped
|
||||
// double-quotes by switching the outer wrapper to single quotes and
|
||||
// leaving inner quotes alone.
|
||||
{
|
||||
const lines = working.split('\n');
|
||||
let firstNonEmpty = -1;
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
if (lines[i].trim().length > 0) { firstNonEmpty = i; break; }
|
||||
}
|
||||
if (firstNonEmpty >= 0 && lines[firstNonEmpty].trim() === '---') {
|
||||
let closeIdx = lines.length;
|
||||
for (let i = firstNonEmpty + 1; i < lines.length; i++) {
|
||||
if (lines[i].trim() === '---') { closeIdx = i; break; }
|
||||
}
|
||||
let fixedAny = false;
|
||||
for (let i = firstNonEmpty + 1; i < closeIdx; i++) {
|
||||
const m = lines[i].match(/^(\s*[A-Za-z_][\w-]*\s*:\s*)"(.*)"\s*(.*)$/);
|
||||
if (!m) continue;
|
||||
const [, prefix, inner, trailing] = m;
|
||||
let count = 0;
|
||||
for (let j = 0; j < inner.length; j++) {
|
||||
if (inner[j] === '"' && (j === 0 || inner[j - 1] !== '\\')) count++;
|
||||
}
|
||||
// Total " on the line includes the two outer quotes the regex
|
||||
// captured, plus whatever's in inner. We need 3+ to trigger.
|
||||
if (count >= 1) {
|
||||
// Inner already has unescaped " — outer wrap is causing the YAML
|
||||
// parse failure. Rewrite to 'single-quoted'. YAML escapes `'` inside
|
||||
// a single-quoted string by doubling it.
|
||||
const escapedInner = inner.replace(/'/g, "''");
|
||||
lines[i] = `${prefix}'${escapedInner}'${trailing ? ' ' + trailing : ''}`.replace(/\s+$/, '');
|
||||
fixedAny = true;
|
||||
}
|
||||
}
|
||||
if (fixedAny) {
|
||||
working = lines.join('\n');
|
||||
fixes.push({
|
||||
code: 'NESTED_QUOTES',
|
||||
description: 'Rewrote nested double-quoted YAML values to single-quoted',
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 4. SLUG_MISMATCH — remove `slug:` line if filePath is provided and the
|
||||
// declared slug doesn't match the path-derived one. Per PR #392 spec,
|
||||
// gbrain derives slug from path; the field shouldn't be in frontmatter.
|
||||
if (opts?.filePath) {
|
||||
const expectedSlug = slugifyPath(opts.filePath);
|
||||
// Use the (possibly partially-fixed) working content to detect whether
|
||||
// the slug field is present and mismatched.
|
||||
const re = /^slug:\s*(.+?)\s*$/m;
|
||||
const m = working.match(re);
|
||||
if (m && m[1].replace(/^["']|["']$/g, '') !== expectedSlug) {
|
||||
working = working.replace(re, '').replace(/\n{3,}/g, '\n\n');
|
||||
fixes.push({
|
||||
code: 'SLUG_MISMATCH',
|
||||
description: `Removed mismatched slug field (was "${m[1]}", expected "${expectedSlug}")`,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
return { content: working, fixes };
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// writeBrainPage — path-guarded write with .bak backup
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export class BrainWriterError extends Error {
|
||||
code: string;
|
||||
hint?: string;
|
||||
constructor(code: string, message: string, hint?: string) {
|
||||
super(message);
|
||||
this.name = 'BrainWriterError';
|
||||
this.code = code;
|
||||
this.hint = hint;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Path-guarded brain page writer. Always writes `<filePath>.bak` before any
|
||||
* in-place mutation (the contract that replaces git-tree-clean for non-git
|
||||
* brain repos). Throws BrainWriterError if filePath is not under sourcePath.
|
||||
*/
|
||||
export function writeBrainPage(
|
||||
filePath: string,
|
||||
content: string,
|
||||
opts: { sourcePath: string; autoFix?: boolean },
|
||||
): { fixes: AuditFix[] } {
|
||||
const resolvedSource = resolve(opts.sourcePath);
|
||||
const resolvedTarget = resolve(filePath);
|
||||
if (resolvedTarget !== resolvedSource && !resolvedTarget.startsWith(resolvedSource + '/')) {
|
||||
throw new BrainWriterError(
|
||||
'PATH_OUTSIDE_SOURCE',
|
||||
`writeBrainPage: ${filePath} is not under ${opts.sourcePath}`,
|
||||
'Pass --source <id> matching the source the file lives in.',
|
||||
);
|
||||
}
|
||||
|
||||
let toWrite = content;
|
||||
let fixes: AuditFix[] = [];
|
||||
if (opts.autoFix) {
|
||||
const result = autoFixFrontmatter(content, { filePath });
|
||||
toWrite = result.content;
|
||||
fixes = result.fixes;
|
||||
}
|
||||
|
||||
if (existsSync(filePath)) {
|
||||
copyFileSync(filePath, filePath + '.bak');
|
||||
} else {
|
||||
mkdirSync(dirname(filePath), { recursive: true });
|
||||
}
|
||||
writeFileSync(filePath, toWrite, 'utf8');
|
||||
return { fixes };
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// scanBrainSources
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
interface SourceRow {
|
||||
id: string;
|
||||
local_path: string | null;
|
||||
}
|
||||
|
||||
export interface ScanOpts {
|
||||
/** Limit scan to one source. When omitted, all registered sources with a
|
||||
* local_path are scanned. */
|
||||
sourceId?: string;
|
||||
onProgress?: ProgressReporter;
|
||||
signal?: AbortSignal;
|
||||
}
|
||||
|
||||
export async function scanBrainSources(
|
||||
engine: BrainEngine,
|
||||
opts: ScanOpts = {},
|
||||
): Promise<AuditReport> {
|
||||
const sources = await listSources(engine, opts.sourceId);
|
||||
const totals: Partial<Record<ParseValidationCode, number>> = {};
|
||||
const perSource: PerSourceReport[] = [];
|
||||
let grandTotal = 0;
|
||||
|
||||
for (const src of sources) {
|
||||
if (opts.signal?.aborted) break;
|
||||
if (!src.local_path) continue;
|
||||
if (!existsSync(src.local_path)) {
|
||||
// Source registered but path is missing on disk; surface as a zero-row
|
||||
// entry with a synthetic SCAN_PATH_MISSING note via warn-and-skip.
|
||||
perSource.push({
|
||||
source_id: src.id,
|
||||
source_path: src.local_path,
|
||||
total: 0,
|
||||
errors_by_code: {},
|
||||
sample: [],
|
||||
});
|
||||
continue;
|
||||
}
|
||||
const report = scanOneSource(src.id, src.local_path, opts);
|
||||
perSource.push(report);
|
||||
grandTotal += report.total;
|
||||
for (const [code, n] of Object.entries(report.errors_by_code)) {
|
||||
const k = code as ParseValidationCode;
|
||||
totals[k] = (totals[k] ?? 0) + (n as number);
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
ok: grandTotal === 0,
|
||||
total: grandTotal,
|
||||
errors_by_code: totals,
|
||||
per_source: perSource,
|
||||
scanned_at: new Date().toISOString(),
|
||||
};
|
||||
}
|
||||
|
||||
function scanOneSource(
|
||||
sourceId: string,
|
||||
sourcePath: string,
|
||||
opts: ScanOpts,
|
||||
): PerSourceReport {
|
||||
const errorsByCode: Partial<Record<ParseValidationCode, number>> = {};
|
||||
const sample: PerSourceReport['sample'] = [];
|
||||
const rootResolved = resolve(sourcePath);
|
||||
let scanned = 0;
|
||||
let total = 0;
|
||||
|
||||
walkDir(rootResolved, (absPath) => {
|
||||
if (opts.signal?.aborted) return false;
|
||||
const relPath = relative(rootResolved, absPath);
|
||||
if (!isSyncable(relPath, { strategy: 'markdown' })) return true;
|
||||
scanned++;
|
||||
let content: string;
|
||||
try {
|
||||
content = readFileSync(absPath, 'utf8');
|
||||
} catch {
|
||||
return true; // skip unreadable
|
||||
}
|
||||
const expectedSlug = slugifyPath(relPath);
|
||||
const parsed = parseMarkdown(content, relPath, { validate: true, expectedSlug });
|
||||
const errs = parsed.errors ?? [];
|
||||
if (errs.length > 0) {
|
||||
total += errs.length;
|
||||
const codes: ParseValidationCode[] = [];
|
||||
for (const e of errs) {
|
||||
errorsByCode[e.code] = (errorsByCode[e.code] ?? 0) + 1;
|
||||
codes.push(e.code);
|
||||
}
|
||||
if (sample.length < SAMPLE_PER_SOURCE) {
|
||||
sample.push({ path: relPath, codes });
|
||||
}
|
||||
}
|
||||
if (opts.onProgress && scanned % 50 === 0) {
|
||||
opts.onProgress.tick(50);
|
||||
}
|
||||
return true;
|
||||
});
|
||||
|
||||
if (opts.onProgress) {
|
||||
opts.onProgress.heartbeat(`scanned ${scanned} pages in ${sourceId}`);
|
||||
}
|
||||
|
||||
return {
|
||||
source_id: sourceId,
|
||||
source_path: sourcePath,
|
||||
total,
|
||||
errors_by_code: errorsByCode,
|
||||
sample,
|
||||
};
|
||||
}
|
||||
|
||||
/** Recursive directory walker with symlink-loop protection (via lstat).
|
||||
* Calls `visit` for each regular file. Returning false from `visit` stops
|
||||
* the walk. Skips entries lstat reports as symlinks (sync's no-symlink
|
||||
* policy). */
|
||||
function walkDir(root: string, visit: (absPath: string) => boolean | void): void {
|
||||
const stack: string[] = [root];
|
||||
const visited = new Set<string>();
|
||||
while (stack.length > 0) {
|
||||
const dir = stack.pop()!;
|
||||
let entries: string[];
|
||||
try {
|
||||
entries = readdirSync(dir);
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
for (const name of entries) {
|
||||
const full = join(dir, name);
|
||||
let st: ReturnType<typeof lstatSync>;
|
||||
try {
|
||||
st = lstatSync(full);
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
if (st.isSymbolicLink()) continue; // matches sync's no-symlink policy
|
||||
if (st.isDirectory()) {
|
||||
const real = resolve(full);
|
||||
if (visited.has(real)) continue;
|
||||
visited.add(real);
|
||||
stack.push(full);
|
||||
} else if (st.isFile()) {
|
||||
const result = visit(full);
|
||||
if (result === false) return;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function listSources(engine: BrainEngine, sourceId?: string): Promise<SourceRow[]> {
|
||||
if (sourceId) {
|
||||
const rows = await engine.executeRaw<SourceRow>(
|
||||
`SELECT id, local_path FROM sources WHERE id = $1`,
|
||||
[sourceId],
|
||||
);
|
||||
return rows;
|
||||
}
|
||||
return engine.executeRaw<SourceRow>(
|
||||
`SELECT id, local_path FROM sources WHERE local_path IS NOT NULL ORDER BY id`,
|
||||
);
|
||||
}
|
||||
+7
-94
@@ -140,14 +140,6 @@ export interface CycleOpts {
|
||||
* + refreshes the cycle-lock-table TTL.
|
||||
*/
|
||||
yieldBetweenPhases?: () => Promise<void>;
|
||||
/**
|
||||
* AbortSignal from the Minions worker. When aborted (timeout, cancel,
|
||||
* lock-loss), runCycle bails between phases and returns a 'failed' report
|
||||
* instead of running the next phase. Without this, a timed-out
|
||||
* autopilot-cycle handler ignores the abort and runs until the worker
|
||||
* wedges (the 98-waiting-0-active incident on 2026-04-24).
|
||||
*/
|
||||
signal?: AbortSignal;
|
||||
}
|
||||
|
||||
// ─── Lock primitives ───────────────────────────────────────────────
|
||||
@@ -352,20 +344,6 @@ async function safeYield(hook?: () => Promise<void>) {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Check if the abort signal has fired. Called between phases so that a
|
||||
* timed-out Minions job bails promptly instead of grinding through all
|
||||
* remaining phases while the worker thinks it's still at capacity.
|
||||
*/
|
||||
function checkAborted(signal?: AbortSignal): void {
|
||||
if (signal?.aborted) {
|
||||
const reason = signal.reason instanceof Error
|
||||
? signal.reason.message
|
||||
: String(signal.reason || 'aborted');
|
||||
throw new Error(`[cycle] aborted between phases: ${reason}`);
|
||||
}
|
||||
}
|
||||
|
||||
// ─── Phase runners ─────────────────────────────────────────────────
|
||||
|
||||
async function runPhaseLint(brainDir: string, dryRun: boolean): Promise<PhaseResult> {
|
||||
@@ -438,55 +416,19 @@ async function runPhaseBacklinks(brainDir: string, dryRun: boolean): Promise<Pha
|
||||
}
|
||||
}
|
||||
|
||||
/** Extended sync result that also carries the changed slug list for downstream phases. */
|
||||
interface SyncPhaseResult extends PhaseResult {
|
||||
/** Slugs that sync added or modified. Used by extract for incremental processing. */
|
||||
pagesAffected?: string[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve the source id for a brain directory by looking up the sources
|
||||
* table. Returns undefined when no registered source matches (falls back
|
||||
* to pre-v0.18 global config.sync.* keys).
|
||||
*/
|
||||
async function resolveSourceForDir(
|
||||
engine: BrainEngine,
|
||||
brainDir: string,
|
||||
): Promise<string | undefined> {
|
||||
try {
|
||||
const rows = await engine.executeRaw<{ id: string }>(
|
||||
`SELECT id FROM sources WHERE local_path = $1 LIMIT 1`,
|
||||
[brainDir],
|
||||
);
|
||||
return rows[0]?.id;
|
||||
} catch {
|
||||
// sources table might not exist on very old brains — fall through.
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
async function runPhaseSync(
|
||||
engine: BrainEngine,
|
||||
brainDir: string,
|
||||
dryRun: boolean,
|
||||
pull: boolean,
|
||||
willRunExtractPhase: boolean,
|
||||
): Promise<SyncPhaseResult> {
|
||||
): Promise<PhaseResult> {
|
||||
try {
|
||||
const { performSync } = await import('../commands/sync.ts');
|
||||
// Resolve the per-source id so sync reads source-scoped last_commit
|
||||
// instead of the global config key. The global key can drift out of
|
||||
// git history (force push, GC) causing a full reimport of all files.
|
||||
const sourceId = await resolveSourceForDir(engine, brainDir);
|
||||
const result = await performSync(engine, {
|
||||
repoPath: brainDir,
|
||||
sourceId,
|
||||
dryRun,
|
||||
noPull: !pull,
|
||||
noEmbed: true, // embed is a separate phase
|
||||
noExtract: willRunExtractPhase, // dedupe ONLY when cycle's extract phase will also run.
|
||||
// If extract isn't scheduled (e.g. `gbrain dream --phase sync`),
|
||||
// sync's inline extract still runs to preserve prior behavior.
|
||||
noEmbed: true, // embed is a separate phase
|
||||
});
|
||||
const syncedCount = result.added + result.modified;
|
||||
return {
|
||||
@@ -506,7 +448,6 @@ async function runPhaseSync(
|
||||
syncStatus: result.status,
|
||||
dryRun,
|
||||
},
|
||||
pagesAffected: result.pagesAffected,
|
||||
};
|
||||
} catch (e) {
|
||||
return {
|
||||
@@ -524,7 +465,6 @@ async function runPhaseExtract(
|
||||
engine: BrainEngine,
|
||||
brainDir: string,
|
||||
dryRun: boolean,
|
||||
changedSlugs?: string[],
|
||||
): Promise<PhaseResult> {
|
||||
try {
|
||||
const { runExtractCore } = await import('../commands/extract.ts');
|
||||
@@ -540,29 +480,15 @@ async function runPhaseExtract(
|
||||
details: { dryRun: true, reason: 'no_dry_run_support' },
|
||||
};
|
||||
}
|
||||
// Incremental path: if sync told us which slugs changed, only extract those.
|
||||
// On a 54K-page brain this turns a 10-minute full walk into a sub-second pass.
|
||||
const result = await runExtractCore(engine, {
|
||||
mode: 'all',
|
||||
dir: brainDir,
|
||||
slugs: changedSlugs, // undefined = full walk (first run / manual)
|
||||
});
|
||||
const result = await runExtractCore(engine, { mode: 'all', dir: brainDir });
|
||||
const linksCreated = result?.links_created ?? 0;
|
||||
const timelineCreated = result?.timeline_entries_created ?? 0;
|
||||
const incremental = changedSlugs !== undefined;
|
||||
return {
|
||||
phase: 'extract',
|
||||
status: 'ok',
|
||||
duration_ms: 0,
|
||||
summary: incremental
|
||||
? `${linksCreated} link(s), ${timelineCreated} timeline entries (incremental: ${changedSlugs.length} slugs)`
|
||||
: `${linksCreated} link(s), ${timelineCreated} timeline entries`,
|
||||
details: {
|
||||
linksCreated, timelineCreated,
|
||||
pages_processed: result?.pages_processed ?? 0,
|
||||
incremental,
|
||||
...(incremental ? { slugs_targeted: changedSlugs.length } : {}),
|
||||
},
|
||||
summary: `${linksCreated} link(s), ${timelineCreated} timeline entries`,
|
||||
details: { linksCreated, timelineCreated, pages_processed: result?.pages_processed ?? 0 },
|
||||
};
|
||||
} catch (e) {
|
||||
return {
|
||||
@@ -718,7 +644,6 @@ export async function runCycle(
|
||||
try {
|
||||
// ── Phase 1: lint ────────────────────────────────────────────
|
||||
if (phases.includes('lint')) {
|
||||
checkAborted(opts.signal);
|
||||
progress.start('cycle.lint');
|
||||
const { result, duration_ms } = await timePhase(() => runPhaseLint(opts.brainDir, dryRun));
|
||||
result.duration_ms = duration_ms;
|
||||
@@ -729,7 +654,6 @@ export async function runCycle(
|
||||
|
||||
// ── Phase 2: backlinks ──────────────────────────────────────
|
||||
if (phases.includes('backlinks')) {
|
||||
checkAborted(opts.signal);
|
||||
progress.start('cycle.backlinks');
|
||||
const { result, duration_ms } = await timePhase(() => runPhaseBacklinks(opts.brainDir, dryRun));
|
||||
result.duration_ms = duration_ms;
|
||||
@@ -739,10 +663,7 @@ export async function runCycle(
|
||||
}
|
||||
|
||||
// ── Phase 3: sync ───────────────────────────────────────────
|
||||
// Track which slugs sync touched so extract can run incrementally.
|
||||
let syncPagesAffected: string[] | undefined;
|
||||
if (phases.includes('sync')) {
|
||||
checkAborted(opts.signal);
|
||||
if (!engine) {
|
||||
phaseResults.push({
|
||||
phase: 'sync',
|
||||
@@ -753,10 +674,8 @@ export async function runCycle(
|
||||
});
|
||||
} else {
|
||||
progress.start('cycle.sync');
|
||||
const { result, duration_ms } = await timePhase(() => runPhaseSync(engine, opts.brainDir, dryRun, pull, phases.includes('extract')));
|
||||
const { result, duration_ms } = await timePhase(() => runPhaseSync(engine, opts.brainDir, dryRun, pull));
|
||||
result.duration_ms = duration_ms;
|
||||
// Capture changed slugs for incremental extract.
|
||||
syncPagesAffected = (result as SyncPhaseResult).pagesAffected;
|
||||
phaseResults.push(result);
|
||||
progress.finish();
|
||||
}
|
||||
@@ -765,7 +684,6 @@ export async function runCycle(
|
||||
|
||||
// ── Phase 4: extract ────────────────────────────────────────
|
||||
if (phases.includes('extract')) {
|
||||
checkAborted(opts.signal);
|
||||
if (!engine) {
|
||||
phaseResults.push({
|
||||
phase: 'extract',
|
||||
@@ -775,11 +693,8 @@ export async function runCycle(
|
||||
details: { reason: 'no_database' },
|
||||
});
|
||||
} else {
|
||||
// Pass changed slugs from sync for incremental extract.
|
||||
// If sync didn't run (phases exclude it) or failed, syncPagesAffected
|
||||
// is undefined → extract falls back to full walk (safe default).
|
||||
progress.start('cycle.extract');
|
||||
const { result, duration_ms } = await timePhase(() => runPhaseExtract(engine, opts.brainDir, dryRun, syncPagesAffected));
|
||||
const { result, duration_ms } = await timePhase(() => runPhaseExtract(engine, opts.brainDir, dryRun));
|
||||
result.duration_ms = duration_ms;
|
||||
phaseResults.push(result);
|
||||
progress.finish();
|
||||
@@ -789,7 +704,6 @@ export async function runCycle(
|
||||
|
||||
// ── Phase 5: embed ──────────────────────────────────────────
|
||||
if (phases.includes('embed')) {
|
||||
checkAborted(opts.signal);
|
||||
if (!engine) {
|
||||
phaseResults.push({
|
||||
phase: 'embed',
|
||||
@@ -810,7 +724,6 @@ export async function runCycle(
|
||||
|
||||
// ── Phase 6: orphans ────────────────────────────────────────
|
||||
if (phases.includes('orphans')) {
|
||||
checkAborted(opts.signal);
|
||||
if (!engine) {
|
||||
phaseResults.push({
|
||||
phase: 'orphans',
|
||||
|
||||
@@ -1,140 +0,0 @@
|
||||
/**
|
||||
* Generic DB-backed lock primitive.
|
||||
*
|
||||
* Reuses the gbrain_cycle_locks table (id PK + holder_pid + ttl_expires_at)
|
||||
* with a parameterized lock id. Both `gbrain-cycle` (the broad cycle lock)
|
||||
* and `gbrain-sync` (performSync's writer lock) live here.
|
||||
*
|
||||
* Why not pg_advisory_xact_lock: it is session-scoped, and PgBouncer
|
||||
* transaction pooling drops session state between calls. This row-based
|
||||
* lock survives PgBouncer because it's plain INSERT/UPDATE/DELETE with
|
||||
* a TTL fallback (a crashed holder's row times out).
|
||||
*
|
||||
* Why a separate table-row per lock id rather than reusing the cycle lock:
|
||||
* the cycle lock is broader (covers every phase). performSync's write-window
|
||||
* is narrower. If performSync reused the cycle lock and the cycle handler
|
||||
* called performSync, the inner acquire would deadlock against itself. Two
|
||||
* lock ids let callers nest cleanly: cycle holds gbrain-cycle for its run;
|
||||
* performSync (called from anywhere — cycle, jobs handler, CLI) takes
|
||||
* gbrain-sync just for the write window.
|
||||
*
|
||||
* v0.22.13 — added in PR #490 to fix CODEX-2 (no cross-process lock for
|
||||
* direct sync paths). The cycle path was already protected.
|
||||
*/
|
||||
import { hostname } from 'os';
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
|
||||
export interface DbLockHandle {
|
||||
id: string;
|
||||
release: () => Promise<void>;
|
||||
refresh: () => Promise<void>;
|
||||
}
|
||||
|
||||
/** Default TTL: 30 minutes, same as cycle lock. */
|
||||
const DEFAULT_TTL_MINUTES = 30;
|
||||
|
||||
/**
|
||||
* Try to acquire a named DB lock.
|
||||
*
|
||||
* Returns a handle on success. Returns `null` if another live holder has
|
||||
* the lock (its row exists and ttl_expires_at is in the future).
|
||||
*
|
||||
* The acquire is upsert-style:
|
||||
* INSERT ... ON CONFLICT (id) DO UPDATE
|
||||
* ... WHERE existing.ttl_expires_at < NOW()
|
||||
* RETURNING id
|
||||
*
|
||||
* Empty RETURNING means the existing row is still live. An expired holder
|
||||
* (worker crashed without releasing) is auto-superseded by the UPDATE
|
||||
* branch.
|
||||
*/
|
||||
export async function tryAcquireDbLock(
|
||||
engine: BrainEngine,
|
||||
lockId: string,
|
||||
ttlMinutes: number = DEFAULT_TTL_MINUTES,
|
||||
): Promise<DbLockHandle | null> {
|
||||
const pid = process.pid;
|
||||
const host = hostname();
|
||||
|
||||
// Engine-agnostic: prefer the engine's raw escape hatch (`sql` for postgres-js,
|
||||
// `db.query` for PGLite). Mirrors cycle.ts's pattern so behavior stays identical.
|
||||
const maybePG = engine as unknown as { sql?: (...args: unknown[]) => Promise<unknown> };
|
||||
const maybePGLite = engine as unknown as {
|
||||
db?: { query: (sql: string, params?: unknown[]) => Promise<{ rows: unknown[] }> };
|
||||
};
|
||||
|
||||
if (engine.kind === 'postgres' && maybePG.sql) {
|
||||
const sql = maybePG.sql as any;
|
||||
const ttl = `${ttlMinutes} minutes`;
|
||||
const rows: Array<{ id: string }> = await sql`
|
||||
INSERT INTO gbrain_cycle_locks (id, holder_pid, holder_host, acquired_at, ttl_expires_at)
|
||||
VALUES (${lockId}, ${pid}, ${host}, NOW(), NOW() + ${ttl}::interval)
|
||||
ON CONFLICT (id) DO UPDATE
|
||||
SET holder_pid = ${pid},
|
||||
holder_host = ${host},
|
||||
acquired_at = NOW(),
|
||||
ttl_expires_at = NOW() + ${ttl}::interval
|
||||
WHERE gbrain_cycle_locks.ttl_expires_at < NOW()
|
||||
RETURNING id
|
||||
`;
|
||||
if (rows.length === 0) return null;
|
||||
return {
|
||||
id: lockId,
|
||||
refresh: async () => {
|
||||
await sql`
|
||||
UPDATE gbrain_cycle_locks
|
||||
SET ttl_expires_at = NOW() + ${ttl}::interval
|
||||
WHERE id = ${lockId} AND holder_pid = ${pid}
|
||||
`;
|
||||
},
|
||||
release: async () => {
|
||||
await sql`
|
||||
DELETE FROM gbrain_cycle_locks
|
||||
WHERE id = ${lockId} AND holder_pid = ${pid}
|
||||
`;
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
if (engine.kind === 'pglite' && maybePGLite.db) {
|
||||
const db = maybePGLite.db;
|
||||
const ttl = `${ttlMinutes} minutes`;
|
||||
const { rows } = await db.query(
|
||||
`INSERT INTO gbrain_cycle_locks (id, holder_pid, holder_host, acquired_at, ttl_expires_at)
|
||||
VALUES ($1, $2, $3, NOW(), NOW() + $4::interval)
|
||||
ON CONFLICT (id) DO UPDATE
|
||||
SET holder_pid = $2,
|
||||
holder_host = $3,
|
||||
acquired_at = NOW(),
|
||||
ttl_expires_at = NOW() + $4::interval
|
||||
WHERE gbrain_cycle_locks.ttl_expires_at < NOW()
|
||||
RETURNING id`,
|
||||
[lockId, pid, host, ttl],
|
||||
);
|
||||
if (rows.length === 0) return null;
|
||||
return {
|
||||
id: lockId,
|
||||
refresh: async () => {
|
||||
await db.query(
|
||||
`UPDATE gbrain_cycle_locks
|
||||
SET ttl_expires_at = NOW() + $1::interval
|
||||
WHERE id = $2 AND holder_pid = $3`,
|
||||
[ttl, lockId, pid],
|
||||
);
|
||||
},
|
||||
release: async () => {
|
||||
await db.query(
|
||||
`DELETE FROM gbrain_cycle_locks WHERE id = $1 AND holder_pid = $2`,
|
||||
[lockId, pid],
|
||||
);
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
throw new Error(`Unknown engine kind for db-lock: ${engine.kind}`);
|
||||
}
|
||||
|
||||
/** Lock id for performSync's writer window. Distinct from gbrain-cycle so the
|
||||
* cycle handler can hold gbrain-cycle while performSync (called from inside
|
||||
* the cycle) acquires gbrain-sync. */
|
||||
export const SYNC_LOCK_ID = 'gbrain-sync';
|
||||
+17
-130
@@ -1,8 +1,6 @@
|
||||
import postgres from 'postgres';
|
||||
import { GBrainError, type EngineConfig } from './types.ts';
|
||||
import { SCHEMA_SQL } from './schema-embedded.ts';
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
import { verifySchema } from './schema-verify.ts';
|
||||
|
||||
let sql: ReturnType<typeof postgres> | null = null;
|
||||
let connectedUrl: string | null = null;
|
||||
@@ -74,78 +72,26 @@ export function resolvePoolSize(explicit?: number): number {
|
||||
}
|
||||
|
||||
/**
|
||||
* Session-level GUCs applied to every new backend connection. Prevents
|
||||
* orphan pgbouncer sessions from holding locks or running queries
|
||||
* indefinitely when the postgres.js client disconnects mid-transaction
|
||||
* (typical cause: autopilot SIGKILL'd by launchd, worker crash-loop,
|
||||
* or transient network drop).
|
||||
* Apply session-level defaults to a fresh connection. Called from both
|
||||
* the module-level `connect()` singleton and the PostgresEngine
|
||||
* instance-level pool so the idle-in-transaction session timeout is set
|
||||
* uniformly.
|
||||
*
|
||||
* Observed failure mode these prevent: a single autopilot UPDATE on
|
||||
* `minion_jobs.lock_until` left a pooler backend in `state='active'`
|
||||
* / `wait_event='ClientRead'` for 24h+, holding a RowExclusiveLock
|
||||
* that blocked every subsequent `ALTER TABLE minion_jobs ...`.
|
||||
* `idle_in_transaction_session_timeout = 5 min` was the v0.18.0 field
|
||||
* report's headline production issue: a 24-hour idle connection was
|
||||
* holding a lock on `pages` and blocking all DDL. 5 minutes is generous
|
||||
* for any legitimate transaction but catches crashed writers. The GUC
|
||||
* is session-scoped (safe for shared pools — no cross-statement leak).
|
||||
*
|
||||
* Defaults are conservative (chosen not to interfere with bulk work
|
||||
* like long-running embed passes or CREATE INDEX on large tables):
|
||||
* - statement_timeout = '5min'
|
||||
* - idle_in_transaction_session_timeout = '5min' (matches v0.18.0
|
||||
* posture; #363's original 2min default was tightened to 5min on
|
||||
* merge with v0.21.0's setSessionDefaults to avoid regressing
|
||||
* long-running embed passes)
|
||||
*
|
||||
* Override per-GUC with env vars:
|
||||
* - GBRAIN_STATEMENT_TIMEOUT
|
||||
* - GBRAIN_IDLE_TX_TIMEOUT
|
||||
* - GBRAIN_CLIENT_CHECK_INTERVAL (Postgres 14+; empty default - opt-in
|
||||
* only since older self-hosted Postgres rejects this startup param)
|
||||
*
|
||||
* Set any env var to '0' or 'off' to disable that GUC entirely.
|
||||
*
|
||||
* Delivered via postgres.js's `connection` option, which sends these as
|
||||
* startup parameters in the initial connection packet. Works correctly
|
||||
* with PgBouncer session mode AND transaction mode: startup parameters
|
||||
* pass through to the backend on connection creation and persist for the
|
||||
* backend's lifetime (unlike `SET` commands which transaction-mode
|
||||
* PgBouncer strips between transactions).
|
||||
*
|
||||
* Supersedes the v0.21.0 `setSessionDefaults(sql)` helper, which used
|
||||
* a post-pool `SET` command. That approach is unreliable in PgBouncer
|
||||
* transaction mode (transaction-mode poolers strip session-state SETs
|
||||
* between transactions); startup parameters are durable.
|
||||
* Wrapped in try/catch because some managed Postgres tenants restrict
|
||||
* SET on the GUC; non-fatal if it fails.
|
||||
*/
|
||||
const DEFAULT_STATEMENT_TIMEOUT = '5min';
|
||||
const DEFAULT_IDLE_TX_TIMEOUT = '5min';
|
||||
|
||||
export function resolveSessionTimeouts(): Record<string, string> {
|
||||
const out: Record<string, string> = {};
|
||||
const add = (envKey: string, gucKey: string, defaultVal: string) => {
|
||||
const raw = process.env[envKey];
|
||||
if (raw === '0' || raw === 'off') return; // explicitly disabled
|
||||
const val = raw ?? defaultVal;
|
||||
if (val) out[gucKey] = val;
|
||||
};
|
||||
add('GBRAIN_STATEMENT_TIMEOUT', 'statement_timeout', DEFAULT_STATEMENT_TIMEOUT);
|
||||
add('GBRAIN_IDLE_TX_TIMEOUT', 'idle_in_transaction_session_timeout', DEFAULT_IDLE_TX_TIMEOUT);
|
||||
// client_connection_check_interval is opt-in: Postgres 14+ only, and some
|
||||
// managed pooler tiers reject unknown startup parameters. Users can enable
|
||||
// it explicitly once they know their Postgres version supports it.
|
||||
add('GBRAIN_CLIENT_CHECK_INTERVAL', 'client_connection_check_interval', '');
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* Backward-compat shim for v0.21.0's `setSessionDefaults` callers.
|
||||
* The current implementation no-ops because session timeouts are now
|
||||
* applied at connection-startup time via `resolveSessionTimeouts()` +
|
||||
* postgres.js's `connection` option (more durable across PgBouncer
|
||||
* transaction mode).
|
||||
*
|
||||
* Kept as a callable function so existing call sites in `connect()` and
|
||||
* `PostgresEngine.connect()` don't need to be touched on the merge —
|
||||
* the work has already happened by the time this function would run.
|
||||
*/
|
||||
export async function setSessionDefaults(_sql: ReturnType<typeof postgres>): Promise<void> {
|
||||
// No-op: timeouts are now applied as startup parameters in resolveSessionTimeouts().
|
||||
export async function setSessionDefaults(sql: ReturnType<typeof postgres>): Promise<void> {
|
||||
try {
|
||||
await sql`SET idle_in_transaction_session_timeout = '300000'`;
|
||||
} catch {
|
||||
// Non-fatal: some managed Postgres may restrict this GUC
|
||||
}
|
||||
}
|
||||
|
||||
export function getConnection(): ReturnType<typeof postgres> {
|
||||
@@ -179,7 +125,6 @@ export async function connect(config: EngineConfig): Promise<void> {
|
||||
|
||||
try {
|
||||
const prepare = resolvePrepare(url);
|
||||
const timeouts = resolveSessionTimeouts();
|
||||
const opts: Record<string, unknown> = {
|
||||
max: resolvePoolSize(),
|
||||
idle_timeout: 20,
|
||||
@@ -189,9 +134,6 @@ export async function connect(config: EngineConfig): Promise<void> {
|
||||
bigint: postgres.BigInt,
|
||||
},
|
||||
};
|
||||
if (Object.keys(timeouts).length > 0) {
|
||||
opts.connection = timeouts;
|
||||
}
|
||||
if (typeof prepare === 'boolean') {
|
||||
opts.prepare = prepare;
|
||||
if (!prepare) {
|
||||
@@ -238,64 +180,9 @@ export async function initSchema(): Promise<void> {
|
||||
}
|
||||
}
|
||||
|
||||
export { verifySchema } from './schema-verify.ts';
|
||||
|
||||
export async function withTransaction<T>(fn: (tx: ReturnType<typeof postgres>) => Promise<T>): Promise<T> {
|
||||
const conn = getConnection();
|
||||
return conn.begin(async (tx) => {
|
||||
return fn(tx as unknown as ReturnType<typeof postgres>);
|
||||
}) as Promise<T>;
|
||||
}
|
||||
|
||||
const RETRYABLE_DB_CONNECT_PATTERNS = [
|
||||
/password authentication failed/i,
|
||||
/connection refused/i,
|
||||
/the database system is starting up/i,
|
||||
/Connection terminated unexpectedly/i,
|
||||
/ECONNRESET/i,
|
||||
];
|
||||
|
||||
export function isRetryableDbConnectError(err: unknown): boolean {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
if (!msg) return false;
|
||||
return RETRYABLE_DB_CONNECT_PATTERNS.some(p => p.test(msg));
|
||||
}
|
||||
|
||||
export interface ConnectWithRetryOpts {
|
||||
attempts?: number;
|
||||
baseDelayMs?: number;
|
||||
noRetry?: boolean;
|
||||
log?: (line: string) => void;
|
||||
}
|
||||
|
||||
export async function connectWithRetry(
|
||||
engine: BrainEngine,
|
||||
config: EngineConfig & { poolSize?: number },
|
||||
opts: ConnectWithRetryOpts = {},
|
||||
): Promise<void> {
|
||||
const noRetry = opts.noRetry ?? (process.env.GBRAIN_NO_RETRY_CONNECT === '1');
|
||||
const attempts = noRetry ? 1 : (opts.attempts ?? 3);
|
||||
const baseDelayMs = opts.baseDelayMs ?? 1000;
|
||||
const log = opts.log ?? ((line) => console.warn(line));
|
||||
|
||||
let lastErr: unknown;
|
||||
for (let i = 0; i < attempts; i++) {
|
||||
try {
|
||||
await engine.connect(config);
|
||||
return;
|
||||
} catch (e: unknown) {
|
||||
lastErr = e;
|
||||
const retryable = isRetryableDbConnectError(e);
|
||||
const isLast = i === attempts - 1;
|
||||
if (!retryable || isLast) {
|
||||
throw e;
|
||||
}
|
||||
const delay = baseDelayMs * Math.pow(2, i);
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
log(`[connect] attempt ${i + 1} failed (${msg.slice(0, 80)}), retrying in ${delay}ms`);
|
||||
await new Promise(resolve => setTimeout(resolve, delay));
|
||||
}
|
||||
}
|
||||
// Unreachable, but TS needs the throw.
|
||||
throw lastErr;
|
||||
}
|
||||
|
||||
@@ -1,83 +0,0 @@
|
||||
/**
|
||||
* Recursive filesystem walk into a slug → Stats map.
|
||||
*
|
||||
* Replaces per-page `existsSync` + `statSync` syscall storms (Issue #14 of
|
||||
* the v0.22.3 eng review). On a 200K-page brain the per-page approach was
|
||||
* 400K syscalls in a synchronous loop; this walk is one syscall per directory
|
||||
* plus one stat per file, then O(1) Map lookups for everything downstream.
|
||||
*
|
||||
* The slug key is the on-disk path relative to the brain repo, with the
|
||||
* trailing `.md` stripped, matching how pages are stored: `people/alice.md`
|
||||
* on disk becomes `people/alice` as a slug.
|
||||
*
|
||||
* Skipped entries:
|
||||
* - `.git/`, `node_modules/`, and dot-directories generally — not part of
|
||||
* the brain's page namespace. Speeds up walks significantly on dirty
|
||||
* working copies.
|
||||
* - Files that don't end in `.md`. Sidecar JSON, raw binary attachments,
|
||||
* etc. are tracked by the brain but not via slugs.
|
||||
*/
|
||||
|
||||
import { readdirSync, statSync, type Stats, type Dirent } from 'fs';
|
||||
import { join } from 'path';
|
||||
|
||||
export interface DiskFileEntry {
|
||||
size: number;
|
||||
mtimeMs: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Walk `repoPath` and return a Map of slug → file metadata for every `.md`
|
||||
* file. Skips dot-directories. Synchronous (matches the call-site shape and
|
||||
* the io pattern of stat-heavy scans).
|
||||
*
|
||||
* @param repoPath Absolute path to the brain repo root.
|
||||
* @returns Map keyed by slug (no `.md` suffix). Empty map if repoPath
|
||||
* doesn't exist or contains no markdown files.
|
||||
*/
|
||||
export function walkBrainRepo(repoPath: string): Map<string, DiskFileEntry> {
|
||||
const result = new Map<string, DiskFileEntry>();
|
||||
|
||||
function recurse(dirPath: string, slugPrefix: string): void {
|
||||
// Annotate as Dirent[] explicitly: ReturnType<typeof readdirSync> with
|
||||
// withFileTypes:true picks an overload union that includes
|
||||
// Dirent<Buffer<ArrayBufferLike>>, which makes entry.name a Buffer in
|
||||
// strict tsc mode. Cast to the string-based Dirent[] (same shape sync.ts
|
||||
// uses for its own filesystem walk).
|
||||
let entries: Dirent[];
|
||||
try {
|
||||
entries = readdirSync(dirPath, { withFileTypes: true }) as unknown as Dirent[];
|
||||
} catch {
|
||||
return; // unreadable directory — skip silently
|
||||
}
|
||||
|
||||
for (const entry of entries) {
|
||||
// Skip dot-directories (.git, .gbrain, .vscode, etc) and node_modules.
|
||||
if (entry.name.startsWith('.') || entry.name === 'node_modules') continue;
|
||||
const childPath = join(dirPath, entry.name);
|
||||
|
||||
if (entry.isDirectory()) {
|
||||
recurse(childPath, slugPrefix ? `${slugPrefix}/${entry.name}` : entry.name);
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!entry.isFile()) continue;
|
||||
if (!entry.name.endsWith('.md')) continue;
|
||||
|
||||
let stats: Stats;
|
||||
try {
|
||||
stats = statSync(childPath);
|
||||
} catch {
|
||||
continue; // race: file deleted between readdir and stat
|
||||
}
|
||||
|
||||
const slug = slugPrefix
|
||||
? `${slugPrefix}/${entry.name.slice(0, -3)}`
|
||||
: entry.name.slice(0, -3);
|
||||
result.set(slug, { size: stats.size, mtimeMs: stats.mtimeMs });
|
||||
}
|
||||
}
|
||||
|
||||
recurse(repoPath, '');
|
||||
return result;
|
||||
}
|
||||
+1
-16
@@ -1,6 +1,6 @@
|
||||
import type {
|
||||
Page, PageInput, PageFilters,
|
||||
Chunk, ChunkInput, StaleChunkRow,
|
||||
Chunk, ChunkInput,
|
||||
SearchResult, SearchOpts,
|
||||
Link, GraphNode, GraphPath,
|
||||
TimelineEntry, TimelineInput, TimelineOpts,
|
||||
@@ -133,21 +133,6 @@ export interface BrainEngine {
|
||||
// Chunks
|
||||
upsertChunks(slug: string, chunks: ChunkInput[]): Promise<void>;
|
||||
getChunks(slug: string): Promise<Chunk[]>;
|
||||
/**
|
||||
* Count chunks across the entire brain where embedded_at IS NULL.
|
||||
* Pre-flight short-circuit for `embed --stale` so a 100%-embedded brain
|
||||
* does no further work after a single SELECT count(*) (~50 bytes wire).
|
||||
*/
|
||||
countStaleChunks(): Promise<number>;
|
||||
/**
|
||||
* Return every chunk where embedded_at IS NULL, with the metadata needed
|
||||
* to call embedBatch + upsertChunks. The `embedding` column is omitted
|
||||
* by design — stale rows have NULL embeddings, so shipping them wastes
|
||||
* wire bytes for no gain. Caller groups by slug, embeds, and re-upserts.
|
||||
*
|
||||
* Bounded by an internal LIMIT of 100000 to mirror listPages.
|
||||
*/
|
||||
listStaleChunks(): Promise<StaleChunkRow[]>;
|
||||
deleteChunks(slug: string): Promise<void>;
|
||||
|
||||
// Links
|
||||
|
||||
+6
-201
@@ -2,29 +2,6 @@ import matter from 'gray-matter';
|
||||
import type { PageType } from './types.ts';
|
||||
import { slugifyPath } from './sync.ts';
|
||||
|
||||
export type ParseValidationCode =
|
||||
| 'MISSING_OPEN'
|
||||
| 'MISSING_CLOSE'
|
||||
| 'YAML_PARSE'
|
||||
| 'SLUG_MISMATCH'
|
||||
| 'NULL_BYTES'
|
||||
| 'NESTED_QUOTES'
|
||||
| 'EMPTY_FRONTMATTER';
|
||||
|
||||
export interface ParseValidationError {
|
||||
code: ParseValidationCode;
|
||||
message: string;
|
||||
line?: number;
|
||||
}
|
||||
|
||||
export interface ParseOpts {
|
||||
/** When true, errors[] is populated. Existing callers unaffected. */
|
||||
validate?: boolean;
|
||||
/** When validate is true and frontmatter has a `slug:` field that doesn't
|
||||
* match expectedSlug, emits SLUG_MISMATCH. */
|
||||
expectedSlug?: string;
|
||||
}
|
||||
|
||||
export interface ParsedMarkdown {
|
||||
frontmatter: Record<string, unknown>;
|
||||
compiled_truth: string;
|
||||
@@ -33,8 +10,6 @@ export interface ParsedMarkdown {
|
||||
type: PageType;
|
||||
title: string;
|
||||
tags: string[];
|
||||
/** Present iff opts.validate. Empty array means no errors. */
|
||||
errors?: ParseValidationError[];
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -58,53 +33,26 @@ export interface ParsedMarkdown {
|
||||
* heading (backward-compat for existing files). A bare `---` in body text
|
||||
* is treated as a markdown horizontal rule, not a timeline separator.
|
||||
*/
|
||||
export function parseMarkdown(
|
||||
content: string,
|
||||
filePath?: string,
|
||||
opts?: ParseOpts,
|
||||
): ParsedMarkdown {
|
||||
const errors: ParseValidationError[] = [];
|
||||
|
||||
// gray-matter is forgiving: it returns empty data + original content for
|
||||
// pretty much any input. The validation surface below catches the cases
|
||||
// it silently swallows. Validation only runs when opts.validate is true,
|
||||
// so existing callers are unaffected.
|
||||
let parsed: ReturnType<typeof matter> | null = null;
|
||||
let yamlParseError: Error | null = null;
|
||||
try {
|
||||
parsed = matter(content);
|
||||
} catch (e) {
|
||||
yamlParseError = e as Error;
|
||||
}
|
||||
|
||||
if (opts?.validate) {
|
||||
collectValidationErrors(content, errors, {
|
||||
yamlParseError,
|
||||
expectedSlug: opts.expectedSlug,
|
||||
parsedFrontmatter: parsed?.data ?? {},
|
||||
});
|
||||
}
|
||||
|
||||
// When YAML parsing failed (rare; gray-matter is forgiving), fall back to
|
||||
// empty frontmatter + raw content as the body so non-validate callers still
|
||||
// get a usable shape.
|
||||
const frontmatter = (parsed?.data ?? {}) as Record<string, unknown>;
|
||||
const body = parsed?.content ?? content;
|
||||
export function parseMarkdown(content: string, filePath?: string): ParsedMarkdown {
|
||||
const { data: frontmatter, content: body } = matter(content);
|
||||
|
||||
// Split body at first standalone ---
|
||||
const { compiled_truth, timeline } = splitBody(body);
|
||||
|
||||
// Extract metadata from frontmatter
|
||||
const type = (frontmatter.type as PageType) || inferType(filePath);
|
||||
const title = (frontmatter.title as string) || inferTitle(filePath);
|
||||
const tags = extractTags(frontmatter);
|
||||
const slug = (frontmatter.slug as string) || inferSlug(filePath);
|
||||
|
||||
// Remove processed fields from frontmatter (they're stored as columns)
|
||||
const cleanFrontmatter = { ...frontmatter };
|
||||
delete cleanFrontmatter.type;
|
||||
delete cleanFrontmatter.title;
|
||||
delete cleanFrontmatter.tags;
|
||||
delete cleanFrontmatter.slug;
|
||||
|
||||
const result: ParsedMarkdown = {
|
||||
return {
|
||||
frontmatter: cleanFrontmatter,
|
||||
compiled_truth: compiled_truth.trim(),
|
||||
timeline: timeline.trim(),
|
||||
@@ -113,149 +61,6 @@ export function parseMarkdown(
|
||||
title,
|
||||
tags,
|
||||
};
|
||||
if (opts?.validate) result.errors = errors;
|
||||
return result;
|
||||
}
|
||||
|
||||
/**
|
||||
* Inspect raw content for the 7 frontmatter validation classes that gray-matter
|
||||
* silently accepts. Mutates `errors` in place. The order of checks is
|
||||
* deliberate: cheap byte-level checks first, then structural checks, then
|
||||
* YAML-parse-dependent checks.
|
||||
*/
|
||||
function collectValidationErrors(
|
||||
content: string,
|
||||
errors: ParseValidationError[],
|
||||
ctx: {
|
||||
yamlParseError: Error | null;
|
||||
expectedSlug?: string;
|
||||
parsedFrontmatter: Record<string, unknown>;
|
||||
},
|
||||
): void {
|
||||
// 1. NULL_BYTES — binary corruption indicator.
|
||||
const nullIdx = content.indexOf('\x00');
|
||||
if (nullIdx >= 0) {
|
||||
const line = content.slice(0, nullIdx).split('\n').length;
|
||||
errors.push({
|
||||
code: 'NULL_BYTES',
|
||||
message: 'Content contains null bytes (likely binary corruption)',
|
||||
line,
|
||||
});
|
||||
}
|
||||
|
||||
// 2. MISSING_OPEN — first non-empty line must be `---`.
|
||||
const lines = content.split('\n');
|
||||
let firstNonEmpty = -1;
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
if (lines[i].trim().length > 0) {
|
||||
firstNonEmpty = i;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (firstNonEmpty === -1) {
|
||||
// Empty file: treat as MISSING_OPEN. Don't run other structural checks.
|
||||
errors.push({
|
||||
code: 'MISSING_OPEN',
|
||||
message: 'File is empty or whitespace-only; expected frontmatter starting with ---',
|
||||
line: 1,
|
||||
});
|
||||
return;
|
||||
}
|
||||
if (lines[firstNonEmpty].trim() !== '---') {
|
||||
errors.push({
|
||||
code: 'MISSING_OPEN',
|
||||
message: 'Frontmatter must start with --- on the first non-empty line',
|
||||
line: firstNonEmpty + 1,
|
||||
});
|
||||
// Without an opener we can't reason about MISSING_CLOSE / EMPTY_FRONTMATTER
|
||||
// / NESTED_QUOTES inside frontmatter. Stop structural checks here.
|
||||
return;
|
||||
}
|
||||
|
||||
// 3. MISSING_CLOSE — find the next `---` after the opener. If a markdown
|
||||
// heading appears before it, that's a strong signal the closing
|
||||
// delimiter is missing (the heading was meant to be in the body).
|
||||
let closeLine = -1;
|
||||
let headingBeforeClose = -1;
|
||||
for (let i = firstNonEmpty + 1; i < lines.length; i++) {
|
||||
const t = lines[i].trim();
|
||||
if (t === '---') {
|
||||
closeLine = i;
|
||||
break;
|
||||
}
|
||||
if (/^#{1,6}\s/.test(t) && headingBeforeClose === -1) {
|
||||
headingBeforeClose = i;
|
||||
}
|
||||
}
|
||||
if (closeLine === -1) {
|
||||
errors.push({
|
||||
code: 'MISSING_CLOSE',
|
||||
message:
|
||||
headingBeforeClose >= 0
|
||||
? `No closing --- before heading at line ${headingBeforeClose + 1}`
|
||||
: 'No closing --- delimiter found',
|
||||
line: headingBeforeClose >= 0 ? headingBeforeClose + 1 : firstNonEmpty + 1,
|
||||
});
|
||||
return;
|
||||
}
|
||||
if (headingBeforeClose >= 0 && headingBeforeClose < closeLine) {
|
||||
errors.push({
|
||||
code: 'MISSING_CLOSE',
|
||||
message: `Heading at line ${headingBeforeClose + 1} found inside frontmatter zone (closing --- comes after)`,
|
||||
line: headingBeforeClose + 1,
|
||||
});
|
||||
}
|
||||
|
||||
// 4. EMPTY_FRONTMATTER — open and close present but nothing meaningful between.
|
||||
const fmBody = lines.slice(firstNonEmpty + 1, closeLine).join('\n').trim();
|
||||
if (fmBody.length === 0) {
|
||||
errors.push({
|
||||
code: 'EMPTY_FRONTMATTER',
|
||||
message: 'Frontmatter block is empty',
|
||||
line: firstNonEmpty + 1,
|
||||
});
|
||||
}
|
||||
|
||||
// 5. NESTED_QUOTES — common breakage pattern: `title: "Name "Nick" Last"`.
|
||||
// Detect any frontmatter `key: ...` line whose value contains 3 or more
|
||||
// unescaped double-quote characters. A clean quoted value has 2.
|
||||
for (let i = firstNonEmpty + 1; i < closeLine; i++) {
|
||||
const line = lines[i];
|
||||
const m = line.match(/^\s*[A-Za-z_][\w-]*\s*:\s*(.*)$/);
|
||||
if (!m) continue;
|
||||
const value = m[1];
|
||||
let count = 0;
|
||||
for (let j = 0; j < value.length; j++) {
|
||||
if (value[j] === '"' && (j === 0 || value[j - 1] !== '\\')) count++;
|
||||
}
|
||||
if (count >= 3) {
|
||||
errors.push({
|
||||
code: 'NESTED_QUOTES',
|
||||
message: 'Nested double quotes in YAML value (use single quotes for the outer)',
|
||||
line: i + 1,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// 6. YAML_PARSE — gray-matter threw.
|
||||
if (ctx.yamlParseError) {
|
||||
errors.push({
|
||||
code: 'YAML_PARSE',
|
||||
message: `YAML parse failed: ${ctx.yamlParseError.message}`,
|
||||
line: firstNonEmpty + 1,
|
||||
});
|
||||
}
|
||||
|
||||
// 7. SLUG_MISMATCH — only when expectedSlug was provided and a slug field exists.
|
||||
if (ctx.expectedSlug && typeof ctx.parsedFrontmatter.slug === 'string') {
|
||||
const declared = ctx.parsedFrontmatter.slug as string;
|
||||
if (declared !== ctx.expectedSlug) {
|
||||
errors.push({
|
||||
code: 'SLUG_MISMATCH',
|
||||
message: `Frontmatter slug "${declared}" does not match path-derived slug "${ctx.expectedSlug}"`,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -811,14 +811,6 @@ export const MIGRATIONS: Migration[] = [
|
||||
RAISE NOTICE 'v24: RLS backfill complete (role % has BYPASSRLS)', current_user;
|
||||
END $$;
|
||||
`,
|
||||
// PGLite has no RLS engine and is intrinsically single-tenant (local file).
|
||||
// The 8 ALTER TABLE ... ENABLE ROW LEVEL SECURITY statements above also
|
||||
// target tables that may not exist on PGLite (subagent_*, minion_inbox),
|
||||
// since pglite-schema.ts is the canonical PGLite schema source. No-op
|
||||
// override keeps PGLite upgrades unwedged and the version bump intact.
|
||||
sqlFor: {
|
||||
pglite: '',
|
||||
},
|
||||
},
|
||||
{
|
||||
version: 25,
|
||||
|
||||
@@ -75,9 +75,6 @@ export interface SupervisorOpts {
|
||||
allowShellJobs: boolean;
|
||||
/** JSON mode: emit JSONL events on stderr, reserve stdout for data payloads. Default: false. */
|
||||
json: boolean;
|
||||
/** RSS threshold (MB) passed to the spawned worker as `--max-rss N`.
|
||||
* Default: 2048. Set to 0 to spawn the worker without a watchdog. */
|
||||
maxRssMb: number;
|
||||
/** Optional event sink (Lane C audit writer). Called for every lifecycle event. */
|
||||
onEvent?: (event: SupervisorEmission) => void;
|
||||
/**
|
||||
@@ -104,7 +101,6 @@ const DEFAULTS: Omit<SupervisorOpts, 'cliPath'> = {
|
||||
healthInterval: 60_000,
|
||||
allowShellJobs: false,
|
||||
json: false,
|
||||
maxRssMb: 2048,
|
||||
};
|
||||
|
||||
/** Calculate backoff: 1s, 2s, 4s, 8s, 16s, 32s, 60s cap. */
|
||||
@@ -146,7 +142,6 @@ export class MinionSupervisor {
|
||||
private sigtermListener: (() => void) | null = null;
|
||||
private sigintListener: (() => void) | null = null;
|
||||
private lockAcquired = false;
|
||||
private consecutiveHealthFailures = 0;
|
||||
|
||||
constructor(engine: BrainEngine, opts: Partial<SupervisorOpts> & { cliPath: string }) {
|
||||
this.engine = engine;
|
||||
@@ -415,9 +410,6 @@ export class MinionSupervisor {
|
||||
'--concurrency', String(this.opts.concurrency),
|
||||
'--queue', this.opts.queue,
|
||||
];
|
||||
if (this.opts.maxRssMb > 0) {
|
||||
args.push('--max-rss', String(this.opts.maxRssMb));
|
||||
}
|
||||
|
||||
// Build child env. Explicit handling for GBRAIN_ALLOW_SHELL_JOBS:
|
||||
// inherit only when caller opts in, otherwise strip from the clone.
|
||||
@@ -484,26 +476,10 @@ export class MinionSupervisor {
|
||||
}
|
||||
|
||||
const exitReason = signal ? `signal ${signal}` : `code ${code ?? 'null'}`;
|
||||
|
||||
// Classify the likely cause for easier debugging
|
||||
let likelyCause: string;
|
||||
if (signal === 'SIGKILL') {
|
||||
likelyCause = 'oom_or_external_kill';
|
||||
} else if (signal === 'SIGTERM') {
|
||||
likelyCause = 'graceful_shutdown';
|
||||
} else if (code === 1) {
|
||||
likelyCause = 'runtime_error';
|
||||
} else if (code === 0) {
|
||||
likelyCause = 'clean_exit';
|
||||
} else {
|
||||
likelyCause = 'unknown';
|
||||
}
|
||||
|
||||
this.emit('worker_exited', {
|
||||
code: code ?? null,
|
||||
signal: signal ?? null,
|
||||
reason: exitReason,
|
||||
likely_cause: likelyCause,
|
||||
crash_count: this.crashCount,
|
||||
max_crashes: this.opts.maxCrashes,
|
||||
run_duration_ms: runDuration,
|
||||
@@ -547,9 +523,6 @@ export class MinionSupervisor {
|
||||
[this.opts.queue],
|
||||
);
|
||||
|
||||
// Reset consecutive failure counter on successful health check
|
||||
this.consecutiveHealthFailures = 0;
|
||||
|
||||
const row = rows[0] ?? { stalled: '0', waiting: '0', last_completed: null };
|
||||
const stalledCount = parseInt(row.stalled ?? '0', 10);
|
||||
const waitingCount = parseInt(row.waiting ?? '0', 10);
|
||||
@@ -588,41 +561,11 @@ export class MinionSupervisor {
|
||||
});
|
||||
}
|
||||
} catch (e) {
|
||||
this.consecutiveHealthFailures++;
|
||||
const errMsg = e instanceof Error ? e.message : String(e);
|
||||
|
||||
if (this.consecutiveHealthFailures >= 3) {
|
||||
// DB connection is likely dead. Emit a degraded warning.
|
||||
this.emit('health_warn', {
|
||||
reason: 'db_connection_degraded',
|
||||
consecutive_failures: this.consecutiveHealthFailures,
|
||||
error: errMsg,
|
||||
queue: this.opts.queue,
|
||||
});
|
||||
// Attempt to reconnect the engine if it supports it
|
||||
try {
|
||||
if ('reconnect' in this.engine && typeof (this.engine as Record<string, unknown>).reconnect === 'function') {
|
||||
await (this.engine as unknown as { reconnect(): Promise<void> }).reconnect();
|
||||
this.consecutiveHealthFailures = 0;
|
||||
this.emit('health_warn', {
|
||||
reason: 'db_reconnected',
|
||||
queue: this.opts.queue,
|
||||
});
|
||||
}
|
||||
} catch (reconnErr) {
|
||||
this.emit('health_error', {
|
||||
error: `reconnect failed: ${reconnErr instanceof Error ? reconnErr.message : String(reconnErr)}`,
|
||||
reconnect_failed: true,
|
||||
queue: this.opts.queue,
|
||||
});
|
||||
}
|
||||
} else {
|
||||
// Non-fatal single failure
|
||||
this.emit('health_error', {
|
||||
error: errMsg,
|
||||
queue: this.opts.queue,
|
||||
});
|
||||
}
|
||||
// Health check failures are non-fatal.
|
||||
this.emit('health_error', {
|
||||
error: e instanceof Error ? e.message : String(e),
|
||||
queue: this.opts.queue,
|
||||
});
|
||||
} finally {
|
||||
this.healthInFlight = false;
|
||||
}
|
||||
|
||||
@@ -160,16 +160,6 @@ export interface MinionWorkerOpts {
|
||||
stalledInterval?: number; // ms, default 30000
|
||||
maxStalledCount?: number; // default 1
|
||||
pollInterval?: number; // ms, default 5000 (for PGLite fallback)
|
||||
/** RSS threshold in MB. When exceeded, worker triggers graceful shutdown
|
||||
* so a supervisor can respawn it. 0 or undefined = disabled. */
|
||||
maxRssMb?: number;
|
||||
/** Optional injection point for RSS readback. Defaults to
|
||||
* `() => process.memoryUsage().rss`. Tests inject deterministic sequences. */
|
||||
getRss?: () => number;
|
||||
/** Periodic RSS check interval in ms, default 60000. Catches the freeze
|
||||
* case where all concurrency slots are wedged with zero job completions
|
||||
* so the per-job check never fires. */
|
||||
rssCheckInterval?: number;
|
||||
}
|
||||
|
||||
// --- Job Context (passed to handlers) ---
|
||||
|
||||
@@ -56,11 +56,6 @@ export class MinionWorker {
|
||||
* deploy restart — they still get the full 30s cleanup race instead. */
|
||||
private shutdownAbort = new AbortController();
|
||||
|
||||
/** Cumulative jobs that finished (success or failure). Used in watchdog log lines. */
|
||||
private jobsCompleted = 0;
|
||||
/** Idempotency latch for gracefulShutdown — per-job and periodic check sites can race. */
|
||||
private gracefulShutdownFired = false;
|
||||
|
||||
private opts: Required<MinionWorkerOpts>;
|
||||
|
||||
constructor(
|
||||
@@ -78,9 +73,6 @@ export class MinionWorker {
|
||||
stalledInterval: opts?.stalledInterval ?? 30000,
|
||||
maxStalledCount: opts?.maxStalledCount ?? 1,
|
||||
pollInterval: opts?.pollInterval ?? 5000,
|
||||
maxRssMb: opts?.maxRssMb ?? 0,
|
||||
getRss: opts?.getRss ?? (() => process.memoryUsage().rss),
|
||||
rssCheckInterval: opts?.rssCheckInterval ?? 60000,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -144,17 +136,6 @@ export class MinionWorker {
|
||||
}
|
||||
}, this.opts.stalledInterval);
|
||||
|
||||
// Periodic RSS watchdog — closes the production-freeze regression where
|
||||
// all concurrency slots are wedged with zero job completions, so the
|
||||
// per-job check in executeJob().finally() never fires. Disabled when
|
||||
// maxRssMb is 0 (default for bare `gbrain jobs work`; supervisor sets 2048).
|
||||
let rssTimer: ReturnType<typeof setInterval> | null = null;
|
||||
if (this.opts.maxRssMb > 0) {
|
||||
rssTimer = setInterval(() => {
|
||||
this.checkMemoryLimit('periodic');
|
||||
}, this.opts.rssCheckInterval);
|
||||
}
|
||||
|
||||
try {
|
||||
while (this.running) {
|
||||
// Promote delayed jobs
|
||||
@@ -200,7 +181,6 @@ export class MinionWorker {
|
||||
}
|
||||
} finally {
|
||||
clearInterval(stalledTimer);
|
||||
if (rssTimer) clearInterval(rssTimer);
|
||||
process.removeListener('SIGTERM', shutdown);
|
||||
process.removeListener('SIGINT', shutdown);
|
||||
|
||||
@@ -277,55 +257,6 @@ export class MinionWorker {
|
||||
this.running = false;
|
||||
}
|
||||
|
||||
/** RSS watchdog. Called from the per-job finally and the periodic timer.
|
||||
* Idempotent: returns early if already not running or already shut down.
|
||||
* When threshold is exceeded, hands off to gracefulShutdown(). */
|
||||
private checkMemoryLimit(source: 'post-job' | 'periodic'): void {
|
||||
if (this.opts.maxRssMb <= 0) return;
|
||||
if (!this.running) return;
|
||||
if (this.gracefulShutdownFired) return;
|
||||
|
||||
let rss = 0;
|
||||
try {
|
||||
rss = this.opts.getRss();
|
||||
} catch {
|
||||
// process.memoryUsage() effectively cannot throw, but be safe.
|
||||
return;
|
||||
}
|
||||
const rssMb = Math.round(rss / (1024 * 1024));
|
||||
if (rssMb < this.opts.maxRssMb) return;
|
||||
|
||||
const ts = new Date().toISOString().slice(11, 19);
|
||||
console.warn(
|
||||
`[watchdog ${ts}] rss=${rssMb}MB threshold=${this.opts.maxRssMb}MB ` +
|
||||
`jobs_completed=${this.jobsCompleted} source=${source} — draining`,
|
||||
);
|
||||
this.gracefulShutdown('watchdog');
|
||||
}
|
||||
|
||||
/** Trigger a unified-style graceful shutdown. Fires shutdownAbort + per-job
|
||||
* aborts + running=false in that order so:
|
||||
* 1. Shell handlers (and anything subscribed to ctx.shutdownSignal) start
|
||||
* their cleanup sequence (SIGTERM → 5s grace → SIGKILL on children).
|
||||
* 2. Cooperative handlers see ctx.signal.aborted and bail instead of
|
||||
* waiting out the 30s drain.
|
||||
* 3. Main loop exits at the top of the next iteration.
|
||||
* The existing 30s drain in start()'s finally then backstops genuinely
|
||||
* uninterruptible work. */
|
||||
private gracefulShutdown(reason: string): void {
|
||||
if (this.gracefulShutdownFired) return;
|
||||
this.gracefulShutdownFired = true;
|
||||
if (!this.shutdownAbort.signal.aborted) {
|
||||
this.shutdownAbort.abort(new Error(reason));
|
||||
}
|
||||
for (const entry of this.inFlight.values()) {
|
||||
if (!entry.abort.signal.aborted) {
|
||||
entry.abort.abort(new Error(reason));
|
||||
}
|
||||
}
|
||||
this.running = false;
|
||||
}
|
||||
|
||||
/** Launch a job as an independent in-flight promise. */
|
||||
private launchJob(job: MinionJob, lockToken: string): void {
|
||||
const abort = new AbortController();
|
||||
@@ -346,30 +277,12 @@ export class MinionWorker {
|
||||
// The .finally clearTimeout below ensures process exit isn't delayed by a
|
||||
// dangling timer on normal completion.
|
||||
let timeoutTimer: ReturnType<typeof setTimeout> | null = null;
|
||||
let graceTimer: ReturnType<typeof setTimeout> | null = null;
|
||||
if (job.timeout_ms != null) {
|
||||
timeoutTimer = setTimeout(() => {
|
||||
if (!abort.signal.aborted) {
|
||||
console.warn(`Job ${job.id} (${job.name}) hit per-job timeout (${job.timeout_ms}ms), aborting`);
|
||||
abort.abort(new Error('timeout'));
|
||||
}
|
||||
// Safety net: if the handler doesn't resolve within 30s after abort,
|
||||
// force-evict from inFlight so the worker can pick up new jobs.
|
||||
// Without this, a handler that ignores AbortSignal wedges the worker
|
||||
// forever (the 98-waiting-0-active incident on 2026-04-24).
|
||||
graceTimer = setTimeout(() => {
|
||||
if (this.inFlight.has(job.id)) {
|
||||
console.warn(
|
||||
`Job ${job.id} (${job.name}) did not exit within 30s of abort. ` +
|
||||
`Force-evicting from inFlight to unblock worker. ` +
|
||||
`The handler is still running but the worker will claim new jobs.`
|
||||
);
|
||||
clearInterval(lockTimer);
|
||||
this.inFlight.delete(job.id);
|
||||
// Best-effort: mark as dead in DB so it doesn't get reclaimed
|
||||
this.queue.failJob(job.id, lockToken, 'handler ignored abort signal (force-evicted)', 'dead').catch(() => {});
|
||||
}
|
||||
}, 30_000);
|
||||
}, job.timeout_ms);
|
||||
}
|
||||
|
||||
@@ -377,10 +290,7 @@ export class MinionWorker {
|
||||
.finally(() => {
|
||||
clearInterval(lockTimer);
|
||||
if (timeoutTimer) clearTimeout(timeoutTimer);
|
||||
if (graceTimer) clearTimeout(graceTimer);
|
||||
this.inFlight.delete(job.id);
|
||||
this.jobsCompleted += 1;
|
||||
this.checkMemoryLimit('post-job');
|
||||
});
|
||||
|
||||
this.inFlight.set(job.id, { job, lockToken, lockTimer, abort, promise });
|
||||
|
||||
+20
-211
@@ -9,7 +9,7 @@ import { PGLITE_SCHEMA_SQL } from './pglite-schema.ts';
|
||||
import { acquireLock, releaseLock, type LockHandle } from './pglite-lock.ts';
|
||||
import type {
|
||||
Page, PageInput, PageFilters, PageType,
|
||||
Chunk, ChunkInput, StaleChunkRow,
|
||||
Chunk, ChunkInput,
|
||||
SearchResult, SearchOpts,
|
||||
Link, GraphNode, GraphPath,
|
||||
TimelineEntry, TimelineInput, TimelineOpts,
|
||||
@@ -20,8 +20,6 @@ import type {
|
||||
EngineConfig,
|
||||
} from './types.ts';
|
||||
import { validateSlug, contentHash, rowToPage, rowToChunk, rowToSearchResult } from './utils.ts';
|
||||
import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts';
|
||||
import { buildSourceFactorCase, buildHardExcludeClause } from './search/sql-ranking.ts';
|
||||
|
||||
type PGLiteDB = PGlite;
|
||||
|
||||
@@ -86,16 +84,6 @@ export class PGLiteEngine implements BrainEngine {
|
||||
}
|
||||
|
||||
async initSchema(): Promise<void> {
|
||||
// Pre-schema bootstrap: add forward-referenced state the embedded schema
|
||||
// blob requires but that older brains don't have yet. Without this, a
|
||||
// pre-v0.18 brain hits `CREATE INDEX idx_pages_source_id ON pages(source_id)`
|
||||
// (issues #366/#375/#378/#396) or a pre-v0.13 brain hits
|
||||
// `CREATE INDEX idx_links_source ON links(link_source)` (#266/#357), and
|
||||
// initSchema crashes before runMigrations gets a chance to apply the
|
||||
// missing column. Bootstrap is structurally idempotent and a no-op on
|
||||
// fresh installs and modern brains.
|
||||
await this.applyForwardReferenceBootstrap();
|
||||
|
||||
await this.db.exec(PGLITE_SCHEMA_SQL);
|
||||
|
||||
const { applied } = await runMigrations(this);
|
||||
@@ -104,111 +92,6 @@ export class PGLiteEngine implements BrainEngine {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Bootstrap state that PGLITE_SCHEMA_SQL forward-references but that older
|
||||
* brains don't have yet. Currently covers:
|
||||
*
|
||||
* - `sources` table + default seed (FK target of pages.source_id) — v0.18
|
||||
* - `pages.source_id` column (indexed by `idx_pages_source_id`) — v0.18
|
||||
* - `links.link_source` column (indexed by `idx_links_source`) — v0.13
|
||||
* - `links.origin_page_id` column (indexed by `idx_links_origin`) — v0.13
|
||||
* - `content_chunks.symbol_name` column (indexed by `idx_chunks_symbol_name`) — v0.19
|
||||
* - `content_chunks.language` column (indexed by `idx_chunks_language`) — v0.19
|
||||
*
|
||||
* **Maintenance contract:** when a future migration adds a column-with-index
|
||||
* or new-table-with-FK referenced by PGLITE_SCHEMA_SQL, extend this method
|
||||
* AND `test/schema-bootstrap-coverage.test.ts`'s `REQUIRED_BOOTSTRAP_COVERAGE`.
|
||||
* The coverage test fails loudly if the bootstrap drifts behind the schema.
|
||||
*/
|
||||
private async applyForwardReferenceBootstrap(): Promise<void> {
|
||||
// Single round-trip probe for every forward-reference target.
|
||||
const { rows } = await this.db.query(`
|
||||
SELECT
|
||||
EXISTS (SELECT 1 FROM information_schema.tables
|
||||
WHERE table_schema='public' AND table_name='pages') AS pages_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.columns
|
||||
WHERE table_schema='public' AND table_name='pages' AND column_name='source_id') AS source_id_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.tables
|
||||
WHERE table_schema='public' AND table_name='links') AS links_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.columns
|
||||
WHERE table_schema='public' AND table_name='links' AND column_name='link_source') AS link_source_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.columns
|
||||
WHERE table_schema='public' AND table_name='links' AND column_name='origin_page_id') AS origin_page_id_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.tables
|
||||
WHERE table_schema='public' AND table_name='content_chunks') AS chunks_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.columns
|
||||
WHERE table_schema='public' AND table_name='content_chunks' AND column_name='symbol_name') AS symbol_name_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.columns
|
||||
WHERE table_schema='public' AND table_name='content_chunks' AND column_name='language') AS language_exists
|
||||
`);
|
||||
const probe = rows[0] as {
|
||||
pages_exists: boolean;
|
||||
source_id_exists: boolean;
|
||||
links_exists: boolean;
|
||||
link_source_exists: boolean;
|
||||
origin_page_id_exists: boolean;
|
||||
chunks_exists: boolean;
|
||||
symbol_name_exists: boolean;
|
||||
language_exists: boolean;
|
||||
};
|
||||
|
||||
const needsPagesBootstrap = probe.pages_exists && !probe.source_id_exists;
|
||||
const needsLinksBootstrap = probe.links_exists
|
||||
&& (!probe.link_source_exists || !probe.origin_page_id_exists);
|
||||
const needsChunksBootstrap = probe.chunks_exists
|
||||
&& (!probe.symbol_name_exists || !probe.language_exists);
|
||||
|
||||
// Fresh installs (no tables yet) and modern brains both no-op.
|
||||
if (!needsPagesBootstrap && !needsLinksBootstrap && !needsChunksBootstrap) return;
|
||||
|
||||
console.log(' Pre-v0.21 brain detected, applying forward-reference bootstrap');
|
||||
|
||||
if (needsPagesBootstrap) {
|
||||
// Mirror schema-embedded.ts shape for `sources` so the subsequent
|
||||
// PGLITE_SCHEMA_SQL CREATE TABLE IF NOT EXISTS is a true no-op.
|
||||
await this.db.exec(`
|
||||
CREATE TABLE IF NOT EXISTS sources (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL UNIQUE,
|
||||
local_path TEXT,
|
||||
last_commit TEXT,
|
||||
last_sync_at TIMESTAMPTZ,
|
||||
config JSONB NOT NULL DEFAULT '{}'::jsonb,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
INSERT INTO sources (id, name, config)
|
||||
VALUES ('default', 'default', '{"federated": true}'::jsonb)
|
||||
ON CONFLICT (id) DO NOTHING;
|
||||
ALTER TABLE pages ADD COLUMN IF NOT EXISTS source_id TEXT
|
||||
NOT NULL DEFAULT 'default' REFERENCES sources(id) ON DELETE CASCADE;
|
||||
`);
|
||||
}
|
||||
|
||||
if (needsLinksBootstrap) {
|
||||
// v11 (links_provenance_columns) is responsible for the CHECK constraint
|
||||
// and backfill. The bootstrap only adds enough state for SCHEMA_SQL's
|
||||
// `CREATE INDEX idx_links_source/origin` not to crash. v11 runs later
|
||||
// via runMigrations and is idempotent (`IF NOT EXISTS` everywhere).
|
||||
await this.db.exec(`
|
||||
ALTER TABLE links ADD COLUMN IF NOT EXISTS link_source TEXT;
|
||||
ALTER TABLE links ADD COLUMN IF NOT EXISTS origin_page_id INTEGER
|
||||
REFERENCES pages(id) ON DELETE SET NULL;
|
||||
`);
|
||||
}
|
||||
|
||||
if (needsChunksBootstrap) {
|
||||
// v26 (content_chunks_code_metadata) adds the full code-chunk metadata
|
||||
// surface (language, symbol_name, symbol_type, start_line, end_line).
|
||||
// The bootstrap only adds the two columns the schema blob's partial
|
||||
// indexes reference (idx_chunks_symbol_name, idx_chunks_language).
|
||||
// v26 runs later via runMigrations and adds the rest idempotently.
|
||||
await this.db.exec(`
|
||||
ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS language TEXT;
|
||||
ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS symbol_name TEXT;
|
||||
`);
|
||||
}
|
||||
}
|
||||
|
||||
async withReservedConnection<T>(fn: (conn: ReservedConnection) => Promise<T>): Promise<T> {
|
||||
// PGLite has no connection pool. The single backing connection is
|
||||
// always effectively reserved — pass it through.
|
||||
@@ -294,13 +177,6 @@ export class PGLiteEngine implements BrainEngine {
|
||||
params.push(filters.updated_after);
|
||||
where.push(`p.updated_at > $${params.length}::timestamptz`);
|
||||
}
|
||||
// slugPrefix uses the (source_id, slug) UNIQUE btree for index range scans.
|
||||
// Escape LIKE metacharacters so the user prefix is treated as a literal.
|
||||
if (filters?.slugPrefix) {
|
||||
const escaped = filters.slugPrefix.replace(/[\\%_]/g, (c) => '\\' + c) + '%';
|
||||
params.push(escaped);
|
||||
where.push(`p.slug LIKE $${params.length} ESCAPE '\\'`);
|
||||
}
|
||||
|
||||
const whereSql = where.length > 0 ? `WHERE ${where.join(' AND ')}` : '';
|
||||
params.push(limit, offset);
|
||||
@@ -363,12 +239,6 @@ export class PGLiteEngine implements BrainEngine {
|
||||
// Fetch 3x to give dedup headroom, then page-dedup + re-limit.
|
||||
const innerLimit = Math.min(limit * 3, MAX_SEARCH_LIMIT * 3);
|
||||
|
||||
// Source-aware ranking (v0.22): see postgres-engine.ts for rationale.
|
||||
const boostMap = resolveBoostMap();
|
||||
const sourceFactorCase = buildSourceFactorCase('p.slug', boostMap, opts?.detail);
|
||||
const hardExcludePrefixes = resolveHardExcludes(opts?.exclude_slug_prefixes, opts?.include_slug_prefixes);
|
||||
const hardExcludeClause = buildHardExcludeClause('p.slug', hardExcludePrefixes);
|
||||
|
||||
// v0.20.0 Cathedral II Layer 10 C1/C2: language + symbol-kind filters.
|
||||
const params: unknown[] = [query, innerLimit, limit, offset];
|
||||
let extraFilter = '';
|
||||
@@ -386,13 +256,13 @@ export class PGLiteEngine implements BrainEngine {
|
||||
SELECT
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score,
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) AS score,
|
||||
CASE WHEN p.updated_at < (
|
||||
SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id
|
||||
) THEN true ELSE false END AS stale
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) ${detailFilter}${extraFilter} ${hardExcludeClause}
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) ${detailFilter}${extraFilter}
|
||||
ORDER BY score DESC
|
||||
LIMIT $2
|
||||
),
|
||||
@@ -431,13 +301,6 @@ export class PGLiteEngine implements BrainEngine {
|
||||
console.warn(`[gbrain] Warning: search limit clamped from ${opts.limit} to ${MAX_SEARCH_LIMIT}`);
|
||||
}
|
||||
|
||||
// Source-aware ranking applied here too — searchKeywordChunks is the
|
||||
// chunk-grain anchor primitive that two-pass retrieval (Layer 7) uses.
|
||||
const boostMap = resolveBoostMap();
|
||||
const sourceFactorCase = buildSourceFactorCase('p.slug', boostMap, opts?.detail);
|
||||
const hardExcludePrefixes = resolveHardExcludes(opts?.exclude_slug_prefixes, opts?.include_slug_prefixes);
|
||||
const hardExcludeClause = buildHardExcludeClause('p.slug', hardExcludePrefixes);
|
||||
|
||||
const params: unknown[] = [query, limit, offset];
|
||||
let extraFilter = '';
|
||||
if (opts?.language) {
|
||||
@@ -453,13 +316,13 @@ export class PGLiteEngine implements BrainEngine {
|
||||
`SELECT
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score,
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) AS score,
|
||||
CASE WHEN p.updated_at < (
|
||||
SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id
|
||||
) THEN true ELSE false END AS stale
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) ${detailFilter}${extraFilter} ${hardExcludeClause}
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) ${detailFilter}${extraFilter}
|
||||
ORDER BY score DESC
|
||||
LIMIT $2 OFFSET $3`,
|
||||
params
|
||||
@@ -478,23 +341,7 @@ export class PGLiteEngine implements BrainEngine {
|
||||
console.warn(`[gbrain] Warning: search limit clamped from ${opts.limit} to ${MAX_SEARCH_LIMIT}`);
|
||||
}
|
||||
|
||||
// Two-stage CTE (v0.22): pure-distance ORDER BY in inner CTE preserves
|
||||
// HNSW; outer SELECT re-ranks by raw_score * source_factor over the
|
||||
// narrow candidate pool. innerLimit scales with offset to preserve the
|
||||
// pagination contract. See postgres-engine.ts searchVector for rationale.
|
||||
const boostMap = resolveBoostMap();
|
||||
// Outer SELECT references the aliased CTE column. Aliasing the CTE as `hc`
|
||||
// disambiguates the correlated subquery (`te.page_id = hc.page_id`) from
|
||||
// the inner column. Without the alias, an unqualified `page_id` in the
|
||||
// subquery's WHERE would lexically resolve back to `te.page_id` itself
|
||||
// and degrade to `te.page_id = te.page_id` (always true), making every
|
||||
// result stale=true. Codex caught this in adversarial review.
|
||||
const sourceFactorCaseOnSlug = buildSourceFactorCase('hc.slug', boostMap, opts?.detail);
|
||||
const hardExcludePrefixes = resolveHardExcludes(opts?.exclude_slug_prefixes, opts?.include_slug_prefixes);
|
||||
const hardExcludeClause = buildHardExcludeClause('p.slug', hardExcludePrefixes);
|
||||
const innerLimit = offset + Math.max(limit * 5, 100);
|
||||
|
||||
const params: unknown[] = [vecStr, innerLimit, limit, offset];
|
||||
const params: unknown[] = [vecStr, limit, offset];
|
||||
let extraFilter = '';
|
||||
if (opts?.language) {
|
||||
params.push(opts.language);
|
||||
@@ -506,28 +353,19 @@ export class PGLiteEngine implements BrainEngine {
|
||||
}
|
||||
|
||||
const { rows } = await this.db.query(
|
||||
`WITH hnsw_candidates AS (
|
||||
SELECT
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id, p.updated_at,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
1 - (cc.embedding <=> $1::vector) AS raw_score
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NOT NULL ${detailFilter}${extraFilter} ${hardExcludeClause}
|
||||
ORDER BY cc.embedding <=> $1::vector
|
||||
LIMIT $2
|
||||
)
|
||||
SELECT
|
||||
hc.slug, hc.page_id, hc.title, hc.type, hc.source_id,
|
||||
hc.chunk_id, hc.chunk_index, hc.chunk_text, hc.chunk_source,
|
||||
hc.raw_score * ${sourceFactorCaseOnSlug} AS score,
|
||||
CASE WHEN hc.updated_at < (
|
||||
SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = hc.page_id
|
||||
) THEN true ELSE false END AS stale
|
||||
FROM hnsw_candidates hc
|
||||
ORDER BY score DESC
|
||||
LIMIT $3
|
||||
OFFSET $4`,
|
||||
`SELECT
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
1 - (cc.embedding <=> $1::vector) AS score,
|
||||
CASE WHEN p.updated_at < (
|
||||
SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id
|
||||
) THEN true ELSE false END AS stale
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NOT NULL ${detailFilter}${extraFilter}
|
||||
ORDER BY cc.embedding <=> $1::vector
|
||||
LIMIT $2
|
||||
OFFSET $3`,
|
||||
params
|
||||
);
|
||||
|
||||
@@ -613,9 +451,6 @@ export class PGLiteEngine implements BrainEngine {
|
||||
}
|
||||
}
|
||||
|
||||
// CONSISTENCY: when chunk_text changes and no new embedding is supplied, BOTH embedding AND
|
||||
// embedded_at must reset to NULL so `embed --stale` correctly picks up the row for re-embedding.
|
||||
// See postgres-engine.ts upsertChunks for the full rationale — pglite mirrors it for parity.
|
||||
await this.db.query(
|
||||
`INSERT INTO content_chunks ${cols} VALUES ${rowParts.join(', ')}
|
||||
ON CONFLICT (page_id, chunk_index) DO UPDATE SET
|
||||
@@ -624,10 +459,7 @@ export class PGLiteEngine implements BrainEngine {
|
||||
embedding = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.embedding ELSE COALESCE(EXCLUDED.embedding, content_chunks.embedding) END,
|
||||
model = COALESCE(EXCLUDED.model, content_chunks.model),
|
||||
token_count = EXCLUDED.token_count,
|
||||
embedded_at = CASE
|
||||
WHEN EXCLUDED.chunk_text != content_chunks.chunk_text AND EXCLUDED.embedding IS NULL THEN NULL
|
||||
ELSE COALESCE(EXCLUDED.embedded_at, content_chunks.embedded_at)
|
||||
END,
|
||||
embedded_at = COALESCE(EXCLUDED.embedded_at, content_chunks.embedded_at),
|
||||
language = EXCLUDED.language,
|
||||
symbol_name = EXCLUDED.symbol_name,
|
||||
symbol_type = EXCLUDED.symbol_type,
|
||||
@@ -651,29 +483,6 @@ export class PGLiteEngine implements BrainEngine {
|
||||
return (rows as Record<string, unknown>[]).map(r => rowToChunk(r));
|
||||
}
|
||||
|
||||
async countStaleChunks(): Promise<number> {
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT count(*)::int AS count
|
||||
FROM content_chunks
|
||||
WHERE embedding IS NULL`,
|
||||
);
|
||||
const count = (rows[0] as { count: number } | undefined)?.count ?? 0;
|
||||
return Number(count);
|
||||
}
|
||||
|
||||
async listStaleChunks(): Promise<StaleChunkRow[]> {
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
cc.model, cc.token_count
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NULL
|
||||
ORDER BY p.id, cc.chunk_index
|
||||
LIMIT 100000`,
|
||||
);
|
||||
return rows as unknown as StaleChunkRow[];
|
||||
}
|
||||
|
||||
async deleteChunks(slug: string): Promise<void> {
|
||||
await this.db.query(
|
||||
`DELETE FROM content_chunks
|
||||
|
||||
+83
-426
@@ -3,10 +3,9 @@ import type { BrainEngine, LinkBatchInput, TimelineBatchInput, ReservedConnectio
|
||||
import { MAX_SEARCH_LIMIT, clampSearchLimit } from './engine.ts';
|
||||
import { runMigrations } from './migrate.ts';
|
||||
import { SCHEMA_SQL } from './schema-embedded.ts';
|
||||
import { verifySchema } from './schema-verify.ts';
|
||||
import type {
|
||||
Page, PageInput, PageFilters, PageType,
|
||||
Chunk, ChunkInput, StaleChunkRow,
|
||||
Chunk, ChunkInput,
|
||||
SearchResult, SearchOpts,
|
||||
Link, GraphNode, GraphPath,
|
||||
TimelineEntry, TimelineInput, TimelineOpts,
|
||||
@@ -19,26 +18,10 @@ import type {
|
||||
import { GBrainError } from './types.ts';
|
||||
import * as db from './db.ts';
|
||||
import { validateSlug, contentHash, rowToPage, rowToChunk, rowToSearchResult, parseEmbedding, tryParseEmbedding } from './utils.ts';
|
||||
import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts';
|
||||
import { buildSourceFactorCase, buildHardExcludeClause } from './search/sql-ranking.ts';
|
||||
|
||||
// CONNECTION_ERROR_PATTERNS / isConnectionError were used by the per-call
|
||||
// executeRaw retry that #406 originally shipped. Eng-review D3 dropped that
|
||||
// retry as unsound (regex idempotence-boundary doesn't hold for writable
|
||||
// CTEs or side-effecting SELECTs). Recovery now happens at the supervisor
|
||||
// level (3-strikes-then-reconnect). The unit tests in
|
||||
// test/connection-resilience.test.ts retain a self-contained copy of the
|
||||
// helper so the regression-against-future-reintroduction guard still works.
|
||||
// See TODOS.md item: "err.code-based connection-error matching" for the
|
||||
// follow-up that will reintroduce a typed retry mechanism.
|
||||
|
||||
export class PostgresEngine implements BrainEngine {
|
||||
readonly kind = 'postgres' as const;
|
||||
private _sql: ReturnType<typeof postgres> | null = null;
|
||||
/** Saved config for reconnection. */
|
||||
private _savedConfig: (EngineConfig & { poolSize?: number }) | null = null;
|
||||
/** Whether a reconnect is in progress (prevents concurrent reconnects). */
|
||||
private _reconnecting = false;
|
||||
|
||||
// Instance connection (for workers) or fall back to module global (backward compat)
|
||||
get sql(): ReturnType<typeof postgres> {
|
||||
@@ -48,7 +31,6 @@ export class PostgresEngine implements BrainEngine {
|
||||
|
||||
// Lifecycle
|
||||
async connect(config: EngineConfig & { poolSize?: number }): Promise<void> {
|
||||
this._savedConfig = config;
|
||||
if (config.poolSize) {
|
||||
// Instance-level connection for worker isolation. resolvePoolSize lets
|
||||
// GBRAIN_POOL_SIZE cap below the caller's requested size when set — the
|
||||
@@ -61,20 +43,12 @@ export class PostgresEngine implements BrainEngine {
|
||||
// "prepared statement does not exist" under load just like the module
|
||||
// singleton did before v0.15.4.
|
||||
const prepare = db.resolvePrepare(url);
|
||||
// Session timeouts (statement_timeout + idle_in_transaction_session_timeout)
|
||||
// keep orphan pgbouncer backends from holding locks for hours when the
|
||||
// postgres.js client disconnects mid-transaction. See resolveSessionTimeouts
|
||||
// in db.ts for context + env var overrides.
|
||||
const timeouts = db.resolveSessionTimeouts();
|
||||
const opts: Record<string, unknown> = {
|
||||
max: size,
|
||||
idle_timeout: 20,
|
||||
connect_timeout: 10,
|
||||
types: { bigint: postgres.BigInt },
|
||||
};
|
||||
if (Object.keys(timeouts).length > 0) {
|
||||
opts.connection = timeouts;
|
||||
}
|
||||
if (typeof prepare === 'boolean') {
|
||||
opts.prepare = prepare;
|
||||
}
|
||||
@@ -99,26 +73,9 @@ export class PostgresEngine implements BrainEngine {
|
||||
async initSchema(): Promise<void> {
|
||||
const conn = this.sql;
|
||||
// Advisory lock prevents concurrent initSchema() calls from deadlocking
|
||||
// on DDL statements (DROP TRIGGER + CREATE TRIGGER acquire AccessExclusiveLock).
|
||||
//
|
||||
// Honest limitation: pg_advisory_lock(42) is session-scoped to this pooled
|
||||
// connection. runMigrations() below uses engine.transaction() and
|
||||
// withReservedConnection() which may hop to a different backend in the
|
||||
// pool. Cross-process serialization of initSchema is best-effort, not a
|
||||
// correctness guarantee. Pre-existing concern; the bootstrap doesn't
|
||||
// change it.
|
||||
// on DDL statements (DROP TRIGGER + CREATE TRIGGER acquire AccessExclusiveLock)
|
||||
await conn`SELECT pg_advisory_lock(42)`;
|
||||
try {
|
||||
// Pre-schema bootstrap: add forward-referenced state the embedded schema
|
||||
// blob requires but that older brains don't have yet. Without this, a
|
||||
// pre-v0.18 brain hits `CREATE INDEX idx_pages_source_id ON pages(source_id)`
|
||||
// (issues #366/#375/#378/#396), or a pre-v0.13 brain hits
|
||||
// `CREATE INDEX idx_links_source ON links(link_source)` (#266/#357), and
|
||||
// SCHEMA_SQL crashes before runMigrations gets a chance to apply the
|
||||
// missing column. Bootstrap is structurally idempotent and a no-op on
|
||||
// fresh installs and modern brains.
|
||||
await this.applyForwardReferenceBootstrap();
|
||||
|
||||
await conn.unsafe(SCHEMA_SQL);
|
||||
|
||||
// Run any pending migrations automatically
|
||||
@@ -126,126 +83,11 @@ export class PostgresEngine implements BrainEngine {
|
||||
if (applied > 0) {
|
||||
console.log(` ${applied} migration(s) applied`);
|
||||
}
|
||||
|
||||
// Post-migration schema verification: catches columns that migrations
|
||||
// defined but PgBouncer transaction-mode silently failed to create.
|
||||
// Self-heals missing columns via ALTER TABLE ADD COLUMN IF NOT EXISTS.
|
||||
const verify = await verifySchema(this);
|
||||
if (verify.healed.length > 0) {
|
||||
console.log(` Schema verify: self-healed ${verify.healed.length} missing column(s)`);
|
||||
}
|
||||
} finally {
|
||||
await conn`SELECT pg_advisory_unlock(42)`;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Bootstrap state that SCHEMA_SQL forward-references but that older brains
|
||||
* don't have yet. Mirror of `PGLiteEngine#applyForwardReferenceBootstrap`
|
||||
* in shape and intent. Currently covers:
|
||||
*
|
||||
* - `sources` table + default seed (FK target of pages.source_id) — v0.18
|
||||
* - `pages.source_id` column (indexed by `idx_pages_source_id`) — v0.18
|
||||
* - `links.link_source` column (indexed by `idx_links_source`) — v0.13
|
||||
* - `links.origin_page_id` column (indexed by `idx_links_origin`) — v0.13
|
||||
* - `content_chunks.symbol_name` column (indexed by `idx_chunks_symbol_name`) — v0.19
|
||||
* - `content_chunks.language` column (indexed by `idx_chunks_language`) — v0.19
|
||||
*
|
||||
* Keep this in sync with the PGLite version; covered by
|
||||
* `test/schema-bootstrap-coverage.test.ts` (PGLite side) and
|
||||
* `test/e2e/postgres-bootstrap.test.ts` (Postgres side).
|
||||
*/
|
||||
private async applyForwardReferenceBootstrap(): Promise<void> {
|
||||
const conn = this.sql;
|
||||
|
||||
// Single round-trip probe for every forward-reference target.
|
||||
// current_schema() resolves to whatever search_path the connection uses,
|
||||
// which matches schema-embedded.ts's `public.` references.
|
||||
const probeRows = await conn<{
|
||||
pages_exists: boolean;
|
||||
source_id_exists: boolean;
|
||||
links_exists: boolean;
|
||||
link_source_exists: boolean;
|
||||
origin_page_id_exists: boolean;
|
||||
chunks_exists: boolean;
|
||||
symbol_name_exists: boolean;
|
||||
language_exists: boolean;
|
||||
}[]>`
|
||||
SELECT
|
||||
EXISTS (SELECT 1 FROM information_schema.tables
|
||||
WHERE table_schema = current_schema() AND table_name = 'pages') AS pages_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.columns
|
||||
WHERE table_schema = current_schema() AND table_name = 'pages' AND column_name = 'source_id') AS source_id_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.tables
|
||||
WHERE table_schema = current_schema() AND table_name = 'links') AS links_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.columns
|
||||
WHERE table_schema = current_schema() AND table_name = 'links' AND column_name = 'link_source') AS link_source_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.columns
|
||||
WHERE table_schema = current_schema() AND table_name = 'links' AND column_name = 'origin_page_id') AS origin_page_id_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.tables
|
||||
WHERE table_schema = current_schema() AND table_name = 'content_chunks') AS chunks_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.columns
|
||||
WHERE table_schema = current_schema() AND table_name = 'content_chunks' AND column_name = 'symbol_name') AS symbol_name_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.columns
|
||||
WHERE table_schema = current_schema() AND table_name = 'content_chunks' AND column_name = 'language') AS language_exists
|
||||
`;
|
||||
const probe = probeRows[0]!;
|
||||
|
||||
const needsPagesBootstrap = probe.pages_exists && !probe.source_id_exists;
|
||||
const needsLinksBootstrap = probe.links_exists
|
||||
&& (!probe.link_source_exists || !probe.origin_page_id_exists);
|
||||
const needsChunksBootstrap = probe.chunks_exists
|
||||
&& (!probe.symbol_name_exists || !probe.language_exists);
|
||||
|
||||
if (!needsPagesBootstrap && !needsLinksBootstrap && !needsChunksBootstrap) return;
|
||||
|
||||
console.log(' Pre-v0.21 brain detected, applying forward-reference bootstrap');
|
||||
|
||||
if (needsPagesBootstrap) {
|
||||
// Mirror schema-embedded.ts's `sources` shape so the subsequent
|
||||
// SCHEMA_SQL CREATE TABLE IF NOT EXISTS is a true no-op.
|
||||
await conn.unsafe(`
|
||||
CREATE TABLE IF NOT EXISTS sources (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL UNIQUE,
|
||||
local_path TEXT,
|
||||
last_commit TEXT,
|
||||
last_sync_at TIMESTAMPTZ,
|
||||
config JSONB NOT NULL DEFAULT '{}'::jsonb,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
INSERT INTO sources (id, name, config)
|
||||
VALUES ('default', 'default', '{"federated": true}'::jsonb)
|
||||
ON CONFLICT (id) DO NOTHING;
|
||||
ALTER TABLE pages ADD COLUMN IF NOT EXISTS source_id TEXT
|
||||
NOT NULL DEFAULT 'default' REFERENCES sources(id) ON DELETE CASCADE;
|
||||
`);
|
||||
}
|
||||
|
||||
if (needsLinksBootstrap) {
|
||||
// v11 (links_provenance_columns) handles the CHECK constraint, the
|
||||
// UNIQUE swap, and the backfill. The bootstrap only adds enough state
|
||||
// for SCHEMA_SQL's `CREATE INDEX idx_links_source/origin` not to crash.
|
||||
// v11 runs later via runMigrations and is idempotent.
|
||||
await conn.unsafe(`
|
||||
ALTER TABLE links ADD COLUMN IF NOT EXISTS link_source TEXT;
|
||||
ALTER TABLE links ADD COLUMN IF NOT EXISTS origin_page_id INTEGER
|
||||
REFERENCES pages(id) ON DELETE SET NULL;
|
||||
`);
|
||||
}
|
||||
|
||||
if (needsChunksBootstrap) {
|
||||
// v26 (content_chunks_code_metadata) adds the full code-chunk metadata
|
||||
// surface. The bootstrap only adds the two columns the schema blob's
|
||||
// partial indexes reference (idx_chunks_symbol_name, idx_chunks_language).
|
||||
// v26 runs later via runMigrations and adds the rest idempotently.
|
||||
await conn.unsafe(`
|
||||
ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS language TEXT;
|
||||
ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS symbol_name TEXT;
|
||||
`);
|
||||
}
|
||||
}
|
||||
|
||||
async transaction<T>(fn: (engine: BrainEngine) => Promise<T>): Promise<T> {
|
||||
const conn = this._sql || db.getConnection();
|
||||
return conn.begin(async (tx) => {
|
||||
@@ -334,17 +176,11 @@ export class PostgresEngine implements BrainEngine {
|
||||
const tagJoin = filters?.tag ? sql`JOIN tags t ON t.page_id = p.id` : sql``;
|
||||
const tagCondition = filters?.tag ? sql`AND t.tag = ${filters.tag}` : sql``;
|
||||
const updatedCondition = updatedAfter ? sql`AND p.updated_at > ${updatedAfter}::timestamptz` : sql``;
|
||||
// slugPrefix uses the (source_id, slug) UNIQUE btree index for range scans.
|
||||
// Escape LIKE metacharacters so the user prefix is treated as a literal.
|
||||
const slugPrefix = filters?.slugPrefix;
|
||||
const slugCondition = slugPrefix
|
||||
? sql`AND p.slug LIKE ${slugPrefix.replace(/[\\%_]/g, (c) => '\\' + c) + '%'} ESCAPE '\\'`
|
||||
: sql``;
|
||||
|
||||
const rows = await sql`
|
||||
SELECT p.* FROM pages p
|
||||
${tagJoin}
|
||||
WHERE 1=1 ${typeCondition} ${tagCondition} ${updatedCondition} ${slugCondition}
|
||||
WHERE 1=1 ${typeCondition} ${tagCondition} ${updatedCondition}
|
||||
ORDER BY p.updated_at DESC LIMIT ${limit} OFFSET ${offset}
|
||||
`;
|
||||
|
||||
@@ -400,83 +236,51 @@ export class PostgresEngine implements BrainEngine {
|
||||
// ship < limit pages. 3x gives dedup enough to pick top N distinct pages.
|
||||
const innerLimit = Math.min(limit * 3, MAX_SEARCH_LIMIT * 3);
|
||||
|
||||
// Source-aware ranking (v0.22): boost curated content (originals/,
|
||||
// concepts/, writing/) and dampen bulk content (chat/, daily/, media/x/)
|
||||
// by multiplying the chunk-grain ts_rank with a source-factor CASE.
|
||||
// Detail-gated — disabled for `detail='high'` (temporal queries) so
|
||||
// chat surfaces normally for date-framed lookups. Hard-exclude prefixes
|
||||
// (test/, archive/, attachments/, .raw/ by default) filter at the
|
||||
// chunk-rank stage so they never enter the candidate set.
|
||||
const boostMap = resolveBoostMap();
|
||||
const sourceFactorCase = buildSourceFactorCase('p.slug', boostMap, opts?.detail);
|
||||
const hardExcludePrefixes = resolveHardExcludes(opts?.exclude_slug_prefixes, opts?.include_slug_prefixes);
|
||||
const hardExcludeClause = buildHardExcludeClause('p.slug', hardExcludePrefixes);
|
||||
|
||||
const params: unknown[] = [query];
|
||||
let typeClause = '';
|
||||
if (type) {
|
||||
params.push(type);
|
||||
typeClause = `AND p.type = $${params.length}`;
|
||||
}
|
||||
let excludeSlugsClause = '';
|
||||
if (excludeSlugs?.length) {
|
||||
params.push(excludeSlugs);
|
||||
excludeSlugsClause = `AND p.slug != ALL($${params.length}::text[])`;
|
||||
}
|
||||
let languageClause = '';
|
||||
if (language) {
|
||||
params.push(language);
|
||||
languageClause = `AND cc.language = $${params.length}`;
|
||||
}
|
||||
let symbolKindClause = '';
|
||||
if (symbolKind) {
|
||||
params.push(symbolKind);
|
||||
symbolKindClause = `AND cc.symbol_type = $${params.length}`;
|
||||
}
|
||||
params.push(innerLimit);
|
||||
const innerLimitParam = `$${params.length}`;
|
||||
params.push(limit);
|
||||
const limitParam = `$${params.length}`;
|
||||
params.push(offset);
|
||||
const offsetParam = `$${params.length}`;
|
||||
|
||||
const rawQuery = `
|
||||
WITH ranked_chunks AS (
|
||||
SELECT
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1)
|
||||
${typeClause}
|
||||
${excludeSlugsClause}
|
||||
${detailLow ? `AND cc.chunk_source = 'compiled_truth'` : ''}
|
||||
${languageClause}
|
||||
${symbolKindClause}
|
||||
${hardExcludeClause}
|
||||
ORDER BY score DESC
|
||||
LIMIT ${innerLimitParam}
|
||||
),
|
||||
best_per_page AS (
|
||||
SELECT DISTINCT ON (slug) *
|
||||
FROM ranked_chunks
|
||||
ORDER BY slug, score DESC
|
||||
)
|
||||
SELECT slug, page_id, title, type, source_id,
|
||||
chunk_id, chunk_index, chunk_text, chunk_source, score,
|
||||
false AS stale
|
||||
FROM best_per_page
|
||||
ORDER BY score DESC
|
||||
LIMIT ${limitParam}
|
||||
OFFSET ${offsetParam}
|
||||
`;
|
||||
|
||||
// Search-only timeout. SET LOCAL inside sql.begin() scopes the GUC
|
||||
// to the transaction so it can never leak onto a pooled connection.
|
||||
// Search-only timeout: prevents DoS via expensive queries without
|
||||
// affecting long-running operations like embed --all or bulk import.
|
||||
// SET LOCAL inside sql.begin() scopes the GUC to the transaction so
|
||||
// it can never leak onto a pooled connection returned to other
|
||||
// callers. A bare `SET statement_timeout` goes to an arbitrary
|
||||
// connection from the pool, lives past this method, and either
|
||||
// clips an unrelated caller's long-running query (DoS) or — via
|
||||
// `SET statement_timeout = 0` — disables the guard for them.
|
||||
const rows = await sql.begin(async sql => {
|
||||
await sql`SET LOCAL statement_timeout = '8s'`;
|
||||
return await sql.unsafe(rawQuery, params as Parameters<typeof sql.unsafe>[1]);
|
||||
// CTE chain: rank chunks by FTS → DISTINCT ON (slug) to pick best
|
||||
// chunk per page → order by score → limit. The external shape is
|
||||
// page-grain; chunk-grain ranking wins because A4 weights mean
|
||||
// doc-comment hits (and, once Layer 5 populates them, qualified
|
||||
// symbol hits) beat body-text hits at the chunk level.
|
||||
return await sql`
|
||||
WITH ranked_chunks AS (
|
||||
SELECT
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('english', ${query})) AS score
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('english', ${query})
|
||||
${type ? sql`AND p.type = ${type}` : sql``}
|
||||
${excludeSlugs?.length ? sql`AND p.slug != ALL(${excludeSlugs})` : sql``}
|
||||
${detailLow ? sql`AND cc.chunk_source = 'compiled_truth'` : sql``}
|
||||
${language ? sql`AND cc.language = ${language}` : sql``}
|
||||
${symbolKind ? sql`AND cc.symbol_type = ${symbolKind}` : sql``}
|
||||
ORDER BY score DESC
|
||||
LIMIT ${innerLimit}
|
||||
),
|
||||
best_per_page AS (
|
||||
SELECT DISTINCT ON (slug) *
|
||||
FROM ranked_chunks
|
||||
ORDER BY slug, score DESC
|
||||
)
|
||||
SELECT slug, page_id, title, type, source_id,
|
||||
chunk_id, chunk_index, chunk_text, chunk_source, score,
|
||||
false AS stale
|
||||
FROM best_per_page
|
||||
ORDER BY score DESC
|
||||
LIMIT ${limit}
|
||||
OFFSET ${offset}
|
||||
`;
|
||||
});
|
||||
return rows.map(rowToSearchResult);
|
||||
}
|
||||
@@ -504,64 +308,26 @@ export class PostgresEngine implements BrainEngine {
|
||||
console.warn(`[gbrain] Warning: search limit clamped from ${opts.limit} to ${MAX_SEARCH_LIMIT}`);
|
||||
}
|
||||
|
||||
// Source-aware ranking applies here too — searchKeywordChunks is the
|
||||
// chunk-grain anchor primitive that two-pass retrieval (Layer 7) uses,
|
||||
// so curated-vs-bulk dampening should affect the anchor pool. Same
|
||||
// detail-gate, same hard-exclude behavior as searchKeyword.
|
||||
const boostMap = resolveBoostMap();
|
||||
const sourceFactorCase = buildSourceFactorCase('p.slug', boostMap, opts?.detail);
|
||||
const hardExcludePrefixes = resolveHardExcludes(opts?.exclude_slug_prefixes, opts?.include_slug_prefixes);
|
||||
const hardExcludeClause = buildHardExcludeClause('p.slug', hardExcludePrefixes);
|
||||
|
||||
const params: unknown[] = [query];
|
||||
let typeClause = '';
|
||||
if (type) {
|
||||
params.push(type);
|
||||
typeClause = `AND p.type = $${params.length}`;
|
||||
}
|
||||
let excludeSlugsClause = '';
|
||||
if (excludeSlugs?.length) {
|
||||
params.push(excludeSlugs);
|
||||
excludeSlugsClause = `AND p.slug != ALL($${params.length}::text[])`;
|
||||
}
|
||||
let languageClause = '';
|
||||
if (language) {
|
||||
params.push(language);
|
||||
languageClause = `AND cc.language = $${params.length}`;
|
||||
}
|
||||
let symbolKindClause = '';
|
||||
if (symbolKind) {
|
||||
params.push(symbolKind);
|
||||
symbolKindClause = `AND cc.symbol_type = $${params.length}`;
|
||||
}
|
||||
params.push(limit);
|
||||
const limitParam = `$${params.length}`;
|
||||
params.push(offset);
|
||||
const offsetParam = `$${params.length}`;
|
||||
|
||||
const rawQuery = `
|
||||
SELECT
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score,
|
||||
false AS stale
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1)
|
||||
${typeClause}
|
||||
${excludeSlugsClause}
|
||||
${detailLow ? `AND cc.chunk_source = 'compiled_truth'` : ''}
|
||||
${languageClause}
|
||||
${symbolKindClause}
|
||||
${hardExcludeClause}
|
||||
ORDER BY score DESC
|
||||
LIMIT ${limitParam}
|
||||
OFFSET ${offsetParam}
|
||||
`;
|
||||
|
||||
const rows = await sql.begin(async sql => {
|
||||
await sql`SET LOCAL statement_timeout = '8s'`;
|
||||
return await sql.unsafe(rawQuery, params as Parameters<typeof sql.unsafe>[1]);
|
||||
return await sql`
|
||||
SELECT
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('english', ${query})) AS score,
|
||||
false AS stale
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('english', ${query})
|
||||
${type ? sql`AND p.type = ${type}` : sql``}
|
||||
${excludeSlugs?.length ? sql`AND p.slug != ALL(${excludeSlugs})` : sql``}
|
||||
${detailLow ? sql`AND cc.chunk_source = 'compiled_truth'` : sql``}
|
||||
${language ? sql`AND cc.language = ${language}` : sql``}
|
||||
${symbolKind ? sql`AND cc.symbol_type = ${symbolKind}` : sql``}
|
||||
ORDER BY score DESC
|
||||
LIMIT ${limit}
|
||||
OFFSET ${offset}
|
||||
`;
|
||||
});
|
||||
return rows.map(rowToSearchResult);
|
||||
}
|
||||
@@ -582,80 +348,29 @@ export class PostgresEngine implements BrainEngine {
|
||||
|
||||
const vecStr = '[' + Array.from(embedding).join(',') + ']';
|
||||
|
||||
// Two-stage CTE (v0.22): inner CTE keeps a pure-distance ORDER BY so
|
||||
// the HNSW index stays usable. Folding source-boost into the inner
|
||||
// ORDER BY would force a sequential scan over every chunk (seconds vs
|
||||
// ~10ms with HNSW). Outer SELECT re-ranks the candidate pool by
|
||||
// raw_score * source_factor.
|
||||
//
|
||||
// innerLimit scales with offset to preserve the pagination contract:
|
||||
// a fixed cap of 100 would silently empty offset > 100.
|
||||
const boostMap = resolveBoostMap();
|
||||
const sourceFactorCaseOnSlug = buildSourceFactorCase('slug', boostMap, opts?.detail);
|
||||
const hardExcludePrefixes = resolveHardExcludes(opts?.exclude_slug_prefixes, opts?.include_slug_prefixes);
|
||||
const hardExcludeClause = buildHardExcludeClause('p.slug', hardExcludePrefixes);
|
||||
const innerLimit = offset + Math.max(limit * 5, 100);
|
||||
|
||||
const params: unknown[] = [vecStr];
|
||||
let typeClause = '';
|
||||
if (type) {
|
||||
params.push(type);
|
||||
typeClause = `AND p.type = $${params.length}`;
|
||||
}
|
||||
let excludeSlugsClause = '';
|
||||
if (excludeSlugs?.length) {
|
||||
params.push(excludeSlugs);
|
||||
excludeSlugsClause = `AND p.slug != ALL($${params.length}::text[])`;
|
||||
}
|
||||
let languageClause = '';
|
||||
if (language) {
|
||||
params.push(language);
|
||||
languageClause = `AND cc.language = $${params.length}`;
|
||||
}
|
||||
let symbolKindClause = '';
|
||||
if (symbolKind) {
|
||||
params.push(symbolKind);
|
||||
symbolKindClause = `AND cc.symbol_type = $${params.length}`;
|
||||
}
|
||||
params.push(innerLimit);
|
||||
const innerLimitParam = `$${params.length}`;
|
||||
params.push(limit);
|
||||
const limitParam = `$${params.length}`;
|
||||
params.push(offset);
|
||||
const offsetParam = `$${params.length}`;
|
||||
|
||||
const rawQuery = `
|
||||
WITH hnsw_candidates AS (
|
||||
// Search-only timeout (see searchKeyword for rationale). SET LOCAL +
|
||||
// sql.begin ensures the GUC stays transaction-scoped on the pooled
|
||||
// connection.
|
||||
const rows = await sql.begin(async sql => {
|
||||
await sql`SET LOCAL statement_timeout = '8s'`;
|
||||
return await sql`
|
||||
SELECT
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
1 - (cc.embedding <=> $1::vector) AS raw_score
|
||||
1 - (cc.embedding <=> ${vecStr}::vector) AS score,
|
||||
false AS stale
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NOT NULL
|
||||
${detailLow ? `AND cc.chunk_source = 'compiled_truth'` : ''}
|
||||
${typeClause}
|
||||
${excludeSlugsClause}
|
||||
${languageClause}
|
||||
${symbolKindClause}
|
||||
${hardExcludeClause}
|
||||
ORDER BY cc.embedding <=> $1::vector
|
||||
LIMIT ${innerLimitParam}
|
||||
)
|
||||
SELECT
|
||||
slug, page_id, title, type, source_id,
|
||||
chunk_id, chunk_index, chunk_text, chunk_source,
|
||||
raw_score * ${sourceFactorCaseOnSlug} AS score,
|
||||
false AS stale
|
||||
FROM hnsw_candidates
|
||||
ORDER BY score DESC
|
||||
LIMIT ${limitParam}
|
||||
OFFSET ${offsetParam}
|
||||
`;
|
||||
|
||||
const rows = await sql.begin(async sql => {
|
||||
await sql`SET LOCAL statement_timeout = '8s'`;
|
||||
return await sql.unsafe(rawQuery, params as Parameters<typeof sql.unsafe>[1]);
|
||||
${detailLow ? sql`AND cc.chunk_source = 'compiled_truth'` : sql``}
|
||||
${type ? sql`AND p.type = ${type}` : sql``}
|
||||
${excludeSlugs?.length ? sql`AND p.slug != ALL(${excludeSlugs})` : sql``}
|
||||
${language ? sql`AND cc.language = ${language}` : sql``}
|
||||
${symbolKind ? sql`AND cc.symbol_type = ${symbolKind}` : sql``}
|
||||
ORDER BY cc.embedding <=> ${vecStr}::vector
|
||||
LIMIT ${limit}
|
||||
OFFSET ${offset}
|
||||
`;
|
||||
});
|
||||
return rows.map(rowToSearchResult);
|
||||
}
|
||||
@@ -734,13 +449,7 @@ export class PostgresEngine implements BrainEngine {
|
||||
}
|
||||
}
|
||||
|
||||
// Single statement upsert: preserves existing embeddings via COALESCE when new value is NULL.
|
||||
// CONSISTENCY: when chunk_text changes and no new embedding is supplied, BOTH embedding AND
|
||||
// embedded_at must reset to NULL so `embed --stale` correctly picks up the row for re-embedding.
|
||||
// Without this, embedded_at lies (says "embedded" while embedding=NULL), and any staleness
|
||||
// predicate on embedded_at would silently skip the row. This is why the egress fix predicates
|
||||
// on `embedding IS NULL` rather than `embedded_at IS NULL` — and it's why we now keep both
|
||||
// columns honest at write time.
|
||||
// Single statement upsert: preserves existing embeddings via COALESCE when new value is NULL
|
||||
await sql.unsafe(
|
||||
`INSERT INTO content_chunks ${cols} VALUES ${rows.join(', ')}
|
||||
ON CONFLICT (page_id, chunk_index) DO UPDATE SET
|
||||
@@ -749,10 +458,7 @@ export class PostgresEngine implements BrainEngine {
|
||||
embedding = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.embedding ELSE COALESCE(EXCLUDED.embedding, content_chunks.embedding) END,
|
||||
model = COALESCE(EXCLUDED.model, content_chunks.model),
|
||||
token_count = EXCLUDED.token_count,
|
||||
embedded_at = CASE
|
||||
WHEN EXCLUDED.chunk_text != content_chunks.chunk_text AND EXCLUDED.embedding IS NULL THEN NULL
|
||||
ELSE COALESCE(EXCLUDED.embedded_at, content_chunks.embedded_at)
|
||||
END,
|
||||
embedded_at = COALESCE(EXCLUDED.embedded_at, content_chunks.embedded_at),
|
||||
language = EXCLUDED.language,
|
||||
symbol_name = EXCLUDED.symbol_name,
|
||||
symbol_type = EXCLUDED.symbol_type,
|
||||
@@ -776,30 +482,6 @@ export class PostgresEngine implements BrainEngine {
|
||||
return rows.map((r) => rowToChunk(r as Record<string, unknown>));
|
||||
}
|
||||
|
||||
async countStaleChunks(): Promise<number> {
|
||||
const sql = this.sql;
|
||||
const [row] = await sql`
|
||||
SELECT count(*)::int AS count
|
||||
FROM content_chunks
|
||||
WHERE embedding IS NULL
|
||||
`;
|
||||
return Number((row as { count?: number } | undefined)?.count ?? 0);
|
||||
}
|
||||
|
||||
async listStaleChunks(): Promise<StaleChunkRow[]> {
|
||||
const sql = this.sql;
|
||||
const rows = await sql`
|
||||
SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
cc.model, cc.token_count
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NULL
|
||||
ORDER BY p.id, cc.chunk_index
|
||||
LIMIT 100000
|
||||
`;
|
||||
return rows as unknown as StaleChunkRow[];
|
||||
}
|
||||
|
||||
async deleteChunks(slug: string): Promise<void> {
|
||||
const sql = this.sql;
|
||||
await sql`
|
||||
@@ -1523,34 +1205,9 @@ export class PostgresEngine implements BrainEngine {
|
||||
return rows.map((r) => rowToChunk(r as Record<string, unknown>, true));
|
||||
}
|
||||
|
||||
/**
|
||||
* Reconnect the engine by tearing down the current pool and creating a fresh one.
|
||||
* No-ops if no saved config (module-singleton mode) or if already reconnecting.
|
||||
*/
|
||||
async reconnect(): Promise<void> {
|
||||
if (!this._savedConfig || this._reconnecting) return;
|
||||
this._reconnecting = true;
|
||||
try {
|
||||
// Tear down old pool (best-effort — it may already be dead)
|
||||
try { await this.disconnect(); } catch { /* swallow */ }
|
||||
// Create fresh pool
|
||||
await this.connect(this._savedConfig);
|
||||
} finally {
|
||||
this._reconnecting = false;
|
||||
}
|
||||
}
|
||||
|
||||
async executeRaw<T = Record<string, unknown>>(sql: string, params?: unknown[]): Promise<T[]> {
|
||||
const conn = this.sql;
|
||||
return conn.unsafe(sql, params as Parameters<typeof conn.unsafe>[1]) as unknown as T[];
|
||||
// Pre-#406 behavior: throw on any error including connection death.
|
||||
// Per-call auto-retry is not safe here because executeRaw is also used
|
||||
// for non-transactional mutations (DELETE/UPDATE/INSERT in sources.ts,
|
||||
// ALTER TABLE in migrations) where retrying after a connection-mid-statement
|
||||
// death can phantom-write a row that already committed on the server.
|
||||
// Recovery instead happens at the supervisor level: the watchdog detects
|
||||
// 3 consecutive health-check failures and calls engine.reconnect() to
|
||||
// swap in a fresh pool. See db.ts setSessionDefaults / supervisor.ts.
|
||||
}
|
||||
|
||||
// ============================================================
|
||||
|
||||
@@ -1,282 +0,0 @@
|
||||
/**
|
||||
* Post-migration schema verification with self-healing.
|
||||
*
|
||||
* PgBouncer transaction-mode poolers can silently swallow ALTER TABLE
|
||||
* statements: the SQL doesn't error, but the column never gets created.
|
||||
* The migration system increments the schema version counter anyway, so
|
||||
* gbrain thinks it's on v29 but the actual table is missing columns.
|
||||
*
|
||||
* This module parses the canonical CREATE TABLE definitions in
|
||||
* schema-embedded.ts and diffs them against information_schema.columns.
|
||||
* Missing columns are self-healed via ALTER TABLE ADD COLUMN IF NOT EXISTS.
|
||||
*
|
||||
* Called at the end of initSchema(), after all migrations complete.
|
||||
*/
|
||||
|
||||
import { SCHEMA_SQL } from './schema-embedded.ts';
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
|
||||
/** A column expected to exist in the database. */
|
||||
export interface ExpectedColumn {
|
||||
table: string;
|
||||
column: string;
|
||||
/** The full column definition (type + constraints) from the CREATE TABLE. */
|
||||
definition: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse CREATE TABLE statements from SCHEMA_SQL to extract expected columns.
|
||||
*
|
||||
* This is a best-effort parser that handles the gbrain schema conventions:
|
||||
* - Standard column definitions with types and constraints
|
||||
* - Skips CONSTRAINT lines, CHECK lines, and UNIQUE lines
|
||||
* - Handles multi-line definitions
|
||||
*
|
||||
* Returns only tables and columns — not constraints, indexes, or triggers.
|
||||
*/
|
||||
export function parseExpectedColumns(): ExpectedColumn[] {
|
||||
const results: ExpectedColumn[] = [];
|
||||
|
||||
// Match CREATE TABLE IF NOT EXISTS <name> ( ... );
|
||||
const tableRegex = /CREATE\s+TABLE\s+IF\s+NOT\s+EXISTS\s+(\w+)\s*\(([\s\S]*?)\);/gi;
|
||||
|
||||
const SQL_KEYWORDS = new Set(['constraint', 'unique', 'check', 'primary', 'foreign', 'exclude']);
|
||||
|
||||
function processLine(tableName: string, line: string) {
|
||||
line = line.trim().replace(/,\s*$/, '');
|
||||
if (!line) return;
|
||||
|
||||
// Skip CONSTRAINT, UNIQUE, CHECK, PRIMARY KEY lines
|
||||
if (/^\s*(CONSTRAINT|UNIQUE|CHECK|PRIMARY\s+KEY)/i.test(line)) return;
|
||||
|
||||
const colMatch = line.match(/^\s*(\w+)\s+(.+)$/);
|
||||
if (colMatch) {
|
||||
const colName = colMatch[1].toLowerCase();
|
||||
if (SQL_KEYWORDS.has(colName)) return;
|
||||
|
||||
results.push({
|
||||
table: tableName,
|
||||
column: colName,
|
||||
definition: colMatch[2].trim(),
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
let match: RegExpExecArray | null;
|
||||
while ((match = tableRegex.exec(SCHEMA_SQL)) !== null) {
|
||||
const tableName = match[1];
|
||||
const body = match[2];
|
||||
|
||||
const lines = body.split('\n');
|
||||
let currentLine = '';
|
||||
|
||||
for (const rawLine of lines) {
|
||||
const trimmed = rawLine.trim();
|
||||
|
||||
// Skip empty lines and comments
|
||||
if (!trimmed || trimmed.startsWith('--')) {
|
||||
// If we have accumulated content and hit a blank/comment line,
|
||||
// the accumulated content is a complete line
|
||||
if (currentLine.trim()) {
|
||||
processLine(tableName, currentLine);
|
||||
currentLine = '';
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
currentLine += ' ' + trimmed;
|
||||
|
||||
// If line ends with comma, it's a complete column definition
|
||||
if (trimmed.endsWith(',')) {
|
||||
processLine(tableName, currentLine);
|
||||
currentLine = '';
|
||||
}
|
||||
}
|
||||
|
||||
// Handle any remaining accumulated line (last column before closing paren)
|
||||
if (currentLine.trim()) {
|
||||
processLine(tableName, currentLine);
|
||||
}
|
||||
}
|
||||
|
||||
// Also parse ALTER TABLE ... ADD COLUMN IF NOT EXISTS statements.
|
||||
// These are used for columns added outside CREATE TABLE blocks
|
||||
// (e.g., pages.search_vector, files.source_id).
|
||||
const alterRegex = /ALTER\s+TABLE\s+(\w+)\s+ADD\s+COLUMN\s+IF\s+NOT\s+EXISTS\s+(\w+)\s+([^;,]+)/gi;
|
||||
let alterMatch: RegExpExecArray | null;
|
||||
const seen = new Set(results.map(r => `${r.table}.${r.column}`));
|
||||
while ((alterMatch = alterRegex.exec(SCHEMA_SQL)) !== null) {
|
||||
const table = alterMatch[1];
|
||||
const column = alterMatch[2].toLowerCase();
|
||||
const definition = alterMatch[3].trim().replace(/,\s*$/, '');
|
||||
const key = `${table}.${column}`;
|
||||
if (!seen.has(key)) {
|
||||
seen.add(key);
|
||||
results.push({ table, column, definition });
|
||||
}
|
||||
}
|
||||
|
||||
return results;
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a simplified type expression suitable for ALTER TABLE ADD COLUMN.
|
||||
*
|
||||
* Strips inline REFERENCES, CHECK, UNIQUE, and complex constraints that
|
||||
* can't be used in ADD COLUMN IF NOT EXISTS. Preserves NOT NULL, DEFAULT,
|
||||
* and the base type.
|
||||
*/
|
||||
export function simplifyColumnDef(definition: string): string {
|
||||
let def = definition;
|
||||
|
||||
// Remove REFERENCES ... (with optional ON DELETE/UPDATE clauses)
|
||||
def = def.replace(/REFERENCES\s+\w+\([^)]*\)(\s+ON\s+(DELETE|UPDATE)\s+\w+(\s+\w+)?)*\s*/gi, '');
|
||||
|
||||
// Remove CHECK constraints (handle nested parens)
|
||||
def = def.replace(/CHECK\s*\((?:[^()]*|\([^()]*\))*\)/gi, '');
|
||||
|
||||
// Remove inline UNIQUE
|
||||
def = def.replace(/\bUNIQUE\b/gi, '');
|
||||
|
||||
// Remove trailing commas and whitespace
|
||||
def = def.replace(/,\s*$/, '').trim();
|
||||
|
||||
// Collapse multiple spaces
|
||||
def = def.replace(/\s+/g, ' ').trim();
|
||||
|
||||
return def;
|
||||
}
|
||||
|
||||
/**
|
||||
* Query the database for actual columns in the public schema.
|
||||
* Returns a Set of "table.column" strings for fast lookup.
|
||||
*/
|
||||
async function getActualColumns(engine: BrainEngine): Promise<Set<string>> {
|
||||
const rows = await engine.executeRaw<{ table_name: string; column_name: string }>(
|
||||
`SELECT table_name, column_name
|
||||
FROM information_schema.columns
|
||||
WHERE table_schema = 'public'`
|
||||
);
|
||||
const set = new Set<string>();
|
||||
for (const row of rows) {
|
||||
set.add(`${row.table_name}.${row.column_name}`);
|
||||
}
|
||||
return set;
|
||||
}
|
||||
|
||||
/**
|
||||
* Get the set of tables that actually exist in the database.
|
||||
*/
|
||||
async function getActualTables(engine: BrainEngine): Promise<Set<string>> {
|
||||
const rows = await engine.executeRaw<{ table_name: string }>(
|
||||
`SELECT table_name
|
||||
FROM information_schema.tables
|
||||
WHERE table_schema = 'public' AND table_type = 'BASE TABLE'`
|
||||
);
|
||||
return new Set(rows.map(r => r.table_name));
|
||||
}
|
||||
|
||||
export interface VerifyResult {
|
||||
/** Total columns checked */
|
||||
checked: number;
|
||||
/** Columns that were missing */
|
||||
missing: Array<{ table: string; column: string }>;
|
||||
/** Columns successfully self-healed */
|
||||
healed: Array<{ table: string; column: string }>;
|
||||
/** Columns that failed to self-heal */
|
||||
failed: Array<{ table: string; column: string; error: string }>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Verify that every column defined in schema-embedded.ts actually exists
|
||||
* in the database. Self-heals missing columns via ALTER TABLE ADD COLUMN.
|
||||
*
|
||||
* Should be called after initSchema() + runMigrations() complete.
|
||||
*
|
||||
* @returns VerifyResult with details of what was checked and fixed.
|
||||
* @throws Error if any columns could not be healed (after attempting all).
|
||||
*/
|
||||
export async function verifySchema(engine: BrainEngine): Promise<VerifyResult> {
|
||||
const expected = parseExpectedColumns();
|
||||
const actualColumns = await getActualColumns(engine);
|
||||
const actualTables = await getActualTables(engine);
|
||||
|
||||
const result: VerifyResult = {
|
||||
checked: 0,
|
||||
missing: [],
|
||||
healed: [],
|
||||
failed: [],
|
||||
};
|
||||
|
||||
// Group expected columns by table for cleaner logging
|
||||
for (const col of expected) {
|
||||
// Skip tables that don't exist yet — they'll be created by schema.sql
|
||||
// on the next initSchema() call. We only verify columns on tables that
|
||||
// DO exist (the failure mode is: table exists, migration ran, but ALTER
|
||||
// TABLE silently failed).
|
||||
if (!actualTables.has(col.table)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
result.checked++;
|
||||
|
||||
const key = `${col.table}.${col.column}`;
|
||||
if (!actualColumns.has(key)) {
|
||||
result.missing.push({ table: col.table, column: col.column });
|
||||
}
|
||||
}
|
||||
|
||||
if (result.missing.length === 0) {
|
||||
return result;
|
||||
}
|
||||
|
||||
// Log missing columns
|
||||
console.warn(`\n⚠️ Schema verification found ${result.missing.length} missing column(s):`);
|
||||
for (const m of result.missing) {
|
||||
console.warn(` ${m.table}.${m.column}`);
|
||||
}
|
||||
console.warn(' Attempting self-heal via ALTER TABLE ADD COLUMN...\n');
|
||||
|
||||
// Build a map from table.column -> definition for self-healing
|
||||
const defMap = new Map<string, string>();
|
||||
for (const col of expected) {
|
||||
defMap.set(`${col.table}.${col.column}`, col.definition);
|
||||
}
|
||||
|
||||
// Attempt to add each missing column
|
||||
for (const m of result.missing) {
|
||||
const rawDef = defMap.get(`${m.table}.${m.column}`);
|
||||
if (!rawDef) {
|
||||
result.failed.push({ ...m, error: 'No definition found in schema' });
|
||||
continue;
|
||||
}
|
||||
|
||||
const simpleDef = simplifyColumnDef(rawDef);
|
||||
|
||||
try {
|
||||
const sql = `ALTER TABLE ${m.table} ADD COLUMN IF NOT EXISTS ${m.column} ${simpleDef}`;
|
||||
await engine.runMigration(0, sql);
|
||||
result.healed.push({ table: m.table, column: m.column });
|
||||
console.log(` ✓ Added ${m.table}.${m.column}`);
|
||||
} catch (err: unknown) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
result.failed.push({ ...m, error: msg });
|
||||
console.error(` ✗ Failed to add ${m.table}.${m.column}: ${msg}`);
|
||||
}
|
||||
}
|
||||
|
||||
if (result.healed.length > 0) {
|
||||
console.log(`\n Schema self-heal: ${result.healed.length}/${result.missing.length} column(s) recovered.`);
|
||||
}
|
||||
|
||||
if (result.failed.length > 0) {
|
||||
const failList = result.failed.map(f => `${f.table}.${f.column}: ${f.error}`).join('\n ');
|
||||
throw new Error(
|
||||
`Schema verification failed: ${result.failed.length} column(s) could not be added:\n ${failList}\n` +
|
||||
'This usually means PgBouncer transaction-mode silently dropped ALTER TABLE statements.\n' +
|
||||
'Fix: connect directly to Postgres (not through PgBouncer) and run: gbrain apply-migrations --yes'
|
||||
);
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
@@ -1,116 +0,0 @@
|
||||
/**
|
||||
* Source-Type Boost Map
|
||||
*
|
||||
* Multiplies into ts_rank / vector cosine score at SQL build time so that
|
||||
* curated content (originals/, concepts/, writing/) outranks bulk content
|
||||
* (wintermute/chat/, daily/, media/x/) for non-temporal queries.
|
||||
*
|
||||
* Keyed by slug prefix. Longest-prefix-match wins (sorted at lookup time
|
||||
* inside sql-ranking.ts). Defaults grounded in the composition of the
|
||||
* canonical brain at ~/git/brain/.
|
||||
*
|
||||
* Override via env: GBRAIN_SOURCE_BOOST="originals/:1.8,wintermute/chat/:0.3"
|
||||
* Hard-exclude via env: GBRAIN_SEARCH_EXCLUDE="test/,scratch/"
|
||||
*/
|
||||
|
||||
export const DEFAULT_SOURCE_BOOSTS: Record<string, number> = {
|
||||
// Curated, opinionated, high-signal — Garry's own writing
|
||||
'originals/': 1.5,
|
||||
// Reusable knowledge frameworks
|
||||
'concepts/': 1.3,
|
||||
// Long-form essays / articles
|
||||
'writing/': 1.4,
|
||||
// Entity pages
|
||||
'people/': 1.2,
|
||||
'companies/': 1.2,
|
||||
'deals/': 1.2,
|
||||
// Notes from real meetings
|
||||
'meetings/': 1.1,
|
||||
// Ingested third-party content
|
||||
'media/articles/': 1.1,
|
||||
'media/repos/': 1.1,
|
||||
// Neutral baselines (explicit for clarity)
|
||||
'yc/': 1.0,
|
||||
'civic/': 1.0,
|
||||
// Bulk / noisy
|
||||
'daily/': 0.8,
|
||||
'media/x/': 0.7,
|
||||
// Chat transcripts — massive, noisy, swamp keyword queries
|
||||
'wintermute/chat/': 0.5,
|
||||
};
|
||||
|
||||
/**
|
||||
* Hard-excludes — slug prefixes that should never enter search results
|
||||
* (unless explicitly opted-in via include_slug_prefixes).
|
||||
*/
|
||||
export const DEFAULT_HARD_EXCLUDES: string[] = [
|
||||
'test/',
|
||||
'archive/',
|
||||
'attachments/',
|
||||
'.raw/',
|
||||
];
|
||||
|
||||
/**
|
||||
* Parse GBRAIN_SOURCE_BOOST env var.
|
||||
* Format: comma-separated prefix:factor pairs.
|
||||
* Example: "originals/:1.8,wintermute/chat/:0.3"
|
||||
*
|
||||
* Malformed entries are skipped silently. Returns empty object if env is
|
||||
* unset or unparseable in its entirety.
|
||||
*/
|
||||
export function parseSourceBoostEnv(env: string | undefined): Record<string, number> {
|
||||
if (!env) return {};
|
||||
const out: Record<string, number> = {};
|
||||
for (const pair of env.split(',')) {
|
||||
const idx = pair.lastIndexOf(':');
|
||||
if (idx <= 0) continue;
|
||||
const prefix = pair.slice(0, idx).trim();
|
||||
const factor = Number.parseFloat(pair.slice(idx + 1).trim());
|
||||
if (!prefix || !Number.isFinite(factor) || factor < 0) continue;
|
||||
out[prefix] = factor;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse GBRAIN_SEARCH_EXCLUDE env var.
|
||||
* Format: comma-separated slug prefixes.
|
||||
* Example: "test/,scratch/,private/"
|
||||
*
|
||||
* Blank entries skipped. Returns empty array if env is unset.
|
||||
*/
|
||||
export function parseHardExcludesEnv(env: string | undefined): string[] {
|
||||
if (!env) return [];
|
||||
return env.split(',').map(s => s.trim()).filter(s => s.length > 0);
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve the effective boost map by merging defaults with env override.
|
||||
* Env entries override defaults (shallow merge); env-only entries are added.
|
||||
*/
|
||||
export function resolveBoostMap(
|
||||
envValue: string | undefined = process.env.GBRAIN_SOURCE_BOOST,
|
||||
): Record<string, number> {
|
||||
const override = parseSourceBoostEnv(envValue);
|
||||
return { ...DEFAULT_SOURCE_BOOSTS, ...override };
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve the effective hard-exclude prefix list.
|
||||
*
|
||||
* - Defaults union with env-supplied excludes
|
||||
* - Subtract any caller-supplied include_slug_prefixes (opt-back-in)
|
||||
* - Caller-supplied exclude_slug_prefixes adds to the union
|
||||
*/
|
||||
export function resolveHardExcludes(
|
||||
excludeOpt?: string[],
|
||||
includeOpt?: string[],
|
||||
envValue: string | undefined = process.env.GBRAIN_SEARCH_EXCLUDE,
|
||||
): string[] {
|
||||
const envExcludes = parseHardExcludesEnv(envValue);
|
||||
const union = new Set<string>([...DEFAULT_HARD_EXCLUDES, ...envExcludes, ...(excludeOpt ?? [])]);
|
||||
if (includeOpt?.length) {
|
||||
for (const p of includeOpt) union.delete(p);
|
||||
}
|
||||
return Array.from(union);
|
||||
}
|
||||
@@ -1,105 +0,0 @@
|
||||
/**
|
||||
* SQL Ranking Builders
|
||||
*
|
||||
* Pure string builders for the source-aware ranking signal that both
|
||||
* postgres-engine and pglite-engine inject into searchKeyword / searchVector.
|
||||
*
|
||||
* Returns RAW SQL FRAGMENTS. Call sites must embed via the engine's "unsafe"
|
||||
* SQL tag (`sql.unsafe(fragment)` for postgres.js, equivalent for pglite).
|
||||
*
|
||||
* Inputs to these builders that originate from env vars or caller options
|
||||
* (slug prefixes) are LIKE-pattern-escaped (`%`, `_`, `\`) AND SQL-string
|
||||
* escaped (single-quote doubling) before inlining. The slugColumn parameter
|
||||
* is supplied by us at the call site and is never user-controllable.
|
||||
*
|
||||
* Numeric factors come from `parseSourceBoostEnv` which calls Number.parseFloat
|
||||
* and validates `Number.isFinite(factor) && factor >= 0`, so they're safe to
|
||||
* inline as bare literals.
|
||||
*/
|
||||
|
||||
/** Escape `%`, `_`, and `\` so a string can be used as a LIKE prefix literal. */
|
||||
function escapeLikePattern(s: string): string {
|
||||
return s.replace(/[%_\\]/g, '\\$&');
|
||||
}
|
||||
|
||||
/** Escape a SQL string literal: replace single-quote with two single-quotes. */
|
||||
function escapeSqlLiteral(s: string): string {
|
||||
return s.replace(/'/g, "''");
|
||||
}
|
||||
|
||||
/** Escape a slug prefix for use as `LIKE 'prefix%'` (both LIKE-escape and SQL-escape). */
|
||||
function buildLikePrefixLiteral(prefix: string): string {
|
||||
return `'${escapeSqlLiteral(escapeLikePattern(prefix))}%'`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a CASE expression that returns the source-boost factor for a slug.
|
||||
*
|
||||
* Returns a literal `'1.0'` when `detail === 'high'` so temporal queries
|
||||
* bypass source-boost entirely (mirrors the existing COMPILED_TRUTH_BOOST
|
||||
* gate in hybrid.ts).
|
||||
*
|
||||
* Prefixes are sorted by length descending so longest-match wins:
|
||||
* `media/articles/` (1.1) wins over `media/x/` (0.7) without caller-order
|
||||
* dependencies.
|
||||
*
|
||||
* @param slugColumn — qualified column reference (e.g. `'p.slug'`). MUST be
|
||||
* supplied by the engine, never from user input.
|
||||
* @param boostMap — prefix → factor map (defaults merged with env override)
|
||||
* @param detail — query detail level; `'high'` disables source-boost
|
||||
*
|
||||
* @returns raw SQL fragment, e.g. `(CASE WHEN p.slug LIKE 'originals/%' THEN 1.5 ... ELSE 1.0 END)`
|
||||
*/
|
||||
export function buildSourceFactorCase(
|
||||
slugColumn: string,
|
||||
boostMap: Record<string, number>,
|
||||
detail: 'low' | 'medium' | 'high' | undefined,
|
||||
): string {
|
||||
// Loose-string guard: agents passing `"HIGH"` or `"high "` over MCP/JSON
|
||||
// should still hit the temporal-bypass path. TypeScript narrows `detail`
|
||||
// for typed callers; this guard catches the untyped boundary.
|
||||
const normalized = typeof detail === 'string' ? detail.trim().toLowerCase() : detail;
|
||||
if (normalized === 'high') return '1.0';
|
||||
|
||||
const entries = Object.entries(boostMap)
|
||||
.filter(([prefix, factor]) => prefix.length > 0 && Number.isFinite(factor) && factor >= 0)
|
||||
.sort((a, b) => b[0].length - a[0].length); // longest-prefix-match wins
|
||||
|
||||
if (entries.length === 0) return '1.0';
|
||||
|
||||
const whens = entries.map(([prefix, factor]) =>
|
||||
`WHEN ${slugColumn} LIKE ${buildLikePrefixLiteral(prefix)} THEN ${factor}`
|
||||
).join(' ');
|
||||
|
||||
return `(CASE ${whens} ELSE 1.0 END)`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a `NOT (col LIKE 'p1%' OR col LIKE 'p2%' OR ...)` exclusion clause.
|
||||
*
|
||||
* Why OR-chain wrapped in NOT, not `NOT LIKE ALL/ANY(array)`:
|
||||
* - `NOT LIKE ALL(array)` means "doesn't match every pattern" — still
|
||||
* keeps rows that match one. Wrong for set-exclusion.
|
||||
* - `NOT LIKE ANY(array)` is non-standard and behavior varies.
|
||||
* - Boolean-friendly OR-chain wrapped in NOT is unambiguous and indexable.
|
||||
*
|
||||
* Returns empty string when prefixes is empty, so callers can interpolate
|
||||
* unconditionally with a leading `AND`.
|
||||
*
|
||||
* @param slugColumn — qualified column reference (engine-supplied, trusted)
|
||||
* @param prefixes — list of slug prefixes to exclude (env + caller-supplied; escaped)
|
||||
*
|
||||
* @returns raw SQL fragment (with leading space) or empty string
|
||||
*/
|
||||
export function buildHardExcludeClause(slugColumn: string, prefixes: string[]): string {
|
||||
if (!prefixes.length) return '';
|
||||
const likes = prefixes
|
||||
.filter(p => p.length > 0)
|
||||
.map(p => `${slugColumn} LIKE ${buildLikePrefixLiteral(p)}`)
|
||||
.join(' OR ');
|
||||
if (!likes) return '';
|
||||
return `AND NOT (${likes})`;
|
||||
}
|
||||
|
||||
// Exported for unit tests
|
||||
export const __test__ = { escapeLikePattern, escapeSqlLiteral, buildLikePrefixLiteral };
|
||||
@@ -183,11 +183,7 @@ function acquireLock(workspace: string, opts: InstallOptions): void {
|
||||
const existing = readLock(workspace);
|
||||
const staleMs = opts.lockStaleMs ?? DEFAULT_LOCK_STALE_MS;
|
||||
if (existing) {
|
||||
// Clamp to 0. On Linux ext4, statSync().mtimeMs has sub-ms precision;
|
||||
// Date.now() is integer ms. A file written microseconds ago can report
|
||||
// a negative age here, which would break the staleMs:0 "any age is stale"
|
||||
// contract the force-unlock path relies on (CI passes, local macOS masks it).
|
||||
const age = Math.max(0, Date.now() - existing.mtimeMs);
|
||||
const age = Date.now() - existing.mtimeMs;
|
||||
// `staleMs: 0` in tests means "any age counts as stale". Use >=
|
||||
// so a just-written lock qualifies when the threshold is 0.
|
||||
// Negative age (mtime in the future) happens on fast CI filesystems
|
||||
|
||||
@@ -132,42 +132,6 @@ async function assertSourceExists(engine: BrainEngine, id: string): Promise<void
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Get the local_path of the resolved source (per the resolveSourceId chain).
|
||||
*
|
||||
* Returns the on-disk brain repo path for the source the user is currently
|
||||
* operating against. Used by `gbrain storage status` and `gbrain export
|
||||
* --restore-only` to find the brain repo without raw SQL or bare try/catch.
|
||||
*
|
||||
* Resolution order:
|
||||
* 1. `sources.local_path` for the resolved source id (multi-source v0.18+ path)
|
||||
* 2. Legacy global `sync.repo_path` config key (pre-v0.18 default-source brains)
|
||||
* 3. null
|
||||
*
|
||||
* @returns local_path string, or null if no path is configured anywhere.
|
||||
* @throws If DB error occurs (does NOT silently swallow). Callers handle
|
||||
* the null case to provide their own fallback (typically a hard error
|
||||
* telling the user to pass --repo).
|
||||
*/
|
||||
export async function getDefaultSourcePath(
|
||||
engine: BrainEngine,
|
||||
cwd: string = process.cwd(),
|
||||
): Promise<string | null> {
|
||||
const sourceId = await resolveSourceId(engine, null, cwd);
|
||||
const rows = await engine.executeRaw<{ local_path: string | null }>(
|
||||
`SELECT local_path FROM sources WHERE id = $1`,
|
||||
[sourceId],
|
||||
);
|
||||
if (rows[0]?.local_path) return rows[0].local_path;
|
||||
|
||||
// Legacy fallback: pre-v0.18 brains stored the repo path in the global
|
||||
// config table under sync.repo_path. The sources table exists but its
|
||||
// local_path is NULL for the seeded 'default' row. Fall back so storage
|
||||
// tiering works without forcing a `gbrain sources add . --path .` migration.
|
||||
const legacyPath = await engine.getConfig('sync.repo_path');
|
||||
return legacyPath ?? null;
|
||||
}
|
||||
|
||||
/** Exposed for tests. */
|
||||
export const __testing = {
|
||||
readDotfileWalk,
|
||||
|
||||
@@ -1,377 +0,0 @@
|
||||
import { readFileSync, existsSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
|
||||
/**
|
||||
* Storage tier configuration loaded from gbrain.yml.
|
||||
*
|
||||
* The canonical key names are `db_tracked` and `db_only` (engine-agnostic).
|
||||
* The deprecated keys `git_tracked` and `supabase_only` are still read for
|
||||
* backward compatibility but emit a once-per-process deprecation warning.
|
||||
* Sunset: future release will reject the deprecated names.
|
||||
*/
|
||||
export interface StorageConfig {
|
||||
db_tracked: string[];
|
||||
db_only: string[];
|
||||
}
|
||||
|
||||
export type StorageTier = 'db_tracked' | 'db_only' | 'unspecified';
|
||||
|
||||
/** Recognized YAML keys (canonical and deprecated). */
|
||||
const STORAGE_KEYS = new Set([
|
||||
'db_tracked', 'db_only',
|
||||
'git_tracked', 'supabase_only', // deprecated aliases
|
||||
]);
|
||||
|
||||
/**
|
||||
* Parse the gbrain.yml shape: a top-level `storage:` section with up to four
|
||||
* array-valued nested keys (canonical `db_tracked` / `db_only` plus the
|
||||
* deprecated aliases `git_tracked` / `supabase_only`).
|
||||
*
|
||||
* Intentionally narrow. Does NOT handle the full YAML spec — only the file
|
||||
* shape gbrain controls. Trades expressiveness for zero-dep parsing and
|
||||
* predictable behavior. Returns null if the file has no `storage:` section
|
||||
* (so callers can distinguish "no config" from "empty config").
|
||||
*
|
||||
* Replaces gray-matter, which silently returned `{data: {}}` on
|
||||
* delimiter-less YAML and broke the entire feature on every install.
|
||||
* The defect that prompted this rewrite: storage-config.ts:24 in the
|
||||
* pre-v0.22.3 implementation.
|
||||
*
|
||||
* Returns the raw key map. The caller (loadStorageConfig) is responsible
|
||||
* for normalizing deprecated keys → canonical, emitting deprecation
|
||||
* warnings, and merging if both old and new keys appear.
|
||||
*/
|
||||
type RawStorage = {
|
||||
db_tracked?: string[];
|
||||
db_only?: string[];
|
||||
git_tracked?: string[];
|
||||
supabase_only?: string[];
|
||||
};
|
||||
|
||||
function parseStorageYaml(content: string): RawStorage | null {
|
||||
const lines = content.split('\n').map((line) => line.replace(/\r$/, ''));
|
||||
|
||||
let inStorage = false;
|
||||
let currentList: keyof RawStorage | null = null;
|
||||
const raw: RawStorage = {};
|
||||
let sawStorage = false;
|
||||
|
||||
for (const line of lines) {
|
||||
// Strip comments. Conservative: drop trailing `# ...` and full-line `#`.
|
||||
const noComment = line.replace(/\s+#.*$/, '').replace(/^#.*$/, '');
|
||||
if (noComment.trim() === '') continue;
|
||||
|
||||
// Top-level key (no leading whitespace).
|
||||
if (!noComment.startsWith(' ') && !noComment.startsWith('\t')) {
|
||||
const colon = noComment.indexOf(':');
|
||||
if (colon === -1) continue;
|
||||
const key = noComment.slice(0, colon).trim();
|
||||
if (key === 'storage') {
|
||||
inStorage = true;
|
||||
sawStorage = true;
|
||||
currentList = null;
|
||||
continue;
|
||||
}
|
||||
inStorage = false;
|
||||
currentList = null;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!inStorage) continue;
|
||||
|
||||
const indented = noComment.replace(/^\s+/, '');
|
||||
|
||||
if (indented.startsWith('-')) {
|
||||
if (!currentList) continue;
|
||||
const value = indented.slice(1).trim().replace(/^["']|["']$/g, '');
|
||||
if (value) {
|
||||
if (!raw[currentList]) raw[currentList] = [];
|
||||
raw[currentList]!.push(value);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
const colon = indented.indexOf(':');
|
||||
if (colon === -1) continue;
|
||||
const key = indented.slice(0, colon).trim();
|
||||
if (STORAGE_KEYS.has(key)) {
|
||||
currentList = key as keyof RawStorage;
|
||||
// Inline empty list: `db_only: []`.
|
||||
const remainder = indented.slice(colon + 1).trim();
|
||||
if (remainder === '[]' && !raw[currentList]) {
|
||||
raw[currentList] = [];
|
||||
}
|
||||
continue;
|
||||
}
|
||||
currentList = null;
|
||||
}
|
||||
|
||||
if (!sawStorage) return null;
|
||||
return raw;
|
||||
}
|
||||
|
||||
/**
|
||||
* Normalize raw parsed keys into canonical StorageConfig shape.
|
||||
*
|
||||
* Resolution order (per plan eng-review pass 2 finding #2):
|
||||
* 1. If canonical keys present, use them.
|
||||
* 2. Else if deprecated keys present, map to canonical AND emit a
|
||||
* once-per-process deprecation warning suggesting `gbrain doctor --fix`.
|
||||
* 3. If both are present, canonical wins. Deprecated keys are ignored
|
||||
* with a stronger warning (the user is mid-migration).
|
||||
*
|
||||
* Validation (validateStorageConfig) always runs against the canonical
|
||||
* shape, so error messages reference `db_only` / `db_tracked` regardless
|
||||
* of which keys the user wrote.
|
||||
*/
|
||||
let _deprecationWarned = false;
|
||||
|
||||
function normalizeStorageConfig(raw: RawStorage): StorageConfig {
|
||||
const hasCanonical = Boolean(raw.db_tracked || raw.db_only);
|
||||
const hasDeprecated = Boolean(raw.git_tracked || raw.supabase_only);
|
||||
|
||||
if (hasDeprecated && !_deprecationWarned) {
|
||||
_deprecationWarned = true;
|
||||
const which = [
|
||||
raw.git_tracked ? '`git_tracked`' : null,
|
||||
raw.supabase_only ? '`supabase_only`' : null,
|
||||
].filter(Boolean).join(' and ');
|
||||
if (hasCanonical) {
|
||||
console.warn(
|
||||
`Warning: ${which} in gbrain.yml is deprecated and ignored ` +
|
||||
`(canonical keys db_tracked/db_only are present). ` +
|
||||
`Remove the deprecated keys, or run \`gbrain doctor --fix\`.`,
|
||||
);
|
||||
} else {
|
||||
console.warn(
|
||||
`Warning: ${which} in gbrain.yml is deprecated. ` +
|
||||
`Rename to db_tracked / db_only — see docs/storage-tiering.md. ` +
|
||||
`Run \`gbrain doctor --fix\` for an automated rename.`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
if (hasCanonical) {
|
||||
return {
|
||||
db_tracked: raw.db_tracked ?? [],
|
||||
db_only: raw.db_only ?? [],
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
db_tracked: raw.git_tracked ?? [],
|
||||
db_only: raw.supabase_only ?? [],
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Load gbrain.yml configuration from the brain repository root.
|
||||
*
|
||||
* Returns null when:
|
||||
* - repoPath is null/undefined
|
||||
* - gbrain.yml doesn't exist at the repo root
|
||||
* - gbrain.yml exists but has no `storage:` section (with sanity warning)
|
||||
*
|
||||
* Throws when:
|
||||
* - gbrain.yml exists but is unreadable (permission denied, etc.) — D36 lock:
|
||||
* fail loud rather than silently disable the feature.
|
||||
*
|
||||
* Logs a console.warn (once per process) when:
|
||||
* - File parses but `storage:` section is empty or missing — Issue #1 lock:
|
||||
* surface "your config didn't take" rather than silently no-op.
|
||||
*/
|
||||
let _missingStorageWarned = false;
|
||||
|
||||
export function loadStorageConfig(repoPath?: string | null): StorageConfig | null {
|
||||
if (!repoPath) return null;
|
||||
|
||||
const yamlPath = join(repoPath, 'gbrain.yml');
|
||||
if (!existsSync(yamlPath)) return null;
|
||||
|
||||
// Read failure is a real error (not a "feature not configured" signal).
|
||||
// Throwing here lets the caller decide whether to crash or fall back.
|
||||
const content = readFileSync(yamlPath, 'utf-8');
|
||||
|
||||
let raw: RawStorage | null;
|
||||
try {
|
||||
raw = parseStorageYaml(content);
|
||||
} catch (error) {
|
||||
console.warn(
|
||||
`Warning: Failed to parse gbrain.yml: ${error instanceof Error ? error.message : String(error)}`,
|
||||
);
|
||||
return null;
|
||||
}
|
||||
|
||||
// No storage section at all → null (with sanity warning).
|
||||
if (raw === null) {
|
||||
if (!_missingStorageWarned) {
|
||||
_missingStorageWarned = true;
|
||||
console.warn(
|
||||
`Warning: ${yamlPath} exists but has no storage configuration. ` +
|
||||
`Add a "storage:" section with db_tracked / db_only arrays, ` +
|
||||
`or remove gbrain.yml to suppress this warning.`,
|
||||
);
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
const merged = normalizeStorageConfig(raw);
|
||||
|
||||
// Empty storage section → return as-is but warn.
|
||||
if (merged.db_tracked.length === 0 && merged.db_only.length === 0) {
|
||||
if (!_missingStorageWarned) {
|
||||
_missingStorageWarned = true;
|
||||
console.warn(
|
||||
`Warning: ${yamlPath} exists but has no storage configuration. ` +
|
||||
`Add a "storage:" section with db_tracked / db_only arrays, ` +
|
||||
`or remove gbrain.yml to suppress this warning.`,
|
||||
);
|
||||
}
|
||||
return merged;
|
||||
}
|
||||
|
||||
// Normalize cosmetic issues + throw on semantic overlap (D7).
|
||||
// Throws StorageConfigError on overlap — propagates to the caller.
|
||||
return normalizeAndValidateStorageConfig(merged);
|
||||
}
|
||||
|
||||
export class StorageConfigError extends Error {
|
||||
constructor(message: string) {
|
||||
super(message);
|
||||
this.name = 'StorageConfigError';
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Validate storage configuration for conflicts and issues.
|
||||
* Returns warning strings; callers decide how to surface them.
|
||||
*
|
||||
* Always runs against the canonical (db_tracked / db_only) shape — error
|
||||
* messages reference canonical names regardless of which keys the user
|
||||
* wrote in gbrain.yml.
|
||||
*
|
||||
* Pure: does not mutate. For the auto-normalize behavior (D7), see
|
||||
* `normalizeAndValidateStorageConfig` below.
|
||||
*/
|
||||
export function validateStorageConfig(config: StorageConfig): string[] {
|
||||
const warnings: string[] = [];
|
||||
|
||||
const trackedSet = new Set(config.db_tracked);
|
||||
for (const path of config.db_only) {
|
||||
if (trackedSet.has(path)) {
|
||||
warnings.push(`Directory "${path}" appears in both db_tracked and db_only`);
|
||||
}
|
||||
}
|
||||
|
||||
const allPaths = [...config.db_tracked, ...config.db_only];
|
||||
for (const path of allPaths) {
|
||||
if (!path.endsWith('/')) {
|
||||
warnings.push(`Directory path "${path}" should end with "/" for consistency`);
|
||||
}
|
||||
}
|
||||
|
||||
return warnings;
|
||||
}
|
||||
|
||||
/**
|
||||
* Auto-normalize and strict-validate per D7+D8.
|
||||
*
|
||||
* 1. Cosmetic fixups are applied silently with a one-time info message
|
||||
* naming what changed:
|
||||
* - missing trailing `/` is added
|
||||
* The message helps the user learn the canonical form without nagging.
|
||||
* 2. Semantic problems THROW (don't return warnings):
|
||||
* - same directory in both tiers (ambiguous routing)
|
||||
*
|
||||
* Caller passes a fresh raw config; this returns the normalized shape that
|
||||
* the rest of the code (matcher, sync, etc.) sees.
|
||||
*/
|
||||
let _normalizationInfoEmitted = false;
|
||||
|
||||
export function normalizeAndValidateStorageConfig(input: StorageConfig): StorageConfig {
|
||||
const normalize = (paths: string[]): { normalized: string[]; changed: string[] } => {
|
||||
const normalized: string[] = [];
|
||||
const changed: string[] = [];
|
||||
for (const p of paths) {
|
||||
if (p.endsWith('/')) {
|
||||
normalized.push(p);
|
||||
} else {
|
||||
normalized.push(p + '/');
|
||||
changed.push(`"${p}" → "${p}/"`);
|
||||
}
|
||||
}
|
||||
return { normalized, changed };
|
||||
};
|
||||
|
||||
const tracked = normalize(input.db_tracked);
|
||||
const dbonly = normalize(input.db_only);
|
||||
const allChanged = [...tracked.changed, ...dbonly.changed];
|
||||
|
||||
if (allChanged.length > 0 && !_normalizationInfoEmitted) {
|
||||
_normalizationInfoEmitted = true;
|
||||
console.warn(
|
||||
`Note: normalized ${allChanged.length} storage path(s) in gbrain.yml — ` +
|
||||
`${allChanged.join(', ')}. Add trailing "/" to suppress this note.`,
|
||||
);
|
||||
}
|
||||
|
||||
// Semantic check: overlap between tiers throws. Ambiguous routing.
|
||||
const trackedSet = new Set(tracked.normalized);
|
||||
for (const path of dbonly.normalized) {
|
||||
if (trackedSet.has(path)) {
|
||||
throw new StorageConfigError(
|
||||
`gbrain.yml: directory "${path}" appears in both db_tracked and db_only — ` +
|
||||
`pick one tier. Edit gbrain.yml to remove the overlap.`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
return { db_tracked: tracked.normalized, db_only: dbonly.normalized };
|
||||
}
|
||||
|
||||
/**
|
||||
* Path-segment match: a slug belongs to a tier directory iff the directory
|
||||
* is a complete path-segment ancestor of the slug. `media/x/` matches
|
||||
* `media/x/foo` but NOT `media/xerox/foo` — eliminates the prefix-collision
|
||||
* class of bug (Issue #5 of the eng review, D6 lock).
|
||||
*
|
||||
* Strict: requires the configured directory to end with `/`. The validator
|
||||
* (per D7+D8) auto-normalizes input so the matcher only ever sees canonical
|
||||
* trailing-`/` directories.
|
||||
*/
|
||||
function matchesTierDir(slug: string, dir: string): boolean {
|
||||
if (!dir.endsWith('/')) return false; // not normalized — matcher refuses
|
||||
// slug must equal dir's bare prefix OR start with the trailing-slash form.
|
||||
// Example: dir = 'media/x/' matches 'media/x/anything' but not 'media/x'
|
||||
// or 'media/xerox'. (A slug that exactly equals 'media/x' is a directory-
|
||||
// level entry the brain doesn't write.)
|
||||
return slug.startsWith(dir);
|
||||
}
|
||||
|
||||
export function isDbTracked(slug: string, config: StorageConfig): boolean {
|
||||
return config.db_tracked.some((dir) => matchesTierDir(slug, dir));
|
||||
}
|
||||
|
||||
export function isDbOnly(slug: string, config: StorageConfig): boolean {
|
||||
return config.db_only.some((dir) => matchesTierDir(slug, dir));
|
||||
}
|
||||
|
||||
export function getStorageTier(slug: string, config: StorageConfig): StorageTier {
|
||||
if (isDbTracked(slug, config)) return 'db_tracked';
|
||||
if (isDbOnly(slug, config)) return 'db_only';
|
||||
return 'unspecified';
|
||||
}
|
||||
|
||||
// ── Deprecated aliases — to be removed in a future release ────────
|
||||
// Kept so existing callers (storage.ts, export.ts) compile during the
|
||||
// step-by-step refactor. Will be deleted once those call sites migrate
|
||||
// to the canonical names.
|
||||
export const isGitTracked = isDbTracked;
|
||||
export const isSupabaseOnly = isDbOnly;
|
||||
|
||||
/** Reset once-per-process warning flags. Test-only. */
|
||||
export function __resetMissingStorageWarning(): void {
|
||||
_missingStorageWarned = false;
|
||||
_deprecationWarned = false;
|
||||
_normalizationInfoEmitted = false;
|
||||
}
|
||||
@@ -1,101 +0,0 @@
|
||||
/**
|
||||
* Shared concurrency policy for sync + import + jobs paths.
|
||||
*
|
||||
* Three callers used to embed three different policies:
|
||||
* - performSync (incremental): >100 files → 4 workers
|
||||
* - performFullSync: Postgres → 4 workers
|
||||
* - jobs.ts sync handler: hardcoded 4
|
||||
*
|
||||
* They drift over time and confuse users ("why does my sync not parallelize?"
|
||||
* is a different answer in each path). This module is one source of truth.
|
||||
*
|
||||
* v0.22.13 — extracted as part of the parallel-sync hardening (PR #490).
|
||||
*/
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
|
||||
/** Threshold above which auto-concurrency fires for incremental sync paths. */
|
||||
export const AUTO_CONCURRENCY_FILE_THRESHOLD = 100;
|
||||
|
||||
/** Minimum file count below which the parallel branch is skipped even when
|
||||
* auto-concurrency would otherwise fire. Prevents spawning workers for trivial
|
||||
* diffs where setup cost exceeds parallelism gains. Only consulted on the
|
||||
* auto path; explicit `--workers N` bypasses this. */
|
||||
export const PARALLEL_FILE_FLOOR = 50;
|
||||
|
||||
/** Default worker count when auto-concurrency fires. */
|
||||
export const DEFAULT_PARALLEL_WORKERS = 4;
|
||||
|
||||
/**
|
||||
* Resolve effective worker count for a sync/import operation.
|
||||
*
|
||||
* Inputs:
|
||||
* - engine.kind: 'pglite' always returns 1 (single-connection)
|
||||
* - override: caller's explicit --workers / opts.concurrency value
|
||||
* - fileCount: size of the work batch
|
||||
*
|
||||
* Rules:
|
||||
* - PGLite → always 1 (the engine is single-connection regardless)
|
||||
* - explicit override → respect it (clamped to >=1)
|
||||
* - auto path → DEFAULT_PARALLEL_WORKERS when fileCount > AUTO_CONCURRENCY_FILE_THRESHOLD, else 1
|
||||
*
|
||||
* Note: this function does NOT consult PARALLEL_FILE_FLOOR. The floor is a
|
||||
* caller-side gate that decides whether to take the parallel code path even
|
||||
* when the worker count is > 1. It only applies to the auto path; explicit
|
||||
* --workers bypasses the floor entirely (per Q1 in PR #490).
|
||||
*/
|
||||
export function autoConcurrency(
|
||||
engine: BrainEngine,
|
||||
fileCount: number,
|
||||
override?: number,
|
||||
): number {
|
||||
if (engine.kind === 'pglite') return 1;
|
||||
if (override !== undefined) return Math.max(1, override);
|
||||
return fileCount > AUTO_CONCURRENCY_FILE_THRESHOLD
|
||||
? DEFAULT_PARALLEL_WORKERS
|
||||
: 1;
|
||||
}
|
||||
|
||||
/**
|
||||
* Decide whether the parallel code path should run.
|
||||
*
|
||||
* - workers <= 1 → never parallel
|
||||
* - workers > 1 + explicit override → always parallel (user opted in,
|
||||
* respect them even on small diffs — Q1 in PR #490)
|
||||
* - workers > 1 + auto path → parallel only when fileCount > PARALLEL_FILE_FLOOR
|
||||
*/
|
||||
export function shouldRunParallel(
|
||||
workers: number,
|
||||
fileCount: number,
|
||||
explicit: boolean,
|
||||
): boolean {
|
||||
if (workers <= 1) return false;
|
||||
if (explicit) return true;
|
||||
return fileCount > PARALLEL_FILE_FLOOR;
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse a `--workers N` / `--concurrency N` CLI argument value.
|
||||
*
|
||||
* Returns:
|
||||
* - undefined when the flag was not provided
|
||||
* - a positive integer when the flag was provided with a valid value
|
||||
*
|
||||
* Throws on:
|
||||
* - non-integer ("foo", "1.5", "")
|
||||
* - zero or negative ("0", "-3")
|
||||
* - NaN / Infinity
|
||||
*
|
||||
* Q2 in PR #490: the prior parseInt-with-no-validation accepted `--workers 0`
|
||||
* and silently fell through to auto-concurrency (4 workers), the opposite of
|
||||
* what the user typed. Fail loud instead.
|
||||
*/
|
||||
export function parseWorkers(s: string | undefined): number | undefined {
|
||||
if (s === undefined) return undefined;
|
||||
const n = parseInt(s, 10);
|
||||
if (!Number.isFinite(n) || n < 1 || String(n) !== s.trim()) {
|
||||
throw new Error(
|
||||
`--workers must be a positive integer, got: ${JSON.stringify(s)}`,
|
||||
);
|
||||
}
|
||||
return n;
|
||||
}
|
||||
+6
-108
@@ -307,8 +307,6 @@ import { createHash as _createHash } from 'crypto';
|
||||
export interface SyncFailure {
|
||||
path: string;
|
||||
error: string;
|
||||
/** Structured error code extracted from the error message. */
|
||||
code?: string;
|
||||
commit: string;
|
||||
line?: number;
|
||||
ts: string;
|
||||
@@ -316,91 +314,6 @@ export interface SyncFailure {
|
||||
acknowledged_at?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Best-effort extraction of a structured error code from a sync failure
|
||||
* message. Matches known ParseValidationCode patterns (SLUG_MISMATCH,
|
||||
* YAML_PARSE, etc.) and common DB / timeout errors. Returns 'UNKNOWN'
|
||||
* when no pattern matches.
|
||||
*
|
||||
* Order matters: DB-layer errors are checked BEFORE YAML-layer ones so
|
||||
* Postgres `duplicate key value violates unique constraint` doesn't get
|
||||
* mislabeled as a YAML duplicate-key. Frontmatter patterns key off the
|
||||
* canonical messages emitted by `collectValidationErrors()` in markdown.ts.
|
||||
*/
|
||||
export function classifyErrorCode(errorMsg: string): string {
|
||||
// SLUG_MISMATCH: thrown by importFromFile() at src/core/import-file.ts:374.
|
||||
if (/slug.*does not match|SLUG_MISMATCH/i.test(errorMsg)) return 'SLUG_MISMATCH';
|
||||
|
||||
// DB-layer errors come BEFORE the YAML duplicate-key check. Postgres unique-
|
||||
// constraint violations contain "duplicate key" but are not a YAML problem.
|
||||
if (/duplicate key value violates unique constraint|DB_DUPLICATE_KEY/i.test(errorMsg)) {
|
||||
return 'DB_DUPLICATE_KEY';
|
||||
}
|
||||
if (/canceling statement due to statement timeout|STATEMENT_TIMEOUT/i.test(errorMsg)) {
|
||||
return 'STATEMENT_TIMEOUT';
|
||||
}
|
||||
|
||||
// YAML / frontmatter patterns. These match either the canonical message
|
||||
// strings in src/core/markdown.ts (collectValidationErrors) or the literal
|
||||
// ParseValidationCode token, so they fire whether the caller stores the
|
||||
// message or just the code.
|
||||
if (/YAML parse failed|YAML_PARSE/i.test(errorMsg)) return 'YAML_PARSE';
|
||||
if (/YAMLException|duplicated mapping key|YAML_DUPLICATE_KEY/i.test(errorMsg)) {
|
||||
return 'YAML_DUPLICATE_KEY';
|
||||
}
|
||||
if (/File is empty or whitespace-only|Frontmatter must start with ---|MISSING_OPEN/i.test(errorMsg)) {
|
||||
return 'MISSING_OPEN';
|
||||
}
|
||||
if (/No closing --- delimiter|Heading at line .* found inside frontmatter|MISSING_CLOSE/i.test(errorMsg)) {
|
||||
return 'MISSING_CLOSE';
|
||||
}
|
||||
if (/Frontmatter block is empty|EMPTY_FRONTMATTER/i.test(errorMsg)) return 'EMPTY_FRONTMATTER';
|
||||
if (/Content contains null bytes|NULL_BYTES|null byte/i.test(errorMsg)) return 'NULL_BYTES';
|
||||
if (/Nested double quotes|NESTED_QUOTES/i.test(errorMsg)) return 'NESTED_QUOTES';
|
||||
|
||||
// Generic fallbacks.
|
||||
if (/invalid UTF-?8|INVALID_UTF8/i.test(errorMsg)) return 'INVALID_UTF8';
|
||||
|
||||
// v0.22.12 additions: covers the four real production sites in src/core/import-file.ts
|
||||
// (lines 199, 347, 352, 401) that previously bucketed to UNKNOWN.
|
||||
if (/file too large|content too large|FILE_TOO_LARGE/i.test(errorMsg)) return 'FILE_TOO_LARGE';
|
||||
if (/skipping symlink|symlink|SYMLINK_NOT_ALLOWED/i.test(errorMsg)) return 'SYMLINK_NOT_ALLOWED';
|
||||
return 'UNKNOWN';
|
||||
}
|
||||
|
||||
/** Group failures by error code and return a sorted summary. */
|
||||
export function summarizeFailuresByCode(
|
||||
failures: Array<{ error: string; code?: string }>,
|
||||
): Array<{ code: string; count: number }> {
|
||||
const counts: Record<string, number> = {};
|
||||
for (const f of failures) {
|
||||
const code = f.code ?? classifyErrorCode(f.error);
|
||||
counts[code] = (counts[code] ?? 0) + 1;
|
||||
}
|
||||
return Object.entries(counts)
|
||||
.sort(([, a], [, b]) => b - a)
|
||||
.map(([code, count]) => ({ code, count }));
|
||||
}
|
||||
|
||||
/**
|
||||
* Format a code-grouped summary as a human-readable multi-line string for
|
||||
* stderr / doctor output. Accepts either raw failures (which are summarized
|
||||
* internally) or an already-summarized `{code, count}[]` shape (the return
|
||||
* value of `summarizeFailuresByCode` or `AcknowledgeResult.summary`).
|
||||
* Returns an empty string when the input is empty.
|
||||
*/
|
||||
export function formatCodeBreakdown(
|
||||
input: Array<{ error: string; code?: string }> | Array<{ code: string; count: number }>,
|
||||
): string {
|
||||
// Distinguish by shape: summary entries have a numeric `count`. Empty array
|
||||
// returns '' from either branch — both paths produce a 0-length join.
|
||||
const summary =
|
||||
input.length > 0 && typeof (input[0] as { count?: unknown }).count === 'number'
|
||||
? (input as Array<{ code: string; count: number }>)
|
||||
: summarizeFailuresByCode(input as Array<{ error: string; code?: string }>);
|
||||
return summary.map(s => ` ${s.code}: ${s.count}`).join('\n');
|
||||
}
|
||||
|
||||
function _failuresDir(): string {
|
||||
return _joinPath(_homedir(), '.gbrain');
|
||||
}
|
||||
@@ -457,7 +370,6 @@ export function recordSyncFailures(
|
||||
const entry: SyncFailure = {
|
||||
path: f.path,
|
||||
error: f.error,
|
||||
code: classifyErrorCode(f.error),
|
||||
commit,
|
||||
line: f.line,
|
||||
ts: now,
|
||||
@@ -468,42 +380,28 @@ export function recordSyncFailures(
|
||||
}
|
||||
}
|
||||
|
||||
export interface AcknowledgeResult {
|
||||
count: number;
|
||||
summary: Array<{ code: string; count: number }>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Mark all unacknowledged failures as acknowledged. Used by
|
||||
* `gbrain sync --skip-failed`. Returns count and a structured summary
|
||||
* grouped by error code so the operator can see *why* files were skipped.
|
||||
* `gbrain sync --skip-failed`. Returns the number newly acknowledged.
|
||||
*
|
||||
* We do not delete — acknowledged entries stay as historical record so
|
||||
* doctor can still show them under a "previously skipped" bucket.
|
||||
*/
|
||||
export function acknowledgeSyncFailures(): AcknowledgeResult {
|
||||
export function acknowledgeSyncFailures(): number {
|
||||
const entries = loadSyncFailures();
|
||||
if (entries.length === 0) return { count: 0, summary: [] };
|
||||
if (entries.length === 0) return 0;
|
||||
const now = new Date().toISOString();
|
||||
let changed = 0;
|
||||
const newlyAcked: SyncFailure[] = [];
|
||||
const updated = entries.map(e => {
|
||||
if (e.acknowledged) return e;
|
||||
changed++;
|
||||
// Backfill code for entries that predate the code field.
|
||||
const code = e.code ?? classifyErrorCode(e.error);
|
||||
const acked = { ...e, code, acknowledged: true, acknowledged_at: now };
|
||||
newlyAcked.push(acked);
|
||||
return acked;
|
||||
return { ...e, acknowledged: true, acknowledged_at: now };
|
||||
});
|
||||
if (changed === 0) return { count: 0, summary: [] };
|
||||
if (changed === 0) return 0;
|
||||
_mkdirSync(_failuresDir(), { recursive: true });
|
||||
const fd = require('fs').writeFileSync;
|
||||
fd(syncFailuresPath(), updated.map(e => JSON.stringify(e)).join('\n') + '\n');
|
||||
return {
|
||||
count: changed,
|
||||
summary: summarizeFailuresByCode(newlyAcked),
|
||||
};
|
||||
return changed;
|
||||
}
|
||||
|
||||
/** Return only unacknowledged failures. */
|
||||
|
||||
@@ -45,14 +45,6 @@ export interface PageFilters {
|
||||
offset?: number;
|
||||
/** ISO date string (YYYY-MM-DD or full ISO timestamp). Filter to pages updated_at > value. */
|
||||
updated_after?: string;
|
||||
/**
|
||||
* Prefix-match filter on slug. Implemented as `WHERE slug LIKE prefix || '%'`
|
||||
* in both engines so it uses the (source_id, slug) UNIQUE constraint's btree
|
||||
* index for efficient range scans on large brains. Used by storage-tiering
|
||||
* commands (gbrain storage status, gbrain export --restore-only) to scope
|
||||
* queries to a tier directory without loading every page into memory.
|
||||
*/
|
||||
slugPrefix?: string;
|
||||
}
|
||||
|
||||
// Chunks
|
||||
@@ -78,21 +70,6 @@ export interface Chunk {
|
||||
symbol_name_qualified?: string | null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Lightweight row shape returned by `BrainEngine.listStaleChunks()`.
|
||||
* Excludes the `embedding` column on purpose — only chunks needing
|
||||
* an embedding come back, and we don't ship the (always-null on stale
|
||||
* rows) embedding bytes over the wire. See `embed --stale` egress fix.
|
||||
*/
|
||||
export interface StaleChunkRow {
|
||||
slug: string;
|
||||
chunk_index: number;
|
||||
chunk_text: string;
|
||||
chunk_source: 'compiled_truth' | 'timeline';
|
||||
model: string | null;
|
||||
token_count: number | null;
|
||||
}
|
||||
|
||||
export interface ChunkInput {
|
||||
chunk_index: number;
|
||||
chunk_text: string;
|
||||
@@ -145,18 +122,6 @@ export interface SearchOpts {
|
||||
offset?: number;
|
||||
type?: PageType;
|
||||
exclude_slugs?: string[];
|
||||
/**
|
||||
* Slug-prefix excludes — additive over DEFAULT_HARD_EXCLUDES (test/, archive/,
|
||||
* attachments/, .raw/) and the GBRAIN_SEARCH_EXCLUDE env var. Stacks with
|
||||
* `exclude_slugs` (exact match) — a row is filtered if it matches either set.
|
||||
*/
|
||||
exclude_slug_prefixes?: string[];
|
||||
/**
|
||||
* Opt-back-in list — subtracts entries from the resolved hard-exclude set.
|
||||
* E.g. `include_slug_prefixes: ['test/']` lets a query see test/ pages even
|
||||
* though they're hard-excluded by default.
|
||||
*/
|
||||
include_slug_prefixes?: string[];
|
||||
detail?: 'low' | 'medium' | 'high';
|
||||
/**
|
||||
* v0.20.0 Cathedral II: filter by content_chunks.language (e.g., 'typescript',
|
||||
|
||||
@@ -1,103 +0,0 @@
|
||||
/**
|
||||
* Shared MCP tool-call dispatch — single source of truth for stdio + HTTP transports.
|
||||
*
|
||||
* Both transports validate the same params, build the same OperationContext shape,
|
||||
* and serialize errors identically. Drift between transports caused PR #483's reversed-args
|
||||
* + missing-context bugs; this module exists to prevent that recurring.
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { operations, OperationError } from '../core/operations.ts';
|
||||
import type { Operation, OperationContext } from '../core/operations.ts';
|
||||
import { loadConfig } from '../core/config.ts';
|
||||
|
||||
export interface ToolResult {
|
||||
content: { type: 'text'; text: string }[];
|
||||
isError?: boolean;
|
||||
}
|
||||
|
||||
export interface DispatchOpts {
|
||||
/** Defaults to true (remote/untrusted). Local CLI callers (`gbrain call`) pass false. */
|
||||
remote?: boolean;
|
||||
/** Override the default stderr logger (e.g. CLI uses console.* directly). */
|
||||
logger?: OperationContext['logger'];
|
||||
}
|
||||
|
||||
/** Validate required params exist and have the expected type. Returns null on success, error message on failure. */
|
||||
export function validateParams(op: Operation, params: Record<string, unknown>): string | null {
|
||||
for (const [key, def] of Object.entries(op.params)) {
|
||||
if (def.required && (params[key] === undefined || params[key] === null)) {
|
||||
return `Missing required parameter: ${key}`;
|
||||
}
|
||||
if (params[key] !== undefined && params[key] !== null) {
|
||||
const val = params[key];
|
||||
const expected = def.type;
|
||||
if (expected === 'string' && typeof val !== 'string') return `Parameter "${key}" must be a string`;
|
||||
if (expected === 'number' && typeof val !== 'number') return `Parameter "${key}" must be a number`;
|
||||
if (expected === 'boolean' && typeof val !== 'boolean') return `Parameter "${key}" must be a boolean`;
|
||||
if (expected === 'object' && (typeof val !== 'object' || Array.isArray(val))) return `Parameter "${key}" must be an object`;
|
||||
if (expected === 'array' && !Array.isArray(val)) return `Parameter "${key}" must be an array`;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
const stderrLogger: OperationContext['logger'] = {
|
||||
info: (msg: string) => process.stderr.write(`[info] ${msg}\n`),
|
||||
warn: (msg: string) => process.stderr.write(`[warn] ${msg}\n`),
|
||||
error: (msg: string) => process.stderr.write(`[error] ${msg}\n`),
|
||||
};
|
||||
|
||||
export function buildOperationContext(
|
||||
engine: BrainEngine,
|
||||
params: Record<string, unknown>,
|
||||
opts: DispatchOpts = {},
|
||||
): OperationContext {
|
||||
return {
|
||||
engine,
|
||||
config: loadConfig() || { engine: 'postgres' },
|
||||
logger: opts.logger || stderrLogger,
|
||||
dryRun: !!params.dry_run,
|
||||
remote: opts.remote ?? true,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve operation, validate params, build context, invoke handler, format result.
|
||||
*
|
||||
* Returns a `ToolResult` with the same shape both MCP transports need:
|
||||
* `{ content: [{ type: 'text', text }], isError?: boolean }`.
|
||||
*/
|
||||
export async function dispatchToolCall(
|
||||
engine: BrainEngine,
|
||||
name: string,
|
||||
params: Record<string, unknown> | undefined,
|
||||
opts: DispatchOpts = {},
|
||||
): Promise<ToolResult> {
|
||||
const op = operations.find(o => o.name === name);
|
||||
if (!op) {
|
||||
return { content: [{ type: 'text', text: `Error: Unknown tool: ${name}` }], isError: true };
|
||||
}
|
||||
|
||||
const safeParams = params || {};
|
||||
const validationError = validateParams(op, safeParams);
|
||||
if (validationError) {
|
||||
return {
|
||||
content: [{ type: 'text', text: JSON.stringify({ error: 'invalid_params', message: validationError }, null, 2) }],
|
||||
isError: true,
|
||||
};
|
||||
}
|
||||
|
||||
const ctx = buildOperationContext(engine, safeParams, opts);
|
||||
|
||||
try {
|
||||
const result = await op.handler(ctx, safeParams);
|
||||
return { content: [{ type: 'text', text: JSON.stringify(result, null, 2) }] };
|
||||
} catch (e: unknown) {
|
||||
if (e instanceof OperationError) {
|
||||
return { content: [{ type: 'text', text: JSON.stringify(e.toJSON(), null, 2) }], isError: true };
|
||||
}
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
return { content: [{ type: 'text', text: `Error: ${msg}` }], isError: true };
|
||||
}
|
||||
}
|
||||
@@ -1,354 +0,0 @@
|
||||
/**
|
||||
* HTTP transport for `gbrain serve --http`.
|
||||
*
|
||||
* Postgres-only. PGLite users get a clear fail-fast at startup (the access_tokens
|
||||
* table doesn't exist on PGLite per pglite-schema.ts).
|
||||
*
|
||||
* Security model:
|
||||
* - Every request must include `Authorization: Bearer <token>` (except /health)
|
||||
* - Tokens are validated against SHA-256 hashes in the access_tokens table
|
||||
* - Create/manage tokens with auth.ts (gbrain auth create/list/revoke)
|
||||
* - No open OAuth, no client_credentials, no self-service tokens
|
||||
*
|
||||
* Hardening:
|
||||
* - CORS default-deny: allowlist via GBRAIN_HTTP_CORS_ORIGIN (comma-separated)
|
||||
* - Rate limit: per-IP pre-auth (protects DB from brute-force load) + per-token-id post-auth
|
||||
* (limits runaway clients). Default 30 req/min per IP, 60 req/min per token. Bounded LRU
|
||||
* so attacker-controlled keys can't grow memory unbounded.
|
||||
* - Body cap: 1 MiB default (GBRAIN_HTTP_MAX_BODY_BYTES). Stream-counted, not buffered —
|
||||
* chunked transfers without Content-Length are still capped.
|
||||
* - last_used_at debounce: only one UPDATE per token per 60s (SQL-level WHERE clause).
|
||||
* - mcp_request_log: one row per request with token_name + operation + status + latency.
|
||||
*
|
||||
* Replaces the standalone HTTP+OAuth wrapper that was vulnerable to unauthenticated
|
||||
* client registration (see SECURITY.md).
|
||||
*/
|
||||
|
||||
import { createHash } from 'crypto';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { buildToolDefs } from './tool-defs.ts';
|
||||
import { operations } from '../core/operations.ts';
|
||||
import { VERSION } from '../version.ts';
|
||||
import { dispatchToolCall } from './dispatch.ts';
|
||||
import { buildDefaultLimiters, type RateLimiter } from './rate-limit.ts';
|
||||
|
||||
const DEFAULT_BODY_CAP = 1024 * 1024; // 1 MiB
|
||||
|
||||
function hashToken(token: string): string {
|
||||
return createHash('sha256').update(token).digest('hex');
|
||||
}
|
||||
|
||||
function envInt(name: string, fallback: number): number {
|
||||
const v = process.env[name];
|
||||
if (!v) return fallback;
|
||||
const n = parseInt(v, 10);
|
||||
return Number.isFinite(n) && n > 0 ? n : fallback;
|
||||
}
|
||||
|
||||
function parseCorsAllowlist(): Set<string> | null {
|
||||
const v = process.env.GBRAIN_HTTP_CORS_ORIGIN;
|
||||
if (!v) return null;
|
||||
return new Set(v.split(',').map(s => s.trim()).filter(Boolean));
|
||||
}
|
||||
|
||||
interface HttpTransportOptions {
|
||||
port: number;
|
||||
engine: BrainEngine;
|
||||
/** Override limiters (for tests). Defaults to env-driven buildDefaultLimiters. */
|
||||
limiters?: { ip: RateLimiter; token: RateLimiter };
|
||||
}
|
||||
|
||||
interface AuthResult {
|
||||
ok: boolean;
|
||||
tokenId?: string;
|
||||
tokenName?: string;
|
||||
}
|
||||
|
||||
/** Read up to `cap` bytes off req.body. Returns null if cap exceeded. */
|
||||
async function readBodyWithCap(req: Request, cap: number): Promise<string | null> {
|
||||
const cl = req.headers.get('content-length');
|
||||
if (cl) {
|
||||
const n = parseInt(cl, 10);
|
||||
if (Number.isFinite(n) && n > cap) return null;
|
||||
}
|
||||
const reader = req.body?.getReader();
|
||||
if (!reader) return '';
|
||||
const chunks: Uint8Array[] = [];
|
||||
let total = 0;
|
||||
while (true) {
|
||||
const { value, done } = await reader.read();
|
||||
if (done) break;
|
||||
if (!value) continue;
|
||||
total += value.byteLength;
|
||||
if (total > cap) {
|
||||
try { await reader.cancel(); } catch { /* noop */ }
|
||||
return null;
|
||||
}
|
||||
chunks.push(value);
|
||||
}
|
||||
// Concatenate without Buffer to keep this Node-vs-Bun-portable.
|
||||
const merged = new Uint8Array(total);
|
||||
let offset = 0;
|
||||
for (const c of chunks) {
|
||||
merged.set(c, offset);
|
||||
offset += c.byteLength;
|
||||
}
|
||||
return new TextDecoder().decode(merged);
|
||||
}
|
||||
|
||||
/** Resolve client IP. Honors X-Forwarded-For only when GBRAIN_HTTP_TRUST_PROXY=1. */
|
||||
function resolveClientIp(req: Request, server: { requestIP: (r: Request) => { address: string } | null }): string {
|
||||
if (process.env.GBRAIN_HTTP_TRUST_PROXY === '1') {
|
||||
const xff = req.headers.get('x-forwarded-for');
|
||||
if (xff) {
|
||||
const first = xff.split(',')[0]?.trim();
|
||||
if (first) return first;
|
||||
}
|
||||
const xRealIp = req.headers.get('x-real-ip');
|
||||
if (xRealIp) return xRealIp.trim();
|
||||
}
|
||||
const sock = server.requestIP(req);
|
||||
return sock?.address || 'unknown';
|
||||
}
|
||||
|
||||
export async function startHttpTransport(opts: HttpTransportOptions) {
|
||||
const { port, engine } = opts;
|
||||
|
||||
// Fail-fast: HTTP transport requires Postgres because access_tokens / mcp_request_log
|
||||
// only exist in the Postgres schema (see src/core/pglite-schema.ts:5-6).
|
||||
if ((engine as { kind?: string }).kind !== 'postgres') {
|
||||
console.error('Error: gbrain serve --http requires a Postgres engine for remote auth tokens.');
|
||||
console.error('PGLite is local-only by design (access_tokens table is Postgres-only).');
|
||||
console.error('Either:');
|
||||
console.error(' - Use stdio: gbrain serve');
|
||||
console.error(' - Migrate to Postgres: gbrain migrate --to supabase');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const sql = (engine as unknown as { sql: any }).sql;
|
||||
if (!sql) {
|
||||
console.error('Error: Postgres engine has no .sql client. Engine may not be connected.');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const limiters = opts.limiters || buildDefaultLimiters();
|
||||
const bodyCap = envInt('GBRAIN_HTTP_MAX_BODY_BYTES', DEFAULT_BODY_CAP);
|
||||
const corsAllowlist = parseCorsAllowlist();
|
||||
const tools = buildToolDefs(operations);
|
||||
|
||||
function corsHeaders(origin: string | null, extra: Record<string, string> = {}): Record<string, string> {
|
||||
const headers: Record<string, string> = { ...extra };
|
||||
if (corsAllowlist && origin && corsAllowlist.has(origin)) {
|
||||
headers['Access-Control-Allow-Origin'] = origin;
|
||||
headers['Vary'] = 'Origin';
|
||||
}
|
||||
return headers;
|
||||
}
|
||||
|
||||
function corsPreflightHeaders(origin: string | null): Record<string, string> {
|
||||
const headers: Record<string, string> = {
|
||||
'Access-Control-Allow-Methods': 'POST, GET, OPTIONS',
|
||||
'Access-Control-Allow-Headers': 'Content-Type, Authorization, Accept',
|
||||
};
|
||||
if (corsAllowlist && origin && corsAllowlist.has(origin)) {
|
||||
headers['Access-Control-Allow-Origin'] = origin;
|
||||
headers['Vary'] = 'Origin';
|
||||
}
|
||||
return headers;
|
||||
}
|
||||
|
||||
async function validateToken(authHeader: string | null): Promise<AuthResult> {
|
||||
if (!authHeader?.startsWith('Bearer ')) return { ok: false };
|
||||
const token = authHeader.slice(7);
|
||||
const hash = hashToken(token);
|
||||
try {
|
||||
const [row] = await sql`
|
||||
SELECT id, name FROM access_tokens
|
||||
WHERE token_hash = ${hash} AND revoked_at IS NULL
|
||||
`;
|
||||
if (!row) return { ok: false };
|
||||
// Debounced last_used_at update — only writes once per token per 60s.
|
||||
// SQL-level WHERE clause keeps this race-tolerant even under concurrent requests.
|
||||
sql`UPDATE access_tokens
|
||||
SET last_used_at = now()
|
||||
WHERE id = ${row.id}
|
||||
AND (last_used_at IS NULL OR last_used_at < now() - interval '60 seconds')`
|
||||
.catch(() => { /* fire-and-forget */ });
|
||||
return { ok: true, tokenId: row.id, tokenName: row.name };
|
||||
} catch {
|
||||
return { ok: false };
|
||||
}
|
||||
}
|
||||
|
||||
function logRequest(tokenName: string | null, operation: string, status: string, latencyMs: number) {
|
||||
sql`INSERT INTO mcp_request_log (token_name, operation, latency_ms, status)
|
||||
VALUES (${tokenName}, ${operation}, ${latencyMs}, ${status})`
|
||||
.catch(() => { /* best-effort */ });
|
||||
}
|
||||
|
||||
const server = Bun.serve({
|
||||
port,
|
||||
async fetch(req, server) {
|
||||
const startedMs = Date.now();
|
||||
const url = new URL(req.url);
|
||||
const path = url.pathname;
|
||||
const origin = req.headers.get('origin');
|
||||
|
||||
// CORS preflight
|
||||
if (req.method === 'OPTIONS') {
|
||||
return new Response(null, { headers: corsPreflightHeaders(origin) });
|
||||
}
|
||||
|
||||
// Health check — no auth, no rate limit. Probes the DB so orchestration
|
||||
// doesn't see "ok" while clients are getting misleading 401s during a DB outage.
|
||||
if (path === '/health') {
|
||||
try {
|
||||
await sql`SELECT 1`;
|
||||
return Response.json(
|
||||
{ status: 'ok', version: VERSION, transport: 'http', db: 'ok' },
|
||||
{ headers: corsHeaders(origin) },
|
||||
);
|
||||
} catch (e: any) {
|
||||
return Response.json(
|
||||
{ status: 'unhealthy', version: VERSION, transport: 'http', db: 'unreachable', error: e?.message ?? 'unknown' },
|
||||
{ status: 503, headers: corsHeaders(origin) },
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
if (path !== '/mcp') {
|
||||
return Response.json({ error: 'not_found' }, { status: 404, headers: corsHeaders(origin) });
|
||||
}
|
||||
if (req.method !== 'POST') {
|
||||
return Response.json({ error: 'method_not_allowed' }, { status: 405, headers: corsHeaders(origin) });
|
||||
}
|
||||
|
||||
const ip = resolveClientIp(req, server);
|
||||
|
||||
// Pre-auth IP rate limit. Fires BEFORE the DB lookup so we actually limit brute-force load.
|
||||
const ipCheck = limiters.ip.check(ip);
|
||||
if (!ipCheck.allowed) {
|
||||
logRequest(null, 'unknown', 'rate_limited', Date.now() - startedMs);
|
||||
return Response.json(
|
||||
{ error: 'rate_limited', message: 'Too many requests' },
|
||||
{
|
||||
status: 429,
|
||||
headers: corsHeaders(origin, { 'Retry-After': String(ipCheck.retryAfter ?? 60) }),
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
// Body cap (stream-counted; chunked transfers caught here, not at req.json).
|
||||
const bodyText = await readBodyWithCap(req, bodyCap);
|
||||
if (bodyText === null) {
|
||||
logRequest(null, 'unknown', 'body_too_large', Date.now() - startedMs);
|
||||
return Response.json(
|
||||
{ error: 'payload_too_large', message: `Request body exceeds ${bodyCap} bytes` },
|
||||
{ status: 413, headers: corsHeaders(origin) },
|
||||
);
|
||||
}
|
||||
|
||||
// Auth.
|
||||
const auth = await validateToken(req.headers.get('Authorization'));
|
||||
if (!auth.ok) {
|
||||
logRequest(null, 'unknown', 'auth_failed', Date.now() - startedMs);
|
||||
return Response.json(
|
||||
{ error: 'invalid_token', message: 'Bearer token required. Create one: gbrain auth create <name>' },
|
||||
{ status: 401, headers: corsHeaders(origin) },
|
||||
);
|
||||
}
|
||||
|
||||
// Post-auth token-id rate limit. Limits runaway authed clients.
|
||||
const tokCheck = limiters.token.check(auth.tokenId!);
|
||||
if (!tokCheck.allowed) {
|
||||
logRequest(auth.tokenName!, 'unknown', 'rate_limited', Date.now() - startedMs);
|
||||
return Response.json(
|
||||
{ error: 'rate_limited', message: 'Too many requests for this token' },
|
||||
{
|
||||
status: 429,
|
||||
headers: corsHeaders(origin, { 'Retry-After': String(tokCheck.retryAfter ?? 60) }),
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
// Parse JSON-RPC body.
|
||||
let body: { method?: string; params?: any; id?: any };
|
||||
try {
|
||||
body = JSON.parse(bodyText);
|
||||
} catch (e: any) {
|
||||
logRequest(auth.tokenName!, 'unknown', 'parse_error', Date.now() - startedMs);
|
||||
return Response.json(
|
||||
{ error: 'parse_error', message: e?.message ?? 'invalid JSON' },
|
||||
{ status: 400, headers: corsHeaders(origin) },
|
||||
);
|
||||
}
|
||||
|
||||
const { method, params, id } = body;
|
||||
|
||||
// initialize
|
||||
if (method === 'initialize') {
|
||||
logRequest(auth.tokenName!, 'initialize', 'success', Date.now() - startedMs);
|
||||
return Response.json(
|
||||
{
|
||||
result: {
|
||||
protocolVersion: '2025-03-26',
|
||||
serverInfo: { name: 'gbrain', version: VERSION },
|
||||
capabilities: { tools: {} },
|
||||
},
|
||||
jsonrpc: '2.0',
|
||||
id,
|
||||
},
|
||||
{ headers: corsHeaders(origin) },
|
||||
);
|
||||
}
|
||||
|
||||
// notifications/initialized — acknowledge with 204
|
||||
if (method === 'notifications/initialized') {
|
||||
return new Response(null, { status: 204, headers: corsHeaders(origin) });
|
||||
}
|
||||
|
||||
// tools/list
|
||||
if (method === 'tools/list') {
|
||||
logRequest(auth.tokenName!, 'tools/list', 'success', Date.now() - startedMs);
|
||||
return Response.json(
|
||||
{ result: { tools }, jsonrpc: '2.0', id },
|
||||
{ headers: corsHeaders(origin) },
|
||||
);
|
||||
}
|
||||
|
||||
// tools/call — dispatch through shared dispatch.ts (parity with stdio)
|
||||
if (method === 'tools/call') {
|
||||
const toolName: string = params?.name ?? 'unknown';
|
||||
const args: Record<string, unknown> = params?.arguments ?? {};
|
||||
const result = await dispatchToolCall(engine, toolName, args, { remote: true });
|
||||
const status = result.isError ? 'error' : 'success';
|
||||
logRequest(auth.tokenName!, `tools/call:${toolName}`, status, Date.now() - startedMs);
|
||||
return Response.json(
|
||||
{ result, jsonrpc: '2.0', id },
|
||||
{ headers: corsHeaders(origin) },
|
||||
);
|
||||
}
|
||||
|
||||
logRequest(auth.tokenName!, method ?? 'unknown', 'unknown_method', Date.now() - startedMs);
|
||||
return Response.json(
|
||||
{ error: 'unknown_method', message: `Unknown method: ${method}` },
|
||||
{ status: 400, headers: corsHeaders(origin) },
|
||||
);
|
||||
},
|
||||
});
|
||||
|
||||
console.error(`GBrain HTTP MCP server running on port ${port}`);
|
||||
console.error(` Health: http://localhost:${port}/health`);
|
||||
console.error(` MCP: http://localhost:${port}/mcp`);
|
||||
console.error(` Auth: Bearer token required (create with: gbrain auth create <name>)`);
|
||||
if (!corsAllowlist) {
|
||||
console.error(' CORS: default-deny. Set GBRAIN_HTTP_CORS_ORIGIN=https://your.app to allow browser clients.');
|
||||
} else {
|
||||
console.error(` CORS: allowlist = ${[...corsAllowlist].join(', ')}`);
|
||||
}
|
||||
console.error('');
|
||||
console.error('⚠️ Do NOT use open OAuth registration for remote MCP access.');
|
||||
console.error(' Tokens are managed via: gbrain auth create/list/revoke');
|
||||
|
||||
return server;
|
||||
}
|
||||
@@ -1,142 +0,0 @@
|
||||
/**
|
||||
* Rate limiter for `gbrain serve --http`.
|
||||
*
|
||||
* Token-bucket per key, stored in a bounded LRU map so attacker-controlled keys
|
||||
* can't grow memory unbounded. TTL prune on every access (entries older than
|
||||
* 2× window are evicted) so abandoned keys don't sit around forever.
|
||||
*
|
||||
* Two buckets in the request pipeline (see http-transport.ts):
|
||||
* 1. Pre-auth IP bucket — fires BEFORE the DB lookup so we actually limit
|
||||
* brute-force load against access_tokens, not just response codes.
|
||||
* 2. Post-auth token-id bucket — fires after auth so legitimate-but-runaway
|
||||
* clients get throttled at the right principal.
|
||||
*
|
||||
* Both buckets behave identically; only the key differs.
|
||||
*/
|
||||
|
||||
export interface RateLimitOpts {
|
||||
/** Maximum requests in the window. */
|
||||
limit: number;
|
||||
/** Window length in milliseconds. */
|
||||
windowMs: number;
|
||||
/** LRU cap on distinct keys. Evicts least-recently-used on overflow. */
|
||||
lruCap: number;
|
||||
}
|
||||
|
||||
export interface RateLimitResult {
|
||||
allowed: boolean;
|
||||
/** Seconds until next request would be allowed (only set when !allowed). */
|
||||
retryAfter?: number;
|
||||
/** Tokens remaining in the bucket after this check. */
|
||||
remaining: number;
|
||||
}
|
||||
|
||||
interface Bucket {
|
||||
tokens: number;
|
||||
/** Used for refill math: tokens accrue based on elapsed time since this. */
|
||||
lastRefillMs: number;
|
||||
/** Used for TTL eviction: time of last check, regardless of refill. Prevents bucket-reset attack
|
||||
* where an exhausted key would otherwise get TTL-evicted and recreated fresh. */
|
||||
lastTouchedMs: number;
|
||||
}
|
||||
|
||||
/** Clock function — defaults to Date.now, overridable for tests. */
|
||||
type Clock = () => number;
|
||||
|
||||
export class RateLimiter {
|
||||
readonly opts: RateLimitOpts;
|
||||
private readonly buckets: Map<string, Bucket> = new Map();
|
||||
private readonly clock: Clock;
|
||||
|
||||
constructor(opts: RateLimitOpts, clock: Clock = Date.now) {
|
||||
if (opts.limit <= 0) throw new Error('RateLimiter: limit must be > 0');
|
||||
if (opts.windowMs <= 0) throw new Error('RateLimiter: windowMs must be > 0');
|
||||
if (opts.lruCap <= 0) throw new Error('RateLimiter: lruCap must be > 0');
|
||||
this.opts = opts;
|
||||
this.clock = clock;
|
||||
}
|
||||
|
||||
check(key: string): RateLimitResult {
|
||||
const now = this.clock();
|
||||
this.prune(now);
|
||||
|
||||
let bucket = this.buckets.get(key);
|
||||
if (!bucket) {
|
||||
bucket = { tokens: this.opts.limit, lastRefillMs: now, lastTouchedMs: now };
|
||||
} else {
|
||||
// Refill: tokens accrue continuously over the window. limit/windowMs tokens per ms.
|
||||
const elapsed = now - bucket.lastRefillMs;
|
||||
const refilled = Math.floor((elapsed * this.opts.limit) / this.opts.windowMs);
|
||||
if (refilled > 0) {
|
||||
bucket.tokens = Math.min(this.opts.limit, bucket.tokens + refilled);
|
||||
bucket.lastRefillMs = now;
|
||||
}
|
||||
bucket.lastTouchedMs = now;
|
||||
// LRU bookkeeping: re-insert to move to end (Map iteration order = insertion).
|
||||
this.buckets.delete(key);
|
||||
}
|
||||
|
||||
if (bucket.tokens > 0) {
|
||||
bucket.tokens -= 1;
|
||||
this.buckets.set(key, bucket);
|
||||
this.evictIfOver();
|
||||
return { allowed: true, remaining: bucket.tokens };
|
||||
}
|
||||
|
||||
// No tokens. Compute Retry-After from when the next token will accrue.
|
||||
const msPerToken = this.opts.windowMs / this.opts.limit;
|
||||
const msUntilNext = msPerToken - (now - bucket.lastRefillMs);
|
||||
const retryAfter = Math.max(1, Math.ceil(msUntilNext / 1000));
|
||||
this.buckets.set(key, bucket);
|
||||
this.evictIfOver();
|
||||
return { allowed: false, retryAfter, remaining: 0 };
|
||||
}
|
||||
|
||||
/** Evict TTL-expired entries (older than 2× window since last touch). Cheap: O(n) but n is bounded by lruCap.
|
||||
* Uses lastTouchedMs (not lastRefillMs) so an attacker can't reset their bucket by hammering an exhausted key
|
||||
* past the TTL — every check updates lastTouchedMs even when refill produces 0 tokens. */
|
||||
private prune(now: number): void {
|
||||
const ttl = this.opts.windowMs * 2;
|
||||
for (const [key, bucket] of this.buckets) {
|
||||
if (now - bucket.lastTouchedMs > ttl) {
|
||||
this.buckets.delete(key);
|
||||
} else {
|
||||
// Map iteration is in insertion order; once we hit a fresh entry, the rest are also fresh
|
||||
// ONLY if we maintain insertion-order = recency. That holds because check() does delete+set on every call.
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private evictIfOver(): void {
|
||||
while (this.buckets.size > this.opts.lruCap) {
|
||||
// Map iteration starts at oldest (first-inserted). Delete it.
|
||||
const oldestKey = this.buckets.keys().next().value;
|
||||
if (oldestKey === undefined) break;
|
||||
this.buckets.delete(oldestKey);
|
||||
}
|
||||
}
|
||||
|
||||
/** Test helper: current key count. */
|
||||
get size(): number {
|
||||
return this.buckets.size;
|
||||
}
|
||||
}
|
||||
|
||||
/** Parse a positive integer env var, falling back to default. */
|
||||
function envInt(name: string, fallback: number): number {
|
||||
const v = process.env[name];
|
||||
if (!v) return fallback;
|
||||
const n = parseInt(v, 10);
|
||||
return Number.isFinite(n) && n > 0 ? n : fallback;
|
||||
}
|
||||
|
||||
/** Build limiters from env. Keep this lazy — tests can construct RateLimiter directly. */
|
||||
export function buildDefaultLimiters(clock: Clock = Date.now): { ip: RateLimiter; token: RateLimiter } {
|
||||
const lruCap = envInt('GBRAIN_HTTP_RATE_LIMIT_LRU', 10000);
|
||||
const windowMs = 60_000;
|
||||
return {
|
||||
ip: new RateLimiter({ limit: envInt('GBRAIN_HTTP_RATE_LIMIT_IP', 30), windowMs, lruCap }, clock),
|
||||
token: new RateLimiter({ limit: envInt('GBRAIN_HTTP_RATE_LIMIT_TOKEN', 60), windowMs, lruCap }, clock),
|
||||
};
|
||||
}
|
||||
+66
-13
@@ -2,10 +2,30 @@ import { Server } from '@modelcontextprotocol/sdk/server/index.js';
|
||||
import { StdioServerTransport } from '@modelcontextprotocol/sdk/server/stdio.js';
|
||||
import { ListToolsRequestSchema, CallToolRequestSchema } from '@modelcontextprotocol/sdk/types.js';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { operations } from '../core/operations.ts';
|
||||
import { operations, OperationError } from '../core/operations.ts';
|
||||
import type { Operation, OperationContext } from '../core/operations.ts';
|
||||
import { loadConfig } from '../core/config.ts';
|
||||
import { VERSION } from '../version.ts';
|
||||
import { buildToolDefs } from './tool-defs.ts';
|
||||
import { dispatchToolCall, validateParams, buildOperationContext } from './dispatch.ts';
|
||||
|
||||
/** Validate required params exist and have the expected type */
|
||||
function validateParams(op: Operation, params: Record<string, unknown>): string | null {
|
||||
for (const [key, def] of Object.entries(op.params)) {
|
||||
if (def.required && (params[key] === undefined || params[key] === null)) {
|
||||
return `Missing required parameter: ${key}`;
|
||||
}
|
||||
if (params[key] !== undefined && params[key] !== null) {
|
||||
const val = params[key];
|
||||
const expected = def.type;
|
||||
if (expected === 'string' && typeof val !== 'string') return `Parameter "${key}" must be a string`;
|
||||
if (expected === 'number' && typeof val !== 'number') return `Parameter "${key}" must be a number`;
|
||||
if (expected === 'boolean' && typeof val !== 'boolean') return `Parameter "${key}" must be a boolean`;
|
||||
if (expected === 'object' && (typeof val !== 'object' || Array.isArray(val))) return `Parameter "${key}" must be an object`;
|
||||
if (expected === 'array' && !Array.isArray(val)) return `Parameter "${key}" must be an array`;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
export async function startMcpServer(engine: BrainEngine) {
|
||||
const server = new Server(
|
||||
@@ -20,21 +40,50 @@ export async function startMcpServer(engine: BrainEngine) {
|
||||
tools: buildToolDefs(operations),
|
||||
}));
|
||||
|
||||
// Dispatch tool calls via shared dispatch.ts (parity with HTTP transport).
|
||||
// MCP stdio callers are remote/untrusted; dispatch defaults remote=true.
|
||||
// The MCP SDK's response type widened in 1.29 to allow a managed-task wrapper;
|
||||
// gbrain ops are synchronous, so we return the legacy `{ content, isError? }`
|
||||
// shape and cast through `any` (the SDK accepts it via the ServerResult union).
|
||||
server.setRequestHandler(CallToolRequestSchema, async (request: any): Promise<any> => {
|
||||
// Dispatch tool calls to operation handlers
|
||||
server.setRequestHandler(CallToolRequestSchema, async (request: any) => {
|
||||
const { name, arguments: params } = request.params;
|
||||
return dispatchToolCall(engine, name, params, { remote: true });
|
||||
const op = operations.find(o => o.name === name);
|
||||
if (!op) {
|
||||
return { content: [{ type: 'text', text: `Error: Unknown tool: ${name}` }], isError: true };
|
||||
}
|
||||
|
||||
const ctx: OperationContext = {
|
||||
engine,
|
||||
config: loadConfig() || { engine: 'postgres' },
|
||||
logger: {
|
||||
info: (msg: string) => process.stderr.write(`[info] ${msg}\n`),
|
||||
warn: (msg: string) => process.stderr.write(`[warn] ${msg}\n`),
|
||||
error: (msg: string) => process.stderr.write(`[error] ${msg}\n`),
|
||||
},
|
||||
dryRun: !!(params?.dry_run),
|
||||
// MCP stdio callers are remote/untrusted; enforce strict file confinement.
|
||||
remote: true,
|
||||
};
|
||||
|
||||
const safeParams = params || {};
|
||||
const validationError = validateParams(op, safeParams);
|
||||
if (validationError) {
|
||||
return { content: [{ type: 'text', text: JSON.stringify({ error: 'invalid_params', message: validationError }, null, 2) }], isError: true };
|
||||
}
|
||||
|
||||
try {
|
||||
const result = await op.handler(ctx, safeParams);
|
||||
return { content: [{ type: 'text', text: JSON.stringify(result, null, 2) }] };
|
||||
} catch (e: unknown) {
|
||||
if (e instanceof OperationError) {
|
||||
return { content: [{ type: 'text', text: JSON.stringify(e.toJSON(), null, 2) }], isError: true };
|
||||
}
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
return { content: [{ type: 'text', text: `Error: ${msg}` }], isError: true };
|
||||
}
|
||||
});
|
||||
|
||||
const transport = new StdioServerTransport();
|
||||
await server.connect(transport);
|
||||
}
|
||||
|
||||
// Backward compat: used by `gbrain call` command (trusted local path).
|
||||
// Backward compat: used by `gbrain call` command
|
||||
export async function handleToolCall(
|
||||
engine: BrainEngine,
|
||||
tool: string,
|
||||
@@ -46,10 +95,14 @@ export async function handleToolCall(
|
||||
const validationError = validateParams(op, params);
|
||||
if (validationError) throw new Error(validationError);
|
||||
|
||||
const ctx = buildOperationContext(engine, params, {
|
||||
remote: false,
|
||||
const ctx: OperationContext = {
|
||||
engine,
|
||||
config: loadConfig() || { engine: 'postgres' },
|
||||
logger: { info: console.log, warn: console.warn, error: console.error },
|
||||
});
|
||||
dryRun: !!(params?.dry_run),
|
||||
// Backing path for `gbrain call` CLI command — trusted local invocation.
|
||||
remote: false,
|
||||
};
|
||||
|
||||
return op.handler(ctx, params);
|
||||
}
|
||||
|
||||
@@ -108,7 +108,7 @@ describe('buildPlan — diff against completed + installed VERSION', () => {
|
||||
// autopilot cooperative, v0.16.0 = subagent runtime, v0.18.0 = multi-
|
||||
// source brains, v0.18.1 = RLS hardening, v0.21.0 = Cathedral II
|
||||
// (renumbered from v0.20.0 after master shipped v0.20.x in parallel).
|
||||
expect(plan.skippedFuture.map(m => m.version)).toEqual(['0.12.0', '0.12.2', '0.13.0', '0.13.1', '0.14.0', '0.16.0', '0.18.0', '0.18.1', '0.21.0', '0.22.4']);
|
||||
expect(plan.skippedFuture.map(m => m.version)).toEqual(['0.12.0', '0.12.2', '0.13.0', '0.13.1', '0.14.0', '0.16.0', '0.18.0', '0.18.1', '0.21.0']);
|
||||
});
|
||||
|
||||
test('already applied → v0.11.0 lands in `applied` bucket, not pending', () => {
|
||||
@@ -148,7 +148,7 @@ describe('buildPlan — diff against completed + installed VERSION', () => {
|
||||
// were added later; installed=0.12.0 means they belong in skippedFuture,
|
||||
// not pending. v0.11.0 and v0.12.0 stay pending despite being ≤ installed —
|
||||
// that is the H9 invariant.
|
||||
expect(plan.skippedFuture.map(m => m.version)).toEqual(['0.12.2', '0.13.0', '0.13.1', '0.14.0', '0.16.0', '0.18.0', '0.18.1', '0.21.0', '0.22.4']);
|
||||
expect(plan.skippedFuture.map(m => m.version)).toEqual(['0.12.2', '0.13.0', '0.13.1', '0.14.0', '0.16.0', '0.18.0', '0.18.1', '0.21.0']);
|
||||
});
|
||||
|
||||
test('--migration filter narrows to one version', () => {
|
||||
|
||||
@@ -1,192 +0,0 @@
|
||||
/**
|
||||
* PGLite forward-reference bootstrap tests.
|
||||
*
|
||||
* Validates the contract of `PGLiteEngine#applyForwardReferenceBootstrap`:
|
||||
* given a brain that lacks the schema-blob's forward-referenced state, the
|
||||
* bootstrap adds enough state for PGLITE_SCHEMA_SQL to replay safely.
|
||||
*
|
||||
* The bootstrap covers the wedge incidents from issues
|
||||
* #239/#266/#357/#366/#374/#375/#378/#396 — every gbrain release that added
|
||||
* a column-with-index in the schema blob without a corresponding bootstrap
|
||||
* triggered the same wedge family.
|
||||
*
|
||||
* Honest limitation: test 4 simulates a v20 brain by dropping known forward
|
||||
* state from a fresh-LATEST instance. This is the same down-mutation pattern
|
||||
* codex flagged as "weak simulation" — it can't simulate every possible
|
||||
* historical state. Acceptable here because the bootstrap's contract is
|
||||
* narrow ("given a brain that lacks the specific forward-references,
|
||||
* initSchema produces a brain at LATEST"), and that contract is exactly
|
||||
* what this test exercises.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { LATEST_VERSION } from '../src/core/migrate.ts';
|
||||
|
||||
describe('PGLiteEngine#applyForwardReferenceBootstrap', () => {
|
||||
test('no-op on fresh install (no pages or links table)', async () => {
|
||||
const engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
try {
|
||||
// Don't call initSchema — verify bootstrap alone does nothing on empty DB
|
||||
await (engine as any).applyForwardReferenceBootstrap();
|
||||
const { rows } = await (engine as any).db.query(`
|
||||
SELECT COUNT(*)::int AS c FROM information_schema.tables
|
||||
WHERE table_schema = 'public'
|
||||
`);
|
||||
expect(rows[0].c).toBe(0);
|
||||
} finally {
|
||||
await engine.disconnect();
|
||||
}
|
||||
}, 30000);
|
||||
|
||||
test('idempotent: calling twice produces same result', async () => {
|
||||
const engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
try {
|
||||
await engine.initSchema();
|
||||
const db = (engine as any).db;
|
||||
|
||||
// Mutate to pre-v0.18 shape: drop source_id and the sources FK target
|
||||
await db.exec(`
|
||||
ALTER TABLE pages DROP CONSTRAINT IF EXISTS pages_source_slug_key;
|
||||
ALTER TABLE pages ADD CONSTRAINT pages_slug_key UNIQUE (slug);
|
||||
DROP INDEX IF EXISTS idx_pages_source_id;
|
||||
ALTER TABLE pages DROP COLUMN IF EXISTS source_id;
|
||||
DROP TABLE IF EXISTS sources CASCADE;
|
||||
`);
|
||||
|
||||
// First call: applies bootstrap
|
||||
await (engine as any).applyForwardReferenceBootstrap();
|
||||
// Second call: must not error, must not duplicate state
|
||||
await (engine as any).applyForwardReferenceBootstrap();
|
||||
|
||||
const { rows: cols } = await db.query(`
|
||||
SELECT column_name FROM information_schema.columns
|
||||
WHERE table_name = 'pages' AND column_name = 'source_id'
|
||||
`);
|
||||
expect(cols).toHaveLength(1);
|
||||
|
||||
const { rows: src } = await db.query(`SELECT COUNT(*)::int AS c FROM sources`);
|
||||
expect(src[0].c).toBe(1); // 'default' seed not duplicated
|
||||
} finally {
|
||||
await engine.disconnect();
|
||||
}
|
||||
}, 30000);
|
||||
|
||||
test('no-op on modern brain (source_id and links provenance already present)', async () => {
|
||||
const engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
try {
|
||||
await engine.initSchema();
|
||||
const db = (engine as any).db;
|
||||
|
||||
const before = await db.query(`SELECT COUNT(*)::int AS c FROM sources`);
|
||||
await (engine as any).applyForwardReferenceBootstrap();
|
||||
const after = await db.query(`SELECT COUNT(*)::int AS c FROM sources`);
|
||||
|
||||
// Bootstrap probe should detect the brain is modern and skip the seed insert
|
||||
expect(after.rows[0].c).toBe(before.rows[0].c);
|
||||
} finally {
|
||||
await engine.disconnect();
|
||||
}
|
||||
}, 30000);
|
||||
|
||||
test('full path: pre-v0.18 brain reaches LATEST_VERSION via initSchema', async () => {
|
||||
const engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
try {
|
||||
await engine.initSchema();
|
||||
const db = (engine as any).db;
|
||||
|
||||
// Mutate to pre-v0.18 shape: strip the forward-referenced state.
|
||||
// Match the shape from #399's regression fixture; constraints first
|
||||
// (so dropping columns succeeds).
|
||||
await db.exec(`
|
||||
ALTER TABLE pages DROP CONSTRAINT IF EXISTS pages_source_slug_key;
|
||||
ALTER TABLE pages ADD CONSTRAINT pages_slug_key UNIQUE (slug);
|
||||
DROP INDEX IF EXISTS idx_pages_source_id;
|
||||
ALTER TABLE pages DROP COLUMN IF EXISTS source_id;
|
||||
DROP TABLE IF EXISTS sources CASCADE;
|
||||
ALTER TABLE links DROP CONSTRAINT IF EXISTS links_resolution_type_check;
|
||||
ALTER TABLE links DROP COLUMN IF EXISTS resolution_type;
|
||||
`);
|
||||
await engine.setConfig('version', '20');
|
||||
|
||||
// Path under test: bootstrap → SCHEMA_SQL → runMigrations
|
||||
await engine.initSchema();
|
||||
|
||||
expect(await engine.getConfig('version')).toBe(String(LATEST_VERSION));
|
||||
|
||||
const { rows: srcCol } = await db.query(`
|
||||
SELECT column_name FROM information_schema.columns
|
||||
WHERE table_name = 'pages' AND column_name = 'source_id'
|
||||
`);
|
||||
expect(srcCol).toHaveLength(1);
|
||||
|
||||
const { rows: defaultSrc } = await db.query(`SELECT id FROM sources WHERE id = 'default'`);
|
||||
expect(defaultSrc).toHaveLength(1);
|
||||
} finally {
|
||||
await engine.disconnect();
|
||||
}
|
||||
}, 30000);
|
||||
|
||||
test('fresh install regression: initSchema on empty DB produces LATEST', async () => {
|
||||
// The bootstrap's table-existence probe must not mis-classify "no table"
|
||||
// as "pre-v0.18 brain." Without the table-existence guard, the bootstrap
|
||||
// would call runMigrations against an empty DB and crash on
|
||||
// `relation "config" does not exist`. Regression test for that path.
|
||||
const engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
try {
|
||||
await engine.initSchema();
|
||||
expect(await engine.getConfig('version')).toBe(String(LATEST_VERSION));
|
||||
|
||||
const db = (engine as any).db;
|
||||
const pages = await db.query(`SELECT 1 FROM pages LIMIT 0`);
|
||||
const sources = await db.query(`SELECT 1 FROM sources LIMIT 0`);
|
||||
const config = await db.query(`SELECT 1 FROM config LIMIT 0`);
|
||||
expect(pages).toBeDefined();
|
||||
expect(sources).toBeDefined();
|
||||
expect(config).toBeDefined();
|
||||
} finally {
|
||||
await engine.disconnect();
|
||||
}
|
||||
}, 30000);
|
||||
|
||||
test('pre-v0.13 links shape: bootstrap adds link_source + origin_page_id', async () => {
|
||||
// Issues #266 / #357 — pre-v0.13 brains had `links` without
|
||||
// `link_source` / `origin_page_id`. Schema blob's
|
||||
// `CREATE INDEX idx_links_source` would crash before v11 ran.
|
||||
const engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
try {
|
||||
await engine.initSchema();
|
||||
const db = (engine as any).db;
|
||||
|
||||
await db.exec(`
|
||||
DROP INDEX IF EXISTS idx_links_source;
|
||||
DROP INDEX IF EXISTS idx_links_origin;
|
||||
ALTER TABLE links DROP CONSTRAINT IF EXISTS links_from_to_type_source_origin_unique;
|
||||
ALTER TABLE links DROP COLUMN IF EXISTS link_source;
|
||||
ALTER TABLE links DROP COLUMN IF EXISTS origin_page_id;
|
||||
`);
|
||||
|
||||
await (engine as any).applyForwardReferenceBootstrap();
|
||||
|
||||
const { rows: lsCol } = await db.query(`
|
||||
SELECT column_name FROM information_schema.columns
|
||||
WHERE table_name = 'links' AND column_name = 'link_source'
|
||||
`);
|
||||
expect(lsCol).toHaveLength(1);
|
||||
|
||||
const { rows: opCol } = await db.query(`
|
||||
SELECT column_name FROM information_schema.columns
|
||||
WHERE table_name = 'links' AND column_name = 'origin_page_id'
|
||||
`);
|
||||
expect(opCol).toHaveLength(1);
|
||||
} finally {
|
||||
await engine.disconnect();
|
||||
}
|
||||
}, 30000);
|
||||
});
|
||||
@@ -1,229 +0,0 @@
|
||||
import { describe, expect, test, beforeAll, afterAll, beforeEach, afterEach } from 'bun:test';
|
||||
import { mkdtempSync, rmSync, writeFileSync, readFileSync, existsSync, mkdirSync, symlinkSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import {
|
||||
autoFixFrontmatter,
|
||||
writeBrainPage,
|
||||
scanBrainSources,
|
||||
BrainWriterError,
|
||||
} from '../src/core/brain-writer.ts';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
|
||||
const fence = '---';
|
||||
|
||||
describe('autoFixFrontmatter', () => {
|
||||
test('strips null bytes', () => {
|
||||
const input = `${fence}\ntitle: ok\n${fence}\n\nbody\x00drop\x00here`;
|
||||
const { content, fixes } = autoFixFrontmatter(input);
|
||||
expect(content.includes('\x00')).toBe(false);
|
||||
expect(fixes.some(f => f.code === 'NULL_BYTES')).toBe(true);
|
||||
});
|
||||
|
||||
test('inserts closing --- before heading when MISSING_CLOSE', () => {
|
||||
const input = `${fence}\ntype: concept\ntitle: ok\n# A heading\n\nbody`;
|
||||
const { content, fixes } = autoFixFrontmatter(input);
|
||||
expect(fixes.some(f => f.code === 'MISSING_CLOSE')).toBe(true);
|
||||
// After fix, parsing should find a closing --- before the heading.
|
||||
const idxClose = content.indexOf('---', 3);
|
||||
const idxHeading = content.indexOf('# A heading');
|
||||
expect(idxClose).toBeGreaterThan(0);
|
||||
expect(idxClose).toBeLessThan(idxHeading);
|
||||
});
|
||||
|
||||
test('rewrites nested-quote title to single-quoted', () => {
|
||||
const input = `${fence}\ntype: concept\ntitle: "Phil "Nick" Last"\n${fence}\n\nbody`;
|
||||
const { content, fixes } = autoFixFrontmatter(input);
|
||||
expect(fixes.some(f => f.code === 'NESTED_QUOTES')).toBe(true);
|
||||
// Outer wrapper is now single quotes.
|
||||
expect(content).toMatch(/^title: '.*'\s*$/m);
|
||||
});
|
||||
|
||||
test('removes mismatched slug field', () => {
|
||||
const input = `${fence}\ntype: concept\ntitle: hi\nslug: wrong-slug\n${fence}\n\nbody`;
|
||||
const { content, fixes } = autoFixFrontmatter(input, { filePath: 'people/jane-doe.md' });
|
||||
expect(fixes.some(f => f.code === 'SLUG_MISMATCH')).toBe(true);
|
||||
expect(content).not.toMatch(/^slug:/m);
|
||||
});
|
||||
|
||||
test('idempotent: running twice produces no diff and no fixes on second pass', () => {
|
||||
const input = `${fence}\ntype: concept\ntitle: "Phil "Nick" Last"\n${fence}\n\nbody\x00`;
|
||||
const first = autoFixFrontmatter(input);
|
||||
const second = autoFixFrontmatter(first.content);
|
||||
expect(second.content).toBe(first.content);
|
||||
expect(second.fixes).toEqual([]);
|
||||
});
|
||||
|
||||
test('clean input: no fixes, content unchanged', () => {
|
||||
const input = `${fence}\ntype: concept\ntitle: ok\n${fence}\n\nbody`;
|
||||
const { content, fixes } = autoFixFrontmatter(input);
|
||||
expect(content).toBe(input);
|
||||
expect(fixes).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('writeBrainPage', () => {
|
||||
let tmp: string;
|
||||
|
||||
beforeEach(() => {
|
||||
tmp = mkdtempSync(join(tmpdir(), 'brain-writer-'));
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
rmSync(tmp, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
test('happy path: writes file inside source', () => {
|
||||
const file = join(tmp, 'people', 'jane.md');
|
||||
const content = `${fence}\ntype: person\ntitle: Jane\n${fence}\n\nhello`;
|
||||
writeBrainPage(file, content, { sourcePath: tmp });
|
||||
expect(readFileSync(file, 'utf8')).toBe(content);
|
||||
});
|
||||
|
||||
test('throws BrainWriterError when path is outside sourcePath', () => {
|
||||
const elsewhere = mkdtempSync(join(tmpdir(), 'brain-writer-other-'));
|
||||
try {
|
||||
const offending = join(elsewhere, 'evil.md');
|
||||
expect(() =>
|
||||
writeBrainPage(offending, 'content', { sourcePath: tmp }),
|
||||
).toThrow(BrainWriterError);
|
||||
} finally {
|
||||
rmSync(elsewhere, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('writes .bak before mutating an existing file', () => {
|
||||
const file = join(tmp, 'people', 'jane.md');
|
||||
mkdirSync(join(tmp, 'people'), { recursive: true });
|
||||
const original = `${fence}\ntype: person\ntitle: Old\n${fence}\n\nold`;
|
||||
writeFileSync(file, original);
|
||||
writeBrainPage(file, `${fence}\ntype: person\ntitle: New\n${fence}\n\nnew`, { sourcePath: tmp });
|
||||
expect(existsSync(file + '.bak')).toBe(true);
|
||||
expect(readFileSync(file + '.bak', 'utf8')).toBe(original);
|
||||
});
|
||||
|
||||
test('autoFix: true repairs nested quotes before writing', () => {
|
||||
const file = join(tmp, 'people', 'jane.md');
|
||||
const broken = `${fence}\ntype: person\ntitle: "Phil "Nick" Last"\n${fence}\n\nbody`;
|
||||
const { fixes } = writeBrainPage(file, broken, { sourcePath: tmp, autoFix: true });
|
||||
expect(fixes.some(f => f.code === 'NESTED_QUOTES')).toBe(true);
|
||||
expect(readFileSync(file, 'utf8')).toMatch(/^title: '.*'\s*$/m);
|
||||
});
|
||||
});
|
||||
|
||||
describe('scanBrainSources (PGLite)', () => {
|
||||
let tmp: string;
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
// One PGLite per file — beforeEach wipes data only. PGLite cold-start is
|
||||
// ~20s on CI; sharing one engine across 6 tests in this block saves ~2 min.
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetPgliteState(engine);
|
||||
tmp = mkdtempSync(join(tmpdir(), 'brain-writer-scan-'));
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
rmSync(tmp, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
async function registerSource(id: string, path: string) {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, local_path) VALUES ($1, $1, $2)
|
||||
ON CONFLICT (id) DO UPDATE SET local_path = EXCLUDED.local_path`,
|
||||
[id, path],
|
||||
);
|
||||
}
|
||||
|
||||
test('returns ok=true for empty source', async () => {
|
||||
await registerSource('empty', tmp);
|
||||
const report = await scanBrainSources(engine);
|
||||
expect(report.ok).toBe(true);
|
||||
expect(report.total).toBe(0);
|
||||
const empty = report.per_source.find(s => s.source_id === 'empty');
|
||||
expect(empty).toBeDefined();
|
||||
expect(empty!.total).toBe(0);
|
||||
});
|
||||
|
||||
test('detects errors across multiple sources', async () => {
|
||||
const srcA = join(tmp, 'a');
|
||||
const srcB = join(tmp, 'b');
|
||||
mkdirSync(srcA, { recursive: true });
|
||||
mkdirSync(srcB, { recursive: true });
|
||||
writeFileSync(join(srcA, 'p1.md'), `${fence}\ntype: x\ntitle: ok\n${fence}\n\nbody\x00`);
|
||||
writeFileSync(join(srcB, 'p2.md'), `${fence}\ntype: x\ntitle: "P "I" L"\n${fence}\n\nbody`);
|
||||
await registerSource('alpha', srcA);
|
||||
await registerSource('beta', srcB);
|
||||
|
||||
const report = await scanBrainSources(engine);
|
||||
expect(report.ok).toBe(false);
|
||||
expect(report.total).toBeGreaterThan(0);
|
||||
const alpha = report.per_source.find(s => s.source_id === 'alpha')!;
|
||||
const beta = report.per_source.find(s => s.source_id === 'beta')!;
|
||||
expect(alpha.errors_by_code.NULL_BYTES).toBeGreaterThanOrEqual(1);
|
||||
expect(beta.errors_by_code.NESTED_QUOTES).toBeGreaterThanOrEqual(1);
|
||||
});
|
||||
|
||||
test('respects sourceId filter', async () => {
|
||||
const srcA = join(tmp, 'a');
|
||||
const srcB = join(tmp, 'b');
|
||||
mkdirSync(srcA, { recursive: true });
|
||||
mkdirSync(srcB, { recursive: true });
|
||||
writeFileSync(join(srcA, 'bad.md'), `${fence}\ntype: x\ntitle: ok\n${fence}\n\nbody\x00`);
|
||||
writeFileSync(join(srcB, 'bad.md'), `${fence}\ntype: x\ntitle: ok\n${fence}\n\nbody\x00`);
|
||||
await registerSource('alpha', srcA);
|
||||
await registerSource('beta', srcB);
|
||||
|
||||
const onlyA = await scanBrainSources(engine, { sourceId: 'alpha' });
|
||||
expect(onlyA.per_source.length).toBe(1);
|
||||
expect(onlyA.per_source[0]!.source_id).toBe('alpha');
|
||||
});
|
||||
|
||||
test('skips registered source with missing path', async () => {
|
||||
await registerSource('ghost', join(tmp, 'does-not-exist'));
|
||||
const report = await scanBrainSources(engine);
|
||||
const ghost = report.per_source.find(s => s.source_id === 'ghost')!;
|
||||
expect(ghost.total).toBe(0);
|
||||
});
|
||||
|
||||
test('skips symlinks (matches sync no-symlink policy)', async () => {
|
||||
mkdirSync(join(tmp, 'real'), { recursive: true });
|
||||
writeFileSync(join(tmp, 'real', 'good.md'), `${fence}\ntype: x\ntitle: ok\n${fence}\n\nbody`);
|
||||
// Create a symlink loop: tmp/real/loop -> tmp/real
|
||||
try {
|
||||
symlinkSync(join(tmp, 'real'), join(tmp, 'real', 'loop'));
|
||||
} catch {
|
||||
// Some CI environments forbid symlink creation; skip the assertion.
|
||||
return;
|
||||
}
|
||||
await registerSource('with-symlink', tmp);
|
||||
const report = await scanBrainSources(engine);
|
||||
// The walk should complete without infinite-looping; at most one .md
|
||||
// entry visited (via the real path, not the symlink).
|
||||
expect(report.per_source[0]!.total).toBe(0);
|
||||
});
|
||||
|
||||
test('AbortSignal mid-scan stops walking', async () => {
|
||||
const src = join(tmp, 'big');
|
||||
mkdirSync(src, { recursive: true });
|
||||
for (let i = 0; i < 50; i++) {
|
||||
writeFileSync(join(src, `p${i}.md`), `${fence}\ntype: x\ntitle: t${i}\n${fence}\n\nbody`);
|
||||
}
|
||||
await registerSource('big', src);
|
||||
const ctrl = new AbortController();
|
||||
ctrl.abort();
|
||||
const report = await scanBrainSources(engine, { signal: ctrl.signal });
|
||||
// Aborted before any source ran; per_source array stays empty (or has zero reports).
|
||||
expect(report.per_source.length).toBe(0);
|
||||
});
|
||||
});
|
||||
@@ -255,21 +255,6 @@ describe("DRY detection — checkResolvable", () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe("v0.22.4 regression — actual repo skills/ has 0 warnings", () => {
|
||||
test("repo skills/ pass check-resolvable cleanly", () => {
|
||||
// The contract for v0.22.4 (Part A): zero warnings, zero errors
|
||||
// against the actual checked-in skills/ tree. Guards against future
|
||||
// regressions that re-introduce trigger overlap, DRY violations, or
|
||||
// routing-eval fixture drift.
|
||||
const report = checkResolvable(SKILLS_DIR);
|
||||
const errors = report.issues.filter(i => i.severity === "error");
|
||||
const warnings = report.issues.filter(i => i.severity === "warning");
|
||||
expect(errors).toEqual([]);
|
||||
expect(warnings).toEqual([]);
|
||||
expect(report.ok).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
// bun:test has no beforeEach/afterEach at module scope cleanly interacting
|
||||
// with closures; a small helper keeps cleanup readable and per-test.
|
||||
function afterEachCleanup(fn: () => void) {
|
||||
|
||||
@@ -1,314 +0,0 @@
|
||||
import { describe, it, expect } from 'bun:test';
|
||||
|
||||
/**
|
||||
* Tests for connection resilience features:
|
||||
* 1. PostgresEngine.executeRaw retries on connection errors
|
||||
* 2. PostgresEngine.reconnect creates fresh connection pool
|
||||
* 3. Supervisor health check tracks consecutive failures
|
||||
* 4. Supervisor classifies worker exit reasons
|
||||
*/
|
||||
|
||||
// --- Unit tests for isConnectionError (extracted pattern) ---
|
||||
|
||||
const CONNECTION_ERROR_PATTERNS = [
|
||||
'ECONNREFUSED',
|
||||
'ECONNRESET',
|
||||
'EPIPE',
|
||||
'connection terminated',
|
||||
'Client has encountered a connection error',
|
||||
'password authentication failed',
|
||||
'Connection terminated unexpectedly',
|
||||
'no pg_hba.conf entry',
|
||||
'server closed the connection unexpectedly',
|
||||
'SSL connection has been closed unexpectedly',
|
||||
'connection is insecure',
|
||||
'too many connections',
|
||||
'remaining connection slots are reserved',
|
||||
];
|
||||
|
||||
function isConnectionError(err: unknown): boolean {
|
||||
if (!err) return false;
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
const code = (err as NodeJS.ErrnoException)?.code;
|
||||
if (code && CONNECTION_ERROR_PATTERNS.includes(code)) return true;
|
||||
return CONNECTION_ERROR_PATTERNS.some(p => msg.includes(p));
|
||||
}
|
||||
|
||||
describe('isConnectionError', () => {
|
||||
it('detects password authentication failure', () => {
|
||||
expect(isConnectionError(new Error('password authentication failed for user "postgres"'))).toBe(true);
|
||||
});
|
||||
|
||||
it('detects ECONNREFUSED via error code', () => {
|
||||
const err = new Error('connect ECONNREFUSED 127.0.0.1:5432') as NodeJS.ErrnoException;
|
||||
err.code = 'ECONNREFUSED';
|
||||
expect(isConnectionError(err)).toBe(true);
|
||||
});
|
||||
|
||||
it('detects ECONNRESET via error code', () => {
|
||||
const err = new Error('read ECONNRESET') as NodeJS.ErrnoException;
|
||||
err.code = 'ECONNRESET';
|
||||
expect(isConnectionError(err)).toBe(true);
|
||||
});
|
||||
|
||||
it('detects connection terminated message', () => {
|
||||
expect(isConnectionError(new Error('connection terminated'))).toBe(true);
|
||||
});
|
||||
|
||||
it('detects Connection terminated unexpectedly', () => {
|
||||
expect(isConnectionError(new Error('Connection terminated unexpectedly'))).toBe(true);
|
||||
});
|
||||
|
||||
it('detects server closed the connection', () => {
|
||||
expect(isConnectionError(new Error('server closed the connection unexpectedly'))).toBe(true);
|
||||
});
|
||||
|
||||
it('detects SSL connection closed', () => {
|
||||
expect(isConnectionError(new Error('SSL connection has been closed unexpectedly'))).toBe(true);
|
||||
});
|
||||
|
||||
it('detects too many connections', () => {
|
||||
expect(isConnectionError(new Error('FATAL: too many connections for role "postgres"'))).toBe(true);
|
||||
});
|
||||
|
||||
it('does not match regular query errors', () => {
|
||||
expect(isConnectionError(new Error('relation "foo" does not exist'))).toBe(false);
|
||||
});
|
||||
|
||||
it('does not match null/undefined', () => {
|
||||
expect(isConnectionError(null)).toBe(false);
|
||||
expect(isConnectionError(undefined)).toBe(false);
|
||||
});
|
||||
|
||||
it('does not match syntax errors', () => {
|
||||
expect(isConnectionError(new Error('syntax error at or near "SELECT"'))).toBe(false);
|
||||
});
|
||||
|
||||
it('does not match constraint violations', () => {
|
||||
expect(isConnectionError(new Error('duplicate key value violates unique constraint'))).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
// --- Unit tests for worker exit classification ---
|
||||
|
||||
function classifyWorkerExit(code: number | null, signal: string | null): string {
|
||||
if (signal === 'SIGKILL') return 'oom_or_external_kill';
|
||||
if (signal === 'SIGTERM') return 'graceful_shutdown';
|
||||
if (code === 1) return 'runtime_error';
|
||||
if (code === 0) return 'clean_exit';
|
||||
return 'unknown';
|
||||
}
|
||||
|
||||
describe('classifyWorkerExit', () => {
|
||||
it('classifies SIGKILL as OOM/external kill', () => {
|
||||
expect(classifyWorkerExit(null, 'SIGKILL')).toBe('oom_or_external_kill');
|
||||
});
|
||||
|
||||
it('classifies SIGTERM as graceful shutdown', () => {
|
||||
expect(classifyWorkerExit(null, 'SIGTERM')).toBe('graceful_shutdown');
|
||||
});
|
||||
|
||||
it('classifies exit code 1 as runtime error', () => {
|
||||
expect(classifyWorkerExit(1, null)).toBe('runtime_error');
|
||||
});
|
||||
|
||||
it('classifies exit code 0 as clean exit', () => {
|
||||
expect(classifyWorkerExit(0, null)).toBe('clean_exit');
|
||||
});
|
||||
|
||||
it('classifies unknown codes as unknown', () => {
|
||||
expect(classifyWorkerExit(137, null)).toBe('unknown');
|
||||
expect(classifyWorkerExit(null, null)).toBe('unknown');
|
||||
});
|
||||
|
||||
// Signal takes precedence over code
|
||||
it('SIGKILL takes precedence over any exit code', () => {
|
||||
expect(classifyWorkerExit(1, 'SIGKILL')).toBe('oom_or_external_kill');
|
||||
});
|
||||
});
|
||||
|
||||
// --- Mock-based tests for reconnect logic ---
|
||||
|
||||
describe('PostgresEngine reconnect behavior', () => {
|
||||
it('reconnect flag prevents concurrent reconnections', async () => {
|
||||
// Simulate the _reconnecting guard
|
||||
let reconnecting = false;
|
||||
let reconnectCount = 0;
|
||||
|
||||
async function reconnect() {
|
||||
if (reconnecting) return;
|
||||
reconnecting = true;
|
||||
try {
|
||||
reconnectCount++;
|
||||
await new Promise(r => setTimeout(r, 10));
|
||||
} finally {
|
||||
reconnecting = false;
|
||||
}
|
||||
}
|
||||
|
||||
// Fire 3 concurrent reconnects — only 1 should run
|
||||
await Promise.all([reconnect(), reconnect(), reconnect()]);
|
||||
expect(reconnectCount).toBe(1);
|
||||
});
|
||||
|
||||
it('executeRaw retry does not infinite-loop on persistent connection failure', async () => {
|
||||
// Simulate: first call fails (connection error), reconnect succeeds,
|
||||
// but retry also fails with a NON-connection error
|
||||
let callCount = 0;
|
||||
|
||||
async function executeRawWithRetry(): Promise<unknown[]> {
|
||||
callCount++;
|
||||
if (callCount === 1) {
|
||||
throw new Error('connection terminated'); // connection error → triggers retry
|
||||
}
|
||||
if (callCount === 2) {
|
||||
throw new Error('relation "foo" does not exist'); // NOT a connection error → throw
|
||||
}
|
||||
return [{ ok: true }];
|
||||
}
|
||||
|
||||
try {
|
||||
await (async () => {
|
||||
try {
|
||||
return await executeRawWithRetry();
|
||||
} catch (err) {
|
||||
if (isConnectionError(err)) {
|
||||
// "reconnect" would happen here
|
||||
return await executeRawWithRetry();
|
||||
}
|
||||
throw err;
|
||||
}
|
||||
})();
|
||||
} catch (err) {
|
||||
expect((err as Error).message).toBe('relation "foo" does not exist');
|
||||
}
|
||||
|
||||
expect(callCount).toBe(2); // Only 2 attempts, no infinite loop
|
||||
});
|
||||
|
||||
it('executeRaw succeeds on retry after connection error', async () => {
|
||||
let callCount = 0;
|
||||
|
||||
async function executeRawWithRetry(): Promise<unknown[]> {
|
||||
callCount++;
|
||||
if (callCount === 1) {
|
||||
throw new Error('password authentication failed for user "postgres"');
|
||||
}
|
||||
return [{ ok: true }];
|
||||
}
|
||||
|
||||
const result = await (async () => {
|
||||
try {
|
||||
return await executeRawWithRetry();
|
||||
} catch (err) {
|
||||
if (isConnectionError(err)) {
|
||||
// reconnect would happen here
|
||||
return await executeRawWithRetry();
|
||||
}
|
||||
throw err;
|
||||
}
|
||||
})();
|
||||
|
||||
expect(result).toEqual([{ ok: true }]);
|
||||
expect(callCount).toBe(2);
|
||||
});
|
||||
});
|
||||
|
||||
// --- Supervisor health check failure tracking ---
|
||||
|
||||
describe('Supervisor health check failure tracking', () => {
|
||||
it('emits db_connection_degraded after 3 consecutive failures', () => {
|
||||
let consecutiveFailures = 0;
|
||||
const emitted: Array<{ event: string; reason?: string }> = [];
|
||||
|
||||
function emit(event: string, fields: Record<string, unknown> = {}) {
|
||||
emitted.push({ event, ...fields } as { event: string; reason?: string });
|
||||
}
|
||||
|
||||
// Simulate 3 health check failures
|
||||
for (let i = 0; i < 4; i++) {
|
||||
consecutiveFailures++;
|
||||
if (consecutiveFailures >= 3) {
|
||||
emit('health_warn', { reason: 'db_connection_degraded', consecutive_failures: consecutiveFailures });
|
||||
} else {
|
||||
emit('health_error', { error: 'connection terminated' });
|
||||
}
|
||||
}
|
||||
|
||||
const degradedWarnings = emitted.filter(e => e.reason === 'db_connection_degraded');
|
||||
expect(degradedWarnings.length).toBe(2); // fires at count 3 and 4
|
||||
|
||||
// First two were regular health_error
|
||||
expect(emitted[0].event).toBe('health_error');
|
||||
expect(emitted[1].event).toBe('health_error');
|
||||
// Third triggers the degraded warning
|
||||
expect(emitted[2].reason).toBe('db_connection_degraded');
|
||||
});
|
||||
|
||||
it('resets failure counter on successful health check', () => {
|
||||
let consecutiveFailures = 0;
|
||||
|
||||
// 2 failures
|
||||
consecutiveFailures++;
|
||||
consecutiveFailures++;
|
||||
expect(consecutiveFailures).toBe(2);
|
||||
|
||||
// Success resets
|
||||
consecutiveFailures = 0;
|
||||
expect(consecutiveFailures).toBe(0);
|
||||
|
||||
// 1 more failure — should not trigger degraded (need 3 consecutive)
|
||||
consecutiveFailures++;
|
||||
expect(consecutiveFailures).toBeLessThan(3);
|
||||
});
|
||||
});
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
// Eng-review D3 regression guards — executeRaw retry wrapper dropped
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
//
|
||||
// The original #406 wrapped PostgresEngine.executeRaw in a per-call
|
||||
// try/catch that retried on connection errors. Eng-review D3 dropped
|
||||
// that wrapper as unsound (regex idempotence boundary doesn't hold
|
||||
// for writable CTEs or side-effecting SELECTs). Recovery now happens
|
||||
// at the supervisor level via the 3-strikes-then-reconnect path.
|
||||
//
|
||||
// These guards prevent reintroduction of the per-call retry without
|
||||
// a typed-idempotency boundary.
|
||||
|
||||
import { readFileSync } from 'fs';
|
||||
import { resolve } from 'path';
|
||||
|
||||
describe('Eng-review D3 — executeRaw has no per-call retry wrapper', () => {
|
||||
it('PostgresEngine.executeRaw is a single-statement passthrough (no try/catch on connection errors)', () => {
|
||||
const src = readFileSync(resolve('src/core/postgres-engine.ts'), 'utf-8');
|
||||
|
||||
// Find the executeRaw method in the class (not the helper inside withReservedConnection)
|
||||
// Pattern: must be a method on the class taking (sql, params)
|
||||
const fnMatch = src.match(/async executeRaw<T = Record<string, unknown>>\(sql: string, params\?: unknown\[\]\): Promise<T\[\]> \{([\s\S]*?)\n \}/);
|
||||
expect(fnMatch).not.toBeNull();
|
||||
const body = fnMatch![1];
|
||||
|
||||
// Must not have any try/catch
|
||||
expect(body).not.toContain('try {');
|
||||
expect(body).not.toContain('catch');
|
||||
// Must not call reconnect() from this method
|
||||
expect(body).not.toContain('this.reconnect()');
|
||||
// Must call conn.unsafe directly
|
||||
expect(body).toContain('conn.unsafe(');
|
||||
});
|
||||
|
||||
it('PostgresEngine.reconnect() still exists for supervisor-driven recovery', () => {
|
||||
const src = readFileSync(resolve('src/core/postgres-engine.ts'), 'utf-8');
|
||||
expect(src).toContain('async reconnect()');
|
||||
expect(src).toContain('await this.disconnect()');
|
||||
});
|
||||
|
||||
it('Supervisor still has the 3-strikes-then-reconnect path', () => {
|
||||
const src = readFileSync(resolve('src/core/minions/supervisor.ts'), 'utf-8');
|
||||
expect(src).toContain('consecutiveHealthFailures');
|
||||
// Supervisor invokes reconnect via a typed cast after 3 consecutive failures.
|
||||
expect(src).toMatch(/reconnect\(\): Promise<void>/);
|
||||
expect(src).toContain('this.consecutiveHealthFailures >= 3');
|
||||
});
|
||||
});
|
||||
+5
-153
@@ -17,8 +17,8 @@ import { existsSync, unlinkSync } from 'fs';
|
||||
|
||||
let lintCalls: Array<{ target: string; fix: boolean; dryRun: boolean | undefined }> = [];
|
||||
let backlinksCalls: Array<{ action: string; dir: string; dryRun: boolean | undefined }> = [];
|
||||
let syncCalls: Array<{ dryRun: boolean | undefined; noPull: boolean | undefined; noExtract: boolean | undefined; sourceId: string | undefined }> = [];
|
||||
let extractCalls: Array<{ mode: string; dir: string; slugs: string[] | undefined }> = [];
|
||||
let syncCalls: Array<{ dryRun: boolean | undefined; noPull: boolean | undefined }> = [];
|
||||
let extractCalls: Array<{ mode: string; dir: string }> = [];
|
||||
let embedCalls: Array<{ stale: boolean | undefined; dryRun: boolean | undefined }> = [];
|
||||
let orphansCalls: number = 0;
|
||||
|
||||
@@ -49,7 +49,7 @@ mock.module('../../src/commands/backlinks.ts', () => ({
|
||||
// Mock sync
|
||||
mock.module('../../src/commands/sync.ts', () => ({
|
||||
performSync: async (_engine: any, opts: any) => {
|
||||
syncCalls.push({ dryRun: opts.dryRun, noPull: opts.noPull, noExtract: opts.noExtract, sourceId: opts.sourceId });
|
||||
syncCalls.push({ dryRun: opts.dryRun, noPull: opts.noPull });
|
||||
return {
|
||||
status: opts.dryRun ? 'dry_run' : 'synced',
|
||||
fromCommit: 'abcd',
|
||||
@@ -72,8 +72,8 @@ mock.module('../../src/commands/sync.ts', () => ({
|
||||
// Mock extract
|
||||
mock.module('../../src/commands/extract.ts', () => ({
|
||||
runExtractCore: async (_engine: any, opts: any) => {
|
||||
extractCalls.push({ mode: opts.mode, dir: opts.dir, slugs: opts.slugs });
|
||||
return { links_created: 7, timeline_entries_created: 3, pages_processed: opts.slugs?.length ?? 5 };
|
||||
extractCalls.push({ mode: opts.mode, dir: opts.dir });
|
||||
return { links_created: 7, timeline_entries_created: 3, pages_processed: 5 };
|
||||
},
|
||||
walkMarkdownFiles: () => [],
|
||||
extractMarkdownLinks: () => [],
|
||||
@@ -392,151 +392,3 @@ describe('runCycle — yieldBetweenPhases hook', () => {
|
||||
expect(report.phases.length).toBe(6);
|
||||
});
|
||||
});
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
// Wave regression guards (#417 + Codex F2)
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
|
||||
describe('runCycle — incremental extract slug propagation (#417)', () => {
|
||||
beforeEach(async () => {
|
||||
await truncateCycleLocks(sharedEngine);
|
||||
syncCalls = [];
|
||||
extractCalls = [];
|
||||
});
|
||||
|
||||
test('cycle threads sync.pagesAffected into extract phase as the slugs argument', async () => {
|
||||
// performSync mock returns pagesAffected = ['a', 'b']. The extract phase
|
||||
// must receive those exact slugs, not undefined (which would trigger a full walk).
|
||||
await runCycle(sharedEngine, { brainDir: '/tmp/brain' });
|
||||
|
||||
// Sync ran once
|
||||
expect(syncCalls.length).toBe(1);
|
||||
// Extract ran once with the slugs from sync (not undefined)
|
||||
expect(extractCalls.length).toBe(1);
|
||||
expect(extractCalls[0].slugs).toEqual(['a', 'b']);
|
||||
});
|
||||
|
||||
test('extract phase falls back to full walk when sync was skipped (slugs undefined)', async () => {
|
||||
// Run only the extract phase — sync didn't run, so syncPagesAffected
|
||||
// is undefined and extract should walk the full directory (slugs:undefined).
|
||||
await runCycle(sharedEngine, { brainDir: '/tmp/brain', phases: ['extract'] });
|
||||
|
||||
expect(syncCalls.length).toBe(0);
|
||||
expect(extractCalls.length).toBe(1);
|
||||
expect(extractCalls[0].slugs).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe('runCycle — Codex F2: noExtract is gated on whether extract phase runs', () => {
|
||||
beforeEach(async () => {
|
||||
await truncateCycleLocks(sharedEngine);
|
||||
syncCalls = [];
|
||||
extractCalls = [];
|
||||
});
|
||||
|
||||
test('full cycle (sync + extract): noExtract=true so sync skips inline extraction (extract phase handles it)', async () => {
|
||||
await runCycle(sharedEngine, { brainDir: '/tmp/brain', phases: ['sync', 'extract'] });
|
||||
|
||||
expect(syncCalls.length).toBe(1);
|
||||
expect(syncCalls[0].noExtract).toBe(true); // dedupe enabled
|
||||
expect(extractCalls.length).toBe(1); // extract phase ran
|
||||
});
|
||||
|
||||
test('phases:[sync] only: noExtract=false so sync runs inline extraction (no silent extract drop)', async () => {
|
||||
await runCycle(sharedEngine, { brainDir: '/tmp/brain', phases: ['sync'] });
|
||||
|
||||
expect(syncCalls.length).toBe(1);
|
||||
// Critical: noExtract must be false here. If it were true, the user just lost
|
||||
// their extraction without any indication. This is the F2 regression guard.
|
||||
expect(syncCalls[0].noExtract).toBe(false);
|
||||
expect(extractCalls.length).toBe(0); // extract phase did NOT run
|
||||
});
|
||||
});
|
||||
|
||||
// ─── sourceId resolution (regression #475) ─────────────────────────
|
||||
//
|
||||
// Production OpenClaw deployment hit a 30+ min hang on every autopilot
|
||||
// cycle because runPhaseSync was calling performSync without sourceId,
|
||||
// so sync read the global config.sync.last_commit key (which had drifted
|
||||
// out of git history after a force-push GC'd the commit). The per-source
|
||||
// sources.last_commit anchor was valid the entire time. PR #475 added
|
||||
// resolveSourceForDir() so the cycle reads the per-source anchor instead.
|
||||
//
|
||||
// These tests pin the resolver -> performSync(opts.sourceId) plumbing.
|
||||
|
||||
describe('runCycle — sourceId resolution (regression #475)', () => {
|
||||
beforeEach(async () => {
|
||||
await truncateCycleLocks(sharedEngine);
|
||||
await (sharedEngine as any).db.query('DELETE FROM sources');
|
||||
});
|
||||
|
||||
test('seeded sources row → performSync receives matching sourceId', async () => {
|
||||
await (sharedEngine as any).db.query(
|
||||
`INSERT INTO sources (id, name, local_path) VALUES ($1, $2, $3)`,
|
||||
['default', 'default', '/tmp/brain-475-a'],
|
||||
);
|
||||
await runCycle(sharedEngine, { brainDir: '/tmp/brain-475-a' });
|
||||
expect(syncCalls.at(-1)?.sourceId).toBe('default');
|
||||
});
|
||||
|
||||
test('no matching sources row → performSync receives sourceId=undefined', async () => {
|
||||
await runCycle(sharedEngine, { brainDir: '/tmp/brain-475-b' });
|
||||
expect(syncCalls.at(-1)?.sourceId).toBeUndefined();
|
||||
});
|
||||
|
||||
test('different brainDir than registered source → undefined (no cross-match)', async () => {
|
||||
await (sharedEngine as any).db.query(
|
||||
`INSERT INTO sources (id, name, local_path) VALUES ($1, $2, $3)`,
|
||||
['other', 'other', '/some/other/brain'],
|
||||
);
|
||||
await runCycle(sharedEngine, { brainDir: '/tmp/brain-475-c' });
|
||||
expect(syncCalls.at(-1)?.sourceId).toBeUndefined();
|
||||
});
|
||||
|
||||
test('sources table missing (very old brain) → catch returns undefined, sync still runs', async () => {
|
||||
// CRITICAL: do NOT DROP TABLE on the shared engine. initSchema() only
|
||||
// re-runs PENDING migrations; once schema_version is at latest, the
|
||||
// v20 migration that creates `sources` will not re-execute. Use a
|
||||
// fresh one-shot engine so the shared engine isn't degraded for
|
||||
// every later test in this file.
|
||||
const fresh = new PGLiteEngine();
|
||||
await fresh.connect({});
|
||||
await fresh.initSchema();
|
||||
await (fresh as any).db.query('DROP TABLE IF EXISTS sources CASCADE');
|
||||
try {
|
||||
await runCycle(fresh, { brainDir: '/tmp/brain-475-d' });
|
||||
expect(syncCalls.at(-1)?.sourceId).toBeUndefined();
|
||||
} finally {
|
||||
await fresh.disconnect();
|
||||
}
|
||||
});
|
||||
|
||||
test('multiple rows with same local_path → resolver returns one matching id (non-deterministic)', async () => {
|
||||
// Schema has no UNIQUE on local_path; SQL has no ORDER BY. Either id
|
||||
// is acceptable; the contract is "any matching id, never null when
|
||||
// matches exist." This test pins behavior so the follow-up
|
||||
// UNIQUE-constraint TODO has a regression target.
|
||||
await (sharedEngine as any).db.query(
|
||||
`INSERT INTO sources (id, name, local_path) VALUES
|
||||
('first', 'first', '/tmp/brain-475-e'),
|
||||
('second', 'second', '/tmp/brain-475-e')`,
|
||||
);
|
||||
await runCycle(sharedEngine, { brainDir: '/tmp/brain-475-e' });
|
||||
const sourceId = syncCalls.at(-1)?.sourceId;
|
||||
expect(sourceId).toBeDefined();
|
||||
expect(['first', 'second']).toContain(sourceId as string);
|
||||
});
|
||||
|
||||
test('empty-string id row → resolver propagates as "" (defensive)', async () => {
|
||||
// Schema has id as PRIMARY KEY (NOT NULL), so NULL id can't happen.
|
||||
// Empty string CAN be inserted, and the resolver's `rows[0]?.id`
|
||||
// would treat any falsy id as "no source" via the optional chain.
|
||||
// This test pins the current behavior (we DO pass '' through to
|
||||
// performSync) so a future refactor doesn't silently regress it.
|
||||
await (sharedEngine as any).db.query(
|
||||
`INSERT INTO sources (id, name, local_path) VALUES ('', 'empty', '/tmp/brain-475-f')`,
|
||||
);
|
||||
await runCycle(sharedEngine, { brainDir: '/tmp/brain-475-f' });
|
||||
expect(syncCalls.at(-1)?.sourceId).toBe('');
|
||||
});
|
||||
});
|
||||
|
||||
@@ -1,148 +0,0 @@
|
||||
/**
|
||||
* test/cycle-abort.test.ts — Verify runCycle respects AbortSignal.
|
||||
*
|
||||
* Regression test for the 2026-04-24 incident where 98 jobs piled up
|
||||
* because autopilot-cycle's handler didn't propagate AbortSignal to
|
||||
* runCycle, and runCycle had no signal-checking between phases.
|
||||
*
|
||||
* Tests the three-layer fix:
|
||||
* 1. CycleOpts.signal — runCycle checks signal between phases
|
||||
* 2. Handler wiring — autopilot-cycle passes job.signal
|
||||
* 3. Worker force-eviction — last resort if handler ignores abort
|
||||
*
|
||||
* Layer 3 is tested in minions.test.ts (worker-level). This file
|
||||
* covers layers 1 and 2 via the cycle interface.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
|
||||
// We can't easily import runCycle with a real engine for unit tests,
|
||||
// but we CAN test the checkAborted pattern and CycleOpts contract.
|
||||
|
||||
describe('CycleOpts.signal contract (v0.20.5)', () => {
|
||||
test('signal field exists on CycleOpts interface', async () => {
|
||||
// Type-level test: importing the type should work
|
||||
const mod = await import('../src/core/cycle.ts');
|
||||
// runCycle exists and is callable
|
||||
expect(typeof mod.runCycle).toBe('function');
|
||||
});
|
||||
|
||||
test('runCycle accepts signal in opts without error', async () => {
|
||||
// Verify runCycle doesn't crash when signal is passed but no engine
|
||||
const { runCycle } = await import('../src/core/cycle.ts');
|
||||
const abort = new AbortController();
|
||||
|
||||
// Call with null engine + minimal opts — should return a report
|
||||
// (phases that need engine will be skipped)
|
||||
const report = await runCycle(null, {
|
||||
brainDir: '/nonexistent-for-test',
|
||||
phases: [], // empty phases = no work
|
||||
signal: abort.signal,
|
||||
});
|
||||
|
||||
expect(report.schema_version).toBe('1');
|
||||
expect(report.status).toBeDefined();
|
||||
});
|
||||
|
||||
test('runCycle bails on pre-aborted signal', async () => {
|
||||
const { runCycle } = await import('../src/core/cycle.ts');
|
||||
const abort = new AbortController();
|
||||
abort.abort(new Error('timeout'));
|
||||
|
||||
// With a pre-aborted signal and phases that would run, it should
|
||||
// throw or return failed (depending on which phase catches it first)
|
||||
try {
|
||||
const report = await runCycle(null, {
|
||||
brainDir: '/nonexistent-for-test',
|
||||
phases: ['lint'], // lint doesn't need engine, would normally run
|
||||
signal: abort.signal,
|
||||
});
|
||||
// If it returns instead of throwing, status should reflect the abort
|
||||
expect(['failed', 'partial']).toContain(report.status);
|
||||
} catch (err) {
|
||||
// checkAborted threw — this is the expected behavior
|
||||
expect(err instanceof Error).toBe(true);
|
||||
expect((err as Error).message).toContain('aborted');
|
||||
}
|
||||
});
|
||||
|
||||
test('runCycle bails mid-flight when signal fires between phases', async () => {
|
||||
const { runCycle } = await import('../src/core/cycle.ts');
|
||||
const abort = new AbortController();
|
||||
|
||||
// Abort after 50ms — should catch between phases
|
||||
setTimeout(() => abort.abort(new Error('timeout')), 50);
|
||||
|
||||
try {
|
||||
const report = await runCycle(null, {
|
||||
brainDir: '/nonexistent-for-test',
|
||||
phases: ['lint', 'backlinks', 'orphans'],
|
||||
signal: abort.signal,
|
||||
yieldBetweenPhases: async () => {
|
||||
// Slow yield to give the abort time to fire
|
||||
await new Promise(r => setTimeout(r, 100));
|
||||
},
|
||||
});
|
||||
// If it returned cleanly, not all phases should have run
|
||||
// (abort should have prevented later phases)
|
||||
const completedPhases = report.phases.length;
|
||||
expect(completedPhases).toBeLessThan(3);
|
||||
} catch (err) {
|
||||
// checkAborted threw between phases — expected
|
||||
expect(err instanceof Error).toBe(true);
|
||||
expect((err as Error).message).toContain('aborted');
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('autopilot-cycle handler contract (v0.20.5)', () => {
|
||||
test('handler registration passes signal to runCycle', async () => {
|
||||
// Verify the handler code in jobs.ts includes job.signal
|
||||
const fs = await import('fs');
|
||||
const jobsSource = fs.readFileSync(
|
||||
new URL('../src/commands/jobs.ts', import.meta.url),
|
||||
'utf8',
|
||||
);
|
||||
|
||||
// The autopilot-cycle handler MUST pass signal to runCycle
|
||||
// This is a source-level regression guard
|
||||
const handlerBlock = jobsSource.slice(
|
||||
jobsSource.indexOf("worker.register('autopilot-cycle'"),
|
||||
jobsSource.indexOf("worker.register('autopilot-cycle'") + 2000,
|
||||
);
|
||||
|
||||
expect(handlerBlock).toContain('signal: job.signal');
|
||||
});
|
||||
|
||||
test('worker.ts has force-eviction safety net after timeout', async () => {
|
||||
// Verify the worker code includes the grace timer
|
||||
const fs = await import('fs');
|
||||
const workerSource = fs.readFileSync(
|
||||
new URL('../src/core/minions/worker.ts', import.meta.url),
|
||||
'utf8',
|
||||
);
|
||||
|
||||
// Must have the force-eviction pattern
|
||||
expect(workerSource).toContain('Force-evicting from inFlight');
|
||||
expect(workerSource).toContain('graceTimer');
|
||||
expect(workerSource).toContain('handler ignored abort signal');
|
||||
});
|
||||
|
||||
test('cycle.ts has checkAborted calls between phases', async () => {
|
||||
// Verify the cycle code checks abort between every phase
|
||||
const fs = await import('fs');
|
||||
const cycleSource = fs.readFileSync(
|
||||
new URL('../src/core/cycle.ts', import.meta.url),
|
||||
'utf8',
|
||||
);
|
||||
|
||||
// Count checkAborted calls in the runCycle function body
|
||||
const runCycleBody = cycleSource.slice(
|
||||
cycleSource.indexOf('export async function runCycle'),
|
||||
);
|
||||
const checkCalls = (runCycleBody.match(/checkAborted\(opts\.signal\)/g) || []).length;
|
||||
|
||||
// Should have at least 6 (one per phase)
|
||||
expect(checkCalls).toBeGreaterThanOrEqual(6);
|
||||
});
|
||||
});
|
||||
@@ -1,92 +0,0 @@
|
||||
/**
|
||||
* Tests for src/core/disk-walk.ts — single-walk filesystem scan.
|
||||
*
|
||||
* Replaces the per-page existsSync+statSync syscall storm in storage.ts
|
||||
* (Issue #14 of the v0.22.3 eng review).
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
|
||||
import { mkdtempSync, mkdirSync, rmSync, writeFileSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import { walkBrainRepo } from '../src/core/disk-walk.ts';
|
||||
|
||||
let tmp: string;
|
||||
|
||||
beforeEach(() => {
|
||||
tmp = mkdtempSync(join(tmpdir(), 'gbrain-walk-test-'));
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
rmSync(tmp, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
function write(relPath: string, content: string): void {
|
||||
const full = join(tmp, relPath);
|
||||
mkdirSync(join(full, '..'), { recursive: true });
|
||||
writeFileSync(full, content);
|
||||
}
|
||||
|
||||
describe('walkBrainRepo', () => {
|
||||
test('returns empty map for empty directory', () => {
|
||||
expect(walkBrainRepo(tmp).size).toBe(0);
|
||||
});
|
||||
|
||||
test('returns empty map for nonexistent directory', () => {
|
||||
expect(walkBrainRepo(join(tmp, 'does-not-exist')).size).toBe(0);
|
||||
});
|
||||
|
||||
test('finds top-level .md files keyed by slug (no .md suffix)', () => {
|
||||
write('alice.md', '# Alice');
|
||||
const result = walkBrainRepo(tmp);
|
||||
expect(result.has('alice')).toBe(true);
|
||||
expect(result.get('alice')!.size).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
test('walks nested directories and produces slash-joined slugs', () => {
|
||||
write('people/alice.md', '# Alice');
|
||||
write('media/x/tweet-1.md', 'tweet');
|
||||
write('media/articles/post-1.md', 'post');
|
||||
const result = walkBrainRepo(tmp);
|
||||
expect(new Set(result.keys())).toEqual(
|
||||
new Set(['people/alice', 'media/x/tweet-1', 'media/articles/post-1']),
|
||||
);
|
||||
});
|
||||
|
||||
test('skips dot-directories (.git, .gbrain, .vscode)', () => {
|
||||
write('.git/HEAD', 'ref: refs/heads/main');
|
||||
write('.gbrain/config.json', '{}');
|
||||
write('.vscode/settings.json', '{}');
|
||||
write('people/alice.md', '# Alice');
|
||||
const result = walkBrainRepo(tmp);
|
||||
expect(new Set(result.keys())).toEqual(new Set(['people/alice']));
|
||||
});
|
||||
|
||||
test('skips node_modules', () => {
|
||||
write('node_modules/foo/bar.md', 'noise');
|
||||
write('people/alice.md', '# Alice');
|
||||
const result = walkBrainRepo(tmp);
|
||||
expect(new Set(result.keys())).toEqual(new Set(['people/alice']));
|
||||
});
|
||||
|
||||
test('ignores non-.md files', () => {
|
||||
write('people/alice.md', '# Alice');
|
||||
write('people/alice.json', '{}');
|
||||
write('people/photo.png', 'binary');
|
||||
const result = walkBrainRepo(tmp);
|
||||
expect(new Set(result.keys())).toEqual(new Set(['people/alice']));
|
||||
});
|
||||
|
||||
test('captures size from stat', () => {
|
||||
const content = '# Alice\n'.repeat(100);
|
||||
write('people/alice.md', content);
|
||||
const result = walkBrainRepo(tmp);
|
||||
expect(result.get('people/alice')!.size).toBe(content.length);
|
||||
});
|
||||
|
||||
test('captures mtimeMs', () => {
|
||||
write('people/alice.md', '# Alice');
|
||||
const result = walkBrainRepo(tmp);
|
||||
expect(result.get('people/alice')!.mtimeMs).toBeGreaterThan(0);
|
||||
});
|
||||
});
|
||||
@@ -146,9 +146,7 @@ describe('gbrain doctor — half-migrated Minions detection', () => {
|
||||
|
||||
test('filesystem: multiple versions each need their own complete entry', () => {
|
||||
// v0.10 is fully migrated but v0.11 is only partial. Doctor should
|
||||
// flag v0.11 by name. The forward-progress override only kicks in
|
||||
// when a NEWER version completed; v0.10 is older than v0.11 so the
|
||||
// partial still stands.
|
||||
// flag v0.11 by name.
|
||||
const migrationsDir = join(tmp, '.gbrain', 'migrations');
|
||||
mkdirSync(migrationsDir, { recursive: true });
|
||||
writeFileSync(
|
||||
@@ -168,65 +166,6 @@ describe('gbrain doctor — half-migrated Minions detection', () => {
|
||||
expect(minions!.message).not.toContain('0.10.0');
|
||||
});
|
||||
|
||||
test('filesystem: stale partial superseded by newer complete → NO warning (forward-progress override)', () => {
|
||||
// v0.16.0 completed AFTER v0.11.0 went partial. The schema clearly
|
||||
// advanced past v0.11.0, so the partial record is stale historical
|
||||
// noise — not a real "MINIONS HALF-INSTALLED" condition.
|
||||
//
|
||||
// Without this override, every install that ever went through a
|
||||
// v0.11.0 stopgap and then upgraded carries the FAIL flag forever,
|
||||
// even on installs that have been at v0.22+ for months. Real cause:
|
||||
// long-running gbrain installs accumulate partial entries from
|
||||
// historical stopgap runs; a doctor flag with no time decay or
|
||||
// forward-progress detection becomes meaningless once you've
|
||||
// moved past those versions.
|
||||
const migrationsDir = join(tmp, '.gbrain', 'migrations');
|
||||
mkdirSync(migrationsDir, { recursive: true });
|
||||
writeFileSync(
|
||||
join(migrationsDir, 'completed.jsonl'),
|
||||
[
|
||||
JSON.stringify({ version: '0.16.0', status: 'complete', ts: '2026-04-26T06:13:50.825Z' }),
|
||||
JSON.stringify({ version: '0.11.0', status: 'partial', ts: '2026-04-26T06:16:56.298Z' }),
|
||||
JSON.stringify({ version: '0.11.0', status: 'partial', ts: '2026-04-26T06:19:03.617Z' }),
|
||||
].join('\n') + '\n',
|
||||
);
|
||||
|
||||
const result = run(['doctor', '--fast', '--json']);
|
||||
// No FAIL on minions_migration — the v0.11.0 partials are stale
|
||||
// because v0.16.0 (a newer release) completed.
|
||||
const checks = JSON.parse(result.stdout).checks as Array<{ name: string; status: string }>;
|
||||
const minions = checks.find(c => c.name === 'minions_migration');
|
||||
if (minions) {
|
||||
expect(minions.status).not.toBe('fail');
|
||||
}
|
||||
// Critically: the test fixture would have caused exit 1 under the old
|
||||
// (no-override) logic because of the stale partial flag. Under the new
|
||||
// logic, doctor exits 0 (or only warns about non-related checks).
|
||||
expect(result.exitCode).toBe(0);
|
||||
});
|
||||
|
||||
test('filesystem: stale partial NOT superseded → still flagged', () => {
|
||||
// The override only fires when a >= partial version has completed.
|
||||
// Older completes (e.g. v0.10 complete + v0.16 partial) do NOT
|
||||
// supersede the partial; the partial still indicates a real problem.
|
||||
const migrationsDir = join(tmp, '.gbrain', 'migrations');
|
||||
mkdirSync(migrationsDir, { recursive: true });
|
||||
writeFileSync(
|
||||
join(migrationsDir, 'completed.jsonl'),
|
||||
[
|
||||
JSON.stringify({ version: '0.10.0', status: 'complete' }),
|
||||
JSON.stringify({ version: '0.16.0', status: 'partial' }),
|
||||
].join('\n') + '\n',
|
||||
);
|
||||
|
||||
const result = run(['doctor', '--fast', '--json']);
|
||||
expect(result.exitCode).toBe(1);
|
||||
const checks = JSON.parse(result.stdout).checks as Array<{ name: string; status: string; message: string }>;
|
||||
const minions = checks.find(c => c.name === 'minions_migration');
|
||||
expect(minions!.status).toBe('fail');
|
||||
expect(minions!.message).toContain('0.16.0');
|
||||
});
|
||||
|
||||
test('human output: prints MINIONS HALF-INSTALLED loud banner', () => {
|
||||
// Same fixture as the first test, but check the human-readable output
|
||||
// includes the exact banner phrase an OpenClaw host's cron script
|
||||
|
||||
@@ -21,16 +21,6 @@ describe('doctor command', () => {
|
||||
expect(stdout).toContain('--fast');
|
||||
});
|
||||
|
||||
test('frontmatter_integrity subcheck added in v0.22.4', async () => {
|
||||
const fs = await import('fs');
|
||||
const src = fs.readFileSync('src/commands/doctor.ts', 'utf8');
|
||||
// Subcheck name and call into shared scanner are present.
|
||||
expect(src).toContain("name: 'frontmatter_integrity'");
|
||||
expect(src).toContain('scanBrainSources');
|
||||
// Fix hint points at the right CLI command.
|
||||
expect(src).toContain('gbrain frontmatter validate');
|
||||
});
|
||||
|
||||
test('Check interface supports issues array', async () => {
|
||||
// `Check` is a TypeScript interface — type-only, no runtime value.
|
||||
// Importing it for type assertion is enough to validate the shape.
|
||||
|
||||
@@ -1,227 +0,0 @@
|
||||
/**
|
||||
* Engine Parity E2E
|
||||
*
|
||||
* Codex flagged that searchKeyword behavior differs structurally between
|
||||
* the two engines (Postgres uses a CTE that ranks pages then picks best
|
||||
* chunk; PGLite returns chunks directly). Without verification, source-aware
|
||||
* ranking could pass on PGLite and silently fail on Postgres.
|
||||
*
|
||||
* Strategy: seed identical corpora into both engines, run identical queries,
|
||||
* assert top-5 slug ordering matches.
|
||||
*
|
||||
* Gated by DATABASE_URL — skips gracefully if no real Postgres. Always runs
|
||||
* the PGLite half so the seed/query path is at least exercised.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
|
||||
import type { ChunkInput, SearchResult } from '../../src/core/types.ts';
|
||||
import type { BrainEngine } from '../../src/core/engine.ts';
|
||||
import { hasDatabase, setupDB, teardownDB, getEngine } from './helpers.ts';
|
||||
|
||||
const SKIP_PG = !hasDatabase();
|
||||
const describeBoth = SKIP_PG ? describe.skip : describe;
|
||||
|
||||
function basisEmbedding(idx: number, dim = 1536): Float32Array {
|
||||
const emb = new Float32Array(dim);
|
||||
emb[idx % dim] = 1.0;
|
||||
return emb;
|
||||
}
|
||||
|
||||
interface SeedPage {
|
||||
slug: string;
|
||||
type: 'writing' | 'concept' | 'note' | 'person' | 'company';
|
||||
title: string;
|
||||
body: string;
|
||||
embeddingDim: number;
|
||||
}
|
||||
|
||||
const SEED_PAGES: SeedPage[] = [
|
||||
{
|
||||
slug: 'originals/talks/article-outline-fat-code',
|
||||
type: 'writing',
|
||||
title: 'Fat Code Thin Harness — Part 3',
|
||||
body: 'fat code thin harness pattern part 3 production case studies',
|
||||
embeddingDim: 7,
|
||||
},
|
||||
{
|
||||
slug: 'concepts/fat-code-thin-harness',
|
||||
type: 'concept',
|
||||
title: 'Fat Code Thin Harness',
|
||||
body: 'reusable concept fat code thin harness architecture',
|
||||
embeddingDim: 14,
|
||||
},
|
||||
{
|
||||
slug: 'wintermute/chat/2026-04-15',
|
||||
type: 'note',
|
||||
title: '2026-04-15 chat',
|
||||
body:
|
||||
'fat code thin harness fat code thin harness discussion went on at length, ' +
|
||||
'fat code thin harness came up again and again, fat code thin harness fat code thin harness.',
|
||||
embeddingDim: 8,
|
||||
},
|
||||
{
|
||||
slug: 'wintermute/chat/2026-04-16',
|
||||
type: 'note',
|
||||
title: '2026-04-16 chat',
|
||||
body:
|
||||
'fat code thin harness once more, fat code thin harness fat code thin harness, ' +
|
||||
'still talking about fat code thin harness fat code thin harness.',
|
||||
embeddingDim: 9,
|
||||
},
|
||||
{
|
||||
slug: 'people/example-founder',
|
||||
type: 'person',
|
||||
title: 'Example Founder',
|
||||
body: 'example founder unrelated content for distraction',
|
||||
embeddingDim: 50,
|
||||
},
|
||||
];
|
||||
|
||||
async function seedEngine(eng: BrainEngine) {
|
||||
for (const p of SEED_PAGES) {
|
||||
await eng.putPage(p.slug, {
|
||||
type: p.type,
|
||||
title: p.title,
|
||||
compiled_truth: p.body,
|
||||
timeline: '',
|
||||
});
|
||||
const chunks: ChunkInput[] = [
|
||||
{
|
||||
chunk_index: 0,
|
||||
chunk_text: p.body,
|
||||
chunk_source: 'compiled_truth',
|
||||
embedding: basisEmbedding(p.embeddingDim),
|
||||
token_count: p.body.split(/\s+/).length,
|
||||
},
|
||||
];
|
||||
await eng.upsertChunks(p.slug, chunks);
|
||||
}
|
||||
}
|
||||
|
||||
const QUERIES = [
|
||||
'fat code thin harness',
|
||||
'fat code thin harness part 3',
|
||||
'fat code production',
|
||||
];
|
||||
|
||||
describeBoth('Engine parity — Postgres vs PGLite', () => {
|
||||
let pgEngine: BrainEngine;
|
||||
let pgliteEngine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
pgEngine = await setupDB();
|
||||
await seedEngine(pgEngine);
|
||||
|
||||
pgliteEngine = new PGLiteEngine();
|
||||
await pgliteEngine.connect({});
|
||||
await pgliteEngine.initSchema();
|
||||
await seedEngine(pgliteEngine);
|
||||
}, 90_000);
|
||||
|
||||
afterAll(async () => {
|
||||
await pgliteEngine.disconnect();
|
||||
await teardownDB();
|
||||
});
|
||||
|
||||
for (const q of QUERIES) {
|
||||
test(`searchKeyword: top-5 slugs match for "${q}"`, async () => {
|
||||
const pgResults = await pgEngine.searchKeyword(q, { limit: 5 });
|
||||
const pgliteResults = await pgliteEngine.searchKeyword(q, { limit: 5 });
|
||||
|
||||
const pgSlugs = pgResults.map((r: SearchResult) => r.slug);
|
||||
const pgliteSlugs = pgliteResults.map((r: SearchResult) => r.slug);
|
||||
|
||||
// Top result MUST match (the swamp-resistance guarantee).
|
||||
expect(pgSlugs[0]).toBe(pgliteSlugs[0]);
|
||||
// Sets should match (allowing some ordering drift on lower-ranked
|
||||
// results since FTS rank function differences between engines are
|
||||
// out of scope for this fix).
|
||||
expect(new Set(pgSlugs)).toEqual(new Set(pgliteSlugs));
|
||||
});
|
||||
}
|
||||
|
||||
test('searchVector: top result matches between engines', async () => {
|
||||
const queryVec = basisEmbedding(7); // article direction
|
||||
const pgResults = await pgEngine.searchVector(queryVec, { limit: 5 });
|
||||
const pgliteResults = await pgliteEngine.searchVector(queryVec, { limit: 5 });
|
||||
|
||||
expect(pgResults[0]?.slug).toBe(pgliteResults[0]?.slug);
|
||||
});
|
||||
|
||||
test('hard-exclude is consistent across engines', async () => {
|
||||
// Both engines should hide test/ pages by default; both should opt
|
||||
// them back in via include_slug_prefixes.
|
||||
await pgEngine.putPage('test/parity-fixture', {
|
||||
type: 'note',
|
||||
title: 'parity test fixture',
|
||||
compiled_truth: 'parity test fixture content',
|
||||
timeline: '',
|
||||
});
|
||||
await pgEngine.upsertChunks('test/parity-fixture', [{
|
||||
chunk_index: 0,
|
||||
chunk_text: 'parity test fixture content',
|
||||
chunk_source: 'compiled_truth',
|
||||
embedding: basisEmbedding(20),
|
||||
token_count: 5,
|
||||
}] satisfies ChunkInput[]);
|
||||
|
||||
await pgliteEngine.putPage('test/parity-fixture', {
|
||||
type: 'note',
|
||||
title: 'parity test fixture',
|
||||
compiled_truth: 'parity test fixture content',
|
||||
timeline: '',
|
||||
});
|
||||
await pgliteEngine.upsertChunks('test/parity-fixture', [{
|
||||
chunk_index: 0,
|
||||
chunk_text: 'parity test fixture content',
|
||||
chunk_source: 'compiled_truth',
|
||||
embedding: basisEmbedding(20),
|
||||
token_count: 5,
|
||||
}] satisfies ChunkInput[]);
|
||||
|
||||
const pgDefault = await pgEngine.searchKeyword('parity test fixture');
|
||||
const pgliteDefault = await pgliteEngine.searchKeyword('parity test fixture');
|
||||
expect(pgDefault.map((r: SearchResult) => r.slug)).not.toContain('test/parity-fixture');
|
||||
expect(pgliteDefault.map((r: SearchResult) => r.slug)).not.toContain('test/parity-fixture');
|
||||
|
||||
const pgOptIn = await pgEngine.searchKeyword('parity test fixture', {
|
||||
include_slug_prefixes: ['test/'],
|
||||
});
|
||||
const pgliteOptIn = await pgliteEngine.searchKeyword('parity test fixture', {
|
||||
include_slug_prefixes: ['test/'],
|
||||
});
|
||||
expect(pgOptIn.map((r: SearchResult) => r.slug)).toContain('test/parity-fixture');
|
||||
expect(pgliteOptIn.map((r: SearchResult) => r.slug)).toContain('test/parity-fixture');
|
||||
});
|
||||
|
||||
test('detail=high produces a different ranking than default on at least one engine', async () => {
|
||||
// Source-boost gates on `detail !== 'high'`. If the gate works on both
|
||||
// engines, the ordering for `detail=high` should differ from default in
|
||||
// any case where the swamp / curated pages have different raw scores.
|
||||
//
|
||||
// Postgres's CTE ranks pages then picks best chunk; ts_rank normalizes
|
||||
// by doc length so chat pages don't always swamp at the page level.
|
||||
// PGLite scores chunks directly — chat chunks beat article chunks on
|
||||
// raw ts_rank. The two engines need different parity contracts here.
|
||||
//
|
||||
// Common assertion that holds on both: detail=high must include the
|
||||
// chat pages in its result set (they're not filtered by detail), and
|
||||
// the result set should not be identical to default-detail (the boost
|
||||
// must be doing _something_ visible).
|
||||
const pgDefault = await pgEngine.searchKeyword('fat code thin harness', { limit: 5 });
|
||||
const pgHigh = await pgEngine.searchKeyword('fat code thin harness', { detail: 'high', limit: 5 });
|
||||
const pgliteDefault = await pgliteEngine.searchKeyword('fat code thin harness', { limit: 5 });
|
||||
const pgliteHigh = await pgliteEngine.searchKeyword('fat code thin harness', { detail: 'high', limit: 5 });
|
||||
|
||||
// Chat pages must be present in detail=high results on both engines.
|
||||
expect(pgHigh.some((r: SearchResult) => r.slug.startsWith('wintermute/chat/'))).toBe(true);
|
||||
expect(pgliteHigh.some((r: SearchResult) => r.slug.startsWith('wintermute/chat/'))).toBe(true);
|
||||
|
||||
// The boost must be doing something — at least one engine's ordering
|
||||
// should change between default and detail=high.
|
||||
const pgChanged = pgDefault.map((r: SearchResult) => r.slug).join(',') !== pgHigh.map((r: SearchResult) => r.slug).join(',');
|
||||
const pgliteChanged = pgliteDefault.map((r: SearchResult) => r.slug).join(',') !== pgliteHigh.map((r: SearchResult) => r.slug).join(',');
|
||||
expect(pgChanged || pgliteChanged).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -1,191 +0,0 @@
|
||||
/**
|
||||
* E2E: v0.22.4 frontmatter-guard migration end-to-end on PGLite.
|
||||
*
|
||||
* Closes plan item B14. Runs the v0_22_4 orchestrator against a real PGLite
|
||||
* brain with two registered sources and synthetic malformed brain pages on
|
||||
* disk. Asserts:
|
||||
* - audit phase writes ~/.gbrain/migrations/v0.22.4-audit.json with the
|
||||
* expected per-source counts.
|
||||
* - emit-todo phase appends one entry per source-with-issues to
|
||||
* ~/.gbrain/migrations/pending-host-work.jsonl, each pointing at
|
||||
* skills/migrations/v0.22.4.md (dotted convention) with the exact
|
||||
* gbrain frontmatter validate <source-path> --fix command.
|
||||
* - The migration is audit-only — no fixture page is mutated during
|
||||
* apply-migrations.
|
||||
*
|
||||
* Uses the __setTestEngineOverride() injection point on v0_22_4.ts (mirrors
|
||||
* the repair-jsonb test pattern). Bun's os.homedir() doesn't observe
|
||||
* process.env.HOME mutations mid-process, so we redirect via the explicit
|
||||
* test override rather than relying on env-var redirection of loadConfig().
|
||||
*
|
||||
* No DATABASE_URL needed; runs unconditionally in CI's Tier 1.
|
||||
*
|
||||
* Run: bun test test/e2e/frontmatter-migration.test.ts
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { mkdtempSync, mkdirSync, writeFileSync, readFileSync, existsSync, rmSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
|
||||
import { v0_22_4, __setTestEngineOverride } from '../../src/commands/migrations/v0_22_4.ts';
|
||||
|
||||
const fence = '---';
|
||||
|
||||
let workdir: string;
|
||||
let tmpHome: string;
|
||||
let brainRootA: string;
|
||||
let brainRootB: string;
|
||||
let engine: PGLiteEngine;
|
||||
let originalHome: string | undefined;
|
||||
const originalContents = new Map<string, string>();
|
||||
|
||||
beforeAll(async () => {
|
||||
workdir = mkdtempSync(join(tmpdir(), 'fm-migration-e2e-'));
|
||||
tmpHome = join(workdir, 'home');
|
||||
brainRootA = join(workdir, 'brain-a');
|
||||
brainRootB = join(workdir, 'brain-b');
|
||||
mkdirSync(tmpHome, { recursive: true });
|
||||
mkdirSync(brainRootA, { recursive: true });
|
||||
mkdirSync(brainRootB, { recursive: true });
|
||||
mkdirSync(join(tmpHome, '.gbrain', 'migrations'), { recursive: true });
|
||||
|
||||
// Seed fixture brain pages on disk. Source A has 2 broken pages
|
||||
// (NESTED_QUOTES + NULL_BYTES); source B has 1 broken page (NESTED_QUOTES)
|
||||
// plus 1 clean page.
|
||||
const aBrokenNested = `${fence}\ntype: person\ntitle: "Phil "Nick" Last"\n${fence}\n\nbody-a-nested`;
|
||||
const aBrokenNull = `${fence}\ntype: concept\ntitle: ok\n${fence}\n\nbody-a-null\x00drop`;
|
||||
const bBroken = `${fence}\ntype: company\ntitle: "Co "Inc" Name"\n${fence}\n\nbody-b`;
|
||||
const bClean = `${fence}\ntype: concept\ntitle: clean\n${fence}\n\nbody-b-clean`;
|
||||
|
||||
const filesToTrack: Array<{ path: string; content: string }> = [
|
||||
{ path: join(brainRootA, 'people', 'phil.md'), content: aBrokenNested },
|
||||
{ path: join(brainRootA, 'concepts', 'foo.md'), content: aBrokenNull },
|
||||
{ path: join(brainRootB, 'companies', 'co.md'), content: bBroken },
|
||||
{ path: join(brainRootB, 'concepts', 'bar.md'), content: bClean },
|
||||
];
|
||||
|
||||
for (const f of filesToTrack) {
|
||||
mkdirSync(join(f.path, '..'), { recursive: true });
|
||||
writeFileSync(f.path, f.content);
|
||||
originalContents.set(f.path, f.content);
|
||||
}
|
||||
|
||||
// Single in-memory PGLite for the whole test. We inject it into the
|
||||
// orchestrator via __setTestEngineOverride so phaseBAudit skips loadConfig.
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, local_path) VALUES ($1, $1, $2)`,
|
||||
['alpha', brainRootA],
|
||||
);
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, local_path) VALUES ($1, $1, $2)`,
|
||||
['beta', brainRootB],
|
||||
);
|
||||
__setTestEngineOverride(engine);
|
||||
|
||||
// Redirect ~/.gbrain/migrations/ output. The orchestrator's gbrainDir()
|
||||
// helper reads process.env.HOME at call time, so the override takes
|
||||
// effect even though Bun's os.homedir() does not observe mid-process
|
||||
// mutations.
|
||||
originalHome = process.env.HOME;
|
||||
process.env.HOME = tmpHome;
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
__setTestEngineOverride(null);
|
||||
if (engine) await engine.disconnect();
|
||||
if (originalHome === undefined) delete process.env.HOME;
|
||||
else process.env.HOME = originalHome;
|
||||
rmSync(workdir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
describe('E2E: v0.22.4 frontmatter-guard migration', () => {
|
||||
test('orchestrator runs end-to-end and produces the expected artifacts', async () => {
|
||||
const result = await v0_22_4.orchestrator({
|
||||
yes: true,
|
||||
dryRun: false,
|
||||
noAutopilotInstall: true,
|
||||
});
|
||||
|
||||
expect(result.version).toBe('0.22.4');
|
||||
expect(['complete', 'partial']).toContain(result.status);
|
||||
expect(result.phases.length).toBe(3);
|
||||
const auditPhase = result.phases.find((p) => p.name === 'audit')!;
|
||||
expect(auditPhase.status).toBe('complete');
|
||||
const emitPhase = result.phases.find((p) => p.name === 'emit-todo')!;
|
||||
expect(emitPhase.status).toBe('complete');
|
||||
expect(result.pending_host_work).toBe(2);
|
||||
});
|
||||
|
||||
test('audit JSON report exists and has per-source counts', () => {
|
||||
const reportPath = join(tmpHome, '.gbrain', 'migrations', 'v0.22.4-audit.json');
|
||||
expect(existsSync(reportPath)).toBe(true);
|
||||
const report = JSON.parse(readFileSync(reportPath, 'utf8'));
|
||||
|
||||
expect(report.ok).toBe(false);
|
||||
expect(report.total).toBeGreaterThan(0);
|
||||
expect(report.scanned_at).toMatch(/\d{4}-\d{2}-\d{2}T/);
|
||||
|
||||
const alpha = report.per_source.find((s: any) => s.source_id === 'alpha');
|
||||
const beta = report.per_source.find((s: any) => s.source_id === 'beta');
|
||||
expect(alpha).toBeDefined();
|
||||
expect(beta).toBeDefined();
|
||||
|
||||
// Source A has NESTED_QUOTES (in phil.md) and NULL_BYTES (in foo.md).
|
||||
// YAML_PARSE may also fire on the nested-quote page since gray-matter
|
||||
// throws — assert each expected code shows up at least once.
|
||||
expect(alpha.errors_by_code.NESTED_QUOTES).toBeGreaterThanOrEqual(1);
|
||||
expect(alpha.errors_by_code.NULL_BYTES).toBeGreaterThanOrEqual(1);
|
||||
expect(alpha.total).toBeGreaterThanOrEqual(2);
|
||||
expect(beta.errors_by_code.NESTED_QUOTES).toBeGreaterThanOrEqual(1);
|
||||
expect(beta.total).toBeGreaterThanOrEqual(1);
|
||||
|
||||
// Sample lists carry the affected file paths for each source.
|
||||
expect(alpha.sample.some((s: any) => s.path.includes('phil.md'))).toBe(true);
|
||||
expect(beta.sample.some((s: any) => s.path.includes('co.md'))).toBe(true);
|
||||
});
|
||||
|
||||
test('pending-host-work.jsonl carries one entry per source-with-issues', () => {
|
||||
const jsonlPath = join(tmpHome, '.gbrain', 'migrations', 'pending-host-work.jsonl');
|
||||
expect(existsSync(jsonlPath)).toBe(true);
|
||||
const lines = readFileSync(jsonlPath, 'utf8').split('\n').filter(Boolean);
|
||||
expect(lines.length).toBe(2);
|
||||
|
||||
const entries = lines.map((l) => JSON.parse(l));
|
||||
const ids = entries.map((e: any) => e.source_id).sort();
|
||||
expect(ids).toEqual(['alpha', 'beta']);
|
||||
|
||||
for (const e of entries) {
|
||||
expect(e.migration).toBe('0.22.4');
|
||||
// Dotted-filename convention: the skill pointer matches the user-facing
|
||||
// migration doc at skills/migrations/v0.22.4.md, NOT the underscored
|
||||
// TS module path.
|
||||
expect(e.skill).toBe('skills/migrations/v0.22.4.md');
|
||||
expect(e.command).toContain('gbrain frontmatter validate');
|
||||
expect(e.command).toContain('--fix');
|
||||
expect(e.command).toContain(e.source_path);
|
||||
}
|
||||
});
|
||||
|
||||
test('audit phase did NOT mutate any fixture brain page (audit-only contract)', () => {
|
||||
for (const [path, original] of originalContents) {
|
||||
expect(readFileSync(path, 'utf8')).toBe(original);
|
||||
// Nor should there be a .bak — the migration never invokes writeBrainPage.
|
||||
expect(existsSync(path + '.bak')).toBe(false);
|
||||
}
|
||||
});
|
||||
|
||||
test('orchestrator is idempotent — re-running does not duplicate JSONL entries', async () => {
|
||||
await v0_22_4.orchestrator({
|
||||
yes: true,
|
||||
dryRun: false,
|
||||
noAutopilotInstall: true,
|
||||
});
|
||||
const jsonlPath = join(tmpHome, '.gbrain', 'migrations', 'pending-host-work.jsonl');
|
||||
const lines = readFileSync(jsonlPath, 'utf8').split('\n').filter(Boolean);
|
||||
expect(lines.length).toBe(2);
|
||||
});
|
||||
});
|
||||
@@ -1,233 +0,0 @@
|
||||
/**
|
||||
* E2E tests for src/mcp/http-transport.ts against real Postgres.
|
||||
*
|
||||
* Catches schema drift (column-name typos that would slip past the unit suite's
|
||||
* stubbed engine.sql) and proves the F1+F2+F3 dispatch pipeline works against a
|
||||
* real handler doing real DB work. Also exercises the SQL-level last_used_at
|
||||
* debounce against real Postgres semantics.
|
||||
*
|
||||
* Run: DATABASE_URL=... bun test test/e2e/http-transport.test.ts
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { createHash, randomBytes } from 'crypto';
|
||||
import { startHttpTransport } from '../../src/mcp/http-transport.ts';
|
||||
import { hasDatabase, setupDB, teardownDB, getEngine, getConn } from './helpers.ts';
|
||||
|
||||
const skip = !hasDatabase();
|
||||
const describeE2E = skip ? describe.skip : describe;
|
||||
|
||||
if (skip) {
|
||||
console.log('Skipping E2E http-transport tests (DATABASE_URL not set)');
|
||||
}
|
||||
|
||||
interface ServerHandle {
|
||||
port: number;
|
||||
stop: () => Promise<void>;
|
||||
}
|
||||
|
||||
function generateToken(): string {
|
||||
return 'gbrain_test_' + randomBytes(16).toString('hex');
|
||||
}
|
||||
|
||||
function hashToken(token: string): string {
|
||||
return createHash('sha256').update(token).digest('hex');
|
||||
}
|
||||
|
||||
async function startServer(): Promise<ServerHandle> {
|
||||
const engine = getEngine();
|
||||
const server = await startHttpTransport({ port: 0, engine: engine as any });
|
||||
return {
|
||||
port: (server as any).port,
|
||||
stop: async () => { (server as any).stop(true); },
|
||||
};
|
||||
}
|
||||
|
||||
function rpc(method: string, params?: unknown, id: number = 1) {
|
||||
return JSON.stringify({ jsonrpc: '2.0', id, method, ...(params !== undefined ? { params } : {}) });
|
||||
}
|
||||
|
||||
describeE2E('http-transport E2E (real Postgres)', () => {
|
||||
let srv: ServerHandle;
|
||||
let validToken: string;
|
||||
let revokedToken: string;
|
||||
let validTokenName: string;
|
||||
|
||||
beforeAll(async () => {
|
||||
await setupDB();
|
||||
const conn = getConn();
|
||||
|
||||
// Seed a valid + revoked token directly via SQL (mirrors auth.ts's create path).
|
||||
validToken = generateToken();
|
||||
validTokenName = 'e2e-valid-' + randomBytes(4).toString('hex');
|
||||
await conn.unsafe(
|
||||
'INSERT INTO access_tokens (name, token_hash) VALUES ($1, $2)',
|
||||
[validTokenName, hashToken(validToken)],
|
||||
);
|
||||
revokedToken = generateToken();
|
||||
await conn.unsafe(
|
||||
'INSERT INTO access_tokens (name, token_hash, revoked_at) VALUES ($1, $2, now())',
|
||||
['e2e-revoked-' + randomBytes(4).toString('hex'), hashToken(revokedToken)],
|
||||
);
|
||||
|
||||
srv = await startServer();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
if (srv) await srv.stop();
|
||||
await teardownDB();
|
||||
});
|
||||
|
||||
test('1. /health → 200 with expected JSON shape', async () => {
|
||||
const r = await fetch(`http://localhost:${srv.port}/health`);
|
||||
expect(r.status).toBe(200);
|
||||
const body = await r.json();
|
||||
expect(body.status).toBe('ok');
|
||||
expect(body.transport).toBe('http');
|
||||
expect(body.version).toBeString();
|
||||
});
|
||||
|
||||
test('2. /mcp tools/list with valid Bearer → 200 + ops list', async () => {
|
||||
const r = await fetch(`http://localhost:${srv.port}/mcp`, {
|
||||
method: 'POST',
|
||||
headers: { 'Authorization': `Bearer ${validToken}`, 'Content-Type': 'application/json' },
|
||||
body: rpc('tools/list'),
|
||||
});
|
||||
expect(r.status).toBe(200);
|
||||
const body = await r.json();
|
||||
expect(body.result.tools).toBeArray();
|
||||
expect(body.result.tools.length).toBeGreaterThan(5);
|
||||
expect(r.headers.get('content-type')).toContain('application/json');
|
||||
});
|
||||
|
||||
test('3. /mcp tools/call (real op: list_pages) round-trips successfully — F1+F2+F3 guard', async () => {
|
||||
const r = await fetch(`http://localhost:${srv.port}/mcp`, {
|
||||
method: 'POST',
|
||||
headers: { 'Authorization': `Bearer ${validToken}`, 'Content-Type': 'application/json' },
|
||||
body: rpc('tools/call', { name: 'list_pages', arguments: { limit: 5 } }),
|
||||
});
|
||||
expect(r.status).toBe(200);
|
||||
const body = await r.json();
|
||||
expect(body.jsonrpc).toBe('2.0');
|
||||
expect(body.result.content).toBeArray();
|
||||
// Should NOT be an error — handler ran successfully against the real engine.
|
||||
expect(body.result.isError).toBeUndefined();
|
||||
// Result text should parse as JSON (list_pages returns an object/array)
|
||||
const resultText = body.result.content[0].text;
|
||||
const parsed = JSON.parse(resultText);
|
||||
expect(parsed).toBeDefined();
|
||||
});
|
||||
|
||||
test('4. revoked token → 401', async () => {
|
||||
const r = await fetch(`http://localhost:${srv.port}/mcp`, {
|
||||
method: 'POST',
|
||||
headers: { 'Authorization': `Bearer ${revokedToken}`, 'Content-Type': 'application/json' },
|
||||
body: rpc('tools/list'),
|
||||
});
|
||||
expect(r.status).toBe(401);
|
||||
});
|
||||
|
||||
test('5. last_used_at debounce: two consecutive valid calls → only one UPDATE within 60s', async () => {
|
||||
const conn = getConn();
|
||||
|
||||
// Reset last_used_at to NULL so the first call definitely updates
|
||||
await conn.unsafe('UPDATE access_tokens SET last_used_at = NULL WHERE name = $1', [validTokenName]);
|
||||
|
||||
// First request — should update last_used_at
|
||||
await fetch(`http://localhost:${srv.port}/mcp`, {
|
||||
method: 'POST',
|
||||
headers: { 'Authorization': `Bearer ${validToken}`, 'Content-Type': 'application/json' },
|
||||
body: rpc('tools/list'),
|
||||
});
|
||||
// Give the fire-and-forget UPDATE a moment to land
|
||||
await new Promise(r => setTimeout(r, 50));
|
||||
|
||||
const [row1] = await conn.unsafe(
|
||||
'SELECT last_used_at FROM access_tokens WHERE name = $1',
|
||||
[validTokenName],
|
||||
) as { last_used_at: Date | null }[];
|
||||
expect(row1.last_used_at).not.toBeNull();
|
||||
const firstUpdate = row1.last_used_at;
|
||||
|
||||
// Second request immediately — should NOT trigger another UPDATE (debounced by SQL WHERE)
|
||||
await fetch(`http://localhost:${srv.port}/mcp`, {
|
||||
method: 'POST',
|
||||
headers: { 'Authorization': `Bearer ${validToken}`, 'Content-Type': 'application/json' },
|
||||
body: rpc('tools/list'),
|
||||
});
|
||||
await new Promise(r => setTimeout(r, 50));
|
||||
|
||||
const [row2] = await conn.unsafe(
|
||||
'SELECT last_used_at FROM access_tokens WHERE name = $1',
|
||||
[validTokenName],
|
||||
) as { last_used_at: Date | null }[];
|
||||
// Same timestamp = same UPDATE = debounce held
|
||||
expect(row2.last_used_at?.getTime()).toBe(firstUpdate?.getTime());
|
||||
});
|
||||
|
||||
test('6. last_used_at debounce: simulating 65s gap → second request DOES update', async () => {
|
||||
const conn = getConn();
|
||||
|
||||
// Set last_used_at to 65 seconds ago — simulates the time gap without waiting in real time
|
||||
await conn.unsafe(
|
||||
`UPDATE access_tokens SET last_used_at = now() - interval '65 seconds' WHERE name = $1`,
|
||||
[validTokenName],
|
||||
);
|
||||
const [before] = await conn.unsafe(
|
||||
'SELECT last_used_at FROM access_tokens WHERE name = $1',
|
||||
[validTokenName],
|
||||
) as { last_used_at: Date | null }[];
|
||||
|
||||
await fetch(`http://localhost:${srv.port}/mcp`, {
|
||||
method: 'POST',
|
||||
headers: { 'Authorization': `Bearer ${validToken}`, 'Content-Type': 'application/json' },
|
||||
body: rpc('tools/list'),
|
||||
});
|
||||
await new Promise(r => setTimeout(r, 50));
|
||||
|
||||
const [after] = await conn.unsafe(
|
||||
'SELECT last_used_at FROM access_tokens WHERE name = $1',
|
||||
[validTokenName],
|
||||
) as { last_used_at: Date | null }[];
|
||||
expect(after.last_used_at?.getTime()).toBeGreaterThan(before.last_used_at!.getTime());
|
||||
});
|
||||
|
||||
test('7. mcp_request_log gets a row per request', async () => {
|
||||
const conn = getConn();
|
||||
const beforeRows = await conn.unsafe('SELECT count(*)::int AS n FROM mcp_request_log') as { n: number }[];
|
||||
const beforeN = beforeRows[0].n;
|
||||
|
||||
await fetch(`http://localhost:${srv.port}/mcp`, {
|
||||
method: 'POST',
|
||||
headers: { 'Authorization': `Bearer ${validToken}`, 'Content-Type': 'application/json' },
|
||||
body: rpc('tools/list'),
|
||||
});
|
||||
// Fire-and-forget audit insert — give it a tick
|
||||
await new Promise(r => setTimeout(r, 100));
|
||||
|
||||
const afterRows = await conn.unsafe('SELECT count(*)::int AS n FROM mcp_request_log') as { n: number }[];
|
||||
expect(afterRows[0].n).toBeGreaterThan(beforeN);
|
||||
|
||||
const [row] = await conn.unsafe(
|
||||
`SELECT token_name, operation, status, latency_ms FROM mcp_request_log
|
||||
WHERE token_name = $1 ORDER BY created_at DESC LIMIT 1`,
|
||||
[validTokenName],
|
||||
) as { token_name: string; operation: string; status: string; latency_ms: number }[];
|
||||
expect(row.token_name).toBe(validTokenName);
|
||||
expect(row.operation).toBe('tools/list');
|
||||
expect(row.status).toBe('success');
|
||||
expect(row.latency_ms).toBeGreaterThanOrEqual(0);
|
||||
});
|
||||
|
||||
test('8. tools/call with malformed params → isError result with invalid_params', async () => {
|
||||
const r = await fetch(`http://localhost:${srv.port}/mcp`, {
|
||||
method: 'POST',
|
||||
headers: { 'Authorization': `Bearer ${validToken}`, 'Content-Type': 'application/json' },
|
||||
body: rpc('tools/call', { name: 'get_page', arguments: { slug: 42 } }),
|
||||
});
|
||||
expect(r.status).toBe(200);
|
||||
const body = await r.json();
|
||||
expect(body.result.isError).toBe(true);
|
||||
expect(body.result.content[0].text).toContain('invalid_params');
|
||||
});
|
||||
});
|
||||
@@ -1,168 +0,0 @@
|
||||
/**
|
||||
* E2E parity tests — scanIntegrity batch path vs sequential path.
|
||||
*
|
||||
* The batch path (Postgres-only fast path added in v0.20.x) and the sequential
|
||||
* path (engine.getAllSlugs + getPage loop) MUST return the same result for
|
||||
* every supported case, otherwise gbrain doctor reports different numbers
|
||||
* depending on engine type or whether batch was attempted.
|
||||
*
|
||||
* Codex review of the original perf commit caught a multi-source dedup
|
||||
* regression: the batch SQL scanned raw (source_id, slug) rows while
|
||||
* sequential's getAllSlugs() returned a Set<string>. v0.22.7 adds
|
||||
* SELECT DISTINCT ON (slug) to the batch SQL; these tests prove parity.
|
||||
*
|
||||
* Run: DATABASE_URL=... bun test test/e2e/integrity-batch.test.ts
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { hasDatabase, setupDB, teardownDB, getEngine, getConn } from './helpers.ts';
|
||||
import { scanIntegrity } from '../../src/commands/integrity.ts';
|
||||
|
||||
const skip = !hasDatabase();
|
||||
const describeE2E = skip ? describe.skip : describe;
|
||||
|
||||
if (skip) {
|
||||
console.log('Skipping E2E integrity batch parity tests (DATABASE_URL not set)');
|
||||
}
|
||||
|
||||
describeE2E('scanIntegrity batch parity (E2E, Postgres-only)', () => {
|
||||
beforeAll(async () => {
|
||||
await setupDB();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await teardownDB();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
// Clean slate per case so fixtures don't leak across describes.
|
||||
const conn = getConn();
|
||||
await conn.unsafe(`TRUNCATE pages CASCADE`);
|
||||
});
|
||||
|
||||
describe('dedup', () => {
|
||||
test('multi-source duplicate slugs scan once, not once-per-source', async () => {
|
||||
const engine = getEngine();
|
||||
const conn = getConn();
|
||||
|
||||
// Seed default-source page via the engine.
|
||||
await engine.putPage('people/alice', {
|
||||
type: 'person',
|
||||
title: 'Alice',
|
||||
compiled_truth: 'Alice writes about AI safety.',
|
||||
timeline: '',
|
||||
frontmatter: {},
|
||||
});
|
||||
|
||||
// Seed alt-source row via raw SQL — engine.putPage doesn't take a source_id,
|
||||
// and we specifically need to test that DISTINCT ON (slug) collapses
|
||||
// the multi-source rows into one scan.
|
||||
await conn.unsafe(`
|
||||
INSERT INTO sources (id, name) VALUES ('test-source-2', 'test-source-2')
|
||||
ON CONFLICT DO NOTHING
|
||||
`);
|
||||
await conn.unsafe(`
|
||||
INSERT INTO pages (source_id, slug, type, title, compiled_truth, timeline, frontmatter)
|
||||
VALUES ('test-source-2', 'people/alice', 'person', 'Alice (alt source)',
|
||||
'Alice from another source.', '', '{}'::jsonb)
|
||||
`);
|
||||
|
||||
const batchResult = await scanIntegrity(engine, { limit: 100, batchLoad: true });
|
||||
const seqResult = await scanIntegrity(engine, { limit: 100, batchLoad: false });
|
||||
|
||||
// Both paths must report the same number of distinct slugs scanned.
|
||||
// Pre-fix: batch reported 2 (one per source row), sequential reported 1.
|
||||
expect(batchResult.pagesScanned).toBe(seqResult.pagesScanned);
|
||||
expect(batchResult.pagesScanned).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe('hits', () => {
|
||||
test('bareHits and externalHits arrays match between paths', async () => {
|
||||
const engine = getEngine();
|
||||
|
||||
await engine.putPage('people/alice', {
|
||||
type: 'person',
|
||||
title: 'Alice',
|
||||
compiled_truth: 'Alice tweeted about AI safety last week.',
|
||||
timeline: '',
|
||||
frontmatter: {},
|
||||
});
|
||||
await engine.putPage('people/bob', {
|
||||
type: 'person',
|
||||
title: 'Bob',
|
||||
compiled_truth: 'Bob wrote at [example](https://example.com/bob).',
|
||||
timeline: '',
|
||||
frontmatter: {},
|
||||
});
|
||||
|
||||
const batchResult = await scanIntegrity(engine, { limit: 100, batchLoad: true });
|
||||
const seqResult = await scanIntegrity(engine, { limit: 100, batchLoad: false });
|
||||
|
||||
expect(batchResult.bareHits.length).toBe(seqResult.bareHits.length);
|
||||
expect(batchResult.externalHits.length).toBe(seqResult.externalHits.length);
|
||||
expect(batchResult.bareHits.map(h => h.slug).sort()).toEqual(
|
||||
seqResult.bareHits.map(h => h.slug).sort(),
|
||||
);
|
||||
expect(batchResult.externalHits.map(h => h.slug).sort()).toEqual(
|
||||
seqResult.externalHits.map(h => h.slug).sort(),
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('validate', () => {
|
||||
test('validate:false (boolean) page is skipped on both paths', async () => {
|
||||
const engine = getEngine();
|
||||
|
||||
await engine.putPage('people/alice', {
|
||||
type: 'person',
|
||||
title: 'Alice',
|
||||
compiled_truth: 'Alice tweeted about something.',
|
||||
timeline: '',
|
||||
frontmatter: {},
|
||||
});
|
||||
await engine.putPage('people/legacy', {
|
||||
type: 'person',
|
||||
title: 'Legacy',
|
||||
compiled_truth: 'Legacy tweeted about old stuff.',
|
||||
timeline: '',
|
||||
frontmatter: { validate: false },
|
||||
});
|
||||
|
||||
const batchResult = await scanIntegrity(engine, { limit: 100, batchLoad: true });
|
||||
const seqResult = await scanIntegrity(engine, { limit: 100, batchLoad: false });
|
||||
|
||||
expect(batchResult.pagesScanned).toBe(seqResult.pagesScanned);
|
||||
expect(batchResult.pagesScanned).toBe(1);
|
||||
expect(batchResult.bareHits.map(h => h.slug)).not.toContain('people/legacy');
|
||||
expect(seqResult.bareHits.map(h => h.slug)).not.toContain('people/legacy');
|
||||
});
|
||||
});
|
||||
|
||||
describe('topPages', () => {
|
||||
test('topPages ordering matches between paths', async () => {
|
||||
const engine = getEngine();
|
||||
|
||||
// Alice has 2 bare-tweet hits; Bob has 1.
|
||||
await engine.putPage('people/alice', {
|
||||
type: 'person',
|
||||
title: 'Alice',
|
||||
compiled_truth: 'Alice tweeted today. Alice tweeted yesterday too.',
|
||||
timeline: '',
|
||||
frontmatter: {},
|
||||
});
|
||||
await engine.putPage('people/bob', {
|
||||
type: 'person',
|
||||
title: 'Bob',
|
||||
compiled_truth: 'Bob tweeted once.',
|
||||
timeline: '',
|
||||
frontmatter: {},
|
||||
});
|
||||
|
||||
const batchResult = await scanIntegrity(engine, { limit: 100, batchLoad: true });
|
||||
const seqResult = await scanIntegrity(engine, { limit: 100, batchLoad: false });
|
||||
|
||||
expect(batchResult.topPages).toEqual(seqResult.topPages);
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -43,7 +43,7 @@ beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({}); // in-memory PGLite
|
||||
await engine.initSchema(); // installs pages, minion_jobs, config, etc.
|
||||
}, 30000);
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
|
||||
@@ -43,27 +43,13 @@ async function waitTerminal(queue: MinionQueue, id: number, timeoutMs = 15000):
|
||||
}
|
||||
|
||||
describeE2E('E2E: Minions shell handler', () => {
|
||||
let originalAllowShellJobs: string | undefined;
|
||||
|
||||
beforeAll(async () => {
|
||||
// The shell handler refuses to run unless GBRAIN_ALLOW_SHELL_JOBS=1 is
|
||||
// set on the worker process (defense-in-depth: the env var is the
|
||||
// operator-trust gate, separate from the trusted-add allowProtectedSubmit
|
||||
// flag). The PGLite sibling test sets this in its beforeAll for the same
|
||||
// reason; without it shell jobs land in `dead`.
|
||||
originalAllowShellJobs = process.env.GBRAIN_ALLOW_SHELL_JOBS;
|
||||
process.env.GBRAIN_ALLOW_SHELL_JOBS = '1';
|
||||
await setupDB();
|
||||
await runMigrations(getEngine());
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await teardownDB();
|
||||
if (originalAllowShellJobs === undefined) {
|
||||
delete process.env.GBRAIN_ALLOW_SHELL_JOBS;
|
||||
} else {
|
||||
process.env.GBRAIN_ALLOW_SHELL_JOBS = originalAllowShellJobs;
|
||||
}
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
|
||||
@@ -1,93 +0,0 @@
|
||||
/**
|
||||
* E2E test for PostgresEngine forward-reference bootstrap.
|
||||
*
|
||||
* Codex caught that `test/e2e/helpers.ts:74` uses the standalone
|
||||
* `db.initSchema()` from `src/core/db.ts`, which only runs SCHEMA_SQL and
|
||||
* never calls runMigrations(). A test using that helper would NOT exercise
|
||||
* `PostgresEngine.initSchema()`'s reordered path, producing false-positive
|
||||
* coverage. This test deliberately bypasses the standard helper and
|
||||
* instantiates `PostgresEngine` directly, calling `engine.initSchema()` so
|
||||
* the bootstrap → SCHEMA_SQL → runMigrations sequence runs end-to-end.
|
||||
*
|
||||
* Covers issues #366, #375, #378 — Postgres-side wedges where pre-v0.18
|
||||
* brains crashed on `column "source_id" does not exist`.
|
||||
*
|
||||
* NOTE: snapshot-based historical state simulation is out of scope for this
|
||||
* wave (would require maintaining historical schema dumps). The test
|
||||
* mutates a fresh-LATEST brain to a pre-v0.18 shape; codex flagged this as
|
||||
* approximate. Acceptable here because the bootstrap's contract is narrow:
|
||||
* "given a brain that lacks the specific forward-references, initSchema
|
||||
* produces a brain at LATEST." The test exercises exactly that contract.
|
||||
*
|
||||
* Run: DATABASE_URL=postgresql://... bun run test:e2e test/e2e/postgres-bootstrap.test.ts
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { PostgresEngine } from '../../src/core/postgres-engine.ts';
|
||||
import { LATEST_VERSION } from '../../src/core/migrate.ts';
|
||||
|
||||
const DATABASE_URL = process.env.DATABASE_URL;
|
||||
const skip = !DATABASE_URL;
|
||||
|
||||
describe.skipIf(skip)('PostgresEngine forward-reference bootstrap (E2E)', () => {
|
||||
let engine: PostgresEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PostgresEngine();
|
||||
await engine.connect({ database_url: DATABASE_URL! });
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
test('PostgresEngine.initSchema applies bootstrap → SCHEMA_SQL → migrations on pre-v0.18 brain', async () => {
|
||||
// First call: bring the test DB to LATEST shape so we have something to mutate.
|
||||
await engine.initSchema();
|
||||
|
||||
// Clear data from prior tests in the suite. Adding a UNIQUE(slug)
|
||||
// constraint below would fail if multi-source fixtures left rows with
|
||||
// duplicate slugs across sources (which is valid under the composite
|
||||
// UNIQUE this test is undoing).
|
||||
const conn = (engine as any).sql;
|
||||
await conn.unsafe(`TRUNCATE pages, content_chunks, links, tags, raw_data, timeline_entries, page_versions, ingest_log RESTART IDENTITY CASCADE`);
|
||||
|
||||
// Mutate to pre-v0.18 shape: drop source_id and the sources table.
|
||||
// The advisory lock is released between initSchema calls, so this
|
||||
// direct DDL won't deadlock.
|
||||
await conn.unsafe(`
|
||||
ALTER TABLE pages DROP CONSTRAINT IF EXISTS pages_source_slug_key;
|
||||
ALTER TABLE pages ADD CONSTRAINT pages_slug_key UNIQUE (slug);
|
||||
DROP INDEX IF EXISTS idx_pages_source_id;
|
||||
ALTER TABLE pages DROP COLUMN IF EXISTS source_id CASCADE;
|
||||
DROP TABLE IF EXISTS sources CASCADE;
|
||||
`);
|
||||
await engine.setConfig('version', '20');
|
||||
|
||||
// The path under test: full PostgresEngine.initSchema() including the
|
||||
// bootstrap call, SCHEMA_SQL replay, and runMigrations chain.
|
||||
await engine.initSchema();
|
||||
|
||||
expect(await engine.getConfig('version')).toBe(String(LATEST_VERSION));
|
||||
|
||||
// Verify the forward-referenced column exists after upgrade.
|
||||
const colCheck = await conn`
|
||||
SELECT column_name FROM information_schema.columns
|
||||
WHERE table_schema = current_schema()
|
||||
AND table_name = 'pages'
|
||||
AND column_name = 'source_id'
|
||||
`;
|
||||
expect(colCheck).toHaveLength(1);
|
||||
|
||||
// Verify the default source row was seeded.
|
||||
const srcCheck = await conn`SELECT id FROM sources WHERE id = 'default'`;
|
||||
expect(srcCheck).toHaveLength(1);
|
||||
});
|
||||
|
||||
test('PostgresEngine.initSchema is idempotent on a brain already at LATEST', async () => {
|
||||
// Fresh-LATEST brain. Calling initSchema again must not error and must
|
||||
// not regress the version.
|
||||
await engine.initSchema();
|
||||
expect(await engine.getConfig('version')).toBe(String(LATEST_VERSION));
|
||||
});
|
||||
});
|
||||
@@ -1,150 +0,0 @@
|
||||
/**
|
||||
* Hard-Exclude E2E
|
||||
*
|
||||
* Verifies the new exclude_slug_prefixes / include_slug_prefixes plumbing.
|
||||
* test/, archive/, attachments/, .raw/ are hard-excluded by default.
|
||||
* include_slug_prefixes opts back in.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
|
||||
import type { ChunkInput } from '../../src/core/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
function basisEmbedding(idx: number, dim = 1536): Float32Array {
|
||||
const emb = new Float32Array(dim);
|
||||
emb[idx % dim] = 1.0;
|
||||
return emb;
|
||||
}
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
|
||||
await engine.putPage('test/fixtures/widget', {
|
||||
type: 'note',
|
||||
title: 'Widget test fixture',
|
||||
compiled_truth: 'widget test fixture for the test suite',
|
||||
timeline: '',
|
||||
});
|
||||
await engine.upsertChunks('test/fixtures/widget', [
|
||||
{
|
||||
chunk_index: 0,
|
||||
chunk_text: 'widget test fixture for the test suite',
|
||||
chunk_source: 'compiled_truth',
|
||||
embedding: basisEmbedding(11),
|
||||
token_count: 8,
|
||||
},
|
||||
] satisfies ChunkInput[]);
|
||||
|
||||
await engine.putPage('archive/old-stuff/widget-2020', {
|
||||
type: 'note',
|
||||
title: 'Widget 2020',
|
||||
compiled_truth: 'widget archived from 2020',
|
||||
timeline: '',
|
||||
});
|
||||
await engine.upsertChunks('archive/old-stuff/widget-2020', [
|
||||
{
|
||||
chunk_index: 0,
|
||||
chunk_text: 'widget archived from 2020 — stale info about widget',
|
||||
chunk_source: 'compiled_truth',
|
||||
embedding: basisEmbedding(12),
|
||||
token_count: 8,
|
||||
},
|
||||
] satisfies ChunkInput[]);
|
||||
|
||||
await engine.putPage('concepts/widget-pattern', {
|
||||
type: 'concept',
|
||||
title: 'Widget Pattern',
|
||||
compiled_truth: 'the widget pattern is a useful design pattern',
|
||||
timeline: '',
|
||||
});
|
||||
await engine.upsertChunks('concepts/widget-pattern', [
|
||||
{
|
||||
chunk_index: 0,
|
||||
chunk_text: 'the widget pattern is a useful widget design pattern',
|
||||
chunk_source: 'compiled_truth',
|
||||
embedding: basisEmbedding(13),
|
||||
token_count: 9,
|
||||
},
|
||||
] satisfies ChunkInput[]);
|
||||
}, 60_000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
describe('searchKeyword default hard-excludes', () => {
|
||||
test('test/ pages are hidden by default', async () => {
|
||||
const results = await engine.searchKeyword('widget');
|
||||
const slugs = results.map(r => r.slug);
|
||||
expect(slugs).not.toContain('test/fixtures/widget');
|
||||
});
|
||||
|
||||
test('archive/ pages are hidden by default', async () => {
|
||||
const results = await engine.searchKeyword('widget');
|
||||
const slugs = results.map(r => r.slug);
|
||||
expect(slugs).not.toContain('archive/old-stuff/widget-2020');
|
||||
});
|
||||
|
||||
test('curated content is unaffected', async () => {
|
||||
const results = await engine.searchKeyword('widget');
|
||||
const slugs = results.map(r => r.slug);
|
||||
expect(slugs).toContain('concepts/widget-pattern');
|
||||
});
|
||||
});
|
||||
|
||||
describe('searchKeyword include_slug_prefixes opt-back-in', () => {
|
||||
test('include_slug_prefixes: ["test/"] surfaces test pages', async () => {
|
||||
const results = await engine.searchKeyword('widget', {
|
||||
include_slug_prefixes: ['test/'],
|
||||
});
|
||||
const slugs = results.map(r => r.slug);
|
||||
expect(slugs).toContain('test/fixtures/widget');
|
||||
// archive/ is still excluded.
|
||||
expect(slugs).not.toContain('archive/old-stuff/widget-2020');
|
||||
});
|
||||
|
||||
test('include_slug_prefixes lets caller opt back into both', async () => {
|
||||
const results = await engine.searchKeyword('widget', {
|
||||
include_slug_prefixes: ['test/', 'archive/'],
|
||||
});
|
||||
const slugs = results.map(r => r.slug);
|
||||
expect(slugs).toContain('test/fixtures/widget');
|
||||
expect(slugs).toContain('archive/old-stuff/widget-2020');
|
||||
});
|
||||
});
|
||||
|
||||
describe('searchVector hard-excludes', () => {
|
||||
test('test/ pages are excluded by default in vector search', async () => {
|
||||
const results = await engine.searchVector(basisEmbedding(11));
|
||||
// basisEmbedding(11) is the closest direction to test/fixtures/widget,
|
||||
// so without exclude it would be at top. With default exclude, it's gone.
|
||||
const slugs = results.map(r => r.slug);
|
||||
expect(slugs).not.toContain('test/fixtures/widget');
|
||||
});
|
||||
|
||||
test('include_slug_prefixes lets it back in', async () => {
|
||||
const results = await engine.searchVector(basisEmbedding(11), {
|
||||
include_slug_prefixes: ['test/'],
|
||||
});
|
||||
const slugs = results.map(r => r.slug);
|
||||
expect(slugs).toContain('test/fixtures/widget');
|
||||
});
|
||||
});
|
||||
|
||||
describe('caller-supplied exclude_slug_prefixes (additive)', () => {
|
||||
test('caller can add a custom exclude prefix on top of defaults', async () => {
|
||||
const results = await engine.searchKeyword('widget', {
|
||||
exclude_slug_prefixes: ['concepts/'],
|
||||
});
|
||||
const slugs = results.map(r => r.slug);
|
||||
// concepts/ now also excluded; with all three categories filtered, no
|
||||
// hits remain.
|
||||
expect(slugs).not.toContain('concepts/widget-pattern');
|
||||
expect(slugs).not.toContain('test/fixtures/widget');
|
||||
expect(slugs).not.toContain('archive/old-stuff/widget-2020');
|
||||
});
|
||||
});
|
||||
@@ -1,148 +0,0 @@
|
||||
/**
|
||||
* Search Swamp Resistance E2E
|
||||
*
|
||||
* Reproduces the v3-plan repro case: a curated article (originals/) competes
|
||||
* with two chat-log pages (wintermute/chat/) on similar ts_rank. With v0.21+
|
||||
* source-aware ranking, the article must rank #0.
|
||||
*
|
||||
* Mirrors the structure of search-quality.test.ts. Uses PGLite in-memory.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
|
||||
import type { ChunkInput } from '../../src/core/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
function basisEmbedding(idx: number, dim = 1536): Float32Array {
|
||||
const emb = new Float32Array(dim);
|
||||
emb[idx % dim] = 1.0;
|
||||
return emb;
|
||||
}
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
|
||||
// Curated article — short, dense, opinionated. The page that should win.
|
||||
await engine.putPage('originals/talks/article-outline-fat-code', {
|
||||
type: 'writing',
|
||||
title: 'Fat Code Thin Harness — Part 3',
|
||||
compiled_truth:
|
||||
'Fat code thin harness is the architectural pattern where business logic ' +
|
||||
'lives in fat skill files and the runtime stays thin. Part 3 covers the ' +
|
||||
'production case studies.',
|
||||
timeline: '2026-04-10: Drafted Part 3 outline.',
|
||||
});
|
||||
await engine.upsertChunks('originals/talks/article-outline-fat-code', [
|
||||
{
|
||||
chunk_index: 0,
|
||||
chunk_text:
|
||||
'Fat code thin harness — the pattern where business logic lives in fat skill files. Part 3.',
|
||||
chunk_source: 'compiled_truth',
|
||||
embedding: basisEmbedding(7),
|
||||
token_count: 20,
|
||||
},
|
||||
] satisfies ChunkInput[]);
|
||||
|
||||
// Chat swamp #1 — long page, mentions the phrase repeatedly.
|
||||
await engine.putPage('wintermute/chat/2026-04-15', {
|
||||
type: 'note',
|
||||
title: '2026-04-15 chat',
|
||||
compiled_truth: '',
|
||||
timeline:
|
||||
'fat code thin harness fat code thin harness — discussed at length. ' +
|
||||
'fat code thin harness came up again. ' +
|
||||
'The fat code thin harness pattern is something we keep returning to. ' +
|
||||
'fat code thin harness fat code thin harness fat code thin harness.',
|
||||
});
|
||||
await engine.upsertChunks('wintermute/chat/2026-04-15', [
|
||||
{
|
||||
chunk_index: 0,
|
||||
chunk_text:
|
||||
'fat code thin harness fat code thin harness discussed at length, ' +
|
||||
'the fat code thin harness pattern keeps coming back, ' +
|
||||
'fat code thin harness fat code thin harness fat code thin harness.',
|
||||
chunk_source: 'timeline',
|
||||
embedding: basisEmbedding(8),
|
||||
token_count: 30,
|
||||
},
|
||||
] satisfies ChunkInput[]);
|
||||
|
||||
// Chat swamp #2 — same shape.
|
||||
await engine.putPage('wintermute/chat/2026-04-16', {
|
||||
type: 'note',
|
||||
title: '2026-04-16 chat',
|
||||
compiled_truth: '',
|
||||
timeline:
|
||||
'fat code thin harness once more. fat code thin harness fat code thin harness. ' +
|
||||
'still talking about fat code thin harness. fat code thin harness.',
|
||||
});
|
||||
await engine.upsertChunks('wintermute/chat/2026-04-16', [
|
||||
{
|
||||
chunk_index: 0,
|
||||
chunk_text:
|
||||
'fat code thin harness once more, fat code thin harness fat code thin harness, ' +
|
||||
'still talking about fat code thin harness fat code thin harness.',
|
||||
chunk_source: 'timeline',
|
||||
embedding: basisEmbedding(9),
|
||||
token_count: 25,
|
||||
},
|
||||
] satisfies ChunkInput[]);
|
||||
}, 60_000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
describe('searchKeyword swamp resistance', () => {
|
||||
test('curated originals/ page outranks chat swamp on multi-word query', async () => {
|
||||
const results = await engine.searchKeyword('fat code thin harness');
|
||||
expect(results.length).toBeGreaterThan(0);
|
||||
const top = results[0];
|
||||
expect(top.slug).toBe('originals/talks/article-outline-fat-code');
|
||||
});
|
||||
|
||||
test('detail=high (temporal bypass) lets chat swamp re-surface', async () => {
|
||||
// With source-boost disabled, raw ts_rank wins → chat pages, which have
|
||||
// many more keyword hits, are allowed back to the top. This guards the
|
||||
// temporal-query workflow ("what did we discuss about X").
|
||||
const results = await engine.searchKeyword('fat code thin harness', { detail: 'high' });
|
||||
expect(results.length).toBeGreaterThan(0);
|
||||
// Top result should be a chat page (more keyword density per chunk).
|
||||
const topSlugs = results.slice(0, 2).map(r => r.slug);
|
||||
const anyChat = topSlugs.some(s => s.startsWith('wintermute/chat/'));
|
||||
expect(anyChat).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe('searchVector swamp resistance', () => {
|
||||
test('curated originals/ page outranks chat swamp when boost is meaningful', async () => {
|
||||
// Query vector is close to all three pages (mixed direction). Without
|
||||
// source-boost the chat pages would tie or win on raw cosine; with
|
||||
// source-boost the originals/ page dominates.
|
||||
const queryVec = new Float32Array(1536);
|
||||
queryVec[7] = 0.6; // article direction
|
||||
queryVec[8] = 0.55; // chat-1 direction (slightly higher, simulating swamp)
|
||||
queryVec[9] = 0.55; // chat-2 direction
|
||||
// Normalize so cosine math is well-formed.
|
||||
const norm = Math.sqrt(0.6 * 0.6 + 0.55 * 0.55 + 0.55 * 0.55);
|
||||
for (let i = 0; i < queryVec.length; i++) queryVec[i] = queryVec[i] / norm;
|
||||
|
||||
const results = await engine.searchVector(queryVec);
|
||||
expect(results.length).toBeGreaterThan(0);
|
||||
expect(results[0].slug).toBe('originals/talks/article-outline-fat-code');
|
||||
});
|
||||
|
||||
test('two-stage CTE returns p.source_id (regression for v0.18 multi-source)', async () => {
|
||||
const queryVec = basisEmbedding(7);
|
||||
const results = await engine.searchVector(queryVec);
|
||||
expect(results.length).toBeGreaterThan(0);
|
||||
// source_id is added by v0.18 multi-source brains; carrying it through
|
||||
// the inner→outer CTE is one of the v3 plan's pass-4 findings.
|
||||
for (const r of results) {
|
||||
expect(r.source_id).toBeDefined();
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -1,273 +0,0 @@
|
||||
/**
|
||||
* E2E test for storage tiering — Postgres-only.
|
||||
*
|
||||
* Per the v0.23.0 plan: full lifecycle. Container restart simulation:
|
||||
* write pages via Postgres, delete files from disk, run gbrain export
|
||||
* --restore-only, assert files restored. Real .gitignore round-trip.
|
||||
* Real source-resolver path through getDefaultSourcePath().
|
||||
*
|
||||
* Skips gracefully when DATABASE_URL is unset (per CLAUDE.md E2E pattern).
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { mkdtempSync, rmSync, writeFileSync, readFileSync, existsSync, mkdirSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import { setupDB, teardownDB, getEngine, hasDatabase, getConn } from './helpers.ts';
|
||||
import {
|
||||
getStorageStatus,
|
||||
formatStorageStatusHuman,
|
||||
__resetPGLiteWarn,
|
||||
} from '../../src/commands/storage.ts';
|
||||
import { manageGitignore, __resetPGLiteTierWarn } from '../../src/commands/sync.ts';
|
||||
import { getDefaultSourcePath } from '../../src/core/source-resolver.ts';
|
||||
import { __resetMissingStorageWarning } from '../../src/core/storage-config.ts';
|
||||
|
||||
if (!hasDatabase()) {
|
||||
describe('storage-tiering E2E', () => {
|
||||
test.skip('DATABASE_URL not set — skipping E2E', () => {});
|
||||
});
|
||||
} else {
|
||||
describe('storage-tiering E2E (Postgres lifecycle)', () => {
|
||||
let tmp: string;
|
||||
|
||||
beforeAll(async () => {
|
||||
await setupDB();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await teardownDB();
|
||||
});
|
||||
|
||||
beforeEach(() => {
|
||||
tmp = mkdtempSync(join(tmpdir(), 'gbrain-e2e-storage-'));
|
||||
__resetMissingStorageWarning();
|
||||
__resetPGLiteWarn();
|
||||
__resetPGLiteTierWarn();
|
||||
});
|
||||
|
||||
function cleanup(): void {
|
||||
rmSync(tmp, { recursive: true, force: true });
|
||||
}
|
||||
|
||||
function writeGbrainYml(): void {
|
||||
writeFileSync(
|
||||
join(tmp, 'gbrain.yml'),
|
||||
`storage:
|
||||
db_tracked:
|
||||
- people/
|
||||
db_only:
|
||||
- media/x/
|
||||
- media/articles/
|
||||
`,
|
||||
);
|
||||
}
|
||||
|
||||
test('engine.kind is postgres', () => {
|
||||
try {
|
||||
expect(getEngine().kind).toBe('postgres');
|
||||
} finally {
|
||||
cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
test('full lifecycle: write pages → status reports tiers → manage .gitignore → restore-only path', async () => {
|
||||
try {
|
||||
const engine = getEngine();
|
||||
|
||||
// Truncate sources + pages so this test has a clean slate.
|
||||
const conn = getConn();
|
||||
await conn.unsafe(`TRUNCATE pages, content_chunks, sources CASCADE`);
|
||||
await conn.unsafe(
|
||||
`INSERT INTO sources (id, name, local_path) VALUES ('default', 'Default', $1)`,
|
||||
[tmp],
|
||||
);
|
||||
|
||||
writeGbrainYml();
|
||||
|
||||
// Seed 4 pages: 1 db_tracked, 2 db_only, 1 unspecified.
|
||||
await engine.putPage('people/alice', {
|
||||
type: 'person',
|
||||
title: 'Alice',
|
||||
compiled_truth: 'Alice is a founder.',
|
||||
timeline: '',
|
||||
});
|
||||
await engine.putPage('media/x/tweet-1', {
|
||||
type: 'media',
|
||||
title: 'Tweet 1',
|
||||
compiled_truth: 'tweet body',
|
||||
timeline: '',
|
||||
});
|
||||
await engine.putPage('media/x/tweet-2', {
|
||||
type: 'media',
|
||||
title: 'Tweet 2',
|
||||
compiled_truth: 'tweet body 2',
|
||||
timeline: '',
|
||||
});
|
||||
await engine.putPage('random/note', {
|
||||
type: 'note',
|
||||
title: 'Random',
|
||||
compiled_truth: 'random',
|
||||
timeline: '',
|
||||
});
|
||||
|
||||
// Storage status reports tier counts correctly.
|
||||
const status = await getStorageStatus(engine, tmp);
|
||||
expect(status.totalPages).toBe(4);
|
||||
expect(status.pagesByTier.db_tracked).toBe(1);
|
||||
expect(status.pagesByTier.db_only).toBe(2);
|
||||
expect(status.pagesByTier.unspecified).toBe(1);
|
||||
|
||||
// Human formatter renders without errors.
|
||||
const out = formatStorageStatusHuman(status);
|
||||
expect(out).toContain('DB tracked: 1 pages');
|
||||
expect(out).toContain('DB only: 2 pages');
|
||||
|
||||
// .gitignore management: empty .gitignore → managed block written.
|
||||
manageGitignore(tmp, 'postgres');
|
||||
const gitignore = readFileSync(join(tmp, '.gitignore'), 'utf-8');
|
||||
expect(gitignore).toContain('# Auto-managed by gbrain');
|
||||
expect(gitignore).toContain('media/x/');
|
||||
expect(gitignore).toContain('media/articles/');
|
||||
|
||||
// Idempotency: second run adds nothing new.
|
||||
manageGitignore(tmp, 'postgres');
|
||||
const gitignore2 = readFileSync(join(tmp, '.gitignore'), 'utf-8');
|
||||
const xCount = (gitignore2.match(/^media\/x\/$/gm) || []).length;
|
||||
expect(xCount).toBe(1);
|
||||
|
||||
// Source resolution finds the local_path we registered.
|
||||
const resolvedPath = await getDefaultSourcePath(engine);
|
||||
expect(resolvedPath).toBe(tmp);
|
||||
} finally {
|
||||
cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
test('container restart simulation: db_only files missing on disk are restorable from DB', async () => {
|
||||
try {
|
||||
const engine = getEngine();
|
||||
const conn = getConn();
|
||||
|
||||
// Fresh slate.
|
||||
await conn.unsafe(`TRUNCATE pages, content_chunks, sources CASCADE`);
|
||||
await conn.unsafe(
|
||||
`INSERT INTO sources (id, name, local_path) VALUES ('default', 'Default', $1)`,
|
||||
[tmp],
|
||||
);
|
||||
|
||||
writeGbrainYml();
|
||||
|
||||
// Write some db_only pages to the database.
|
||||
await engine.putPage('media/x/tweet-1', {
|
||||
type: 'media',
|
||||
title: 'Tweet 1',
|
||||
compiled_truth: 'tweet body 1',
|
||||
timeline: '',
|
||||
});
|
||||
await engine.putPage('media/x/tweet-2', {
|
||||
type: 'media',
|
||||
title: 'Tweet 2',
|
||||
compiled_truth: 'tweet body 2',
|
||||
timeline: '',
|
||||
});
|
||||
|
||||
// Simulate "files were on disk, but the container restarted."
|
||||
// Storage status: missingFiles should list them.
|
||||
const status = await getStorageStatus(engine, tmp);
|
||||
expect(status.pagesByTier.db_only).toBe(2);
|
||||
expect(status.missingFiles.length).toBe(2);
|
||||
|
||||
// Verify slugPrefix engine filter (Issue #13) works on Postgres for
|
||||
// the prefix that --restore-only would use.
|
||||
const tierPages = await engine.listPages({ slugPrefix: 'media/x/', limit: 100 });
|
||||
expect(tierPages.map((p) => p.slug).sort()).toEqual(['media/x/tweet-1', 'media/x/tweet-2']);
|
||||
|
||||
// Source-default path resolution returns the configured local_path
|
||||
// (the typed accessor that replaces the original raw-SQL try/catch
|
||||
// in storage.ts:38).
|
||||
const path = await getDefaultSourcePath(engine);
|
||||
expect(path).toBe(tmp);
|
||||
} finally {
|
||||
cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
test('slugPrefix filter on Postgres uses index-based range scan (regression for Issue #13)', async () => {
|
||||
try {
|
||||
const engine = getEngine();
|
||||
const conn = getConn();
|
||||
await conn.unsafe(`TRUNCATE pages, content_chunks, sources CASCADE`);
|
||||
await conn.unsafe(`INSERT INTO sources (id, name) VALUES ('default', 'Default')`);
|
||||
|
||||
// Seed enough data to make a difference between scan types.
|
||||
for (let i = 0; i < 50; i++) {
|
||||
await engine.putPage(`media/x/item-${i}`, {
|
||||
type: 'media',
|
||||
title: `Item ${i}`,
|
||||
compiled_truth: 'x',
|
||||
timeline: '',
|
||||
});
|
||||
}
|
||||
for (let i = 0; i < 50; i++) {
|
||||
await engine.putPage(`people/p-${i}`, {
|
||||
type: 'person',
|
||||
title: `Person ${i}`,
|
||||
compiled_truth: 'x',
|
||||
timeline: '',
|
||||
});
|
||||
}
|
||||
|
||||
// Prefix query should return exactly 50 (people not included).
|
||||
const xResults = await engine.listPages({ slugPrefix: 'media/x/', limit: 200 });
|
||||
expect(xResults.length).toBe(50);
|
||||
for (const p of xResults) {
|
||||
expect(p.slug.startsWith('media/x/')).toBe(true);
|
||||
}
|
||||
|
||||
// Path-segment risk: slugPrefix 'media/x' (no /) would match
|
||||
// 'media/xerox' if any existed. The engine treats slugPrefix as a
|
||||
// literal string prefix; trailing-/ semantics are the matcher's
|
||||
// responsibility (storage-config.ts).
|
||||
const looseResults = await engine.listPages({ slugPrefix: 'media/x', limit: 200 });
|
||||
expect(looseResults.length).toBe(50); // no media/xerox/* exists yet
|
||||
} finally {
|
||||
cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
test('hard-error path: storage status without local_path or --repo gets null repoPath', async () => {
|
||||
try {
|
||||
const engine = getEngine();
|
||||
const conn = getConn();
|
||||
await conn.unsafe(`TRUNCATE sources CASCADE`);
|
||||
// Default source with NO local_path.
|
||||
await conn.unsafe(
|
||||
`INSERT INTO sources (id, name, local_path) VALUES ('default', 'Default', NULL)`,
|
||||
);
|
||||
|
||||
const path = await getDefaultSourcePath(engine);
|
||||
expect(path).toBeNull();
|
||||
} finally {
|
||||
cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
test('manageGitignore on Postgres engine does NOT emit PGLite warning', async () => {
|
||||
try {
|
||||
writeGbrainYml();
|
||||
const warnings: string[] = [];
|
||||
const orig = console.warn;
|
||||
console.warn = (...a: unknown[]) => warnings.push(a.map(String).join(' '));
|
||||
try {
|
||||
manageGitignore(tmp, 'postgres');
|
||||
} finally {
|
||||
console.warn = orig;
|
||||
}
|
||||
expect(warnings.filter((w) => /limited effect on PGLite/.test(w))).toEqual([]);
|
||||
} finally {
|
||||
cleanup();
|
||||
}
|
||||
});
|
||||
});
|
||||
}
|
||||
@@ -1,167 +0,0 @@
|
||||
/**
|
||||
* E2E test for parallel sync against real Postgres.
|
||||
*
|
||||
* T2 — happy path: 60-file sync at concurrency=4 against PostgresEngine
|
||||
* actually constructs N worker engines, imports correctly, and does
|
||||
* not leak connections (probe pg_stat_activity before/after).
|
||||
* P4 — benchmark: serial vs concurrency=4 timing on the same fixture so
|
||||
* the v0.22.13 CHANGELOG can quote a real number instead of "~4×".
|
||||
*
|
||||
* Gated on DATABASE_URL. Run via:
|
||||
* docker run -d --name gbrain-test-pg -e POSTGRES_USER=postgres \
|
||||
* -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=gbrain_test \
|
||||
* -p 5435:5432 pgvector/pgvector:pg16
|
||||
* DATABASE_URL=postgresql://postgres:postgres@localhost:5435/gbrain_test \
|
||||
* bun test test/e2e/sync-parallel.test.ts
|
||||
*/
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { mkdtempSync, rmSync, writeFileSync, mkdirSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import { execSync } from 'child_process';
|
||||
import { hasDatabase, setupDB, teardownDB, getEngine, getConn } from './helpers.ts';
|
||||
|
||||
const skip = !hasDatabase();
|
||||
const describeE2E = skip ? describe.skip : describe;
|
||||
|
||||
if (skip) {
|
||||
console.log('Skipping E2E sync-parallel tests (DATABASE_URL not set)');
|
||||
}
|
||||
|
||||
function seedRepo(repoPath: string, fileCount: number): string {
|
||||
execSync('git init', { cwd: repoPath, stdio: 'pipe' });
|
||||
execSync('git config user.email "test@test.com"', { cwd: repoPath, stdio: 'pipe' });
|
||||
execSync('git config user.name "Test"', { cwd: repoPath, stdio: 'pipe' });
|
||||
mkdirSync(join(repoPath, 'people'), { recursive: true });
|
||||
for (let i = 0; i < fileCount; i++) {
|
||||
writeFileSync(join(repoPath, `people/p${i}.md`), [
|
||||
'---',
|
||||
'type: person',
|
||||
`title: Person ${i}`,
|
||||
'---',
|
||||
'',
|
||||
`Person ${i} body — some text long enough to chunk.`,
|
||||
`Iteration index ${i}, generated by sync-parallel E2E.`,
|
||||
].join('\n'));
|
||||
}
|
||||
execSync('git add -A && git commit -m "initial"', { cwd: repoPath, stdio: 'pipe' });
|
||||
return execSync('git rev-parse HEAD', { cwd: repoPath, encoding: 'utf-8' }).trim();
|
||||
}
|
||||
|
||||
async function activeConnections(): Promise<number> {
|
||||
const conn = getConn();
|
||||
const rows = await conn.unsafe(`
|
||||
SELECT count(*) AS n FROM pg_stat_activity
|
||||
WHERE datname = current_database()
|
||||
AND state IS NOT NULL
|
||||
`) as Array<{ n: string }>;
|
||||
return parseInt(rows[0]?.n ?? '0', 10);
|
||||
}
|
||||
|
||||
describeE2E('E2E sync-parallel: T2 happy path + leak probe', () => {
|
||||
let repoPath: string;
|
||||
|
||||
beforeAll(async () => {
|
||||
await setupDB();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
if (repoPath) rmSync(repoPath, { recursive: true, force: true });
|
||||
await teardownDB();
|
||||
});
|
||||
|
||||
test('60-file Postgres sync at concurrency=4 imports all + no connection leak', async () => {
|
||||
repoPath = mkdtempSync(join(tmpdir(), 'gbrain-e2e-par-'));
|
||||
seedRepo(repoPath, 60);
|
||||
|
||||
const before = await activeConnections();
|
||||
|
||||
const { performSync } = await import('../../src/commands/sync.ts');
|
||||
const engine = getEngine();
|
||||
const result = await performSync(engine, {
|
||||
repoPath,
|
||||
noPull: true,
|
||||
noEmbed: true,
|
||||
concurrency: 4,
|
||||
});
|
||||
|
||||
// First sync routes through performFullSync (delegates to runImport which
|
||||
// also accepts --workers); status is 'first_sync'.
|
||||
expect(result.status).toBe('first_sync');
|
||||
|
||||
const after = await activeConnections();
|
||||
|
||||
// Allow some slack — the helper engine + sync's normal pool stay open.
|
||||
// Worker engines (4 × 2 = 8 connections) MUST have closed; if they
|
||||
// hadn't, after - before would be at least 8.
|
||||
expect(after - before).toBeLessThan(4);
|
||||
|
||||
// Verify pages are actually in the DB (via raw SQL — engine API also works).
|
||||
const conn = getConn();
|
||||
const pageRows = await conn.unsafe(
|
||||
`SELECT count(*) AS n FROM pages WHERE slug LIKE 'people/p%'`,
|
||||
) as Array<{ n: string }>;
|
||||
const count = parseInt(pageRows[0]?.n ?? '0', 10);
|
||||
expect(count).toBe(60);
|
||||
}, 60_000);
|
||||
});
|
||||
|
||||
describeE2E('E2E sync-parallel: P4 benchmark serial vs concurrency=4', () => {
|
||||
let repoSerial: string;
|
||||
let repoParallel: string;
|
||||
|
||||
beforeAll(async () => {
|
||||
await setupDB();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
if (repoSerial) rmSync(repoSerial, { recursive: true, force: true });
|
||||
if (repoParallel) rmSync(repoParallel, { recursive: true, force: true });
|
||||
await teardownDB();
|
||||
});
|
||||
|
||||
test('120-file benchmark: report serial and parallel wall-clock', async () => {
|
||||
// Two separate repos so neither sync's chunks bleed into the other.
|
||||
repoSerial = mkdtempSync(join(tmpdir(), 'gbrain-bench-serial-'));
|
||||
repoParallel = mkdtempSync(join(tmpdir(), 'gbrain-bench-parallel-'));
|
||||
seedRepo(repoSerial, 120);
|
||||
seedRepo(repoParallel, 120);
|
||||
|
||||
const { performSync } = await import('../../src/commands/sync.ts');
|
||||
const engine = getEngine();
|
||||
|
||||
// Truncate between runs to keep the benchmark honest.
|
||||
const conn = getConn();
|
||||
|
||||
const t1 = Date.now();
|
||||
await performSync(engine, {
|
||||
repoPath: repoSerial,
|
||||
noPull: true,
|
||||
noEmbed: true,
|
||||
concurrency: 1,
|
||||
});
|
||||
const serialMs = Date.now() - t1;
|
||||
|
||||
// Wipe pages before second run so neither one is "incremental".
|
||||
await conn.unsafe(`TRUNCATE pages CASCADE`);
|
||||
await conn.unsafe(`TRUNCATE config CASCADE`);
|
||||
|
||||
const t2 = Date.now();
|
||||
await performSync(engine, {
|
||||
repoPath: repoParallel,
|
||||
noPull: true,
|
||||
noEmbed: true,
|
||||
concurrency: 4,
|
||||
});
|
||||
const parallelMs = Date.now() - t2;
|
||||
|
||||
const speedup = (serialMs / parallelMs).toFixed(2);
|
||||
// Emit as a single line stdout consumers can grep for.
|
||||
console.log(`SYNC_PARALLEL_BENCH 120 files | serial=${serialMs}ms | parallel(4)=${parallelMs}ms | speedup=${speedup}x`);
|
||||
|
||||
// Soft assertion: parallel must not be slower than serial. The actual
|
||||
// speedup ratio depends heavily on Postgres latency profile and is what
|
||||
// the CHANGELOG quotes — don't gate the test on a specific multiplier.
|
||||
expect(parallelMs).toBeLessThanOrEqual(serialMs * 1.5); // +50% slack for noisy CI
|
||||
}, 120_000);
|
||||
});
|
||||
+2
-162
@@ -10,10 +10,10 @@
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { mkdtempSync, writeFileSync, rmSync, mkdirSync, unlinkSync, existsSync, readFileSync } from 'fs';
|
||||
import { mkdtempSync, writeFileSync, rmSync, mkdirSync, unlinkSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { execSync } from 'child_process';
|
||||
import { tmpdir, homedir } from 'os';
|
||||
import { tmpdir } from 'os';
|
||||
import {
|
||||
hasDatabase, setupDB, teardownDB, getEngine,
|
||||
} from './helpers.ts';
|
||||
@@ -394,163 +394,3 @@ describeE2E('E2E: Git-to-DB Sync Pipeline', () => {
|
||||
expect(page!.title).toBe('Draft Meeting Notes');
|
||||
});
|
||||
});
|
||||
|
||||
/**
|
||||
* E2E: --skip-failed loop with structured error code summary.
|
||||
*
|
||||
* Closes the v0.22.12 ship-blocker gap from issue #500 — the whole code path
|
||||
* (record → classify → block → skip → doctor render → second cycle) had only
|
||||
* mocked-JSONL unit coverage. This is the integration test that proves the
|
||||
* chain holds together with a real Postgres engine, real git history, and
|
||||
* real frontmatter validation.
|
||||
*
|
||||
* Owns its own repo + sync-failures.jsonl lifecycle so it can't leak state
|
||||
* into the shared describeE2E above. Saves and restores the user's real
|
||||
* ~/.gbrain/sync-failures.jsonl so running E2E on a developer machine
|
||||
* doesn't trash their local sync state.
|
||||
*/
|
||||
describeE2E('E2E: sync --skip-failed structured summary loop (v0.22.12, issue #500)', () => {
|
||||
let repoPath: string;
|
||||
const realFailuresPath = join(homedir(), '.gbrain', 'sync-failures.jsonl');
|
||||
let savedFailuresContent: string | null = null;
|
||||
|
||||
beforeAll(async () => {
|
||||
await setupDB();
|
||||
|
||||
// Save+clear the real ~/.gbrain/sync-failures.jsonl so the test starts from
|
||||
// a known-empty state. Restored in afterAll. This file is per-machine, NOT
|
||||
// per-repo, so we have to be defensive about a developer running this
|
||||
// suite on their actual brain machine.
|
||||
if (existsSync(realFailuresPath)) {
|
||||
savedFailuresContent = readFileSync(realFailuresPath, 'utf-8');
|
||||
unlinkSync(realFailuresPath);
|
||||
}
|
||||
|
||||
// Fresh git repo with one valid file. Mirrors createTestRepo above but
|
||||
// scoped to this describe block.
|
||||
repoPath = mkdtempSync(join(tmpdir(), 'gbrain-skipfailed-e2e-'));
|
||||
execSync('git init', { cwd: repoPath, stdio: 'pipe' });
|
||||
execSync('git config user.email "test@test.com"', { cwd: repoPath, stdio: 'pipe' });
|
||||
execSync('git config user.name "Test"', { cwd: repoPath, stdio: 'pipe' });
|
||||
mkdirSync(join(repoPath, 'people'), { recursive: true });
|
||||
writeFileSync(join(repoPath, 'people/alice.md'), [
|
||||
'---', 'type: person', 'title: Alice', '---', '', 'Body.',
|
||||
].join('\n'));
|
||||
execSync('git add -A && git commit -m "initial"', { cwd: repoPath, stdio: 'pipe' });
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await teardownDB();
|
||||
if (repoPath) rmSync(repoPath, { recursive: true, force: true });
|
||||
|
||||
// Restore the user's real sync-failures.jsonl, if any.
|
||||
if (savedFailuresContent !== null) {
|
||||
mkdirSync(join(homedir(), '.gbrain'), { recursive: true });
|
||||
writeFileSync(realFailuresPath, savedFailuresContent);
|
||||
} else if (existsSync(realFailuresPath)) {
|
||||
// Test wrote one but there was none before. Clean up.
|
||||
unlinkSync(realFailuresPath);
|
||||
}
|
||||
});
|
||||
|
||||
test('full --skip-failed loop: blocks on bad file, skip advances bookmark, doctor shows code breakdown', async () => {
|
||||
const { performSync } = await import('../../src/commands/sync.ts');
|
||||
const { loadSyncFailures, summarizeFailuresByCode } = await import('../../src/core/sync.ts');
|
||||
const engine = getEngine();
|
||||
|
||||
// Step 1: First sync of the clean repo — should succeed.
|
||||
let result = await performSync(engine, { repoPath, noPull: true, noEmbed: true });
|
||||
expect(result.status).toBe('first_sync');
|
||||
const firstCommit = await engine.getConfig('sync.last_commit');
|
||||
expect(firstCommit).toBeTruthy();
|
||||
|
||||
// Step 2: Add a broken file — frontmatter slug doesn't match path-derived slug.
|
||||
// The file path is people/bob.md so the path-derived slug is "people/bob",
|
||||
// but we declare slug: "wrong-slug" in frontmatter. import-file.ts:368-377
|
||||
// raises "Frontmatter slug ... does not match path-derived slug ..." which
|
||||
// classifier hits as SLUG_MISMATCH.
|
||||
writeFileSync(join(repoPath, 'people/bob.md'), [
|
||||
'---', 'type: person', 'title: Bob', 'slug: wrong-slug', '---', '', 'Body.',
|
||||
].join('\n'));
|
||||
execSync('git add -A && git commit -m "add broken bob"', { cwd: repoPath, stdio: 'pipe' });
|
||||
|
||||
// Step 3: Sync should block. Bookmark must NOT advance.
|
||||
result = await performSync(engine, { repoPath, noPull: true, noEmbed: true });
|
||||
expect(result.status).toBe('blocked_by_failures');
|
||||
const afterBlockedCommit = await engine.getConfig('sync.last_commit');
|
||||
expect(afterBlockedCommit).toBe(firstCommit); // bookmark stuck at the pre-broken commit
|
||||
|
||||
// JSONL has one unacked entry with code SLUG_MISMATCH.
|
||||
let failures = loadSyncFailures();
|
||||
expect(failures.length).toBe(1);
|
||||
expect(failures[0].code).toBe('SLUG_MISMATCH');
|
||||
expect(failures[0].acknowledged).toBeFalsy();
|
||||
// Group summary aggregates correctly across the unacked set.
|
||||
expect(summarizeFailuresByCode(failures)).toEqual([{ code: 'SLUG_MISMATCH', count: 1 }]);
|
||||
|
||||
// Step 4: Run with skipFailed — bookmark advances, entry gets acked.
|
||||
result = await performSync(engine, { repoPath, noPull: true, noEmbed: true, skipFailed: true });
|
||||
expect(result.status).toBe('synced');
|
||||
const afterSkipCommit = await engine.getConfig('sync.last_commit');
|
||||
expect(afterSkipCommit).not.toBe(firstCommit); // bookmark moved past the broken commit
|
||||
failures = loadSyncFailures();
|
||||
expect(failures.length).toBe(1);
|
||||
expect(failures[0].acknowledged).toBe(true);
|
||||
expect(typeof failures[0].acknowledged_at).toBe('string');
|
||||
|
||||
// Step 5: Verify what doctor would render for the historical entry.
|
||||
// We call the same primitives doctor's `sync_failures` check uses
|
||||
// (src/commands/doctor.ts:252-275) — loadSyncFailures + summarizeFailuresByCode —
|
||||
// and assert the rendering string. Directly invoking runDoctor() here is a CLI
|
||||
// entrypoint with stdout/exit side effects that would truncate this test mid-flow.
|
||||
{
|
||||
const all = loadSyncFailures();
|
||||
const ackedSummary = summarizeFailuresByCode(all);
|
||||
const ackedBreakdown = ackedSummary.map(s => `${s.code}=${s.count}`).join(', ');
|
||||
// This is the literal string interpolation doctor.ts:271-274 produces.
|
||||
const doctorMessage = `${all.length} historical sync failure(s), all acknowledged [${ackedBreakdown}].`;
|
||||
expect(doctorMessage).toContain('SLUG_MISMATCH=1');
|
||||
expect(doctorMessage).toContain('1 historical');
|
||||
}
|
||||
|
||||
// Step 6: Add a second broken file — this one with a different failure code
|
||||
// (also SLUG_MISMATCH but on a different file) so the JSONL has 2 entries
|
||||
// with DIFFERENT paths but the same code. This proves both: per-file dedup
|
||||
// honors path identity, and summary aggregation sums across files.
|
||||
//
|
||||
// We'd ideally test a different code class here, but the sync path uses
|
||||
// parseMarkdown WITHOUT {validate:true}, so the markdown.ts validation
|
||||
// codes (MISSING_OPEN/CLOSE, NESTED_QUOTES, EMPTY_FRONTMATTER, NULL_BYTES)
|
||||
// don't naturally surface — they'd need {validate:true} plumbed in. That
|
||||
// plumbing is the v0.22.13+ follow-up. For v0.22.12, two SLUG_MISMATCH
|
||||
// entries from different files still proves the dedup + aggregation chain.
|
||||
writeFileSync(join(repoPath, 'people/carol.md'), [
|
||||
'---', 'type: person', 'title: Carol', 'slug: also-wrong-slug', '---', '', 'Body.',
|
||||
].join('\n'));
|
||||
execSync('git add -A && git commit -m "add carol with bad slug"', { cwd: repoPath, stdio: 'pipe' });
|
||||
|
||||
// Step 7: Sync blocks again on the new failure. Old entry stays acked.
|
||||
result = await performSync(engine, { repoPath, noPull: true, noEmbed: true });
|
||||
expect(result.status).toBe('blocked_by_failures');
|
||||
failures = loadSyncFailures();
|
||||
expect(failures.length).toBe(2);
|
||||
const acked = failures.filter(f => f.acknowledged);
|
||||
const unacked = failures.filter(f => !f.acknowledged);
|
||||
expect(acked.length).toBe(1);
|
||||
expect(acked[0].code).toBe('SLUG_MISMATCH');
|
||||
expect(acked[0].path).toContain('bob');
|
||||
expect(unacked.length).toBe(1);
|
||||
expect(unacked[0].code).toBe('SLUG_MISMATCH');
|
||||
expect(unacked[0].path).toContain('carol');
|
||||
|
||||
// Step 8: Skip again — both entries acked, summary aggregates the count.
|
||||
result = await performSync(engine, { repoPath, noPull: true, noEmbed: true, skipFailed: true });
|
||||
expect(result.status).toBe('synced');
|
||||
failures = loadSyncFailures();
|
||||
expect(failures.length).toBe(2);
|
||||
expect(failures.every(f => f.acknowledged)).toBe(true);
|
||||
|
||||
const finalSummary = summarizeFailuresByCode(failures);
|
||||
expect(finalSummary).toEqual([{ code: 'SLUG_MISMATCH', count: 2 }]);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -1,178 +0,0 @@
|
||||
/**
|
||||
* test/e2e/worker-abort-recovery.test.ts — E2E smoke test for worker
|
||||
* recovery after handler timeout.
|
||||
*
|
||||
* Exercises the full path: submit job → handler runs → timeout fires →
|
||||
* abort propagates → worker recovers → claims next job.
|
||||
*
|
||||
* This is the end-to-end regression test for the 2026-04-24 incident
|
||||
* where a stuck autopilot-cycle handler wedged the worker with 98 jobs
|
||||
* waiting and 0 active.
|
||||
*
|
||||
* Uses PGLite (in-memory), no external services needed.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
|
||||
import { MinionQueue } from '../../src/core/minions/queue.ts';
|
||||
import { MinionWorker } from '../../src/core/minions/worker.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let queue: MinionQueue;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({ database_url: '' });
|
||||
await engine.initSchema();
|
||||
queue = new MinionQueue(engine);
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await engine.executeRaw('DELETE FROM minion_jobs');
|
||||
});
|
||||
|
||||
describe('E2E: worker abort recovery (2026-04-24 regression)', () => {
|
||||
test('worker recovers from timed-out handler and processes next job', async () => {
|
||||
// Step 1: Submit a slow job with a short timeout
|
||||
const slowJob = await queue.add('slow-handler', { type: 'slow' }, {
|
||||
timeout_ms: 200,
|
||||
max_attempts: 1,
|
||||
});
|
||||
|
||||
// Step 2: Submit a fast job that should run AFTER the slow one times out
|
||||
const fastJob = await queue.add('fast-handler', { type: 'fast' }, {
|
||||
max_attempts: 1,
|
||||
});
|
||||
|
||||
let slowHandlerAborted = false;
|
||||
let fastHandlerExecuted = false;
|
||||
|
||||
const worker = new MinionWorker(engine, {
|
||||
pollInterval: 50,
|
||||
concurrency: 1, // Single slot — forces sequential execution
|
||||
});
|
||||
|
||||
// Slow handler: respects AbortSignal (the fix path)
|
||||
worker.register('slow-handler', async (ctx) => {
|
||||
// Simulate expensive work (like extract scanning 54K pages)
|
||||
while (!ctx.signal.aborted) {
|
||||
await new Promise(r => setTimeout(r, 20));
|
||||
}
|
||||
slowHandlerAborted = true;
|
||||
throw ctx.signal.reason || new Error('aborted');
|
||||
});
|
||||
|
||||
// Fast handler: just completes
|
||||
worker.register('fast-handler', async () => {
|
||||
fastHandlerExecuted = true;
|
||||
return { done: true };
|
||||
});
|
||||
|
||||
// Step 3: Start worker
|
||||
const workerPromise = worker.start();
|
||||
|
||||
// Step 4: Wait for slow job timeout (200ms) + handler abort + fast job execution
|
||||
await new Promise(r => setTimeout(r, 1000));
|
||||
|
||||
// Step 5: Stop worker
|
||||
worker.stop();
|
||||
await workerPromise;
|
||||
|
||||
// Step 6: Verify
|
||||
expect(slowHandlerAborted).toBe(true);
|
||||
expect(fastHandlerExecuted).toBe(true);
|
||||
|
||||
const slowResult = await queue.getJob(slowJob.id);
|
||||
expect(slowResult!.status).toBe('dead');
|
||||
|
||||
const fastResult = await queue.getJob(fastJob.id);
|
||||
expect(fastResult!.status).toBe('completed');
|
||||
expect(fastResult!.result).toEqual({ done: true });
|
||||
});
|
||||
|
||||
test('concurrency=2 worker still processes jobs while one slot is timing out', async () => {
|
||||
const slowJob = await queue.add('slow-c2', {}, {
|
||||
timeout_ms: 200,
|
||||
max_attempts: 1,
|
||||
});
|
||||
const fastJob = await queue.add('fast-c2', {}, { max_attempts: 1 });
|
||||
|
||||
let slowAborted = false;
|
||||
let fastDone = false;
|
||||
|
||||
const worker = new MinionWorker(engine, {
|
||||
pollInterval: 50,
|
||||
concurrency: 2, // Two slots — fast job can run in parallel
|
||||
});
|
||||
|
||||
worker.register('slow-c2', async (ctx) => {
|
||||
while (!ctx.signal.aborted) {
|
||||
await new Promise(r => setTimeout(r, 10));
|
||||
}
|
||||
slowAborted = true;
|
||||
throw new Error('aborted');
|
||||
});
|
||||
|
||||
worker.register('fast-c2', async () => {
|
||||
fastDone = true;
|
||||
return { fast: true };
|
||||
});
|
||||
|
||||
const workerPromise = worker.start();
|
||||
await new Promise(r => setTimeout(r, 600));
|
||||
worker.stop();
|
||||
await workerPromise;
|
||||
|
||||
expect(slowAborted).toBe(true);
|
||||
expect(fastDone).toBe(true);
|
||||
|
||||
const slowResult = await queue.getJob(slowJob.id);
|
||||
expect(slowResult!.status).toBe('dead');
|
||||
|
||||
const fastResult = await queue.getJob(fastJob.id);
|
||||
expect(fastResult!.status).toBe('completed');
|
||||
});
|
||||
|
||||
test('multiple timeouts in sequence dont permanently wedge worker', async () => {
|
||||
// Submit 3 slow jobs that all timeout + 1 fast job
|
||||
// The fast job MUST execute
|
||||
const slow1 = await queue.add('multi-slow', {}, { timeout_ms: 100, max_attempts: 1 });
|
||||
const slow2 = await queue.add('multi-slow', {}, { timeout_ms: 100, max_attempts: 1 });
|
||||
const slow3 = await queue.add('multi-slow', {}, { timeout_ms: 100, max_attempts: 1 });
|
||||
const fast = await queue.add('multi-fast', {}, { max_attempts: 1 });
|
||||
|
||||
let timeoutsHit = 0;
|
||||
let fastDone = false;
|
||||
|
||||
const worker = new MinionWorker(engine, { pollInterval: 50, concurrency: 1 });
|
||||
|
||||
worker.register('multi-slow', async (ctx) => {
|
||||
while (!ctx.signal.aborted) {
|
||||
await new Promise(r => setTimeout(r, 10));
|
||||
}
|
||||
timeoutsHit++;
|
||||
throw new Error('aborted');
|
||||
});
|
||||
|
||||
worker.register('multi-fast', async () => {
|
||||
fastDone = true;
|
||||
return { ok: true };
|
||||
});
|
||||
|
||||
const workerPromise = worker.start();
|
||||
// 3 slow jobs × (100ms timeout + overhead) + fast job + margin
|
||||
await new Promise(r => setTimeout(r, 2000));
|
||||
worker.stop();
|
||||
await workerPromise;
|
||||
|
||||
expect(timeoutsHit).toBe(3);
|
||||
expect(fastDone).toBe(true);
|
||||
|
||||
const fastResult = await queue.getJob(fast.id);
|
||||
expect(fastResult!.status).toBe('completed');
|
||||
});
|
||||
});
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user