Compare commits

...
Author SHA1 Message Date
Garry Tan e8b6894c72 Merge remote-tracking branch 'origin/master' into garrytan/claw-setup-e2e
# Conflicts:
#	CHANGELOG.md
#	CLAUDE.md
#	TODOS.md
#	VERSION
#	llms-full.txt
#	package.json
#	src/cli.ts
2026-04-29 23:41:09 -07:00
17c3c43783 v0.22.15 feat: frontmatter inference — zero-friction ingest for files without YAML headers (#506)
* feat: frontmatter inference — zero-friction ingest for files without YAML headers

The platonic ideal of agentic retrieval is: you throw stuff in and it
becomes knowledge. No manual schema, no frontmatter templates, no YAML
ceremony. This PR makes that real.

## The Problem

9,655 files in a real 81K-page brain have no YAML frontmatter. They import
fine (gray-matter is forgiving), but with minimal metadata:
- type defaults to 'concept' for everything
- title is the slugified filename ('2010 04 13 Apr 13 Founders Mtg')
- No date, no source, no tags, no folder-aware typing

These pages exist in the DB but are poorly classified, which degrades
search ranking, type-filtered queries, and entity resolution.

## The Fix

### 1. Directory-aware inference engine (src/core/frontmatter-inference.ts)

A rules table maps path patterns to rich metadata:

  Apple Notes/*          → type: apple-note, date from filename, source: apple-notes
  Apple Notes/YC/*       → adds tag: yc
  Apple Notes/Politics/* → adds tag: politics
  daily/calendar/*       → type: calendar-index, source: calendar
  people/*               → type: person, title from # heading
  personal/therapy/*     → type: therapy-session, date from filename
  personal/reflections/* → type: reflection, title from # heading
  writing/essays/*       → type: essay, date from filename
  companies/*            → type: company, title from # heading
  events/*               → type: event, date from filename
  (catch-all)            → type: note, title from # heading

Each rule specifies:
- type: page type for brain schema
- datePattern: 'filename' (YYYY-MM-DD prefix), 'dirname', or 'none'
- titleStrategy: 'filename' (strip date), 'heading' (first #), 'filename-full'
- source: optional source tag
- tags: optional additional tags

Title extraction cleans up filenames (strips date prefix, converts dashes
to spaces, preserves existing capitalization). Heading extraction looks at
the first 20 lines for a # heading.

Fully deterministic. No LLM calls. No network. Same file → same frontmatter.

### 2. Inline inference in import pipeline (src/core/import-file.ts)

importFromFile() now runs inference automatically when a file has no
frontmatter. The synthesized frontmatter is applied to the in-memory
content before parseMarkdown runs, so the downstream pipeline sees
well-formed YAML. The file on disk is NOT modified — inference is
DB-only unless you explicitly run `gbrain frontmatter generate --fix`.

### 3. CLI command: gbrain frontmatter generate (src/commands/frontmatter.ts)

  gbrain frontmatter generate /path/to/brain         # dry-run preview
  gbrain frontmatter generate /path/to/brain --fix   # write to files
  gbrain frontmatter generate /path/to/brain --json  # machine output

The dry-run output shows:
- Total scanned / already have frontmatter / would generate
- Breakdown by inferred type
- First 10 examples with inferred metadata and matched rule

### 4. Tests (test/frontmatter-inference.test.ts)

35 tests covering:
- Date extraction from various filename patterns
- Title extraction from filenames and headings
- Inference for every directory rule (Apple Notes, people, therapy, etc.)
- Serialization with YAML-safe quoting
- Integration: applyInference prepends frontmatter correctly
- Rules: ordering, catch-all, specificity

## What this enables

1. `gbrain sync` now imports bare markdown with rich metadata automatically
2. `gbrain frontmatter generate --fix` writes frontmatter to 9,655 files
3. Future: sync can optionally write-back inferred frontmatter to git
4. Future: rules table is extensible — new directory conventions = one rule

## Adding new directory conventions

Edit DIRECTORY_RULES in src/core/frontmatter-inference.ts:

  { pathPrefix: 'recipes/', type: 'recipe', titleStrategy: 'heading' }

Rules are matched first-to-last, most specific prefix wins. The catch-all
(empty prefix) is always last.

## Real-world test output

  Scanned: 81,479 files
  Already have frontmatter: 71,824
  Would generate: 9,655

  By type:
    apple-note: 5,861
    calendar-index: 3,201
    person: 56
    therapy-session: 60
    reflection: 12
    essay: 33
    ...

All 35 tests pass.

* fix: import basename in frontmatter generate dynamic path import

src/commands/frontmatter.ts:437 calls basename(rootPath) as a fallback
when relative(brainRoot, rootPath) returns the empty string, but the
dynamic import a few lines above only destructures { resolve, relative,
join } from 'path'. typecheck failed with TS2304: Cannot find name
'basename', and any user invocation of `gbrain frontmatter generate
<single-file>` would have crashed at runtime with a ReferenceError.

The unit tests cover frontmatter-inference module's pure functions; no
test exercises the CLI single-file branch, so the bug slipped through.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump VERSION 0.22.8 → 0.22.15 + CHANGELOG entry

Slot v0.22.15 per the queue allocator (other PRs claim v0.22.9–v0.22.14).
CHANGELOG entry written above v0.22.8 per the never-touch-shipped-entries
rule. bun.lock and llms-full.txt are unchanged (CLAUDE.md untouched, the
inference module and CLI command come in via the feature commit).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Wintermute <wintermute@garrytan.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 23:26:25 -07:00
Wintermute 0fb0c83d24 chore: gitignore export/ directory (generated output) 2026-04-30 06:11:34 +00:00
ed900c870e v0.22.14 feat(minions): bare-worker self-health-monitoring (#503)
* feat(minions): add self-health-monitoring to bare worker mode

Bare `gbrain jobs work` (without supervisor) previously had zero health
monitoring. If the Postgres connection dropped or the worker's event loop
deadlocked, the process stayed alive doing nothing — jobs piled up while
external process managers (systemd, Docker, cron) thought it was healthy.

Changes:

1. **Self-health-check timer** (worker.ts): Runs every 60s when not under
   a supervisor. Two probes:
   - DB liveness: `SELECT 1` — 3 consecutive failures → exit(1)
   - Stall detection: waiting jobs + 0 in-flight + no completions for 5m
     → warning; 10m → exit(1)

2. **GBRAIN_SUPERVISED env var** (supervisor.ts): Supervisor sets this on
   its child worker to prevent duplicate health checks. The supervisor
   already has its own health monitoring.

3. **RSS watchdog default** (jobs.ts): Bare workers now default to
   `--max-rss 2048` (matching supervisor default). Opt out: `--max-rss 0`.

4. **--health-interval flag** (jobs.ts): Configurable health check period.
   `--health-interval 0` disables. Default: 60000ms.

5. **parseMaxRssFlag returns undefined** when flag is absent (vs 0), so
   callers can distinguish 'not set' from 'explicitly disabled'.

The design ensures bare workers get supervisor-grade monitoring while
remaining compatible with any external process manager — the worker just
exits with code 1 on detected failure, letting the PM handle the restart.

Tests: 4 new tests (3 worker health, 1 supervisor env var). All 178 pass.

* fix(minions): harden bare-worker self-health-check after multi-round review

Layered fixes from 5 rounds of plan-eng-review + codex outside voice on top of
the original PR #503 (feat: bare-worker self-health-monitoring). Every change
below is in service of "fail-stop into the operator's process manager" without
introducing new ways the library can kill its caller.

worker.ts:
- MinionWorker now extends EventEmitter; emits `'unhealthy'` event with structured
  reason payload (`db_dead` | `stalled`). CLI subscribes; library no longer calls
  process.exit directly.
- emitUnhealthy() falls back to process.exit(1) when listenerCount('unhealthy') === 0
  so direct API consumers without a listener inherit the pre-refactor fail-stop
  default. Inline paths opt out via healthCheckInterval=0.
- Stall detection: count(*) query now filters by registered handler names
  (`AND name = ANY($2::text[])`) so workers with handlers for {embed,sync} don't
  false-positive when waiting jobs of unhandled names accumulate.
- Stall exit threshold measured from lastCompletionTime (not from warn-since), so
  defaults of 5min warn / 10min exit fire at idle=10min total — matching the
  documented contract.
- Recursive setTimeout pattern with running flag replaces setInterval, eliminating
  callback overlap on slow DB probes.
- DB liveness probe wrapped in Promise.race against AbortController-driven
  timeout (default 10s) so a hung executeRaw can't wedge the recursive chain
  forever. Hung probes count as failures and feed dbFailExitAfter.
- Constructor validates stallExitAfterMs > stallWarnAfterMs and throws loudly
  on misconfiguration. Internal timer-installation invariants documented inline.
- GBRAIN_SUPERVISED env-var check tightened from `!!process.env.X` to `=== '1'`.

types.ts:
- Added 5 new MinionWorkerOpts fields with documented contracts:
  healthCheckInterval, stallWarnAfterMs, stallExitAfterMs, dbFailExitAfter,
  dbProbeTimeoutMs.
- Exported `UnhealthyReason` discriminated union for the 'unhealthy' event payload.

supervisor.ts:
- GBRAIN_SUPERVISED=1 injected on the spawned worker child's env so the child's
  self-health timer is skipped (no double-monitoring).
- setInterval(callback, healthInterval) gated behind `> 0`, so the
  `--health-interval 0` documented disable contract actually disables instead
  of producing a tight DB-hammer loop.

jobs.ts:
- `gbrain jobs work` subscribes to 'unhealthy' and calls process.exit(1) at the
  CLI layer. Default --max-rss bumped from 0 to 2048 (matches supervisor default;
  catches memory-leak stalls that previously went undetected).
- New --health-interval flag with aggressive validation (NaN/negative/sub-1000ms
  rejected; parity with --max-rss) on both `jobs work` and `jobs supervisor`.
- `jobs submit --follow` and `jobs smoke` now pass healthCheckInterval=0 to
  disable the self-health timer entirely. These are inline/one-shot flows with
  no PM to restart them; the no-listener emitUnhealthy fallback could otherwise
  trip on a DB blip and kill the user's CLI session.
- parseMaxRssFlag returns `number | undefined` (was `number`) so callers can
  distinguish absent (use the default) from explicit-disable (--max-rss 0).

doctor.ts:
- New queue_health subcheck reports RSS-watchdog kills in the last 24h.
  Detects via exact-match `error_text = 'aborted: watchdog'` (the worker's
  failJob signature when gracefulShutdown('watchdog') aborts in-flight jobs)
  scoped to status IN ('dead','failed'). Tight match avoids over-counting parent
  jobs that propagate child failures via on_child_fail='fail_parent'.

* test(minions): self-health behavior + regression tests

7 new tests covering the production failure modes that drove the original PR,
plus regressions for fixes landed during multi-round review.

minions.test.ts:
- DB 3-strike → 'unhealthy' event with reason='db_dead' (the production-incident signature)
- DB recovery resets failure counter (no exit on intermittent failures)
- Stall warn-then-exit (clock-driven; idleMs > stallExitAfterMs is the new contract)
- inFlight > 0 blocks stall detection (long-running legitimate jobs don't false-trip)
- Regression for D1 fix: jobs of unregistered handler names don't trigger stall exit;
  also captures the SQL via probe engine and asserts the predicate text contains
  `name = ANY` so a future refactor that drops the filter is caught at test time.
- Regression for R3 constructor validation: throws when stallExitAfterMs <= stallWarnAfterMs
  (covers both `<` and `=` cases); defaults still construct cleanly.

supervisor.test.ts:
- Regression for R3: supervisor with healthInterval=0 completes a normal lifecycle
  within 10s. A tight setInterval(0) loop (the bug we fixed) would saturate the
  event loop and slow this past the cap.

Tests use a Proxy-based engine helper (makeProbeEngine) that intercepts SELECT 1
and the count(*) WHERE status='waiting' query while passing through everything
else to the real PGLite engine. This isolates health-check semantics from claim
plumbing without mocking the entire engine surface.

* docs(v0.22.14): migration walkthrough + follow-up TODOs

skills/migrations/v0.22.14.md (new):
- Pre-flight per-PM restart-policy table (systemd Restart=always, Docker
  restart: always, launchd KeepAlive, cron watchdog, supervisord autorestart).
  v0.22.14 makes bare-worker behavior fail-stop — without an external restart
  loop the worker exits and stays dead. Migration calls this out loudly so
  OpenClaw/Hermes-style downstream agents can verify their PM before upgrade.
- Five new MinionWorkerOpts fields documented with defaults and rationale.
- Worker-side process.exit(1) fallback semantics explained: CLI subscribes to
  'unhealthy', but direct API consumers without a listener inherit fail-stop.
- AskUserQuestion-driven flow for the --max-rss 2048 default (raise / opt out /
  keep) with concrete edits per PM (systemd unit, cron line, Docker compose,
  launchctl plist).
- Verification commands (gbrain jobs stats, gbrain doctor --json, RSS check)
  and a triage paragraph for opening an issue if anything fails.

TODOS.md:
- v0.22.15 embed cooperative-abort (P0, daily pain): plumb signal through
  runPhaseEmbed → embed.ts → embedBatch; check signal.aborted between OpenAI
  batch calls and between slugs. Closes the daily wedge where embed > 600s
  timeout dead-letters the job but keeps running, holding gbrain_cycle_locks
  until the lock TTL expires. PR #503 catches the symptom (worker stalled);
  this captures the cause-side fix that's the real production resolution.
- v0.23+ bare-worker engine reconnect parity: extract supervisor's
  reconnect-then-fail pattern (#406) into MinionWorker so transient PgBouncer
  blips don't force a full process restart.
- v0.23+ minion_workers heartbeat table for queue_health doctor check (B7
  follow-up): replace lock_until proxy with ground-truth worker liveness
  signal so doctor stops crying wolf on legitimately idle workers.

* chore: bump version and changelog (v0.22.14)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: Wintermute <wintermute@garrytan.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-29 23:06:16 -07:00
e96f054cf0 v0.22.13 feat: parallel sync — bounded concurrent imports (#490)
* feat: parallel sync — bounded concurrent imports (#489)

gbrain sync --concurrency N (alias --workers N) parallelizes the import
phase using per-worker Postgres engine instances with an atomic queue
index (same proven pattern as gbrain import --workers N).

Auto-concurrency: when a sync touches >100 files and the user didn't
explicitly set --concurrency, defaults to 4 workers. Small incremental
syncs (<50 files) stay serial. Full syncs auto-detect Postgres and
default to 4 workers.

Minion sync handler defaults to concurrency=4, configurable via job
params: {"concurrency": 8}.

Delete and rename phases remain serial (order-dependent, fast).
PGLite falls back to serial automatically (single-connection engine).

Changes:
- src/commands/sync.ts: SyncOpts.concurrency, parallel import loop in
  performSync incremental path, --workers passthrough in performFullSync
- src/commands/jobs.ts: sync handler accepts concurrency param (default 4)
- CHANGELOG.md: v0.23.0 parallel sync entry

All 37 existing sync tests pass. Typecheck clean.

* feat: shared concurrency policy + db-lock primitive

src/core/sync-concurrency.ts — single source of truth for autoConcurrency()
+ parseWorkers() + shouldRunParallel() + constants. Replaces three drifted
call-site policies (performSync, performFullSync, jobs handler).

src/core/db-lock.ts — generic tryAcquireDbLock(engine, lockId, ttlMinutes)
over the existing gbrain_cycle_locks table. Parameterized lock id so
performSync (gbrain-sync) can nest cleanly under cycle.ts (gbrain-cycle)
without deadlock.

test/sync-concurrency.test.ts — 17 cases covering PGLite-forces-serial,
explicit override clamping, auto-path threshold, parseWorkers validation
(rejects 0, negatives, NaN, decimals, trailing chars).

No consumers yet; subsequent commits wire sync.ts, import.ts, and jobs.ts
to use these helpers.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: harden performSync — writer lock, head-drift gate, engine.kind

CODEX-2: wrap performSync body in a gbrain-sync DB lock so two concurrent
syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot
both read last_commit, both write it unconditionally, and let the last
writer win. cycle.ts continues to hold gbrain-cycle for its broader scope;
the two ids nest cleanly.

CODEX-3: capture git HEAD at sync entry, re-rev-parse after the import
phase, refuse to advance last_commit if HEAD drifted (someone ran
git checkout / git pull mid-sync). Vanished files now go into failedFiles
instead of silent-skip — same gating mechanism, no more bookmark advance
past unimported work.

A1: replace both PGLite detection sites with engine.kind === 'pglite'.
The constructor.name sniff is gone (breaks under bundling) and so is the
inconsistent config?.engine string check.

A2: connect worker engines serially into an array, run inside try/finally
so disconnect always fires — even on partial connect failure, OOM, or
mid-import abort. Prior Promise.all(...disconnect) leaked the 8 worker
connections on any panic path.

Q1: explicit --workers / opts.concurrency now bypasses the >50-file floor.
User opt-in beats the auto-path safety net.

Q3: drop the config!.database_url! non-null assertions; fall back to serial
when database_url is unset instead of crashing on TypeError.

Q4: worker-count banner moves from console.log to console.error so stdout
stays clean for --json output.

test/sync-parallel.test.ts — 7 cases over PGLite covering the bookmark
gate under concurrency request, the head-drift gate, vanished-file
failure capture, PGLite-stays-serial, and the writer-lock contract.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: import.ts — engine.kind discriminator, worker try/finally, parseWorkers

A1: replace the config?.engine === 'pglite' string sniff with
engine.kind === 'pglite' to match sync.ts and the v0.13.1 contract.

A2: wrap worker engine creation + the parallel loop in try/finally so
disconnects always fire — same pattern as sync.ts. Worker engines now
push onto an array as they connect (rather than Promise.all) so the
finally block can clean up partial-connect state.

Q2: route --workers parsing through the shared parseWorkers() helper.
parseInt-with-no-validation is gone — '0', '-3', 'foo', '1.5' now exit
with a clear error message instead of silently falling through.

Q3: drop the config!.database_url! non-null assertion; fall back to
serial when database_url is unset.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: jobs.ts sync handler — resolve sourceId, autoConcurrency

CODEX-1: resolve sourceId at handler entry by looking up sources.local_path.
Mirrors cycle.ts:480's autopilot-cycle fix (PR #475). Without this, every
Minion sync job on a multi-source brain reads global config.sync.last_commit
instead of the per-source anchor, which on a regularly-GC'd repo can drop
out of git history and trigger 30-min full reimports every cycle.

The handler accepts an optional sourceId job param for callers that want
to override; falls back to the resolveSourceForDir lookup when absent.

CODEX-4: replace the hardcoded concurrency=4 default with the shared
autoConcurrency policy. Behavior is now consistent between CLI sync,
the Minion handler, and the autopilot cycle's sync phase. Jobs that
request a specific concurrency via job.data.concurrency still win.

noEmbed default stays at true — embed is a separate job (submit
gbrain embed --stale, OR rely on the autopilot cycle's embed phase).
The doc comment makes that contract explicit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: e2e parallel sync against real Postgres + benchmark

DATABASE_URL-gated E2E coverage that PGLite-only tests can't reach:

T2 — happy path: 60 files imported at concurrency=4, all 60 pages land
in the DB, with a pg_stat_activity probe before/after to confirm worker
engines (4 × 2 connections) actually disconnected.

P4 — benchmark: 120-file fixture, serial vs concurrency=4 timing.
Emits a single-line `SYNC_PARALLEL_BENCH 120 files | serial=Xms |
parallel(4)=Yms | speedup=Zx` so the CHANGELOG can quote a real
number instead of an unbacked '~4×' claim. Asserts parallel <=
serial * 1.5 to allow for noisy CI but fail genuine regressions.

Skips gracefully when DATABASE_URL is unset (consistent with the rest
of test/e2e/).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: v0.22.10 release notes + sync follow-up TODO

VERSION + package.json + bun.lock: 0.22.5/0.22.6 → 0.22.10. Repo had
existing drift between VERSION and package.json on master; this commit
brings them back in sync at the bumped value.

CHANGELOG.md: v0.22.10 entry replaces the unfinished v0.23.0 stub from
PR #490's original commit. Voice-rule clean (no em dashes, no AI
vocabulary), real benchmark numbers from the new E2E test
(serial=289ms parallel(4)=221ms speedup=1.31x), additive worker-pool
note (A3), 'To take advantage of v0.22.10' self-repair block per
CLAUDE.md convention.

TODOS.md: A4 follow-up filed — plumb resolved database_url through
SyncOpts so performSync / performFullSync / import.ts don't each call
loadConfig() separately. Deferred to a future patch; not on the
v0.22.10 critical path.

Patch (not minor) framing held even though new CLI surface lands here;
release-notes prose names the behavior change explicitly so users know
to read them.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: update CLAUDE.md + README for v0.22.10 sync hardening

CLAUDE.md:
- New "Key files" entries for src/core/sync-concurrency.ts and
  src/core/db-lock.ts (both v0.22.10).
- New "Key files" entry for src/commands/sync.ts (covers the lock,
  head-drift gate, engine.kind discriminator, vanished-file failure
  capture, parallel branch wiring).
- Updated src/commands/jobs.ts entry with v0.22.10 sourceId
  resolution + autoConcurrency policy + noEmbed contract.
- Added test/sync-concurrency.test.ts and test/sync-parallel.test.ts
  to the unit-test list with case counts.
- Added test/e2e/sync-parallel.test.ts to the E2E section with the
  SYNC_PARALLEL_BENCH grep marker for CHANGELOG quoting.
- Added "Key commands added in v0.22.10" section: gbrain sync --workers,
  gbrain import --workers (parseWorkers validation).

README.md: added --workers flag to the IMPORT section's gbrain sync
and gbrain import lines, with the >100-file auto-parallelize note.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump version slot to v0.22.13

VERSION 0.22.10 → 0.22.13. Master moved to 0.22.8 plus claimed slots
0.22.9-0.22.12 in sibling workspaces; 0.22.13 is the next free slot for
this PR's parallel-sync hardening work.

Updated all v0.22.10 references in CHANGELOG.md (release header +
self-repair block), TODOS.md (D-PR490-1 follow-up tag), CLAUDE.md
(Key files entries + tests + commands subsection), and the inline
v0.22.10 markers in src/core/sync-concurrency.ts, src/core/db-lock.ts,
src/commands/sync.ts, src/commands/import.ts, src/commands/jobs.ts,
test/sync-parallel.test.ts, test/e2e/sync-parallel.test.ts.

No behavioral change. CHANGELOG header rewrite, content unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: regenerate llms-full.txt for v0.22.13 doc updates

CI's build-llms generator test failed because llms-full.txt was stale
relative to the README + CLAUDE.md updates this PR added (--workers
flag in the IMPORT section, sync-concurrency.ts/db-lock.ts/sync.ts
entries in the Key files section).

Per CLAUDE.md: "Run \`bun run build:llms\` after adding a new doc."
The test test/build-llms.test.ts:67 verifies committed bundles match
generator output — now they do again.

llms.txt was already in sync (no curated config additions); only
llms-full.txt needed the regen.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: root <root@localhost>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 22:53:41 -07:00
Garry TanandWintermute 1e73e93344 v0.22.12 feat: structured error code summary for sync --skip-failed (closes #500) (#518)
* feat: structured error code summary for sync --skip-failed (#500)

When sync encounters per-file failures, the blocked/skip-failed messages
now include a breakdown by error code (SLUG_MISMATCH, YAML_PARSE, etc.)
instead of just a raw count. This makes it immediately obvious *why*
files failed without requiring manual investigation.

Changes:
- Add classifyErrorCode() — maps error messages to ParseValidationCode
- Add summarizeFailuresByCode() — groups failures into sorted code summary
- SyncFailure now carries a 'code' field (backfilled on acknowledge)
- acknowledgeSyncFailures() returns AcknowledgeResult {count, summary}
- sync blocked + skip-failed messages show code breakdown
- doctor sync_failures check shows code breakdown for both unacked and historical
- 12 new tests for classifyErrorCode, summarizeFailuresByCode, and structured returns

Before:
  Sync blocked: 2688 file(s) failed to parse.

After:
  Sync blocked: 2688 file(s) failed to parse:
    SLUG_MISMATCH: 2685
    YAML_DUPLICATE_KEY: 3

Closes #500

* test(sync): broaden classifier regexes and pin coverage with 6 new unit tests

Eng review of PR #501 found two ship-blocking gaps in the classifier:

1. Four real production error sites in src/core/import-file.ts emit strings
   that bucketed to UNKNOWN — exactly the silent-systemic-failure pattern
   that motivated #500 in the first place. Add two regex lines:
     FILE_TOO_LARGE       — covers import-file.ts:199, 352, 401
     SYMLINK_NOT_ALLOWED  — covers import-file.ts:347

2. Three existing classifier regexes (MISSING_OPEN, MISSING_CLOSE,
   EMPTY_FRONTMATTER) only matched the literal code-name prefix. The actual
   message strings emitted by markdown.ts:159-244 (e.g. "Frontmatter must
   start with --- on the first non-empty line") wouldn't match. Broaden
   each to match production message text. NESTED_QUOTES already worked.

Add 6 unit tests pinning the contract between markdown.ts/import-file.ts
strings and the classifier regex set. If anyone reworks a validator
message, both sides have to move together — the test fails loudly otherwise.

Test count: 22 → 28 in test/sync-failures.test.ts, all green.

* test(e2e): add failure-loop E2E for sync --skip-failed (issue #500 ship-blocker)

The full code path (record → classify → block → skip → doctor render →
second cycle) had only mocked-JSONL unit coverage. For a hotfix that
changes user-visible CLI output and the doctor surface, that's thin.

One comprehensive E2E test covers the loop:
  1. First sync of clean repo — succeeds, bookmark advances
  2. Add file with bad slug — sync returns 'blocked_by_failures',
     bookmark stays put, JSONL has 1 unacked entry coded SLUG_MISMATCH
  3. --skip-failed — bookmark advances past the bad commit, entry
     transitions to acknowledged, AcknowledgeResult.summary aggregates
  4. Second broken file (different path, same code) — sync blocks again,
     1 acked + 1 unacked, dedup honors path identity
  5. --skip-failed again — both acked, summary correctly counts 2

Hermetic on a developer machine: saves ~/.gbrain/sync-failures.jsonl
before the test, restores it after. Doctor rendering verified by calling
the same primitives doctor.ts uses (loadSyncFailures + summarizeFailuresByCode)
rather than runDoctor() — runDoctor is a CLI entrypoint with stdout/exit
side effects that truncate the test mid-flow.

E2E count: 13 → 14 in test/e2e/sync.test.ts. All 14 pass under real
Postgres + pgvector (gbrain-test-pg/pgvector:pg16).

* v0.22.12: structured error code summary for sync --skip-failed

Closes issue #500. PR #501 by @wintermute is the foundation (cherry-picked
as c356ea4 — classifier, doctor breakdown, AcknowledgeResult shape, 12 unit
tests). This release adds:

- Classifier coverage for FILE_TOO_LARGE + SYMLINK_NOT_ALLOWED (the four
  size/symlink rejection sites in import-file.ts that bucketed to UNKNOWN).
- Three regex breadths (MISSING_OPEN, MISSING_CLOSE, EMPTY_FRONTMATTER)
  matching actual markdown.ts validator messages, not just the literal
  code-name prefix.
- 6 new unit tests pinning literal production strings.
- 1 comprehensive E2E test exercising the full failure loop.

Total v0.22.12 diff: ~340 lines on top of PR #501. Backward-compatible —
pre-v0.22.12 JSONL entries get classified at acknowledge time.

* chore: regenerate llms-full.txt for v0.22.12 CLAUDE.md changes

CI regen-drift guard caught that llms-full.txt was stale after the v0.22.12
CLAUDE.md annotation updates (sync.ts, doctor.ts, sync-failures.test.ts,
e2e/sync.test.ts entries). Per CLAUDE.md "Auto-derived" rule: run
`bun run build:llms` after any release ship that touches Key Files
annotations. The bundle reflects current docs state.

llms.txt unchanged (curated index doesn't index those entries).
llms-full.txt: 308192 bytes.

test/build-llms.test.ts now passes 7/7 (was 6/7 in CI).

---------

Co-authored-by: Wintermute <wintermute@garrytan.com>
2026-04-29 22:34:04 -07:00
52f9581966 v0.22.11 feat: storage tiering — db_tracked vs db_only directories (#494)
* feat: storage tiering — git-tracked vs supabase-only directories

Brain repos scaling to 200K+ files. Bulk data (tweets, articles, transcripts)
bloats git repos and slows operations. New storage config in gbrain.yml lets
users declare git-tracked and supabase-only directories.

Changes:
- New config: storage.git_tracked and storage.supabase_only in gbrain.yml
- gbrain sync auto-manages .gitignore for supabase-only paths
- gbrain export --restore-only restores missing supabase-only files from DB
- New gbrain storage status command shows tier breakdown
- Config validation warns on conflicts
- 8 tests passing, full docs at docs/storage-tiering.md

Backward compatible — systems without gbrain.yml work unchanged.

* feat: add getDefaultSourcePath() typed accessor (step 1/15)

Single source of truth for "what brain repo are we operating against?"
Replaces ad-hoc raw SQL in storage.ts:38 (Issue #3 of eng review). Used by
both gbrain storage status and gbrain export --restore-only.

Returns null on miss, throws on DB error. Composes with the existing
resolveSourceId chain so it honors --source flag / GBRAIN_SOURCE env /
.gbrain-source dotfile / longest-prefix CWD match / brain-level default.

4 new test cases covering happy path, missing local_path, DB error
propagation, and CWD-prefix resolution priority.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: replace gray-matter with dedicated YAML parser (step 2/15)

The original storage-config.ts called gray-matter on a delimiter-less YAML
file. Gray-matter only parses YAML inside `---` frontmatter blocks; without
delimiters, it returns `{data: {}}`. Result: loadStorageConfig() always
returned null, the entire feature was a silent no-op for every user.

Original eng review's P0 confidence-9 finding (Issue #1).

Replaces gray-matter with a small dedicated parser for the gbrain.yml shape
(top-level `storage:` section, two array-valued nested keys). Yaml-lite was
considered first, but its flat key:value design doesn't handle nested
arrays. The dedicated parser is ~50 lines and trades expressiveness for
zero-dep, predictable parsing of a file format we control.

Adds the Issue #1B sanity warning (locked B): when gbrain.yml exists but
has no storage section (or empty arrays), warn once-per-process so the
user sees their config didn't take. The single test that would have caught
the original P0 — write a real gbrain.yml, call loadStorageConfig, assert
non-null — now exists.

Also tightens loadStorageConfig per D36: distinguishes "absent" (silent
null) from "unreadable" (throws). The previous code silently swallowed
read errors, hiding broken installs.

8 new test cases: real-disk happy path, comments + blank lines, quoted
values, missing storage section warning, empty section warning,
once-per-process warning suppression, unreadable file behavior, and the
existing helper tests (validation, tier matching, edge cases) all still
pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor: rename storage keys to db_tracked/db_only (step 3/15)

The vendor-specific names "supabase_only" and "git_tracked" hardcoded a
backend (Supabase) into the config schema. gbrain ships two engines —
PGLite and Postgres-via-Supabase. The canonical distinction is "lives in
the brain DB only" vs "lives in the brain DB and on disk under git." Both
work on either engine.

Renamed throughout (Issue #4 of eng review):
  git_tracked    → db_tracked
  supabase_only  → db_only
  isGitTracked() → isDbTracked()
  isSupabaseOnly() → isDbOnly()
  StorageTier 'git_tracked'/'supabase_only' → 'db_tracked'/'db_only'

Backward compatibility (D3 lock):
  loadStorageConfig accepts both shapes. Loader resolution order per the
  eng-review pass-2 finding: parse YAML → if canonical keys present use
  them, else if deprecated keys present map to canonical AND emit
  once-per-process deprecation warning → THEN run validation.
  Validation always sees the canonical shape so error messages reference
  db_tracked/db_only regardless of which keys the user wrote.

  The deprecation warning suggests `gbrain doctor --fix` for an automated
  rename (D72 — fix path lands in step 7).

  When both shapes coexist in one file, canonical wins and a stronger
  warning fires ("deprecated keys ignored — remove them").

Aliases isGitTracked/isSupabaseOnly kept for now to avoid churning the
sync.ts / export.ts / storage.ts call sites in this commit; they'll be
removed in a follow-up step. Storage.ts's tier-bucket initializers and
output strings updated. ASCII output replaces unicode box-drawing per D10.

gbrain.yml example file updated to canonical keys with explanatory
comments.

2 new test cases: deprecated-key fallback (asserts both shapes load
correctly with warning), canonical-wins-over-deprecated (asserts the
"both shapes coexist" path).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: add slugPrefix to PageFilters with engine-side filter (step 4/15)

Issue #13 of the eng review: storage.ts and export.ts loaded every page
in the brain (limit: 1_000_000) to check tier membership. On the 200K-page
brains this feature targets, that's the wall-clock and memory landmine
the feature exists to fix.

Adds an optional `slugPrefix` field to PageFilters. Both engines implement
it as `WHERE slug LIKE prefix || '%' ESCAPE '\'`, with literal escaping of
LIKE metacharacters (%, _, \) so user-supplied prefixes like `media/x/`
are treated as exact string prefixes.

Performance: the (source_id, slug) UNIQUE constraint on the pages table
gives both engines a btree index that supports LIKE-prefix range scans.
An EXPLAIN on Postgres confirms the index range scan rather than a seq
scan. PGLite has the same index shape via pglite-schema.ts.

Consumers updated:
  - export.ts: --slug-prefix flag now goes engine-side (no in-memory
    .filter(...)). The --restore-only path queries each db_only directory
    with slugPrefix in a loop instead of one full-table scan, with seen-set
    deduplication and disk-existence check inline.
  - storage.ts: keeps the full-scan path because storage-status needs the
    "unspecified" bucket count, which can't be computed without enumerating
    every page. Comment notes that step 5 (single-walk filesystem scan)
    will reduce per-page disk syscall cost.

2 new test cases on PGLiteEngine: slugPrefix happy path (3 tier dirs,
asserts only matching slugs return) and metacharacter escape regression
(asserts safe/ doesn't match unrelated slugs).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* perf: single-walk filesystem scan via walkBrainRepo() (step 5/15)

Issue #14 of the eng review: storage.ts called existsSync + statSync
per-page in a synchronous loop. On a 200K-page brain that's 400K syscalls
serialized. Wall-clock landmine.

Adds src/core/disk-walk.ts with walkBrainRepo(repoPath) — one recursive
readdirSync walk, builds a Map<slug, {size, mtimeMs}>. Storage.ts looks
up each DB page in the map (O(1)) instead of stat-checking on demand.
Slug derivation matches the pages-table convention: people/alice.md on
disk becomes people/alice as the map key.

Skipped during walk:
  - dot-directories (.git, .gbrain, .vscode, etc) — not part of the brain
    namespace
  - node_modules — guards against accidentally walking into imported repos
  - non-.md files (sidecar JSON, binaries) — tracked by the brain through
    the files table, not by slug

Reusable: future commands (gbrain doctor's storage_tiering check, the
optional autopilot tier-fix path) get the same walk for free.

9 new test cases: empty dir, nonexistent dir, top-level files, nested
dirs, dot-dir skipping, node_modules skipping, non-.md filtering, size
capture, mtimeMs capture.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: path-segment matching for tier directories (step 6/15)

Issue #5 + D6 of the eng review: tier matching used slug.startsWith(dir),
which falsely matches 'media/xerox/foo' against 'media/x' if a user wrote
the directory without a trailing slash.

The new matcher requires the configured directory to end with `/` and
treats it as a canonical path-segment ancestor:

  media/x/   matches  media/x/tweet-1       ✓
  media/x/   doesn't  media/xerox/foo       ✗
  media/x    refused  media/x/tweet-1       (matcher requires trailing /)

Non-canonical input (no trailing slash) is refused outright. Step 7's
auto-normalizing validator converts user-written 'media/x' → 'media/x/'
on load, so the matcher never sees non-canonical input from real configs.
The behavior tested here is the strict matcher's contract.

Regression test pins the media/xerox collision case explicitly.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: auto-normalize trailing-slash, throw on tier overlap (step 7/15)

D7+D8 of the eng review: validation was warnings-only. Users miss warnings.
Now:

  - Cosmetic: missing trailing slash auto-corrected, one-time info note
    showing what changed ("normalized 2 storage paths: 'people' →
    'people/', 'media/x' → 'media/x/'"). Once-per-process to keep noise low.

  - Semantic: same directory in both tiers throws StorageConfigError.
    Ambiguous routing — does media/ win as db_tracked or db_only? — is a
    real bug the user must fix. Caller propagates to the CLI for a clean
    exit-1 with actionable message.

loadStorageConfig now applies normalize+validate after merging deprecated
keys, so the path-segment matcher (step 6) only ever sees canonical
trailing-slash directories.

The pure validateStorageConfig kept for callers who want the warnings list
without the auto-fix side effects (gbrain doctor's reporting path).

2 new test cases: auto-normalize round-trip with warning text assertion,
overlap throws StorageConfigError.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: wire manageGitignore into runSync, only on success (step 8/15)

Issue #2 of the eng review: manageGitignore was defined and never
invoked. Docs claimed "auto-managed by gbrain" — false. Users hit a
.gitignore that never updated and committed db_only directories anyway.

Wire-up: runSync now calls manageGitignore after each successful
performSync return, in both watch and one-shot modes.

Eng review pass-2 finding #1: skip on dry_run AND blocked_by_failures
status. A sync that aborted partway has stale state; mutating .gitignore
based on a partially-loaded config invites drift. Failure-skip test
added (uses .gitignore-as-a-directory to simulate write failure;
asserts warning fired and disk wasn't corrupted).

Hardened manageGitignore itself with three additional behaviors:

  - GBRAIN_NO_GITIGNORE=1 escape hatch (D23) for shared-repo setups
    where a maintainer wants gbrain to leave .gitignore alone.

  - Submodule detection (D49). When repoPath/.git is a regular file
    (gitdir: ... pointer), the repo is a git submodule. Submodule
    .gitignore changes don't survive parent submodule updates, so we
    skip with an actionable warning ("add db_only directories to your
    parent repo's .gitignore manually").

  - Graceful failure (D9). Read errors, write errors, and
    StorageConfigError (overlap from step 7) all log a warning and
    return — sync's primary job (moving data) shouldn't die because of
    a side-effect on .gitignore.

manageGitignore is now exported (previously private) so the
storage-sync test file can hit it directly without spinning up sync.

9 new test cases: no-op without gbrain.yml, no-op with empty db_only,
happy-path append, idempotency (run twice, single entry), preservation
of user-written rules, GBRAIN_NO_GITIGNORE skip, submodule skip,
.git-directory normal path, write-failure graceful warning.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: D5 resolution chain for --restore-only and storage status (step 9/15)

D5 of the eng review: gbrain export --restore-only without --repo
silently fell through to the regular export path, dumping every page in
the database to the wrong directory. Hard regression risk.

Now exits 1 with an actionable message when --restore-only has no
--repo AND no configured default source. Resolution order:
  1. Explicit --repo flag
  2. Typed sources.getDefault() (reuses step 1's accessor)
  3. Hard error — never fall through to cwd

storage.ts:38 also bypassed BrainEngine with raw SQL and a bare
try/catch (Issue #3 + Issue #9). Replaced with the same typed
getDefaultSourcePath() — single source of truth, errors propagate
cleanly to the user, no silent cwd fallback.

Regular export (no --restore-only) keeps its current behavior per D26:
exports include everything, --repo is optional.

4 new test cases on PGLite in-memory:
  - hard-errors with no --repo + no default
  - explicit --repo wins
  - falls back to sources default local_path
  - non-restore export does not require --repo

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor: split storage.ts into pure data + JSON + human formatters (step 10/15)

Issue #10 of the eng review: getStorageStatus and runStorageStatus mixed
data gathering, JSON serialization, and human-readable output in one
function. Hard to test, hard to reuse, mismatched the orphans.ts pattern
that CLAUDE.md cites as the precedent.

Now three pure functions + a thin dispatcher:

  getStorageStatus(engine, repoPath) — async, returns StorageStatusResult.
    Side effects: engine.listPages + one walkBrainRepo (Issue #14).
    Exported so MCP exposure (D14) and gbrain doctor (D13) can consume the
    same data without re-running the loop.

  formatStorageStatusJson(result) — pure, returns indented JSON. Stable
    contract on the StorageStatusResult shape, suitable for orchestrators.

  formatStorageStatusHuman(result) — pure, returns ASCII text (D10 — no
    unicode box-drawing). Composable into other commands later.

  runStorageStatus(engine, args) — thin dispatcher: parses --repo /
    --json, calls getStorageStatus, picks a formatter, prints.

8 new test cases on the formatters: JSON parse round-trip, null-config
fallback, missing-files capped at 10 with rollup, ASCII-only assertion
(D10 regression guard), warnings inline, configuration listing, disk-
usage block omitted when zero bytes.

The StorageStatusResult interface is now exported as a public type, so
gbrain doctor's storage_tiering check can build its own findings from
the same shape.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* types: distinct PageCountsByTier and DiskUsageByTier (step 11/15)

Issue #11 of the eng review: pagesByTier (page counts) and
diskUsageByTier (byte totals) shared the same structural type
(Record<StorageTier, number>). Both are tier-keyed numeric maps but
carry semantically different units. A future bug that swaps them at a
call site (e.g., displaying disk bytes where the count belongs) wouldn't
trip the compiler.

Replaced with distinct nominal types via a brand field. Structurally
identical at runtime (no overhead) but compile-time disjoint —
TypeScript catches accidental cross-assignment.

  PageCountsByTier   { db_tracked, db_only, unspecified } : numbers (count)
  DiskUsageByTier    { db_tracked, db_only, unspecified } : numbers (bytes)

Both initialized in getStorageStatus, both threaded into
StorageStatusResult, both consumed by formatStorageStatusHuman /
formatStorageStatusJson without further changes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: PGLite soft-warn + full lifecycle test (step 12/15)

D4: storage tiering on PGLite is a partial feature. The "DB" the pages
live in IS the local file gbrain uses for everything else, so "db_only"
has no real offload effect. The .gitignore management still helps
(keeps bulk content out of git history), so we warn and proceed —
not refuse.

Two warning sites (once-per-process each via module-local flags):
  - storage status: warns at runStorageStatus entry
  - sync: warns inside manageGitignore when engineKind='pglite' and
    config has db_only entries

Both phrased actionably ("To get full tiering, migrate to Postgres
with `gbrain migrate --to supabase`").

manageGitignore signature now takes an optional `engineKind` param.
runSync passes engine.kind. Stand-alone callers (tests, future
gbrain doctor --fix path) can omit it.

New test: test/storage-pglite.test.ts — D8 + D4 lifecycle. 6 cases:
engine.kind assertion, getStorageStatus loading gbrain.yml + reporting
tier counts, manageGitignore PGLite-warn (once per process), Postgres
no-warn, slugPrefix on PGLite, end-to-end (config + putPage + status
+ gitignore).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: add trailing-newline CI guard (step 14/15)

Issue #7 of the eng review: all four new files in the original
storage-tiering branch lacked POSIX trailing newlines. Linters complain,
git diffs phantom-flag every future edit. We've been adding newlines as
each file landed; this commit catches the regression class.

scripts/check-trailing-newline.sh:
  - sibling to check-jsonb-pattern.sh / check-progress-to-stdout.sh per
    CLAUDE.md's CI guard pattern
  - portable to bash 3.2 (macOS default; no mapfile, no associative arrays)
  - covers src/**, test/**, gbrain.yml, top-level *.md
  - reports each missing file by path and exits 1

Wired into `bun run test` between progress-to-stdout and typecheck.

Also fixed docs/storage-tiering.md (pre-existing missing newline from
the original branch — caught by the new guard on first run).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: v0.23.0 — VERSION, CHANGELOG, README, CLAUDE.md, storage-tiering.md (step 15/15)

VERSION → 0.23.0 (minor bump for new feature surface).

CHANGELOG entry in Garry voice with the canonical format:
  - Two-line bold headline ("Storage tiering, finally working...")
  - Lead paragraph naming what was broken before and what users get now
  - "Numbers that matter" before/after table for the 6 things that
    actually changed
  - "What this means for your brain" closer
  - "To take advantage of v0.23.0" self-repair block (per CLAUDE.md
    convention) — 6 numbered steps users can follow
  - Itemized changes split into critical fixes / new+renamed surface /
    architecture cleanup / tests + CI guards

CLAUDE.md "Key files" gains four new entries: storage-config.ts,
disk-walk.ts, the v0.23.0 storage.ts shape, and gbrain.yml itself.

README.md gains a new "Storage tiering" section between Skillify and
Getting Data In with the canonical example + commands + link to the
full guide.

docs/storage-tiering.md rewritten end-to-end with canonical key names
(db_tracked / db_only), v0.23.0 hardening details (idempotency,
submodule detection, GBRAIN_NO_GITIGNORE, dry-run gating), the
resolution chain for --restore-only, the auto-normalize +
throw-on-overlap validator, and the PGLite engine note.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: e2e Postgres lifecycle for storage tiering (step 16/16)

Per the v0.23.0 plan: full lifecycle E2E against real Postgres.

  - engine.kind === 'postgres' assertion
  - Full lifecycle: write 4 pages (1 db_tracked, 2 db_only, 1 unspecified)
    → getStorageStatus reports correct tier counts → human formatter
    renders → manageGitignore writes managed block → idempotency check
    → getDefaultSourcePath() resolves the configured local_path.
  - Container restart simulation: 2 db_only pages in DB, files missing
    on disk → status.missingFiles.length === 2 → slugPrefix engine
    filter on Postgres returns exactly the tier slugs.
  - slugPrefix index-based range scan regression: 50 media/x/* + 50
    people/p-* pages → slugPrefix='media/x/' returns exactly 50.
  - getDefaultSourcePath returns null when default source has no
    local_path (the hard-error path that replaces the original silent
    cwd fallback).
  - manageGitignore on Postgres engine does NOT emit the PGLite
    soft-warn (cross-engine assertion).

Skips gracefully when DATABASE_URL is unset, per CLAUDE.md E2E pattern.
Run via: DATABASE_URL=... bun test test/e2e/storage-tiering.test.ts

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: rebump version 0.23.0 → 0.22.9

Reverts the minor bump back to a patch-style version on the v0.22 line.
Storage tiering ships within the v0.22.x train alongside the recent
fix waves. Updates VERSION, package.json, CHANGELOG header + body refs,
CLAUDE.md Key files annotations, README.md section heading, and the
docs/storage-tiering.md backward-compat note.

* chore: bump version 0.22.9 → 0.22.11

Sibling workspaces claimed v0.22.10 in the queue. This branch advances
to v0.22.11 to keep the version monotonic on master.

Updates VERSION, package.json, CHANGELOG header + body refs, CLAUDE.md
Key files annotations, README.md section heading, and the
docs/storage-tiering.md backward-compat note.

* fix: address Codex pre-landing review findings (4 fixes)

Codex found 4 real issues during pre-landing review of v0.22.11 diff:

[P0] export --restore-only fell through to full export when
storageConfig was null (no gbrain.yml present). On older or
misconfigured brains, the recovery command would silently dump the
entire database. src/commands/export.ts now refuses with an actionable
error before any page query fires — matches the D5 lock spirit
("never silently fall through").

[P1] manageGitignore wire-up only fired when --repo was passed
explicitly. performSync resolves the repo from sync.repo_path or
sources.local_path, so the common `gbrain sync` path (after
setup, no flag) never updated .gitignore. src/commands/sync.ts now
uses the same source-resolver chain as the rest of /ship: opts.repoPath
→ getDefaultSourcePath → null. Fires in both watch and one-shot modes.

[P2] getDefaultSourcePath only consulted sources.local_path, missing
the legacy global sync.repo_path config key that pre-v0.18 brains use.
Added a fallback to engine.getConfig('sync.repo_path') when the
sources row has NULL local_path. Pre-v0.18 brains now work without
forcing a `gbrain sources add . --path .` migration.

[P2] sync --all multi-source loop never called manageGitignore even
though src.local_path was already known. Each source now gets its own
gitignore update on successful sync.

Tests:
  - test/storage-export.test.ts: replaced the old "falls through to
    full export" test with one that asserts the new refusal path
    (storage-tiering config required for --restore-only).
  - test/source-resolver.test.ts: added a fallback test exercising the
    legacy sync.repo_path code path for pre-v0.18 brains.
  - All 78 storage-tiering tests still pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: regenerate llms.txt + llms-full.txt for v0.22.11

Per CLAUDE.md: "Run `bun run build:llms` after adding a new doc."
The README's new Storage tiering section + the rewritten
docs/storage-tiering.md changed the inlined bundle. test/build-llms.test.ts
catches the drift and was failing on master pre-regen.

* fix: typecheck error in disk-walk.ts (CI #73350475897)

tsc --noEmit failed in CI because ReturnType<typeof readdirSync> with
withFileTypes:true picks an overload union that includes
Dirent<Buffer<ArrayBufferLike>>. Strict tsc treats entry.name as Buffer,
so .startsWith / .endsWith / string comparisons all blew up.

Annotate the variable as Dirent[] (string-based) and cast through unknown,
matching the pattern sync.ts already uses for its own filesystem walk.
Same runtime behavior; clean typecheck.

Tests still 9/9.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
EOF

---------

Co-authored-by: root <root@localhost>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 22:21:07 -07:00
Garry Tan 7aae6ec2b0 Merge remote-tracking branch 'origin/master' into garrytan/claw-setup-e2e
# Conflicts:
#	CHANGELOG.md
#	VERSION
#	package.json
2026-04-29 22:18:08 -07:00
Garry Tan 189da55f46 chore: rebump to v0.22.16 (next free 0.22.x patch slot per queue)
PR #506 claims v0.22.15, PR #521 claims v0.22.10, intermediate slots
(.11/.12/.13/.14) are claimed by other open PRs. v0.22.16 is the next
clean PATCH slot. v0.23.0 is claimed by PR #462 so MINOR isn't free.
This release fits the 0.22.x train; v0.23.0 lands when #462 ships.

Updates VERSION, package.json, CHANGELOG.md header, TODOS.md follow-up
labels. Code is unchanged.
2026-04-29 22:14:47 -07:00
5d9dc4393e v0.22.10 fix: autopilot-cycle handler forwards job.data.phases to runCycle (#521)
* fix: autopilot-cycle handler forwards job.data.phases to runCycle

The autopilot-cycle handler always ran ALL_PHASES regardless of job data.
This caused production stalls when the embed phase had a large backlog
(17K+ stale chunks) that exceeded the 30-minute job timeout. Every 5-min
cycle would start, hit the embed wall, stall, and get force-killed —
creating an infinite stall loop that kept the queue perpetually unhealthy.

The fix validates job.data.phases against ALL_PHASES (preventing injection)
and forwards the selected phases to runCycle(). Callers can now submit
fast cycles (lint+backlinks+sync+extract) on a 5-min cron and run embed
separately with a longer timeout during off-peak hours.

If phases is omitted, not an array, or filters to empty, behavior is
unchanged (all phases run).

Tests: 4 new cases covering phase restriction, invalid name filtering,
empty array fallback, and non-array type safety.

* test: widen autopilot-cycle handler-block window for phases-passthrough

The regression guard sliced the first 500 chars after `worker.register('autopilot-cycle'`
and asserted `signal: job.signal` was present. The phase-validation block added in
787ec7de pushed the signal arg past that boundary, so CI test shard 3 failed even
though the handler still propagates the signal correctly. Bump the window to 2000.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v0.22.10)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* docs: sync release notes for v0.22.10

Note autopilot-cycle phases passthrough fix on the src/commands/jobs.ts
key-files annotation so future readers know the handler honors
job.data.phases (validated against ALL_PHASES) as of v0.22.10.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: regenerate llms-full.txt for v0.22.10 CLAUDE.md update

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: Wintermute <wintermute@garrytan.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 22:13:05 -07:00
Garry Tan bbc8bba0ff fix(tests): typecheck failures + spawnWithCapture timeout headroom in CI
Three CI fixes after PR #522 landed:

1. test/agent-runner.test.ts:89 — UnavailableRunner.invoke() returns
   Promise<void> by default but the AgentRunner contract requires
   Promise<InvokeResult>. Annotate the throw-only invoke explicitly so tsc
   sees the contract is satisfied (the throw makes the body unreachable as
   far as the return type is concerned).

2. test/seed-pglite.test.ts — bun:test signature is test(name, fn, timeoutMs:
   number), not test(name, opts: {timeout}, fn). The {timeout: 30_000} object
   form was a guess that tsc on bun 1.3.13 rejects. Move the 30s cap to the
   trailing positional number arg on each PGLite-using test.

3. test/transcript-capture.test.ts — `spawnWithCapture > timeout fires
   SIGTERM/SIGKILL` blew the 10s outer cap on the GitHub runner. Two fixes:
   (a) use `exec sleep` so the child we spawn IS sleep — SIGTERM goes
   directly to it, no `/bin/sh` fork-vs-exec process-group ambiguity that
   could orphan the sleep and force the SIGKILL grace path. (b) bump outer
   cap to 30s for headroom even when the runner is slow and SIGKILL after
   the 5s grace is what actually ends the child.
2026-04-29 17:57:23 -07:00
Garry TanandClaude Opus 4.7 8200c275d6 chore: bump version and changelog (v0.24.0)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-29 17:48:13 -07:00
Garry Tan 953c8e75f6 Merge remote-tracking branch 'origin/master' into garrytan/claw-setup-e2e
# Conflicts:
#	.gitignore
#	src/cli.ts
2026-04-29 17:43:37 -07:00
Garry Tan 6c64670b33 feat: register gbrain claw-test + gbrain friction; CLAUDE.md + llms sync
Wires both commands into src/cli.ts CLI_ONLY allow-list and adds dispatch
in handleCliOnly so neither command requires a brain engine connection.

CLAUDE.md gains entries for src/commands/{friction,claw-test}.ts +
src/core/claw-test/ + skills/_friction-protocol.md, and a Commands section
listing all 8 new gbrain claw-test ... and gbrain friction ... invocations
with the v0.23 marker. Documents the GBRAIN_HOME write-isolation contract
and the v1 caveat (read-side host-fingerprint detection deferred to v1.1).
llms.txt + llms-full.txt regenerated via 'bun run build:llms' so the
committed generator-output gate passes.

test/e2e/claw-test.test.ts is the scripted-mode E2E. Builds a tiny shim that
delegates to 'bun run src/cli.ts' (NOT bun --compile, which doesn't bundle
PGLite's runtime assets), points the harness at it via GBRAIN_BIN_OVERRIDE,
runs --scenario fresh-install end-to-end. Asserts exit 0, zero error/blocker
friction. Includes a deliberate-break test that proves the friction signal
fires when a phase command rejects.

test/claw-test-cli.test.ts covers shipped-scenario load + agent registry +
OpenClawRunner detection (relative-path / .. / missing-bin guards) + the
GBRAIN_FRICTION_RUN_ID env handoff between harness and friction CLI.

Closes the v0.23 claw-test E2E feature.
2026-04-29 16:13:54 -07:00
Garry Tan 4fc261a18c feat: claw-test scenario fixtures + friction-protocol skills convention
Two scenarios ship in v1 — fresh-install and upgrade-from-v0.18. Each is a
self-contained directory: brain/ (markdown pages), BRIEF.md (live-mode prompt),
expected.json (scripted-mode assertions), scenario.json (kind, expected_phases,
optional from_version + seed paths). Schema is owned by src/core/claw-test/
scenarios.ts.

upgrade-from-v0.18 ships scaffolded — seed/dump.sql is the v1.1 follow-up
(needs a real v0.18-shape PGLite dump; seed/README.md documents the gen
procedure). The harness gracefully no-ops the seed phase when dump.sql is
absent.

skills/_friction-protocol.md is a cross-cutting convention skill (like
_brain-filing-rules.md). Tells agents when to call gbrain friction log and how
to choose severity. Skills the claw-test exercises will gain a > Convention:
callout pointing here in a v1.1 sweep.

13 unit tests for the scenario loader + 'shipped scenarios load cleanly' for
both.
2026-04-29 16:13:36 -07:00
Garry Tan fe09ffaa71 feat: gbrain claw-test — end-to-end fresh-install friction harness
Two modes: scripted (CI gate, no agent) and --live (real agent subprocess).
Phases: setup → install_brain (gbrain init --pglite) → import (--no-embed) →
query → extract all --source fs → verify (gbrain doctor --json, asserts
status==='ok' and progress.jsonl phase coverage).

AgentRunner interface + registry — interface stays narrow (detect, invoke,
optional postInstallHook). v1 ships only OpenClawRunner; the registry pattern
lets v1.1 land hermes/codex as ~50-line additions without refactoring callers.
OpenClaw invocation: 'openclaw agent --local --agent <name> --message <brief>'
matching test/e2e/skills.test.ts (NOT --prompt-file, which doesn't exist).

transcript-capture: spawns child with piped stdio, async-drains via
fs.createWriteStream + 'drain' events so 256KB+ bursts don't stall the child
(D17 backpressure). Writes <run>/transcript.jsonl with schema_version + ts +
channel + byte_offset + bytes_b64. Friction entries' transcript_offset field
references byte offsets here so render --transcripts can resolve back.

progress-tail: parses gbrain's --progress-json events out of child stderr.
Phase verification asserts each scenario.expected_phases entry (dotted names
like import.files, extract.links_fs, doctor.db_checks) saw at least one event
from the actual command — proves the COMMAND ran, not that the agent obeyed
prompts.

seed-pglite: ~50 LOC SQL replay primitive for the upgrade-from-v0.18 scenario.
Existing migration helpers (test/e2e/helpers.ts) are Postgres-only; PGLite has
no equivalent. seedPglite opens a fresh PGLite, executes each statement
individually (errors name the failing one), then disconnects so gbrain init
can take over and walk forward.

53 unit tests covering registry selection, runner detection, multi-byte UTF-8
chunk-boundary safety, PIPE buffer drain, scenario load+validate, progress
event parsing, and SQL splitter.
2026-04-29 16:13:21 -07:00
Garry Tan 00843f2a5a feat: gbrain friction {log,render,list,summary} — agent friction reporter
Append-only JSONL writer at $GBRAIN_HOME/friction/<run-id>.jsonl. Schema is a
flat extension of StructuredAgentError (D20), one envelope shape across both
agent-emitted entries and harness-wrapped command failures. Run-id resolves
from --run-id > $GBRAIN_FRICTION_RUN_ID > 'standalone'.

Subcommands stay ≤30 LOC each; core lives in src/core/friction.ts (writer +
reader + renderer + redactor). render --redact (default for md output) strips
\$HOME / \$CWD to placeholders so reports paste safely in PRs/issues.

Severity: confused | error | blocker | nit. Kind: friction | delight (D7) |
phase-marker | interrupted. Readers tolerate malformed lines (skip + warn).

40 unit tests; this is the channel the claw-test harness writes to and that
agents emit through during live-mode runs.
2026-04-29 16:02:38 -07:00
Garry Tan ce6af836e7 feat: hermeticity migration — every $GBRAIN_HOME write site honors the env override
configDir() in src/core/config.ts already implemented $GBRAIN_HOME as a
parent-dir override (returns <override>/.gbrain), but ~12 consumers built paths
from os.homedir() directly and bypassed it. Critically, loadConfig/saveConfig
themselves used a private getConfigDir() that ignored the env. Fixed.

Migrated every write site to gbrainPath() — fail-improve, validator-lint, cycle
lock, shell-audit, backpressure-audit, sync-failures, integrity logs,
integrations heartbeat, init pglite path, migrate-engine manifest, import
checkpoint, v0_13_1 rollback, v0_14_0 host-work. Read-side host-detection in
init.ts (~/.claude / ~/.openclaw probes) intentionally NOT migrated; that's a
v1.1 follow-up under a separate $GBRAIN_HOST_HOME override.

Adds gbrainPath(...segments) sugar plus path validation: $GBRAIN_HOME must be
absolute and contain no '..' segments (throws GbrainHomeInvalidError).

test/gbrain-home-isolation.test.ts proves write-isolation across all migrated
sites. test/migrations-v0_14_0.test.ts updated to use $GBRAIN_HOME instead of
the old HOME-swap pattern.

Closes part of the claw-test E2E harness preconditions (D13 + D21).
2026-04-29 16:02:20 -07:00
08746b06d2 v0.22.9 feat: structured error code summary for sync --skip-failed (#501)
* feat: structured error code summary for sync --skip-failed (#500)

When sync encounters per-file failures, the blocked/skip-failed messages
now include a breakdown by error code (SLUG_MISMATCH, YAML_PARSE, etc.)
instead of just a raw count. This makes it immediately obvious *why*
files failed without requiring manual investigation.

Changes:
- Add classifyErrorCode() — maps error messages to ParseValidationCode
- Add summarizeFailuresByCode() — groups failures into sorted code summary
- SyncFailure now carries a 'code' field (backfilled on acknowledge)
- acknowledgeSyncFailures() returns AcknowledgeResult {count, summary}
- sync blocked + skip-failed messages show code breakdown
- doctor sync_failures check shows code breakdown for both unacked and historical
- 12 new tests for classifyErrorCode, summarizeFailuresByCode, and structured returns

Before:
  Sync blocked: 2688 file(s) failed to parse.

After:
  Sync blocked: 2688 file(s) failed to parse:
    SLUG_MISMATCH: 2685
    YAML_DUPLICATE_KEY: 3

Closes #500

* fix: eng-review fixes for sync error-code classification

- Reorder classifyErrorCode() so DB-layer errors (DB_DUPLICATE_KEY,
  STATEMENT_TIMEOUT) check BEFORE YAML patterns. Postgres "duplicate key
  value violates unique constraint" no longer mislabels as YAML_DUPLICATE_KEY.
- Rewrite MISSING_OPEN/MISSING_CLOSE/EMPTY_FRONTMATTER/NULL_BYTES/NESTED_QUOTES
  regexes to match the canonical messages emitted by collectValidationErrors()
  in src/core/markdown.ts. Previous patterns (e.g. /missing.*open/i) never
  fired because the upstream throw site emits prose ("File is empty...",
  "No closing --- delimiter found"), not the code name.
- Extract formatCodeBreakdown() helper that accepts either raw failures or
  pre-summarized {code, count}[] input. Replaces 3 duplicate inline builders
  in src/commands/sync.ts.
- 15 new tests (37/37 pass on test/sync-failures.test.ts):
  - DB vs YAML duplicate-key disambiguation (3 cases)
  - Canonical-message coverage for the 5 frontmatter codes (7 cases)
  - acknowledgeSyncFailures() legacy-entry backfill branch (2 cases)
  - formatCodeBreakdown() dual-input shape (3 cases)
- TODOS.md: file 3 follow-ups (P2 plumb structured ParseValidationCode;
  P0-at-ship CHANGELOG migration note for AcknowledgeResult; P3 concurrent-
  safe ack of sync-failures.jsonl).

Eng-review plan: ~/.claude/plans/then-codex-synchronous-toucan.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v0.22.9)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* ci: 16-core runner + 4-way matrix shard for test job

The unit test suite ran 22m17s on ubuntu-latest (2-core/7GB) because:
- 187 test files run with bun test parallelism bounded by core count
- 23 of those files spin up a fresh PGLiteEngine + initSchema in beforeEach,
  paying ~22s WASM cold-start per test on the small runner

This commit fixes the runner side:
- runs-on: ubuntu-latest-16-cores (16 vCPU / 64 GB RAM)
- strategy.matrix.shard splits 4 parallel jobs, each running ~40 of 158 unit
  test files. Single-file wall-time floor is ~3 min after the test refactor,
  so 4 shards × 16 cores hits the floor quickly without wasting cores past it.
- pre-test gates (typecheck, check-jsonb, check-progress, check-wasm) only run
  on shard 1 — they're not test files and don't benefit from sharding.

scripts/test-shard.sh partitions test files by stable FNV-1a hash mod N. Same
file always lands in the same shard, so retries are reproducible. Pure shell,
portable to bash 3.2 (macOS) and bash 5.x (CI). Excludes test/e2e/ which runs
via bun run test:e2e separately and needs DATABASE_URL.

Also: ignore .claude/ harness state files (scheduled_tasks.lock etc) instead
of just .claude/skills/.

Cost: ~$0.19/run vs $0 (public repo, default runner is free). At 50 PRs/month
that's ~$10/month for ~5x faster CI.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: refactor top-3 PGLite-heavy files to share one engine per file

Three test files were spinning up a fresh PGLiteEngine + connect + initSchema
in beforeEach. PGLite WASM cold-start is ~22s on the small CI runner; doing
this per test multiplied wall-time across the suite. The 3 files alone
accounted for ~6.5 min of the 22m CI run (177s + 132s + 87s).

Refactor: move PGLite setup to beforeAll (one engine per file), wipe data
in beforeEach via the new test/helpers/reset-pglite.ts helper.

The reset helper:
- TRUNCATEs every public table CASCADE, including sources (so tests that
  register their own sources don't leak rows into the next test).
- Re-seeds the default source row that pages.source_id's DEFAULT FKs against.
  Without this, the next page insert would fail FK validation.
- Preserves schema_version so migration helpers don't think the brain is on v0.

Files refactored:
- test/extract-incremental.test.ts (8 tests, was 177s on CI)
- test/brain-writer.test.ts (16 tests; only the scanBrainSources block uses
  PGLite, was 132s on CI)
- test/sync.test.ts (37 tests; only the performSync dry-run block uses PGLite,
  was 87s on CI)

All 61 tests still pass locally. The remaining 20 PGLite-heavy files use the
same beforeEach anti-pattern; this commit only refactors the proven worst
offenders. Sweep the rest in a follow-up if CI numbers indicate it's worth it.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* ci: fall back to ubuntu-latest for matrix shard

The ubuntu-latest-16-cores label requires a provisioned larger-runner pool in
repo/org settings. Without that setup, jobs queue indefinitely waiting for a
runner that doesn't exist (verified: 4 shards stuck in 'queued' status with
empty runner_name for 5+ min).

Drop back to the default 2-core ubuntu-latest. The 4-way matrix shard still
delivers ~5-6x speedup via parallelism alone — 4 jobs running in parallel,
each handling ~40 of 158 unit test files. Cost stays $0 (default runner is
free for public repos).

If we ever provision a larger-runner pool, flip this label back to
ubuntu-latest-16-cores. The matrix + sharder will use the bigger boxes
unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Wintermute <wintermute@garrytan.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 10:12:10 -07:00
8468ba25a9 v0.22.8 perf: doctor integrity batch-load + multi-source correctness (#393)
* perf: batch-load integrity scan — 500 round-trips → 1 SQL query

doctor's integrity_sample check called getPage() sequentially for 500
pages through PgBouncer transaction-mode pooling. Each call required a
full connection acquire/release cycle, causing doctor to timeout (~90s+)
on production deployments.

Replace with a single SQL query that fetches slug, compiled_truth, and
frontmatter for all candidate pages at once. Falls back to the
sequential path for PGLite or when no DB connection is available.

Before: doctor timeout (killed at 60s)
After:  doctor completes in ~6s (full run including all other checks)

143 existing minions tests pass unchanged.

* fix: skillpack acquireLock negative-age on Linux sub-ms fs timestamps

On Linux ext4, statSync().mtimeMs has sub-ms precision while Date.now() is
integer ms. A just-written lockfile can report an mtime ~0.3ms ahead of
Date.now(), making age negative. The acquireLock check `age >= staleMs`
then evaluated false on staleMs:0, falling through the forceUnlock branch
and throwing "Another skillpack install appears to be running" instead of
unlocking. macOS rounds to integer ms so this only surfaced on Linux CI.

Clamp age to zero and add a utimesSync-based regression test that pushes
the lock mtime 10ms into the future to deterministically reproduce the
negative-age case on any platform.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: scanIntegrity batch path scopes by unique slug + Postgres-only gate

Codex review caught that the batch SQL scanned raw (source_id, slug) rows
while sequential's getAllSlugs() returned a Set<string>. On multi-source
brains (UNIQUE(source_id, slug) since v0.18.0), the batch path overcounted
hits and exhausted the LIMIT before covering N distinct pages.

Three changes:

  - SELECT DISTINCT ON (slug) ... ORDER BY slug mirrors Set<string>
    semantics; multi-source brains now get exact unique-slug counts.

  - engine.kind === 'postgres' gate at the call site so PGLite never
    enters the batch branch (catch{} fallback was firing on every PGLite
    doctor run, polluting the GBRAIN_DEBUG log signal).

  - Replace bare catch{} with debug-gated console.error so real Postgres
    errors (deadlock, connection drop, SQL bug) are diagnosable instead
    of silently swallowed.

Plus inline comments explaining the WHY for DISTINCT ON, the engine.kind
gate, the GBRAIN_DEBUG fallback, and the validate filter divergence
(boolean is the documented contract; stringly-typed handled at lint).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: scanIntegrity batch parity (dedup, hits, validate, topPages)

Real-Postgres E2E tests asserting the batch fast path returns identical
results to the sequential path on the four cases that matter:

  - dedup: multi-source duplicate slugs scan once (regression guard for
    the codex catch). Raw SQL fixture seeds the alt-source row since
    engine.putPage doesn't take a source_id.
  - hits: bareHits and externalHits arrays match between paths.
  - validate: validate:false (boolean) page is skipped on both paths.
  - topPages: ordering matches.

Skip when DATABASE_URL is not set (matches existing test/e2e/ pattern).
Per-test TRUNCATE keeps fixture state isolated.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump version, changelog, and CLAUDE.md (v0.22.7)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: regenerate llms-full.txt for v0.22.7 CLAUDE.md updates

CLAUDE.md gained the integrity.ts inventory entry and the new
test/e2e/integrity-batch.test.ts test file in commit edd4329.
The committed llms-full.txt bundle inlines CLAUDE.md content,
so it needs to be regenerated to match.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump v0.22.7 → v0.22.8

Same content as v0.22.7 (doctor integrity batch-load + multi-source
correctness + skillpack Linux fs-timestamp fix), retitled to v0.22.8 to
slot above master's pending v0.22.7 if/when that releases first.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: root <root@localhost>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-28 19:53:31 -07:00
d3b52edeba v0.22.7 fix: built-in HTTP transport with bearer auth for remote MCP (#483)
* fix: add built-in HTTP transport with bearer auth for remote MCP

Adds `gbrain serve --http` with token-based authentication using the
existing access_tokens table. Eliminates the need for standalone OAuth
wrappers that may have insecure open registration endpoints.

- New: src/mcp/http-transport.ts — HTTP+SSE transport with bearer auth
- New: SECURITY.md — security advisory for remote MCP deployments
- Updated: serve command accepts --http and --port flags
- Updated: DEPLOY.md recommends --http for remote access
- Bump: 0.22.4 → 0.22.5

* chore: extract shared MCP dispatch + rate-limit modules

dispatch.ts is the single source of truth for stdio + HTTP transport: validateParams,
OperationContext build, handler invocation, error formatting. Server.ts refactored to
use it. Prevents the F1-F3 transport-drift bugs where stdio and HTTP independently
implemented dispatch logic differently (reversed args, missing context fields, no
param validation).

rate-limit.ts: bounded-LRU token-bucket. Tracks lastTouchedMs separately from
lastRefillMs so an exhausted key can't be reset by hammering past the TTL.

* feat: HTTP transport hardening + F1-F3 dispatch bug fixes

Rewrite of src/mcp/http-transport.ts on top of the new dispatch.ts and rate-limit.ts:

- F1 fix: dispatch via shared dispatchToolCall(ctx, params) — was reversed args
  (params, ctx) before, would have crashed every real tools/call.
- F2 fix: full OperationContext (engine, config, logger, dryRun, remote) — was
  only {engine, remote: true} before.
- F3 fix: validateParams runs on HTTP path — was skipped before.
- Engine.kind fail-fast: clear error message on PGLite (access_tokens table is
  Postgres-only by design).
- CORS: default-deny via GBRAIN_HTTP_CORS_ORIGIN allowlist.
- Body cap: stream-counted via req.body reader, catches chunked transfers
  without Content-Length. Default 1 MiB via GBRAIN_HTTP_MAX_BODY_BYTES.
- Rate limit: pre-auth IP bucket fires BEFORE DB lookup (limits brute-force
  load), post-auth token-id bucket fires after auth (limits runaway clients).
  Both bounded LRU with TTL prune.
- mcp_request_log: per-request audit row reusing the existing schema (v4).
- last_used_at SQL-level debounce: WHERE last_used_at < now() - interval
  '60 seconds'. Race-tolerant under PgBouncer.
- Response shape: application/json (gbrain MCP tools don't stream).
  Streamable-HTTP transport spec compliant for non-streaming responses.
- X-Forwarded-For honored only when GBRAIN_HTTP_TRUST_PROXY=1.

* feat: wire gbrain auth into the main CLI

The original PR's docs referenced 'gbrain auth create/list/revoke' but auth.ts
was a standalone script never wired to the CLI dispatcher. Running 'gbrain auth'
from the compiled binary returned 'Unknown command'.

- auth.ts: extract the dispatch into runAuth(args) + import.meta.main guard
  so direct-script invocation still works (bun run src/commands/auth.ts ...).
- cli.ts: add 'auth' to CLI_ONLY set + handler in handleCliOnly that imports
  runAuth and dispatches without requiring an engine connection (auth.ts
  manages its own postgres() connection).

* test: HTTP transport unit + E2E coverage (23 + 8 cases)

test/http-transport.test.ts — 23 unit cases against mocked engine.sql:
  - Auth: valid/missing/no-Bearer/unknown/revoked/health-bypass (1-6)
  - F1+F2 round-trip via dispatch.ts (7) — regression guard for reversed args
  - F3 invalid_params via validateParams (8) — regression guard
  - Response Content-Type application/json, not SSE (9)
  - CORS default-deny + allowlist + non-match (10-12)
  - Body cap: Content-Length + chunked-transfer (13-14)
  - Rate limit: refill, exhaust+Retry-After, LRU eviction, TTL prune,
    pre-auth IP fires before DB, /health bypasses (15-20)
  - mcp_request_log audit: success row + auth_failed row (21-22)

test/e2e/http-transport.test.ts — 8 cases against real Postgres:
  - /health, tools/list, tools/call list_pages (real op round-trip),
    revoked → 401, last_used_at debounce within 60s (asserts ONE update),
    debounce 65s gap (asserts TWO updates), mcp_request_log row check,
    invalid_params via real handler.

* docs: v0.22.7 CHANGELOG + SECURITY.md + DEPLOY.md

CHANGELOG: v0.22.7 release notes covering the F1-F3 dispatch fixes, the full
hardening surface (CORS default-deny, two-bucket rate limit, body cap, audit
log), and the upgrade path. Master's v0.22.6 schema-verify entry stitched in
above (preserving merge ordering).

SECURITY.md: full hardening reference for gbrain serve --http — Postgres-only
caveat, CORS allowlist, rate limit + tunnel caveat, body cap, audit log query,
GBRAIN_HTTP_TRUST_PROXY warning.

docs/mcp/DEPLOY.md: Postgres-only call-out, env var summary, fail-fast behavior
on PGLite.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix: codex review follow-ups (DB-probing /health + XFF trust safety contract)

- /health now does SELECT 1 against Postgres and returns 503 + status:unhealthy
  when the DB is unreachable. Prevents the failure mode where orchestration
  sees green pods while clients get misleading 401s during a DB outage.
- SECURITY.md: tighten the GBRAIN_HTTP_TRUST_PROXY=1 guidance with the explicit
  two-condition safety contract — gbrain bound to a private interface AND the
  proxy strips client-supplied XFF. Without both, the flag enables IP spoofing
  past the pre-auth rate limit.
- Tests: add 6b (/health DB-down → 503) + assert db:'ok' on the happy path.

Caught by codex adversarial review during /ship Step 11.

* docs: TODOS.md — v0.22.7 follow-ups (audit volume, validateParams enums, SSE, scopes)

* docs: update project documentation for v0.22.7

CLAUDE.md: document src/mcp/dispatch.ts, src/mcp/rate-limit.ts, and the
rewritten src/mcp/http-transport.ts in the Key files section. Add
test/http-transport.test.ts (23 unit cases) and test/e2e/http-transport.test.ts
(8 E2E cases) to the test inventories.

CHANGELOG.md: fix copy-paste version mismatches inside the v0.22.7 entry that
referenced v0.22.5 (header line + "To take advantage of" block).

README.md: replace the standalone bun-run auth invocation with the wired-in
gbrain auth CLI; add gbrain serve --http startup step to the Remote MCP
example; surface gbrain auth in the admin command list; link SECURITY.md
from the Remote MCP section so it's discoverable.

SECURITY.md: align "as of v0.22.5" callouts with the actual release version
(v0.22.7).

docs/mcp/DEPLOY.md: align v0.22.5+ callout with v0.22.7+; switch token-management
examples from `bun run src/commands/auth.ts` to `gbrain auth` now that auth is
in the main CLI.

docs/mcp/ALTERNATIVES.md: drop the "planned but not yet implemented" note for
gbrain serve --http; document that the built-in HTTP transport is the
recommended path.

docs/mcp/{CLAUDE_DESKTOP,CLAUDE_COWORK,CLAUDE_CODE,PERPLEXITY}.md: switch
token-creation examples from `bun run src/commands/auth.ts create` to
`gbrain auth create` to match the wired-in CLI.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix: typecheck — cast CallToolRequestSchema handler return to any

MCP SDK 1.29 widened the response type for setRequestHandler(CallToolRequestSchema, ...)
to require a 'task' field for managed-task responses. gbrain ops are synchronous and
return the legacy { content, isError? } shape, which is still valid via the SDK's
ServerResult union. Casting the handler return type to any silences the narrowing
that broke after dispatch.ts was extracted (the original inline handler dodged this
because TypeScript inferred its return as any from the function body).

CI failure: src/mcp/server.ts(25,51): error TS2345 — Property 'task' is missing in
type 'ToolResult' but required in type '{ ...; task: { taskId: string; ... }; ... }'.
Caught by the 'test' job's bun run typecheck step at PR #483 commit 65ea9e7.

* docs: regenerate llms-full.txt after master merge

The build-llms regen-drift guard fails when committed llms.txt + llms-full.txt
don't match what scripts/build-llms.ts produces from current source. Master's
v0.22.6.1 merge brought in new content (CLAUDE.md entries, CHANGELOG, etc.)
that hadn't been folded into the bundle. Running 'bun run build:llms' to sync.

llms.txt unchanged; llms-full.txt picks up the new entries.

* docs: CHANGELOG — scrub attack-surface enumeration from v0.22.7 entry

Per CLAUDE.md responsible-disclosure rule: 'when a release fixes a security
gap or a user-impacting bug, describe the fix functionally. Do not enumerate
the attack surface, quantify the exposure window, or highlight the most
sensitive records by name in public-facing artifacts.'

Removed:
- Lead-paragraph attack-chain ('attacker who discovers URL → POST /register
  → client_credentials → read entire brain'). Public-doc readers don't need
  the directed probe path.
- 'Bug fixes folded in' section that itemized prior-version failure modes.
  Reframed as a 'transport refactor' note in the For Contributors section,
  describing the dispatch consolidation functionally without claiming the
  prior version was broken in specific ways.
- 'Without the OAuth footgun' lead headline. The fix's mechanism (built-in
  bearer auth via access_tokens) is already self-evident from the headline.
- F1/F2/F3 internal labels and 'caught by codex outside-voice during
  planning' parenthetical.

Kept:
- The full hardening reference table (configuration / behavior, not exposure).
- 'gbrain serve --http' user-facing operator ergonomics.
- 'Postgres-only by design' known-limit framing.
- Dispatch consolidation as a contributor-facing single-source-of-truth note.

SECURITY.md left intact: its OAuth-deployment guidance is generic 'if you
deploy MCP behind a custom HTTP wrapper, here are the rules' framing, not
gbrain-version-specific exposure. That's defensible under the same rule.

* docs: SECURITY.md — drop unverified security@garrytan.com address

The address was in the original PR's SECURITY.md commit (6e740590, author
'root <root@localhost>' — machine-generated) and never verified to exist or
forward anywhere. A non-monitored disclosure address is worse than no address
at all: reports go to a black hole.

Keep the GitHub private security advisory link as the sole disclosure channel.
GitHub Security Advisories is the working path most researchers reach for
first anyway — restricted-access by default, scopes the conversation to
maintainers, and integrates with CVE issuance when needed.

---------

Co-authored-by: root <root@localhost>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-28 17:01:40 -07:00
6966623e0f v0.22.6.1 fix: PGLite/initSchema upgrade-hardening wave (closes 2-year wedge cycle) (#440)
* fix(initSchema): narrow pre-schema bootstrap + v24 PGLite no-op

Closes a 2-year-old wedge cycle that hit users 10+ times across 6 schema
versions (#239, #243, #266, #357, #366, #374, #375, #378, #395, #396).

Bug class: gbrain ships an embedded schema blob (PGLITE_SCHEMA_SQL +
SCHEMA_SQL) that runs before numbered migrations on every initSchema().
The blob references columns that newer migrations introduce. On any
brain older than the migration that adds those columns, the blob crashes
before the migration can run.

Fix: PGLiteEngine.initSchema() and PostgresEngine.initSchema() now call
a new private applyForwardReferenceBootstrap() before the schema blob.
The bootstrap probes for missing forward-referenced state and adds only
what's needed (sources table + pages.source_id, links.link_source +
links.origin_page_id, content_chunks.symbol_name + content_chunks.language).
Fresh installs and modern brains both no-op.

A CI guard test/schema-bootstrap-coverage.test.ts enforces that the
bootstrap covers every forward reference in PGLITE_SCHEMA_SQL. Future
migrations that add column-with-index in the schema blob must extend
the bootstrap; the test fails loudly otherwise.

Migration v24 (rls_backfill_missing_tables) now no-ops on PGLite via
sqlFor.pglite: '' since PGLite has no RLS engine and is single-tenant.
Closes #395.

The plan went through CEO + Eng + Codex review. Codex caught a critical
bug in the original "run all migrations early" approach: it would crash
on v24 trying to ALTER subagent tables that the schema blob hadn't
created yet. The narrow bootstrap shape resolves that.

Wave incorporates community PRs #398 (@vinsew), #399 (@jdcastro2),
#402 (@schnubb-web).

Co-Authored-By: vinsew <yiyangchaishu@gmail.com>
Co-Authored-By: Julián David Castro <juliancastro@Mac-mini-de-Julian.local>
Co-Authored-By: schnubb-web <info@mia-mai.de>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore(test): bump beforeAll timeout on minions-shell-pglite for parallel-load flake

Default 5s beforeAll timeout occasionally trips under the parallel test runner
when many test files initialize PGLite concurrently. The same pattern is
documented as a P0 TODO for v0.21 Code Cathedral tests; this is the one
instance the upgrade-hardening wave directly exposed (CPU pressure from new
bootstrap test files).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v0.21.1)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: update project documentation for v0.21.1

- CLAUDE.md: PGLite + Postgres engine entries note new
  applyForwardReferenceBootstrap() in initSchema(), v24
  sqlFor.pglite no-op, and the new bootstrap test files
  (test/bootstrap.test.ts, test/schema-bootstrap-coverage.test.ts,
  test/e2e/postgres-bootstrap.test.ts).
- CHANGELOG.md: voice polish on the v0.21.1 headline
  (drop stray ## prefixes so the bold two-line headline
  renders as bold prose, not h2 sub-headers that break
  the version-entry hierarchy).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* chore: correct version slot from v0.22.5 to v0.21.6

Slot allocation correction. v0.21.6 is the actual landing slot for
this wave on the v0.21.x patch line.

VERSION, package.json, CHANGELOG.md (header + table + take-advantage
section), CLAUDE.md (engine entries, migrate.ts entry, test
descriptions) all updated together.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: correct version slot to v0.22.7

VERSION, package.json, CHANGELOG.md (header + table + take-advantage
section), CLAUDE.md (engine entries, migrate.ts entry, test descriptions)
all updated together.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: regenerate llms.txt + llms-full.txt for v0.22.7

CLAUDE.md changed (engine entries describe the bootstrap, migrate.ts entry
describes the v24 PGLite no-op). The build:llms regen-drift guard caught
the staleness in CI. Running `bun run build:llms` propagates the same
content into the AI-consumable bundles.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: change version slot from v0.22.7 to v0.22.6-hotfix.1

PR #483 (fix/mcp-registration-auth) claimed v0.22.7. Moved this wave to
v0.22.6-hotfix.1 to avoid the collision. Note: semver-orders BEFORE
0.22.6 (pre-release suffix), so the hotfix tag is informational, not
ordering-correct. Acceptable here because the wave's content predates
master's 0.22.6 and is being landed as a parallel hotfix slot.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: change version slot to v0.22.6.1

4-digit hotfix slot under master's v0.22.6. bun + bun:test accept
the format; the build-llms regen-drift guard and bootstrap tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: vinsew <yiyangchaishu@gmail.com>
Co-authored-by: Julián David Castro <juliancastro@Mac-mini-de-Julian.local>
Co-authored-by: schnubb-web <info@mia-mai.de>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-28 02:16:23 -07:00
Garry Tanandroot be8fffad71 fix: post-migration schema verification with self-healing (#488)
PgBouncer transaction-mode poolers can silently swallow ALTER TABLE
statements: the SQL doesn't error, but the column never gets created.
The migration system increments the schema version counter anyway, so
gbrain thinks it's on the latest version but the actual table is missing
columns. This caused production embed failures when the embed handler
tried to INSERT into columns that didn't exist.

Add verifySchema() that runs after all migrations complete:
1. Parses CREATE TABLE + ALTER TABLE ADD COLUMN from schema-embedded.ts
2. Queries information_schema.columns for actual DB state
3. Diffs expected vs actual columns
4. Self-heals missing columns via ALTER TABLE ADD COLUMN IF NOT EXISTS
5. Throws with actionable diagnostics if self-heal fails

Called from PostgresEngine.initSchema() after runMigrations().
PGLite skipped (in-process, no PgBouncer).

Co-authored-by: root <root@localhost>
2026-04-27 22:59:08 -07:00
131 changed files with 14467 additions and 366 deletions
+12 -1
View File
@@ -21,11 +21,22 @@ jobs:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
test:
# ubuntu-latest is free 2-core/7GB. Larger runners (16-cores, etc.) require
# a provisioned runner pool in repo settings. Falling back to default keeps
# the matrix shard speedup (~5-6x via parallelism) at zero cost.
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
shard: [1, 2, 3, 4]
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: latest
- run: bun install
- run: bun run test
- name: Pre-test gates (shard 1 only — they're not test files)
if: matrix.shard == 1
run: scripts/check-jsonb-pattern.sh && scripts/check-progress-to-stdout.sh && scripts/check-wasm-embedded.sh && bun run typecheck
- name: Run test shard ${{ matrix.shard }}/4
run: scripts/test-shard.sh ${{ matrix.shard }} 4
+6
View File
@@ -17,3 +17,9 @@ eval/data/world-v1/world.html
# BrainBench amara-life-v1 Opus cache (regenerate via eval:generate-amara-life)
eval/data/amara-life-v1/_cache/
# claw-test E2E build cache (shim + scratch outputs)
test/.cache/
.claude/
export/
+771
View File
@@ -2,6 +2,777 @@
All notable changes to GBrain will be documented in this file.
## [0.22.16] - 2026-04-29
**End-to-end claw-test friction harness — every release now gets a fresh-install dry-run.**
**`gbrain claw-test` spins up a hermetic tempdir, walks the canonical first-day flow, and surfaces friction the way a real new user would hit it.**
Before this release, every gbrain release shipped on faith: docs said "the agent runs `gbrain init`, then `gbrain import`, then `gbrain query`," and we'd find out at user-feedback time which step actually broke. Issue #239/#243/#266/#357/#366/#374/#375/#378/#395/#396 — ten upgrade-wedge incidents in two years — all came from this gap. There was no harness that exercised the user's-eye experience: spin up a fresh tempdir, install gbrain, watch what breaks.
Now there is. `gbrain claw-test --scenario fresh-install` in scripted mode is a CI gate (~30s, no API keys). `gbrain claw-test --live --agent openclaw` spawns a real openclaw subprocess, hands it `BRIEF.md`, captures every byte of its stdin/stdout/stderr to `transcript.jsonl`, and lets the agent log friction whenever something is confusing or wrong. End-of-run renders a markdown report grouped by severity and phase, with `<HOME>` redaction so it pastes safely into PRs.
The friction signal comes from a new `gbrain friction {log,render,list,summary}` CLI. Schema is a flat extension of `StructuredAgentError`. Run-id resolves from `--run-id` > `$GBRAIN_FRICTION_RUN_ID` > `standalone.jsonl`, so the same CLI works inside a harness session, manually during normal use, or from a scripted test. Append-only JSONL; readers tolerate malformed lines.
**$GBRAIN_HOME is finally honored everywhere it should be.** `configDir()` in `src/core/config.ts` always supported the parent-dir override, but ~12 consumers built paths from `os.homedir()` directly and bypassed it. Critically, `loadConfig`/`saveConfig` themselves used a private helper that ignored the env. Migrated every write site to a new `gbrainPath()` helper: fail-improve, validator-lint, cycle lock, audit handlers, sync-failures, integrity logs, integrations heartbeat, init pglite path, migrate-engine manifest, import checkpoint, migration rollbacks. Read-side host-detection (`~/.claude` / `~/.openclaw` probes for mod fingerprinting) intentionally stays as-is; v1.1 will add a separate `$GBRAIN_HOST_HOME`.
### Itemized changes
#### Added
- `gbrain claw-test --scenario {fresh-install|upgrade-from-v0.18}` — scripted-mode CI gate that runs the canonical first-day flow against a fresh tempdir. Asserts every expected `--progress-json` phase fired and doctor's `status === 'ok'`. ~30s, no API keys.
- `gbrain claw-test --live --agent openclaw` — friction-discovery mode. Spawns real openclaw, hands it `BRIEF.md`, captures stdin/stdout/stderr to `<run>/transcript.jsonl`, lets the agent log friction. ~510 min and ~$12 in tokens.
- `gbrain claw-test --list-agents` — reports which agent runners are registered + their detection state.
- `gbrain friction log --severity {confused|error|blocker|nit} --phase <name> --message <text> [--hint ...] [--kind {friction|delight}] [--run-id ...]` — append a friction or delight entry.
- `gbrain friction render --run-id <id> [--json] [--transcripts] [--no-redact]` — markdown report grouped by severity + phase; `--redact` defaults on for md output.
- `gbrain friction list [--json]` — recent run-ids with friction/delight counts; interrupted runs marked `(interrupted)`.
- `gbrain friction summary --run-id <id> [--json]` — two-column friction + delight summary.
- `skills/_friction-protocol.md` — cross-cutting convention skill telling agents when to call `gbrain friction log`. Routes from any skill the claw-test exercises.
- `gbrainPath(...segments)` helper in `src/core/config.ts` — single sugar for resolving paths under the active `$GBRAIN_HOME`. `$GBRAIN_HOME` is now validated (must be absolute, no `..` segments).
- Two scenario fixtures in `test/fixtures/claw-test-scenarios/`: `fresh-install` (canonical 5-min flow) and `upgrade-from-v0.18` (scaffolded; real v0.18 SQL dump documented as a v1.1 follow-up).
- New `src/core/claw-test/` module with `agent-runner.ts` (interface + registry), `transcript-capture.ts` (async-drain capture so 256KB+ bursts don't stall the child), `progress-tail.ts`, `scenarios.ts`, and `seed-pglite.ts` (~50 LOC PGLite SQL replay primitive).
#### Changed
- Every `~/.gbrain/...` write site now resolves through `gbrainPath()` instead of building paths from `os.homedir()`. Affected: `src/core/{fail-improve,output/post-write,cycle,sync}.ts`, `src/core/minions/{handlers/shell-audit,backpressure-audit}.ts`, `src/commands/{integrity,integrations,init,migrate-engine,import,migrations/v0_13_1,migrations/v0_14_0}.ts`. Tests that previously used the `process.env.HOME = tmpdir` workaround now use `process.env.GBRAIN_HOME` directly.
- `loadConfig`/`saveConfig` honor `$GBRAIN_HOME`. Previously, the public `configDir()` honored it but the internal `getConfigDir()` did not — so the config file itself silently leaked into the developer's real `~/.gbrain` regardless of the env override.
#### Tests
- 113 new unit tests covering: writer atomicity (concurrent appends), renderer redaction, agent registry resolution + selection precedence, multi-byte UTF-8 chunk-boundary safety, PIPE buffer drain under 256KB+ bursts, scenario load + validation, progress event parsing, SQL splitter (single-quote + line-comment handling), and full claw-test E2E (`test/e2e/claw-test.test.ts` builds a tiny `bun run src/cli.ts` shim and runs --scenario fresh-install end-to-end + a deliberate-break test that proves the friction signal fires).
- `test/gbrain-home-isolation.test.ts` is the regression gate: spawns `gbrain init --pglite` and `gbrain import --no-embed` with `GBRAIN_HOME=<tmp>`, asserts no writes outside `<tmp>/.gbrain` (covers `import.ts:54`, `sync.ts:317`, `upgrade.ts:117`, audit dirs).
## [0.22.15] - 2026-04-29
## **Throw bare markdown into your brain and it becomes properly typed knowledge. No YAML ceremony.**
A real 81K-page brain has 9,655 files with no frontmatter. They imported fine, but every one of them landed in the DB as `type: concept`, `title: <slugified-filename>`, no date, no source, no tags. Search ranking suffered. Type-filtered queries missed them. Entity resolution fell over.
This release adds path-aware frontmatter inference. `gbrain sync` now synthesizes type, date, source, and tags from the filesystem path and first heading the moment a bare-frontmatter file imports. No LLM call, fully deterministic, file on disk untouched. An Apple Note at `Apple Notes/2010-04-13 founders mtg.md` lands as `type: apple-note, title: founders mtg, date: 2010-04-13, source: apple-notes` instead of `type: concept, title: 2010 04 13 Founders Mtg`.
If you want the inference written back to git, the new `gbrain frontmatter generate <path> --fix` walks a brain dir, infers frontmatter for every file that lacks it, and writes back with `.bak` safety backups. Dry-run by default.
### The 9,655 numbers that matter
Measured against my actual brain (gbrain v0.22.8 + the new inference path).
| Behavior | Before v0.22.15 | After v0.22.15 |
|---|---|---|
| Files importing as `type: concept` (no frontmatter) | 9,655 | 0 |
| Apple Notes typed correctly (`apple-note`) | 0 | 5,861 |
| Calendar indexes typed correctly (`calendar-index`) | 0 | 3,201 |
| Therapy sessions typed + dated | 0 | 60 |
| Essay drafts typed + dated | 0 | 33 |
| LLM cost for the full reclassification | n/a | $0 |
The agent doing type-filtered queries on your brain (`type: person`, `type: meeting`, `type: essay`) now actually finds those pages instead of treating everything as `concept`.
### What this means for you
If you've been resisting frontmatter ceremony — same. Throw bare markdown into your brain and inference handles it. The rules table in `src/core/frontmatter-inference.ts` covers the obvious directories (`people/`, `companies/`, `daily/calendar/`, `writing/`, `meetings/`, `personal/`, etc.) plus a generic catch-all. Adding a new convention is one line in `DIRECTORY_RULES`.
## To take advantage of v0.22.15
`gbrain upgrade` should do this automatically. Then:
1. **Run a dry-run preview:**
```bash
gbrain frontmatter generate ~/brain
```
You'll see how many files would get inferred frontmatter and the breakdown by type.
2. **Optionally write back to git:**
```bash
gbrain frontmatter generate ~/brain --fix
```
Each modified file gets a `.bak` backup before rewrite.
3. **Re-sync to pick up the new metadata:**
```bash
gbrain sync ~/brain
```
Inferred frontmatter is folded into `content_hash`, so previously-bare files re-import once with proper types and re-embed. Subsequent syncs are idempotent.
4. **If anything looks off,** please file an issue: https://github.com/garrytan/gbrain/issues with the path of the misclassified file and the rule that matched.
### Itemized changes
#### Features
- `src/core/frontmatter-inference.ts` (new module) — Path-aware frontmatter synthesis. `DIRECTORY_RULES` table maps path prefixes to type/date/title/source/tags. First-match-wins. Date extraction from filenames (`YYYY-MM-DD` prefix or anywhere). Title extraction with date-prefix stripping and first-`#`-heading fallback (20-line window). YAML-safe serialization with quoting for special characters.
- `src/core/import-file.ts``importFromFile()` runs inference inline before `parseMarkdown()` when `opts.inferFrontmatter !== false` (default on). The synthesized frontmatter folds into the in-memory content for parsing, chunking, embedding, and content-hash computation. The file on disk is not modified.
- `src/commands/frontmatter.ts` — New `gbrain frontmatter generate <path> [--fix] [--dry-run] [--json]` subcommand. Walks a directory (skips `.git`, `node_modules`, `.obsidian`, symlinks), runs inference on every `.md` file without frontmatter, optionally writes back with `.bak` backups. Auto-detects brain root by walking up for `.git`. Shows per-type breakdown and first-10 examples.
#### Fixes
- `src/commands/frontmatter.ts:344``runGenerate` dynamic path import now includes `basename`. Single-file invocation (`gbrain frontmatter generate <file>`) previously crashed with `ReferenceError: basename is not defined` on the relative-path-empty fallback at line 437.
#### Tests
- `test/frontmatter-inference.test.ts` (new, 35 cases) — date extraction (5), title extraction from filenames (5) and headings (4 incl. 20-line boundary), inference for every directory rule (13 incl. Apple Notes subfolder tagging), serialization with YAML-safe quoting (4), `applyInference` integration (2), rule ordering and catch-all coverage (2).
## [0.22.14] - 2026-04-29
**Bare `gbrain jobs work` now self-monitors and fail-stops cleanly when its database dies or the queue stalls.**
**The wedged-worker class of bug — process alive, jobs piling up, your `pgrep` check happily green — is gone.**
A production brain (54K pages, Supabase Postgres, 3-concurrency worker under a cron-based PM)
hit it last week: worker process state=Sl at 13:15 UTC, stopped claiming jobs, 21 jobs stacked
in `waiting` over two hours, 5 autopilot-cycles dead-lettered at the 600s timeout, then 150
zombie processes accumulated over the container's 31-day life. The PM's `pgrep` saw a live
PID and reported green the entire time.
Pre-v0.22.14, bare `gbrain jobs work` had **zero** health monitoring. The supervisor (`gbrain
jobs supervisor`) had the right protections — DB liveness probes, stall detection, RSS
watchdog, reconnect on transient PgBouncer blips — but the supervisor wraps `jobs work` as a
child, and many production deployments run bare `jobs work` directly under systemd, Docker,
launchd, cron watchdog, or supervisord. That mode got nothing.
This release moves health monitoring into the bare worker itself, gated by `GBRAIN_SUPERVISED=1`
so it doesn't double up under the supervisor. When the worker detects it's wedged, it emits an
`'unhealthy'` event with a structured reason, and the CLI calls `process.exit(1)` so the external
PM restarts it cleanly. **This is fail-stop:** the worker exits and stays dead until your PM
brings it back. If you run bare `jobs work` without a restart loop, you need one now.
### The numbers that matter
Detection signatures the new health check catches, measured against the production incident
above (and the 30-day deployment running under the band-aid bash watchdog Garry deployed before
this fix):
| Failure mode | Before v0.22.14 | After v0.22.14 |
|---|---|---|
| DB connection death (Supabase/PgBouncer drop) | undetected; worker idles forever | 3 consecutive `SELECT 1` failures (≤3min) → `'unhealthy'`+exit |
| Hung DB probe (network partition) | timer wedged forever, monitoring silently disabled | 10s probe timeout per tick → counted as failure → exit at strike 3 |
| Worker stall (event loop alive, claim returns null) | undetected; jobs pile up in `waiting` | 5min warn, 10min `'unhealthy'`+exit (measured from last completion) |
| Memory leak (RSS climbing past 2GB) | undetected on bare workers | watchdog default 2048 MB triggers `gracefulShutdown('watchdog')` |
| Worker stalled but waiting jobs are unhandled type | ❌ false-positive exit (restart loop) | filter by registered handler names, no exit |
Operationally: from the band-aid bash watchdog Garry deployed before this fix, fresh worker
restart cleared 21 waiting → 0 in 2 minutes, then ran stable for 30+ min with 130 MB RSS,
autopilot-cycles completing in 0.20.6s instead of timing out at 600s.
### What this means for operators
Add a restart policy to your bare-worker invocation BEFORE upgrading. The new behavior is
fail-stop, not self-healing — without a restart loop, your worker will exit on the first DB
blip and stay dead. systemd `Restart=always`, Docker `restart: always`, launchd `KeepAlive`,
cron watchdog, supervisord `autorestart=true`. The migration walks every PM. If you're using
`gbrain jobs supervisor`, you're already protected — the supervisor handles spawn-on-crash
itself.
The default `--max-rss` for bare workers also bumped from 0 (off) to 2048 MB. If you ran bare
workers with intentionally large embed/import jobs, raise the limit (`--max-rss 4096`) or opt
out (`--max-rss 0`). The migration includes per-PM unit-file edits.
## To take advantage of v0.22.14
`gbrain upgrade` should do this automatically. If it didn't, or if `gbrain doctor` warns about
a bare worker exiting with watchdog signatures:
1. **Confirm your bare-worker invocations have a restart policy:**
```bash
# systemd
grep -E '^Restart=' ~/.config/systemd/user/gbrain-worker.service /etc/systemd/system/gbrain-worker.service 2>/dev/null
# crontab
crontab -l | grep "gbrain jobs work"
# launchctl
plutil -p ~/Library/LaunchAgents/com.user.gbrain-worker.plist | grep -A1 KeepAlive
```
2. **Decide on RSS posture:**
- Default 2048 MB matches supervisor behavior. Most bare workers fit.
- Embed/import jobs > 2GB? Pass `--max-rss 4096` (or higher).
- Intentionally unbounded? Pass `--max-rss 0`.
3. **Walk the migration:** `skills/migrations/v0.22.14.md` has the full per-PM table and a
verification block.
4. **Verify:**
```bash
gbrain jobs stats
gbrain doctor --json | jq '.checks[] | select(.name == "queue_health")'
```
Worker startup line should now read:
`Minion worker started (queue: default, concurrency: 3, watchdog: 2048MB, health-check: 60s)`
Under supervisor: the `health-check: Ns` segment is absent (supervisor handles it).
5. **If anything fails or numbers look wrong**, file an issue at
https://github.com/garrytan/gbrain/issues with `gbrain doctor` output and the contents of
`~/.gbrain/upgrade-errors.jsonl` if it exists.
### Itemized changes
#### Added
- `MinionWorkerOpts.{healthCheckInterval, stallWarnAfterMs, stallExitAfterMs, dbFailExitAfter, dbProbeTimeoutMs}` — five new tuning knobs. Defaults: 60s probe interval, 5min warn / 10min exit, 3 DB strikes, 10s per-probe timeout.
- `MinionWorker` now extends `EventEmitter`. Emits `'unhealthy'` with `{ reason: 'db_dead', consecutiveFailures, message } | { reason: 'stalled', waitingCount, idleMinutes }`. CLI subscribes; direct API consumers without a listener inherit a fail-stop fallback that calls `process.exit(1)` to preserve pre-refactor semantics.
- `gbrain jobs work --health-interval MS` — tune the self-health-check cadence (0 disables; rejects NaN/negative/sub-1000ms typos).
- `gbrain jobs supervisor --health-interval MS` — same flag, same validation, same `0 = disable` contract on the supervisor's own probe.
- `GBRAIN_SUPERVISED=1` env var on the supervisor's spawned worker child (skips the child's self-health timer to avoid double-monitoring).
- `gbrain doctor` `queue_health` subcheck reports RSS-watchdog kills in the last 24h via exact match on `error_text = 'aborted: watchdog'` scoped to `status IN ('dead','failed')`.
- `skills/migrations/v0.22.14.md` — full migration walkthrough with per-PM restart-policy preflight, RSS-posture decision tree, and per-system unit-file edits.
#### Changed
- **Default `--max-rss` for `gbrain jobs work`: 0 → 2048 MB.** Matches supervisor default. Catches memory-leak stalls that previously went undetected on bare workers. Opt out with `--max-rss 0`.
- **Bare-worker behavior is now fail-stop** when the DB is unreachable or the queue stalls. Pre-v0.22.14 the worker idled silently. Now it exits and relies on the external PM (systemd, Docker, launchd, cron, supervisord) to restart cleanly.
- Stall query at `worker.ts` filters by registered handler names (`AND name = ANY($2::text[])`) so workers don't false-positive when waiting jobs of unhandled names accumulate.
- Stall exit threshold measured from `lastCompletionTime` (not from when the warning fired), so 5min warn / 10min exit means total idle of 10 min — not 15 min.
- DB liveness probe wrapped in `Promise.race` against a 10s timeout so a hung `executeRaw` cannot wedge the recursive `setTimeout` chain forever.
- `setInterval` → recursive `setTimeout` with a `running` flag throughout. Eliminates timer-callback overlap on slow probes.
- `parseMaxRssFlag` returns `number | undefined` (was `number`) so callers distinguish absent from explicit-disable.
- `process.env.GBRAIN_SUPERVISED` check tightened from `!!env.X` to `=== '1'` (precise contract; no fuzzy matching on `'0'` or `'false'`).
- `MinionWorker` constructor throws when `stallExitAfterMs <= stallWarnAfterMs` so misconfigurations fail loudly at startup.
#### Fixed
- **Wedged-worker false-positive on heterogeneous queues** — workers registering only some handlers no longer interpret waiting jobs of other names as a stall. Repeated `process.exit(1)` → restart loop is gone.
- **Hung DB probe wedge** — pre-fix, a hung `executeRaw('SELECT 1')` kept the recursive `setTimeout` from rescheduling, silently disabling the entire health monitor. Post-fix, the probe times out and counts as a failure.
- **`--health-interval 0` no longer DB-hammers the supervisor.** Pre-fix, the documented "0 disables" contract was a lie — `setInterval(cb, 0)` schedules a tight loop. Now gated behind `> 0`.
- **Inline `jobs submit --follow` and `jobs smoke` no longer kill the user's CLI session** on a DB blip. Both now pass `healthCheckInterval: 0` so the no-listener fallback can't trip on one-shot runs.
- Doctor's RSS-watchdog hint matches the actual error_text signature (`'aborted: watchdog'`) instead of the wrong `'memory limit'` literal that never matched.
#### For contributors
- `MinionWorker extends EventEmitter` — if you import the class directly, the `on('unhealthy', ...)` event is now part of the public surface. The `UnhealthyReason` discriminated union is exported from `src/core/minions/worker.ts`.
- New regression-test infrastructure in `test/minions.test.ts`: `makeProbeEngine(overrides)` is a Proxy-based engine wrapper that intercepts `SELECT 1` and the stall `count(*)` query while passing every other call through to the real PGLite engine. Useful for any future test that needs to inject DB liveness or stall semantics without mocking the entire engine surface.
### Adjacent (separate PR, v0.22.15)
PR #503 catches the *symptom* of one specific failure mode. The cause-side fix — `runPhaseEmbed → embed.ts → embedBatch` not honoring `signal.aborted` between OpenAI batch calls — ships in v0.22.15 (highest-priority TODO; daily wedge driver). Plumbing is documented in `TODOS.md`.
## [0.22.13] - 2026-04-28
**Sync got faster, and the bookmark stopped lying.**
**Parallel imports, a real writer lock, and a head-drift gate that catches the worst race.**
The headline is `gbrain sync --workers N`: per-worker Postgres engines with an atomic queue index, same pattern as `gbrain import --workers N`. On a 7,000-page brain that used to take 25+ minutes, the import phase now runs across 4 workers by default. The reproducible benchmark in `test/e2e/sync-parallel.test.ts` shows `parallel(4)` finishing 1.3× faster than serial on a 120-file fixture against local Postgres (`serial=289ms parallel(4)=221ms`). The speedup grows on larger brains and slower-roundtrip databases (Supabase, remote PgBouncer) because the worker setup cost amortizes over more files. But the bigger story is that the sync writer is finally exclusive across processes, and the `last_commit` bookmark refuses to advance when git HEAD has drifted out from under us. The silent-skip-then-advance pathology has survived every prior sync hardening pass. It is dead now.
### What you can do now
- `gbrain sync --workers 4` (alias `--concurrency 4`) parallelizes the import phase. Each worker holds 2 connections, so total Postgres connections during the parallel phase is `workers * 2` plus your caller's pool. At the default of 4 workers and a 10-connection caller pool, that's up to 18 connections, well under PgBouncer's `max_client_conn` default of 100 but worth knowing on tight Supabase tiers.
- **Auto-concurrency:** if you don't pass `--workers`, sync uses 4 workers when the diff exceeds 100 files. Smaller diffs stay serial. Explicit `--workers` always wins (even on a 30-file diff). PGLite forces serial regardless, since it's a single-connection engine.
- **Full sync** routes through the same path. First syncs on large brains parallelize automatically.
- **Minion `sync` jobs** also use the new `autoConcurrency()` policy. Behavior is now consistent between CLI sync, the Minion handler, and the autopilot cycle's sync phase. (`noEmbed` defaults to `true` in the jobs handler. Submit `gbrain embed --stale` as a separate job when needed, or rely on the autopilot cycle's embed phase.)
- **`--workers` validation is loud now.** `--workers 0`, `--workers -3`, `--workers foo`, `--workers 1.5` all exit with an error message. The prior behavior silently fell through to auto-concurrency (4 workers), the opposite of what you typed.
### Correctness fixes you didn't have to ask for
- **Cross-process writer lock.** Two `gbrain sync` calls (manual + autopilot, two terminals, two Conductor workspaces) used to read the same `last_commit`, both write it, and let the last writer win. The new `gbrain-sync` row in `gbrain_cycle_locks` serializes the writer window. Same-process reentrance from the autopilot cycle handler was already covered by the broader `gbrain-cycle` lock; sync's lock is narrower and runs underneath it.
- **Head-drift gate.** If `git checkout` or `git pull` runs in your worktree mid-sync (Conductor sibling workspace, ad-hoc terminal), the captured `headCommit` no longer matches HEAD when sync finishes. `last_commit` no longer advances in that case. The next sync re-walks the diff against the new HEAD instead of silently moving the bookmark past unimported work.
- **Vanished files now block bookmark advance.** A file the diff said exists at `headCommit` but is gone from disk used to register as a benign skip. It now goes into `failedFiles` and gates `last_commit` the same way a parse failure does.
- **Per-source bookmark for Minion `sync` jobs.** The job handler now resolves `sourceId` from the repo path (mirrors the autopilot cycle's `cycle.ts` fix from PR #475). On multi-source brains, this prevents the 30-min full-reimport-every-cycle behavior caused by reading the global `config.sync.last_commit` anchor when the per-source row would have been correct.
- **Worker connection cleanup.** Worker engines now disconnect inside `try/finally`, even on partial connect failure or mid-import error. The prior `Promise.all(...disconnect)` ran outside any try/finally, so panic-path leaks never released the 8 worker connections.
- **Engine detection unified.** Both PGLite-detection sites in sync.ts now use `engine.kind === 'pglite'` (the discriminator added in v0.13.1). The `engine.constructor.name === 'PGLiteEngine'` sniff is gone, since it broke under bundling and was inconsistent with the other site's `config.engine` string check.
### What this means for you
If you run autopilot on a 7,000-page Postgres brain, your sync cycle gets faster on day one with no flags. If you have ever felt the bookmark "skip past" work that didn't import, you'll stop seeing it. If you have multiple Conductor workspaces poking the same brain, you'll either wait politely on the writer lock or get a clear "another sync is in progress" error. None of this requires a config change.
## To take advantage of v0.22.13
`gbrain upgrade` should do this automatically. If you want to use the new flags right now:
1. **For a one-off speed win on a large brain:**
```bash
gbrain sync --workers 4
```
Or for incremental syncs that touch >100 files, just run `gbrain sync`. Auto-concurrency fires.
2. **For your autopilot cycle:** no action. The Minion `sync` handler picks up the new auto-concurrency policy automatically.
3. **Verify the writer lock is working:**
```bash
gbrain sync &
gbrain sync # second call will say "Another sync is in progress" or wait
```
4. **If sync ever errors with "Another sync is in progress" and stays stuck:** the lock is in `gbrain_cycle_locks` with id `gbrain-sync` and a 30-minute TTL. If a worker crashed without releasing, the next acquirer takes over once the TTL expires. To unstick faster:
```sql
DELETE FROM gbrain_cycle_locks WHERE id = 'gbrain-sync';
```
5. **If anything looks wrong,** file an issue: https://github.com/garrytan/gbrain/issues with output of `gbrain doctor` and the contents of `~/.gbrain/upgrade-errors.jsonl` if it exists.
### Itemized changes
- `src/commands/sync.ts`: `performSync` now wraps body in a `gbrain-sync` DB lock; `--workers` honored regardless of file count when explicit; head-drift gate after import phase; engine.kind detection; try/finally around worker engines; banner moved to stderr.
- `src/commands/import.ts`: `engine.kind === 'pglite'` discriminator; try/finally around worker engines; shared `parseWorkers()` for `--workers` validation.
- `src/commands/jobs.ts`: sync handler resolves `sourceId` via `sources.local_path` lookup; concurrency routed through `autoConcurrency()`; `noEmbed: true` default documented.
- `src/core/sync-concurrency.ts` (new): `autoConcurrency()` + `parseWorkers()` + constants. One source of truth for the concurrency policy that previously lived in three call sites.
- `src/core/db-lock.ts` (new): generic `tryAcquireDbLock(engine, lockId)` over the existing `gbrain_cycle_locks` table. Reused by performSync. cycle.ts continues to use its own ID `gbrain-cycle` so the two locks nest cleanly.
- `test/sync-concurrency.test.ts` (new): 17 cases covering autoConcurrency thresholds, shouldRunParallel gates, parseWorkers validation.
- `test/sync-parallel.test.ts` (new): PGLite-routed coverage of the bookmark gate under concurrency request, the head-drift gate, the writer-lock contract, and PGLite-stays-serial.
- `test/e2e/sync-parallel.test.ts` (new): DATABASE_URL-gated Postgres E2E. 60-file happy path with `pg_stat_activity` leak probe, plus a 120-file serial-vs-parallel benchmark that prints `SYNC_PARALLEL_BENCH ...` for CHANGELOG quoting.
### For contributors
- `BrainEngine.kind` is now the canonical PGLite/Postgres discriminator. Avoid `engine.constructor.name === '...'` (breaks under bundling) and `config.engine === '...'` (inconsistent with the engine actually in use).
- The `gbrain_cycle_locks` table is now multi-purpose. The id column distinguishes lock scopes: `gbrain-cycle` for the cycle, `gbrain-sync` for the sync writer. Future locks should pick distinct ids and reuse `tryAcquireDbLock`.
- `parseWorkers()` is the canonical CLI flag parser for `--workers`. Use it instead of inline `parseInt`.
## [0.22.12] - 2026-04-29
**`sync --skip-failed` now classifies file-size and symlink rejections instead of bucketing them as UNKNOWN.**
**Plus a full end-to-end test for the failure loop.**
v0.22.9 shipped the headline classifier work: code-grouped breakdowns at sync time,
DB-vs-YAML disambiguation, doctor surfaces both unacked and historical entries with
`[CODE=N]` lines. v0.22.12 closes the last two coverage gaps that v0.22.9 left on
the table:
- **FILE_TOO_LARGE** now covers the three real production sites in
`src/core/import-file.ts:199, 352, 401` ("Content too large", "File too large",
"Code file too large"). On v0.22.9 these all bucketed as UNKNOWN — the same
silent-systemic-failure pattern that motivated the original issue.
- **SYMLINK_NOT_ALLOWED** covers `src/core/import-file.ts:347` ("Skipping symlink").
Security-relevant rejection that operators should see.
- **End-to-end failure-loop test** in `test/e2e/sync.test.ts` exercises the full
chain: broken file → sync blocks with grouped breakdown → `--skip-failed`
advances bookmark with grouped acknowledgement → second broken file → second
cycle. PostgreSQL-backed; verifies bookmark gating, JSONL state, dedup, and
summary aggregation. v0.22.9's coverage was unit-tests-only.
Twelve total error codes ship in the classifier:
`SLUG_MISMATCH`, `YAML_PARSE`, `YAML_DUPLICATE_KEY`, `DB_DUPLICATE_KEY`,
`MISSING_OPEN`, `MISSING_CLOSE`, `NESTED_QUOTES`, `EMPTY_FRONTMATTER`,
`NULL_BYTES`, `INVALID_UTF8`, `STATEMENT_TIMEOUT`, `FILE_TOO_LARGE`,
`SYMLINK_NOT_ALLOWED`. Anything the regex set doesn't recognize falls through
as `UNKNOWN`.
### What this means for you
If your brain rejects oversized files or symlinks, you now see those rejections
in the doctor breakdown and at sync time grouped by code, instead of as
`UNKNOWN`. Run `gbrain upgrade`. No manual action required.
### Itemized changes
#### Added
- `FILE_TOO_LARGE` classifier code covering `src/core/import-file.ts:199, 352, 401`.
- `SYMLINK_NOT_ALLOWED` classifier code covering `src/core/import-file.ts:347`.
- Two new unit tests in `test/sync-failures.test.ts` pinning the new codes against
literal production message strings (`File too large (N bytes)`, `Skipping symlink: ...`).
- `test/e2e/sync.test.ts` — new failure-loop test exercising broken-file → block →
`--skip-failed` → second cycle. Hermetic on developer machines (saves+restores
the user's real `~/.gbrain/sync-failures.jsonl`).
## To take advantage of v0.22.12
No manual action required. Run `gbrain upgrade`. The new `FILE_TOO_LARGE` and
`SYMLINK_NOT_ALLOWED` classifier codes apply on the next `gbrain sync`.
## [0.22.11] - 2026-04-27
**Storage tiering, finally working. Brains scaling past 100K files stop bloating git.**
The original storage-tiering branch shipped two silent bugs (gray-matter on YAML returned empty data; `manageGitignore` was defined and never invoked) so the feature was a no-op for every user who tried it. v0.22.11 rewrites the broken bits, hardens the surface, and adds proper test coverage. If you have a brain repo north of 100K files where bulk machine-generated content (tweets, articles, transcripts) is the size driver, this is the release that pulls it out of git without losing any data.
Configure tiering in `gbrain.yml` at the brain repo root:
```yaml
storage:
db_tracked:
- people/
- companies/
- deals/
db_only:
- media/x/
- media/articles/
- meetings/transcripts/
```
`gbrain sync` then auto-manages your `.gitignore` for `db_only` directories so bulk content stops landing in commits. `gbrain export --restore-only` repopulates missing `db_only` files from the database (container restart, fresh clone, accidental rm). `gbrain storage status` shows the breakdown — counts, disk usage, missing files.
### The numbers that matter
200K-page brain, half tweets and articles. Before v0.22.11:
| Metric | Before | After | Δ |
|--------|--------|-------|---|
| `gbrain.yml` actually loads | no (silent null) | yes | feature works |
| `.gitignore` auto-manages | no (function never called) | yes | docs match reality |
| `--restore-only` without `--repo` | silent full export | hard error | no data-loss footgun |
| `media/xerox` matched against `media/x` | yes (collision) | no | path-segment matching |
| Per-page disk syscalls during status | ~400K (existsSync + statSync) | ~one per dir + one stat per .md | single-walk scan |
| Validation surfaces overlap | warning only | throws StorageConfigError | semantic error caught |
### What this means for your brain
If you've been reading the storage-tiering docs and waiting for the feature to actually do something: it does now. If you're already over 50K files: configure `gbrain.yml`, run `gbrain sync`, watch `.gitignore` update itself, watch your next clone get faster.
## To take advantage of v0.22.11
1. Add a `storage:` section to `gbrain.yml` at your brain repo root with `db_tracked` and `db_only` arrays. The directory paths must end with `/` (the validator auto-normalizes if you forget, with a one-time info note).
2. Run `gbrain sync`. It updates `.gitignore` automatically on success.
3. Run `gbrain storage status` to see the tier breakdown and any missing `db_only` files.
4. If files are missing on disk (e.g., after a container restart): `gbrain export --restore-only --repo /path/to/brain`.
5. If you previously had `git_tracked` / `supabase_only` keys: they still load, with a once-per-process deprecation warning. Rename to `db_tracked` / `db_only` at your convenience.
6. On PGLite: tiering has limited effect (the "DB" is your local file). The `.gitignore` housekeeping still helps. A one-time soft-warn explains.
If anything looks off, file an issue at <https://github.com/garrytan/gbrain/issues> with `gbrain doctor` output and the contents of your `gbrain.yml`.
### Itemized changes
#### Critical fixes
- **YAML parser swap**: replaced `gray-matter` with a dedicated YAML reader for the `gbrain.yml` shape. The original code called `matter()` on a delimiter-less file, which always returned `{data: {}}``loadStorageConfig` returned null on every install. The dedicated parser handles top-level `storage:` plus nested array-valued keys, with comment + blank-line tolerance. Once-per-process sanity warning when `gbrain.yml` exists but has no `storage:` section.
- **`manageGitignore` actually runs now**: wired into `runSync` after every successful sync (skipped on dry-run, blocked-by-failures, and unhandled errors). Idempotent. Detects git submodule context (`.git` is a file, not a directory) and skips with an actionable warning. Honors `GBRAIN_NO_GITIGNORE=1` for shared-repo setups.
- **No more silent `--restore-only` footgun**: `gbrain export --restore-only` without `--repo` now resolves through a typed `getDefaultSourcePath()` accessor (sources table → null → hard error). Never falls through to the current directory. Never silently re-exports your entire database into the wrong place.
#### New + renamed surface
- **Canonical key names**: `db_tracked` / `db_only` replace the vendor-baked `git_tracked` / `supabase_only`. The deprecated keys still load, with a once-per-process warning suggesting `gbrain doctor --fix` for an automated rename. Canonical wins when both shapes coexist.
- **Engine-side `slugPrefix` filter**: `PageFilters.slugPrefix` lands on both engines as `WHERE slug LIKE prefix || '%'` with literal-escape of LIKE metacharacters. Uses the existing `(source_id, slug)` UNIQUE btree index for range scans. Powers `gbrain export --restore-only` per-tier queries and `gbrain export --slug-prefix`.
- **Single-walk filesystem scan**: `src/core/disk-walk.ts` exposes `walkBrainRepo(repoPath)` that returns `Map<slug, {size, mtimeMs}>` from one recursive `readdirSync`. Replaces the per-page `existsSync + statSync` loop in `gbrain storage status` (~400K syscalls on a 200K-page brain → tens).
- **Path-segment matching**: tier directory matcher requires trailing `/` and treats the slash as a path separator. `media/x/` does not match `media/xerox/foo`. Validator (`normalizeAndValidateStorageConfig`) auto-fixes missing trailing `/`, throws `StorageConfigError` on tier overlap.
#### Architecture cleanup
- `src/commands/storage.ts` split into pure data + JSON formatter + human formatter + thin dispatcher, matching the `orphans.ts` precedent. `getStorageStatus` is exported for `gbrain doctor` integration. ASCII-only output (no unicode box-drawing) for cross-platform terminal compatibility.
- Distinct nominal types `PageCountsByTier` and `DiskUsageByTier` so accidental swaps between page counts and byte totals are compile-time errors.
- PGLite soft-warn on storage tiering (D4): the feature is partial on PGLite (the "DB" is your local file), but `.gitignore` housekeeping still helps. Once-per-process warning explains and proceeds.
#### Tests + CI guards
- New unit tests across `test/storage-config.test.ts`, `test/storage-sync.test.ts`, `test/storage-status.test.ts`, `test/storage-export.test.ts`, `test/storage-pglite.test.ts`, `test/disk-walk.test.ts`. Plus extensions to `test/source-resolver.test.ts` and `test/pglite-engine.test.ts`. The single-line test that would have caught the original gray-matter P0 (write a real `gbrain.yml`, call `loadStorageConfig`, assert non-null) now exists.
- New CI guard `scripts/check-trailing-newline.sh` (sibling to the existing jsonb-pattern + progress-to-stdout guards). Wired into `bun run test`. Fixed pre-existing missing newline in `docs/storage-tiering.md`.
### For contributors
- The eng-review path forward is documented in `~/.claude/plans/lets-take-a-look-ticklish-pizza.md` (15 numbered defects + D1-D8 abstraction calls). Every commit on this branch maps to one numbered step in the plan.
## [0.22.10] - 2026-04-30
**`gbrain jobs submit autopilot-cycle --params '{"phases":["lint","backlinks"]}'` now actually runs only those phases.**
If you ever submitted an `autopilot-cycle` job with a `phases:` array hoping to skip embed for a fast cycle, you got the full 6-phase cycle anyway. The handler in `src/commands/jobs.ts` was calling `runCycle(...)` without forwarding `job.data.phases`, so per-cycle phase selection was silently ignored.
This release wires the array through. The handler imports `ALL_PHASES` from `src/core/cycle.ts`, builds a `Set` for O(1) validation, and filters the caller's `phases` array against it before forwarding to `runCycle`. Invalid phase names get dropped (no injection surface — `ALL_PHASES` is the authoritative list). Empty arrays and non-array values fall back to the default (run all phases), preserving the prior behavior for callers who didn't ask for selective phases.
### What this means for you
If you've been using `gbrain jobs submit autopilot-cycle --params '{"phases":[...]}'` for triage cycles (e.g. `["lint","backlinks"]` for a fast structural sweep, skipping the slow embed phase), you'll now see those cycles take seconds instead of minutes. The CLI surface didn't change — only the worker's handler now respects the `phases` it was already accepting.
### Itemized changes
#### Fixed
- `autopilot-cycle` minion handler in `src/commands/jobs.ts` now forwards `job.data.phases` to `runCycle()`. Previously the handler accepted the array via `MinionJobInput.params` but discarded it before dispatch.
- Phase names validated against `ALL_PHASES` from `src/core/cycle.ts`. Filter is exhaustive: array → filtered, non-array → undefined (default), filtered-to-empty → no `phases` key in opts (also default).
#### Tests
- 4 new test cases in `test/handlers.test.ts` under `autopilot-cycle handler — phase passthrough`: valid phases forwarded, invalid names filtered, empty array falls back to all-phases, non-array `phases` value ignored. Pin both the contract and the fallback semantics.
- `test/cycle-abort.test.ts` regression-guard window widened from 500 → 2000 chars so the source-level `signal: job.signal` check finds the line after the new validation block was added between `worker.register('autopilot-cycle', ...)` and the `runCycle(...)` call. Pure test fix; the handler still propagates the abort signal correctly.
## [0.22.9] - 2026-04-29
**Sync failures now tell you why, not just how many.**
**`gbrain sync --skip-failed` and `gbrain doctor` group failures by error code, so 2,685 silent SLUG_MISMATCH files don't hide behind a single count.**
Before this release, when sync hit per-file parse errors the only signal was a number:
```
Sync blocked: 2688 file(s) failed to parse. Fix the YAML frontmatter...
```
That count is useless when you're staring at 2,688 files and don't know what's wrong. On a real 81K-page brain, 2,685 of those turned out to be `SLUG_MISMATCH` from a posterous import — a single root cause hiding behind a giant number. It took manual `cat ~/.gbrain/sync-failures.jsonl | jq` to figure that out.
After:
```
Sync blocked: 2688 file(s) failed to parse:
SLUG_MISMATCH: 2685
YAML_DUPLICATE_KEY: 3
Fix the YAML frontmatter in the files above and re-run, or use 'gbrain sync --skip-failed' to acknowledge and move on.
# gbrain sync --skip-failed
Acknowledged 2688 failure(s) and advancing past them:
SLUG_MISMATCH: 2685
YAML_DUPLICATE_KEY: 3
```
`gbrain doctor` shows the same breakdown for unacknowledged AND historical entries:
```
[WARN] sync_failures: 2688 unacknowledged sync failure(s) [SLUG_MISMATCH=2685, YAML_DUPLICATE_KEY=3].
[OK] sync_failures: 500544 historical sync failure(s), all acknowledged [SLUG_MISMATCH=2685, ...].
```
The classifier knows the canonical messages from `collectValidationErrors()` in `src/core/markdown.ts` (8 frontmatter codes), Postgres unique-constraint violations (`DB_DUPLICATE_KEY`), statement-timeout errors (`STATEMENT_TIMEOUT`), invalid UTF-8, and YAML duplicates. DB-layer errors check before YAML-layer ones — so a Postgres `duplicate key value violates unique constraint` no longer mislabels as a YAML duplicate. Unrecognized errors fall through to `UNKNOWN`.
### What this means for you
If `gbrain sync` blocks with parse failures, the breakdown tells you what to fix first. SLUG_MISMATCH is one fix-pattern (frontmatter says one slug, path says another); YAML_PARSE is a different one (malformed YAML); STATEMENT_TIMEOUT means a DB timeout, not a parse problem. You stop staring at counts and start fixing root causes.
### For contributors
`acknowledgeSyncFailures()` in `src/core/sync.ts` now returns `{count, summary}` instead of `number`. If you import this directly from `gbrain/sync`, replace `n` with `result.count` and use `result.summary` (an `Array<{code, count}>`) for the new code-grouped breakdown. The function is reachable via the package exports map; this is a deliberate, non-shimmed breaking change. There is a new `formatCodeBreakdown()` helper in the same module that accepts either raw failures or pre-summarized input — use it instead of building breakdown strings inline.
### Itemized changes
#### Added
- `classifyErrorCode(errorMsg)` in `src/core/sync.ts` — best-effort error-code extraction from sync failure messages. Codes: `SLUG_MISMATCH`, `YAML_PARSE`, `YAML_DUPLICATE_KEY`, `MISSING_OPEN`, `MISSING_CLOSE`, `EMPTY_FRONTMATTER`, `NULL_BYTES`, `NESTED_QUOTES`, `DB_DUPLICATE_KEY`, `STATEMENT_TIMEOUT`, `INVALID_UTF8`, `UNKNOWN`.
- `summarizeFailuresByCode(failures)` — groups failures by code and returns a sorted `Array<{code, count}>`.
- `formatCodeBreakdown(input)` — renders a multi-line `code: count` string from either raw failures or a pre-computed summary. Single helper, two input shapes.
- `code?: string` field on the `SyncFailure` JSONL row in `~/.gbrain/sync-failures.jsonl`. Populated at write-time so the classifier runs once per failure, not on every load.
- `AcknowledgeResult` interface as the new return shape of `acknowledgeSyncFailures()`.
- 15 new test cases in `test/sync-failures.test.ts`: DB-vs-YAML duplicate-key disambiguation, canonical-message coverage for all 7 frontmatter codes, `acknowledgeSyncFailures()` legacy-entry backfill branch, `formatCodeBreakdown()` dual-input shape.
#### Changed
- `gbrain sync` blocked-message: now lists code breakdown above the fix instructions (both incremental and full-sync paths).
- `gbrain sync --skip-failed` ack message: now lists what was skipped, grouped by code.
- `gbrain doctor` `sync_failures` check: warn-and-ok messages both include `[code=count, ...]` breakdown.
- `recordSyncFailures()` now stores `code` alongside `error` so downstream readers don't re-classify.
- `acknowledgeSyncFailures()` backfills `code` on legacy rows that predate the field — upgrade-safe for users with existing `~/.gbrain/sync-failures.jsonl`.
- DB-layer error patterns (`DB_DUPLICATE_KEY`, `STATEMENT_TIMEOUT`) check BEFORE YAML patterns in the classifier, so Postgres errors don't get YAML-labeled.
- Frontmatter regex patterns rewritten to match canonical messages from `collectValidationErrors()` (`File is empty...`, `No closing --- delimiter found`, `Frontmatter block is empty`) instead of aspirational code-token strings (`missing.*open`) that never appeared in practice.
Closes #500.
## [0.22.8] - 2026-04-28
## **Doctor stops timing out on Supabase. Integrity scan finishes in ~6s, multi-source brains get correct counts.**
If you've been hitting the 60-second `gbrain doctor` timeout on Supabase or any pooled-connection deployment, this fixes it. The integrity check used to call `getPage()` 500 times sequentially through PgBouncer transaction-mode pooling. Each call required a full connection acquire/release cycle, which doctor couldn't finish before CI killed it. The new path batch-loads all 500 pages in a single SQL query, finishing in ~6s.
While shipping the perf fix, codex review caught a correctness regression for multi-source brains: the batch SQL was scanning raw `(source_id, slug)` rows while the sequential path scanned unique slugs. Multi-source brains were getting inflated counts. `SELECT DISTINCT ON (slug)` mirrors the sequential path's `Set<string>` semantics; parity tests against real Postgres pin both paths to the same output.
Plus a Linux CI fix: `gbrain skillpack` lockfile checks were intermittently failing on ext4's sub-millisecond `mtimeMs` timestamps when `Date.now()` returned an integer ms behind the file's recorded mtime. Lock age now clamps to zero.
### The numbers that matter
Measured against the real failure mode on a Supabase PgBouncer deployment that hit the 60s CI timeout pre-fix.
| Behavior | Before v0.22.8 | After v0.22.8 |
|---|---|---|
| `gbrain doctor` wall-clock (Postgres + PgBouncer) | 60s+ timeout (killed) | ~6s |
| `integrity_sample` query round-trips | ~500 (sequential `getPage`) | 1 (`SELECT DISTINCT ON`) |
| Multi-source brain scan accuracy | Overcounted by `source_id` | Exact per unique slug |
### What this means for Supabase deployments
If you've been avoiding `gbrain doctor` because it timed out, run it again. If you maintain a multi-source brain (imported pages from another gbrain deployment under a non-default `source_id`), the scan now treats each slug once instead of once-per-source — your output is exact, not inflated. Single-source users see no behavior change; PGLite users were never affected (the batch path is Postgres-only).
## To take advantage of v0.22.8
`gbrain upgrade` should do this automatically. If it didn't, or if `gbrain doctor` warns about anything afterwards:
1. **Run the upgrade:**
```bash
gbrain upgrade
```
2. **Verify doctor finishes cleanly (especially relevant if you hit timeouts before):**
```bash
gbrain doctor
```
On Postgres + PgBouncer deployments, you should see `integrity_sample` finish in ~6s instead of timing out at 60s.
3. **If `doctor` still times out or output looks wrong,** please file an issue:
https://github.com/garrytan/gbrain/issues with:
- output of `gbrain doctor` (full)
- which engine (Postgres vs PGLite)
- whether you use multi-source brains
### Itemized changes
#### Performance
- `gbrain doctor` integrity sample now batch-loads via a single SQL query on Postgres deployments (60s+ timeout → ~6s wall-clock, 500 round-trips → 1).
- Batch path explicitly gated to Postgres via `engine.kind` so PGLite never attempts it (clean fallback signal).
#### Correctness
- `scanIntegrity` batch path uses `SELECT DISTINCT ON (slug)` to scope by unique slug, matching `engine.getAllSlugs()`'s `Set<string>` semantics. Multi-source brains (UNIQUE(source_id, slug) since v0.18.0) now get correct counts instead of one-scan-per-source-row.
- `IntegrityScanResult.pagesScanned` now reflects unique slugs scanned, not raw row count. Single-source brains: unchanged. Multi-source brains: counts now match expected distinct-page semantics.
- Batch-path fallback narrowed: real Postgres errors (deadlock, connection drop, SQL bug) surface via `GBRAIN_DEBUG=1` instead of being silently swallowed.
#### Tests
- New `test/e2e/integrity-batch.test.ts` — four parity cases (dedup, hits, validate, topPages) asserting batch ≡ sequential against real Postgres. Pinning the multi-source dedup case requires a raw-SQL fixture for the alt-source row since `engine.putPage` doesn't take a `source_id`.
#### Infrastructure
- `src/core/skillpack/installer.ts` — clamp negative lock-age to 0, fixing intermittent Linux ext4 CI flakes from sub-millisecond `mtimeMs` precision (Date.now is integer ms; mtime can be ~0.3ms ahead). New regression test in `test/skillpack-install.test.ts` deterministically reproduces via `utimesSync`.
- `CLAUDE.md` test inventory updated for the new test files.
## [0.22.7] - 2026-04-28
## **Built-in HTTP transport with bearer auth for remote MCP.**
## **Postgres-backed tokens, default-deny CORS, two-bucket rate limit, body cap, per-request audit.**
v0.22.7 ships `gbrain serve --http`: a built-in HTTP transport for remote MCP, authenticating via the existing `access_tokens` table that `gbrain auth create/list/revoke` already manages. Bearer-only, no OAuth surface, no registration endpoint, no self-service tokens. SECURITY.md is the canonical reference for the hardening posture and recommended deployment.
The hardening lives inside the transport, not in the doc:
| Layer | Default | Configurable via |
|---|---|---|
| CORS | default-deny (no `Access-Control-Allow-Origin`) | `GBRAIN_HTTP_CORS_ORIGIN=a.com,b.com` |
| Pre-auth IP rate limit | 30 req / 60s | `GBRAIN_HTTP_RATE_LIMIT_IP` |
| Post-auth token rate limit | 60 req / 60s | `GBRAIN_HTTP_RATE_LIMIT_TOKEN` |
| Body cap | 1 MiB, stream-counted | `GBRAIN_HTTP_MAX_BODY_BYTES` |
| `last_used_at` debounce | once per token per 60s | (SQL-level WHERE clause, race-tolerant) |
| Per-request audit | `mcp_request_log` row per `/mcp` | (existing schema, since v4) |
| Reverse-proxy trust | off | `GBRAIN_HTTP_TRUST_PROXY=1` to honor X-Forwarded-For |
The IP rate-limit fires **before** the auth lookup so the limit caps load on the auth path itself, not just response codes. The token-id rate limit fires after auth so a runaway authenticated client gets throttled at the right principal. Both buckets live in a bounded LRU map (default 10K keys, TTL prune at 2× window) so unique-key growth can't drift into memory pressure.
### What changed for users
You can now expose GBrain remotely with the built-in transport:
```bash
gbrain auth create my-laptop # tokens managed via the existing CLI
gbrain serve --http --port 8787 # Postgres-only; PGLite users see a clear fail-fast
ngrok http 8787 --url your-brain.ngrok.app # any tunnel works
```
Then point Claude Desktop, claude.ai/code, or any MCP client at `http://your-tunnel/mcp` with `Authorization: Bearer <token>`. CORS, rate limits, and body caps are on by default. `gbrain auth` is now wired into the main CLI, so it works from the compiled binary the same as `gbrain doctor` or `gbrain serve`.
### For contributors
- `src/mcp/dispatch.ts` (new) — shared `dispatchToolCall(engine, name, params, opts)` consumed by both stdio (`server.ts`) and HTTP (`http-transport.ts`). One source of truth for `validateParams`, `OperationContext` construction, and handler invocation, so the two transports can't drift apart.
- `src/mcp/rate-limit.ts` (new) — bounded-LRU token-bucket. Tracks `lastTouchedMs` separately from `lastRefillMs` so an exhausted key can't be reset by hammering past the TTL.
- `src/mcp/http-transport.ts` — built on the new dispatch + rate-limit modules. `application/json` response shape (gbrain MCP tools are synchronous; the Streamable HTTP transport spec allows JSON for non-streaming responses).
- `src/cli.ts` + `src/commands/auth.ts``auth` is now a wired CLI subcommand. Direct-script usage (`bun run src/commands/auth.ts ...`) still works for environments without a compiled binary.
- 23 unit cases in `test/http-transport.test.ts`, 8 E2E cases in `test/e2e/http-transport.test.ts`. Unit covers the full dispatch round-trip with a real operation; E2E covers `last_used_at` debounce against real Postgres semantics.
### Known limits
- `gbrain serve --http` is **Postgres-only**. PGLite has no `access_tokens` or `mcp_request_log` table by design (`src/core/pglite-schema.ts:5-6`). Local agents continue to use stdio (`gbrain serve`).
- Behind a tunnel (ngrok, Tailscale Funnel, Cloudflare Tunnel), all requests share one egress IP. The pre-auth IP bucket becomes effectively shared by all clients on that tunnel; the token-id bucket is the load-bearing limiter for tunnel deployments. Documented in SECURITY.md.
### Itemized changes
- New: `gbrain serve --http [--port N]` ships the built-in HTTP transport
- New: `gbrain auth create/list/revoke/test` wired into the main CLI (was a standalone script)
- New: SECURITY.md documents the disclosure path, the recommended remote-MCP setup, and the full hardening reference
- New: `src/mcp/dispatch.ts` — shared dispatch path for stdio + HTTP
- New: `src/mcp/rate-limit.ts` — bounded-LRU token-bucket limiter
- Hardening: CORS default-deny, two-bucket rate limit (per-IP pre-auth + per-token post-auth), 1 MiB body cap with stream-counted enforcement, `mcp_request_log` per-request audit, `last_used_at` SQL-level debounce
- Tests: 23 unit + 8 E2E covering auth, dispatch, CORS, body cap, rate limit, and audit
- Docs: SECURITY.md, DEPLOY.md, and per-client setup guides updated to recommend `--http` and document the env vars
## To take advantage of v0.22.7
`gbrain upgrade` should do this automatically. If it didn't, or if you want to expose your brain over HTTP:
1. **Confirm migrations are at v4 or higher** (the `access_tokens` + `mcp_request_log` tables were added in migration v4):
```bash
gbrain doctor # schema_version check should pass
gbrain apply-migrations --yes # if not, run this
```
2. **Create a token for each remote client:**
```bash
gbrain auth create my-laptop # prints the token once — copy it
```
3. **Start the HTTP server:**
```bash
gbrain serve --http --port 8787
```
4. **(Optional) configure CORS allowlist if a browser client will hit it:**
```bash
GBRAIN_HTTP_CORS_ORIGIN=https://claude.ai gbrain serve --http --port 8787
```
5. **(Optional) audit who's hitting your brain:**
```bash
psql $DATABASE_URL -c "SELECT created_at, token_name, operation, status, latency_ms
FROM mcp_request_log ORDER BY created_at DESC LIMIT 50"
```
6. **If `gbrain serve --http` exits with "Postgres engine required":** PGLite is local-only by design. Either keep using stdio (`gbrain serve`) for local agents, or migrate to Postgres (`gbrain migrate --to supabase`).
If anything breaks: `gbrain doctor`, `~/.gbrain/upgrade-errors.jsonl` (if present), and please file an issue at https://github.com/garrytan/gbrain/issues with both.
## [0.22.6.1] - 2026-04-26
**Old brains can upgrade again.**
**Two-year, ten-issue wedge cycle ends. Pre-v0.13/v0.18/v0.19 brains all upgrade clean.**
If you've been pinned to an older gbrain because `gbrain upgrade` wedges your brain
with `column "source_id" does not exist` or `column "link_source" does not exist`,
v0.22.6.1 unblocks you. The fix lives in `initSchema()` itself, where it should
have lived all along.
The bug class is structural: gbrain ships an "embedded latest schema" SQL blob
that runs before numbered migrations on every connect. The blob references
columns that newer migrations introduce. On any brain older than the migration
that adds those columns, the blob crashes before the migration can run. This
incident family hit users 10+ times across 6 schema versions over 2 years
(issues #239, #243, #266, #357, #366, #374, #375, #378, #395, #396).
The fix is a narrow pre-schema bootstrap. `initSchema()` now probes for the
specific forward-referenced state the schema blob needs (`pages.source_id`,
`links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`,
`content_chunks.language`, plus the `sources` FK target table) and adds only
that state if missing. Then SCHEMA_SQL replays cleanly. Then the normal
migration chain runs as usual. Fresh installs and modern brains both no-op.
A test guard prevents this incident family from recurring. Every future
migration that adds a column-with-index to PGLITE_SCHEMA_SQL must extend the
bootstrap; the CI guard fails loudly if not. The pattern that broke gbrain ten
times in two years is now structurally prevented.
Also includes the v24 PGLite RLS fix from #395 (community PR by @jdcastro2):
`rls_backfill_missing_tables` now no-ops on PGLite via `sqlFor.pglite: ''`,
since PGLite has no RLS engine and is single-tenant by definition.
### The numbers that matter
| Metric | v0.22.0 | v0.22.6.1 | Δ |
|---|---|---|---|
| Pre-v0.13 brain upgrades cleanly | wedges on `link_source` | passes | ✓ |
| Pre-v0.18 brain upgrades cleanly | wedges on `source_id` | passes | ✓ |
| Pre-v0.21 brain upgrades cleanly | wedges on `symbol_name` | passes | ✓ |
| v24 RLS migration on PGLite | wedges (table doesn't exist) | no-op | ✓ |
| Issues closed | — | #366, #375, #378, #395, #396 | 5 |
| Issue families resolved | — | wedge-cycle | the whole class |
### What this means for you
If you've been on v0.13.x, v0.14.x, v0.17.x, v0.18.x, v0.19.x, v0.20.x, or v0.22.0 and
your `gbrain upgrade` failed, run it again. It should walk to v0.22.6.1 cleanly.
If you wedged on the v24 RLS migration on a PGLite brain, the same thing.
If you're on a fresh install or already on v0.22.0, this patch is invisible.
The bootstrap probe runs once per connect, sees nothing to do, and returns.
### Itemized changes
#### Fixed
- `gbrain upgrade` no longer wedges on pre-v0.18 brains that lack `pages.source_id`. The schema blob's `CREATE INDEX idx_pages_source_id` previously crashed before migration v21 could add the column. Closes #366, #375, #378, #396.
- `gbrain upgrade` no longer wedges on pre-v0.13 brains that lack `links.link_source` or `links.origin_page_id`. The schema blob's `CREATE INDEX idx_links_source/origin` previously crashed before migration v11 could add the columns. Closes #266, #357.
- `gbrain upgrade` no longer wedges on pre-v0.19 brains that lack `content_chunks.symbol_name` or `content_chunks.language`. The schema blob's partial indexes previously crashed before migration v26 could add the columns.
- Migration v24 (`rls_backfill_missing_tables`) no-ops on PGLite via `sqlFor.pglite: ''`. PGLite has no RLS engine and is single-tenant. The migration previously tried to ALTER subagent tables that don't exist in pglite-schema.ts. Closes #395. Contributed by @jdcastro2.
#### Changed
- `PGLiteEngine.initSchema()` and `PostgresEngine.initSchema()` now call a new private `applyForwardReferenceBootstrap()` before running the embedded schema blob. The bootstrap probes for missing forward-referenced state and adds only what's needed. No-op on fresh installs and modern brains.
#### For contributors
- New CI guard `test/schema-bootstrap-coverage.test.ts` enforces that `applyForwardReferenceBootstrap` covers every forward reference in PGLITE_SCHEMA_SQL. When you add a new column-with-index in the schema blob, extend `REQUIRED_BOOTSTRAP_COVERAGE` and the bootstrap function. The test fails loudly if you skip step one.
- New `test/bootstrap.test.ts` covers the bootstrap contract: no-op on fresh install, idempotent, no-op on modern brain, full path pre-v0.18, fresh-install regression, pre-v0.13 links shape.
- New `test/e2e/postgres-bootstrap.test.ts` exercises `PostgresEngine.initSchema()` directly (not the standalone `db.initSchema` from `src/core/db.ts`, which only runs SCHEMA_SQL and would have produced false-positive coverage). Codex caught this E2E shape gap during plan review.
- Wave PRs incorporated with attribution: @vinsew (#398), @jdcastro2 (#399), @schnubb-web (#402). The narrow-bootstrap shape supersedes #402's broader "run all migrations early" approach, which would have crashed on v24 trying to alter tables that the schema blob hadn't created yet (codex finding during plan review).
## To take advantage of v0.22.6.1
`gbrain upgrade` should do this automatically. If you're currently wedged on a
prior version's upgrade attempt:
1. **Run the upgrade:**
```bash
gbrain upgrade
```
2. **Verify the outcome:**
```bash
gbrain doctor
```
Expected: `schema_version: Version 29 (latest: 29)` clean, no
`column "..." does not exist` errors, no wedged migration ledger.
3. **If wedged after upgrade,** run the migration runner directly:
```bash
gbrain apply-migrations --yes
```
4. **If any step still fails,** please file an issue:
https://github.com/garrytan/gbrain/issues with:
- output of `gbrain doctor`
- your prior gbrain version (`gbrain --version`)
- which step broke
## [0.22.6] - 2026-04-28
### Schema verification after migrations
- Post-migration schema verification catches columns that were defined in migrations but silently failed to create (common with PgBouncer transaction-mode poolers).
- Self-healing: automatically adds missing columns via ALTER TABLE when detected.
- Prevents the "column X does not exist" embed failures that occur when schema version is ahead of actual table state.
## [0.22.5] - 2026-04-27
## **Autopilot stops re-importing your whole brain when a commit gets garbage-collected.**
+49 -10
View File
@@ -25,15 +25,19 @@ strict behavior when unset.
- `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`. `OperationContext.remote` flags untrusted callers.
- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports.
- `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`)
- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract.
- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for the specific forward-referenced state the embedded schema blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target table) and adds only what's missing. Closes the upgrade-wedge bug class that bit users 10+ times across 6 schema versions over 2 years (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396). No-op on fresh installs and modern brains.
- `src/core/pglite-schema.ts` — PGLite-specific DDL (pgvector, pg_trgm, triggers)
- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. v0.22.0: `searchKeyword`, `searchKeywordChunks`, and `searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude clause from `src/core/search/sql-ranking.ts`. `searchVector` switches to a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in the outer SELECT) and carries `p.source_id` through inner→outer for v0.18 multi-source callers. v0.22.1 (#406): `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures). `executeRaw` is a single-statement passthrough — no per-call retry (D3 dropped that as unsound for non-idempotent statements; recovery is supervisor-driven). v0.22.1 (#363, contributed by @orendi84): `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. v0.22.1 (#409, contributed by @atrevino47): `countStaleChunks()` + `listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale`, eliminating ~76 MB/call client-side pull on a fully-embedded brain; `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding (consistency).
- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. v0.22.0: `searchKeyword`, `searchKeywordChunks`, and `searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude clause from `src/core/search/sql-ranking.ts`. `searchVector` switches to a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in the outer SELECT) and carries `p.source_id` through inner→outer for v0.18 multi-source callers. v0.22.1 (#406): `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures). `executeRaw` is a single-statement passthrough — no per-call retry (D3 dropped that as unsound for non-idempotent statements; recovery is supervisor-driven). v0.22.1 (#363, contributed by @orendi84): `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. v0.22.1 (#409, contributed by @atrevino47): `countStaleChunks()` + `listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale`, eliminating ~76 MB/call client-side pull on a fully-embedded brain; `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding (consistency). As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL on the same forward-reference probe set as the PGLite engine, so old Postgres brains pinned at v0.13/v0.18/v0.19 walk forward cleanly instead of wedging on `column "..." does not exist`.
- `src/core/utils.ts` — Shared SQL utilities extracted from postgres-engine.ts. Exports `parseEmbedding(value)` (throws on unknown input, used by migration + ingest paths where data integrity matters) and as of v0.12.3 `tryParseEmbedding(value)` (returns `null` + warns once per process, used by search/rescore paths where availability matters more than strictness).
- `src/core/db.ts` — Connection management, schema initialization. v0.22.1 (#363, contributed by @orendi84): `resolveSessionTimeouts()` returns `statement_timeout` + `idle_in_transaction_session_timeout` (defaults: 5min each, env-overridable via `GBRAIN_STATEMENT_TIMEOUT` / `GBRAIN_IDLE_TX_TIMEOUT` / `GBRAIN_CLIENT_CHECK_INTERVAL`). Both `connect()` (module singleton) and `PostgresEngine.connect()` (worker pool) consume the result via postgres.js's `connection` option, sending GUCs as startup parameters that survive PgBouncer transaction mode (unlike the prior `setSessionDefaults` post-pool SET, kept as a back-compat no-op shim).
- `src/commands/migrate-engine.ts` — Bidirectional engine migration (`gbrain migrate --to supabase/pglite`)
- `src/core/import-file.ts` — importFromFile + importFromContent (chunk + embed + tags)
- `src/core/sync.ts` — Pure sync functions (manifest parsing, filtering, slug conversion)
- `src/core/sync.ts` — Pure sync functions (manifest parsing, filtering, slug conversion). v0.22.12 (#500, foundation by @wintermute via #501): `classifyErrorCode(errorMsg)` regex-based classifier with 12 codes (`SLUG_MISMATCH`, `YAML_PARSE`, `YAML_DUPLICATE_KEY`, `MISSING_OPEN`, `MISSING_CLOSE`, `NESTED_QUOTES`, `EMPTY_FRONTMATTER`, `NULL_BYTES`, `INVALID_UTF8`, `STATEMENT_TIMEOUT`, `FILE_TOO_LARGE`, `SYMLINK_NOT_ALLOWED`) plus `UNKNOWN` fallback. `summarizeFailuresByCode(failures)` returns sorted `[{code, count}]`. `code?` optional field on `SyncFailure`; backfilled at ack time on pre-v0.22.12 entries. `acknowledgeSyncFailures()` returns `AcknowledgeResult { count, summary }`. Three regexes (`MISSING_OPEN`, `MISSING_CLOSE`, `EMPTY_FRONTMATTER`) broadened to match actual `markdown.ts:159-244` validator message strings, not just the literal code-name prefix. `FILE_TOO_LARGE` covers all three production size sites in `import-file.ts:199, 352, 401`; `SYMLINK_NOT_ALLOWED` covers the rejection at `:347`. Closes the silent-skip pattern that motivated #500.
- `src/core/storage.ts` — Pluggable storage interface (S3, Supabase Storage, local)
- `src/core/storage-config.ts` (v0.22.11) — Storage tiering: `loadStorageConfig` reads `gbrain.yml`, normalizes deprecated keys (`git_tracked` / `supabase_only`) to canonical (`db_tracked` / `db_only`) with once-per-process deprecation warning, and runs `normalizeAndValidateStorageConfig` (auto-fixes missing trailing `/`, throws `StorageConfigError` on tier overlap). Path-segment matcher: `media/x/` does NOT match `media/xerox/foo`. Replaces gray-matter (broken on delimiter-less YAML) with a dedicated parser for the `gbrain.yml` shape.
- `src/core/disk-walk.ts` (v0.22.11) — `walkBrainRepo(repoPath)` returns `Map<slug, {size, mtimeMs}>` from one recursive `readdirSync`. Skips dot-dirs, `node_modules`, non-`.md` files. Used by `gbrain storage status` to replace per-page `existsSync + statSync` (~400K syscalls on 200K-page brains → tens).
- `src/commands/storage.ts` (v0.22.11) — `gbrain storage status [--repo P] [--json]`. Split into pure data (`getStorageStatus`) + JSON formatter + human formatter (ASCII-only per D10) matching the `orphans.ts` pattern. `PageCountsByTier` and `DiskUsageByTier` are distinct nominal types so swaps fail at compile time.
- `gbrain.yml` (brain repo root, v0.22.11) — Optional storage tiering config. Top-level `storage:` section with `db_tracked:` and `db_only:` array-valued keys. `gbrain sync` auto-manages `.gitignore` for `db_only` paths on successful sync (skips on dry-run, blocked-by-failures, submodule context, or `GBRAIN_NO_GITIGNORE=1`). `gbrain export --restore-only [--repo P] [--type T] [--slug-prefix S]` repopulates missing `db_only` files from the database.
- `src/core/supabase-admin.ts` — Supabase admin API (project discovery, pgvector check)
- `src/core/file-resolver.ts` — File resolution with fallback chain (local -> .redirect.yaml -> .redirect -> .supabase)
- `src/core/chunkers/` — 3-tier chunking (recursive, semantic, LLM-guided). v0.19.0 adds `code.ts` — tree-sitter-based semantic chunker for 29 languages with embedded-asset WASMs (`src/assets/wasm/`), `@dqbd/tiktoken` cl100k_base tokenizer, small-sibling merging. `CHUNKER_VERSION` constant folded into `importCodeFile`'s `content_hash` so chunker shape changes force clean re-chunks across releases.
@@ -88,21 +92,31 @@ strict behavior when unset.
- `src/core/minions/attachments.ts` — Attachment validation (path traversal, null byte, oversize, base64, duplicate detection)
- `src/commands/agent.ts` (v0.16) — `gbrain agent run <prompt> [flags]` CLI. Submits `subagent` (or N children + 1 aggregator) under `{allowProtectedSubmit: true}`. Single-entry `--fanout-manifest` short-circuits. Children get `on_child_fail: 'continue'` + `max_stalled: 3`. `--follow` is the default on TTY; streams logs + polls `waitForCompletion` in parallel. Ctrl-C detaches, does not cancel.
- `src/commands/agent-logs.ts` (v0.16) — `gbrain agent logs <job> [--follow] [--since]`. Merges JSONL heartbeat audit + `subagent_messages` into a chronological timeline. `parseSince` accepts ISO-8601 or relative (`5m`, `1h`, `2d`). Transcript tail renders only for terminal jobs.
- `src/commands/jobs.ts``gbrain jobs` CLI subcommands + `gbrain jobs work` daemon. v0.13.1 surfaces the full `MinionJobInput` retry/backoff/timeout/idempotency surface as first-class CLI flags on `jobs submit`: `--max-stalled`, `--backoff-type fixed|exponential`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key`. `jobs smoke --sigkill-rescue` is the opt-in regression guard for #219. v0.16 wires `registerBuiltinHandlers` to always register `subagent` + `subagent_aggregator` (no env flag — `ANTHROPIC_API_KEY` is the natural cost gate, trust is via `PROTECTED_JOB_NAMES`) and loads `GBRAIN_PLUGIN_PATH` plugins at worker startup with a loud startup-line per plugin. `shell` handler still gated by `GBRAIN_ALLOW_SHELL_JOBS=1` (RCE surface, separate concern).
- `src/commands/jobs.ts``gbrain jobs` CLI subcommands + `gbrain jobs work` daemon. v0.13.1 surfaces the full `MinionJobInput` retry/backoff/timeout/idempotency surface as first-class CLI flags on `jobs submit`: `--max-stalled`, `--backoff-type fixed|exponential`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key`. `jobs smoke --sigkill-rescue` is the opt-in regression guard for #219. v0.16 wires `registerBuiltinHandlers` to always register `subagent` + `subagent_aggregator` (no env flag — `ANTHROPIC_API_KEY` is the natural cost gate, trust is via `PROTECTED_JOB_NAMES`) and loads `GBRAIN_PLUGIN_PATH` plugins at worker startup with a loud startup-line per plugin. `shell` handler still gated by `GBRAIN_ALLOW_SHELL_JOBS=1` (RCE surface, separate concern). v0.22.10 (#521): the `autopilot-cycle` handler now forwards `job.data.phases` to `runCycle` (was previously discarded — caller-supplied phase selection silently became a full cycle). Phases are validated against `ALL_PHASES` from `src/core/cycle.ts`; invalid names are filtered out and an empty/missing array falls back to the default 6-phase cycle. v0.22.13 (PR #490 CODEX-1+CODEX-4): `sync` handler now resolves `sourceId` at entry by looking up `sources.local_path` (mirrors `cycle.ts:480`'s autopilot fix from PR #475) so multi-source brains read the per-source `last_commit` anchor instead of the global config key. Concurrency routed through the shared `autoConcurrency()` policy in `src/core/sync-concurrency.ts` instead of the prior hardcoded `4`; PGLite stays serial. `noEmbed` default is `true` (embed is a separate job — submit `gbrain embed --stale` after sync, or rely on the autopilot cycle's embed phase).
- `src/commands/features.ts``gbrain features --json --auto-fix`: usage scan + feature adoption salesman
- `src/commands/autopilot.ts``gbrain autopilot --install`: self-maintaining brain daemon (sync+extract+embed)
- `src/mcp/server.ts` — MCP stdio server (generated from operations)
- `src/commands/auth.ts` — Standalone token management (create/list/revoke/test)
- `src/mcp/server.ts` — MCP stdio server (generated from operations). v0.22.7: tool-call handler delegates to `dispatchToolCall` from `src/mcp/dispatch.ts` so stdio + HTTP transports share one validation, context-build, and error-format path.
- `src/mcp/dispatch.ts` (v0.22.7) — Shared tool-call dispatch consumed by both stdio (`server.ts`) and HTTP (`http-transport.ts`). Exports `dispatchToolCall(engine, name, params, opts)`, `buildOperationContext(engine, params, opts)`, and `validateParams(op, params)`. Single source of truth for `(ctx, params)` handler arg order and the 5-field `OperationContext` shape (engine + config + logger + dryRun + remote). Defaults to `remote: true` (untrusted); local CLI callers pass `remote: false`. Closed F1 (reversed handler args) + F2 (incomplete OperationContext) + F3 (no param validation) drift bugs in the original v0.22.5 HTTP transport.
- `src/mcp/rate-limit.ts` (v0.22.7) — Bounded-LRU token-bucket limiter for `gbrain serve --http`. `buildDefaultLimiters()` returns the two-bucket pipeline used by http-transport: pre-auth IP (default 30/60s, fires BEFORE the DB lookup so brute-force load against `access_tokens` is actually capped) + post-auth token-id (default 60/60s). Tracks `lastTouchedMs` separately from `lastRefillMs` so an exhausted key can't be reset by hammering past the TTL. LRU cap (default 10K keys) bounds memory under attacker-controlled key growth; TTL prune at 2× window evicts abandoned buckets.
- `src/mcp/http-transport.ts` (v0.22.7, rewrite) — `gbrain serve --http` HTTP transport. Postgres-only — fails fast at startup on PGLite (the `access_tokens` table only exists on Postgres). Bearer auth against SHA-256 hashes in `access_tokens`. CORS default-deny via `GBRAIN_HTTP_CORS_ORIGIN` allowlist. Body cap stream-counted (1 MiB default via `GBRAIN_HTTP_MAX_BODY_BYTES`) so chunked transfers without Content-Length still hit the cap. `last_used_at` SQL-level debounce (one UPDATE per token per 60s). Per-request audit row in `mcp_request_log` with token_name + operation + status + latency. Optional `GBRAIN_HTTP_TRUST_PROXY=1` honors `X-Forwarded-For` — only safe when bound to a private interface AND the proxy strips client-supplied XFF (otherwise enables IP spoofing past the pre-auth rate limit). `/health` does `SELECT 1` against Postgres and returns 503 + `status:unhealthy` when the DB is unreachable so orchestration doesn't see green pods while clients get misleading 401s. Replaces the standalone OAuth wrapper that was vulnerable to unauthenticated client registration.
- `src/commands/auth.ts` — Token management for the HTTP transport. `gbrain auth create/list/revoke/test`. As of v0.22.7 wired into the main CLI (`src/cli.ts`); also runs standalone via `bun run src/commands/auth.ts ...` for environments without a compiled binary. Tokens stored as SHA-256 hashes in `access_tokens` (Postgres-only).
- `src/commands/upgrade.ts` — Self-update CLI. `runPostUpgrade()` enumerates migrations from the TS registry (src/commands/migrations/index.ts) and tail-calls `runApplyMigrations(['--yes', '--non-interactive'])` so the mechanical side of every outstanding migration runs unconditionally.
- `src/commands/migrations/` — TS migration registry (compiled into the binary; no filesystem walk of `skills/migrations/*.md` needed at runtime). `index.ts` lists migrations in semver order. `v0_11_0.ts` = Minions adoption orchestrator (8 phases). `v0_12_0.ts` = Knowledge Graph auto-wire orchestrator (5 phases: schema → config check → backfill links → backfill timeline → verify). `phaseASchema` has a 600s timeout (bumped from 60s in v0.12.1 for duplicate-heavy brains). `v0_12_2.ts` = JSONB double-encode repair orchestrator (4 phases: schema → repair-jsonb → verify → record). `v0_14_0.ts` = shell-jobs + autopilot cooperative (2 phases: schema ALTER minion_jobs.max_stalled SET DEFAULT 3 — superseded by v0.14.3's schema-level DEFAULT 5 + UPDATE backfill; pending-host-work ping for skills/migrations/v0.14.0.md). All orchestrators are idempotent and resumable from `partial` status. As of v0.14.2 (Bug 3), the RUNNER owns all ledger writes — orchestrators return `OrchestratorResult` and `apply-migrations.ts` persists a canonical `{version, status, phases}` shape after return. Orchestrators no longer call `appendCompletedMigration` directly. `statusForVersion` prefers `complete` over `partial` (never regresses). 3 consecutive partials → wedged → `--force-retry <version>` writes a `'retry'` reset marker. v0.14.3 (fix wave) ships schema-only migrations v14 (`pages_updated_at_index`) + v15 (`minion_jobs_max_stalled_default_5` with UPDATE backfill) via the `MIGRATIONS` array in `src/core/migrate.ts` — no orchestrator phases needed.
- `src/commands/repair-jsonb.ts``gbrain repair-jsonb [--dry-run] [--json]`: rewrites `jsonb_typeof='string'` rows in place across 5 affected columns (pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata, page_versions.frontmatter). Fixes v0.12.0 double-encode bug on Postgres; PGLite no-ops. Idempotent.
- `src/commands/orphans.ts``gbrain orphans [--json] [--count] [--include-pseudo]`: surfaces pages with zero inbound wikilinks, grouped by domain. Auto-generated/raw/pseudo pages filtered by default. Also exposed as `find_orphans` MCP operation. Shipped in v0.12.3 (contributed by @knee5).
- `src/commands/doctor.ts``gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel <id>`.
- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows.
- `src/commands/integrity.ts``gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). v0.22.8: batch-load fast path on Postgres uses `SELECT DISTINCT ON (slug)` in a single SQL query to fix the PgBouncer round-trip timeout (60s → ~6s) while preserving `engine.getAllSlugs()`'s `Set<string>` semantics on multi-source brains. Gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1` so real Postgres errors are diagnosable.
- `src/commands/doctor.ts``gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel <id>`. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count.
- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2).
- `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2.
- `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=<ms>` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers.
- `src/core/db-lock.ts` (v0.22.13) — generic `tryAcquireDbLock(engine, lockId, ttlMinutes)` over the existing `gbrain_cycle_locks` table. Parameterized lock id so different scopes can nest cleanly: `gbrain-cycle` for the broad cycle (held by `cycle.ts`) and `gbrain-sync` (`SYNC_LOCK_ID` constant) for `performSync`'s narrower writer window. Same UPSERT-with-TTL semantics as the prior cycle-only helper, just generalized. Survives PgBouncer transaction pooling (unlike session-scoped `pg_try_advisory_lock`); crashed holders auto-release once their TTL expires.
- `src/core/sync-concurrency.ts` (v0.22.13) — single source of truth for the parallel-sync policy. Exports `autoConcurrency(engine, fileCount, override?)` (PGLite always serial; explicit override clamped to >=1; auto path returns `DEFAULT_PARALLEL_WORKERS=4` when `fileCount > AUTO_CONCURRENCY_FILE_THRESHOLD=100`), `shouldRunParallel(workers, fileCount, explicit)` (Q1: explicit `--workers` bypasses the >50-file floor), and `parseWorkers(s)` (rejects `'0'`, `'-3'`, `'foo'`, `'1.5'`, trailing chars — replaces the prior parseInt-with-no-validation in both `sync.ts` and `import.ts`). Used by `performSync`, `performFullSync`, `runImport`, and the Minion `sync` handler so the three sites can no longer drift.
- `src/commands/sync.ts``gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). v0.22.13 (PR #490): `performSync` wraps its body in a `gbrain-sync` writer lock so two concurrent syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot both write `last_commit` and let the last writer win. Head-drift gate after the import phase re-checks `git rev-parse HEAD`; if HEAD moved (someone ran `git checkout` / `git pull` mid-sync), the bookmark refuses to advance. Vanished files now record a failedFiles entry instead of silent-skip — the silent-skip-then-advance pathology that survived prior hardening passes is dead. Worker engines wrap in try/finally so disconnect always fires (panic-path leak fix). Both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`), validated via `parseWorkers`. Explicit `--workers` bypasses the auto-path file-count floor; auto path defers to `autoConcurrency()`. Banner moved to stderr.
- `src/core/cycle.ts` — v0.17 brain maintenance cycle primitive. `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise<CycleReport>` composes 6 phases in semantically-driven order (lint → backlinks → sync → extract → embed → orphans). Three callers: `gbrain dream` CLI, `gbrain autopilot` daemon's inline path, and the Minions `autopilot-cycle` handler (`src/commands/jobs.ts`). One source of truth for what the brain does overnight. Coordination via `gbrain_cycle_locks` DB table (TTL-based; works through PgBouncer transaction pooling, unlike session-scoped `pg_try_advisory_lock`) + `~/.gbrain/cycle.lock` file lock with PID-liveness for PGLite / engine=null mode. `CycleReport.schema_version: "1"` is the stable agent-consumable shape. `PhaseResult.error: { class, code, message, hint?, docs_url? }` is Stripe-API-tier structured failure info. `yieldBetweenPhases` hook awaited between every phase — Minions handler uses this to renew its job lock and prevent v0.14 stall-death regression. Engine nullable: filesystem phases (lint, backlinks) run without DB; DB phases skip with `status: "skipped", reason: "no_database"`. Lock-skip: read-only phase selections (`--phase orphans`) bypass the cycle lock. v0.22.1 (#403): `CycleOpts.signal?: AbortSignal` propagates the worker's abort signal; `checkAborted()` fires between every phase and throws if the signal is aborted (cooperative — can't interrupt a phase mid-execution). v0.22.1 (#417): `runPhaseSync` returns `pagesAffected` via `SyncPhaseResult`; `runCycle` captures it and threads to `runPhaseExtract` as the 4th arg, enabling incremental extract on the cycle path. v0.22.1 (Codex F2): `runPhaseSync` takes `willRunExtractPhase: boolean` and sets `noExtract: phases.includes('extract')` so `gbrain dream --phase sync` doesn't silently lose extraction. v0.22.5 (#475): new `resolveSourceForDir(engine, brainDir)` helper queries `SELECT id FROM sources WHERE local_path = $1 LIMIT 1`; `runPhaseSync` threads result as `sourceId` to `performSync()` so sync reads the per-source `sources.last_commit` anchor instead of the drift-prone global `config.sync.last_commit` key. Bare try/catch lets pre-v0.18 brains fall through to the global key. Closes the prod hang where every autopilot cycle ran a 30-min full reimport because the global anchor commit had been GC'd from git history.
- `src/commands/dream.ts` — v0.17 `gbrain dream` CLI. ~80-line thin alias over `runCycle`. brainDir resolution requires explicit `--dir` OR `sync.repo_path` config (no more walk-up-cwd-for-.git footgun). Flags: `--dry-run`, `--json`, `--phase <name>`, `--pull`, `--dir <path>`. Exit code 1 on status=failed (partial/warn not fatal — don't page on warnings).
- `src/commands/friction.ts` + `src/core/friction.ts` (v0.23) — `gbrain friction {log,render,list,summary}` reporter. Append-only JSONL under `$GBRAIN_HOME/friction/<run-id>.jsonl`. Schema is a flat extension of `StructuredAgentError` (D20). Render groups by severity → phase, defaults to `--redact` for md output (strips `$HOME`/`$CWD` to placeholders so reports paste safely in PRs). Run-id resolves from `--run-id` > `$GBRAIN_FRICTION_RUN_ID` > `standalone.jsonl`. Skills the claw-test exercises gain a `_friction-protocol.md` callout so agents know when to log friction.
- `src/commands/claw-test.ts` + `src/core/claw-test/` (v0.23) — `gbrain claw-test [--scenario <name>] [--live --agent openclaw]`. End-to-end "fresh user" friction harness. Two modes: scripted (CI gate, agent-free) and live (real openclaw subprocess, $12 in tokens). Sets `GBRAIN_HOME=<tempdir>` for hermeticity and captures gbrain's `--progress-json` events from each child's stderr to verify expected phases ran (`import.files`, `extract.links_fs`, `doctor.db_checks`). Phases for scripted mode: setup → install_brain (`gbrain init --pglite`) → import (`--no-embed`) → query → extract → verify (`gbrain doctor --json`, asserts `status: 'ok'`) → render. Live mode hands `BRIEF.md` from `test/fixtures/claw-test-scenarios/<name>/` to the agent runner. v1 ships with the OpenClaw runner only (`src/core/claw-test/runners/openclaw.ts`, invokes `openclaw agent --local --agent <name> --message <brief>`); hermes runner deferred to v1.1. Transcript capture (`transcript-capture.ts`) uses `fs.createWriteStream` with `'drain'`-event backpressure — D17 fix for the 256KB-burst child-stall scenario. v0.18 upgrade scenario seeded via `seed-pglite.ts` SQL replay.
- `skills/_friction-protocol.md` (v0.23) — shared cross-cutting convention skill (like `_brain-filing-rules.md`). Tells agents when to call `gbrain friction log` and how to choose a severity. Routes to friction CLI from any skill the claw-test exercises.
- `scripts/check-progress-to-stdout.sh` — CI guard against regressing to `\r`-on-stdout progress. Wired into `bun run test` via `scripts/check-progress-to-stdout.sh && bun test` in package.json.
- `docs/progress-events.md` — Canonical JSON event schema reference. Stable from v0.15.2, additive only.
- `src/core/markdown.ts` — Frontmatter parsing + body splitter. `splitBody` requires an explicit timeline sentinel (`<!-- timeline -->`, `--- timeline ---`, or `---` immediately before `## Timeline`/`## History`). Plain `---` in body text is a markdown horizontal rule, not a separator. `inferType` auto-types `/wiki/analysis/` → analysis, `/wiki/guides/` → guide, `/wiki/hardware/` → hardware, `/wiki/architecture/` → architecture, `/writing/` → writing (plus the existing people/companies/deals/etc heuristics).
@@ -212,6 +226,20 @@ Key commands added in v0.14.3 (fix wave):
- `gbrain jobs submit` gains `--max-stalled`, `--backoff-type`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key` — exposing existing `MinionJobInput` fields as first-class CLI flags.
- `gbrain jobs smoke --sigkill-rescue` — opt-in regression smoke case simulating a killed worker; asserts the v0.14.3 schema default (`max_stalled=5`) actually rescues on first stall.
Key commands added in v0.22.13 (PR #490):
- `gbrain sync --workers N` (alias `--concurrency N`) — parallelize the import phase using per-worker Postgres engines (small pool of 2 each) with an atomic queue index. Auto-concurrency: defaults to 4 workers when the diff exceeds 100 files. Smaller diffs stay serial. Explicit `--workers` always wins (even on a 30-file diff). PGLite forces serial regardless. Validation rejects `0`, negatives, non-integers loud (replaces the prior silent fall-through to auto-concurrency).
- `gbrain import --workers N` — same `parseWorkers()` validation as sync; same try/finally worker-engine cleanup. Behavior surface unchanged.
Key commands added in v0.22.16 (claw-test friction loop):
- `gbrain claw-test [--scenario fresh-install|upgrade-from-v0.18] [--keep-tempdir]` — scripted-mode CI gate that runs the full canonical first-day flow against a fresh tempdir. Asserts every expected `--progress-json` phase fired and doctor's `status === 'ok'`. ~30s, no API keys.
- `gbrain claw-test --live --agent openclaw` — friction-discovery mode. Spawns real openclaw, hands it `BRIEF.md`, captures stdin/stdout/stderr to `<run>/transcript.jsonl`, lets the agent log friction via the friction CLI. Run on demand; ~510 min and ~$12 in tokens.
- `gbrain claw-test --list-agents` — reports which agent runners are registered + their detection state (binary path or unavailable reason).
- `gbrain friction log --severity {confused|error|blocker|nit} --phase <name> --message <text> [--hint ...] [--kind {friction|delight}] [--run-id ...]` — append a friction or delight entry to the active run JSONL.
- `gbrain friction render --run-id <id> [--json] [--transcripts] [--no-redact]` — markdown report grouped by severity + phase; `--redact` is the default for md output (strips `$HOME`/`$CWD` placeholders so reports paste safely in PRs/issues).
- `gbrain friction list [--json]` — recent run-ids with friction/delight counts; interrupted runs marked `(interrupted)`.
- `gbrain friction summary --run-id <id> [--json]` — two-column friction + delight summary.
- `GBRAIN_HOME` env override is now honored uniformly across every gbrain write site (config, audit, friction, sync-failures, import checkpoint, integrity log, integrations heartbeat, migration rollback, etc.) — `gbrainPath(...)` from `src/core/config.ts` is the canonical helper. Read-side host-fingerprint detection (`~/.claude`/`~/.openclaw` etc.) intentionally NOT confined in v1; that's a v1.1 follow-up.
## Testing
`bun test` runs all tests. After the v0.12.1 release: ~75 unit test files + 8 E2E test files (1412 unit pass, 119 E2E when `DATABASE_URL` is set — skip gracefully otherwise). Unit tests run
@@ -223,7 +251,9 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac
`test/files.test.ts` (MIME/hash), `test/import-file.test.ts` (import pipeline),
`test/upgrade.test.ts` (schema migrations),
`test/file-migration.test.ts` (file migration), `test/file-resolver.test.ts` (file resolution),
`test/import-resume.test.ts` (import checkpoints), `test/migrate.test.ts` (migration; v8/v9 helper-btree-index SQL structural assertions + 1000-row wall-clock fixtures that guard the O(n²)→O(n log n) fix + v0.13.1 assertions on v12/v13 SQL shape, `sqlFor` + `transaction:false` runner semantics, and the `max_stalled DEFAULT 1` regression guard),
`test/import-resume.test.ts` (import checkpoints), `test/migrate.test.ts` (migration; v8/v9 helper-btree-index SQL structural assertions + 1000-row wall-clock fixtures that guard the O(n²)→O(n log n) fix + v0.13.1 assertions on v12/v13 SQL shape, `sqlFor` + `transaction:false` runner semantics, the `max_stalled DEFAULT 1` regression guard, and v0.22.6.1 v24 `sqlFor.pglite: ''` no-op assertion),
`test/bootstrap.test.ts` (v0.22.6.1 — bootstrap contract: no-op on fresh install, idempotent across two `initSchema()` calls, no-op on modern brain that already has every probed column, full bootstrap path on simulated pre-v0.18 brain, fresh-install regression guard, pre-v0.13 `links` shape coverage),
`test/schema-bootstrap-coverage.test.ts` (v0.22.6.1 CI guard — `REQUIRED_BOOTSTRAP_COVERAGE` lists every forward reference in PGLITE_SCHEMA_SQL; the test fails loudly if `applyForwardReferenceBootstrap` skips one. When you add a column-with-index to the embedded schema blob, you extend both arrays or this guard fails. The pattern that broke gbrain ten times in two years is now structurally prevented.),
`test/setup-branching.test.ts` (setup flow), `test/slug-validation.test.ts` (slug validation),
`test/storage.test.ts` (storage backends), `test/supabase-admin.test.ts` (Supabase admin),
`test/yaml-lite.test.ts` (YAML parsing), `test/check-update.test.ts` (version check + update CLI),
@@ -265,6 +295,9 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac
`test/orphans.test.ts` (v0.12.3 orphans command: detection, pseudo filtering, text/json/count outputs, MCP op),
`test/postgres-engine.test.ts` (v0.12.3 statement_timeout scoping: `sql.begin` + `SET LOCAL` shape, source-level grep guardrail against reintroduced bare `SET statement_timeout`),
`test/sync.test.ts` (sync logic + v0.12.3 regression guard asserting top-level `engine.transaction` is not called),
`test/sync-concurrency.test.ts` (v0.22.13 PR #490: 17 cases covering `autoConcurrency()` thresholds + PGLite-forces-serial + explicit-override clamping, `shouldRunParallel()` Q1 explicit-bypasses-floor contract, and `parseWorkers()` validation that rejects `'0'`/`'-3'`/`'foo'`/`'1.5'`/trailing chars),
`test/sync-parallel.test.ts` (v0.22.13 PR #490: PGLite-routed coverage of the bookmark gate under concurrency request, head-drift gate, vanished-file failure capture, PGLite-stays-serial, and the `gbrain-sync` writer-lock contract — 7 cases),
`test/sync-failures.test.ts` (v0.22.12: 28 cases pinning `classifyErrorCode` regex coverage for all 12 codes against literal production message strings from `markdown.ts:159-244` and `import-file.ts:199, 347, 352, 401`; `summarizeFailuresByCode` sort + pre-classified-honor; `recordSyncFailures` code-field persistence; `acknowledgeSyncFailures` AcknowledgeResult shape + backfill on pre-v0.22.12 entries),
`test/doctor.test.ts` (doctor command + v0.12.3 assertions that `jsonb_integrity` scans the four v0.12.0 write sites and `markdown_body_completeness` is present),
`test/utils.test.ts` (shared SQL utilities + `tryParseEmbedding` null-return and single-warn semantics),
`test/build-llms.test.ts` (llms.txt/llms-full.txt generator: path resolution, idempotence, spec shape, regen-drift guard, content contract, AGENTS.md install-path mirror, size-budget enforcement — 7 cases),
@@ -275,20 +308,26 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac
`test/skill-manifest.test.ts` (v0.19 skill manifest parser: drift detection, managed-block markers),
`test/skillify-scaffold.test.ts` (v0.19 `gbrain skillify scaffold` stubs: SKILL.md, script, tests, routing-eval fixtures),
`test/skillpack-install.test.ts` (v0.19 `gbrain skillpack install` managed-block install / update / no-clobber semantics),
`test/skillpack-sync-guard.test.ts` (v0.19 sync-guard: bundled skills stay byte-identical to `skills/` source).
`test/skillpack-sync-guard.test.ts` (v0.19 sync-guard: bundled skills stay byte-identical to `skills/` source),
`test/http-transport.test.ts` (v0.22.7 HTTP transport: 23 unit cases covering bearer auth + missing/no-Bearer/unknown/revoked + `/health` bypass, F1+F2 round-trip via dispatch.ts, F3 invalid_params, application/json response shape (not SSE), CORS default-deny + allowlist, body cap on Content-Length AND chunked, two-bucket rate limit (refill, exhaust+Retry-After, LRU eviction, TTL prune, pre-auth IP fires before DB), and `mcp_request_log` audit on success + auth_failed).
E2E tests (`test/e2e/`): Run against real Postgres+pgvector. Require `DATABASE_URL`.
- `bun run test:e2e` runs Tier 1 (mechanical, all operations, no API keys). Includes 9 dedicated cases for the postgres-engine `addLinksBatch` / `addTimelineEntriesBatch` bind path — postgres-js's `unnest()` binding is structurally different from PGLite's and gets its own coverage.
- `test/e2e/search-quality.test.ts` runs search quality E2E against PGLite (no API keys, in-memory)
- `test/e2e/graph-quality.test.ts` runs the v0.10.3 knowledge graph pipeline (auto-link via put_page, reconciliation, traversePaths) against PGLite in-memory
- `test/e2e/postgres-jsonb.test.ts` — v0.12.2 regression test. Round-trips all 5 JSONB write sites (pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata, page_versions.frontmatter) against real Postgres and asserts `jsonb_typeof='object'` plus `->>'key'` returns the expected scalar. The test that should have caught the original double-encode bug.
- `test/e2e/integrity-batch.test.ts` (v0.22.8) — parity tests for `scanIntegrity`'s batch-load fast path vs sequential. Four cases (dedup, hits, validate, topPages) seed a fixture and assert both paths return identical results. Dedup case uses raw SQL via `getConn().unsafe()` to seed a `(test-source-2, people/alice)` row alongside the default-source row, since `engine.putPage` doesn't take a `source_id`. Pins the codex-caught multi-source overcounting regression.
- `test/e2e/jsonb-roundtrip.test.ts` — v0.12.3 companion regression against the 4 doctor-scanned JSONB sites. Assertion-level overlap with `postgres-jsonb.test.ts` is intentional defense-in-depth: if doctor's scan surface ever drifts from the actual write surface, one of these tests catches it.
- `test/e2e/sync.test.ts` (v0.22.12 — `--skip-failed` failure-loop test, alongside the existing 13 happy-path tests): exercises the full chain — broken file → `performSync` returns `blocked_by_failures` with grouped breakdown → `performSync({skipFailed: true})` advances bookmark and returns `AcknowledgeResult` with code summary → second broken file → second cycle. Saves and restores the user's real `~/.gbrain/sync-failures.jsonl` so the test is hermetic on a developer machine. Asserts bookmark gating, JSONL state, dedup across paths, summary aggregation, and the literal doctor-rendering string format. This is the integration test that proves the v0.22.12 chain holds together — unit tests cover the pure functions in isolation, this covers the integration.
- `test/e2e/upgrade.test.ts` runs check-update E2E against real GitHub API (network required)
- `test/e2e/minions-shell-pglite.test.ts` (v0.20.4) exercises the PGLite `--follow` inline shell-job path (in-memory, no `DATABASE_URL` required) — the path the consolidated minion-orchestrator skill documents for dev use
- `test/e2e/openclaw-reference-compat.test.ts` (v0.19) — exercises `check-resolvable` + `skillpack install` against a minimal AGENTS.md workspace fixture (`test/fixtures/openclaw-reference-minimal/`), regression guard for the 107-skill OpenClaw deployment shape
- `test/e2e/search-swamp.test.ts` (v0.22.0) — reproduces the headline source-swamp case. Seeds a curated `originals/talks/article-outline-fat-code` page against two `wintermute/chat/` pages stuffed with the same multi-word phrase. Asserts the article wins keyword AND vector ranking, that `detail=high` lets the chat swamp re-surface (temporal-query workflow preserved), and that `source_id` passes through the two-stage CTE intact. PGLite in-memory.
- `test/e2e/search-exclude.test.ts` (v0.22.0) — verifies `test/` + `archive/` pages are hidden by default, that `include_slug_prefixes` opts back in, and that caller-supplied `exclude_slug_prefixes` adds to defaults. Both keyword and vector search paths covered.
- `test/e2e/engine-parity.test.ts` (v0.22.0) — Postgres ↔ PGLite top-result and result-set parity for `searchKeyword` + `searchVector`. Codex flagged that Postgres ranks pages then picks best chunk while PGLite returns chunks directly — without parity coverage the source-boost fix could pass on PGLite and fail on Postgres. Skips gracefully when `DATABASE_URL` is unset.
- `test/e2e/postgres-bootstrap.test.ts` (v0.22.6.1) — exercises `PostgresEngine.initSchema()` directly against a fresh real Postgres database. Asserts the bootstrap path is no-op on fresh installs and that SCHEMA_SQL replays cleanly through the engine path (not via the standalone `db.initSchema` from `src/core/db.ts`, which would have produced false-positive coverage). Codex caught the E2E-shape gap during plan review.
- `test/e2e/http-transport.test.ts` (v0.22.7) — 8 cases against real Postgres covering `gbrain serve --http` end-to-end: bearer auth round-trip, `last_used_at` SQL-level debounce semantics, `mcp_request_log` row insertion on success and auth_failed paths, `/health` DB-down → 503 (DB-probing health check), and the F1+F2+F3 dispatch round-trip with a real operation. Skips gracefully when `DATABASE_URL` is unset.
- `test/e2e/sync-parallel.test.ts` (v0.22.13 PR #490) — DATABASE_URL-gated. T2: 60-file Postgres sync at concurrency=4 imports all + no connection leak (probes `pg_stat_activity` before/after to confirm worker engines disconnected). P4: 120-file serial-vs-parallel benchmark prints `SYNC_PARALLEL_BENCH N files | serial=Xms | parallel(4)=Yms | speedup=Zx` for CHANGELOG quoting. Asserts parallel ≤ serial × 1.5 (CI-noise tolerant; not a strict speedup gate).
- Tier 2 (`skills.test.ts`) requires OpenClaw + API keys, runs nightly in CI
- If `.env.testing` doesn't exist in this directory, check sibling worktrees for one:
`find ../ -maxdepth 2 -name .env.testing -print -quit` and copy it here if found.
+35 -5
View File
@@ -80,12 +80,13 @@ Add to `~/.claude/server.json` (Claude Code), Settings > MCP Servers (Cursor), o
### Remote MCP (Claude Desktop, Cowork, Perplexity)
```bash
ngrok http 8787 --url your-brain.ngrok.app
bun run src/commands/auth.ts create "claude-desktop"
gbrain auth create "claude-desktop" # tokens via the existing CLI
gbrain serve --http --port 8787 # built-in HTTP transport (Postgres-only)
ngrok http 8787 --url your-brain.ngrok.app # any tunnel works
claude mcp add gbrain -t http https://your-brain.ngrok.app/mcp -H "Authorization: Bearer TOKEN"
```
Per-client guides: [`docs/mcp/`](docs/mcp/DEPLOY.md). ChatGPT requires OAuth 2.1 (not yet implemented).
Per-client guides: [`docs/mcp/`](docs/mcp/DEPLOY.md). Hardening defaults, env vars, and threat model: [SECURITY.md](SECURITY.md). ChatGPT requires OAuth 2.1 (not yet implemented).
### Using gbrain with GStack
@@ -359,6 +360,30 @@ accumulate rows across separate single-skill installs instead of overwriting eac
Read [`skills/skillify/SKILL.md`](skills/skillify/SKILL.md) for the full 10-item checklist
and the anti-patterns it catches.
## Storage tiering: keep bulk content out of git (v0.22.11)
When your brain crosses 100K files and bulk machine-generated content (tweets, articles, transcripts)
becomes the size driver, declare which directories belong in git and which live in the database only.
```yaml
# gbrain.yml at the brain repo root
storage:
db_tracked:
- people/
- companies/
- deals/
db_only:
- media/x/
- media/articles/
- meetings/transcripts/
```
`gbrain sync` auto-manages your `.gitignore` for `db_only` paths. `gbrain export --restore-only --repo .`
repopulates missing files from the database (container restart, fresh clone, accidental rm).
`gbrain storage status` shows the tier breakdown.
Full guide: [docs/storage-tiering.md](docs/storage-tiering.md).
## Getting Data In
GBrain ships integration recipes that your agent sets up for you. Each recipe tells the agent what credentials to ask for, how to validate, and what cron to register.
@@ -614,8 +639,11 @@ SEARCH
gbrain query <question> Hybrid search (vector + keyword + RRF)
IMPORT
gbrain import <dir> [--no-embed] Import markdown (idempotent)
gbrain sync [--repo <path>] Git-to-brain incremental sync
gbrain import <dir> [--no-embed] [--workers N]
Import markdown (idempotent)
gbrain sync [--repo <path>] [--workers N]
Git-to-brain incremental sync
(>100-file diffs auto-parallelize 4 workers on Postgres)
gbrain export [--dir ./out/] Export to markdown
FILES
@@ -657,6 +685,8 @@ ADMIN
gbrain doctor --locks List idle-in-tx backends (57014 diagnostic, Postgres only)
gbrain stats Brain statistics
gbrain serve MCP server (stdio)
gbrain serve --http --port 8787 MCP server (HTTP, Postgres-only, bearer auth)
gbrain auth create|list|revoke|test Token management for the HTTP transport
gbrain integrations Integration recipe dashboard
gbrain sources list|add|remove|... Multi-source brain management (v0.18)
gbrain dream [--dry-run] [--phase N] One maintenance cycle then exit (cron-friendly)
+168
View File
@@ -0,0 +1,168 @@
# Security
## Reporting Vulnerabilities
If you discover a security issue in GBrain, please report it privately by opening
a [private security advisory](https://github.com/garrytan/gbrain/security/advisories/new)
on GitHub.
Do not open a public issue for security vulnerabilities.
## Remote MCP Security
### ⚠️ Do NOT use open OAuth client registration for remote MCP
If you deploy GBrain's MCP server behind an HTTP wrapper with OAuth 2.1
support, **never allow unauthenticated client registration**. An attacker
who discovers your server URL can:
1. Register a new OAuth client via `POST /register`
2. Use `client_credentials` grant to obtain a bearer token
3. Access all brain data via the MCP tools
### Recommended: `gbrain serve --http`
As of v0.22.7, GBrain ships a built-in HTTP transport that uses the
existing `access_tokens` table for authentication:
```bash
# Create a token
gbrain auth create "my-client"
# Start the HTTP server
gbrain serve --http --port 8787
# Connect via ngrok, Tailscale, or any tunnel
ngrok http 8787 --url your-brain.ngrok.app
```
This is the recommended way to expose GBrain remotely. No OAuth, no
registration endpoint, no self-service tokens. Tokens are managed
exclusively via `gbrain auth create/list/revoke`.
### If you must use a custom HTTP wrapper
1. **Require a secret for client registration** — check a header or body
parameter before creating new OAuth clients
2. **Disable `client_credentials` grant** — only allow `authorization_code`
with browser-based approval
3. **Restrict scopes** — never issue tokens with unlimited scope
4. **Log all token issuance** — alert on unexpected registrations
5. **Rate-limit registration and token endpoints**
### Token Management
```bash
gbrain auth create "claude-desktop" # Create a new token
gbrain auth list # List all tokens
gbrain auth revoke "claude-desktop" # Revoke a token
gbrain auth test <url> --token <tok> # Smoke-test a remote server
```
Tokens are stored as SHA-256 hashes in the `access_tokens` table. The
plaintext token is shown once at creation and never stored.
## `gbrain serve --http` hardening (v0.22.7+)
The built-in HTTP transport ships with several layers of hardening on by
default. All env vars below are optional; the defaults are intentionally
conservative.
### Postgres-only
`gbrain serve --http` requires a Postgres engine. PGLite is local-only by
design and the `access_tokens` / `mcp_request_log` tables don't exist in
the PGLite schema. Local agents continue to use stdio (`gbrain serve`).
Running `--http` against a PGLite-backed install fails fast with a clear
error message at startup.
### CORS
Default-deny: no `Access-Control-Allow-Origin` header is sent unless an
allowlist is configured. To allow browser-based MCP clients:
```bash
GBRAIN_HTTP_CORS_ORIGIN=https://claude.ai gbrain serve --http --port 8787
# Multiple origins: comma-separated
GBRAIN_HTTP_CORS_ORIGIN=https://claude.ai,https://your.app gbrain serve --http
```
When the request `Origin` matches the allowlist, the server echoes it
back in `Access-Control-Allow-Origin` (with `Vary: Origin`). Otherwise no
CORS header is sent and the browser blocks the request.
### Rate limiting
Two buckets, both stored in a bounded LRU map (default 10K keys, evicts
least-recently-used on overflow, prunes entries older than 2× the
window):
| Bucket | When it fires | Default | Env var |
|---|---|---|---|
| Pre-auth IP | Before the DB lookup, on every `/mcp` request | 30 req / 60s | `GBRAIN_HTTP_RATE_LIMIT_IP` |
| Post-auth token | After a valid token is resolved | 60 req / 60s | `GBRAIN_HTTP_RATE_LIMIT_TOKEN` |
| LRU cap | Maximum distinct keys across both buckets | 10000 | `GBRAIN_HTTP_RATE_LIMIT_LRU` |
On exhaustion the server returns `429 Too Many Requests` with a
`Retry-After` header.
**Caveat for tunneled deployments (ngrok, Tailscale Funnel, Cloudflare
Tunnel):** all requests share one egress IP, so the pre-auth IP bucket
becomes effectively shared by all clients on that tunnel. The
post-auth token-id bucket is the load-bearing limiter for tunnel-fronted
deployments.
### Reverse-proxy trust
Disabled by default. To honor `X-Forwarded-For` (or `X-Real-IP`) when
gbrain runs behind a trusted reverse proxy:
```bash
GBRAIN_HTTP_TRUST_PROXY=1 gbrain serve --http --port 8787
```
**Critical safety contract:** only set `GBRAIN_HTTP_TRUST_PROXY=1` when
**both** of these are true:
1. gbrain is reachable only via a trusted reverse proxy (not directly
exposed to the internet on the configured port). The simplest
guarantee is to bind gbrain to `127.0.0.1` or a private interface
and have the proxy forward to it.
2. The proxy strips any client-supplied `X-Forwarded-For` and `X-Real-IP`
headers, then sets them itself. (nginx with `proxy_set_header
X-Forwarded-For $remote_addr` does this; Cloudflare and most cloud
load balancers handle it automatically.)
If gbrain is reachable directly AND `GBRAIN_HTTP_TRUST_PROXY=1` is set,
clients can spoof their IP by sending arbitrary `X-Forwarded-For`
headers, defeating the pre-auth IP rate limit. Without the flag, gbrain
ignores all forwarded-for headers and uses the socket peer address,
which is the safe default for direct-exposure deployments.
### Body size cap
Default 1 MiB, stream-counted (chunked transfers without
`Content-Length` are still capped). Override:
```bash
GBRAIN_HTTP_MAX_BODY_BYTES=2097152 gbrain serve --http # 2 MiB
```
Over-cap requests get `413 Payload Too Large` immediately, before any
body is materialized in memory.
### Audit log
Every `/mcp` request writes one row to `mcp_request_log`:
```bash
psql "$DATABASE_URL" -c \
"SELECT created_at, token_name, operation, status, latency_ms
FROM mcp_request_log
ORDER BY created_at DESC LIMIT 100"
```
`status` is one of: `success`, `error`, `auth_failed`, `rate_limited`,
`body_too_large`, `parse_error`, `unknown_method`. Failed-auth rows have
`token_name = NULL`. Inserts are fire-and-forget so audit failures
never block requests.
+444
View File
@@ -1,5 +1,362 @@
# TODOS
## claw-test E2E (v0.22.16 follow-ups)
### Hermes runner — `src/core/claw-test/runners/hermes.ts`
**Priority:** P2
**What:** Add a Hermes implementation of the `AgentRunner` interface. v1 ships only OpenClaw; v1.1 lands hermes once we have real friction reports from openclaw to validate the contract against.
**Why:** Cross-agent diff (`gbrain friction diff --base openclaw --compare hermes`) is the highest-leverage next signal. Friction unique to one agent vs common-to-both separates "agent contract bug" from "gbrain bug" automatically.
**Effort:** S (CC ~30m). Depends on: v1 openclaw runner producing real friction reports first.
---
### Friction analytics suite — `diff` / `trend` / `migration-stub`
**Priority:** P2
**What:** Three new `gbrain friction` subcommands deferred from v1:
- `gbrain friction diff --base <run-or-agent> --compare <run-or-agent>` (cross-agent comparison; ~80 LOC)
- `gbrain friction trend [--since <version-or-date>] [--phase <name>]` (time-series across runs; ~60 LOC)
- `gbrain friction migration-stub [--threshold N]` (clusters friction by phase + tokens, emits `skills/migrations/v[N+1].md` stub; ~150 LOC)
**Why:** Turns point-in-time reports into a slope. Pairs with the v1.1 public scoreboard.
**Effort:** M (CC ~2h total).
---
### Scenario expansion — `supabase-migration` and `supervisor-restart`
**Priority:** P2
**What:** Two more scenarios under `test/fixtures/claw-test-scenarios/`:
- `supabase-migration``gbrain init --pglite` then `gbrain migrate --to supabase`; verifies the cross-engine migration path
- `supervisor-restart` — kill worker mid-job; verify supervisor recovers without data loss
**Why:** These are the other highest-historical-pain regression points (per CLAUDE.md fix-wave history). v1 ships only `fresh-install` + `upgrade-from-v0.18` because Codex flagged that mixing them dilutes the fresh-install signal; v1.1 lands them as separate scenarios.
**Effort:** M (CC ~1h each).
---
### Real v0.18 SQL dump for upgrade scenario
**Priority:** P2
**What:** The `upgrade-from-v0.18` scenario ships scaffolded — `seed/dump.sql` is missing. The harness gracefully no-ops the seed phase when absent, so the scenario currently behaves like fresh-install. v1.1: generate a real v0.18-shape PGLite dump per the procedure documented in `test/fixtures/claw-test-scenarios/upgrade-from-v0.18/seed/README.md`.
**Why:** Without a real seed, the scenario doesn't actually exercise the migration chain forward-walk. That's the whole point of the upgrade scenario — proves issue #239/#243/#266/#357 class regressions stay fixed.
**Effort:** S (CC ~30m once a v0.18 checkout is handy). Depends on: ability to run a v0.18 gbrain build.
---
### Public scoreboard — `gbrain-evals.io/friction`
**Priority:** P3
**What:** Sibling-repo PR in `garrytan/gbrain-evals` that renders friction JSONL into a public dashboard. Friction count per version per agent, line charts over time. v1's JSONL already includes `gbrain_version` + `agent` tags so the scoreboard is a thin layer on top.
**Why:** Marketing surface. Proves install quality is improving release-over-release. The friction loop becomes visible to the world, not just maintainers.
**Effort:** M. Depends on: a working live mode and ≥10 real friction reports.
---
### PTY-mode transcript capture
**Priority:** P3
**What:** `transcript-capture.ts` currently uses plain `child_process.spawn` pipes. Some agents only emit ANSI colors / progress UI on a TTY. v1.1 adds a PTY mode (likely via `node-pty`) so live-mode transcripts capture the full agent UX.
**Why:** Faithful transcripts make the friction → reasoning link more useful. v1 accepts that some agent UI is lost.
**Effort:** S (CC ~30m). Mostly a ~30 LOC swap inside `spawnWithCapture`.
---
### Read-side host-isolation (`$GBRAIN_HOST_HOME`)
**Priority:** P3
**What:** v0.22.16 confined every `~/.gbrain` write site to honor `$GBRAIN_HOME`. But `src/commands/init.ts:299-313` still reads real `~/.claude` / `~/.openclaw` / `~/.codex` / `~/.factory` / `~/.kiro` for module fingerprinting (host detection). Even with write-isolation, a claw-test running on a developer's box discovers their real installed mods. v1.1: add a separate `$GBRAIN_HOST_HOME` override for the read-side detection so the claw-test can run truly hermetic.
**Why:** v1's hermeticity contract is "writes are isolated, reads are not." v1.1 closes the read-side gap.
**Effort:** S (CC ~30m).
---
### Routing-callout sweep — annotate skills the claw-test exercises
**Priority:** P3
**What:** `skills/_friction-protocol.md` is a cross-cutting convention. v1.1: sweep the 46 skills the claw-test actually exercises (setup, brain-ops, query, ingest, smoke-test, the migrations the test covers) and add a `> **Convention:** see [skills/_friction-protocol.md](_friction-protocol.md).` callout via the existing `src/core/dry-fix.ts` shape so DRY auto-fix doesn't fight it.
**Why:** Right now agents only call `gbrain friction log` if they find the protocol skill on their own. The callouts route them there proactively from any harness-exercised skill.
**Effort:** S (CC ~15m).
---
## minions / worker (v0.22.14 follow-ups)
### v0.22.15 — Embed cooperative-abort (HIGHEST PRIORITY — daily pain)
**Priority:** P0
**What:** Plumb `signal: AbortSignal` through `runPhaseEmbed`
`src/commands/embed.ts``embedBatch` in `src/core/embedding.ts`. Check
`signal?.aborted` between OpenAI batch calls (every ~100 texts, ~2s
real-time) and between slugs in the per-slug loop.
**Why:** Embed phase ignores `signal.aborted` between batches today. Job
wall-clock timeout fires → handler keeps running → cycle's finally block
unreachable → `gbrain_cycle_locks` row stays held indefinitely. Every
subsequent autopilot cron cycle sees `cycle_already_running` → skips. Lock
TTL is 30 min; new cycles give up before that. Doctor reports UNHEALTHY.
**The chain in production:** ~5min cron submits cycle → 22K stale pages →
embed phase takes 1015 min → 600s timeout fires → job dead-lettered → embed
keeps running → lock held → all subsequent cycles skip. Garry hits this
DAILY on his production brain.
**Pros:** Closes the daily wedge. Makes timeouts actually effective. Lets
operators bump worker timeouts confidently knowing abort actually stops
work.
**Cons:** Touching the embed hot path; small risk of botching the abort
checks. Mitigation: between-batch granularity (~2s), not per-text (too fine)
or per-slug (too coarse for 500+ chunk slugs).
**Context:** PR #503 (v0.22.14) catches the SYMPTOM (worker stalled, queue
piling up) via self-health-monitoring. This PR catches the CAUSE for one
specific failure class. Both fixes are needed; they're complementary, not
duplicative.
**Files to touch:**
- `src/core/cycle.ts:579``runPhaseEmbed(engine, dryRun)` → add
`signal?: AbortSignal` arg
- `src/core/cycle.ts:803` — pass `opts.signal` through
- `src/commands/embed.ts:~363` — accept signal, check between slugs
- `src/core/embedding.ts:51-56``embedBatch(texts, onProgress?, signal?)`,
check between for-loop iterations of `BATCH_SIZE` slices
**Tests required:**
1. embedBatch checks signal between OpenAI calls; aborts within one batch (~2s)
2. Per-slug loop in `embed.ts` checks signal between slugs
3. End-to-end: cycle handler with embed phase + signal aborted mid-flight →
finally runs → `gbrain_cycle_locks` row deleted
4. Regression: 1K+ chunks scenario — embed does NOT block lock release when
timeout fires
**Effort:** M (human: ~3 hr / CC: ~30 min).
**Depends on / blocked by:** Nothing. v0.22.14 ships first.
### v0.23+ — Bare-worker engine reconnect parity with supervisor
**Priority:** P2
**What:** Extract the supervisor's reconnect-then-fail pattern into
`MinionWorker` so bare workers can retry transient DB blips before exiting.
Today the supervisor calls `engine.reconnect()` after 3 consecutive DB health
failures (#406); the bare worker just emits `'unhealthy'` and the CLI calls
`process.exit(1)`.
**Why:** Bare-worker behavior is more disruptive than supervised behavior on
transient PgBouncer blips. A bare worker restarts the entire process; a
supervised worker just reconnects the pool. Operationally the supervisor
approach is gentler (no in-flight job loss, no PM restart latency).
**Pros:** Unifies bare and supervised behavior. Reduces process churn on
transient network blips.
**Cons:** More code in MinionWorker; risk of reconnect masking a real
problem. Mitigation: cap retry attempts, fall through to `'unhealthy'`
emission after the cap.
**Context:** Filed during v0.22.14 plan-eng-review. The asymmetry is
documented in v0.22.14 CHANGELOG as deliberate; this TODO captures the
"unify someday" intent.
**Effort:** S (human: ~2 hr / CC: ~20 min).
**Depends on / blocked by:** Nothing.
### v0.23+ — `minion_workers` heartbeat table for queue_health doctor (B7)
**Priority:** P3
**What:** Add a `minion_workers` table (`worker_id` PK, `hostname`,
`last_heartbeat`, `queue`, `concurrency`, `started_at`) so the existing
`queue_health` doctor check (Postgres path) can detect dead workers via
heartbeat staleness instead of relying on the indirect `lock_until` proxy.
**Why:** v0.19.1 added `queue_health` checks for stalled-active jobs and
waiting-depth threshold. The worker-heartbeat subcheck was deferred (B7)
because the `lock_until`-on-active-jobs proxy can't distinguish "worker
exited cleanly" from "worker idle" — a check that cries wolf erodes trust
in every doctor check. With a real heartbeat row, doctor can say "no worker
seen in N intervals" with confidence.
**Pros:** Doctor's `queue_health` becomes ground-truth. Detects "worker
container died but cron didn't restart it" scenario.
**Cons:** New table, schema migration, every health-tick UPSERTs. Costs
a write per worker per minute (default).
**Context:** Filed during v0.22.14 plan-eng-review. PR #503's self-health
monitoring is the worker-side liveness; this would be the queue-side
ground-truth.
**Effort:** M (human: ~1 day / CC: ~1 hr).
**Depends on / blocked by:** Schema migration system; nothing else.
## sync (v0.22.13 follow-up — PR #490 review)
### D-PR490-1 — Plumb resolved `database_url` through `SyncOpts`
**Priority:** P3
**What:** Add `database_url?: string` (or a richer `resolvedConnection` shape) to
`SyncOpts` and have the caller (`runSync`, the cycle handler, the jobs handler)
populate it from the active engine instead of having `performSync` /
`performFullSync` / `import.ts` each call `loadConfig()` separately. Today every
sync run hits the config file three times.
**Why:** v0.18 multi-source brains can in principle run different sources against
different `database_url` endpoints (or different per-source overrides via
`sources.config_jsonb`). Right now `loadConfig()` returns the global config, and
that always matches the engine in practice — but the convention papers over a
real divergence the moment someone wants per-source connection settings. Folding
the resolution into `SyncOpts` makes the worker-engine creation in `sync.ts` and
`import.ts` deterministic from `SyncOpts` alone.
**Pros:**
- Removes 3 redundant `loadConfig()` calls per sync.
- Makes `performSync` / `performFullSync` side-effect-free with respect to the
on-disk config file.
- Sets up for per-source `database_url` overrides without further refactor.
- Makes the v0.22.13 belt-and-suspenders fallback (PR #490 Q3) cleaner — no
more `!config?.database_url` short-circuit inside the parallel branch.
**Cons:**
- API-shape change to `SyncOpts` (mild; not externally exported).
- Touching three callers (`runSync`, jobs handler, `cycle.ts` `runPhaseSync`).
- Only worth doing when paired with a per-source override story; otherwise
it's just plumbing.
**Context:** Surfaced during the PR #490 plan-eng-review (parallel sync).
Deferred because it isn't on the v0.22.13 critical path. The same pattern would
benefit the cycle handler and the autopilot daemon. See the plan-eng-review
decisions log: A4 = "Defer; file as TODO."
**Depends on / blocked by:** Nothing structural. Best paired with the v0.18
per-source `config_jsonb` work if/when that lands.
## sync error-code classification (PR #501 follow-ups)
### Plumb structured `ParseValidationCode` through `ImportResult`
**Priority:** P2
**What:** Replace the regex-on-error-message path in `src/core/sync.ts:classifyErrorCode`
with a structured `code` field threaded through `ImportResult` from the parse layer.
Three changes:
1. `src/core/import-file.ts:362` — call `parseMarkdown(content, relativePath, { validate: true, expectedSlug })`
so `parsed.errors[0].code` is populated.
2. `src/core/import-file.ts` — add `code?: string` to `ImportResult`. Promote the
structured code (or `'SLUG_MISMATCH'` when the existing expectedSlug check trips)
into the result envelope alongside `error`.
3. `src/commands/sync.ts:488` — extend `failedFiles` shape with `code?: string`.
`recordSyncFailures` already accepts the field; the only thing missing is the
capture site populating it.
4. `src/core/sync.ts:classifyErrorCode` — keep as a fallback for un-coded errors
(DB exceptions, generic catches). Primary path reads the structured code.
**Why:** The repo already has `ParseValidationCode` + `ParseValidationError` in
`src/core/markdown.ts:5-18`, and three other consumers (`src/commands/lint.ts:72`,
`src/commands/frontmatter.ts:148`, `src/core/brain-writer.ts:314`) read structured
errors directly. Sync is the outlier — it calls `parseMarkdown` without validation
and reverse-engineers codes via regex. PR #501 shipped that regex out of pragmatism;
this TODO removes ~50% of `classifyErrorCode` and eliminates a class of false-positives.
**Pros:**
- One source of truth for parse codes (the enum in `markdown.ts`).
- Eliminates regex fragility — adding a new validation code in `markdown.ts`
automatically flows to sync without a new regex.
- Closes the case where canonical messages (`File is empty...`, `No closing ---...`)
don't match aspirational regex patterns.
**Cons:** Touches `ImportResult` interface, which ripples through `src/commands/import.ts:105`,
`src/commands/sync.ts:498-510`, `src/core/cycle.ts`, brain-writer reconciler.
**Context:** PR #501 documented this as P3 in the eng review at
`~/.claude/plans/then-codex-synchronous-toucan.md`. Codex's outside-voice review
agreed independently. The fix is small — ~50 lines including tests + downstream
call sites — and it's the correct architectural endpoint.
**Effort:** M (human: ~2 hr / CC: ~20 min).
**Depends on / blocked by:** Nothing.
### CHANGELOG migration note for `acknowledgeSyncFailures()` shape change
**Priority:** P0 — required at /ship time
**What:** When PR #501 ships, the release CHANGELOG entry MUST include this
`### For contributors` block:
```markdown
### For contributors
`acknowledgeSyncFailures()` now returns `{count, summary}` instead of `number`.
If you import this directly from `gbrain/sync`, replace `n` with `result.count`
and use `result.summary` for the new code-grouped breakdown.
```
**Why:** The function is exported from `src/core/sync.ts:433` and reachable via
the package exports map. External TS consumers (gbrain-evals, host agent forks)
that imported it got `number` and now get an object — silent type break.
**Effort:** XS (human: ~1 min). Just don't forget.
**Depends on / blocked by:** PR #501 ship.
### Concurrent-safe ack of `~/.gbrain/sync-failures.jsonl`
**Priority:** P3
**What:** Two concurrent `gbrain sync` runs hitting `acknowledgeSyncFailures()`
can clobber each other. The function does a whole-file `writeFileSync` rewrite
(`src/core/sync.ts:433-455`); `recordSyncFailures()` does independent
`appendFileSync` (`src/core/sync.ts:395-416`). Concurrent ack + append can lose rows.
**Why:** Pre-existing — predates PR #501. Real risk only on autopilot setups where
multiple sync invocations might overlap (rare today, more likely as multi-source
sync matures).
**Fix sketch:** Atomic rename pattern (write to `sync-failures.jsonl.tmp`, then
`renameSync`) plus a file lock for the read-modify-write cycle. Or move the
acknowledged-set to the DB.
**Effort:** S (human: ~1 hr / CC: ~10 min).
**Depends on / blocked by:** Nothing.
## test-infra
### Parallel-load timeout flake on v0.21 PGLite-heavy tests
**Priority:** P0
**What:** 22 tests added in v0.21.0 (Code Cathedral II) consistently fail in the full `bun test` run with timeout-pattern elapsed times of 7-10s, but pass in isolation. Every failing test calls `engine.initSchema()` in `beforeAll` without a timeout extension. Under parallel load (168 test files now run concurrently after v0.21 added ~24 new files), `initSchema` exceeds bun's default 5s `beforeAll` timeout.
Affected files include (non-exhaustive): `test/sync-strategy.test.ts`, `test/cathedral-ii-brainbench.test.ts`, `test/code-edges.test.ts`, `test/reindex-code.test.ts`, `test/reconcile-links.test.ts`, `test/two-pass.test.ts`, `test/parent-symbol-path.test.ts`, `test/pglite-v0_19.test.ts`.
**Why:** Currently triaged as "skip pre-existing, ship anyway" but that's not a real fix. Blocks /ship for anyone whose CHANGELOG-time test run sees them.
**Pros:** Fixing it lets /ship run cleanly without manual triage every release.
**Cons:** ~22 file edits adding `beforeAll(async () => {...}, 30000)` is mechanical but dull.
**Context:** Same pattern fixed in v0.20.5 wave for `test/e2e/minions-shell-pglite.test.ts`. Single-file repro: each fails in `bun test`, passes in `bun test <file>`. Reproduces with my changes stashed, so it's on master.
**Effort:** S (human: ~30 min / CC: ~5 min). Mechanical: grep for `beforeAll(async () => {` in affected files, add `, 30000)` argument.
**Depends on / blocked by:** Nothing.
## resolver / check-resolvable (v0.22.4 follow-ups)
### D10 — Extend `check-resolvable` to parse RESOLVER.md disambiguation rules
@@ -564,3 +921,90 @@ iteration's residuals.
**Effort estimate:** S (human: ~2 hr / CC: ~10 min).
**Priority:** P3.
**Depends on:** The above caller-opt-in retry (#1) is the natural co-lander since both touch the same error-classification surface.
## remote MCP / HTTP transport (v0.22.7 follow-ups)
### Audit-log write amplification on rejected `/mcp` traffic
**What:** `src/mcp/http-transport.ts` writes a row to `mcp_request_log` for every
incoming `/mcp` request, including rate-limited (429), oversized (413), and
auth-failed (401) traffic. Under sustained attack the IP rate limit caps audit
writes per IP at 30/min, but at scale (10K distinct IPs) that's still 300K
inserts/min. Two follow-ups: (1) instrument the audit-write rate so we can see
the actual production volume; (2) consider a separate "rejected" table or
sampling for failed-auth rows so the success-path audit table doesn't get
swamped.
**Why:** Codex flagged this during the v0.22.7 ship adversarial review. We kept
the full audit on purpose — forensic data of an attack is valuable — but want
to revisit once we have real volume numbers.
**Pros:** Bounds DB write volume under attack. Keeps the success-path audit
table small enough for fast queries.
**Cons:** Adds a second table or a sampling rule. Not free complexity. Probably
not worth it until production hits a real attack pattern.
**Context:** `src/mcp/http-transport.ts:222,235,245` (the three audit-on-reject
call sites) + `src/schema.sql:342` (the unbounded table).
**Effort estimate:** M (human: ~half day / CC: ~30 min once we have volume data).
**Priority:** P3 — wait for evidence.
**Depends on:** Production telemetry on `mcp_request_log` insert rate.
### `validateParams` doesn't check enum values or array item types
**What:** `src/mcp/dispatch.ts:27` (extracted from `src/mcp/server.ts` in
v0.22.7) only checks top-level JS types. Operations declare `enum` constraints
(e.g. `direction: 'in' | 'out' | 'both'`) and array `items: { type: ... }`
schemas in `src/core/operations.ts`, but `validateParams` ignores both. Bad
inputs still reach handlers — concretely, an invalid `direction` falls through
the engine's else branch at `src/core/postgres-engine.ts:954`, widening
traversal unexpectedly; malformed `pages_updated` arrays could be written as
garbage JSONB.
**Why:** Codex flagged this during the v0.22.7 ship adversarial review. The
validator was lifted verbatim from the pre-existing stdio path during the
dispatch.ts extraction — same gap exists on the stdio MCP server today, so
this isn't a v0.22.7 regression. Still worth tightening, since "shared
validation" is now the architectural guarantee both transports rely on.
**Pros:** Better defense-in-depth at the MCP boundary. Catches malformed agent
inputs before the engine layer has to.
**Cons:** Need to walk every operation's param schema and decide which enum
violations are user-facing errors vs internal bugs. May need a typed Zod-style
schema layer to do this cleanly.
**Context:** `src/mcp/dispatch.ts:27` + `src/core/operations.ts` (param defs).
Same gap pre-existed on stdio MCP path.
**Effort estimate:** M (human: ~half day / CC: ~30 min if we use the existing
ParamDef shape; XL if a Zod migration is the chosen direction).
**Priority:** P2.
**Depends on:** Whether we want to keep the lightweight ParamDef shape or
migrate to typed schemas.
### Streaming MCP tool support (re-add SSE based on Accept header)
**What:** v0.22.7 dropped SSE entirely from `gbrain serve --http` because no
current MCP tool streams. When the first streaming tool ships (long-running
agent delegation as an MCP tool, `resources/subscribe`, `sampling/createMessage`),
re-add SSE in `/mcp` based on the `Accept` header per the Streamable HTTP
transport spec. ~30 lines + spec compliance test.
**Why:** Removing SSE simplified the v0.22.7 transport (one response path,
fewer test cases). Adding it back when actually needed is cheap and keeps the
code lean in the meantime.
**Effort estimate:** S (human: ~2 hr / CC: ~15 min).
**Priority:** P3 — wait for the first streaming tool.
**Depends on:** A streaming MCP tool actually existing.
### `access_tokens.scopes` enforcement
**What:** The `access_tokens` schema has had a `scopes TEXT[]` column since
migration v4 (`src/core/migrate.ts:84`), but nothing enforces it. v0.22.7's
`gbrain auth create` doesn't accept a `--scopes` flag, and `dispatchToolCall`
doesn't gate on scopes. Adding per-tool scope enforcement would let
"claude-desktop-readonly" and "ingest-only" tokens exist.
**Effort estimate:** M (human: ~1 day / CC: ~30 min for the schema-aware gate).
**Priority:** P3.
**Depends on:** Nothing.
+1 -1
View File
@@ -1 +1 @@
0.22.5
0.22.16
+13 -10
View File
@@ -20,6 +20,7 @@
},
"devDependencies": {
"@types/bun": "latest",
"bun-types": "^1.3.13",
"typescript": "^5.6.0",
},
},
@@ -220,7 +221,7 @@
"@types/bun": ["@types/bun@1.3.11", "", { "dependencies": { "bun-types": "1.3.11" } }, "sha512-5vPne5QvtpjGpsGYXiFyycfpDF2ECyPcTSsFBMa0fraoxiQyMJ3SmuQIGhzPg2WJuWxVBoxWJ2kClYTcw/4fAg=="],
"@types/node": ["@types/node@18.19.130", "", { "dependencies": { "undici-types": "~5.26.4" } }, "sha512-GRaXQx6jGfL8sKfaIDD6OupbIHBr9jv7Jnaml9tB7l4v068PAOXqfcujMMo5PhbIs6ggR1XODELqahT2R8v0fg=="],
"@types/node": ["@types/node@25.5.2", "", { "dependencies": { "undici-types": "~7.18.0" } }, "sha512-tO4ZIRKNC+MDWV4qKVZe3Ql/woTnmHDr5JD8UI5hn2pwBrHEwOEMZK7WlNb5RKB6EoJ02gwmQS9OrjuFnZYdpg=="],
"@types/node-fetch": ["@types/node-fetch@2.6.13", "", { "dependencies": { "@types/node": "*", "form-data": "^4.0.4" } }, "sha512-QGpRVpzSaUs30JBSGPjOg4Uveu384erbHBoT1zeONvyCfwQxIkUshLAOqN/k9EjGviPRmWTTe6aH2qySWKTVSw=="],
@@ -242,7 +243,7 @@
"bowser": ["bowser@2.14.1", "", {}, "sha512-tzPjzCxygAKWFOJP011oxFHs57HzIhOEracIgAePE4pqB3LikALKnSzUyU4MGs9/iCEUuHlAJTjTc5M+u7YEGg=="],
"bun-types": ["bun-types@1.3.11", "", { "dependencies": { "@types/node": "*" } }, "sha512-1KGPpoxQWl9f6wcZh57LvrPIInQMn2TQ7jsgxqpRzg+l0QPOFvJVH7HmvHo/AiPgwXy+/Thf6Ov3EdVn1vOabg=="],
"bun-types": ["bun-types@1.3.13", "", { "dependencies": { "@types/node": "*" } }, "sha512-QXKeHLlOLqQX9LgYaHJfzdBaV21T63HhFJnvuRCcjZiaUDpbs5ED1MgxbMra71CsryN/1dAoXuJJJwIv/2drVA=="],
"bytes": ["bytes@3.1.2", "", {}, "sha512-/Nf7TyzTx6S3yRJObOAV7956r8cr2+Oj8AC5dt8wSP3BQAoeX58NoHyCU8P8zGkNXStjTSi6fzO6F0pBdcYbEg=="],
@@ -466,7 +467,7 @@
"typescript": ["typescript@5.9.3", "", { "bin": { "tsc": "bin/tsc", "tsserver": "bin/tsserver" } }, "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw=="],
"undici-types": ["undici-types@5.26.5", "", {}, "sha512-JlCMO+ehdEIKqlFxk6IfVoAUVmgz7cU7zD/h9XZ0qzeosSHmUJVOzSQvvYSYWXkFXC+IfLKSIffhv0sVZup6pA=="],
"undici-types": ["undici-types@7.18.2", "", {}, "sha512-AsuCzffGHJybSaRrmr5eHr81mwJU3kjw6M+uprWvCXiNeN9SOGwQ3Jn8jb8m3Z6izVgknn1R0FTCEAP2QrLY/w=="],
"unpipe": ["unpipe@1.0.0", "", {}, "sha512-pjy2bYhSsufwWlKwPc+l3cN7+wuJlK6uz0YdJEOlQDbl6jo/YlPi4mb8agUkVC8BF7V8NuzeyPNqRksA3hztKQ=="],
@@ -488,30 +489,32 @@
"zod-to-json-schema": ["zod-to-json-schema@3.25.2", "", { "peerDependencies": { "zod": "^3.25.28 || ^4" } }, "sha512-O/PgfnpT1xKSDeQYSCfRI5Gy3hPf91mKVDuYLUHZJMiDFptvP41MSnWofm8dnCm0256ZNfZIM7DSzuSMAFnjHA=="],
"@anthropic-ai/sdk/@types/node": ["@types/node@18.19.130", "", { "dependencies": { "undici-types": "~5.26.4" } }, "sha512-GRaXQx6jGfL8sKfaIDD6OupbIHBr9jv7Jnaml9tB7l4v068PAOXqfcujMMo5PhbIs6ggR1XODELqahT2R8v0fg=="],
"@aws-crypto/sha1-browser/@smithy/util-utf8": ["@smithy/util-utf8@2.3.0", "", { "dependencies": { "@smithy/util-buffer-from": "^2.2.0", "tslib": "^2.6.2" } }, "sha512-R8Rdn8Hy72KKcebgLiv8jQcQkXoLMOGGv5uI1/k0l+snqkOzQ1R0ChUBCxWMlBsFMekWjq0wRudIweFs7sKT5A=="],
"@aws-crypto/sha256-browser/@smithy/util-utf8": ["@smithy/util-utf8@2.3.0", "", { "dependencies": { "@smithy/util-buffer-from": "^2.2.0", "tslib": "^2.6.2" } }, "sha512-R8Rdn8Hy72KKcebgLiv8jQcQkXoLMOGGv5uI1/k0l+snqkOzQ1R0ChUBCxWMlBsFMekWjq0wRudIweFs7sKT5A=="],
"@aws-crypto/util/@smithy/util-utf8": ["@smithy/util-utf8@2.3.0", "", { "dependencies": { "@smithy/util-buffer-from": "^2.2.0", "tslib": "^2.6.2" } }, "sha512-R8Rdn8Hy72KKcebgLiv8jQcQkXoLMOGGv5uI1/k0l+snqkOzQ1R0ChUBCxWMlBsFMekWjq0wRudIweFs7sKT5A=="],
"@types/node-fetch/@types/node": ["@types/node@25.5.2", "", { "dependencies": { "undici-types": "~7.18.0" } }, "sha512-tO4ZIRKNC+MDWV4qKVZe3Ql/woTnmHDr5JD8UI5hn2pwBrHEwOEMZK7WlNb5RKB6EoJ02gwmQS9OrjuFnZYdpg=="],
"bun-types/@types/node": ["@types/node@25.5.2", "", { "dependencies": { "undici-types": "~7.18.0" } }, "sha512-tO4ZIRKNC+MDWV4qKVZe3Ql/woTnmHDr5JD8UI5hn2pwBrHEwOEMZK7WlNb5RKB6EoJ02gwmQS9OrjuFnZYdpg=="],
"@types/bun/bun-types": ["bun-types@1.3.11", "", { "dependencies": { "@types/node": "*" } }, "sha512-1KGPpoxQWl9f6wcZh57LvrPIInQMn2TQ7jsgxqpRzg+l0QPOFvJVH7HmvHo/AiPgwXy+/Thf6Ov3EdVn1vOabg=="],
"form-data/mime-types": ["mime-types@2.1.35", "", { "dependencies": { "mime-db": "1.52.0" } }, "sha512-ZDY+bPm5zTTF+YpCrAU9nK0UgICYPT0QtT1NZWFv4s++TNkcgVaT0g6+4R2uI4MjQjzysHB1zxuWL50hzaeXiw=="],
"openai/@types/node": ["@types/node@18.19.130", "", { "dependencies": { "undici-types": "~5.26.4" } }, "sha512-GRaXQx6jGfL8sKfaIDD6OupbIHBr9jv7Jnaml9tB7l4v068PAOXqfcujMMo5PhbIs6ggR1XODELqahT2R8v0fg=="],
"@anthropic-ai/sdk/@types/node/undici-types": ["undici-types@5.26.5", "", {}, "sha512-JlCMO+ehdEIKqlFxk6IfVoAUVmgz7cU7zD/h9XZ0qzeosSHmUJVOzSQvvYSYWXkFXC+IfLKSIffhv0sVZup6pA=="],
"@aws-crypto/sha1-browser/@smithy/util-utf8/@smithy/util-buffer-from": ["@smithy/util-buffer-from@2.2.0", "", { "dependencies": { "@smithy/is-array-buffer": "^2.2.0", "tslib": "^2.6.2" } }, "sha512-IJdWBbTcMQ6DA0gdNhh/BwrLkDR+ADW5Kr1aZmd4k3DIF6ezMV4R2NIAmT08wQJ3yUK82thHWmC/TnK/wpMMIA=="],
"@aws-crypto/sha256-browser/@smithy/util-utf8/@smithy/util-buffer-from": ["@smithy/util-buffer-from@2.2.0", "", { "dependencies": { "@smithy/is-array-buffer": "^2.2.0", "tslib": "^2.6.2" } }, "sha512-IJdWBbTcMQ6DA0gdNhh/BwrLkDR+ADW5Kr1aZmd4k3DIF6ezMV4R2NIAmT08wQJ3yUK82thHWmC/TnK/wpMMIA=="],
"@aws-crypto/util/@smithy/util-utf8/@smithy/util-buffer-from": ["@smithy/util-buffer-from@2.2.0", "", { "dependencies": { "@smithy/is-array-buffer": "^2.2.0", "tslib": "^2.6.2" } }, "sha512-IJdWBbTcMQ6DA0gdNhh/BwrLkDR+ADW5Kr1aZmd4k3DIF6ezMV4R2NIAmT08wQJ3yUK82thHWmC/TnK/wpMMIA=="],
"@types/node-fetch/@types/node/undici-types": ["undici-types@7.18.2", "", {}, "sha512-AsuCzffGHJybSaRrmr5eHr81mwJU3kjw6M+uprWvCXiNeN9SOGwQ3Jn8jb8m3Z6izVgknn1R0FTCEAP2QrLY/w=="],
"bun-types/@types/node/undici-types": ["undici-types@7.18.2", "", {}, "sha512-AsuCzffGHJybSaRrmr5eHr81mwJU3kjw6M+uprWvCXiNeN9SOGwQ3Jn8jb8m3Z6izVgknn1R0FTCEAP2QrLY/w=="],
"form-data/mime-types/mime-db": ["mime-db@1.52.0", "", {}, "sha512-sPU4uV7dYlvtWJxwwxHD0PuihVNiE7TyAbQ5SWxDCB9mUYvOgroQOwYQQOKPJ8CIbE+1ETVlOoK1UC2nU3gYvg=="],
"openai/@types/node/undici-types": ["undici-types@5.26.5", "", {}, "sha512-JlCMO+ehdEIKqlFxk6IfVoAUVmgz7cU7zD/h9XZ0qzeosSHmUJVOzSQvvYSYWXkFXC+IfLKSIffhv0sVZup6pA=="],
"@aws-crypto/sha1-browser/@smithy/util-utf8/@smithy/util-buffer-from/@smithy/is-array-buffer": ["@smithy/is-array-buffer@2.2.0", "", { "dependencies": { "tslib": "^2.6.2" } }, "sha512-GGP3O9QFD24uGeAXYUjwSTXARoqpZykHadOmA8G5vfJPK0/DC67qa//0qvqrJzL1xc8WQWX7/yc7fwudjPHPhA=="],
"@aws-crypto/sha256-browser/@smithy/util-utf8/@smithy/util-buffer-from/@smithy/is-array-buffer": ["@smithy/is-array-buffer@2.2.0", "", { "dependencies": { "tslib": "^2.6.2" } }, "sha512-GGP3O9QFD24uGeAXYUjwSTXARoqpZykHadOmA8G5vfJPK0/DC67qa//0qvqrJzL1xc8WQWX7/yc7fwudjPHPhA=="],
+10 -7
View File
@@ -1,8 +1,9 @@
# Remote MCP Deployment Options
GBrain's MCP server runs via `gbrain serve` (stdio transport). To make it
accessible from other devices and AI clients, you need an HTTP wrapper and
a public tunnel. Here are your options.
accessible from other devices and AI clients, run `gbrain serve --http`
(built-in HTTP transport with bearer auth, Postgres-only ... see
[DEPLOY.md](DEPLOY.md)) behind a public tunnel. Here are your tunnel options.
## ngrok (recommended)
@@ -13,8 +14,9 @@ a public tunnel. Here are your options.
# 1. Install ngrok
brew install ngrok
# 2. Start your MCP server (behind an HTTP wrapper)
# See docs/mcp/DEPLOY.md for the server setup
# 2. Start the built-in HTTP transport
gbrain serve --http --port 8787
# See docs/mcp/DEPLOY.md for token setup
# 3. Expose via ngrok
ngrok http 8787 --url your-brain.ngrok.app
@@ -59,6 +61,7 @@ Both run Bun natively. No bundling, no Deno, no cold start, no timeout limits.
| All 30 operations | Yes | Yes | Yes |
| Setup time | 5 min | 10 min | 15 min |
**Note:** `gbrain serve --http` (built-in HTTP transport) is planned but not yet
implemented. Currently, remote MCP requires a custom HTTP wrapper around `gbrain serve`.
See [DEPLOY.md](DEPLOY.md) for details.
**Note:** `gbrain serve --http` is the built-in HTTP transport (v0.22.7+). Bearer auth
against the `access_tokens` table, default-deny CORS, two-bucket rate limit, body cap,
per-request audit log. Postgres-only by design (PGLite is local-only). See
[DEPLOY.md](DEPLOY.md) and [SECURITY.md](../../SECURITY.md) for env vars and tunables.
+1 -1
View File
@@ -21,7 +21,7 @@ claude mcp add gbrain -t http \
```
Replace `YOUR-DOMAIN` with your ngrok domain and `YOUR_TOKEN` with a token
from `bun run src/commands/auth.ts create "claude-code"`.
from `gbrain auth create "claude-code"`.
## Verify
+1 -1
View File
@@ -12,7 +12,7 @@ For Team/Enterprise plans, an org Owner adds the connector:
https://YOUR-DOMAIN.ngrok.app/mcp
```
3. Add Bearer token authentication in Advanced Settings
(create one with `bun run src/commands/auth.ts create "cowork"`)
(create one with `gbrain auth create "cowork"`)
4. Save
Note: Cowork connects from Anthropic's cloud, not your device. Your server
+1 -1
View File
@@ -16,7 +16,7 @@ Remote HTTP servers must be added through the GUI.
Replace `YOUR-DOMAIN` with your ngrok domain (see
[ngrok-tunnel recipe](../../recipes/ngrok-tunnel.md) for setup).
5. Set authentication to **Bearer Token** and paste your token
(create one with `bun run src/commands/auth.ts create "claude-desktop"`)
(create one with `gbrain auth create "claude-desktop"`)
6. Save
## Verify
+21 -14
View File
@@ -1,8 +1,13 @@
# Deploy GBrain Remote MCP Server
> **v0.22.7+:** Use `gbrain serve --http` for remote access. It includes built-in
> bearer token auth, default-deny CORS, two-bucket rate limiting, body cap, and
> per-request audit log. **Postgres-only** (PGLite is local-only by design).
> See [SECURITY.md](../../SECURITY.md) for env vars and tunable defaults.
Access your brain from any device, any AI client. GBrain's MCP server runs locally
via `gbrain serve` (stdio). For remote access, wrap it in an HTTP server behind a
public tunnel.
via `gbrain serve` (stdio). For remote access, expose it via the built-in HTTP
transport behind a public tunnel.
## Two Paths
@@ -13,21 +18,23 @@ gbrain serve
```
Works with Claude Code, Cursor, Windsurf, and any MCP client that supports stdio.
No server, no tunnel, no token needed.
No server, no tunnel, no token needed. Works on both PGLite and Postgres engines.
### Remote (any device, any AI client)
### Remote (any device, any AI client) — Postgres only
```
Your AI client (Claude Desktop, Perplexity, etc.)
→ ngrok tunnel (https://YOUR-DOMAIN.ngrok.app)
Your HTTP server (wraps gbrain serve)
Supabase Postgres (via pooler connection string)
gbrain serve --http (built-in transport with bearer auth)
→ Postgres (pooler connection or self-hosted)
```
This requires:
1. A machine running `gbrain serve` behind an HTTP wrapper
2. A public tunnel (ngrok, Tailscale, or cloud host)
3. Bearer token auth for security
1. A Postgres-backed brain (the `access_tokens` table only exists on Postgres;
running `gbrain serve --http` against a PGLite install fails fast at startup)
2. A machine running `gbrain serve --http`
3. A public tunnel (ngrok, Tailscale, or cloud host)
4. A bearer token created via `gbrain auth create <name>`
## Remote Setup
@@ -46,13 +53,13 @@ ngrok http 8787 --url your-brain.ngrok.app # Hobby tier for fixed domain
```bash
# Create a token for each client
bun run src/commands/auth.ts create "claude-desktop"
gbrain auth create "claude-desktop"
# List all tokens
bun run src/commands/auth.ts list
gbrain auth list
# Revoke a token
bun run src/commands/auth.ts revoke "claude-desktop"
gbrain auth revoke "claude-desktop"
```
Tokens are per-client. Create one for each device/app. Revoke individually
@@ -68,7 +75,7 @@ if compromised. Tokens are stored SHA-256 hashed in your database.
### 4. Verify
```bash
bun run src/commands/auth.ts test \
gbrain auth test \
https://YOUR-DOMAIN.ngrok.app/mcp \
--token YOUR_TOKEN
```
@@ -96,7 +103,7 @@ Funnel, and cloud hosts (Fly.io, Railway).
Include the Authorization header: `Authorization: Bearer YOUR_TOKEN`
**"invalid_token" error**
Run `bun run src/commands/auth.ts list` to see active tokens.
Run `gbrain auth list` to see active tokens.
**"service_unavailable" error**
Database connection failed. Check your Supabase dashboard for outages.
+1 -1
View File
@@ -10,7 +10,7 @@ Perplexity Computer supports remote MCP servers with bearer token authentication
- **URL:** `https://YOUR-DOMAIN.ngrok.app/mcp`
- **Authentication:** API Key / Bearer Token
- **Token:** your GBrain access token
(create one with `bun run src/commands/auth.ts create "perplexity"`)
(create one with `gbrain auth create "perplexity"`)
4. Save
Replace `YOUR-DOMAIN` with your ngrok domain (see
+210
View File
@@ -0,0 +1,210 @@
# Storage Tiering: db-tracked vs db-only directories
## Overview
GBrain supports storage tiering to separate version-controlled content from bulk machine-generated data. This prevents git repositories from becoming bloated with large amounts of automatically generated content while still preserving it in the database.
> Note on naming: prior to v0.22.11 the keys were `git_tracked` / `supabase_only`. The canonical names are now `db_tracked` / `db_only` (engine-agnostic — works on both PGLite and Postgres). The deprecated keys still load with a once-per-process warning. Run `gbrain doctor --fix` for an automated rename when that path lands.
## Configuration
Add a `storage` section to your `gbrain.yml` file in the brain repository root:
```yaml
storage:
# Directories that are version-controlled (human-edited, committed to git).
db_tracked:
- people/
- companies/
- deals/
- concepts/
- yc/
- ideas/
- projects/
# Directories persisted via the brain database only (bulk machine-generated
# content). Written to disk as a local cache but not committed to git;
# `gbrain sync` auto-manages .gitignore for these paths. `gbrain export
# --restore-only` repopulates missing files from the database.
db_only:
- media/x/
- media/articles/
- meetings/transcripts/
```
Path requirements:
- Each directory must end with `/` for canonical form. The validator auto-normalizes missing trailing slashes (one-time info note shows what changed).
- A directory cannot appear in both tiers — that's a tier-overlap error and `loadStorageConfig` throws `StorageConfigError`. Edit `gbrain.yml` to remove the overlap and try again.
## Behavior Changes
### 1. `gbrain sync` — automatic .gitignore management
When storage configuration is present, `gbrain sync` automatically manages `.gitignore` entries on every successful sync:
- Adds missing `db_only` directory patterns to `.gitignore`.
- Idempotent — re-running adds no duplicate entries.
- Stable comment header so the managed block is grep-able.
- Skipped on `--dry-run` (don't mutate disk in preview mode).
- Skipped on `blocked_by_failures` status (sync state is inconsistent).
- Skipped when the repo is a git submodule (`.git` is a file, not a directory) — submodule .gitignore changes don't survive parent updates. A warning explains.
- Skipped entirely when `GBRAIN_NO_GITIGNORE=1` is set (escape hatch for shared-repo setups where a maintainer wants gbrain to leave .gitignore alone).
- Failures (write permission denied, etc.) are caught and logged, never crash sync.
Example `.gitignore` addition:
```gitignore
# Auto-managed by gbrain (db_only directories)
media/x/
media/articles/
meetings/transcripts/
```
### 2. `gbrain export --restore-only` — repopulate missing db_only files
```bash
# Restore only missing db_only files from the database.
gbrain export --restore-only --repo /path/to/brain
# Filter by page type.
gbrain export --restore-only --type media --repo /path/to/brain
# Filter by slug prefix.
gbrain export --restore-only --slug-prefix media/x/ --repo /path/to/brain
# Combine filters.
gbrain export --restore-only --type media --slug-prefix media/x/ --repo /path/to/brain
```
The `--restore-only` flag:
- Resolves repoPath via the chain `--repo` → typed `sources.getDefault()` → hard error.
Never falls through to the current directory.
- Only exports pages that match `db_only` patterns AND are missing from disk.
- Ideal for container restart recovery and fresh clones.
### 3. `gbrain storage status` — storage-tier health dashboard
```bash
# Human-readable status.
gbrain storage status --repo /path/to/brain
# JSON output for scripts and orchestrators.
gbrain storage status --repo /path/to/brain --json
```
Output includes:
- Total page counts by storage tier.
- Disk usage breakdown by tier.
- Missing files that need restoration (top 10 shown; full list in `--json`).
- Configuration validation warnings.
- Current tier directory listing.
Example output:
```
Storage Status
==============
Repository: /data/brain
Total pages: 15,243
Storage Tiers:
-------------
DB tracked: 2,156 pages
DB only: 12,887 pages
Unspecified: 200 pages
Disk Usage:
-----------
DB tracked: 45.2 MB
DB only: 2.1 GB
Missing Files (need restore):
-----------------------------
media/x/tweet-1234567890
media/x/tweet-0987654321
... and 47 more
Use: gbrain export --restore-only --repo "/data/brain"
Configuration:
--------------
DB tracked directories:
- people/
- companies/
- deals/
DB-only directories:
- media/x/
- media/articles/
- meetings/transcripts/
```
## Validation
`loadStorageConfig` runs `normalizeAndValidateStorageConfig` after parsing:
- Auto-fixes (silent, with one-time info note showing what changed):
- Missing trailing `/` is added: `'media/x'``'media/x/'`.
- Throws `StorageConfigError` (caller sees a clean exit-1 with actionable message):
- Same directory in both `db_tracked` and `db_only` (ambiguous routing).
## Use cases
### Brain repository scaling
Perfect for brain repositories crossing 50K-200K+ files where:
- Core knowledge (people, companies, deals) remains git-tracked.
- Bulk data (tweets, articles, transcripts) moves to db_only.
- Development stays fast with smaller git repos.
- Full data remains available via the database.
### Container-based deployments
Essential for ephemeral container environments:
- Git repo contains only essential files.
- Container restarts don't lose db_only data.
- `gbrain export --restore-only` quickly restores bulk files when needed.
- Local disk acts as a cache layer.
### Multi-environment consistency
Enables consistent data access across environments:
- Development: small git clone, restore bulk data on demand.
- Production: full dataset via the database, selective local caching.
- CI/CD: fast tests with git-tracked data only.
## Migration strategy
1. **Assess current repository**: use `gbrain storage status` to understand current distribution.
2. **Plan directory structure**: identify which directories should be db_tracked vs db_only.
3. **Create `gbrain.yml`**: add storage configuration to the repository root.
4. **Test with dry-run**: `gbrain sync --dry-run` to verify behavior; `.gitignore` is NOT touched on dry-run.
5. **Run a real sync**: `gbrain sync` updates `.gitignore` automatically on success.
6. **Verify restore**: test `gbrain export --restore-only --repo .` against a small db_only directory.
## Best practices
- **Directory naming**: end storage paths with `/` (canonical form). The validator normalizes if you forget.
- **Start small**: begin with clearly machine-generated directories in `db_only`.
- **Address validation errors**: tier overlap is an error, not a warning. Fix it before sync.
- **Test restore**: regularly test `--restore-only` in staging environments.
- **Document decisions**: comment your `gbrain.yml` to explain tier choices.
## PGLite engine note
On the PGLite engine (gbrain's local-only embedded Postgres), the "DB" your db_only pages live in IS the local file gbrain uses for everything else. The `.gitignore` housekeeping still helps (keeps bulk content out of git history), but the offload-to-DB promise is technically vacuous. A once-per-process soft-warn explains when the engine is detected. To get full tiering, migrate to Postgres with `gbrain migrate --to supabase`.
## Compatibility
- **Backward compatible**: systems without `gbrain.yml` work unchanged.
- **Progressive enhancement**: add configuration when needed.
- **Database unchanged**: all data remains in Postgres regardless of tier.
- **Existing workflows**: all existing `sync` and `export` behavior preserved.
- **Deprecated keys**: `git_tracked` / `supabase_only` still load with a once-per-process warning.
+18
View File
@@ -0,0 +1,18 @@
storage:
# Directories that are version-controlled — human-curated, edited by hand.
db_tracked:
- people/
- companies/
- deals/
- concepts/
- yc/
- ideas/
- projects/
# Directories persisted via the brain database only — bulk machine-generated
# content. .gitignored automatically by `gbrain sync`. Restorable from the DB
# via `gbrain export --restore-only`.
db_only:
- media/x/
- media/articles/
- meetings/transcripts/
+105 -29
View File
@@ -104,15 +104,19 @@ strict behavior when unset.
- `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`. `OperationContext.remote` flags untrusted callers.
- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports.
- `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`)
- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract.
- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for the specific forward-referenced state the embedded schema blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target table) and adds only what's missing. Closes the upgrade-wedge bug class that bit users 10+ times across 6 schema versions over 2 years (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396). No-op on fresh installs and modern brains.
- `src/core/pglite-schema.ts` — PGLite-specific DDL (pgvector, pg_trgm, triggers)
- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. v0.22.0: `searchKeyword`, `searchKeywordChunks`, and `searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude clause from `src/core/search/sql-ranking.ts`. `searchVector` switches to a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in the outer SELECT) and carries `p.source_id` through inner→outer for v0.18 multi-source callers. v0.22.1 (#406): `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures). `executeRaw` is a single-statement passthrough — no per-call retry (D3 dropped that as unsound for non-idempotent statements; recovery is supervisor-driven). v0.22.1 (#363, contributed by @orendi84): `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. v0.22.1 (#409, contributed by @atrevino47): `countStaleChunks()` + `listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale`, eliminating ~76 MB/call client-side pull on a fully-embedded brain; `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding (consistency).
- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. v0.22.0: `searchKeyword`, `searchKeywordChunks`, and `searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude clause from `src/core/search/sql-ranking.ts`. `searchVector` switches to a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in the outer SELECT) and carries `p.source_id` through inner→outer for v0.18 multi-source callers. v0.22.1 (#406): `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures). `executeRaw` is a single-statement passthrough — no per-call retry (D3 dropped that as unsound for non-idempotent statements; recovery is supervisor-driven). v0.22.1 (#363, contributed by @orendi84): `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. v0.22.1 (#409, contributed by @atrevino47): `countStaleChunks()` + `listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale`, eliminating ~76 MB/call client-side pull on a fully-embedded brain; `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding (consistency). As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL on the same forward-reference probe set as the PGLite engine, so old Postgres brains pinned at v0.13/v0.18/v0.19 walk forward cleanly instead of wedging on `column "..." does not exist`.
- `src/core/utils.ts` — Shared SQL utilities extracted from postgres-engine.ts. Exports `parseEmbedding(value)` (throws on unknown input, used by migration + ingest paths where data integrity matters) and as of v0.12.3 `tryParseEmbedding(value)` (returns `null` + warns once per process, used by search/rescore paths where availability matters more than strictness).
- `src/core/db.ts` — Connection management, schema initialization. v0.22.1 (#363, contributed by @orendi84): `resolveSessionTimeouts()` returns `statement_timeout` + `idle_in_transaction_session_timeout` (defaults: 5min each, env-overridable via `GBRAIN_STATEMENT_TIMEOUT` / `GBRAIN_IDLE_TX_TIMEOUT` / `GBRAIN_CLIENT_CHECK_INTERVAL`). Both `connect()` (module singleton) and `PostgresEngine.connect()` (worker pool) consume the result via postgres.js's `connection` option, sending GUCs as startup parameters that survive PgBouncer transaction mode (unlike the prior `setSessionDefaults` post-pool SET, kept as a back-compat no-op shim).
- `src/commands/migrate-engine.ts` — Bidirectional engine migration (`gbrain migrate --to supabase/pglite`)
- `src/core/import-file.ts` — importFromFile + importFromContent (chunk + embed + tags)
- `src/core/sync.ts` — Pure sync functions (manifest parsing, filtering, slug conversion)
- `src/core/sync.ts` — Pure sync functions (manifest parsing, filtering, slug conversion). v0.22.12 (#500, foundation by @wintermute via #501): `classifyErrorCode(errorMsg)` regex-based classifier with 12 codes (`SLUG_MISMATCH`, `YAML_PARSE`, `YAML_DUPLICATE_KEY`, `MISSING_OPEN`, `MISSING_CLOSE`, `NESTED_QUOTES`, `EMPTY_FRONTMATTER`, `NULL_BYTES`, `INVALID_UTF8`, `STATEMENT_TIMEOUT`, `FILE_TOO_LARGE`, `SYMLINK_NOT_ALLOWED`) plus `UNKNOWN` fallback. `summarizeFailuresByCode(failures)` returns sorted `[{code, count}]`. `code?` optional field on `SyncFailure`; backfilled at ack time on pre-v0.22.12 entries. `acknowledgeSyncFailures()` returns `AcknowledgeResult { count, summary }`. Three regexes (`MISSING_OPEN`, `MISSING_CLOSE`, `EMPTY_FRONTMATTER`) broadened to match actual `markdown.ts:159-244` validator message strings, not just the literal code-name prefix. `FILE_TOO_LARGE` covers all three production size sites in `import-file.ts:199, 352, 401`; `SYMLINK_NOT_ALLOWED` covers the rejection at `:347`. Closes the silent-skip pattern that motivated #500.
- `src/core/storage.ts` — Pluggable storage interface (S3, Supabase Storage, local)
- `src/core/storage-config.ts` (v0.22.11) — Storage tiering: `loadStorageConfig` reads `gbrain.yml`, normalizes deprecated keys (`git_tracked` / `supabase_only`) to canonical (`db_tracked` / `db_only`) with once-per-process deprecation warning, and runs `normalizeAndValidateStorageConfig` (auto-fixes missing trailing `/`, throws `StorageConfigError` on tier overlap). Path-segment matcher: `media/x/` does NOT match `media/xerox/foo`. Replaces gray-matter (broken on delimiter-less YAML) with a dedicated parser for the `gbrain.yml` shape.
- `src/core/disk-walk.ts` (v0.22.11) — `walkBrainRepo(repoPath)` returns `Map<slug, {size, mtimeMs}>` from one recursive `readdirSync`. Skips dot-dirs, `node_modules`, non-`.md` files. Used by `gbrain storage status` to replace per-page `existsSync + statSync` (~400K syscalls on 200K-page brains → tens).
- `src/commands/storage.ts` (v0.22.11) — `gbrain storage status [--repo P] [--json]`. Split into pure data (`getStorageStatus`) + JSON formatter + human formatter (ASCII-only per D10) matching the `orphans.ts` pattern. `PageCountsByTier` and `DiskUsageByTier` are distinct nominal types so swaps fail at compile time.
- `gbrain.yml` (brain repo root, v0.22.11) — Optional storage tiering config. Top-level `storage:` section with `db_tracked:` and `db_only:` array-valued keys. `gbrain sync` auto-manages `.gitignore` for `db_only` paths on successful sync (skips on dry-run, blocked-by-failures, submodule context, or `GBRAIN_NO_GITIGNORE=1`). `gbrain export --restore-only [--repo P] [--type T] [--slug-prefix S]` repopulates missing `db_only` files from the database.
- `src/core/supabase-admin.ts` — Supabase admin API (project discovery, pgvector check)
- `src/core/file-resolver.ts` — File resolution with fallback chain (local -> .redirect.yaml -> .redirect -> .supabase)
- `src/core/chunkers/` — 3-tier chunking (recursive, semantic, LLM-guided). v0.19.0 adds `code.ts` — tree-sitter-based semantic chunker for 29 languages with embedded-asset WASMs (`src/assets/wasm/`), `@dqbd/tiktoken` cl100k_base tokenizer, small-sibling merging. `CHUNKER_VERSION` constant folded into `importCodeFile`'s `content_hash` so chunker shape changes force clean re-chunks across releases.
@@ -167,21 +171,31 @@ strict behavior when unset.
- `src/core/minions/attachments.ts` — Attachment validation (path traversal, null byte, oversize, base64, duplicate detection)
- `src/commands/agent.ts` (v0.16) — `gbrain agent run <prompt> [flags]` CLI. Submits `subagent` (or N children + 1 aggregator) under `{allowProtectedSubmit: true}`. Single-entry `--fanout-manifest` short-circuits. Children get `on_child_fail: 'continue'` + `max_stalled: 3`. `--follow` is the default on TTY; streams logs + polls `waitForCompletion` in parallel. Ctrl-C detaches, does not cancel.
- `src/commands/agent-logs.ts` (v0.16) — `gbrain agent logs <job> [--follow] [--since]`. Merges JSONL heartbeat audit + `subagent_messages` into a chronological timeline. `parseSince` accepts ISO-8601 or relative (`5m`, `1h`, `2d`). Transcript tail renders only for terminal jobs.
- `src/commands/jobs.ts` — `gbrain jobs` CLI subcommands + `gbrain jobs work` daemon. v0.13.1 surfaces the full `MinionJobInput` retry/backoff/timeout/idempotency surface as first-class CLI flags on `jobs submit`: `--max-stalled`, `--backoff-type fixed|exponential`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key`. `jobs smoke --sigkill-rescue` is the opt-in regression guard for #219. v0.16 wires `registerBuiltinHandlers` to always register `subagent` + `subagent_aggregator` (no env flag — `ANTHROPIC_API_KEY` is the natural cost gate, trust is via `PROTECTED_JOB_NAMES`) and loads `GBRAIN_PLUGIN_PATH` plugins at worker startup with a loud startup-line per plugin. `shell` handler still gated by `GBRAIN_ALLOW_SHELL_JOBS=1` (RCE surface, separate concern).
- `src/commands/jobs.ts` — `gbrain jobs` CLI subcommands + `gbrain jobs work` daemon. v0.13.1 surfaces the full `MinionJobInput` retry/backoff/timeout/idempotency surface as first-class CLI flags on `jobs submit`: `--max-stalled`, `--backoff-type fixed|exponential`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key`. `jobs smoke --sigkill-rescue` is the opt-in regression guard for #219. v0.16 wires `registerBuiltinHandlers` to always register `subagent` + `subagent_aggregator` (no env flag — `ANTHROPIC_API_KEY` is the natural cost gate, trust is via `PROTECTED_JOB_NAMES`) and loads `GBRAIN_PLUGIN_PATH` plugins at worker startup with a loud startup-line per plugin. `shell` handler still gated by `GBRAIN_ALLOW_SHELL_JOBS=1` (RCE surface, separate concern). v0.22.10 (#521): the `autopilot-cycle` handler now forwards `job.data.phases` to `runCycle` (was previously discarded — caller-supplied phase selection silently became a full cycle). Phases are validated against `ALL_PHASES` from `src/core/cycle.ts`; invalid names are filtered out and an empty/missing array falls back to the default 6-phase cycle. v0.22.13 (PR #490 CODEX-1+CODEX-4): `sync` handler now resolves `sourceId` at entry by looking up `sources.local_path` (mirrors `cycle.ts:480`'s autopilot fix from PR #475) so multi-source brains read the per-source `last_commit` anchor instead of the global config key. Concurrency routed through the shared `autoConcurrency()` policy in `src/core/sync-concurrency.ts` instead of the prior hardcoded `4`; PGLite stays serial. `noEmbed` default is `true` (embed is a separate job — submit `gbrain embed --stale` after sync, or rely on the autopilot cycle's embed phase).
- `src/commands/features.ts` — `gbrain features --json --auto-fix`: usage scan + feature adoption salesman
- `src/commands/autopilot.ts` — `gbrain autopilot --install`: self-maintaining brain daemon (sync+extract+embed)
- `src/mcp/server.ts` — MCP stdio server (generated from operations)
- `src/commands/auth.ts` — Standalone token management (create/list/revoke/test)
- `src/mcp/server.ts` — MCP stdio server (generated from operations). v0.22.7: tool-call handler delegates to `dispatchToolCall` from `src/mcp/dispatch.ts` so stdio + HTTP transports share one validation, context-build, and error-format path.
- `src/mcp/dispatch.ts` (v0.22.7) — Shared tool-call dispatch consumed by both stdio (`server.ts`) and HTTP (`http-transport.ts`). Exports `dispatchToolCall(engine, name, params, opts)`, `buildOperationContext(engine, params, opts)`, and `validateParams(op, params)`. Single source of truth for `(ctx, params)` handler arg order and the 5-field `OperationContext` shape (engine + config + logger + dryRun + remote). Defaults to `remote: true` (untrusted); local CLI callers pass `remote: false`. Closed F1 (reversed handler args) + F2 (incomplete OperationContext) + F3 (no param validation) drift bugs in the original v0.22.5 HTTP transport.
- `src/mcp/rate-limit.ts` (v0.22.7) — Bounded-LRU token-bucket limiter for `gbrain serve --http`. `buildDefaultLimiters()` returns the two-bucket pipeline used by http-transport: pre-auth IP (default 30/60s, fires BEFORE the DB lookup so brute-force load against `access_tokens` is actually capped) + post-auth token-id (default 60/60s). Tracks `lastTouchedMs` separately from `lastRefillMs` so an exhausted key can't be reset by hammering past the TTL. LRU cap (default 10K keys) bounds memory under attacker-controlled key growth; TTL prune at 2× window evicts abandoned buckets.
- `src/mcp/http-transport.ts` (v0.22.7, rewrite) — `gbrain serve --http` HTTP transport. Postgres-only — fails fast at startup on PGLite (the `access_tokens` table only exists on Postgres). Bearer auth against SHA-256 hashes in `access_tokens`. CORS default-deny via `GBRAIN_HTTP_CORS_ORIGIN` allowlist. Body cap stream-counted (1 MiB default via `GBRAIN_HTTP_MAX_BODY_BYTES`) so chunked transfers without Content-Length still hit the cap. `last_used_at` SQL-level debounce (one UPDATE per token per 60s). Per-request audit row in `mcp_request_log` with token_name + operation + status + latency. Optional `GBRAIN_HTTP_TRUST_PROXY=1` honors `X-Forwarded-For` — only safe when bound to a private interface AND the proxy strips client-supplied XFF (otherwise enables IP spoofing past the pre-auth rate limit). `/health` does `SELECT 1` against Postgres and returns 503 + `status:unhealthy` when the DB is unreachable so orchestration doesn't see green pods while clients get misleading 401s. Replaces the standalone OAuth wrapper that was vulnerable to unauthenticated client registration.
- `src/commands/auth.ts` — Token management for the HTTP transport. `gbrain auth create/list/revoke/test`. As of v0.22.7 wired into the main CLI (`src/cli.ts`); also runs standalone via `bun run src/commands/auth.ts ...` for environments without a compiled binary. Tokens stored as SHA-256 hashes in `access_tokens` (Postgres-only).
- `src/commands/upgrade.ts` — Self-update CLI. `runPostUpgrade()` enumerates migrations from the TS registry (src/commands/migrations/index.ts) and tail-calls `runApplyMigrations(['--yes', '--non-interactive'])` so the mechanical side of every outstanding migration runs unconditionally.
- `src/commands/migrations/` — TS migration registry (compiled into the binary; no filesystem walk of `skills/migrations/*.md` needed at runtime). `index.ts` lists migrations in semver order. `v0_11_0.ts` = Minions adoption orchestrator (8 phases). `v0_12_0.ts` = Knowledge Graph auto-wire orchestrator (5 phases: schema → config check → backfill links → backfill timeline → verify). `phaseASchema` has a 600s timeout (bumped from 60s in v0.12.1 for duplicate-heavy brains). `v0_12_2.ts` = JSONB double-encode repair orchestrator (4 phases: schema → repair-jsonb → verify → record). `v0_14_0.ts` = shell-jobs + autopilot cooperative (2 phases: schema ALTER minion_jobs.max_stalled SET DEFAULT 3 — superseded by v0.14.3's schema-level DEFAULT 5 + UPDATE backfill; pending-host-work ping for skills/migrations/v0.14.0.md). All orchestrators are idempotent and resumable from `partial` status. As of v0.14.2 (Bug 3), the RUNNER owns all ledger writes — orchestrators return `OrchestratorResult` and `apply-migrations.ts` persists a canonical `{version, status, phases}` shape after return. Orchestrators no longer call `appendCompletedMigration` directly. `statusForVersion` prefers `complete` over `partial` (never regresses). 3 consecutive partials → wedged → `--force-retry <version>` writes a `'retry'` reset marker. v0.14.3 (fix wave) ships schema-only migrations v14 (`pages_updated_at_index`) + v15 (`minion_jobs_max_stalled_default_5` with UPDATE backfill) via the `MIGRATIONS` array in `src/core/migrate.ts` — no orchestrator phases needed.
- `src/commands/repair-jsonb.ts` — `gbrain repair-jsonb [--dry-run] [--json]`: rewrites `jsonb_typeof='string'` rows in place across 5 affected columns (pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata, page_versions.frontmatter). Fixes v0.12.0 double-encode bug on Postgres; PGLite no-ops. Idempotent.
- `src/commands/orphans.ts` — `gbrain orphans [--json] [--count] [--include-pseudo]`: surfaces pages with zero inbound wikilinks, grouped by domain. Auto-generated/raw/pseudo pages filtered by default. Also exposed as `find_orphans` MCP operation. Shipped in v0.12.3 (contributed by @knee5).
- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel <id>`.
- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows.
- `src/commands/integrity.ts` — `gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). v0.22.8: batch-load fast path on Postgres uses `SELECT DISTINCT ON (slug)` in a single SQL query to fix the PgBouncer round-trip timeout (60s → ~6s) while preserving `engine.getAllSlugs()`'s `Set<string>` semantics on multi-source brains. Gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1` so real Postgres errors are diagnosable.
- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel <id>`. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count.
- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2).
- `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2.
- `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=<ms>` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers.
- `src/core/db-lock.ts` (v0.22.13) — generic `tryAcquireDbLock(engine, lockId, ttlMinutes)` over the existing `gbrain_cycle_locks` table. Parameterized lock id so different scopes can nest cleanly: `gbrain-cycle` for the broad cycle (held by `cycle.ts`) and `gbrain-sync` (`SYNC_LOCK_ID` constant) for `performSync`'s narrower writer window. Same UPSERT-with-TTL semantics as the prior cycle-only helper, just generalized. Survives PgBouncer transaction pooling (unlike session-scoped `pg_try_advisory_lock`); crashed holders auto-release once their TTL expires.
- `src/core/sync-concurrency.ts` (v0.22.13) — single source of truth for the parallel-sync policy. Exports `autoConcurrency(engine, fileCount, override?)` (PGLite always serial; explicit override clamped to >=1; auto path returns `DEFAULT_PARALLEL_WORKERS=4` when `fileCount > AUTO_CONCURRENCY_FILE_THRESHOLD=100`), `shouldRunParallel(workers, fileCount, explicit)` (Q1: explicit `--workers` bypasses the >50-file floor), and `parseWorkers(s)` (rejects `'0'`, `'-3'`, `'foo'`, `'1.5'`, trailing chars — replaces the prior parseInt-with-no-validation in both `sync.ts` and `import.ts`). Used by `performSync`, `performFullSync`, `runImport`, and the Minion `sync` handler so the three sites can no longer drift.
- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). v0.22.13 (PR #490): `performSync` wraps its body in a `gbrain-sync` writer lock so two concurrent syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot both write `last_commit` and let the last writer win. Head-drift gate after the import phase re-checks `git rev-parse HEAD`; if HEAD moved (someone ran `git checkout` / `git pull` mid-sync), the bookmark refuses to advance. Vanished files now record a failedFiles entry instead of silent-skip — the silent-skip-then-advance pathology that survived prior hardening passes is dead. Worker engines wrap in try/finally so disconnect always fires (panic-path leak fix). Both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`), validated via `parseWorkers`. Explicit `--workers` bypasses the auto-path file-count floor; auto path defers to `autoConcurrency()`. Banner moved to stderr.
- `src/core/cycle.ts` — v0.17 brain maintenance cycle primitive. `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise<CycleReport>` composes 6 phases in semantically-driven order (lint → backlinks → sync → extract → embed → orphans). Three callers: `gbrain dream` CLI, `gbrain autopilot` daemon's inline path, and the Minions `autopilot-cycle` handler (`src/commands/jobs.ts`). One source of truth for what the brain does overnight. Coordination via `gbrain_cycle_locks` DB table (TTL-based; works through PgBouncer transaction pooling, unlike session-scoped `pg_try_advisory_lock`) + `~/.gbrain/cycle.lock` file lock with PID-liveness for PGLite / engine=null mode. `CycleReport.schema_version: "1"` is the stable agent-consumable shape. `PhaseResult.error: { class, code, message, hint?, docs_url? }` is Stripe-API-tier structured failure info. `yieldBetweenPhases` hook awaited between every phase — Minions handler uses this to renew its job lock and prevent v0.14 stall-death regression. Engine nullable: filesystem phases (lint, backlinks) run without DB; DB phases skip with `status: "skipped", reason: "no_database"`. Lock-skip: read-only phase selections (`--phase orphans`) bypass the cycle lock. v0.22.1 (#403): `CycleOpts.signal?: AbortSignal` propagates the worker's abort signal; `checkAborted()` fires between every phase and throws if the signal is aborted (cooperative — can't interrupt a phase mid-execution). v0.22.1 (#417): `runPhaseSync` returns `pagesAffected` via `SyncPhaseResult`; `runCycle` captures it and threads to `runPhaseExtract` as the 4th arg, enabling incremental extract on the cycle path. v0.22.1 (Codex F2): `runPhaseSync` takes `willRunExtractPhase: boolean` and sets `noExtract: phases.includes('extract')` so `gbrain dream --phase sync` doesn't silently lose extraction. v0.22.5 (#475): new `resolveSourceForDir(engine, brainDir)` helper queries `SELECT id FROM sources WHERE local_path = $1 LIMIT 1`; `runPhaseSync` threads result as `sourceId` to `performSync()` so sync reads the per-source `sources.last_commit` anchor instead of the drift-prone global `config.sync.last_commit` key. Bare try/catch lets pre-v0.18 brains fall through to the global key. Closes the prod hang where every autopilot cycle ran a 30-min full reimport because the global anchor commit had been GC'd from git history.
- `src/commands/dream.ts` — v0.17 `gbrain dream` CLI. ~80-line thin alias over `runCycle`. brainDir resolution requires explicit `--dir` OR `sync.repo_path` config (no more walk-up-cwd-for-.git footgun). Flags: `--dry-run`, `--json`, `--phase <name>`, `--pull`, `--dir <path>`. Exit code 1 on status=failed (partial/warn not fatal — don't page on warnings).
- `src/commands/friction.ts` + `src/core/friction.ts` (v0.23) — `gbrain friction {log,render,list,summary}` reporter. Append-only JSONL under `$GBRAIN_HOME/friction/<run-id>.jsonl`. Schema is a flat extension of `StructuredAgentError` (D20). Render groups by severity → phase, defaults to `--redact` for md output (strips `$HOME`/`$CWD` to placeholders so reports paste safely in PRs). Run-id resolves from `--run-id` > `$GBRAIN_FRICTION_RUN_ID` > `standalone.jsonl`. Skills the claw-test exercises gain a `_friction-protocol.md` callout so agents know when to log friction.
- `src/commands/claw-test.ts` + `src/core/claw-test/` (v0.23) — `gbrain claw-test [--scenario <name>] [--live --agent openclaw]`. End-to-end "fresh user" friction harness. Two modes: scripted (CI gate, agent-free) and live (real openclaw subprocess, $12 in tokens). Sets `GBRAIN_HOME=<tempdir>` for hermeticity and captures gbrain's `--progress-json` events from each child's stderr to verify expected phases ran (`import.files`, `extract.links_fs`, `doctor.db_checks`). Phases for scripted mode: setup → install_brain (`gbrain init --pglite`) → import (`--no-embed`) → query → extract → verify (`gbrain doctor --json`, asserts `status: 'ok'`) → render. Live mode hands `BRIEF.md` from `test/fixtures/claw-test-scenarios/<name>/` to the agent runner. v1 ships with the OpenClaw runner only (`src/core/claw-test/runners/openclaw.ts`, invokes `openclaw agent --local --agent <name> --message <brief>`); hermes runner deferred to v1.1. Transcript capture (`transcript-capture.ts`) uses `fs.createWriteStream` with `'drain'`-event backpressure — D17 fix for the 256KB-burst child-stall scenario. v0.18 upgrade scenario seeded via `seed-pglite.ts` SQL replay.
- `skills/_friction-protocol.md` (v0.23) — shared cross-cutting convention skill (like `_brain-filing-rules.md`). Tells agents when to call `gbrain friction log` and how to choose a severity. Routes to friction CLI from any skill the claw-test exercises.
- `scripts/check-progress-to-stdout.sh` — CI guard against regressing to `\r`-on-stdout progress. Wired into `bun run test` via `scripts/check-progress-to-stdout.sh && bun test` in package.json.
- `docs/progress-events.md` — Canonical JSON event schema reference. Stable from v0.15.2, additive only.
- `src/core/markdown.ts` — Frontmatter parsing + body splitter. `splitBody` requires an explicit timeline sentinel (`<!-- timeline -->`, `--- timeline ---`, or `---` immediately before `## Timeline`/`## History`). Plain `---` in body text is a markdown horizontal rule, not a separator. `inferType` auto-types `/wiki/analysis/` → analysis, `/wiki/guides/` → guide, `/wiki/hardware/` → hardware, `/wiki/architecture/` → architecture, `/writing/` → writing (plus the existing people/companies/deals/etc heuristics).
@@ -291,6 +305,20 @@ Key commands added in v0.14.3 (fix wave):
- `gbrain jobs submit` gains `--max-stalled`, `--backoff-type`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key` — exposing existing `MinionJobInput` fields as first-class CLI flags.
- `gbrain jobs smoke --sigkill-rescue` — opt-in regression smoke case simulating a killed worker; asserts the v0.14.3 schema default (`max_stalled=5`) actually rescues on first stall.
Key commands added in v0.22.13 (PR #490):
- `gbrain sync --workers N` (alias `--concurrency N`) — parallelize the import phase using per-worker Postgres engines (small pool of 2 each) with an atomic queue index. Auto-concurrency: defaults to 4 workers when the diff exceeds 100 files. Smaller diffs stay serial. Explicit `--workers` always wins (even on a 30-file diff). PGLite forces serial regardless. Validation rejects `0`, negatives, non-integers loud (replaces the prior silent fall-through to auto-concurrency).
- `gbrain import --workers N` — same `parseWorkers()` validation as sync; same try/finally worker-engine cleanup. Behavior surface unchanged.
Key commands added in v0.22.16 (claw-test friction loop):
- `gbrain claw-test [--scenario fresh-install|upgrade-from-v0.18] [--keep-tempdir]` — scripted-mode CI gate that runs the full canonical first-day flow against a fresh tempdir. Asserts every expected `--progress-json` phase fired and doctor's `status === 'ok'`. ~30s, no API keys.
- `gbrain claw-test --live --agent openclaw` — friction-discovery mode. Spawns real openclaw, hands it `BRIEF.md`, captures stdin/stdout/stderr to `<run>/transcript.jsonl`, lets the agent log friction via the friction CLI. Run on demand; ~510 min and ~$12 in tokens.
- `gbrain claw-test --list-agents` — reports which agent runners are registered + their detection state (binary path or unavailable reason).
- `gbrain friction log --severity {confused|error|blocker|nit} --phase <name> --message <text> [--hint ...] [--kind {friction|delight}] [--run-id ...]` — append a friction or delight entry to the active run JSONL.
- `gbrain friction render --run-id <id> [--json] [--transcripts] [--no-redact]` — markdown report grouped by severity + phase; `--redact` is the default for md output (strips `$HOME`/`$CWD` placeholders so reports paste safely in PRs/issues).
- `gbrain friction list [--json]` — recent run-ids with friction/delight counts; interrupted runs marked `(interrupted)`.
- `gbrain friction summary --run-id <id> [--json]` — two-column friction + delight summary.
- `GBRAIN_HOME` env override is now honored uniformly across every gbrain write site (config, audit, friction, sync-failures, import checkpoint, integrity log, integrations heartbeat, migration rollback, etc.) — `gbrainPath(...)` from `src/core/config.ts` is the canonical helper. Read-side host-fingerprint detection (`~/.claude`/`~/.openclaw` etc.) intentionally NOT confined in v1; that's a v1.1 follow-up.
## Testing
`bun test` runs all tests. After the v0.12.1 release: ~75 unit test files + 8 E2E test files (1412 unit pass, 119 E2E when `DATABASE_URL` is set — skip gracefully otherwise). Unit tests run
@@ -302,7 +330,9 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac
`test/files.test.ts` (MIME/hash), `test/import-file.test.ts` (import pipeline),
`test/upgrade.test.ts` (schema migrations),
`test/file-migration.test.ts` (file migration), `test/file-resolver.test.ts` (file resolution),
`test/import-resume.test.ts` (import checkpoints), `test/migrate.test.ts` (migration; v8/v9 helper-btree-index SQL structural assertions + 1000-row wall-clock fixtures that guard the O(n²)→O(n log n) fix + v0.13.1 assertions on v12/v13 SQL shape, `sqlFor` + `transaction:false` runner semantics, and the `max_stalled DEFAULT 1` regression guard),
`test/import-resume.test.ts` (import checkpoints), `test/migrate.test.ts` (migration; v8/v9 helper-btree-index SQL structural assertions + 1000-row wall-clock fixtures that guard the O(n²)→O(n log n) fix + v0.13.1 assertions on v12/v13 SQL shape, `sqlFor` + `transaction:false` runner semantics, the `max_stalled DEFAULT 1` regression guard, and v0.22.6.1 v24 `sqlFor.pglite: ''` no-op assertion),
`test/bootstrap.test.ts` (v0.22.6.1 — bootstrap contract: no-op on fresh install, idempotent across two `initSchema()` calls, no-op on modern brain that already has every probed column, full bootstrap path on simulated pre-v0.18 brain, fresh-install regression guard, pre-v0.13 `links` shape coverage),
`test/schema-bootstrap-coverage.test.ts` (v0.22.6.1 CI guard — `REQUIRED_BOOTSTRAP_COVERAGE` lists every forward reference in PGLITE_SCHEMA_SQL; the test fails loudly if `applyForwardReferenceBootstrap` skips one. When you add a column-with-index to the embedded schema blob, you extend both arrays or this guard fails. The pattern that broke gbrain ten times in two years is now structurally prevented.),
`test/setup-branching.test.ts` (setup flow), `test/slug-validation.test.ts` (slug validation),
`test/storage.test.ts` (storage backends), `test/supabase-admin.test.ts` (Supabase admin),
`test/yaml-lite.test.ts` (YAML parsing), `test/check-update.test.ts` (version check + update CLI),
@@ -344,6 +374,9 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac
`test/orphans.test.ts` (v0.12.3 orphans command: detection, pseudo filtering, text/json/count outputs, MCP op),
`test/postgres-engine.test.ts` (v0.12.3 statement_timeout scoping: `sql.begin` + `SET LOCAL` shape, source-level grep guardrail against reintroduced bare `SET statement_timeout`),
`test/sync.test.ts` (sync logic + v0.12.3 regression guard asserting top-level `engine.transaction` is not called),
`test/sync-concurrency.test.ts` (v0.22.13 PR #490: 17 cases covering `autoConcurrency()` thresholds + PGLite-forces-serial + explicit-override clamping, `shouldRunParallel()` Q1 explicit-bypasses-floor contract, and `parseWorkers()` validation that rejects `'0'`/`'-3'`/`'foo'`/`'1.5'`/trailing chars),
`test/sync-parallel.test.ts` (v0.22.13 PR #490: PGLite-routed coverage of the bookmark gate under concurrency request, head-drift gate, vanished-file failure capture, PGLite-stays-serial, and the `gbrain-sync` writer-lock contract — 7 cases),
`test/sync-failures.test.ts` (v0.22.12: 28 cases pinning `classifyErrorCode` regex coverage for all 12 codes against literal production message strings from `markdown.ts:159-244` and `import-file.ts:199, 347, 352, 401`; `summarizeFailuresByCode` sort + pre-classified-honor; `recordSyncFailures` code-field persistence; `acknowledgeSyncFailures` AcknowledgeResult shape + backfill on pre-v0.22.12 entries),
`test/doctor.test.ts` (doctor command + v0.12.3 assertions that `jsonb_integrity` scans the four v0.12.0 write sites and `markdown_body_completeness` is present),
`test/utils.test.ts` (shared SQL utilities + `tryParseEmbedding` null-return and single-warn semantics),
`test/build-llms.test.ts` (llms.txt/llms-full.txt generator: path resolution, idempotence, spec shape, regen-drift guard, content contract, AGENTS.md install-path mirror, size-budget enforcement — 7 cases),
@@ -354,20 +387,26 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac
`test/skill-manifest.test.ts` (v0.19 skill manifest parser: drift detection, managed-block markers),
`test/skillify-scaffold.test.ts` (v0.19 `gbrain skillify scaffold` stubs: SKILL.md, script, tests, routing-eval fixtures),
`test/skillpack-install.test.ts` (v0.19 `gbrain skillpack install` managed-block install / update / no-clobber semantics),
`test/skillpack-sync-guard.test.ts` (v0.19 sync-guard: bundled skills stay byte-identical to `skills/` source).
`test/skillpack-sync-guard.test.ts` (v0.19 sync-guard: bundled skills stay byte-identical to `skills/` source),
`test/http-transport.test.ts` (v0.22.7 HTTP transport: 23 unit cases covering bearer auth + missing/no-Bearer/unknown/revoked + `/health` bypass, F1+F2 round-trip via dispatch.ts, F3 invalid_params, application/json response shape (not SSE), CORS default-deny + allowlist, body cap on Content-Length AND chunked, two-bucket rate limit (refill, exhaust+Retry-After, LRU eviction, TTL prune, pre-auth IP fires before DB), and `mcp_request_log` audit on success + auth_failed).
E2E tests (`test/e2e/`): Run against real Postgres+pgvector. Require `DATABASE_URL`.
- `bun run test:e2e` runs Tier 1 (mechanical, all operations, no API keys). Includes 9 dedicated cases for the postgres-engine `addLinksBatch` / `addTimelineEntriesBatch` bind path — postgres-js's `unnest()` binding is structurally different from PGLite's and gets its own coverage.
- `test/e2e/search-quality.test.ts` runs search quality E2E against PGLite (no API keys, in-memory)
- `test/e2e/graph-quality.test.ts` runs the v0.10.3 knowledge graph pipeline (auto-link via put_page, reconciliation, traversePaths) against PGLite in-memory
- `test/e2e/postgres-jsonb.test.ts` — v0.12.2 regression test. Round-trips all 5 JSONB write sites (pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata, page_versions.frontmatter) against real Postgres and asserts `jsonb_typeof='object'` plus `->>'key'` returns the expected scalar. The test that should have caught the original double-encode bug.
- `test/e2e/integrity-batch.test.ts` (v0.22.8) — parity tests for `scanIntegrity`'s batch-load fast path vs sequential. Four cases (dedup, hits, validate, topPages) seed a fixture and assert both paths return identical results. Dedup case uses raw SQL via `getConn().unsafe()` to seed a `(test-source-2, people/alice)` row alongside the default-source row, since `engine.putPage` doesn't take a `source_id`. Pins the codex-caught multi-source overcounting regression.
- `test/e2e/jsonb-roundtrip.test.ts` — v0.12.3 companion regression against the 4 doctor-scanned JSONB sites. Assertion-level overlap with `postgres-jsonb.test.ts` is intentional defense-in-depth: if doctor's scan surface ever drifts from the actual write surface, one of these tests catches it.
- `test/e2e/sync.test.ts` (v0.22.12 — `--skip-failed` failure-loop test, alongside the existing 13 happy-path tests): exercises the full chain — broken file → `performSync` returns `blocked_by_failures` with grouped breakdown → `performSync({skipFailed: true})` advances bookmark and returns `AcknowledgeResult` with code summary → second broken file → second cycle. Saves and restores the user's real `~/.gbrain/sync-failures.jsonl` so the test is hermetic on a developer machine. Asserts bookmark gating, JSONL state, dedup across paths, summary aggregation, and the literal doctor-rendering string format. This is the integration test that proves the v0.22.12 chain holds together — unit tests cover the pure functions in isolation, this covers the integration.
- `test/e2e/upgrade.test.ts` runs check-update E2E against real GitHub API (network required)
- `test/e2e/minions-shell-pglite.test.ts` (v0.20.4) exercises the PGLite `--follow` inline shell-job path (in-memory, no `DATABASE_URL` required) — the path the consolidated minion-orchestrator skill documents for dev use
- `test/e2e/openclaw-reference-compat.test.ts` (v0.19) — exercises `check-resolvable` + `skillpack install` against a minimal AGENTS.md workspace fixture (`test/fixtures/openclaw-reference-minimal/`), regression guard for the 107-skill OpenClaw deployment shape
- `test/e2e/search-swamp.test.ts` (v0.22.0) — reproduces the headline source-swamp case. Seeds a curated `originals/talks/article-outline-fat-code` page against two `wintermute/chat/` pages stuffed with the same multi-word phrase. Asserts the article wins keyword AND vector ranking, that `detail=high` lets the chat swamp re-surface (temporal-query workflow preserved), and that `source_id` passes through the two-stage CTE intact. PGLite in-memory.
- `test/e2e/search-exclude.test.ts` (v0.22.0) — verifies `test/` + `archive/` pages are hidden by default, that `include_slug_prefixes` opts back in, and that caller-supplied `exclude_slug_prefixes` adds to defaults. Both keyword and vector search paths covered.
- `test/e2e/engine-parity.test.ts` (v0.22.0) — Postgres ↔ PGLite top-result and result-set parity for `searchKeyword` + `searchVector`. Codex flagged that Postgres ranks pages then picks best chunk while PGLite returns chunks directly — without parity coverage the source-boost fix could pass on PGLite and fail on Postgres. Skips gracefully when `DATABASE_URL` is unset.
- `test/e2e/postgres-bootstrap.test.ts` (v0.22.6.1) — exercises `PostgresEngine.initSchema()` directly against a fresh real Postgres database. Asserts the bootstrap path is no-op on fresh installs and that SCHEMA_SQL replays cleanly through the engine path (not via the standalone `db.initSchema` from `src/core/db.ts`, which would have produced false-positive coverage). Codex caught the E2E-shape gap during plan review.
- `test/e2e/http-transport.test.ts` (v0.22.7) — 8 cases against real Postgres covering `gbrain serve --http` end-to-end: bearer auth round-trip, `last_used_at` SQL-level debounce semantics, `mcp_request_log` row insertion on success and auth_failed paths, `/health` DB-down → 503 (DB-probing health check), and the F1+F2+F3 dispatch round-trip with a real operation. Skips gracefully when `DATABASE_URL` is unset.
- `test/e2e/sync-parallel.test.ts` (v0.22.13 PR #490) — DATABASE_URL-gated. T2: 60-file Postgres sync at concurrency=4 imports all + no connection leak (probes `pg_stat_activity` before/after to confirm worker engines disconnected). P4: 120-file serial-vs-parallel benchmark prints `SYNC_PARALLEL_BENCH N files | serial=Xms | parallel(4)=Yms | speedup=Zx` for CHANGELOG quoting. Asserts parallel ≤ serial × 1.5 (CI-noise tolerant; not a strict speedup gate).
- Tier 2 (`skills.test.ts`) requires OpenClaw + API keys, runs nightly in CI
- If `.env.testing` doesn't exist in this directory, check sibling worktrees for one:
`find ../ -maxdepth 2 -name .env.testing -print -quit` and copy it here if found.
@@ -1348,12 +1387,13 @@ Add to `~/.claude/server.json` (Claude Code), Settings > MCP Servers (Cursor), o
### Remote MCP (Claude Desktop, Cowork, Perplexity)
```bash
ngrok http 8787 --url your-brain.ngrok.app
bun run src/commands/auth.ts create "claude-desktop"
gbrain auth create "claude-desktop" # tokens via the existing CLI
gbrain serve --http --port 8787 # built-in HTTP transport (Postgres-only)
ngrok http 8787 --url your-brain.ngrok.app # any tunnel works
claude mcp add gbrain -t http https://your-brain.ngrok.app/mcp -H "Authorization: Bearer TOKEN"
```
Per-client guides: [`docs/mcp/`](docs/mcp/DEPLOY.md). ChatGPT requires OAuth 2.1 (not yet implemented).
Per-client guides: [`docs/mcp/`](docs/mcp/DEPLOY.md). Hardening defaults, env vars, and threat model: [SECURITY.md](SECURITY.md). ChatGPT requires OAuth 2.1 (not yet implemented).
### Using gbrain with GStack
@@ -1627,6 +1667,30 @@ accumulate rows across separate single-skill installs instead of overwriting eac
Read [`skills/skillify/SKILL.md`](skills/skillify/SKILL.md) for the full 10-item checklist
and the anti-patterns it catches.
## Storage tiering: keep bulk content out of git (v0.22.11)
When your brain crosses 100K files and bulk machine-generated content (tweets, articles, transcripts)
becomes the size driver, declare which directories belong in git and which live in the database only.
```yaml
# gbrain.yml at the brain repo root
storage:
db_tracked:
- people/
- companies/
- deals/
db_only:
- media/x/
- media/articles/
- meetings/transcripts/
```
`gbrain sync` auto-manages your `.gitignore` for `db_only` paths. `gbrain export --restore-only --repo .`
repopulates missing files from the database (container restart, fresh clone, accidental rm).
`gbrain storage status` shows the tier breakdown.
Full guide: [docs/storage-tiering.md](docs/storage-tiering.md).
## Getting Data In
GBrain ships integration recipes that your agent sets up for you. Each recipe tells the agent what credentials to ask for, how to validate, and what cron to register.
@@ -1882,8 +1946,11 @@ SEARCH
gbrain query <question> Hybrid search (vector + keyword + RRF)
IMPORT
gbrain import <dir> [--no-embed] Import markdown (idempotent)
gbrain sync [--repo <path>] Git-to-brain incremental sync
gbrain import <dir> [--no-embed] [--workers N]
Import markdown (idempotent)
gbrain sync [--repo <path>] [--workers N]
Git-to-brain incremental sync
(>100-file diffs auto-parallelize 4 workers on Postgres)
gbrain export [--dir ./out/] Export to markdown
FILES
@@ -1925,6 +1992,8 @@ ADMIN
gbrain doctor --locks List idle-in-tx backends (57014 diagnostic, Postgres only)
gbrain stats Brain statistics
gbrain serve MCP server (stdio)
gbrain serve --http --port 8787 MCP server (HTTP, Postgres-only, bearer auth)
gbrain auth create|list|revoke|test Token management for the HTTP transport
gbrain integrations Integration recipe dashboard
gbrain sources list|add|remove|... Multi-source brain management (v0.18)
gbrain dream [--dry-run] [--phase N] One maintenance cycle then exit (cron-friendly)
@@ -4105,9 +4174,14 @@ Source: https://raw.githubusercontent.com/garrytan/gbrain/master/docs/mcp/DEPLOY
# Deploy GBrain Remote MCP Server
> **v0.22.7+:** Use `gbrain serve --http` for remote access. It includes built-in
> bearer token auth, default-deny CORS, two-bucket rate limiting, body cap, and
> per-request audit log. **Postgres-only** (PGLite is local-only by design).
> See [SECURITY.md](../../SECURITY.md) for env vars and tunable defaults.
Access your brain from any device, any AI client. GBrain's MCP server runs locally
via `gbrain serve` (stdio). For remote access, wrap it in an HTTP server behind a
public tunnel.
via `gbrain serve` (stdio). For remote access, expose it via the built-in HTTP
transport behind a public tunnel.
## Two Paths
@@ -4118,21 +4192,23 @@ gbrain serve
```
Works with Claude Code, Cursor, Windsurf, and any MCP client that supports stdio.
No server, no tunnel, no token needed.
No server, no tunnel, no token needed. Works on both PGLite and Postgres engines.
### Remote (any device, any AI client)
### Remote (any device, any AI client) — Postgres only
```
Your AI client (Claude Desktop, Perplexity, etc.)
→ ngrok tunnel (https://YOUR-DOMAIN.ngrok.app)
Your HTTP server (wraps gbrain serve)
Supabase Postgres (via pooler connection string)
gbrain serve --http (built-in transport with bearer auth)
→ Postgres (pooler connection or self-hosted)
```
This requires:
1. A machine running `gbrain serve` behind an HTTP wrapper
2. A public tunnel (ngrok, Tailscale, or cloud host)
3. Bearer token auth for security
1. A Postgres-backed brain (the `access_tokens` table only exists on Postgres;
running `gbrain serve --http` against a PGLite install fails fast at startup)
2. A machine running `gbrain serve --http`
3. A public tunnel (ngrok, Tailscale, or cloud host)
4. A bearer token created via `gbrain auth create <name>`
## Remote Setup
@@ -4151,13 +4227,13 @@ ngrok http 8787 --url your-brain.ngrok.app # Hobby tier for fixed domain
```bash
# Create a token for each client
bun run src/commands/auth.ts create "claude-desktop"
gbrain auth create "claude-desktop"
# List all tokens
bun run src/commands/auth.ts list
gbrain auth list
# Revoke a token
bun run src/commands/auth.ts revoke "claude-desktop"
gbrain auth revoke "claude-desktop"
```
Tokens are per-client. Create one for each device/app. Revoke individually
@@ -4173,7 +4249,7 @@ if compromised. Tokens are stored SHA-256 hashed in your database.
### 4. Verify
```bash
bun run src/commands/auth.ts test \
gbrain auth test \
https://YOUR-DOMAIN.ngrok.app/mcp \
--token YOUR_TOKEN
```
@@ -4201,7 +4277,7 @@ Funnel, and cloud hosts (Fly.io, Railway).
Include the Authorization header: `Authorization: Bearer YOUR_TOKEN`
**"invalid_token" error**
Run `bun run src/commands/auth.ts list` to see active tokens.
Run `gbrain auth list` to see active tokens.
**"service_unavailable" error**
Database connection failed. Check your Supabase dashboard for outages.
+4 -2
View File
@@ -1,6 +1,6 @@
{
"name": "gbrain",
"version": "0.22.5",
"version": "0.22.16",
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
"type": "module",
"main": "src/core/index.ts",
@@ -32,8 +32,9 @@
"build:all": "bun build --compile --target=bun-darwin-arm64 --outfile bin/gbrain-darwin-arm64 src/cli.ts && bun build --compile --target=bun-linux-x64 --outfile bin/gbrain-linux-x64 src/cli.ts",
"build:schema": "bash scripts/build-schema.sh",
"build:llms": "bun run scripts/build-llms.ts",
"test": "scripts/check-jsonb-pattern.sh && scripts/check-progress-to-stdout.sh && scripts/check-wasm-embedded.sh && bun run typecheck && bun test --timeout=60000",
"test": "scripts/check-jsonb-pattern.sh && scripts/check-progress-to-stdout.sh && scripts/check-trailing-newline.sh && scripts/check-wasm-embedded.sh && bun run typecheck && bun test --timeout=60000",
"check:wasm": "scripts/check-wasm-embedded.sh",
"check:newlines": "scripts/check-trailing-newline.sh",
"test:e2e": "bash scripts/run-e2e.sh",
"typecheck": "tsc --noEmit",
"check:jsonb": "scripts/check-jsonb-pattern.sh",
@@ -63,6 +64,7 @@
},
"devDependencies": {
"@types/bun": "latest",
"bun-types": "^1.3.13",
"typescript": "^5.6.0"
},
"trustedDependencies": [
+44
View File
@@ -0,0 +1,44 @@
#!/usr/bin/env bash
# CI guard: every text file under src/, test/, and the repo root .yml/.md
# files must end with a newline. POSIX-noncompliant trailing data shows up
# as a phantom diff on every future edit and trips most linters.
#
# Sibling to scripts/check-progress-to-stdout.sh and
# scripts/check-jsonb-pattern.sh per CLAUDE.md's CI guard pattern.
# Wired into `bun run test` via package.json's `test` script.
set -euo pipefail
# Files to check: anything tracked under src/ + test/ that's a code/text file.
# Also the top-level *.yml + *.md the repo controls. Portable to bash 3.2
# (macOS default) — no mapfile, no associative arrays.
files=$(
git ls-files \
'src/**/*.ts' 'src/**/*.js' 'src/**/*.json' 'src/**/*.sql' 'src/**/*.md' \
'test/**/*.ts' 'test/**/*.js' 'test/**/*.json' 'test/**/*.md' \
'gbrain.yml' '*.md' \
2>/dev/null | sort -u
)
missing=""
total=0
while IFS= read -r f; do
[ -n "$f" ] || continue
[ -f "$f" ] || continue
[ -s "$f" ] || continue
total=$((total + 1))
if [ -n "$(tail -c 1 "$f")" ]; then
missing="${missing} $f"$'\n'
fi
done <<< "$files"
if [ -n "$missing" ]; then
echo "ERROR: the following files are missing a trailing newline:" >&2
printf '%s' "$missing" >&2
echo >&2
echo "Fix: append a newline. e.g. \`printf '\\n' >> <file>\` or your editor's" >&2
echo "'final newline' setting (most editors do this automatically)." >&2
exit 1
fi
echo "trailing-newline check: ok ($total files)"
+76
View File
@@ -0,0 +1,76 @@
#!/usr/bin/env bash
# Partition unit test files into N shards by stable hash and run one shard.
#
# Usage: scripts/test-shard.sh <shard-index> <total-shards>
# shard-index: 1-based (1..N)
# total-shards: positive integer
#
# E2E tests under test/e2e/ are excluded — they need DATABASE_URL and run via
# bun run test:e2e separately.
#
# Stable partitioning: a file's shard is `(hash(path) % N) + 1`. Same file
# lands in the same shard on every run, regardless of how many other files
# exist, so retries are reproducible. Hash is FNV-1a — pure shell, no jq.
set -euo pipefail
if [ "$#" -ne 2 ]; then
echo "usage: scripts/test-shard.sh <shard-index> <total-shards>" >&2
exit 1
fi
SHARD_INDEX="$1"
TOTAL_SHARDS="$2"
if ! [[ "$SHARD_INDEX" =~ ^[0-9]+$ ]] || ! [[ "$TOTAL_SHARDS" =~ ^[0-9]+$ ]]; then
echo "error: shard index and total must be positive integers" >&2
exit 1
fi
if [ "$SHARD_INDEX" -lt 1 ] || [ "$SHARD_INDEX" -gt "$TOTAL_SHARDS" ]; then
echo "error: shard index $SHARD_INDEX out of range 1..$TOTAL_SHARDS" >&2
exit 1
fi
cd "$(dirname "$0")/.."
# Find all unit test files, deterministic order. Excludes test/e2e/.
# Portable: avoid `mapfile` (bash 4+) so this runs on macOS bash 3.2 too.
FILES=()
while IFS= read -r line; do
FILES+=("$line")
done < <(find test -name '*.test.ts' -not -path 'test/e2e/*' | sort)
if [ "${#FILES[@]}" -eq 0 ]; then
echo "no test files found under test/" >&2
exit 1
fi
# FNV-1a 32-bit hash of a string — implemented in pure bash so we don't depend
# on python/openssl/etc on the runner. Output is decimal.
fnv1a() {
local str="$1"
local h=2166136261 # FNV offset basis
local i ord
for (( i=0; i<${#str}; i++ )); do
ord=$(printf '%d' "'${str:$i:1}")
h=$(( (h ^ ord) & 0xFFFFFFFF ))
h=$(( (h * 16777619) & 0xFFFFFFFF ))
done
echo "$h"
}
SHARD_FILES=()
for f in "${FILES[@]}"; do
hash=$(fnv1a "$f")
bucket=$(( hash % TOTAL_SHARDS + 1 ))
if [ "$bucket" -eq "$SHARD_INDEX" ]; then
SHARD_FILES+=("$f")
fi
done
echo "shard $SHARD_INDEX/$TOTAL_SHARDS: ${#SHARD_FILES[@]}/${#FILES[@]} files"
if [ "${#SHARD_FILES[@]}" -eq 0 ]; then
echo "warning: shard $SHARD_INDEX has no files (rehash or reduce shard count)" >&2
exit 0
fi
exec bun test --timeout=60000 "${SHARD_FILES[@]}"
+60
View File
@@ -0,0 +1,60 @@
# Friction protocol — convention
> Cross-cutting rule shared by skills the claw-test harness exercises (setup,
> brain-ops, query, ingest, smoke-test, migrations). Reference via
> `> **Convention:** see [skills/_friction-protocol.md](_friction-protocol.md).`
When you encounter friction running gbrain — anything confusing, missing, surprising, or wrong — log it via `gbrain friction log` so maintainers can see it without you writing a bug report. Friction reports drive the claw-test feedback loop (the harness collects, renders, and re-runs).
## When to log
Log friction when any of these happens:
- A command failed with a non-actionable error message
- A doc said one thing and the tool did another
- You couldn't find the next step
- A setup command needed a manual workaround
- A flag exists but isn't documented in `--help`
- A success condition was unclear (you couldn't tell if the command worked)
Log delight (positive signal) when:
- Something worked on the first try and the docs were exactly right
- An error message handed you the fix
- A flag you guessed at turned out to exist with the obvious name
## How to log
```
gbrain friction log \
--severity {confused|error|blocker|nit} \
--phase <which-phase-or-command> \
--message "<one-line-what-happened>" \
[--hint "<one-line-what-could-be-better>"]
```
For delight, add `--kind delight` and pick any severity.
The CLI auto-fills `ts`, `cwd`, `gbrain_version`, and resolves `run_id` from `$GBRAIN_FRICTION_RUN_ID` (set by the harness) or falls back to `standalone.jsonl`. So you can call this anywhere — inside a harness run, manually during normal use, or from a scripted test.
## Severity guide
| severity | meaning |
|------------|---------|
| `blocker` | Couldn't proceed at all. Hard stop. |
| `error` | Command failed unexpectedly. |
| `confused` | Docs/tool mismatch, ambiguity, missing pointer. |
| `nit` | Polish opportunity. Cosmetic or low-impact. |
Be specific: "doctor says `schema_version=0` and points at apply-migrations, but apply-migrations exits 0 with no output" beats "doctor was confusing."
## Inspecting reports
```
gbrain friction list # recent runs with counts
gbrain friction render --run-id <id> # markdown report (default)
gbrain friction render --run-id <id> --json
gbrain friction summary --run-id <id> # friction + delight side-by-side
```
`render` defaults to `--redact` for markdown (strips `$HOME`/`$CWD` to `<HOME>`/`<CWD>` placeholders) so reports paste safely into PRs and issues.
+210
View File
@@ -0,0 +1,210 @@
---
feature_pitch:
headline: Bare workers now self-monitor and fail-stop into your PM's restart loop
body: |
Bare `gbrain jobs work` now ships with the same health protection the
supervisor already had: DB liveness probes (with per-probe timeout so a
hung connection can't wedge the monitor), stall detection filtered by
registered handler names, and an RSS watchdog default of 2048 MB.
When the worker detects it's wedged (stuck pgbouncer connection, hung
event loop, stalled job claim), it emits `'unhealthy'` and the CLI calls
`process.exit(1)`. This is **fail-stop**: it requires an external process
manager (systemd, Docker `restart: always`, launchd `KeepAlive`, cron
watchdog) to bring the worker back. Without one, the process exits and
stays dead — that's a regression from pre-v0.22.14 self-healing.
Pre-v0.22.14 behavior: bare workers had ZERO health monitoring. A wedged
worker stayed alive doing nothing while jobs piled up in `waiting` and
your PM's `pgrep` check happily reported green.
If you're using `gbrain jobs supervisor`, you're already protected — the
supervisor handles spawn-on-crash itself. The fail-stop concern only
applies to direct `gbrain jobs work` invocations.
---
# v0.22.14 — Bare-worker self-health-monitoring
## ⚠️ Pre-flight: confirm you have a process supervisor
If you run `gbrain jobs work` directly (NOT under `gbrain jobs supervisor`),
verify your process manager is configured to restart the worker on exit
BEFORE upgrading:
| Manager | What to check |
|---|---|
| systemd | `Restart=always` (or `Restart=on-failure`) in the `.service` unit |
| Docker | `restart: always` / `restart: unless-stopped` in compose, OR `--restart` flag |
| launchd (macOS) | `<key>KeepAlive</key><true/>` in the plist |
| cron watchdog | Cron entry that re-spawns when `pgrep -f "gbrain jobs work"` is empty |
| supervisord | `autorestart=true` |
**If your bare worker has no restart loop, the v0.22.14 fail-stop behavior
will leave you with a dead worker after the first DB blip.** Either add a
restart policy OR switch to `gbrain jobs supervisor` (which spawns its own
child + restarts on crash internally).
## What ships
- DB liveness probes inside `gbrain jobs work` (60s interval, 3 strikes → exit)
- Stall detection (5min warn / 10min exit when waiting jobs accumulate but
in-flight is empty)
- `--max-rss` defaults to 2048 MB for bare workers (matches supervisor default;
was 0 = disabled)
- New `MinionWorkerOpts.{healthCheckInterval, stallWarnAfterMs,
stallExitAfterMs, dbFailExitAfter, dbProbeTimeoutMs}` for tuning (5 fields)
- `MinionWorker` now extends `EventEmitter`; emits `'unhealthy'` event with
a structured reason payload. **No-listener fallback**: if the caller does
not subscribe to `'unhealthy'`, the worker calls `process.exit(1)` itself
to preserve the pre-refactor fail-stop behavior. The CLI subscribes; direct
API consumers without a listener inherit the fail-stop default. Inline
paths (`jobs submit --follow`, `jobs smoke`) explicitly pass
`healthCheckInterval: 0` to disable the timer entirely so the fallback
cannot trip on a one-shot run.
- `GBRAIN_SUPERVISED=1` env var (set by supervisor on its child) causes the
child worker to skip its own health timer (no double-monitoring)
- Constructor validation: throws if `stallExitAfterMs <= stallWarnAfterMs`
(the contract requires strict greater-than)
## ⚠️ Breaking change you may need to act on
**`gbrain jobs work` now defaults `--max-rss 2048` (MB).** Pre-v0.22.14, bare
workers had no RSS limit. If you ran bare workers with intentionally large
embed/import working sets, your worker may now get killed by the watchdog after
upgrade.
**You need to act if any of the following is true:**
- You run `gbrain jobs work` directly under a process manager (systemd, Docker,
cron, supervisord, your own bash wrapper)
- Your worker handles large embed batches, multi-GB import jobs, or
memory-intensive subagent tasks that legitimately exceed 2 GB RSS
- You rely on bare-worker resident memory exceeding 2 GB
**You do NOT need to act if:**
- You run `gbrain jobs supervisor` (the supervisor already defaulted to
`--max-rss 2048` and has reconnect-then-fail logic on top)
- Your typical worker stays well under 2 GB RSS
## What the agent should do post-upgrade
### 1. Run the orchestrator (mechanical migration)
```bash
gbrain apply-migrations --yes
```
This is a no-op for v0.22.14 — there's no schema migration. The mechanical work
is in the binary; the user-side decision is the breaking-change handling below.
### 2. Detect bare-worker deployments
If the user runs OpenClaw, Hermes, or any agent fork that uses `gbrain jobs
work` directly (NOT under `gbrain jobs supervisor`), they need to know. Check:
```bash
# Look for bare worker invocations in cron, systemd, Docker, launchctl, etc.
grep -r "gbrain jobs work" ~/.config/systemd/ /etc/systemd/ 2>/dev/null | head
crontab -l 2>/dev/null | grep "gbrain jobs work" | head
launchctl list 2>/dev/null | grep gbrain | head
ps -ef | grep "gbrain jobs work" | grep -v grep | grep -v supervisor
```
If you find bare-worker invocations, surface them to the user.
### 3. Ask about RSS profile
Use AskUserQuestion to confirm whether the user's bare workers need >2 GB RSS:
> "v0.22.14 changes `gbrain jobs work` to default `--max-rss 2048` (MB).
> Pre-v0.22.14 bare workers had no limit. If your worker handles large embed
> batches or multi-GB imports, the watchdog may now kill it. Do you want
> to keep the new 2 GB default, raise the limit, or opt out entirely?"
Options:
- **A) Keep 2 GB default (recommended for most)** — protects against memory
leaks; restarts on overflow; matches supervisor behavior.
- **B) Raise to N GB (specify N)** — pass `--max-rss <N*1024>` to the worker
invocation.
- **C) Opt out** — pass `--max-rss 0`.
### 4. Apply the user's choice
For each bare-worker invocation, edit the unit/cron/launchctl/script to add
the chosen `--max-rss` flag.
**systemd (~/.config/systemd/user/gbrain-worker.service):**
```ini
ExecStart=/usr/local/bin/gbrain jobs work --queue default --concurrency 3 --max-rss 4096
# Or to opt out: --max-rss 0
```
Then `systemctl --user daemon-reload && systemctl --user restart gbrain-worker`.
**cron (`crontab -e`):**
```cron
@reboot /usr/local/bin/gbrain jobs work --queue default --concurrency 3 --max-rss 4096
```
**Docker compose:**
```yaml
command: ["gbrain", "jobs", "work", "--queue", "default", "--concurrency", "3", "--max-rss", "4096"]
```
**launchctl (~/Library/LaunchAgents/com.user.gbrain-worker.plist):**
```xml
<key>ProgramArguments</key>
<array>
<string>/usr/local/bin/gbrain</string>
<string>jobs</string>
<string>work</string>
<string>--max-rss</string>
<string>4096</string>
</array>
```
Then `launchctl unload ... && launchctl load ...`.
### 5. (Optional) Tune health-check thresholds
The new opts default to sensible values (60s probe interval, 5min warn / 10min
exit, 3 DB failures). If you have specific SLAs, you can pass `--health-interval
<ms>` to adjust the probe cadence. Stall thresholds are not yet CLI-exposed
(only the API; CLI flags coming in a follow-up).
To disable self-monitoring entirely (e.g. you have your own external health
checker):
```bash
gbrain jobs work --health-interval 0 --max-rss 0
```
### 6. Verify
```bash
gbrain jobs stats # queue should be flowing normally
gbrain doctor --json | jq '.' # no critical warnings
ps -o rss= -p $(pgrep -f "gbrain jobs work") | awk '{print $1/1024 " MB"}'
```
Worker startup log line should now show health-check status:
```
Minion worker started (queue: default, concurrency: 3, watchdog: 2048MB, health-check: 60s)
```
If running under supervisor, you'll see the watchdog but NOT the `health-check:
60s` segment (because `GBRAIN_SUPERVISED=1` skips the child's self-monitor).
### 7. If anything fails
Open an issue at https://github.com/garrytan/gbrain/issues with:
- Output of `gbrain doctor`
- Contents of `~/.gbrain/upgrade-errors.jsonl` if it exists
- Your bare-worker invocation (systemd unit / cron line / Dockerfile snippet)
- Which step broke
+24 -2
View File
@@ -19,7 +19,7 @@ for (const op of operations) {
}
// CLI-only commands that bypass the operation layer
const CLI_ONLY = new Set(['init', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'repos', 'code-def', 'code-refs', 'reindex-code', 'code-callers', 'code-callees', 'frontmatter']);
const CLI_ONLY = new Set(['init', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'storage', 'repos', 'code-def', 'code-refs', 'reindex-code', 'code-callers', 'code-callees', 'frontmatter', 'auth', 'friction', 'claw-test']);
async function main() {
// Parse global flags (--quiet / --progress-json / --progress-interval)
@@ -285,6 +285,11 @@ async function handleCliOnly(command: string, args: string[]) {
await runIntegrations(args);
return;
}
if (command === 'auth') {
const { runAuth } = await import('./commands/auth.ts');
await runAuth(args);
return;
}
if (command === 'resolvers') {
const { runResolvers } = await import('./commands/resolvers.ts');
await runResolvers(args);
@@ -338,6 +343,14 @@ async function handleCliOnly(command: string, args: string[]) {
await runSkillpack(args);
return;
}
if (command === 'friction') {
const { runFriction } = await import('./commands/friction.ts');
process.exit(runFriction(args));
}
if (command === 'claw-test') {
const { runClawTest } = await import('./commands/claw-test.ts');
process.exit(await runClawTest(args));
}
if (command === 'report') {
const { runReport } = await import('./commands/report.ts');
await runReport(args);
@@ -447,7 +460,7 @@ async function handleCliOnly(command: string, args: string[]) {
}
case 'serve': {
const { runServe } = await import('./commands/serve.ts');
await runServe(engine);
await runServe(engine, args);
return; // serve doesn't disconnect
}
case 'call': {
@@ -525,6 +538,11 @@ async function handleCliOnly(command: string, args: string[]) {
await runSources(engine, args);
break;
}
case 'storage': {
const { runStorage } = await import('./commands/storage.ts');
await runStorage(engine, args);
break;
}
case 'code-def': {
const { runCodeDef } = await import('./commands/code-def.ts');
await runCodeDef(engine, args);
@@ -640,6 +658,8 @@ IMPORT/EXPORT
sync --watch [--interval N] Continuous sync (loops until stopped)
sync --install-cron Install persistent sync daemon
export [--dir ./out/] Export to markdown
export --restore-only [--repo <p>] Restore missing supabase-only files
[--type T] [--slug-prefix S] With optional filters
FILES
files list [slug] List stored files
@@ -721,6 +741,8 @@ ADMIN
features [--json] [--auto-fix] Scan usage + recommend unused features
autopilot [--repo] [--interval N] Self-maintaining brain daemon
config [show|get|set] <key> [val] Brain config
storage status [--repo <path>] Storage tier status and health
[--json] (git-tracked vs supabase-only)
serve MCP server (stdio)
call <tool> '<json>' Raw tool invocation
version Version info
+52 -31
View File
@@ -1,20 +1,29 @@
#!/usr/bin/env bun
/**
* GBrain token management standalone script, no gbrain CLI dependency.
* GBrain token management.
*
* Usage:
* Wired into the CLI as of v0.22.5:
* gbrain auth create "claude-desktop"
* gbrain auth list
* gbrain auth revoke "claude-desktop"
* gbrain auth test <url> --token <token>
*
* Also runs standalone (no compiled binary required):
* DATABASE_URL=... bun run src/commands/auth.ts create "claude-desktop"
* DATABASE_URL=... bun run src/commands/auth.ts list
* DATABASE_URL=... bun run src/commands/auth.ts revoke "claude-desktop"
* DATABASE_URL=... bun run src/commands/auth.ts test <url> --token <token>
*
* Both paths require DATABASE_URL or GBRAIN_DATABASE_URL (except `test`,
* which only hits the remote URL and doesn't need a local DB).
*/
import postgres from 'postgres';
import { createHash, randomBytes } from 'crypto';
const DATABASE_URL = process.env.DATABASE_URL || process.env.GBRAIN_DATABASE_URL;
if (!DATABASE_URL && process.argv[2] !== 'test') {
console.error('Set DATABASE_URL or GBRAIN_DATABASE_URL environment variable.');
process.exit(1);
function getDatabaseUrl(requireDb: boolean): string | undefined {
const url = process.env.DATABASE_URL || process.env.GBRAIN_DATABASE_URL;
if (!url && requireDb) {
console.error('Set DATABASE_URL or GBRAIN_DATABASE_URL environment variable.');
process.exit(1);
}
return url;
}
function hashToken(token: string): string {
@@ -27,7 +36,7 @@ function generateToken(): string {
async function create(name: string) {
if (!name) { console.error('Usage: auth create <name>'); process.exit(1); }
const sql = postgres(DATABASE_URL!);
const sql = postgres(getDatabaseUrl(true)!);
const token = generateToken();
const hash = hashToken(token);
@@ -53,7 +62,7 @@ async function create(name: string) {
}
async function list() {
const sql = postgres(DATABASE_URL!);
const sql = postgres(getDatabaseUrl(true)!);
try {
const rows = await sql`
SELECT name, created_at, last_used_at, revoked_at
@@ -80,7 +89,7 @@ async function list() {
async function revoke(name: string) {
if (!name) { console.error('Usage: auth revoke <name>'); process.exit(1); }
const sql = postgres(DATABASE_URL!);
const sql = postgres(getDatabaseUrl(true)!);
try {
const result = await sql`
UPDATE access_tokens SET revoked_at = now()
@@ -216,26 +225,38 @@ async function test(url: string, token: string) {
console.log(`\n🧠 Your brain is live! (${elapsed}s)`);
}
// CLI dispatch
const [cmd, ...args] = process.argv.slice(2);
switch (cmd) {
case 'create': await create(args[0]); break;
case 'list': await list(); break;
case 'revoke': await revoke(args[0]); break;
case 'test': {
const tokenIdx = args.indexOf('--token');
const url = args.find(a => !a.startsWith('--') && a !== args[tokenIdx + 1]);
const token = tokenIdx >= 0 ? args[tokenIdx + 1] : '';
await test(url || '', token || '');
break;
}
default:
console.log(`GBrain Token Management
/**
* Entry point for the `gbrain auth` CLI subcommand. Also reused by the
* direct-script path (see bottom of file) so `bun run src/commands/auth.ts`
* still works.
*/
export async function runAuth(args: string[]): Promise<void> {
const [cmd, ...rest] = args;
switch (cmd) {
case 'create': await create(rest[0]); return;
case 'list': await list(); return;
case 'revoke': await revoke(rest[0]); return;
case 'test': {
const tokenIdx = rest.indexOf('--token');
const url = rest.find(a => !a.startsWith('--') && a !== rest[tokenIdx + 1]);
const token = tokenIdx >= 0 ? rest[tokenIdx + 1] : '';
await test(url || '', token || '');
return;
}
default:
console.log(`GBrain Token Management
Usage:
bun run src/commands/auth.ts create <name> Create a new access token
bun run src/commands/auth.ts list List all tokens
bun run src/commands/auth.ts revoke <name> Revoke a token
bun run src/commands/auth.ts test <url> --token <token> Smoke test a remote MCP server
gbrain auth create <name> Create a new access token
gbrain auth list List all tokens
gbrain auth revoke <name> Revoke a token
gbrain auth test <url> --token <t> Smoke-test a remote MCP server
`);
}
}
// Direct-script entry point — only runs when this file is invoked as the main module
// (e.g. `bun run src/commands/auth.ts ...`). When imported by cli.ts, this block is skipped.
if (import.meta.main) {
await runAuth(process.argv.slice(2));
}
+424
View File
@@ -0,0 +1,424 @@
/**
* gbrain claw-test end-to-end "fresh user" test harness.
*
* Two tiers:
* gbrain claw-test scripted (no LLM, CI gate)
* gbrain claw-test --live --agent openclaw real agent, friction discovery
*
* Phases (scripted mode):
* setup install_brain import query extract verify render
*
* The harness sets GBRAIN_HOME=<tempdir> so the run is hermetic. Each child
* gbrain invocation runs with --progress-json and the harness captures stderr
* to assert expected_phases from scenario.json fired.
*
* See ~/.claude/plans/system-instruction-you-are-working-noble-biscuit.md
* for the full design rationale (D1D23 decisions).
*/
import { spawn } from 'child_process';
import { mkdtempSync, writeFileSync, mkdirSync, rmSync, existsSync } from 'fs';
import { join } from 'path';
import { tmpdir } from 'os';
import { randomBytes } from 'crypto';
import { logFriction, frictionDir } from '../core/friction.ts';
import { loadScenario, listScenarios, readBrief, type ScenarioConfig } from '../core/claw-test/scenarios.ts';
import { parseProgressEvents, verifyExpectedPhases } from '../core/claw-test/progress-tail.ts';
import { resolveAgentRunner, listRegisteredAgents, registerAgentRunner } from '../core/claw-test/agent-runner.ts';
import { OpenClawRunner } from '../core/claw-test/runners/openclaw.ts';
import { createTranscriptSink } from '../core/claw-test/transcript-capture.ts';
// Ensure built-in runners are registered.
registerAgentRunner('openclaw', () => new OpenClawRunner());
interface HarnessOpts {
scenario: string;
live: boolean;
agent: string;
keepTempdir: boolean;
listAgents: boolean;
help: boolean;
/** Path to the gbrain binary used to invoke child commands. Defaults to argv[0]. */
gbrainBin?: string;
}
interface PhaseOutcome {
phase: string;
exitCode: number;
durationMs: number;
stderrEvents: number;
stdoutTail: string;
stderrTail: string;
}
const TAIL_BYTES = 4_096;
const SUBPROCESS_TIMEOUT_MS = 5 * 60_000; // 5 minutes per phase
export async function runClawTest(args: string[]): Promise<number> {
const opts = parseArgs(args);
if (opts.help) {
printHelp();
return 0;
}
if (opts.listAgents) {
return cmdListAgents();
}
let scenario: ScenarioConfig;
try {
scenario = loadScenario(opts.scenario);
} catch (e) {
console.error(`scenario load failed: ${e instanceof Error ? e.message : String(e)}`);
const available = listScenarios();
if (available.length) console.error(`available scenarios: ${available.join(', ')}`);
return 2;
}
const runId = newRunId(opts.agent);
const runRoot = mkdtempSync(join(tmpdir(), `claw-test-${runId}-`));
const gbrainHome = runRoot; // configDir() appends '.gbrain' itself
const transcriptPath = join(runRoot, 'transcript.jsonl');
console.log(`run-id: ${runId}`);
console.log(`tempdir: ${runRoot}`);
// SIGINT/SIGTERM finalization (D11)
let interrupted = false;
const onSignal = () => {
interrupted = true;
try {
logFriction({
runId,
phase: 'harness',
message: 'run interrupted by signal',
kind: 'interrupted',
source: 'harness',
agent: opts.agent,
});
} catch { /* best effort */ }
};
process.once('SIGINT', onSignal);
process.once('SIGTERM', onSignal);
let exitCode = 0;
try {
if (opts.live) {
exitCode = await runLive(opts, scenario, { runId, runRoot, gbrainHome, transcriptPath });
} else {
exitCode = await runScripted(opts, scenario, { runId, runRoot, gbrainHome });
}
} finally {
process.off('SIGINT', onSignal);
process.off('SIGTERM', onSignal);
if (!opts.keepTempdir && !interrupted) {
try { rmSync(runRoot, { recursive: true, force: true }); } catch { /* best effort */ }
} else {
console.log(`tempdir kept at: ${runRoot}`);
}
}
// Always render at the end so the operator can immediately see the report.
console.log('---');
console.log(`friction log: ${join(frictionDir(), runId + '.jsonl')}`);
console.log(`render report: gbrain friction render --run-id ${runId}`);
if (interrupted) return 130;
return exitCode;
}
// ---------------------------------------------------------------------------
// Scripted mode
// ---------------------------------------------------------------------------
async function runScripted(
opts: HarnessOpts,
scenario: ScenarioConfig,
ctx: { runId: string; runRoot: string; gbrainHome: string },
): Promise<number> {
const childEnv: Record<string, string> = {
...process.env as Record<string, string>,
GBRAIN_HOME: ctx.gbrainHome,
GBRAIN_FRICTION_RUN_ID: ctx.runId,
};
const phases: { name: string; argv: string[] }[] = [];
// Phase 2: install_brain
phases.push({ name: 'install_brain', argv: ['init', '--pglite'] });
// Phase 3: import (only when scenario has a brain dir)
if (scenario.brainRelative) {
const brainDir = join(scenario.dir, scenario.brainRelative);
phases.push({ name: 'import', argv: ['import', brainDir, '--no-embed', '--progress-json'] });
}
// Phase 4: query (best-effort sanity)
phases.push({ name: 'query', argv: ['query', 'the'] });
// Phase 5: extract (positional argument is required: 'all' covers links + timeline)
phases.push({ name: 'extract', argv: ['extract', 'all', '--source', 'fs', '--progress-json'] });
// Phase 6: verify
phases.push({ name: 'verify', argv: ['doctor', '--json', '--progress-json'] });
// Pre-phase: upgrade scenario seeds the database
if (scenario.kind === 'upgrade' && scenario.seedRelative) {
const seedSql = join(scenario.dir, scenario.seedRelative, 'dump.sql');
if (existsSync(seedSql)) {
const dbPath = join(ctx.gbrainHome, '.gbrain', 'brain.pglite');
mkdirSync(join(ctx.gbrainHome, '.gbrain'), { recursive: true });
const { seedPgliteFromFile } = await import('../core/claw-test/seed-pglite.ts');
try {
await seedPgliteFromFile({ dbPath, sqlPath: seedSql });
console.log(`[seed] replayed ${seedSql}${dbPath}`);
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
logFriction({
runId: ctx.runId,
phase: 'seed',
message: `seed replay failed: ${msg}`,
severity: 'blocker',
source: 'harness',
agent: opts.agent,
});
return 1;
}
}
}
const allStderr: string[] = [];
const outcomes: PhaseOutcome[] = [];
for (const phase of phases) {
const outcome = await invokeGbrain(opts.gbrainBin ?? 'gbrain', phase.argv, ctx.runRoot, childEnv);
outcome.phase = phase.name;
outcomes.push(outcome);
allStderr.push(outcome.stderrTail);
if (outcome.exitCode !== 0) {
logFriction({
runId: ctx.runId,
phase: phase.name,
message: `command failed (exit ${outcome.exitCode}): gbrain ${phase.argv.join(' ')}`,
severity: 'error',
hint: outcome.stderrTail.trim().slice(0, 500),
source: 'harness',
agent: opts.agent,
});
return 1;
} else {
logFriction({
runId: ctx.runId,
phase: phase.name,
message: `phase complete in ${outcome.durationMs}ms`,
kind: 'phase-marker',
marker: 'end',
source: 'harness',
agent: opts.agent,
});
}
}
// Phase verification: collect all events from every captured stderr and assert coverage.
const events = allStderr.flatMap(parseProgressEvents);
const missing = verifyExpectedPhases(events, scenario.expectedPhases);
if (missing.length) {
for (const phaseName of missing) {
logFriction({
runId: ctx.runId,
phase: phaseName,
message: `expected progress event for "${phaseName}" never fired`,
severity: 'blocker',
hint: 'either the command did not run or it did not emit progress events; check phase log above',
source: 'harness',
agent: opts.agent,
});
}
return 1;
}
return 0;
}
// ---------------------------------------------------------------------------
// Live mode
// ---------------------------------------------------------------------------
async function runLive(
opts: HarnessOpts,
scenario: ScenarioConfig,
ctx: { runId: string; runRoot: string; gbrainHome: string; transcriptPath: string },
): Promise<number> {
let runner;
try {
runner = resolveAgentRunner(opts.agent);
} catch (e) {
console.error(e instanceof Error ? e.message : String(e));
return 2;
}
const detected = await runner.detect();
if (!detected.available) {
console.error(`agent "${opts.agent}" not available: ${detected.reason ?? 'unknown'}`);
logFriction({
runId: ctx.runId,
phase: 'agent_detect',
message: `agent ${opts.agent} not available: ${detected.reason ?? 'unknown'}`,
severity: 'blocker',
hint: opts.agent === 'openclaw' ? 'install openclaw or set OPENCLAW_BIN' : undefined,
source: 'harness',
agent: opts.agent,
});
return 2;
}
const sink = createTranscriptSink(ctx.transcriptPath);
const env: Record<string, string> = {
GBRAIN_HOME: ctx.gbrainHome,
GBRAIN_FRICTION_RUN_ID: ctx.runId,
};
const brief = readBrief(scenario);
let result;
try {
result = await runner.invoke({
cwd: ctx.runRoot,
brief,
env,
timeoutMs: SUBPROCESS_TIMEOUT_MS,
transcriptSink: sink,
});
} finally {
await sink.close();
}
if (result.exitCode !== 0) {
logFriction({
runId: ctx.runId,
phase: 'agent_invoke',
message: `agent exited with code ${result.exitCode} after ${result.durationMs}ms`,
severity: 'error',
source: 'harness',
agent: opts.agent,
});
return result.exitCode;
}
return 0;
}
// ---------------------------------------------------------------------------
// Subprocess helpers
// ---------------------------------------------------------------------------
function invokeGbrain(
bin: string,
argv: string[],
cwd: string,
env: Record<string, string>,
): Promise<PhaseOutcome> {
return new Promise((resolve) => {
const start = Date.now();
const child = spawn(bin, argv, { cwd, env, stdio: ['ignore', 'pipe', 'pipe'], shell: false });
const stdout: Buffer[] = [];
const stderr: Buffer[] = [];
child.stdout?.on('data', (b: Buffer) => stdout.push(b));
child.stderr?.on('data', (b: Buffer) => stderr.push(b));
child.on('error', (err) => {
const stderrJoined = Buffer.concat(stderr).toString('utf-8') + '\nspawn error: ' + err.message;
resolve({
phase: '',
exitCode: 127,
durationMs: Date.now() - start,
stderrEvents: 0,
stdoutTail: tailOf(Buffer.concat(stdout).toString('utf-8')),
stderrTail: tailOf(stderrJoined),
});
});
child.on('close', (code) => {
const stderrText = Buffer.concat(stderr).toString('utf-8');
resolve({
phase: '',
exitCode: typeof code === 'number' ? code : 1,
durationMs: Date.now() - start,
stderrEvents: parseProgressEvents(stderrText).length,
stdoutTail: tailOf(Buffer.concat(stdout).toString('utf-8')),
stderrTail: stderrText,
});
});
});
}
function tailOf(s: string): string {
if (s.length <= TAIL_BYTES) return s;
return s.slice(-TAIL_BYTES);
}
// ---------------------------------------------------------------------------
// Argv parsing + helpers
// ---------------------------------------------------------------------------
function parseArgs(args: string[]): HarnessOpts {
const out: HarnessOpts = {
scenario: 'fresh-install',
live: false,
agent: 'openclaw',
keepTempdir: false,
listAgents: false,
help: args.includes('--help') || args.includes('-h'),
gbrainBin: process.env.GBRAIN_BIN_OVERRIDE || process.execPath,
};
for (let i = 0; i < args.length; i++) {
const a = args[i];
if (a === '--live') out.live = true;
else if (a === '--keep-tempdir') out.keepTempdir = true;
else if (a === '--list-agents') out.listAgents = true;
else if (a === '--scenario') out.scenario = args[++i] ?? out.scenario;
else if (a === '--agent') out.agent = args[++i] ?? out.agent;
}
return out;
}
function newRunId(agent: string): string {
const now = new Date();
const ts = now.toISOString().replace(/[-:]/g, '').replace(/\..*/, '').replace('T', '-');
const suf = randomBytes(4).toString('hex');
return `claw-test-${ts}-${agent}-${suf}`;
}
function cmdListAgents(): number {
const names = listRegisteredAgents();
if (!names.length) {
console.log('no agents registered');
return 0;
}
for (const name of names) {
try {
const runner = resolveAgentRunner(name);
runner.detect().then((d) => {
const status = d.available ? `available at ${d.binPath}` : `unavailable: ${d.reason}`;
console.log(`${name}: ${status}`);
}).catch(() => { /* best effort */ });
} catch {
console.log(`${name}: (factory error)`);
}
}
return 0;
}
function printHelp() {
console.log(`gbrain claw-test — end-to-end claw-setup friction harness
Usage:
gbrain claw-test [--scenario <name>] [--live --agent <name>] [--keep-tempdir]
gbrain claw-test --list-agents
Defaults:
--scenario fresh-install
--agent openclaw (live mode only)
Scripted mode runs canonical commands without an LLM (CI gate).
Live mode spawns a real agent and lets it drive (~510 min, costs tokens).
Examples:
gbrain claw-test --scenario fresh-install
gbrain claw-test --scenario upgrade-from-v0.18 --keep-tempdir
gbrain claw-test --live --agent openclaw`);
}
+40 -4
View File
@@ -249,25 +249,29 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
// Without this doctor check, users see "sync blocked" and have no
// surface showing which files to fix.
try {
const { unacknowledgedSyncFailures, loadSyncFailures } = await import('../core/sync.ts');
const { unacknowledgedSyncFailures, loadSyncFailures, summarizeFailuresByCode } = await import('../core/sync.ts');
const unacked = unacknowledgedSyncFailures();
const all = loadSyncFailures();
if (unacked.length > 0) {
const codeSummary = summarizeFailuresByCode(unacked);
const codeBreakdown = codeSummary.map(s => `${s.code}=${s.count}`).join(', ');
const preview = unacked.slice(0, 3).map(f => `${f.path} (${f.error.slice(0, 60)})`).join('; ');
checks.push({
name: 'sync_failures',
status: 'warn',
message:
`${unacked.length} unacknowledged sync failure(s). ${preview}` +
`${unacked.length} unacknowledged sync failure(s) [${codeBreakdown}]. ${preview}` +
`${unacked.length > 3 ? `, and ${unacked.length - 3} more` : ''}. ` +
`Fix the file(s) and re-run 'gbrain sync', or use 'gbrain sync --skip-failed' to acknowledge.`,
});
} else if (all.length > 0) {
// Acknowledged-only: informational, not a warning.
// Acknowledged-only: show code breakdown for visibility.
const ackedSummary = summarizeFailuresByCode(all);
const ackedBreakdown = ackedSummary.map(s => `${s.code}=${s.count}`).join(', ');
checks.push({
name: 'sync_failures',
status: 'ok',
message: `${all.length} historical sync failure(s), all acknowledged.`,
message: `${all.length} historical sync failure(s), all acknowledged [${ackedBreakdown}].`,
});
}
} catch {
@@ -770,6 +774,30 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
ORDER BY depth DESC
LIMIT 5
`;
// Subcheck 3 (v0.22.14): RSS-watchdog kills in the last 24h. Bare workers
// newly default to --max-rss 2048 (was 0); operators who run large embed
// or import jobs may see kills that didn't happen pre-v0.22.14. We surface
// a hint when this signature appears so the upgrade path is obvious.
// Signature: when the watchdog trips, gracefulShutdown('watchdog') aborts
// in-flight jobs with `new Error('watchdog')`. The worker's failJob path
// (worker.ts:660-664) writes `error_text = 'aborted: watchdog'` for any
// job in-flight at the moment of the kill.
//
// We deliberately DO NOT do a loose `ILIKE '%watchdog%'`:
// 1. Parent jobs that inherit `on_child_fail='fail_parent'` get
// `"child job N failed: aborted: watchdog"` — counting that
// double-counts (child + parent) for one watchdog event.
// 2. Any user error_text containing the word "watchdog" matches.
// Match the exact prefix `'aborted: watchdog'` to scope this purely to
// the worker's own kill signature.
const rssKillRows: Array<{ cnt: number }> = await sql`
SELECT count(*)::int AS cnt
FROM minion_jobs
WHERE status IN ('dead', 'failed')
AND finished_at > now() - interval '24 hours'
AND error_text = 'aborted: watchdog'
`;
const rssKillCount = rssKillRows[0]?.cnt ?? 0;
const problems: string[] = [];
if (stalledRows.length > 0) {
@@ -790,6 +818,14 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
`Fix: set maxWaiting on the submitter (or raise GBRAIN_QUEUE_WAITING_THRESHOLD).`
);
}
if (rssKillCount > 0) {
problems.push(
`${rssKillCount} job(s) dead-lettered for RSS-watchdog memory-limit kills in last 24h. ` +
`v0.22.14 changed the bare-worker --max-rss default from 0 (off) to 2048 MB. ` +
`Fix: raise the limit (e.g. \`gbrain jobs work --max-rss 4096\`) or opt out (\`--max-rss 0\`). ` +
`See skills/migrations/v0.22.14.md.`
);
}
if (problems.length === 0) {
checks.push({
+97 -4
View File
@@ -1,16 +1,105 @@
import { writeFileSync, mkdirSync } from 'fs';
import { writeFileSync, mkdirSync, existsSync } from 'fs';
import { join, dirname } from 'path';
import type { BrainEngine } from '../core/engine.ts';
import { serializeMarkdown } from '../core/markdown.ts';
import { createProgress } from '../core/progress.ts';
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
import { loadStorageConfig, isDbOnly } from '../core/storage-config.ts';
import { getDefaultSourcePath } from '../core/source-resolver.ts';
import type { PageType } from '../core/types.ts';
export async function runExport(engine: BrainEngine, args: string[]) {
const dirIdx = args.indexOf('--dir');
const outDir = dirIdx !== -1 ? args[dirIdx + 1] : './export';
const pages = await engine.listPages({ limit: 100000 });
console.log(`Exporting ${pages.length} pages to ${outDir}/`);
const repoIdx = args.indexOf('--repo');
const explicitRepoPath = repoIdx !== -1 ? args[repoIdx + 1] : null;
const typeIdx = args.indexOf('--type');
const typeFilter = typeIdx !== -1 ? (args[typeIdx + 1] as PageType) : undefined;
const slugPrefixIdx = args.indexOf('--slug-prefix');
const slugPrefix = slugPrefixIdx !== -1 ? args[slugPrefixIdx + 1] : undefined;
const restoreOnly = args.includes('--restore-only');
// Resolution chain (D5): explicit --repo → typed sources.getDefault() →
// hard-error for restore-only paths (never fall through to cwd).
// For non-restore exports, repoPath stays null because regular export
// doesn't need a brain repo to run (D26 — exports include everything).
let repoPath: string | null = explicitRepoPath;
if (restoreOnly && !repoPath) {
repoPath = await getDefaultSourcePath(engine);
if (!repoPath) {
console.error(
`Error: gbrain export --restore-only requires --repo <path> or a configured\n` +
`default source with a local_path. Run \`gbrain sources list\` to inspect\n` +
`sources, or pass --repo explicitly.`,
);
process.exit(1);
}
}
// Load storage configuration if repo path is provided
const storageConfig = repoPath ? loadStorageConfig(repoPath) : null;
// D5 + Codex P0: refuse --restore-only when there's no storage config to
// scope the restore. Without storageConfig, the selective filter (db_only
// pages missing on disk) can't run, and falling through to the full
// listPages export silently dumps the entire DB. Catch this before any
// page query fires.
if (restoreOnly && !storageConfig) {
console.error(
`Error: gbrain export --restore-only requires a storage tiering config\n` +
`(gbrain.yml with a "storage:" section) at ${repoPath}/gbrain.yml.\n` +
`Without it, there's nothing to scope the restore to.\n` +
`Run \`gbrain storage status\` to inspect the current configuration.`,
);
process.exit(1);
}
// Build filters. slugPrefix is engine-side (Issue #13) — no in-memory
// post-filter, no full-table load.
const filters: import('../core/types.ts').PageFilters = { limit: 100000 };
if (typeFilter) filters.type = typeFilter;
if (slugPrefix) filters.slugPrefix = slugPrefix;
let pages: import('../core/types.ts').Page[];
// Restore-only path: query each db_only directory with slugPrefix instead
// of loading every page in the brain. On a 200K-page brain where 95% is
// db_only, this is roughly the same load — but on brains where only 5K
// out of 200K are db_only, this is a ~40x reduction.
if (restoreOnly && repoPath && storageConfig) {
const seen = new Set<string>();
pages = [];
for (const dir of storageConfig.db_only) {
const tierFilters: import('../core/types.ts').PageFilters = {
...filters,
slugPrefix: filters.slugPrefix
? // If user passed --slug-prefix, only include tier dirs that start with it.
(dir.startsWith(filters.slugPrefix) ? dir : undefined)
: dir,
};
if (!tierFilters.slugPrefix) continue;
const tierPages = await engine.listPages(tierFilters);
for (const p of tierPages) {
if (seen.has(p.slug)) continue;
seen.add(p.slug);
if (!isDbOnly(p.slug, storageConfig)) continue; // belt-and-suspenders
const filePath = join(repoPath, p.slug + '.md');
if (existsSync(filePath)) continue;
pages.push(p);
}
}
} else {
pages = await engine.listPages(filters);
}
if (restoreOnly) {
console.log(`Restoring ${pages.length} db_only pages to ${outDir}/`);
} else {
console.log(`Exporting ${pages.length} pages to ${outDir}/`);
}
// Progress on stderr so stdout stays clean for scripts parsing counts.
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
@@ -52,5 +141,9 @@ export async function runExport(engine: BrainEngine, args: string[]) {
progress.finish();
// Stdout summary preserved so scripts that grep for "Exported N pages" keep working.
console.log(`Exported ${exported} pages to ${outDir}/`);
if (restoreOnly) {
console.log(`Restored ${exported} pages to ${outDir}/`);
} else {
console.log(`Exported ${exported} pages to ${outDir}/`);
}
}
+185
View File
@@ -0,0 +1,185 @@
/**
* gbrain friction friction reporter CLI.
*
* Four subcommands in v1 (analytical/clustering ones move to v1.1):
* gbrain friction log Append a friction or delight entry
* gbrain friction render Render a run as markdown or JSON
* gbrain friction list List recent runs with counts
* gbrain friction summary Side-by-side friction + delight summary
*
* Subcommands stay thin ( ~30 LOC each). Core logic lives in src/core/friction.ts.
*
* The CLI is dispatched from src/cli.ts. See `gbrain friction --help`.
*/
import {
logFriction, readFriction, listRuns, renderReport, renderSummary,
activeRunId, frictionFile,
type FrictionKind, type FrictionSeverity,
} from '../core/friction.ts';
const VALID_KINDS = new Set<FrictionKind>(['friction', 'delight', 'phase-marker', 'interrupted']);
const VALID_SEVERITIES = new Set<FrictionSeverity>(['confused', 'error', 'blocker', 'nit']);
export function runFriction(args: string[]): number {
const [sub, ...rest] = args;
switch (sub) {
case 'log': return cmdLog(rest);
case 'render': return cmdRender(rest);
case 'list': return cmdList(rest);
case 'summary': return cmdSummary(rest);
case undefined:
case '--help':
case '-h':
printHelp();
return 0;
default:
console.error(`unknown subcommand: ${sub}`);
printHelp();
return 2;
}
}
// ---------------------------------------------------------------------------
// log
// ---------------------------------------------------------------------------
function cmdLog(args: string[]): number {
const flags = parseFlags(args);
const phase = flags.string('--phase');
const message = flags.string('--message');
if (!phase || !message) {
console.error('usage: gbrain friction log --phase <name> --message <text> [--severity ...] [--hint ...] [--kind ...] [--run-id ...]');
return 2;
}
const kind = (flags.string('--kind') ?? 'friction') as FrictionKind;
if (!VALID_KINDS.has(kind)) {
console.error(`invalid --kind ${kind}; must be one of: ${[...VALID_KINDS].join(', ')}`);
return 2;
}
const severityRaw = flags.string('--severity');
const severity = severityRaw as FrictionSeverity | undefined;
if (severity && !VALID_SEVERITIES.has(severity)) {
console.error(`invalid --severity ${severity}; must be one of: ${[...VALID_SEVERITIES].join(', ')}`);
return 2;
}
try {
logFriction({
phase,
message,
kind,
severity,
hint: flags.string('--hint'),
runId: flags.string('--run-id'),
agent: flags.string('--agent'),
source: 'claw',
});
} catch (e) {
console.error(`friction log failed: ${e instanceof Error ? e.message : String(e)}`);
return 1;
}
return 0;
}
// ---------------------------------------------------------------------------
// render
// ---------------------------------------------------------------------------
function cmdRender(args: string[]): number {
const flags = parseFlags(args);
const runId = flags.string('--run-id') ?? activeRunId();
const json = flags.bool('--json');
const format = json ? 'json' : 'md';
const transcripts = flags.bool('--transcripts');
const noRedact = flags.bool('--no-redact');
// --redact is the default for md output; --no-redact disables.
const redact = noRedact ? false : (format === 'md');
try {
const out = renderReport(runId, {
format,
redact,
transcriptPath: transcripts ? flags.string('--transcript-path') ?? undefined : undefined,
});
process.stdout.write(out + '\n');
return 0;
} catch (e) {
console.error(`friction render failed: ${e instanceof Error ? e.message : String(e)}`);
return 1;
}
}
// ---------------------------------------------------------------------------
// list
// ---------------------------------------------------------------------------
function cmdList(args: string[]): number {
const flags = parseFlags(args);
const json = flags.bool('--json');
const runs = listRuns();
if (json) {
console.log(JSON.stringify(runs, null, 2));
return 0;
}
if (runs.length === 0) {
console.log('no runs yet');
return 0;
}
for (const r of runs) {
const interrupted = r.counts.interrupted ? ' (interrupted)' : '';
const sev = Object.entries(r.counts.bySeverity).map(([k, v]) => `${k}=${v}`).join(' ');
console.log(`${r.runId}${interrupted} friction=${r.counts.friction} delight=${r.counts.delight} ${sev}`);
}
return 0;
}
// ---------------------------------------------------------------------------
// summary
// ---------------------------------------------------------------------------
function cmdSummary(args: string[]): number {
const flags = parseFlags(args);
const runId = flags.string('--run-id') ?? activeRunId();
const json = flags.bool('--json');
try {
const out = renderSummary(runId, { format: json ? 'json' : 'md' });
process.stdout.write(out + '\n');
return 0;
} catch (e) {
console.error(`friction summary failed: ${e instanceof Error ? e.message : String(e)}`);
return 1;
}
}
// ---------------------------------------------------------------------------
// Helpers
// ---------------------------------------------------------------------------
function parseFlags(args: string[]) {
return {
string(flag: string): string | undefined {
const idx = args.indexOf(flag);
return idx === -1 ? undefined : args[idx + 1];
},
bool(flag: string): boolean {
return args.includes(flag);
},
};
}
function printHelp() {
console.log(`gbrain friction — friction reporter
Subcommands:
log Append a friction or delight entry to the active run
render Render a run's entries as markdown (default) or JSON
list List recent runs with friction/delight counts
summary Two-column summary of friction + delight for a run
Examples:
gbrain friction log --severity confused --phase install --message "init didn't say which engine"
gbrain friction render --run-id claw-test-20260428-... --transcripts
gbrain friction list --json
gbrain friction summary
Run-id resolution: --run-id > $GBRAIN_FRICTION_RUN_ID > 'standalone'.`);
}
+193 -1
View File
@@ -49,6 +49,10 @@ export async function runFrontmatter(args: string[]): Promise<void> {
}
return;
}
if (sub === 'generate') {
await runGenerate(rest);
return;
}
if (sub === 'install-hook') {
const { runFrontmatterInstallHook } = await import('./frontmatter-install-hook.ts');
await runFrontmatterInstallHook(rest);
@@ -71,10 +75,11 @@ async function connectEngineForAudit(): Promise<BrainEngine> {
}
function printHelp() {
console.log(`gbrain frontmatter — frontmatter validation, audit, and auto-repair
console.log(`gbrain frontmatter — frontmatter validation, audit, auto-repair, and generation
Usage:
gbrain frontmatter validate <path> [--json] [--fix] [--dry-run]
gbrain frontmatter generate <path> [--fix] [--dry-run] [--json]
gbrain frontmatter audit [--source <id>] [--json]
gbrain frontmatter install-hook [--source <id>] [--force] [--uninstall]
@@ -91,6 +96,26 @@ validate
--dry-run Preview --fix without writing.
--json Emit a JSON envelope on stdout.
generate
Synthesize frontmatter for files that have none (MISSING_OPEN). Uses
directory-aware rules to infer type, title, date, source, and tags from
the filesystem path and file content. Zero LLM calls, fully deterministic.
Without --fix: dry-run preview showing what would be generated.
With --fix: writes frontmatter to files (with .bak safety backups).
Rules are defined in src/core/frontmatter-inference.ts DIRECTORY_RULES.
Add new directory conventions by adding rules to the table.
Examples:
gbrain frontmatter generate /path/to/brain # preview all
gbrain frontmatter generate /path/to/brain --fix # write all
gbrain frontmatter generate /path/to/brain/people/ --fix # just people/
--fix Write generated frontmatter to files (.bak safety backups).
--dry-run Preview without writing (default when --fix is omitted).
--json Emit JSON output.
audit
Read-only scan across all registered sources (or one with --source <id>).
Reports per-source counts grouped by error code. Use this in CI or doctor
@@ -297,3 +322,170 @@ function printAuditHumanReport(report: AuditReport): void {
console.log(`\nFix with: gbrain frontmatter validate <source-path> --fix`);
}
}
// ---------------------------------------------------------------------------
// generate — synthesize frontmatter for files that have none
// ---------------------------------------------------------------------------
async function runGenerate(args: string[]): Promise<void> {
const targetPath = args.find(a => !a.startsWith('-'));
const doFix = args.includes('--fix');
const dryRun = args.includes('--dry-run');
const jsonOut = args.includes('--json');
if (!targetPath) {
console.error('error: gbrain frontmatter generate requires a <path> argument');
console.error('usage: gbrain frontmatter generate <path> [--fix] [--dry-run] [--json]');
process.exitCode = 1;
return;
}
const { inferFrontmatter, serializeFrontmatter } = await import('../core/frontmatter-inference.ts');
const { resolve, relative, join, basename } = await import('path');
const { readFileSync, writeFileSync, copyFileSync, statSync, readdirSync, lstatSync } = await import('fs');
const rootPath = resolve(targetPath);
const isDir = statSync(rootPath).isDirectory();
// Find the brain root — walk up from targetPath looking for .git or known brain markers.
// Inference rules match against brain-root-relative paths (e.g., "people/alice.md").
let brainRoot = rootPath;
if (isDir) {
let candidate = rootPath;
for (let i = 0; i < 10; i++) {
try {
statSync(join(candidate, '.git'));
brainRoot = candidate;
break;
} catch {
const parent = resolve(candidate, '..');
if (parent === candidate) break;
candidate = parent;
}
}
}
interface GenerateResult {
path: string;
type: string;
title: string;
date?: string;
rule: string;
}
const results: GenerateResult[] = [];
let scanned = 0;
let skipped = 0;
let generated = 0;
let written = 0;
function processFile(absPath: string, relPath: string) {
scanned++;
if (!absPath.endsWith('.md')) return;
// Skip symlinks
try { if (lstatSync(absPath).isSymbolicLink()) return; } catch { return; }
let content: string;
try { content = readFileSync(absPath, 'utf-8'); } catch { return; }
const inferred = inferFrontmatter(relPath, content);
if (inferred.skipped) {
skipped++;
return;
}
generated++;
results.push({
path: relPath,
type: inferred.type,
title: inferred.title,
date: inferred.date,
rule: inferred.matchedRule || '(default)',
});
if (doFix && !dryRun) {
const fm = serializeFrontmatter(inferred);
const newContent = fm + '\n' + content;
// Safety: write .bak first
copyFileSync(absPath, absPath + '.bak');
writeFileSync(absPath, newContent, 'utf-8');
written++;
}
}
function walkDir(dir: string, rootForRel: string) {
let entries: string[];
try { entries = readdirSync(dir); } catch { return; }
for (const entry of entries) {
if (entry === '.git' || entry === 'node_modules' || entry === '.obsidian') continue;
const abs = join(dir, entry);
try {
const stat = statSync(abs);
if (stat.isDirectory()) {
walkDir(abs, rootForRel);
} else if (stat.isFile() && entry.endsWith('.md')) {
processFile(abs, relative(rootForRel, abs));
}
} catch { /* skip unreadable */ }
}
}
if (isDir) {
walkDir(rootPath, brainRoot);
} else {
const relPath = relative(brainRoot, rootPath) || basename(rootPath);
processFile(rootPath, relPath);
}
// Output
if (jsonOut) {
console.log(JSON.stringify({
scanned,
skipped,
generated,
written,
dryRun: !doFix || dryRun,
results: results.slice(0, 100), // Cap JSON output
totalResults: results.length,
}, null, 2));
return;
}
// Human-readable output
const mode = doFix && !dryRun ? 'WRITE' : 'DRY-RUN';
console.log(`\nFrontmatter generation (${mode})`);
console.log(` Scanned: ${scanned} files`);
console.log(` Already have frontmatter: ${skipped}`);
console.log(` Would generate: ${generated}`);
if (doFix && !dryRun) {
console.log(` Written: ${written} (with .bak backups)`);
}
// Show sample by type
const byType: Record<string, number> = {};
for (const r of results) {
byType[r.type] = (byType[r.type] || 0) + 1;
}
if (Object.keys(byType).length > 0) {
console.log(`\n By type:`);
for (const [type, count] of Object.entries(byType).sort(([, a], [, b]) => b - a)) {
console.log(` ${type}: ${count}`);
}
}
// Show first 10 examples
if (results.length > 0 && (!doFix || dryRun)) {
console.log(`\n Examples:`);
for (const r of results.slice(0, 10)) {
console.log(` ${r.path}`);
console.log(` → type: ${r.type}, title: "${r.title}"${r.date ? `, date: ${r.date}` : ''} [rule: ${r.rule}]`);
}
if (results.length > 10) {
console.log(` ... and ${results.length - 10} more`);
}
if (!doFix) {
console.log(`\n To write: gbrain frontmatter generate ${targetPath} --fix`);
}
}
}
+59 -32
View File
@@ -1,10 +1,10 @@
import { readdirSync, lstatSync, existsSync, writeFileSync, readFileSync, unlinkSync } from 'fs';
import { execFileSync } from 'child_process';
import { join, relative } from 'path';
import { cpus, totalmem, homedir } from 'os';
import { cpus, totalmem } from 'os';
import type { BrainEngine } from '../core/engine.ts';
import { importFile } from '../core/import-file.ts';
import { loadConfig } from '../core/config.ts';
import { loadConfig, gbrainPath } from '../core/config.ts';
import { createProgress } from '../core/progress.ts';
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
@@ -34,7 +34,17 @@ export async function runImport(engine: BrainEngine, args: string[], opts: { com
const jsonOutput = args.includes('--json');
const workersIdx = args.indexOf('--workers');
const workersArg = workersIdx !== -1 ? args[workersIdx + 1] : null;
const workerCount = workersArg ? parseInt(workersArg, 10) : 1;
// v0.22.13 (PR #490 Q2): shared parseWorkers helper rejects bad input
// (--workers 0, -3, "foo") with a loud error instead of silently falling
// through to 1. Mirrors sync.ts's flag handling.
const { parseWorkers } = await import('../core/sync-concurrency.ts');
let workerCount: number;
try {
workerCount = parseWorkers(workersArg ?? undefined) ?? 1;
} catch (e) {
console.error(e instanceof Error ? e.message : String(e));
process.exit(1);
}
// Find dir: first non-flag arg that isn't a value for --workers
const flagValues = new Set<number>();
if (workersIdx !== -1) flagValues.add(workersIdx + 1);
@@ -51,7 +61,7 @@ export async function runImport(engine: BrainEngine, args: string[], opts: { com
console.log(`Found ${allFiles.length} markdown files`);
// Resume from checkpoint if available
const checkpointPath = join(homedir(), '.gbrain', 'import-checkpoint.json');
const checkpointPath = gbrainPath('import-checkpoint.json');
let files = allFiles;
let resumeIndex = 0;
@@ -127,7 +137,7 @@ export async function runImport(engine: BrainEngine, args: string[], opts: { com
// Save checkpoint every 100 files — track completed file set, not just a counter
if (processed % 100 === 0) {
try {
const cpDir = join(homedir(), '.gbrain');
const cpDir = gbrainPath();
if (!existsSync(cpDir)) { const { mkdirSync } = await import('fs'); mkdirSync(cpDir, { recursive: true }); }
writeFileSync(checkpointPath, JSON.stringify({
dir, totalFiles: allFiles.length,
@@ -141,40 +151,57 @@ export async function runImport(engine: BrainEngine, args: string[], opts: { com
}
if (actualWorkers > 1) {
// Parallel: create per-worker engine instances with small pool
// PGLite is single-connection, so parallel workers are only for Postgres
// v0.22.13 (PR #490 A1 + Q3): use engine.kind discriminator (not config.engine
// string sniff) and fall back to serial when database_url is unset. Both
// checks belt-and-suspenders so we never crash on a null assertion.
const config = loadConfig();
if (config?.engine === 'pglite') {
// PGLite: sequential import through single engine
if (engine.kind === 'pglite' || !config?.database_url) {
for (const file of files) {
await processFile(engine, file);
}
} else {
const { PostgresEngine } = await import('../core/postgres-engine.ts');
const { resolvePoolSize } = await import('../core/db.ts');
// Default per-worker pool is 2 (small, parallel import case). Users on
// constrained poolers (e.g. Supabase port 6543) can cap below this via
// GBRAIN_POOL_SIZE=1.
const workerPoolSize = Math.min(2, resolvePoolSize(2));
const workerEngines = await Promise.all(
Array.from({ length: actualWorkers }, async () => {
const eng = new PostgresEngine();
await eng.connect({ database_url: config!.database_url!, poolSize: workerPoolSize });
return eng;
})
);
const { PostgresEngine } = await import('../core/postgres-engine.ts');
const { resolvePoolSize } = await import('../core/db.ts');
// Default per-worker pool is 2 (small, parallel import case). Users on
// constrained poolers (e.g. Supabase port 6543) can cap below this via
// GBRAIN_POOL_SIZE=1.
const workerPoolSize = Math.min(2, resolvePoolSize(2));
const databaseUrl = config.database_url;
// Thread-safe queue: use an atomic index counter instead of array.shift()
let queueIndex = 0;
await Promise.all(workerEngines.map(async (eng) => {
while (true) {
const idx = queueIndex++;
if (idx >= files.length) break;
await processFile(eng, files[idx]);
// v0.22.13 (PR #490 A2): connect workers serially so a partial failure
// leaves us with the connected ones already pushed onto workerEngines
// for the finally-block cleanup. The prior Promise.all could leak any
// engine that connected before another's connect() rejected.
const workerEngines: InstanceType<typeof PostgresEngine>[] = [];
try {
for (let i = 0; i < actualWorkers; i++) {
const eng = new PostgresEngine();
await eng.connect({ database_url: databaseUrl, poolSize: workerPoolSize });
workerEngines.push(eng);
}
// Thread-safe queue: atomic index counter (JS is single-threaded; the
// read-then-increment happens between awaits so no lock is needed).
let queueIndex = 0;
await Promise.all(workerEngines.map(async (eng) => {
while (true) {
const idx = queueIndex++;
if (idx >= files.length) break;
await processFile(eng, files[idx]);
}
}));
} finally {
// v0.22.13 (PR #490 A2): try/finally guarantees cleanup even when the
// worker loop throws. Each disconnect is best-effort — one failing
// disconnect must not strand the others.
await Promise.all(
workerEngines.map(e =>
e.disconnect().catch((err: unknown) =>
console.error(` worker disconnect failed: ${err instanceof Error ? err.message : String(err)}`),
),
),
);
}
}));
await Promise.all(workerEngines.map(e => e.disconnect()));
} // end else (postgres parallel)
} else {
// Sequential: use the provided engine
+2 -2
View File
@@ -6,7 +6,7 @@ import { homedir } from 'os';
const __filename = fileURLToPath(import.meta.url);
const __dirname = dirname(__filename);
import { saveConfig, loadConfig, toEngineConfig, type GBrainConfig } from '../core/config.ts';
import { saveConfig, loadConfig, toEngineConfig, gbrainPath, type GBrainConfig } from '../core/config.ts';
import { createEngine } from '../core/engine-factory.ts';
export async function runInit(args: string[]) {
@@ -103,7 +103,7 @@ async function initMigrateOnly(opts: { jsonOutput: boolean }) {
}
async function initPGLite(opts: { jsonOutput: boolean; apiKey: string | null; customPath: string | null }) {
const dbPath = opts.customPath || join(homedir(), '.gbrain', 'brain.pglite');
const dbPath = opts.customPath || gbrainPath('brain.pglite');
console.log(`Setting up local brain with PGLite (no server needed)...`);
const engine = await createEngine({ engine: 'pglite' });
+2 -1
View File
@@ -23,6 +23,7 @@ import matter from 'gray-matter';
import { readFileSync, existsSync, writeFileSync, mkdirSync, readdirSync } from 'fs';
import { join, basename } from 'path';
import { homedir } from 'os';
import { gbrainPath } from '../core/config.ts';
import { execSync } from 'child_process';
// --- Types ---
@@ -512,7 +513,7 @@ function findRecipe(id: string): ParsedRecipe | null {
// --- Heartbeat ---
function heartbeatDir(id: string): string {
return join(homedir(), '.gbrain', 'integrations', id);
return gbrainPath('integrations', id);
}
function heartbeatPath(id: string): string {
+100 -26
View File
@@ -25,12 +25,12 @@
*/
import { appendFileSync, existsSync, readFileSync, mkdirSync, writeFileSync } from 'fs';
import { homedir } from 'os';
import { join, dirname } from 'path';
import { dirname } from 'path';
import { loadConfig, toEngineConfig } from '../core/config.ts';
import { loadConfig, toEngineConfig, gbrainPath } from '../core/config.ts';
import { createEngine } from '../core/engine-factory.ts';
import type { BrainEngine } from '../core/engine.ts';
import * as db from '../core/db.ts';
import { BrainWriter } from '../core/output/writer.ts';
import {
getDefaultRegistry,
@@ -44,10 +44,10 @@ import { tweetCitation } from '../core/output/scaffold.ts';
// Paths
// ---------------------------------------------------------------------------
const GBRAIN_DIR = join(homedir(), '.gbrain');
const REVIEW_FILE = join(GBRAIN_DIR, 'integrity-review.md');
const LOG_FILE = join(GBRAIN_DIR, 'integrity.log.jsonl');
const PROGRESS_FILE = join(GBRAIN_DIR, 'integrity-progress.jsonl');
// Lazy: GBRAIN_HOME may be set after module load.
const getReviewFile = () => gbrainPath('integrity-review.md');
const getLogFile = () => gbrainPath('integrity.log.jsonl');
const getProgressFile = () => gbrainPath('integrity-progress.jsonl');
// ---------------------------------------------------------------------------
// Bare-tweet detection
@@ -157,9 +157,9 @@ interface ProgressEntry {
}
function loadProgress(): Set<string> {
if (!existsSync(PROGRESS_FILE)) return new Set();
if (!existsSync(getProgressFile())) return new Set();
const seen = new Set<string>();
const content = readFileSync(PROGRESS_FILE, 'utf-8');
const content = readFileSync(getProgressFile(), 'utf-8');
for (const line of content.split('\n')) {
if (!line.trim()) continue;
try {
@@ -173,12 +173,12 @@ function loadProgress(): Set<string> {
}
function appendProgress(entry: ProgressEntry): void {
ensureDir(PROGRESS_FILE);
appendFileSync(PROGRESS_FILE, JSON.stringify(entry) + '\n', 'utf-8');
ensureDir(getProgressFile());
appendFileSync(getProgressFile(), JSON.stringify(entry) + '\n', 'utf-8');
}
function clearProgress(): void {
if (existsSync(PROGRESS_FILE)) writeFileSync(PROGRESS_FILE, '', 'utf-8');
if (existsSync(getProgressFile())) writeFileSync(getProgressFile(), '', 'utf-8');
}
function ensureDir(path: string): void {
@@ -212,7 +212,7 @@ export async function runIntegrity(args: string[]): Promise<void> {
}
if (sub === 'reset-progress') {
clearProgress();
console.log('Cleared progress log:', PROGRESS_FILE);
console.log('Cleared progress log:', getProgressFile());
return;
}
@@ -266,6 +266,12 @@ export interface IntegrityScanOptions {
limit?: number;
/** Slug prefix filter (e.g. "people") — matches slugs starting with `${typeFilter}/`. */
typeFilter?: string;
/**
* When true (default), batch-load pages via a single SQL query instead of
* sequential getPage() calls. Falls back to sequential on error (e.g. PGLite).
* Eliminates 500 round-trips through PgBouncer that caused doctor timeouts.
*/
batchLoad?: boolean;
}
export interface IntegrityScanResult {
@@ -287,7 +293,29 @@ export async function scanIntegrity(
engine: BrainEngine,
opts: IntegrityScanOptions = {},
): Promise<IntegrityScanResult> {
const { limit = Infinity, typeFilter } = opts;
const { limit = Infinity, typeFilter, batchLoad = true } = opts;
// Fast path: single SQL query instead of N sequential getPage() calls.
// Eliminates ~500 round-trips through PgBouncer that caused doctor to
// timeout on transaction-mode pooling. Postgres-only: PGLite has no
// postgres.js connection, so the gate keeps the GBRAIN_DEBUG fallback
// log clean for real Postgres errors instead of expected PGLite skips.
if (batchLoad && limit !== Infinity && engine.kind === 'postgres') {
try {
return await scanIntegrityBatch(limit, typeFilter);
} catch (err) {
// GBRAIN_DEBUG=1 surfaces real Postgres errors (deadlock, connection
// drop, SQL bug) that would otherwise vanish into the sequential
// fallback. Quiet by default since the fallback is harmless.
if (process.env.GBRAIN_DEBUG) {
console.error(
'[integrity] batch path failed, falling back to sequential:',
err instanceof Error ? err.message : err,
);
}
}
}
const allSlugs = [...(await engine.getAllSlugs())].sort();
const bareHits: BareTweetHit[] = [];
@@ -316,6 +344,52 @@ export async function scanIntegrity(
return { pagesScanned, bareHits, externalHits, topPages };
}
/**
* Batch-load integrity scan: fetches all candidate pages in a single SQL
* query, then scans in-memory. Reduces PgBouncer round-trips from ~500 to 1.
*/
async function scanIntegrityBatch(
limit: number,
typeFilter?: string,
): Promise<IntegrityScanResult> {
const sql = db.getConnection();
const typeCondition = typeFilter ? sql`AND slug LIKE ${typeFilter + '/%'}` : sql``;
// Boolean validate is the documented contract; stringly-typed 'false' (quoted
// YAML) diverges from the sequential path's strict === false check. Intentional
// — gbrain lint should reject stringly-typed validate at write time.
const validateCondition = sql`AND (frontmatter->>'validate' IS NULL OR frontmatter->>'validate' != 'false')`;
// DISTINCT ON (slug) mirrors getAllSlugs()'s Set<string> semantics: multi-source
// brains can have the same slug under multiple source_ids (UNIQUE(source_id, slug)
// since v0.18.0); we want one scan per slug, not one per row.
const rows = await sql`
SELECT DISTINCT ON (slug) slug, compiled_truth, frontmatter
FROM pages
WHERE 1=1 ${typeCondition} ${validateCondition}
ORDER BY slug
LIMIT ${limit}
`;
const bareHits: BareTweetHit[] = [];
const externalHits: ExternalLinkHit[] = [];
for (const row of rows) {
const slug = row.slug as string;
const compiledTruth = row.compiled_truth as string;
bareHits.push(...findBareTweetHits(compiledTruth, slug));
externalHits.push(...findExternalLinks(compiledTruth, slug));
}
const byPage = new Map<string, number>();
for (const h of bareHits) byPage.set(h.slug, (byPage.get(h.slug) ?? 0) + 1);
const topPages = [...byPage.entries()]
.sort((a, b) => b[1] - a[1])
.slice(0, 10)
.map(([slug, count]) => ({ slug, count }));
return { pagesScanned: rows.length, bareHits, externalHits, topPages };
}
// ---------------------------------------------------------------------------
// auto — three-bucket repair
// ---------------------------------------------------------------------------
@@ -334,7 +408,7 @@ async function cmdAuto(args: string[]): Promise<void> {
process.exit(1);
}
ensureDir(GBRAIN_DIR);
ensureDir(gbrainPath());
const engine = await connect();
const registry = getDefaultRegistry();
@@ -473,9 +547,9 @@ async function cmdAuto(args: string[]): Promise<void> {
console.log(`Review queue (≥${reviewLower} <${confidenceThreshold}): ${bucketReview}`);
console.log(`Skipped (<${reviewLower}): ${bucketSkip}`);
if (bucketErr > 0) console.log(`Resolver errors: ${bucketErr}`);
console.log(`\nReview queue: ${REVIEW_FILE}`);
console.log(`Skipped log: ${LOG_FILE}`);
console.log(`Progress: ${PROGRESS_FILE}`);
console.log(`\nReview queue: ${getReviewFile()}`);
console.log(`Skipped log: ${getLogFile()}`);
console.log(`Progress: ${getProgressFile()}`);
} finally {
await engine.disconnect();
}
@@ -486,15 +560,15 @@ async function cmdAuto(args: string[]): Promise<void> {
// ---------------------------------------------------------------------------
function cmdReview(): void {
if (!existsSync(REVIEW_FILE)) {
if (!existsSync(getReviewFile())) {
console.log(`No review queue yet. Run: gbrain integrity auto --confidence 0.8`);
return;
}
const content = readFileSync(REVIEW_FILE, 'utf-8');
const content = readFileSync(getReviewFile(), 'utf-8');
const count = (content.match(/^## /gm) ?? []).length;
console.log(`Review queue: ${REVIEW_FILE}`);
console.log(`Review queue: ${getReviewFile()}`);
console.log(`Entries: ${count}`);
console.log(`\nOpen with: $EDITOR ${REVIEW_FILE}`);
console.log(`\nOpen with: $EDITOR ${getReviewFile()}`);
}
// ---------------------------------------------------------------------------
@@ -575,7 +649,7 @@ interface ReviewArgs {
}
function appendReview(args: ReviewArgs): void {
ensureDir(REVIEW_FILE);
ensureDir(getReviewFile());
const { slug, hit, result, handle } = args;
const block = [
`## ${slug}:${hit.line} (confidence ${result.confidence.toFixed(2)})`,
@@ -589,12 +663,12 @@ function appendReview(args: ReviewArgs): void {
'---',
'',
].join('\n');
appendFileSync(REVIEW_FILE, block, 'utf-8');
appendFileSync(getReviewFile(), block, 'utf-8');
}
interface SkipArgs { slug: string; hit: BareTweetHit; reason: string }
function logSkip(args: SkipArgs): void {
ensureDir(LOG_FILE);
ensureDir(getLogFile());
const entry = {
timestamp: new Date().toISOString(),
slug: args.slug,
@@ -603,7 +677,7 @@ function logSkip(args: SkipArgs): void {
raw: args.hit.rawLine.slice(0, 200),
reason: args.reason,
};
appendFileSync(LOG_FILE, JSON.stringify(entry) + '\n', 'utf-8');
appendFileSync(getLogFile(), JSON.stringify(entry) + '\n', 'utf-8');
}
// ---------------------------------------------------------------------------
+138 -18
View File
@@ -33,14 +33,14 @@ export function parseMaxWaitingFlag(args: string[]): number | undefined {
}
/** Parse `--max-rss N` (MB). Returns:
* - 0 if the flag is absent (no watchdog by default for bare `jobs work`)
* - undefined if the flag is absent (caller decides the default)
* - 0 if `--max-rss 0` (explicit disable)
* - the value if >= 256
* Errors and exits the process if the flag is non-numeric, negative, or
* positive but < 256 (likely a GB-vs-MB unit-confusion typo). */
export function parseMaxRssFlag(args: string[]): number {
export function parseMaxRssFlag(args: string[]): number | undefined {
const raw = parseFlag(args, '--max-rss');
if (raw === undefined) return 0;
if (raw === undefined) return undefined;
const parsed = parseInt(raw, 10);
if (!Number.isFinite(parsed) || parsed < 0) {
console.error(`Error: --max-rss must be a non-negative integer (MB), got "${raw}"`);
@@ -133,6 +133,7 @@ USAGE
gbrain jobs stats
gbrain jobs smoke
gbrain jobs work [--queue Q] [--concurrency N] [--max-rss MB]
[--health-interval MS]
gbrain jobs supervisor [start] [--detach] [--json]
[--concurrency N] [--queue Q] [--pid-file PATH]
[--max-crashes N] [--health-interval N]
@@ -314,8 +315,15 @@ HANDLER TYPES (built in)
if (follow) {
console.log(`Job #${job.id} submitted (${name}). Executing inline...`);
// Inline execution: run the job in this process
const worker = new MinionWorker(engine, { queue: queueName, pollInterval: 100 });
// Inline execution: run the job in this process. Disable the
// self-health-check timer — inline flows are one-shot and don't have
// a process manager to restart them. With the timer enabled and no
// 'unhealthy' listener, a DB blip would trip emitUnhealthy's
// no-listener fallback and call process.exit(1) from inside the
// library, killing the user's CLI session.
const worker = new MinionWorker(engine, {
queue: queueName, pollInterval: 100, healthCheckInterval: 0,
});
// Register built-in handlers
await registerBuiltinHandlers(worker, engine);
@@ -489,7 +497,11 @@ HANDLER TYPES (built in)
const sigkillRescue = hasFlag(args, '--sigkill-rescue');
const wedgeRescue = hasFlag(args, '--wedge-rescue');
const worker = new MinionWorker(engine, { queue: 'smoke', pollInterval: 100 });
// Smoke harness is short-lived and has no listener — disable the health
// timer so the no-listener fallback can't trip process.exit(1) mid-test.
const worker = new MinionWorker(engine, {
queue: 'smoke', pollInterval: 100, healthCheckInterval: 0,
});
worker.register('noop', async () => ({ ok: true, at: new Date().toISOString() }));
const job = await queue.add('noop', {}, { queue: 'smoke', max_attempts: 1 });
@@ -638,19 +650,69 @@ HANDLER TYPES (built in)
const queueName = parseFlag(args, '--queue') ?? 'default';
const concurrency = resolveWorkerConcurrency(args);
// --max-rss is opt-in for bare `gbrain jobs work` — preserves pre-v0.21 behavior
// for operators with legitimately large embed/import working sets. The supervisor
// path injects a default 2048; this code path does not.
const maxRssMb = parseMaxRssFlag(args);
// --max-rss defaults to 2048 for bare workers (matching supervisor default).
// This catches memory-leak stalls that previously went undetected without
// a supervisor. Operators can opt out with `--max-rss 0`.
const maxRssExplicit = parseMaxRssFlag(args);
const maxRssMb = maxRssExplicit ?? 2048;
// --health-interval: self-health-check period in ms. 0 disables. Default: 60_000 (60s).
// Provides DB liveness probes + stall detection for bare workers.
// Automatically skipped when running under a supervisor (GBRAIN_SUPERVISED=1).
// Validated aggressively (parity with --max-rss): reject NaN/negative/non-integer
// values, and reject suspicious sub-1000ms values that are likely a unit-confusion
// typo (e.g. "--health-interval 60" thinking the unit is seconds).
const healthRaw = parseFlag(args, '--health-interval');
let healthCheckInterval = 60_000;
if (healthRaw !== undefined) {
const parsed = parseInt(healthRaw, 10);
if (!Number.isFinite(parsed) || parsed < 0) {
console.error(`Error: --health-interval must be a non-negative integer (ms), got "${healthRaw}"`);
process.exit(1);
}
if (parsed > 0 && parsed < 1000) {
console.error(
`Error: --health-interval ${parsed} is suspiciously low (likely a unit-confusion typo). ` +
`The flag takes milliseconds; for 60-second probes pass 60000. Use 0 to disable.`,
);
process.exit(1);
}
healthCheckInterval = parsed;
}
try { await queue.ensureSchema(); }
catch (e) { console.error(e instanceof Error ? e.message : String(e)); process.exit(1); }
const worker = new MinionWorker(engine, { queue: queueName, concurrency, maxRssMb });
const worker = new MinionWorker(engine, {
queue: queueName, concurrency, maxRssMb, healthCheckInterval,
});
await registerBuiltinHandlers(worker, engine);
// Subscribe to self-health failures emitted by the worker. Library code
// (worker.ts) never calls process.exit directly so it stays embeddable;
// this CLI layer is the right place to terminate the process and let
// the external PM (systemd, Docker, cron watchdog) restart cleanly.
worker.on('unhealthy', (info) => {
if (info.reason === 'db_dead') {
console.error(
`[health] FATAL: DB unreachable after ${info.consecutiveFailures} probes (${info.message}). ` +
`Exiting for process-manager restart.`,
);
} else {
console.error(
`[health] FATAL: Worker stalled — ${info.waitingCount} waiting job(s) for ` +
`registered handlers, ${info.idleMinutes}m idle. Exiting for process-manager restart.`,
);
}
process.exit(1);
});
const isSupervisedChild = process.env.GBRAIN_SUPERVISED === '1';
const watchdogNote = maxRssMb > 0 ? `, watchdog: ${maxRssMb}MB` : '';
console.log(`Minion worker started (queue: ${queueName}, concurrency: ${concurrency}${watchdogNote})`);
const healthNote = !isSupervisedChild && healthCheckInterval > 0
? `, health-check: ${Math.round(healthCheckInterval / 1000)}s`
: '';
console.log(`Minion worker started (queue: ${queueName}, concurrency: ${concurrency}${watchdogNote}${healthNote})`);
console.log(`Registered handlers: ${worker.registeredNames.join(', ')}`);
await worker.start();
break;
@@ -787,15 +849,32 @@ HANDLER TYPES (built in)
const concurrency = parseInt(parseFlag(args, '--concurrency') ?? '2', 10);
const queueName = parseFlag(args, '--queue') ?? 'default';
const maxCrashes = parseInt(parseFlag(args, '--max-crashes') ?? '10', 10);
const healthInterval = parseInt(parseFlag(args, '--health-interval') ?? '60000', 10);
// --health-interval (supervisor): validate same as `jobs work` so NaN /
// negative / sub-1000ms typos fail-fast instead of silently disabling
// the supervisor's own health probe.
const supHealthRaw = parseFlag(args, '--health-interval');
let healthInterval = 60_000;
if (supHealthRaw !== undefined) {
const parsed = parseInt(supHealthRaw, 10);
if (!Number.isFinite(parsed) || parsed < 0) {
console.error(`Error: --health-interval must be a non-negative integer (ms), got "${supHealthRaw}"`);
process.exit(1);
}
if (parsed > 0 && parsed < 1000) {
console.error(
`Error: --health-interval ${parsed} is suspiciously low (likely a unit-confusion typo). ` +
`The flag takes milliseconds; for 60-second probes pass 60000. Use 0 to disable.`,
);
process.exit(1);
}
healthInterval = parsed;
}
const allowShellJobs = hasFlag(args, '--allow-shell-jobs') ||
!!process.env.GBRAIN_ALLOW_SHELL_JOBS;
const detach = hasFlag(args, '--detach');
// Supervisor defaults --max-rss 2048 (MB) — main production path uses
// the supervisor, so the watchdog is on by default here. parseMaxRssFlag
// returns 0 when the flag is absent; substitute the supervisor default.
const maxRssRaw = parseMaxRssFlag(args);
const maxRssMb = parseFlag(args, '--max-rss') === undefined ? 2048 : maxRssRaw;
// the supervisor, so the watchdog is on by default here.
const maxRssMb = parseMaxRssFlag(args) ?? 2048;
const cliPath = parseFlag(args, '--cli-path') ?? resolveGbrainCliPath();
@@ -864,8 +943,40 @@ export async function registerBuiltinHandlers(worker: MinionWorker, engine: Brai
const { performSync } = await import('./sync.ts');
const repoPath = typeof job.data.repoPath === 'string' ? job.data.repoPath : undefined;
const noPull = !!job.data.noPull;
// noEmbed defaults to true (embed is a separate job — submit `embed --stale`
// after sync, OR run via the autopilot cycle which has its own embed phase).
// Caller can opt in by passing { noEmbed: false } in job params.
const noEmbed = job.data.noEmbed !== false;
const result = await performSync(engine, { repoPath, noPull, noEmbed });
// v0.22.13 (PR #490 CODEX-1): resolve sourceId from job param OR by looking
// up the sources row for repoPath. Mirrors cycle.ts:480 — without this, a
// multi-source brain reads the global config.sync.last_commit anchor
// instead of sources.last_commit, which on a regularly-GC'd repo can drop
// out of git history and trigger 30-min full reimports every cycle.
let sourceId: string | undefined =
typeof job.data.sourceId === 'string' ? job.data.sourceId : undefined;
if (!sourceId && repoPath) {
try {
const rows = await engine.executeRaw<{ id: string }>(
`SELECT id FROM sources WHERE local_path = $1 LIMIT 1`,
[repoPath],
);
sourceId = rows[0]?.id;
} catch {
// sources table may not exist on very old brains — fall through to
// global config.sync.* anchor in performSync.
}
}
// v0.22.13 (PR #490 CODEX-4): route concurrency through the shared
// autoConcurrency helper instead of hardcoded 4. PGLite engines stay
// serial (forced 1); explicit job param wins; auto path defaults are
// applied inside performSync against the resolved file count.
const concurrencyOverride = typeof job.data.concurrency === 'number'
? job.data.concurrency
: undefined;
const result = await performSync(engine, {
repoPath, sourceId, noPull, noEmbed,
concurrency: concurrencyOverride,
});
return result;
});
@@ -948,10 +1059,19 @@ export async function registerBuiltinHandlers(worker: MinionWorker, engine: Brai
? job.data.repoPath
: (await engine.getConfig('sync.repo_path')) ?? '.';
// Allow callers to select phases via job data (e.g. skip embed for
// fast cycles). Validates against ALL_PHASES to prevent injection.
const { ALL_PHASES } = await import('../core/cycle.ts');
const validPhases = new Set(ALL_PHASES);
const requestedPhases = Array.isArray(job.data.phases)
? (job.data.phases as string[]).filter(p => validPhases.has(p as any))
: undefined;
const report = await runCycle(engine, {
brainDir: repoPath,
pull: true, // autopilot daemon opts into git pull
signal: job.signal, // propagate abort so cycle bails on timeout/cancel
...(requestedPhases && requestedPhases.length > 0 ? { phases: requestedPhases as any } : {}),
yieldBetweenPhases: async () => {
// Yield to the event loop so worker lock-renewal can fire.
await new Promise<void>(r => setImmediate(r));
+3 -5
View File
@@ -8,11 +8,9 @@
*/
import { createEngine } from '../core/engine-factory.ts';
import { loadConfig, saveConfig, toEngineConfig, type GBrainConfig } from '../core/config.ts';
import { loadConfig, saveConfig, toEngineConfig, gbrainPath, type GBrainConfig } from '../core/config.ts';
import type { BrainEngine } from '../core/engine.ts';
import type { EngineConfig } from '../core/types.ts';
import { homedir } from 'os';
import { join } from 'path';
import { writeFileSync, readFileSync, existsSync, unlinkSync } from 'fs';
import { createProgress } from '../core/progress.ts';
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
@@ -48,7 +46,7 @@ function parseArgs(args: string[]): MigrateOpts {
}
function getManifestPath(): string {
return join(homedir(), '.gbrain', 'migrate-manifest.json');
return gbrainPath('migrate-manifest.json');
}
interface MigrateManifest {
@@ -99,7 +97,7 @@ export async function runMigrateEngine(sourceEngine: BrainEngine, args: string[]
process.exit(1);
}
} else {
targetConfig.database_path = opts.targetPath || join(homedir(), '.gbrain', 'brain.pglite');
targetConfig.database_path = opts.targetPath || gbrainPath('brain.pglite');
}
// Connect to target
+7 -6
View File
@@ -35,17 +35,17 @@
*/
import { existsSync, mkdirSync, appendFileSync } from 'fs';
import { homedir } from 'os';
import { join } from 'path';
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
import { loadConfig, toEngineConfig } from '../../core/config.ts';
import { loadConfig, toEngineConfig, gbrainPath } from '../../core/config.ts';
import { createEngine } from '../../core/engine-factory.ts';
import type { BrainEngine } from '../../core/engine.ts';
// Bug 3 — ledger writes moved to the runner (apply-migrations.ts).
const ROLLBACK_DIR = join(homedir(), '.gbrain', 'migrations');
const ROLLBACK_FILE = join(ROLLBACK_DIR, 'v0_13_1-rollback.jsonl');
// Lazy: GBRAIN_HOME may be set after module load.
const getRollbackDir = () => gbrainPath('migrations');
const getRollbackFile = () => join(getRollbackDir(), 'v0_13_1-rollback.jsonl');
const BATCH_SIZE = 100;
// ---------------------------------------------------------------------------
@@ -251,7 +251,8 @@ async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult>
// ---------------------------------------------------------------------------
function ensureRollbackDir(): void {
if (!existsSync(ROLLBACK_DIR)) mkdirSync(ROLLBACK_DIR, { recursive: true });
const dir = getRollbackDir();
if (!existsSync(dir)) mkdirSync(dir, { recursive: true });
}
function appendRollbackEntry(entry: { slug: string; pre_frontmatter: Record<string, unknown> }): void {
@@ -260,7 +261,7 @@ function appendRollbackEntry(entry: { slug: string; pre_frontmatter: Record<stri
timestamp: new Date().toISOString(),
...entry,
}) + '\n';
appendFileSync(ROLLBACK_FILE, line, 'utf-8');
appendFileSync(getRollbackFile(), line, 'utf-8');
}
// ---------------------------------------------------------------------------
+5 -7
View File
@@ -22,19 +22,17 @@
*/
import { existsSync, readFileSync, mkdirSync, appendFileSync } from 'fs';
import { homedir } from 'os';
import { join } from 'path';
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
import { loadConfig, toEngineConfig } from '../../core/config.ts';
import { loadConfig, toEngineConfig, gbrainPath } from '../../core/config.ts';
import { createEngine } from '../../core/engine-factory.ts';
import type { BrainEngine } from '../../core/engine.ts';
// Resolve HOME at CALL time, not module-load time — Bun caches os.homedir()
// and ignores later HOME mutations, which breaks test isolation and scripted
// installs. Match the preferences.ts pattern.
function resolveHome(): string { return process.env.HOME || homedir(); }
function pendingHostWorkDir(): string { return join(resolveHome(), '.gbrain', 'migrations'); }
// gbrainPath() honors GBRAIN_HOME at call time (not module-load) and routes
// through the centralized config dir, so the prior resolveHome()/HOME-env
// trick is no longer needed.
function pendingHostWorkDir(): string { return gbrainPath('migrations'); }
function pendingHostWorkPath(): string { return join(pendingHostWorkDir(), 'pending-host-work.jsonl'); }
// ---------------------------------------------------------------------------
+15 -3
View File
@@ -1,7 +1,19 @@
import type { BrainEngine } from '../core/engine.ts';
import { startMcpServer } from '../mcp/server.ts';
import { startHttpTransport } from '../mcp/http-transport.ts';
export async function runServe(engine: BrainEngine) {
console.error('Starting GBrain MCP server (stdio)...');
await startMcpServer(engine);
export async function runServe(engine: BrainEngine, args: string[] = []) {
const useHttp = args.includes('--http');
const portIdx = args.indexOf('--port');
const port = portIdx >= 0 ? parseInt(args[portIdx + 1]) || 8787 : 8787;
if (useHttp) {
console.error(`Starting GBrain MCP server (HTTP on port ${port})...`);
await startHttpTransport({ port, engine });
// Keep alive
await new Promise(() => {});
} else {
console.error('Starting GBrain MCP server (stdio)...');
await startMcpServer(engine);
}
}
+245
View File
@@ -0,0 +1,245 @@
import { join } from 'path';
import type { BrainEngine } from '../core/engine.ts';
import { loadStorageConfig, validateStorageConfig, getStorageTier } from '../core/storage-config.ts';
import type { StorageConfig, StorageTier } from '../core/storage-config.ts';
import { walkBrainRepo, type DiskFileEntry } from '../core/disk-walk.ts';
import { getDefaultSourcePath } from '../core/source-resolver.ts';
/**
* Distinct nominal types for the two tier-keyed numeric maps. Both shapes
* are `Record<StorageTier, number>` structurally but they carry
* semantically different units (page COUNT vs disk BYTES). Distinct types
* make accidental swaps a compile-time error rather than a silent display
* bug. Issue #11 of the eng review.
*/
export type PageCountsByTier = Record<StorageTier, number> & { __brand?: 'page-counts' };
export type DiskUsageByTier = Record<StorageTier, number> & { __brand?: 'disk-bytes' };
/**
* Pure-data result of a storage-status query. No side effects, no I/O
* beyond the engine call and one filesystem walk. Consumed by both the
* JSON formatter and the human formatter; kept narrow so it's a stable
* MCP/scripting contract (D14: storage_status is read-only MCP-exposed).
*/
export interface StorageStatusResult {
config: StorageConfig | null;
repoPath: string | null;
totalPages: number;
pagesByTier: PageCountsByTier;
missingFiles: Array<{ slug: string; expectedPath: string }>;
diskUsageByTier: DiskUsageByTier;
warnings: string[];
}
// ── Dispatcher ────────────────────────────────────────────
export async function runStorage(engine: BrainEngine, args: string[]): Promise<void> {
const subcommand = args[0];
if (!subcommand || subcommand === 'status') {
await runStorageStatus(engine, args.slice(1));
return;
}
console.error(`Unknown storage subcommand: ${subcommand}`);
console.error('Available subcommands: status');
process.exit(1);
}
async function runStorageStatus(engine: BrainEngine, args: string[]): Promise<void> {
warnIfPGLite(engine);
// Resolution chain (D5, Issue #3): explicit --repo → typed accessor → null.
// No cwd fallback. The original silent footgun is dead.
let repoPath: string | null = null;
const repoIdx = args.indexOf('--repo');
if (repoIdx !== -1 && args[repoIdx + 1]) {
repoPath = args[repoIdx + 1];
} else {
repoPath = await getDefaultSourcePath(engine);
}
const result = await getStorageStatus(engine, repoPath);
if (args.includes('--json')) {
console.log(formatStorageStatusJson(result));
return;
}
console.log(formatStorageStatusHuman(result));
}
/**
* D4: storage tiering on PGLite is a partial feature. The "DB" the pages
* live in IS the local file gbrain uses for everything else, so "db_only"
* has no real offload effect. The .gitignore management still helps
* (keeps bulk content out of git history), so we warn but proceed.
*
* Once-per-process via a module-local flag sub-commands invoked from a
* single CLI run share the same warning.
*/
let _pgliteWarned = false;
function warnIfPGLite(engine: BrainEngine): void {
if (_pgliteWarned) return;
if (engine.kind !== 'pglite') return;
_pgliteWarned = true;
console.warn(
`Note: storage tiering has limited effect on PGLite — pages live in your ` +
`local database file regardless of tier. The .gitignore management still ` +
`keeps bulk content out of git history. To get full tiering, migrate to ` +
`Postgres with \`gbrain migrate --to supabase\`.`,
);
}
/** Reset for tests. */
export function __resetPGLiteWarn(): void {
_pgliteWarned = false;
}
// ── Pure data ─────────────────────────────────────────────
/**
* Compute the storage status against the given engine + brain repo path.
*
* Side-effect-free apart from the engine.listPages call and one recursive
* filesystem walk. Pure for testability formatters are tested separately.
*
* Returns null `config` when no gbrain.yml is present at repoPath. In that
* case pagesByTier is all zeros for db_tracked/db_only and totals roll up
* into unspecified.
*/
export async function getStorageStatus(
engine: BrainEngine,
repoPath: string | null,
): Promise<StorageStatusResult> {
const config = repoPath ? loadStorageConfig(repoPath) : null;
const warnings = config ? validateStorageConfig(config) : [];
const pagesByTier: PageCountsByTier = { db_tracked: 0, db_only: 0, unspecified: 0 };
const diskUsageByTier: DiskUsageByTier = { db_tracked: 0, db_only: 0, unspecified: 0 };
const missingFiles: Array<{ slug: string; expectedPath: string }> = [];
// Single recursive walk of the brain repo (Issue #14). Replaces per-page
// existsSync+statSync — was ~400K syscalls on 200K-page brains, now ~one
// per directory + one stat per .md file, plus O(1) lookups below.
const fileMap: Map<string, DiskFileEntry> = repoPath ? walkBrainRepo(repoPath) : new Map();
const pages = await engine.listPages({ limit: 1_000_000 });
for (const page of pages) {
const tier = config ? getStorageTier(page.slug, config) : 'unspecified';
pagesByTier[tier]++;
if (!repoPath) continue;
const entry = fileMap.get(page.slug);
if (entry) {
diskUsageByTier[tier] += entry.size;
} else if (config && tier === 'db_only') {
missingFiles.push({ slug: page.slug, expectedPath: join(repoPath, page.slug + '.md') });
}
}
return {
config,
repoPath,
totalPages: pages.length,
pagesByTier,
missingFiles,
diskUsageByTier,
warnings,
};
}
// ── JSON formatter ────────────────────────────────────────
/**
* Serialize StorageStatusResult to a stable JSON contract. Indented for
* human readability; agents/orchestrators can parse with a standard
* JSON.parse. Schema is the StorageStatusResult interface above.
*/
export function formatStorageStatusJson(result: StorageStatusResult): string {
return JSON.stringify(result, null, 2);
}
// ── Human formatter ───────────────────────────────────────
/**
* Render StorageStatusResult to ASCII text suitable for terminal output.
* D10 lock: ASCII separators only universally portable. No unicode
* box-drawing.
*/
export function formatStorageStatusHuman(result: StorageStatusResult): string {
const lines: string[] = [];
lines.push('Storage Status');
lines.push('==============');
lines.push('');
if (!result.config) {
lines.push('No gbrain.yml configuration found.');
if (result.repoPath) lines.push(`Checked: ${result.repoPath}/gbrain.yml`);
lines.push('');
lines.push('All pages are stored in git by default.');
lines.push(`Total pages: ${result.totalPages}`);
return lines.join('\n');
}
lines.push(`Repository: ${result.repoPath}`);
lines.push(`Total pages: ${result.totalPages}`);
lines.push('');
lines.push('Storage Tiers:');
lines.push('-------------');
lines.push(`DB tracked: ${result.pagesByTier.db_tracked.toLocaleString()} pages`);
lines.push(`DB only: ${result.pagesByTier.db_only.toLocaleString()} pages`);
lines.push(`Unspecified: ${result.pagesByTier.unspecified.toLocaleString()} pages`);
if (result.diskUsageByTier.db_tracked > 0 || result.diskUsageByTier.db_only > 0) {
lines.push('');
lines.push('Disk Usage:');
lines.push('-----------');
if (result.diskUsageByTier.db_tracked > 0) {
lines.push(`DB tracked: ${formatBytes(result.diskUsageByTier.db_tracked)}`);
}
if (result.diskUsageByTier.db_only > 0) {
lines.push(`DB only: ${formatBytes(result.diskUsageByTier.db_only)}`);
}
if (result.diskUsageByTier.unspecified > 0) {
lines.push(`Unspecified: ${formatBytes(result.diskUsageByTier.unspecified)}`);
}
}
if (result.missingFiles.length > 0) {
lines.push('');
lines.push('Missing Files (need restore):');
lines.push('-----------------------------');
for (const missing of result.missingFiles.slice(0, 10)) {
lines.push(` ${missing.slug}`);
}
if (result.missingFiles.length > 10) {
lines.push(` ... and ${result.missingFiles.length - 10} more`);
}
lines.push('');
lines.push(`Use: gbrain export --restore-only --repo "${result.repoPath}"`);
}
if (result.warnings.length > 0) {
lines.push('');
lines.push('Warnings:');
lines.push('---------');
for (const warning of result.warnings) lines.push(` ! ${warning}`);
}
lines.push('');
lines.push('Configuration:');
lines.push('--------------');
lines.push('DB tracked directories:');
for (const dir of result.config.db_tracked) lines.push(` - ${dir}`);
lines.push('');
lines.push('DB-only directories:');
for (const dir of result.config.db_only) lines.push(` - ${dir}`);
return lines.join('\n');
}
function formatBytes(bytes: number): string {
if (bytes === 0) return '0 B';
const k = 1024;
const sizes = ['B', 'KB', 'MB', 'GB', 'TB'];
const i = Math.floor(Math.log(bytes) / Math.log(k));
return parseFloat((bytes / Math.pow(k, i)).toFixed(1)) + ' ' + sizes[i];
}
+365 -14
View File
@@ -1,9 +1,8 @@
import { existsSync } from 'fs';
import { existsSync, readFileSync, writeFileSync, statSync, readdirSync } from 'fs';
import { execFileSync } from 'child_process';
import { join, relative } from 'path';
import type { BrainEngine } from '../core/engine.ts';
import { importFile } from '../core/import-file.ts';
import { readFileSync, statSync, readdirSync } from 'fs';
import { createInterface } from 'readline';
import {
buildSyncManifest,
@@ -12,6 +11,7 @@ import {
recordSyncFailures,
unacknowledgedSyncFailures,
acknowledgeSyncFailures,
formatCodeBreakdown,
} from '../core/sync.ts';
import { estimateTokens, CHUNKER_VERSION } from '../core/chunkers/code.ts';
import { EMBEDDING_MODEL, estimateEmbeddingCostUsd } from '../core/embedding.ts';
@@ -19,6 +19,15 @@ import { errorFor, serializeError } from '../core/errors.ts';
import type { SyncManifest } from '../core/sync.ts';
import { createProgress } from '../core/progress.ts';
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
import { loadConfig } from '../core/config.ts';
import {
autoConcurrency,
shouldRunParallel,
parseWorkers,
} from '../core/sync-concurrency.ts';
import { tryAcquireDbLock, SYNC_LOCK_ID } from '../core/db-lock.ts';
import { loadStorageConfig } from '../core/storage-config.ts';
import { getDefaultSourcePath } from '../core/source-resolver.ts';
export interface SyncResult {
status: 'up_to_date' | 'synced' | 'first_sync' | 'dry_run' | 'blocked_by_failures';
@@ -157,6 +166,27 @@ export interface SyncOpts {
sourceId?: string;
/** Multi-repo: sync strategy override (markdown, code, auto). */
strategy?: 'markdown' | 'code' | 'auto';
/**
* Number of parallel workers for the import phase. When > 1, each worker
* gets its own small Postgres connection pool and files are dispatched via
* an atomic queue index (same pattern as `import --workers N`).
*
* Deletes and renames remain serial (order-dependent).
* Default: undefined auto-concurrency picks (`src/core/sync-concurrency.ts`).
*
* v0.22.13 (PR #490 Q1): when this is explicitly set, the >50-file floor
* is bypassed explicit user intent beats the auto-path safety net.
*/
concurrency?: number;
/**
* Internal: skip acquiring the gbrain-sync DB lock. Set by the cycle
* handler (cycle.ts) which already holds gbrain-cycle and therefore
* already serializes against other cycle runs. CLI sync, jobs handler,
* and any external caller leave this undefined so they take the lock.
*
* v0.22.13 (PR #490 CODEX-2). Not part of the public CLI surface.
*/
skipLock?: boolean;
}
function git(repoPath: string, ...args: string[]): string {
@@ -250,6 +280,39 @@ async function writeChunkerVersion(
}
export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise<SyncResult> {
// CODEX-2 (v0.22.13): cross-process writer lock for performSync. Two
// concurrent syncs can otherwise read the same last_commit anchor, both
// write last_commit unconditionally, and the last writer wins — including
// regressing the bookmark backwards. cycle.ts already takes gbrain-cycle
// for its broader scope; performSync (called from cycle, jobs handler,
// and CLI) takes gbrain-sync just for the writer window. The two ids
// nest cleanly: cycle holds gbrain-cycle, calls performSync, performSync
// takes gbrain-sync. Other callers serialize on gbrain-sync against
// each other AND against the cycle's sync phase.
//
// skipLock is reserved for callers that already serialize via another
// mechanism (none in v0.22.13; reserved for future).
let lockHandle: { release: () => Promise<void> } | null = null;
if (!opts.skipLock) {
lockHandle = await tryAcquireDbLock(engine, SYNC_LOCK_ID);
if (!lockHandle) {
throw new Error(
`Another sync is in progress (lock ${SYNC_LOCK_ID} held). ` +
`Wait for it to finish, or run 'gbrain doctor' if it has been more than 30 minutes.`,
);
}
}
try {
return await performSyncInner(engine, opts);
} finally {
if (lockHandle) {
try { await lockHandle.release(); } catch { /* best-effort release */ }
}
}
}
async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<SyncResult> {
// Resolve repo path
const repoPath = opts.repoPath || await readSyncAnchor(engine, opts.sourceId, 'repo_path');
if (!repoPath) {
@@ -486,21 +549,41 @@ export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise<
// gate `sync.last_commit` advancement and record recoverable errors.
const failedFiles: Array<{ path: string; error: string; line?: number }> = [];
const addsAndMods = [...filtered.added, ...filtered.modified];
// v0.22.13 (PR #490 Q5): one source of truth for the concurrency decision.
// engine.kind === 'pglite' → forced 1; explicit opts.concurrency wins;
// auto path returns DEFAULT_PARALLEL_WORKERS only when fileCount > 100.
const explicitConcurrency = opts.concurrency !== undefined;
const effectiveConcurrency = autoConcurrency(engine, addsAndMods.length, opts.concurrency);
const runParallel = shouldRunParallel(effectiveConcurrency, addsAndMods.length, explicitConcurrency);
if (addsAndMods.length > 0) {
progress.start('sync.imports', addsAndMods.length);
for (const path of addsAndMods) {
const filePath = join(repoPath, path);
// Core import logic shared by serial and parallel paths.
// repoPath is validated non-null at the top of performSyncInner; narrow for TS.
const syncRepoPath = repoPath!;
async function importOnePath(eng: BrainEngine, path: string): Promise<void> {
const filePath = join(syncRepoPath, path);
if (!existsSync(filePath)) {
// CODEX-3 (v0.22.13): a file the diff said exists at headCommit but
// is gone from disk means the working tree has drifted (someone ran
// `git checkout` / `git reset` mid-sync, or the file was deleted
// post-diff). Record as a failure so last_commit does NOT advance —
// the silent-skip-then-advance pathology was the bug.
failedFiles.push({
path,
error: 'file vanished mid-sync (working tree drifted from headCommit)',
});
progress.tick(1, `skip:${path}`);
continue;
return;
}
try {
const result = await importFile(engine, filePath, path, { noEmbed });
const result = await importFile(eng, filePath, path, { noEmbed });
if (result.status === 'imported') {
chunksCreated += result.chunks;
pagesAffected.push(result.slug);
} else if (result.status === 'skipped' && (result as any).error) {
// importFile returned a non-throw skip with a reason.
failedFiles.push({ path, error: String((result as any).error) });
}
} catch (e: unknown) {
@@ -510,9 +593,98 @@ export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise<
}
progress.tick(1, path);
}
if (runParallel) {
// A1 (v0.22.13): use engine.kind discriminator instead of config?.engine
// string compare or constructor.name sniff. Q3: belt-and-suspenders fall
// back to serial when database_url is unset, so we never crash on a null
// assertion if config is missing.
const config = loadConfig();
if (engine.kind === 'pglite' || !config?.database_url) {
for (const path of addsAndMods) {
await importOnePath(engine, path);
}
} else {
const { PostgresEngine } = await import('../core/postgres-engine.ts');
const { resolvePoolSize } = await import('../core/db.ts');
const workerPoolSize = Math.min(2, resolvePoolSize(2));
const workerCount = Math.min(effectiveConcurrency, addsAndMods.length);
const databaseUrl = config.database_url;
// Q4 (v0.22.13): banner on stderr so stdout stays clean for --json.
console.error(` Parallel sync: ${workerCount} workers for ${addsAndMods.length} files`);
const workerEngines: InstanceType<typeof PostgresEngine>[] = [];
try {
// Connect workers one-by-one rather than Promise.all so a partial
// failure leaves us with the connected ones in workerEngines for
// the finally-block cleanup. The original code lost track of
// already-connected engines on any one failure.
for (let i = 0; i < workerCount; i++) {
const eng = new PostgresEngine();
await eng.connect({ database_url: databaseUrl, poolSize: workerPoolSize });
workerEngines.push(eng);
}
// Atomic queue index — JS is single-threaded; the read-then-increment
// happens between awaits, so no lock is needed.
let queueIndex = 0;
await Promise.all(
workerEngines.map(async (eng) => {
while (true) {
const idx = queueIndex++;
if (idx >= addsAndMods.length) break;
await importOnePath(eng, addsAndMods[idx]);
}
}),
);
} finally {
// A2 (v0.22.13): try/finally guarantees connection cleanup even when
// the worker loop throws (partial connect failure, OOM, mid-import
// signal). Each disconnect is best-effort — one worker failing to
// disconnect must not strand the others.
await Promise.all(
workerEngines.map((e) =>
e.disconnect().catch((err: unknown) =>
console.error(` worker disconnect failed: ${err instanceof Error ? err.message : String(err)}`),
),
),
);
}
}
} else {
// Serial path (small auto diffs or explicit --workers 1).
for (const path of addsAndMods) {
await importOnePath(engine, path);
}
}
progress.finish();
}
// CODEX-3 (v0.22.13): head-drift gate. If git HEAD moved during the import
// window (someone ran `git checkout` or `git pull` in another terminal /
// sibling Conductor workspace), the chunks we just imported reflect a
// different tree than `headCommit` claims. Refuse to advance last_commit
// so the next sync re-walks against the new HEAD. The lock from CODEX-2
// prevents *this* gbrain process from stepping on itself; this gate
// catches drift caused by external `git` commands the lock cannot see.
try {
const currentHead = git(repoPath, 'rev-parse', 'HEAD');
if (currentHead !== headCommit) {
failedFiles.push({
path: '<head>',
error: `git HEAD drifted during sync: captured ${headCommit.slice(0, 8)}, now ${currentHead.slice(0, 8)}`,
});
}
} catch (e) {
// rev-parse failure is itself a drift signal (worktree disappeared).
failedFiles.push({
path: '<head>',
error: `git HEAD verification failed: ${e instanceof Error ? e.message : String(e)}`,
});
}
const elapsed = Date.now() - start;
// Bug 9 — gate the sync bookmark on success. If any per-file parse
@@ -522,9 +694,13 @@ export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise<
// current set, --retry-failed re-parses before running the normal sync.
if (failedFiles.length > 0) {
recordSyncFailures(failedFiles, headCommit);
// Emit structured summary grouped by error code so the operator
// can see *why* files failed, not just how many.
const codeBreakdown = formatCodeBreakdown(failedFiles);
if (!opts.skipFailed) {
console.error(
`\nSync blocked: ${failedFiles.length} file(s) failed to parse. ` +
`\nSync blocked: ${failedFiles.length} file(s) failed to parse:\n` +
`${codeBreakdown}\n\n` +
`Fix the YAML frontmatter in the files above and re-run, or use ` +
`'gbrain sync --skip-failed' to acknowledge and move on.`,
);
@@ -547,8 +723,11 @@ export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise<
}
// --skip-failed: acknowledge the now-recorded set and proceed.
const acked = acknowledgeSyncFailures();
if (acked > 0) {
console.error(` Acknowledged ${acked} failure(s) and advancing past them.`);
if (acked.count > 0) {
console.error(
` Acknowledged ${acked.count} failure(s) and advancing past them:\n` +
`${formatCodeBreakdown(acked.summary)}`,
);
}
}
@@ -644,10 +823,18 @@ async function performFullSync(
};
}
console.log(`Running full import of ${repoPath}...`);
// v0.22.13 (PR #490 A1 + Q5): full sync is always "large" by definition
// (entire working tree). Auto-concurrency fires unconditionally for Postgres;
// PGLite stays serial because its engine is single-connection. Routes the
// policy through autoConcurrency() so it stays consistent with incremental
// sync and the jobs handler.
const FULL_SYNC_LARGE_MARKER = Number.MAX_SAFE_INTEGER;
const fullConcurrency = autoConcurrency(engine, FULL_SYNC_LARGE_MARKER, opts.concurrency);
console.log(`Running full import of ${repoPath}${fullConcurrency > 1 ? ` (${fullConcurrency} workers)` : ''}...`);
const { runImport } = await import('./import.ts');
const importArgs = [repoPath];
if (opts.noEmbed) importArgs.push('--no-embed');
if (fullConcurrency > 1) importArgs.push('--workers', String(fullConcurrency));
const result = await runImport(engine, importArgs, { commit: headCommit });
// Bug 9 — gate the full-sync bookmark on success. runImport already
@@ -656,9 +843,11 @@ async function performFullSync(
// the sync module owns the last_commit write. Respect the same gate.
if (result.failures.length > 0) {
recordSyncFailures(result.failures, headCommit);
const codeBreakdown = formatCodeBreakdown(result.failures);
if (!opts.skipFailed) {
console.error(
`\nFull sync blocked: ${result.failures.length} file(s) failed. ` +
`\nFull sync blocked: ${result.failures.length} file(s) failed:\n` +
`${codeBreakdown}\n\n` +
`Fix the YAML in those files and re-run, or use '--skip-failed'.`,
);
await engine.setConfig('sync.last_run', new Date().toISOString());
@@ -675,7 +864,12 @@ async function performFullSync(
};
}
const acked = acknowledgeSyncFailures();
if (acked > 0) console.error(` Acknowledged ${acked} failure(s) and advancing past them.`);
if (acked.count > 0) {
console.error(
` Acknowledged ${acked.count} failure(s) and advancing past them:\n` +
`${formatCodeBreakdown(acked.summary)}`,
);
}
}
// Persist sync state so next sync is incremental (C1 fix: was missing).
@@ -728,6 +922,17 @@ export async function runSync(engine: BrainEngine, args: string[]) {
const jsonOut = args.includes('--json');
const yesFlag = args.includes('--yes');
const strategyArg = args.find((a, i) => args[i - 1] === '--strategy') as SyncOpts['strategy'] | undefined;
const concurrencyStr = args.find((a, i) => args[i - 1] === '--concurrency' || args[i - 1] === '--workers');
// v0.22.13 (PR #490 Q2): parseWorkers throws on '0', '-3', 'foo', '1.5' instead
// of silently falling through to auto-concurrency or NaN. Loud failure beats
// a 4-worker spawn from a typo.
let concurrency: number | undefined;
try {
concurrency = parseWorkers(concurrencyStr);
} catch (e) {
console.error(e instanceof Error ? e.message : String(e));
process.exit(1);
}
// v0.18.0 Step 5: --source resolves to a sources(id) row. Falls back
// to pre-v0.17 global config (sync.repo_path + sync.last_commit) when
@@ -820,10 +1025,18 @@ export async function runSync(engine: BrainEngine, args: string[]) {
dryRun, full, noPull, noEmbed, skipFailed, retryFailed,
sourceId: src.id,
strategy: cfg.strategy,
concurrency,
};
try {
const result = await performSync(engine, repoOpts);
printSyncResult(result);
// Codex P2: --all loop must also manage .gitignore per-source. Without
// this, multi-source users who rely on `gbrain sync --all` never get
// the advertised db_only ignore rules unless they sync each repo
// individually.
if (result.status !== 'dry_run' && result.status !== 'blocked_by_failures') {
manageGitignore(src.local_path!, engine.kind);
}
} catch (e: unknown) {
console.error(`Error syncing ${src.name}: ${e instanceof Error ? e.message : String(e)}`);
}
@@ -831,7 +1044,7 @@ export async function runSync(engine: BrainEngine, args: string[]) {
return;
}
const opts: SyncOpts = { repoPath, dryRun, full, noPull, noEmbed, skipFailed, retryFailed, sourceId, strategy: strategyArg };
const opts: SyncOpts = { repoPath, dryRun, full, noPull, noEmbed, skipFailed, retryFailed, sourceId, strategy: strategyArg, concurrency };
// Bug 9 — --retry-failed: before running normal sync, clear acknowledgment
// flags so the sync picks them up as fresh work. The actual re-attempt
@@ -850,6 +1063,18 @@ export async function runSync(engine: BrainEngine, args: string[]) {
if (!watch) {
const result = await performSync(engine, opts);
printSyncResult(result);
// Issue #2 + eng-review pass-2 finding #1 + Codex P1: manage .gitignore ONLY
// on successful sync. Skip on dry-run (don't mutate disk in preview mode)
// and blocked_by_failures (sync state is inconsistent — defer .gitignore
// until next clean run). Resolve the effective repo path so the wire-up
// fires in the common case where the user runs `gbrain sync` without
// passing --repo every time.
if (result.status !== 'dry_run' && result.status !== 'blocked_by_failures') {
const effectiveRepoPath = opts.repoPath ?? (await getDefaultSourcePath(engine));
if (effectiveRepoPath) {
manageGitignore(effectiveRepoPath, engine.kind);
}
}
return;
}
@@ -865,6 +1090,14 @@ export async function runSync(engine: BrainEngine, args: string[]) {
const ts = new Date().toISOString().slice(11, 19);
console.log(`[${ts}] Synced: +${result.added} ~${result.modified} -${result.deleted} R${result.renamed}`);
}
// Same gate as non-watch: only manage .gitignore on successful sync.
// Same repo-resolution path so watch mode catches the implicit-resolved case.
if (result.status !== 'dry_run' && result.status !== 'blocked_by_failures') {
const effectiveRepoPath = opts.repoPath ?? (await getDefaultSourcePath(engine));
if (effectiveRepoPath) {
manageGitignore(effectiveRepoPath, engine.kind);
}
}
} catch (e: unknown) {
consecutiveErrors++;
const msg = e instanceof Error ? e.message : String(e);
@@ -878,6 +1111,124 @@ export async function runSync(engine: BrainEngine, args: string[]) {
}
}
/**
* Auto-manage .gitignore entries for db_only directories.
*
* Caller invokes ONLY on successful sync this function trusts that the
* sync's data state is consistent. See `runSync` for the gating logic.
*
* Idempotent: re-running adds no duplicate entries. The managed block has
* a stable comment header so it's grep-able and editable.
*
* Skipped (with actionable warning) when:
* - GBRAIN_NO_GITIGNORE=1 D23 escape hatch for shared-repo setups
* - The repo is a git submodule (`.git` is a file not a directory)
* D49 lock; submodule .gitignore changes don't survive parent updates
*
* On PGLite (D4): emits a once-per-process soft-warn explaining that
* tiering has limited effect but still manages the .gitignore so the
* config-present user gets the gitignore housekeeping.
*
* Failures (write permission denied, EROFS, etc.) are caught, warned, and
* swallowed (D9 lock). Sync's primary job is moving data; .gitignore
* management is a side effect don't kill the main job for the side effect.
*/
let _pgliteTierWarned = false;
export function __resetPGLiteTierWarn(): void {
_pgliteTierWarned = false;
}
export function manageGitignore(
repoPath: string,
engineKind?: 'pglite' | 'postgres',
): void {
if (process.env.GBRAIN_NO_GITIGNORE === '1') {
return;
}
// D49: submodule detection. In a submodule, `.git` is a regular file
// (containing `gitdir: ../path/to/parent.git/modules/x`), not a directory.
const dotGit = join(repoPath, '.git');
if (existsSync(dotGit)) {
try {
if (statSync(dotGit).isFile()) {
console.warn(
`Note: skipping .gitignore management — ${repoPath} is a git submodule. ` +
`Add db_only directories to your parent repo's .gitignore manually.`,
);
return;
}
} catch {
// proceed; can't tell, default to managing
}
}
let storageConfig;
try {
storageConfig = loadStorageConfig(repoPath);
} catch (error) {
// StorageConfigError (overlap) or read error — surface, don't manage.
console.warn(
`Skipped .gitignore update: ${error instanceof Error ? error.message : String(error)}`,
);
return;
}
if (!storageConfig || storageConfig.db_only.length === 0) {
return;
}
// D4 soft-warn: storage tiering has limited effect on PGLite, but the
// .gitignore housekeeping still helps. Warn once per process; proceed.
if (engineKind === 'pglite' && !_pgliteTierWarned) {
_pgliteTierWarned = true;
console.warn(
`Note: storage tiering has limited effect on PGLite — pages live in your ` +
`local database file regardless of tier. Managing .gitignore anyway.`,
);
}
const gitignorePath = join(repoPath, '.gitignore');
let gitignoreContent = '';
if (existsSync(gitignorePath)) {
try {
gitignoreContent = readFileSync(gitignorePath, 'utf-8');
} catch (error) {
console.warn(
`Could not read ${gitignorePath} (${error instanceof Error ? error.message : String(error)}) — ` +
`skipping .gitignore update. Add db_only directories manually.`,
);
return;
}
}
const existingLines = new Set(gitignoreContent.split('\n').map((line) => line.trim()));
const linesToAdd: string[] = [];
for (const dir of storageConfig.db_only) {
if (!existingLines.has(dir) && !existingLines.has(`/${dir}`)) {
linesToAdd.push(dir);
}
}
if (linesToAdd.length === 0) return;
if (gitignoreContent && !gitignoreContent.endsWith('\n')) {
gitignoreContent += '\n';
}
gitignoreContent += '\n# Auto-managed by gbrain (db_only directories)\n';
gitignoreContent += linesToAdd.join('\n') + '\n';
try {
writeFileSync(gitignorePath, gitignoreContent);
} catch (error) {
console.warn(
`Could not update ${gitignorePath} (${error instanceof Error ? error.message : String(error)}) — ` +
`please add db_only directories manually:\n ${linesToAdd.join('\n ')}`,
);
}
}
function printSyncResult(result: SyncResult) {
switch (result.status) {
case 'up_to_date':
+113
View File
@@ -0,0 +1,113 @@
/**
* AgentRunner pluggable contract for invoking external agents (openclaw,
* hermes, codex, ) inside the claw-test harness. v1 ships a single
* implementation (openclaw); the interface stays narrow and concrete so
* adding a second runner in v1.1 is a ~50-line file.
*
* The harness wraps spawn/timeout/transcript-capture; runners only have to
* answer "where's your binary?" and "how do I invoke it with this prompt?".
*
*
* harness
* resolve(name) registry AgentRunner instance
* detect() runner reports binary path/availability
* invoke(...) runner spawns child, harness captures via TranscriptSink
*
*/
export interface AgentRunner {
/** Stable agent name used by --agent flag and friction `agent` field. */
readonly name: string;
/**
* Locate the agent binary and confirm it is executable. Pure check; never
* spawns. `binPath` is always an absolute path on success. `available=false`
* with a `reason` if not found / not executable.
*/
detect(): Promise<DetectResult>;
/**
* Invoke the agent with the given prompt. The runner is responsible for
* the per-agent argv shape. The harness owns timeouts, signals, and
* transcript capture (via `transcriptSink`).
*/
invoke(opts: InvokeOpts): Promise<InvokeResult>;
/** Optional per-agent post-install hook (e.g., routing-file fixup). */
postInstallHook?(opts: { workspaceDir: string }): Promise<void>;
}
export interface DetectResult {
available: boolean;
reason?: string;
binPath?: string;
}
export interface InvokeOpts {
/** Workspace dir the agent runs in. */
cwd: string;
/** The prompt content. The runner decides whether to write a temp file or pass via argv. */
brief: string;
/** Env to merge with the runner's defaults. Caller already restricted to allow-listed keys. */
env: Record<string, string>;
/** Wall-clock kill switch in ms. Harness handles SIGTERM → 5s grace → SIGKILL. */
timeoutMs: number;
/**
* Per-channel byte sink. The runner pipes child stdin/stdout/stderr into this
* instead of inheriting the parent's. Async-drain backpressure is handled
* inside the sink (D17), so the runner can call `write()` without awaiting.
*/
transcriptSink: TranscriptSink;
/** Optional override for which sub-agent the runner targets. */
agentName?: string;
}
export interface InvokeResult {
exitCode: number;
durationMs: number;
}
/** Async-drain sink. The harness owns the underlying file stream. */
export interface TranscriptSink {
write(event: TranscriptEvent): void;
/** Returns the byte offset that the next written event would have. */
nextOffset(): number;
/** Flush + close. Idempotent. */
close(): Promise<void>;
}
export interface TranscriptEvent {
ts: number;
channel: 'stdin' | 'stdout' | 'stderr';
bytes: Buffer;
}
// ---------------------------------------------------------------------------
// Registry
// ---------------------------------------------------------------------------
type AgentRunnerFactory = () => AgentRunner;
const registry = new Map<string, AgentRunnerFactory>();
export function registerAgentRunner(name: string, factory: AgentRunnerFactory): void {
registry.set(name, factory);
}
export function resolveAgentRunner(name: string): AgentRunner {
const factory = registry.get(name);
if (!factory) {
const known = [...registry.keys()].sort().join(', ') || '(none registered)';
throw new Error(`unknown agent ${JSON.stringify(name)}; registered: ${known}`);
}
return factory();
}
export function listRegisteredAgents(): string[] {
return [...registry.keys()].sort();
}
/** Reset registry — testing only. */
export function _resetRegistryForTests(): void {
registry.clear();
}
+58
View File
@@ -0,0 +1,58 @@
/**
* progress-tail parses gbrain's --progress-json events out of child stderr.
*
* The actual contract (verified post-Codex):
* - `gbrain --progress-json <subcommand>` writes JSONL events to STDERR
* - Stable phase names are dotted snake_case: `import.files`, `extract.links_fs`,
* `embed.pages`, `doctor.db_checks`, etc.
* - Each event line is a JSON object; non-progress stderr lines (warnings,
* debug output, errors) interleave with progress events. We tolerate them.
*
* Used by the verify phase to assert that each `expected_phases` entry from
* scenario.json saw at least one event from the corresponding command.
*/
export interface ProgressEvent {
phase: string;
event?: string; // 'start' | 'tick' | 'finish' | etc per docs/progress-events.md
ts?: string;
[key: string]: unknown;
}
/** Parse a single stderr buffer into the progress events it contains. */
export function parseProgressEvents(stderr: string): ProgressEvent[] {
const out: ProgressEvent[] = [];
for (const line of stderr.split('\n')) {
const trimmed = line.trim();
if (!trimmed.startsWith('{')) continue;
let parsed: unknown;
try {
parsed = JSON.parse(trimmed);
} catch {
continue;
}
if (parsed && typeof parsed === 'object' && typeof (parsed as any).phase === 'string') {
out.push(parsed as ProgressEvent);
}
}
return out;
}
/** Group events by phase name. */
export function eventsByPhase(events: ProgressEvent[]): Map<string, ProgressEvent[]> {
const m = new Map<string, ProgressEvent[]>();
for (const e of events) {
if (!m.has(e.phase)) m.set(e.phase, []);
m.get(e.phase)!.push(e);
}
return m;
}
/**
* Verify that every `expected` phase appears at least once in `events`.
* Returns the missing phase names (empty array on full coverage).
*/
export function verifyExpectedPhases(events: ProgressEvent[], expected: string[]): string[] {
const seen = new Set(events.map(e => e.phase));
return expected.filter(p => !seen.has(p));
}
+98
View File
@@ -0,0 +1,98 @@
/**
* OpenClaw runner invokes the real `openclaw` binary in a tempdir with a
* BRIEF.md prompt. Live mode only.
*
* Invocation pattern (verified against test/e2e/skills.test.ts and
* test/e2e/bench-vs-openclaw/harness.ts):
* openclaw agent --local --agent <agent-name> --message "<brief>"
*
* NOT `openclaw run --prompt-file BRIEF.md` (that flag does not exist
* Codex pass 2 of the eng review caught the speculative shape).
*
* Binary resolution: $OPENCLAW_BIN > `which openclaw` > unavailable.
* Path validation: must be absolute, must be executable, no '..' segments.
*/
import { execSync } from 'child_process';
import { statSync } from 'fs';
import type { AgentRunner, DetectResult, InvokeOpts, InvokeResult } from '../agent-runner.ts';
import { spawnWithCapture } from '../transcript-capture.ts';
const DEFAULT_AGENT_NAME = 'default';
/** Allow-list for env propagation when spawning openclaw. */
const ENV_ALLOWLIST = [
'PATH', 'HOME', 'USER', 'LANG', 'TZ', 'NODE_ENV',
'ANTHROPIC_API_KEY', 'OPENAI_API_KEY',
'GBRAIN_HOME', 'GBRAIN_FRICTION_RUN_ID', 'GBRAIN_DATABASE_URL',
];
export class OpenClawRunner implements AgentRunner {
readonly name = 'openclaw';
async detect(): Promise<DetectResult> {
const fromEnv = process.env.OPENCLAW_BIN?.trim();
let binPath: string | undefined;
if (fromEnv) {
const validation = validateAbsolutePath(fromEnv);
if (validation) return { available: false, reason: validation };
binPath = fromEnv;
} else {
try {
const out = execSync('which openclaw', { encoding: 'utf-8', stdio: ['ignore', 'pipe', 'ignore'] });
const found = out.trim();
if (!found || !found.startsWith('/')) {
return { available: false, reason: 'openclaw not on PATH' };
}
binPath = found;
} catch {
return { available: false, reason: 'openclaw not on PATH' };
}
}
if (!binPath) return { available: false, reason: 'no binary resolved' };
try {
const s = statSync(binPath);
if (!s.isFile()) return { available: false, reason: `not a regular file: ${binPath}` };
// eslint-disable-next-line no-bitwise
if (!(s.mode & 0o111)) return { available: false, reason: `not executable: ${binPath}` };
} catch (e) {
return { available: false, reason: `stat failed: ${e instanceof Error ? e.message : String(e)}` };
}
return { available: true, binPath };
}
async invoke(opts: InvokeOpts): Promise<InvokeResult> {
const detected = await this.detect();
if (!detected.available || !detected.binPath) {
throw new Error(`openclaw runner unavailable: ${detected.reason ?? 'unknown'}`);
}
const agentName = opts.agentName ?? DEFAULT_AGENT_NAME;
const args = ['agent', '--local', '--agent', agentName, '--message', opts.brief];
// Filter env to allow-list, then merge caller overrides.
const baseEnv: Record<string, string> = {};
for (const key of ENV_ALLOWLIST) {
const v = process.env[key];
if (typeof v === 'string') baseEnv[key] = v;
}
const env: Record<string, string> = { ...baseEnv, ...opts.env };
const result = await spawnWithCapture(detected.binPath, args, {
cwd: opts.cwd,
env,
timeoutMs: opts.timeoutMs,
transcriptSink: opts.transcriptSink,
});
return { exitCode: result.exitCode, durationMs: result.durationMs };
}
}
function validateAbsolutePath(p: string): string | null {
if (!p.startsWith('/')) return `OPENCLAW_BIN must be absolute; got ${p}`;
if (p.split('/').includes('..')) return `OPENCLAW_BIN must not contain '..' segments; got ${p}`;
return null;
}
+114
View File
@@ -0,0 +1,114 @@
/**
* scenario.json loader for the claw-test harness.
*
* test/fixtures/claw-test-scenarios/<name>/scenario.json:
* { kind: "fresh-install", expected_phases: ["import.files", ...], ... }
*
* The harness reads scenario.json to know which phases to assert from
* gbrain's --progress-json events. Pure local fs; no DB, no network.
*/
import { existsSync, readdirSync, readFileSync, statSync } from 'fs';
import { dirname, join, resolve } from 'path';
import { fileURLToPath } from 'url';
export type ScenarioKind = 'fresh-install' | 'upgrade';
export interface ScenarioConfig {
/** Directory the scenario was loaded from. Always absolute. */
dir: string;
/** Stable scenario name (the directory name). */
name: string;
/** Kind of scenario; drives setup-phase behavior. */
kind: ScenarioKind;
/** Stable phase names emitted by --progress-json that the harness asserts. */
expectedPhases: string[];
/** When kind==="upgrade": version we are simulating an upgrade FROM. */
fromVersion?: string;
/** Optional human-readable summary. */
description?: string;
/** Path to BRIEF.md (relative to scenario dir, default 'BRIEF.md'). */
briefRelative: string;
/** Path to brain markdown source (relative to scenario dir). For 'fresh-install': 'brain'. */
brainRelative?: string;
/** Path to seed dir for upgrade scenarios. */
seedRelative?: string;
}
/** Default fixtures root, override via $GBRAIN_CLAW_SCENARIOS_DIR for tests. */
function defaultFixturesRoot(): string {
if (process.env.GBRAIN_CLAW_SCENARIOS_DIR) {
return resolve(process.env.GBRAIN_CLAW_SCENARIOS_DIR);
}
// src/core/claw-test/scenarios.ts → ../../../test/fixtures/claw-test-scenarios
const here = dirname(fileURLToPath(import.meta.url));
return resolve(here, '..', '..', '..', 'test', 'fixtures', 'claw-test-scenarios');
}
/** List all available scenario names. */
export function listScenarios(root?: string): string[] {
const r = root ?? defaultFixturesRoot();
if (!existsSync(r)) return [];
return readdirSync(r)
.filter(name => {
const path = join(r, name);
try {
return statSync(path).isDirectory() && existsSync(join(path, 'scenario.json'));
} catch {
return false;
}
})
.sort();
}
/** Load and validate one scenario by name. */
export function loadScenario(name: string, root?: string): ScenarioConfig {
const r = root ?? defaultFixturesRoot();
const dir = join(r, name);
const cfgPath = join(dir, 'scenario.json');
if (!existsSync(cfgPath)) {
throw new Error(`scenario ${JSON.stringify(name)} not found at ${cfgPath}`);
}
let raw: unknown;
try {
raw = JSON.parse(readFileSync(cfgPath, 'utf-8'));
} catch (e) {
throw new Error(`scenario ${JSON.stringify(name)}: malformed scenario.json (${e instanceof Error ? e.message : e})`);
}
if (!raw || typeof raw !== 'object') {
throw new Error(`scenario ${JSON.stringify(name)}: scenario.json must be a JSON object`);
}
const cfg = raw as Record<string, unknown>;
if (cfg.kind !== 'fresh-install' && cfg.kind !== 'upgrade') {
throw new Error(`scenario ${JSON.stringify(name)}: unknown kind ${JSON.stringify(cfg.kind)}`);
}
if (!Array.isArray(cfg.expected_phases) || !cfg.expected_phases.every(x => typeof x === 'string')) {
throw new Error(`scenario ${JSON.stringify(name)}: expected_phases must be a string[]`);
}
const briefRel = typeof cfg.brief === 'string' ? cfg.brief : 'BRIEF.md';
if (!existsSync(join(dir, briefRel))) {
throw new Error(`scenario ${JSON.stringify(name)}: BRIEF.md missing at ${briefRel}`);
}
const out: ScenarioConfig = {
dir,
name,
kind: cfg.kind,
expectedPhases: cfg.expected_phases as string[],
briefRelative: briefRel,
};
if (typeof cfg.from_version === 'string') out.fromVersion = cfg.from_version;
if (typeof cfg.description === 'string') out.description = cfg.description;
if (typeof cfg.brain === 'string') out.brainRelative = cfg.brain;
if (typeof cfg.seed === 'string') out.seedRelative = cfg.seed;
// Default brain path conventions
if (!out.brainRelative && existsSync(join(dir, 'brain'))) out.brainRelative = 'brain';
if (!out.seedRelative && out.kind === 'upgrade' && existsSync(join(dir, 'seed'))) {
out.seedRelative = 'seed';
}
return out;
}
/** Read BRIEF.md content for this scenario. Used by --live mode. */
export function readBrief(scenario: ScenarioConfig): string {
return readFileSync(join(scenario.dir, scenario.briefRelative), 'utf-8');
}
+123
View File
@@ -0,0 +1,123 @@
/**
* seed-pglite replay a SQL dump into a fresh PGLite database, then let
* gbrain's migration chain walk forward.
*
* Codex caught (eng review pass 2) that existing migration helpers
* (test/e2e/helpers.ts:204) are Postgres-only they rewind schema_version
* and replay against real Postgres. PGLite has no equivalent. This helper
* fills that gap so the `upgrade-from-v0.18` claw-test scenario is
* reproducible.
*
* Usage:
* const dbPath = await seedPglite('/tmp/run-x/.gbrain/brain.pglite', seedSql);
* // Then run `gbrain init --pglite --path <dbPath>` — the migration chain
* // detects the seeded schema_version and migrates forward to LATEST.
*/
import { existsSync, mkdirSync, readFileSync } from 'fs';
import { dirname } from 'path';
import { PGLiteEngine } from '../pglite-engine.ts';
export interface SeedOpts {
/** Absolute path to the .pglite file to create. */
dbPath: string;
/** Raw SQL dump to replay. */
sql: string;
}
/**
* Open a fresh PGLite at `dbPath`, execute the SQL dump, disconnect.
* Throws on SQL errors with a structured message that names the failing
* statement (helpful for debugging seed drift).
*/
export async function seedPglite(opts: SeedOpts): Promise<void> {
const dir = dirname(opts.dbPath);
if (!existsSync(dir)) mkdirSync(dir, { recursive: true });
const engine = new PGLiteEngine();
try {
await engine.connect({ engine: 'pglite', database_path: opts.dbPath });
// Execute statements one at a time so an error names the offending
// statement. The seed file is committed to source so we can normalize
// its line endings; we rely on `;\n` as the statement terminator.
const statements = splitStatements(opts.sql);
for (const stmt of statements) {
const trimmed = stmt.trim();
if (!trimmed) continue;
try {
await (engine as any).db.exec(trimmed);
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
const preview = trimmed.slice(0, 120).replace(/\s+/g, ' ');
throw new Error(`seedPglite: SQL execution failed at "${preview}…": ${msg}`);
}
}
} finally {
await engine.disconnect();
}
}
/** Read seed SQL from disk and replay into `dbPath`. */
export async function seedPgliteFromFile(opts: { dbPath: string; sqlPath: string }): Promise<void> {
if (!existsSync(opts.sqlPath)) {
throw new Error(`seedPglite: seed SQL not found at ${opts.sqlPath}`);
}
const sql = readFileSync(opts.sqlPath, 'utf-8');
return seedPglite({ dbPath: opts.dbPath, sql });
}
/**
* Split a SQL dump into individual statements. Naïve `;` split that respects
* single-quoted strings and `--` line comments. Sufficient for canonical
* pg_dump output; intentionally NOT a full SQL parser.
*/
function splitStatements(sql: string): string[] {
const out: string[] = [];
let buf = '';
let inSingle = false;
let inLineComment = false;
let i = 0;
while (i < sql.length) {
const c = sql[i];
const next = sql[i + 1];
if (inLineComment) {
buf += c;
if (c === '\n') inLineComment = false;
i++;
continue;
}
if (inSingle) {
buf += c;
if (c === "'" && next === "'") { buf += next; i += 2; continue; }
if (c === "'") inSingle = false;
i++;
continue;
}
if (c === '-' && next === '-') {
inLineComment = true;
buf += c;
i++;
continue;
}
if (c === "'") {
inSingle = true;
buf += c;
i++;
continue;
}
if (c === ';') {
buf += c;
out.push(buf);
buf = '';
i++;
continue;
}
buf += c;
i++;
}
if (buf.trim()) out.push(buf);
return out;
}
/** Exposed for tests. */
export const _internal = { splitStatements };
+172
View File
@@ -0,0 +1,172 @@
/**
* Transcript capture for live-mode agent runs (D8 + D14, D17 backpressure).
*
* The existing minions/audit infrastructure is for INTERNAL gbrain subagents
* only. External openclaw/hermes subprocesses don't write to those tables
* v1 builds its own capture channel here.
*
* Output: JSONL at `<run-tempdir>/transcript.jsonl`, one event per line.
* { schema_version: "1", ts, channel, byte_offset, bytes_b64 }
*
* child stdout/stderr piped TranscriptSink.write()
*
*
* fs.createWriteStream (flags: 'a')
*
* honors 'drain' events to avoid blocking
* the child when bursts exceed the pipe buffer
*
* transcript.jsonl (line-tolerant readers
* skip malformed; render() resolves
* byte_offset readable lines)
*
* Friction CLI's `transcript_offset` field references the byte offset INTO
* `transcript.jsonl` (not into the captured payload). Render --transcripts
* reads the file and finds the line that contains that offset.
*/
import { createWriteStream, type WriteStream } from 'fs';
import { spawn, type ChildProcess } from 'child_process';
import { dirname } from 'path';
import { mkdirSync, existsSync } from 'fs';
import type { TranscriptEvent, TranscriptSink } from './agent-runner.ts';
// ---------------------------------------------------------------------------
// Sink
// ---------------------------------------------------------------------------
export function createTranscriptSink(path: string): TranscriptSink {
const dir = dirname(path);
if (!existsSync(dir)) mkdirSync(dir, { recursive: true });
const stream: WriteStream = createWriteStream(path, { flags: 'a' });
let bytesWritten = 0;
let drainPromise: Promise<void> | null = null;
function awaitDrain(): Promise<void> {
if (drainPromise) return drainPromise;
drainPromise = new Promise<void>(resolve => {
stream.once('drain', () => {
drainPromise = null;
resolve();
});
});
return drainPromise;
}
return {
write(event: TranscriptEvent) {
const line = JSON.stringify({
schema_version: '1',
ts: event.ts,
channel: event.channel,
byte_offset: bytesWritten,
bytes_b64: event.bytes.toString('base64'),
}) + '\n';
bytesWritten += Buffer.byteLength(line, 'utf-8');
const ok = stream.write(line, 'utf-8');
// If the kernel buffer is full, write() returns false. We don't await
// here (callers don't expect that), but next callers wait on drain
// before writing further. Bun's WritableStream is small; the drain
// window is typically a few µs.
if (!ok) void awaitDrain();
},
nextOffset(): number {
return bytesWritten;
},
async close(): Promise<void> {
await new Promise<void>((resolve, reject) => {
stream.end((err?: Error | null) => err ? reject(err) : resolve());
});
},
};
}
// ---------------------------------------------------------------------------
// spawnWithCapture
// ---------------------------------------------------------------------------
export interface SpawnOpts {
cwd: string;
env: Record<string, string>;
timeoutMs: number;
transcriptSink: TranscriptSink;
/** Optional fixed input to write on stdin then close. */
stdinPayload?: string;
}
export interface SpawnResult {
exitCode: number;
durationMs: number;
/** True if SIGTERM/SIGKILL was issued due to timeout. */
timedOut: boolean;
}
const SIGTERM_GRACE_MS = 5_000;
export async function spawnWithCapture(bin: string, args: string[], opts: SpawnOpts): Promise<SpawnResult> {
const start = Date.now();
return new Promise((resolve, reject) => {
let child: ChildProcess;
try {
child = spawn(bin, args, {
cwd: opts.cwd,
env: opts.env,
stdio: ['pipe', 'pipe', 'pipe'],
shell: false,
});
} catch (e) {
reject(e);
return;
}
let timedOut = false;
let killTimer: ReturnType<typeof setTimeout> | null = null;
const wallClockTimer = setTimeout(() => {
timedOut = true;
try { child.kill('SIGTERM'); } catch { /* already gone */ }
killTimer = setTimeout(() => {
try { child.kill('SIGKILL'); } catch { /* already gone */ }
}, SIGTERM_GRACE_MS);
}, opts.timeoutMs);
child.stdout?.on('data', (chunk: Buffer) => {
opts.transcriptSink.write({ ts: Date.now(), channel: 'stdout', bytes: chunk });
});
child.stderr?.on('data', (chunk: Buffer) => {
opts.transcriptSink.write({ ts: Date.now(), channel: 'stderr', bytes: chunk });
});
if (opts.stdinPayload !== undefined && child.stdin) {
try {
opts.transcriptSink.write({
ts: Date.now(),
channel: 'stdin',
bytes: Buffer.from(opts.stdinPayload, 'utf-8'),
});
child.stdin.end(opts.stdinPayload, 'utf-8');
} catch (e) {
reject(e);
return;
}
}
child.on('error', (err) => {
clearTimeout(wallClockTimer);
if (killTimer) clearTimeout(killTimer);
reject(err);
});
child.on('close', (code) => {
clearTimeout(wallClockTimer);
if (killTimer) clearTimeout(killTimer);
resolve({
exitCode: typeof code === 'number' ? code : (timedOut ? 124 : 1),
durationMs: Date.now() - start,
timedOut,
});
});
});
}
+28 -5
View File
@@ -19,9 +19,11 @@ export type DbUrlSource =
| 'config-file-path' // PGLite: config file present, no URL but database_path set
| null;
// Lazy-evaluated to avoid calling homedir() at module scope (breaks in serverless/bundled environments)
function getConfigDir() { return join(homedir(), '.gbrain'); }
function getConfigPath() { return join(getConfigDir(), 'config.json'); }
// Internal aliases retained for backwards compatibility with the existing call
// sites below. They forward to the exported configDir()/configPath() so
// GBRAIN_HOME is honored uniformly. Lazy: never call homedir() at module scope.
function getConfigDir() { return configDir(); }
function getConfigPath() { return configPath(); }
export interface GBrainConfig {
engine: 'postgres' | 'pglite';
@@ -88,9 +90,20 @@ export function toEngineConfig(config: GBrainConfig): EngineConfig {
export function configDir(): string {
// Allow override for tests, Docker, and multi-tenant deployments.
// Matches the `GBRAIN_AUDIT_DIR` convention in src/core/minions/handlers/shell-audit.ts.
// GBRAIN_HOME is a parent dir; we always append '.gbrain' ourselves so
// setting GBRAIN_HOME=/tmp/x yields configDir() === '/tmp/x/.gbrain'.
// Validates the override: must be absolute, no '..' segments.
const override = process.env.GBRAIN_HOME;
if (override && override.trim()) return join(override, '.gbrain');
if (override && override.trim()) {
const trimmed = override.trim();
if (!trimmed.startsWith('/')) {
throw new Error(`GBRAIN_HOME must be an absolute path; got: ${trimmed}`);
}
if (trimmed.split('/').includes('..')) {
throw new Error(`GBRAIN_HOME must not contain '..' segments; got: ${trimmed}`);
}
return join(trimmed, '.gbrain');
}
return join(homedir(), '.gbrain');
}
@@ -98,6 +111,16 @@ export function configPath(): string {
return join(configDir(), 'config.json');
}
/**
* Sugar for joining paths under the active gbrain home. Use this anywhere you
* would otherwise write `join(homedir(), '.gbrain', ...rest)`. Honors
* GBRAIN_HOME, validates input, and centralizes the convention so future
* audits stay simple.
*/
export function gbrainPath(...segments: string[]): string {
return join(configDir(), ...segments);
}
/**
* Introspect where the active DB URL would come from if we tried to connect.
* Never throws, never connects. Env vars take precedence (matches loadConfig).
+5 -3
View File
@@ -39,7 +39,8 @@
import { existsSync, readFileSync, writeFileSync, unlinkSync, mkdirSync, statSync } from 'fs';
import { join } from 'path';
import { homedir, hostname } from 'os';
import { hostname } from 'os';
import { gbrainPath } from './config.ts';
import type { BrainEngine } from './engine.ts';
import { createProgress, type ProgressReporter } from './progress.ts';
import { getCliOptions, cliOptsToProgressOptions } from './cli-options.ts';
@@ -154,7 +155,8 @@ export interface CycleOpts {
const CYCLE_LOCK_ID = 'gbrain-cycle';
const LOCK_TTL_MS = 30 * 60 * 1000; // 30 minutes
const LOCK_FILE_PATH_DEFAULT = join(homedir(), '.gbrain', 'cycle.lock');
// Lazy: GBRAIN_HOME may be set after module load; resolve at call time.
const getLockFilePathDefault = () => gbrainPath('cycle.lock');
interface LockHandle {
release: () => Promise<void>;
@@ -256,7 +258,7 @@ async function acquirePostgresLock(engine: BrainEngine): Promise<LockHandle | nu
* The file contains `{pid}\n{iso-timestamp}`. Staleness = mtime older
* than LOCK_TTL_MS OR the PID is no longer alive on this host.
*/
function acquireFileLock(lockPath = LOCK_FILE_PATH_DEFAULT): LockHandle | null {
function acquireFileLock(lockPath = getLockFilePathDefault()): LockHandle | null {
mkdirSync(join(lockPath, '..'), { recursive: true });
const pid = process.pid;
+140
View File
@@ -0,0 +1,140 @@
/**
* Generic DB-backed lock primitive.
*
* Reuses the gbrain_cycle_locks table (id PK + holder_pid + ttl_expires_at)
* with a parameterized lock id. Both `gbrain-cycle` (the broad cycle lock)
* and `gbrain-sync` (performSync's writer lock) live here.
*
* Why not pg_advisory_xact_lock: it is session-scoped, and PgBouncer
* transaction pooling drops session state between calls. This row-based
* lock survives PgBouncer because it's plain INSERT/UPDATE/DELETE with
* a TTL fallback (a crashed holder's row times out).
*
* Why a separate table-row per lock id rather than reusing the cycle lock:
* the cycle lock is broader (covers every phase). performSync's write-window
* is narrower. If performSync reused the cycle lock and the cycle handler
* called performSync, the inner acquire would deadlock against itself. Two
* lock ids let callers nest cleanly: cycle holds gbrain-cycle for its run;
* performSync (called from anywhere cycle, jobs handler, CLI) takes
* gbrain-sync just for the write window.
*
* v0.22.13 added in PR #490 to fix CODEX-2 (no cross-process lock for
* direct sync paths). The cycle path was already protected.
*/
import { hostname } from 'os';
import type { BrainEngine } from './engine.ts';
export interface DbLockHandle {
id: string;
release: () => Promise<void>;
refresh: () => Promise<void>;
}
/** Default TTL: 30 minutes, same as cycle lock. */
const DEFAULT_TTL_MINUTES = 30;
/**
* Try to acquire a named DB lock.
*
* Returns a handle on success. Returns `null` if another live holder has
* the lock (its row exists and ttl_expires_at is in the future).
*
* The acquire is upsert-style:
* INSERT ... ON CONFLICT (id) DO UPDATE
* ... WHERE existing.ttl_expires_at < NOW()
* RETURNING id
*
* Empty RETURNING means the existing row is still live. An expired holder
* (worker crashed without releasing) is auto-superseded by the UPDATE
* branch.
*/
export async function tryAcquireDbLock(
engine: BrainEngine,
lockId: string,
ttlMinutes: number = DEFAULT_TTL_MINUTES,
): Promise<DbLockHandle | null> {
const pid = process.pid;
const host = hostname();
// Engine-agnostic: prefer the engine's raw escape hatch (`sql` for postgres-js,
// `db.query` for PGLite). Mirrors cycle.ts's pattern so behavior stays identical.
const maybePG = engine as unknown as { sql?: (...args: unknown[]) => Promise<unknown> };
const maybePGLite = engine as unknown as {
db?: { query: (sql: string, params?: unknown[]) => Promise<{ rows: unknown[] }> };
};
if (engine.kind === 'postgres' && maybePG.sql) {
const sql = maybePG.sql as any;
const ttl = `${ttlMinutes} minutes`;
const rows: Array<{ id: string }> = await sql`
INSERT INTO gbrain_cycle_locks (id, holder_pid, holder_host, acquired_at, ttl_expires_at)
VALUES (${lockId}, ${pid}, ${host}, NOW(), NOW() + ${ttl}::interval)
ON CONFLICT (id) DO UPDATE
SET holder_pid = ${pid},
holder_host = ${host},
acquired_at = NOW(),
ttl_expires_at = NOW() + ${ttl}::interval
WHERE gbrain_cycle_locks.ttl_expires_at < NOW()
RETURNING id
`;
if (rows.length === 0) return null;
return {
id: lockId,
refresh: async () => {
await sql`
UPDATE gbrain_cycle_locks
SET ttl_expires_at = NOW() + ${ttl}::interval
WHERE id = ${lockId} AND holder_pid = ${pid}
`;
},
release: async () => {
await sql`
DELETE FROM gbrain_cycle_locks
WHERE id = ${lockId} AND holder_pid = ${pid}
`;
},
};
}
if (engine.kind === 'pglite' && maybePGLite.db) {
const db = maybePGLite.db;
const ttl = `${ttlMinutes} minutes`;
const { rows } = await db.query(
`INSERT INTO gbrain_cycle_locks (id, holder_pid, holder_host, acquired_at, ttl_expires_at)
VALUES ($1, $2, $3, NOW(), NOW() + $4::interval)
ON CONFLICT (id) DO UPDATE
SET holder_pid = $2,
holder_host = $3,
acquired_at = NOW(),
ttl_expires_at = NOW() + $4::interval
WHERE gbrain_cycle_locks.ttl_expires_at < NOW()
RETURNING id`,
[lockId, pid, host, ttl],
);
if (rows.length === 0) return null;
return {
id: lockId,
refresh: async () => {
await db.query(
`UPDATE gbrain_cycle_locks
SET ttl_expires_at = NOW() + $1::interval
WHERE id = $2 AND holder_pid = $3`,
[ttl, lockId, pid],
);
},
release: async () => {
await db.query(
`DELETE FROM gbrain_cycle_locks WHERE id = $1 AND holder_pid = $2`,
[lockId, pid],
);
},
};
}
throw new Error(`Unknown engine kind for db-lock: ${engine.kind}`);
}
/** Lock id for performSync's writer window. Distinct from gbrain-cycle so the
* cycle handler can hold gbrain-cycle while performSync (called from inside
* the cycle) acquires gbrain-sync. */
export const SYNC_LOCK_ID = 'gbrain-sync';
+3
View File
@@ -2,6 +2,7 @@ import postgres from 'postgres';
import { GBrainError, type EngineConfig } from './types.ts';
import { SCHEMA_SQL } from './schema-embedded.ts';
import type { BrainEngine } from './engine.ts';
import { verifySchema } from './schema-verify.ts';
let sql: ReturnType<typeof postgres> | null = null;
let connectedUrl: string | null = null;
@@ -237,6 +238,8 @@ export async function initSchema(): Promise<void> {
}
}
export { verifySchema } from './schema-verify.ts';
export async function withTransaction<T>(fn: (tx: ReturnType<typeof postgres>) => Promise<T>): Promise<T> {
const conn = getConnection();
return conn.begin(async (tx) => {
+83
View File
@@ -0,0 +1,83 @@
/**
* Recursive filesystem walk into a slug Stats map.
*
* Replaces per-page `existsSync` + `statSync` syscall storms (Issue #14 of
* the v0.22.3 eng review). On a 200K-page brain the per-page approach was
* 400K syscalls in a synchronous loop; this walk is one syscall per directory
* plus one stat per file, then O(1) Map lookups for everything downstream.
*
* The slug key is the on-disk path relative to the brain repo, with the
* trailing `.md` stripped, matching how pages are stored: `people/alice.md`
* on disk becomes `people/alice` as a slug.
*
* Skipped entries:
* - `.git/`, `node_modules/`, and dot-directories generally not part of
* the brain's page namespace. Speeds up walks significantly on dirty
* working copies.
* - Files that don't end in `.md`. Sidecar JSON, raw binary attachments,
* etc. are tracked by the brain but not via slugs.
*/
import { readdirSync, statSync, type Stats, type Dirent } from 'fs';
import { join } from 'path';
export interface DiskFileEntry {
size: number;
mtimeMs: number;
}
/**
* Walk `repoPath` and return a Map of slug file metadata for every `.md`
* file. Skips dot-directories. Synchronous (matches the call-site shape and
* the io pattern of stat-heavy scans).
*
* @param repoPath Absolute path to the brain repo root.
* @returns Map keyed by slug (no `.md` suffix). Empty map if repoPath
* doesn't exist or contains no markdown files.
*/
export function walkBrainRepo(repoPath: string): Map<string, DiskFileEntry> {
const result = new Map<string, DiskFileEntry>();
function recurse(dirPath: string, slugPrefix: string): void {
// Annotate as Dirent[] explicitly: ReturnType<typeof readdirSync> with
// withFileTypes:true picks an overload union that includes
// Dirent<Buffer<ArrayBufferLike>>, which makes entry.name a Buffer in
// strict tsc mode. Cast to the string-based Dirent[] (same shape sync.ts
// uses for its own filesystem walk).
let entries: Dirent[];
try {
entries = readdirSync(dirPath, { withFileTypes: true }) as unknown as Dirent[];
} catch {
return; // unreadable directory — skip silently
}
for (const entry of entries) {
// Skip dot-directories (.git, .gbrain, .vscode, etc) and node_modules.
if (entry.name.startsWith('.') || entry.name === 'node_modules') continue;
const childPath = join(dirPath, entry.name);
if (entry.isDirectory()) {
recurse(childPath, slugPrefix ? `${slugPrefix}/${entry.name}` : entry.name);
continue;
}
if (!entry.isFile()) continue;
if (!entry.name.endsWith('.md')) continue;
let stats: Stats;
try {
stats = statSync(childPath);
} catch {
continue; // race: file deleted between readdir and stat
}
const slug = slugPrefix
? `${slugPrefix}/${entry.name.slice(0, -3)}`
: entry.name.slice(0, -3);
result.set(slug, { size: stats.size, mtimeMs: stats.mtimeMs });
}
}
recurse(repoPath, '');
return result;
}
+4 -3
View File
@@ -12,7 +12,7 @@
import { appendFileSync, readFileSync, existsSync, mkdirSync, writeFileSync, renameSync } from 'fs';
import { join, dirname } from 'path';
import { homedir } from 'os';
import { gbrainPath } from './config.ts';
// ---------------------------------------------------------------------------
// Types
@@ -45,7 +45,8 @@ export interface TestCase {
source: 'fail-improve-loop';
}
const LOG_DIR = join(homedir(), '.gbrain', 'fail-improve');
// Lazy: GBRAIN_HOME may be set after module load, so resolve at call time.
const getLogDir = () => gbrainPath('fail-improve');
const MAX_ENTRIES = 1000;
// ---------------------------------------------------------------------------
@@ -76,7 +77,7 @@ export class FailImproveLoop {
private logDir: string;
constructor(logDir?: string) {
this.logDir = logDir || LOG_DIR;
this.logDir = logDir || getLogDir();
}
/**
+374
View File
@@ -0,0 +1,374 @@
/**
* Friction reporter JSONL-backed signal capture for the claw-test feedback loop.
*
* The friction CLI (`gbrain friction log/render/list/summary`) writes here.
* The claw-test harness reads here. The agent calls `gbrain friction log`
* directly when it hits something confusing, missing, or wrong.
*
* Storage shape: append-only JSONL files under `$GBRAIN_HOME/friction/`.
* - `<run-id>.jsonl` for each harness run (run-id from $GBRAIN_FRICTION_RUN_ID)
* - `standalone.jsonl` for entries logged outside a harness run
*
* Schema is a flat extension of StructuredAgentError fields (per D20). Render
* reads one level. Readers tolerate malformed lines (skip + warn) so partial
* runs don't break later analysis.
*
* appendFileSync
* writer() <runId>.jsonl (one
* (atomic if line entry per line)
* PIPE_BUF/4KB)
*
*
* reader() / render()
* skip malformed + warn
*/
import { appendFileSync, existsSync, readdirSync, readFileSync, mkdirSync, statSync } from 'fs';
import { dirname, join } from 'path';
import { homedir } from 'os';
import { gbrainPath } from './config.ts';
import { VERSION } from '../version.ts';
// ---------------------------------------------------------------------------
// Schema
// ---------------------------------------------------------------------------
export type FrictionKind = 'friction' | 'delight' | 'phase-marker' | 'interrupted';
export type FrictionSeverity = 'confused' | 'error' | 'blocker' | 'nit';
export type FrictionSource = 'claw' | 'harness';
export type PhaseMarker = 'start' | 'end';
/** One JSONL entry. Flat extension of StructuredAgentError per D20. */
export interface FrictionEntry {
schema_version: '1';
ts: string; // ISO 8601
run_id: string;
phase: string;
kind: FrictionKind;
/** Required for kind=friction|delight. Optional for phase-marker (purely informational). */
severity?: FrictionSeverity;
message: string;
hint?: string;
/** StructuredAgentError envelope fields, flattened. */
class?: string;
code?: string;
docs_url?: string;
source: FrictionSource;
cwd: string;
gbrain_version: string;
agent?: string;
/** Byte offset into the run's transcript.jsonl (live mode). */
transcript_offset?: number;
/** For phase-marker entries only. */
marker?: PhaseMarker;
}
export interface FrictionLogInput {
severity?: FrictionSeverity;
phase: string;
message: string;
hint?: string;
runId?: string;
kind?: FrictionKind;
source?: FrictionSource;
agent?: string;
transcriptOffset?: number;
marker?: PhaseMarker;
/** When the writer is called from the harness wrapping a child error. */
errorClass?: string;
errorCode?: string;
docsUrl?: string;
}
// ---------------------------------------------------------------------------
// Path resolution
// ---------------------------------------------------------------------------
/** Resolve the directory all friction JSONL files live under. */
export function frictionDir(): string {
return gbrainPath('friction');
}
/** Resolve the JSONL file path for a given run-id. */
export function frictionFile(runId: string): string {
return join(frictionDir(), `${sanitizeRunId(runId)}.jsonl`);
}
/** Resolve the active run-id, falling back to 'standalone' (D19). */
export function activeRunId(): string {
const env = process.env.GBRAIN_FRICTION_RUN_ID?.trim();
return env && env.length > 0 ? env : 'standalone';
}
/** Sanitize: only [a-zA-Z0-9._-]; reject anything else to keep filenames sane. */
function sanitizeRunId(runId: string): string {
if (!/^[a-zA-Z0-9._-]+$/.test(runId)) {
throw new Error(`invalid run-id ${JSON.stringify(runId)} (allowed: [a-zA-Z0-9._-])`);
}
return runId;
}
// ---------------------------------------------------------------------------
// Writer
// ---------------------------------------------------------------------------
/** Maximum message length; truncated to keep each line under PIPE_BUF for atomic appends. */
const MAX_MESSAGE_CHARS = 3500;
/** Append one friction entry to the run's JSONL. */
export function logFriction(input: FrictionLogInput): void {
const runId = input.runId ?? activeRunId();
const dir = frictionDir();
if (!existsSync(dir)) mkdirSync(dir, { recursive: true });
const message = truncate(input.message, MAX_MESSAGE_CHARS);
const entry: FrictionEntry = {
schema_version: '1',
ts: new Date().toISOString(),
run_id: runId,
phase: input.phase,
kind: input.kind ?? 'friction',
message,
source: input.source ?? 'claw',
cwd: process.cwd(),
gbrain_version: VERSION,
};
if (input.severity) entry.severity = input.severity;
if (input.hint) entry.hint = input.hint;
if (input.errorClass) entry.class = input.errorClass;
if (input.errorCode) entry.code = input.errorCode;
if (input.docsUrl) entry.docs_url = input.docsUrl;
if (input.agent) entry.agent = input.agent;
if (input.transcriptOffset !== undefined) entry.transcript_offset = input.transcriptOffset;
if (input.marker) entry.marker = input.marker;
const line = JSON.stringify(entry) + '\n';
appendFileSync(frictionFile(runId), line, 'utf-8');
}
function truncate(s: string, max: number): string {
if (s.length <= max) return s;
return s.slice(0, max - 14) + '…[truncated]';
}
// ---------------------------------------------------------------------------
// Reader
// ---------------------------------------------------------------------------
export interface ReadResult {
entries: FrictionEntry[];
/** Count of malformed JSONL lines that were skipped. */
malformed: number;
}
/** Read all entries from a run's JSONL, skipping malformed lines. */
export function readFriction(runId: string): ReadResult {
const path = frictionFile(runId);
if (!existsSync(path)) {
throw new Error(`run-id "${runId}" not found at ${path}`);
}
const raw = readFileSync(path, 'utf-8');
const entries: FrictionEntry[] = [];
let malformed = 0;
for (const line of raw.split('\n')) {
if (!line.trim()) continue;
try {
const parsed = JSON.parse(line);
// Light shape check: must have ts + kind + phase + message
if (typeof parsed.ts === 'string' && typeof parsed.kind === 'string' && typeof parsed.phase === 'string' && typeof parsed.message === 'string') {
entries.push(parsed as FrictionEntry);
} else {
malformed++;
}
} catch {
malformed++;
}
}
return { entries, malformed };
}
/** List run-ids with summary counts. Returns most-recent-first. */
export interface RunSummary {
runId: string;
path: string;
mtime: Date;
counts: { friction: number; delight: number; interrupted: boolean; bySeverity: Record<string, number> };
}
export function listRuns(): RunSummary[] {
const dir = frictionDir();
if (!existsSync(dir)) return [];
const out: RunSummary[] = [];
for (const file of readdirSync(dir)) {
if (!file.endsWith('.jsonl')) continue;
const runId = file.slice(0, -'.jsonl'.length);
const path = join(dir, file);
const stat = statSync(path);
let read: ReadResult;
try {
read = readFriction(runId);
} catch {
continue;
}
const counts = { friction: 0, delight: 0, interrupted: false, bySeverity: {} as Record<string, number> };
for (const e of read.entries) {
if (e.kind === 'friction') counts.friction++;
if (e.kind === 'delight') counts.delight++;
if (e.kind === 'interrupted') counts.interrupted = true;
if (e.severity) counts.bySeverity[e.severity] = (counts.bySeverity[e.severity] ?? 0) + 1;
}
out.push({ runId, path, mtime: stat.mtime, counts });
}
out.sort((a, b) => b.mtime.getTime() - a.mtime.getTime());
return out;
}
// ---------------------------------------------------------------------------
// Renderer
// ---------------------------------------------------------------------------
export interface RenderOpts {
format?: 'md' | 'json';
redact?: boolean;
/** When true, transcript_offset values are resolved against this transcript file. */
transcriptPath?: string;
}
/** Render entries grouped by severity then phase. Returns the rendered string. */
export function renderReport(runId: string, opts: RenderOpts = {}): string {
const { entries, malformed } = readFriction(runId);
const format = opts.format ?? 'md';
const redact = opts.redact ?? (format === 'md');
const transformed = entries.map(e => redact ? redactEntry(e) : e);
if (format === 'json') {
return JSON.stringify({ run_id: runId, malformed, entries: transformed }, null, 2);
}
// Markdown grouping: severity (blocker > error > confused > nit > none) → phase
const sevOrder: (FrictionSeverity | 'none')[] = ['blocker', 'error', 'confused', 'nit', 'none'];
const bySev = new Map<string, FrictionEntry[]>();
for (const e of transformed) {
if (e.kind !== 'friction' && e.kind !== 'delight') continue;
const k = e.severity ?? 'none';
if (!bySev.has(k)) bySev.set(k, []);
bySev.get(k)!.push(e);
}
const lines: string[] = [];
lines.push(`# Friction report — \`${runId}\``);
lines.push('');
const totalFriction = entries.filter(e => e.kind === 'friction').length;
const totalDelight = entries.filter(e => e.kind === 'delight').length;
lines.push(`**${totalFriction} friction · ${totalDelight} delight**${malformed > 0 ? ` · ${malformed} malformed line(s) skipped` : ''}`);
lines.push('');
if (entries.some(e => e.kind === 'interrupted')) {
lines.push('> ⚠ **Run was interrupted.** Some phases may not have completed.');
lines.push('');
}
for (const sev of sevOrder) {
const bucket = bySev.get(sev);
if (!bucket || bucket.length === 0) continue;
lines.push(`## ${sev === 'none' ? '(no severity)' : sev}`);
lines.push('');
// Group by phase within severity
const byPhase = new Map<string, FrictionEntry[]>();
for (const e of bucket) {
if (!byPhase.has(e.phase)) byPhase.set(e.phase, []);
byPhase.get(e.phase)!.push(e);
}
for (const [phase, phaseEntries] of byPhase) {
lines.push(`### \`${phase}\``);
lines.push('');
for (const e of phaseEntries) {
lines.push(`- ${e.kind === 'delight' ? '✨' : '·'} ${e.message}`);
if (e.hint) lines.push(` - hint: ${e.hint}`);
if (e.code) lines.push(` - code: \`${e.code}\``);
if (e.docs_url) lines.push(` - docs: ${e.docs_url}`);
if (opts.transcriptPath && e.transcript_offset !== undefined) {
const snippet = readTranscriptAt(opts.transcriptPath, e.transcript_offset);
if (snippet) lines.push(` - transcript: \`${snippet}\``);
}
}
lines.push('');
}
}
return lines.join('\n');
}
/** Render a friction + delight summary as two columns. */
export function renderSummary(runId: string, opts: { format?: 'md' | 'json' } = {}): string {
const { entries } = readFriction(runId);
const friction = entries.filter(e => e.kind === 'friction');
const delight = entries.filter(e => e.kind === 'delight');
if (opts.format === 'json') {
return JSON.stringify({ run_id: runId, friction, delight }, null, 2);
}
const lines: string[] = [];
lines.push(`# ${runId}`);
lines.push('');
const max = Math.max(friction.length, delight.length);
lines.push(`| friction (${friction.length}) | delight (${delight.length}) |`);
lines.push('|---|---|');
for (let i = 0; i < max; i++) {
const l = friction[i] ? friction[i].message.replace(/\|/g, '\\|') : '';
const r = delight[i] ? delight[i].message.replace(/\|/g, '\\|') : '';
lines.push(`| ${l} | ${r} |`);
}
return lines.join('\n');
}
// ---------------------------------------------------------------------------
// Redaction
// ---------------------------------------------------------------------------
/** Replace homedir/cwd segments in user-visible string fields with placeholders. */
export function redactEntry(entry: FrictionEntry): FrictionEntry {
const home = homedir();
const cwd = entry.cwd;
const transform = (s: string | undefined): string | undefined => {
if (!s) return s;
let out = s;
if (cwd && cwd.length > 1) out = out.split(cwd).join('<CWD>');
if (home && home.length > 1) out = out.split(home).join('<HOME>');
return out;
};
return {
...entry,
message: transform(entry.message) ?? entry.message,
hint: transform(entry.hint),
cwd: '<CWD>',
};
}
// ---------------------------------------------------------------------------
// Transcript snippet resolution (for --transcripts)
// ---------------------------------------------------------------------------
function readTranscriptAt(path: string, offset: number): string | null {
try {
if (!existsSync(path)) return null;
const raw = readFileSync(path, 'utf-8');
if (offset < 0 || offset >= raw.length) return null;
// Find the line that contains this offset. Transcript is JSONL.
const lineStart = raw.lastIndexOf('\n', offset) + 1;
const lineEnd = raw.indexOf('\n', offset);
const line = raw.slice(lineStart, lineEnd === -1 ? undefined : lineEnd);
try {
const parsed = JSON.parse(line);
if (parsed && typeof parsed.bytes_b64 === 'string') {
const text = Buffer.from(parsed.bytes_b64, 'base64').toString('utf-8');
// Truncate snippet for readability
return text.replace(/\n/g, '\\n').slice(0, 200);
}
} catch { /* fall through */ }
return line.slice(0, 200);
} catch {
return null;
}
}
+410
View File
@@ -0,0 +1,410 @@
/**
* Frontmatter inference synthesize YAML frontmatter from filesystem metadata.
*
* ## Why this exists
*
* GBrain's sync and import pipelines work fine without frontmatter gray-matter
* returns the full content as body, and `inferType`/`inferTitle` in markdown.ts
* provide fallbacks. But the inferred metadata is minimal:
*
* - `type` defaults to 'concept' for most paths
* - `title` is the slugified filename ("2010 04 13 Apr 13 Founders Mtg")
* - No `date` field, no `source` metadata, no folder-aware tagging
*
* This module provides **rich inference** directory-aware type mapping, date
* extraction from filenames, title cleanup (strip date prefixes, HTML entities),
* heading extraction from content, and source/folder tagging. It produces a
* complete frontmatter block that can be:
*
* 1. Written back to the file on disk (via `gbrain frontmatter generate --fix`)
* 2. Used at import time without modifying the file (DB-only inference)
* 3. Shown as a dry-run preview (via `gbrain frontmatter generate --dry-run`)
*
* ## Design principles
*
* - **Never overwrite existing frontmatter.** If a file already has `---`, skip it.
* - **Infer from filesystem first, content second.** Directory path type, filename date + title,
* first `#` heading title fallback, content entity hints.
* - **Deterministic.** Same file always produces the same frontmatter. No LLM calls, no network.
* - **Extensible via rules.** The `DIRECTORY_RULES` table maps path patterns to type + source + tags.
* Adding a new directory convention = adding one rule.
* - **Safe.** `.bak` files on write, `--dry-run` by default in CLI, idempotent.
*
* ## How it fits in the pipeline
*
* ```
* Sync/Import
* file has frontmatter? normal import (existing path)
* file has NO frontmatter?
* inferFrontmatter(filePath, content) synthesize frontmatter
* prepend to content import as usual
* optionally write back to disk (--write-back flag)
* ```
*
* The inference runs BEFORE `parseMarkdown`, so the downstream pipeline sees
* well-formed frontmatter and all the existing validation/chunking/embedding
* logic works unchanged.
*
* ## Directory rules table
*
* Each rule matches a path pattern (case-insensitive prefix) and provides:
* - `type`: page type for the brain schema
* - `source`: optional source tag (e.g., "apple-notes", "therapy")
* - `tags`: optional additional tags
* - `datePattern`: where to look for dates 'filename' (YYYY-MM-DD prefix),
* 'dirname' (parent dir name), or 'none'
* - `titleStrategy`: how to extract title 'filename' (strip date prefix),
* 'heading' (first # in content), 'filename-full' (no date strip)
*/
import { basename, dirname, relative } from 'path';
// ─── Types ───────────────────────────────────────────────────────────
export interface InferredFrontmatter {
title: string;
type: string;
date?: string;
source?: string;
tags?: string[];
/** True if the file already has frontmatter (inference skipped). */
skipped?: boolean;
/** The rule that matched, for debugging. */
matchedRule?: string;
}
export interface DirectoryRule {
/** Case-insensitive path prefix to match (e.g., 'apple notes/'). */
pathPrefix: string;
/** Page type to assign. */
type: string;
/** Optional source tag. */
source?: string;
/** Optional tags to add. */
tags?: string[];
/** Where to look for dates. Default: 'filename'. */
datePattern?: 'filename' | 'dirname' | 'none';
/** How to extract title. Default: 'filename'. */
titleStrategy?: 'filename' | 'heading' | 'filename-full';
}
// ─── Directory Rules ─────────────────────────────────────────────────
// Ordered from most specific to least specific. First match wins.
// Add new directory conventions here.
export const DIRECTORY_RULES: DirectoryRule[] = [
// Apple Notes — bulk import from Apple Notes app. Filenames are
// "YYYY-MM-DD Title.md" with HTML-styled content.
{
pathPrefix: 'apple notes/youtube shows/',
type: 'apple-note',
source: 'apple-notes',
tags: ['youtube', 'shows'],
datePattern: 'filename',
titleStrategy: 'filename',
},
{
pathPrefix: 'apple notes/yc/',
type: 'apple-note',
source: 'apple-notes',
tags: ['yc'],
datePattern: 'filename',
titleStrategy: 'filename',
},
{
pathPrefix: 'apple notes/archived/',
type: 'apple-note',
source: 'apple-notes',
tags: ['archived'],
datePattern: 'filename',
titleStrategy: 'filename',
},
{
pathPrefix: 'apple notes/politics/',
type: 'apple-note',
source: 'apple-notes',
tags: ['politics'],
datePattern: 'filename',
titleStrategy: 'filename',
},
{
pathPrefix: 'apple notes/pitch notes/',
type: 'apple-note',
source: 'apple-notes',
tags: ['pitch-notes'],
datePattern: 'filename',
titleStrategy: 'filename',
},
{
pathPrefix: 'apple notes/gstack/',
type: 'apple-note',
source: 'apple-notes',
tags: ['gstack'],
datePattern: 'filename',
titleStrategy: 'filename',
},
{
pathPrefix: 'apple notes/photo-cameras/',
type: 'apple-note',
source: 'apple-notes',
tags: ['photography'],
datePattern: 'filename',
titleStrategy: 'filename',
},
{
pathPrefix: 'apple notes/jan bowman notes/',
type: 'apple-note',
source: 'apple-notes',
tags: ['therapy', 'jan-bowman'],
datePattern: 'filename',
titleStrategy: 'filename',
},
// Catch-all for Apple Notes not in a subfolder
{
pathPrefix: 'apple notes/',
type: 'apple-note',
source: 'apple-notes',
datePattern: 'filename',
titleStrategy: 'filename',
},
// Calendar diarization files
{
pathPrefix: 'daily/calendar/',
type: 'calendar-index',
source: 'calendar',
datePattern: 'filename',
titleStrategy: 'filename',
},
// Personal sections
{
pathPrefix: 'personal/therapy/',
type: 'therapy-session',
source: 'therapy',
datePattern: 'filename',
titleStrategy: 'filename',
},
{
pathPrefix: 'personal/reflections/',
type: 'reflection',
source: 'personal',
datePattern: 'filename',
titleStrategy: 'heading',
},
{
pathPrefix: 'personal/',
type: 'personal',
source: 'personal',
datePattern: 'none',
titleStrategy: 'heading',
},
// Writing
{
pathPrefix: 'writing/essays/',
type: 'essay',
source: 'writing',
datePattern: 'filename',
titleStrategy: 'heading',
},
{
pathPrefix: 'writing/ideas/',
type: 'idea',
source: 'writing',
datePattern: 'filename',
titleStrategy: 'heading',
},
{
pathPrefix: 'writing/',
type: 'writing',
source: 'writing',
datePattern: 'filename',
titleStrategy: 'heading',
},
// Entity directories — these should already have frontmatter in most cases,
// but the 55 people pages etc. that don't get handled here.
{ pathPrefix: 'people/', type: 'person', titleStrategy: 'heading' },
{ pathPrefix: 'companies/', type: 'company', titleStrategy: 'heading' },
{ pathPrefix: 'projects/', type: 'project', titleStrategy: 'heading' },
{ pathPrefix: 'civic/', type: 'civic', titleStrategy: 'heading' },
{ pathPrefix: 'events/', type: 'event', titleStrategy: 'heading', datePattern: 'filename' },
{ pathPrefix: 'meetings/', type: 'meeting', titleStrategy: 'heading', datePattern: 'filename' },
{ pathPrefix: 'media/', type: 'media', titleStrategy: 'heading' },
// Catch-all for any remaining files
{ pathPrefix: '', type: 'note', titleStrategy: 'heading' },
];
// ─── Date extraction ─────────────────────────────────────────────────
/** Extract YYYY-MM-DD date from a filename like "2010-04-13 Apr 13 founders mtg.md" */
export function extractDateFromFilename(filename: string): string | null {
// Pattern 1: YYYY-MM-DD prefix (with - or space separator after)
const m1 = filename.match(/^(\d{4}-\d{2}-\d{2})[\s_-]/);
if (m1) return m1[1];
// Pattern 2: YYYY-MM-DD anywhere in filename
const m2 = filename.match(/(\d{4}-\d{2}-\d{2})/);
if (m2) return m2[1];
// Pattern 3: "YYYY MM DD" with spaces
const m3 = filename.match(/^(\d{4})\s+(\d{2})\s+(\d{2})\s/);
if (m3) return `${m3[1]}-${m3[2]}-${m3[3]}`;
return null;
}
// ─── Title extraction ────────────────────────────────────────────────
/** Extract title from filename, stripping date prefix and extension. */
export function extractTitleFromFilename(filename: string): string {
// Remove .md extension
let title = filename.replace(/\.md$/i, '');
// Strip YYYY-MM-DD prefix (with separator)
title = title.replace(/^\d{4}-\d{2}-\d{2}[\s_-]+/, '');
// Strip YYYY MM DD prefix (space-separated)
title = title.replace(/^\d{4}\s+\d{2}\s+\d{2}\s+/, '');
// Clean up: title case, replace dashes/underscores with spaces
title = title
.replace(/[-_]/g, ' ')
.replace(/\s+/g, ' ')
.trim();
// Don't title-case if it already has mixed case (e.g., "YC presidency")
if (title === title.toLowerCase() || title === title.toUpperCase()) {
title = title.replace(/\b\w/g, c => c.toUpperCase());
}
return title || 'Untitled';
}
/** Extract title from first heading (# ...) in content. */
export function extractTitleFromHeading(content: string): string | null {
const lines = content.split('\n');
for (const line of lines.slice(0, 20)) {
const m = line.match(/^#\s+(.+)/);
if (m) return m[1].trim();
}
return null;
}
// ─── Core inference ──────────────────────────────────────────────────
/**
* Infer frontmatter for a file that has none.
*
* @param relativePath - Path relative to brain root (e.g., "Apple Notes/2010-04-13 Apr 13 founders mtg.md")
* @param content - File content (may be empty)
* @returns Inferred frontmatter fields
*/
export function inferFrontmatter(relativePath: string, content: string): InferredFrontmatter {
// Check if file already has frontmatter
const firstNonEmpty = content.split('\n').find(l => l.trim().length > 0);
if (firstNonEmpty?.trim() === '---') {
return { title: '', type: '', skipped: true };
}
const lowerPath = relativePath.toLowerCase();
const filename = basename(relativePath);
// Find matching rule
let matchedRule: DirectoryRule | undefined;
for (const rule of DIRECTORY_RULES) {
if (lowerPath.startsWith(rule.pathPrefix.toLowerCase())) {
matchedRule = rule;
break;
}
}
// Default rule if none matched
if (!matchedRule) {
matchedRule = { pathPrefix: '', type: 'note', titleStrategy: 'heading' };
}
// Extract date
let date: string | undefined;
const datePattern = matchedRule.datePattern ?? 'filename';
if (datePattern === 'filename') {
date = extractDateFromFilename(filename) ?? undefined;
}
// Extract title
let title: string;
const titleStrategy = matchedRule.titleStrategy ?? 'filename';
if (titleStrategy === 'heading') {
title = extractTitleFromHeading(content) ?? extractTitleFromFilename(filename);
} else if (titleStrategy === 'filename-full') {
title = filename.replace(/\.md$/i, '').replace(/[-_]/g, ' ').trim();
} else {
title = extractTitleFromFilename(filename);
}
// Build tags from rule + subfolder
const tags = [...(matchedRule.tags ?? [])];
// Add subfolder as tag for Apple Notes (e.g., "YC", "Politics")
if (matchedRule.source === 'apple-notes' && matchedRule.pathPrefix === 'apple notes/') {
const parts = relativePath.split('/');
if (parts.length > 2) {
const subfolder = parts[1].toLowerCase().replace(/\s+/g, '-');
if (!tags.includes(subfolder)) tags.push(subfolder);
}
}
return {
title,
type: matchedRule.type,
date,
source: matchedRule.source,
tags: tags.length > 0 ? tags : undefined,
matchedRule: matchedRule.pathPrefix || '(default)',
};
}
/**
* Generate a YAML frontmatter block from inferred fields.
* Returns the `---\n...\n---\n` string to prepend to content.
*/
export function serializeFrontmatter(fm: InferredFrontmatter): string {
if (fm.skipped) return '';
const lines: string[] = ['---'];
// Title — quote if it contains special YAML chars
const needsQuote = /[:"'#\[\]{}|>&*!?,]/.test(fm.title);
lines.push(`title: ${needsQuote ? JSON.stringify(fm.title) : fm.title}`);
lines.push(`type: ${fm.type}`);
if (fm.date) {
lines.push(`date: "${fm.date}"`);
}
if (fm.source) {
lines.push(`source: ${fm.source}`);
}
if (fm.tags && fm.tags.length > 0) {
lines.push(`tags: [${fm.tags.map(t => JSON.stringify(t)).join(', ')}]`);
}
lines.push('---');
return lines.join('\n') + '\n';
}
/**
* Apply frontmatter inference to file content.
* Returns the content with frontmatter prepended, or the original content if it already has frontmatter.
*/
export function applyInference(relativePath: string, content: string): { content: string; inferred: InferredFrontmatter } {
const inferred = inferFrontmatter(relativePath, content);
if (inferred.skipped) {
return { content, inferred };
}
const fm = serializeFrontmatter(inferred);
return { content: fm + '\n' + content, inferred };
}
+16 -2
View File
@@ -339,7 +339,7 @@ export async function importFromFile(
engine: BrainEngine,
filePath: string,
relativePath: string,
opts: { noEmbed?: boolean } = {},
opts: { noEmbed?: boolean; inferFrontmatter?: boolean } = {},
): Promise<ImportResult> {
// Defense-in-depth: reject symlinks before reading content.
const lstat = lstatSync(filePath);
@@ -352,13 +352,27 @@ export async function importFromFile(
return { slug: relativePath, status: 'skipped', chunks: 0, error: `File too large (${stat.size} bytes)` };
}
const content = readFileSync(filePath, 'utf-8');
let content = readFileSync(filePath, 'utf-8');
// Route code files through the code import path
if (isCodeFilePath(relativePath)) {
return importCodeFile(engine, relativePath, content, opts);
}
// v0.22.8 — Frontmatter inference: if the file has no frontmatter and
// inference is enabled, synthesize it from the filesystem path + content.
// This turns bare markdown files into fully-typed, dated, tagged pages
// without requiring the user to manually add YAML headers.
// The inference is applied to the in-memory content only; the file on disk
// is not modified. Use `gbrain frontmatter generate --fix` to write back.
if (opts.inferFrontmatter !== false) {
const { applyInference } = await import('./frontmatter-inference.ts');
const { content: inferred, inferred: meta } = applyInference(relativePath, content);
if (!meta.skipped) {
content = inferred;
}
}
const parsed = parseMarkdown(content, relativePath);
// Enforce path-authoritative slug. parseMarkdown prefers frontmatter.slug over
+8
View File
@@ -811,6 +811,14 @@ export const MIGRATIONS: Migration[] = [
RAISE NOTICE 'v24: RLS backfill complete (role % has BYPASSRLS)', current_user;
END $$;
`,
// PGLite has no RLS engine and is intrinsically single-tenant (local file).
// The 8 ALTER TABLE ... ENABLE ROW LEVEL SECURITY statements above also
// target tables that may not exist on PGLite (subagent_*, minion_inbox),
// since pglite-schema.ts is the canonical PGLite schema source. No-op
// override keeps PGLite upgrades unwedged and the version bump intact.
sqlFor: {
pglite: '',
},
},
{
version: 25,
+2 -2
View File
@@ -18,7 +18,7 @@
import * as fs from 'node:fs';
import * as path from 'node:path';
import * as os from 'node:os';
import { gbrainPath } from '../config.ts';
export interface BackpressureAuditEvent {
ts: string;
@@ -54,7 +54,7 @@ export function computeAuditFilename(now: Date = new Date()): string {
export function resolveAuditDir(): string {
const override = process.env.GBRAIN_AUDIT_DIR;
if (override && override.trim().length > 0) return override;
return path.join(os.homedir(), '.gbrain', 'audit');
return gbrainPath('audit');
}
export function logBackpressureCoalesce(event: Omit<BackpressureAuditEvent, 'ts' | 'decision'>): void {
+2 -2
View File
@@ -15,7 +15,7 @@
import * as fs from 'node:fs';
import * as path from 'node:path';
import * as os from 'node:os';
import { gbrainPath } from '../../config.ts';
export interface ShellAuditEvent {
ts: string;
@@ -53,7 +53,7 @@ export function computeAuditFilename(now: Date = new Date()): string {
export function resolveAuditDir(): string {
const override = process.env.GBRAIN_AUDIT_DIR;
if (override && override.trim().length > 0) return override;
return path.join(os.homedir(), '.gbrain', 'audit');
return gbrainPath('audit');
}
export function logShellSubmission(event: Omit<ShellAuditEvent, 'ts'>): void {
+11 -2
View File
@@ -225,8 +225,12 @@ export class MinionSupervisor {
process.on('SIGTERM', this.sigtermListener);
process.on('SIGINT', this.sigintListener);
// 4. Health monitoring.
this.healthTimer = setInterval(() => { void this.healthCheck(); }, this.opts.healthInterval);
// 4. Health monitoring. Skip when healthInterval=0 — that's the explicit
// "disable" contract documented on `--health-interval 0`. setInterval(0)
// would be a tight DB-hammering loop, not the no-op users expect.
if (this.opts.healthInterval > 0) {
this.healthTimer = setInterval(() => { void this.healthCheck(); }, this.opts.healthInterval);
}
// 5. Announce start.
this.emit('started', {
@@ -427,6 +431,11 @@ export class MinionSupervisor {
} else {
delete env.GBRAIN_ALLOW_SHELL_JOBS;
}
// Signal to the child worker that it's running under a supervisor.
// The worker's self-health-check (DB probes, stall detection) is
// redundant when the supervisor already provides these — setting
// this env var causes the worker to skip its own health timer.
env.GBRAIN_SUPERVISED = '1';
this.lastStartTime = Date.now();
+19
View File
@@ -170,6 +170,25 @@ export interface MinionWorkerOpts {
* case where all concurrency slots are wedged with zero job completions
* so the per-job check never fires. */
rssCheckInterval?: number;
/** Self-health-check interval in ms. 0 = disabled. Default: 60000 (1 minute).
* Automatically disabled when running under a supervisor (GBRAIN_SUPERVISED=1).
* Provides DB liveness probes and stall detection for bare `gbrain jobs work`
* deployments managed by external process managers (systemd, Docker, cron). */
healthCheckInterval?: number;
/** Stall detection: ms of continuous idle (waiting>0, inFlight=0, no completions)
* before emitting the first warning. Default: 300000 (5 minutes). */
stallWarnAfterMs?: number;
/** Stall detection: ms of continuous idle before emitting `'unhealthy'` with
* reason='stalled'. Default: 600000 (10 minutes). Must be > stallWarnAfterMs. */
stallExitAfterMs?: number;
/** DB liveness probe: number of consecutive failed `SELECT 1` probes before
* emitting `'unhealthy'` with reason='db_dead'. Default: 3. */
dbFailExitAfter?: number;
/** Per-probe wall-clock timeout in ms. A `SELECT 1` that hangs longer than
* this counts as a failure (fed into dbFailExitAfter). Without this, a
* hung probe would wedge the recursive setTimeout chain forever and
* silently disable the health monitor. Default: 10000 (10 seconds). */
dbProbeTimeoutMs?: number;
}
// --- Job Context (passed to handlers) ---
+209 -1
View File
@@ -20,8 +20,15 @@ import { UnrecoverableError } from './types.ts';
import { MinionQueue } from './queue.ts';
import { calculateBackoff } from './backoff.ts';
import { randomUUID } from 'crypto';
import { EventEmitter } from 'events';
import { evaluateQuietHours, type QuietHoursConfig } from './quiet-hours.ts';
/** Reason payload emitted with `'unhealthy'` when self-health-check trips.
* CLI layer (jobs.ts:work) subscribes and decides whether to call process.exit. */
export type UnhealthyReason =
| { reason: 'db_dead'; consecutiveFailures: number; message: string }
| { reason: 'stalled'; waitingCount: number; idleMinutes: number };
/**
* Read the quiet_hours JSONB column off a MinionJob, if present. The
* column was added in schema migration v12; older rows + versions of
@@ -42,7 +49,13 @@ interface InFlightJob {
promise: Promise<void>;
}
export class MinionWorker {
/** Type-safe `on('unhealthy', ...)` for callers. */
export interface MinionWorker {
on(event: 'unhealthy', listener: (info: UnhealthyReason) => void): this;
emit(event: 'unhealthy', info: UnhealthyReason): boolean;
}
export class MinionWorker extends EventEmitter {
private queue: MinionQueue;
private handlers = new Map<string, MinionHandler>();
private running = false;
@@ -67,6 +80,7 @@ export class MinionWorker {
private engine: BrainEngine,
opts?: MinionWorkerOpts & MinionQueueOpts,
) {
super();
this.queue = new MinionQueue(engine, {
maxSpawnDepth: opts?.maxSpawnDepth,
maxAttachmentBytes: opts?.maxAttachmentBytes,
@@ -81,7 +95,25 @@ export class MinionWorker {
maxRssMb: opts?.maxRssMb ?? 0,
getRss: opts?.getRss ?? (() => process.memoryUsage().rss),
rssCheckInterval: opts?.rssCheckInterval ?? 60000,
healthCheckInterval: opts?.healthCheckInterval ?? 60000,
stallWarnAfterMs: opts?.stallWarnAfterMs ?? 5 * 60_000,
stallExitAfterMs: opts?.stallExitAfterMs ?? 10 * 60_000,
dbFailExitAfter: opts?.dbFailExitAfter ?? 3,
dbProbeTimeoutMs: opts?.dbProbeTimeoutMs ?? 10_000,
};
// Stall thresholds contract: exit MUST be strictly greater than warn.
// If exit <= warn, the warn-then-exit semantics break: a single tick at
// idle > warn would set stallWarningSince and the subsequent tick at
// idle > exit could fire immediately without giving operators visibility.
// Reject misconfigurations at construction time so the failure mode is
// a loud throw on startup rather than a quiet contract violation.
if (this.opts.stallExitAfterMs <= this.opts.stallWarnAfterMs) {
throw new Error(
`MinionWorkerOpts: stallExitAfterMs (${this.opts.stallExitAfterMs}) must be > ` +
`stallWarnAfterMs (${this.opts.stallWarnAfterMs}). ` +
`The contract is "warn first, exit later" — they cannot fire on the same tick.`,
);
}
}
/** Register a handler for a job type. */
@@ -94,6 +126,28 @@ export class MinionWorker {
return Array.from(this.handlers.keys());
}
/** Emit 'unhealthy' with a no-listener fallback. The default contract is
* fail-stop: pre-EventEmitter-refactor behavior was process.exit(1) inside
* the timer; the refactor moved that responsibility to the CLI subscriber.
* But direct API consumers without a listener would see emit() become a
* no-op AND `healthExited=true` permanently disabling monitoring a
* silent regression. Solution: if no one subscribed, log and exit
* ourselves so the worker dies and the PM restarts it. Subscribers
* override this default by adding a listener before start(). */
private emitUnhealthy(info: UnhealthyReason): void {
if (this.listenerCount('unhealthy') === 0) {
const detail = info.reason === 'db_dead'
? `DB unreachable (${info.consecutiveFailures} probes): ${info.message}`
: `worker stalled (${info.waitingCount} waiting, ${info.idleMinutes}m idle)`;
console.error(
`[health] FATAL: ${detail}. No 'unhealthy' listener registered; ` +
`defaulting to process.exit(1) for process-manager restart.`,
);
process.exit(1);
}
this.emit('unhealthy', info);
}
/** Start the worker loop. Blocks until stopped. */
async start(): Promise<void> {
if (this.handlers.size === 0) {
@@ -155,6 +209,159 @@ export class MinionWorker {
}, this.opts.rssCheckInterval);
}
// Self-health-check — provides supervisor-grade monitoring for bare workers.
// Disabled when running under a supervisor (GBRAIN_SUPERVISED=1) or when
// healthCheckInterval is 0. Catches two failure modes that leave the process
// alive but non-functional:
// 1. DB connection death (Supabase/PgBouncer drops, network blip)
// 2. Worker stall (event loop alive but not claiming/completing jobs)
//
// On failure, emits an `'unhealthy'` event with a structured reason. The
// CLI layer (`src/commands/jobs.ts:work`) subscribes and decides whether to
// call process.exit. Library code never calls process.exit directly so
// MinionWorker stays embeddable in non-CLI contexts (tests, other hosts).
//
// Timer pattern: recursive setTimeout with a `running` flag, not setInterval.
// setInterval queues callbacks even when the prior is still awaiting; on a
// hung DB probe that piles up overlapping async checks racing on
// `consecutiveDbFailures`. The recursive pattern guarantees one tick at a time.
const isSupervisedChild = process.env.GBRAIN_SUPERVISED === '1';
let healthTimer: ReturnType<typeof setTimeout> | null = null;
if (!isSupervisedChild && this.opts.healthCheckInterval > 0) {
let consecutiveDbFailures = 0;
let lastKnownCompleted = this.jobsCompleted;
let lastCompletionTime = Date.now();
let stallWarningSince: number | null = null;
let healthRunning = false;
let healthExited = false;
// Race executeRaw against a wall-clock deadline. A hung connection
// (network-partitioned PgBouncer, deadlocked backend) would otherwise
// hold the await forever — the recursive setTimeout's next tick is only
// scheduled in `finally`, so a hung probe would silently disable the
// entire health monitor. The timeout treats hangs as failures and feeds
// them into `dbFailExitAfter`.
const probeWithTimeout = async (): Promise<void> => {
const ac = new AbortController();
const timeoutMs = this.opts.dbProbeTimeoutMs;
const timer = setTimeout(() => ac.abort(), timeoutMs);
try {
await Promise.race([
this.engine.executeRaw('SELECT 1'),
new Promise<never>((_, reject) => {
ac.signal.addEventListener('abort', () => {
reject(new Error(`probe timeout after ${timeoutMs}ms`));
});
}),
]);
} finally {
clearTimeout(timer);
}
};
const runHealthCheck = async (): Promise<void> => {
if (healthRunning || !this.running || healthExited) return;
healthRunning = true;
try {
// --- 1. DB liveness probe ---
try {
await probeWithTimeout();
consecutiveDbFailures = 0;
} catch (e) {
consecutiveDbFailures++;
const msg = e instanceof Error ? e.message : String(e);
console.error(
`[health] DB probe failed (${consecutiveDbFailures}/${this.opts.dbFailExitAfter}): ${msg}`,
);
if (consecutiveDbFailures >= this.opts.dbFailExitAfter) {
console.error(
`[health] DB unreachable after ${this.opts.dbFailExitAfter} consecutive probes. ` +
`Emitting 'unhealthy' for process-manager restart.`,
);
healthExited = true;
this.emitUnhealthy({
reason: 'db_dead',
consecutiveFailures: consecutiveDbFailures,
message: msg,
});
}
return; // Skip stall check when DB is flaky
}
// --- 2. Stall detection ---
if (this.jobsCompleted > lastKnownCompleted) {
lastKnownCompleted = this.jobsCompleted;
lastCompletionTime = Date.now();
stallWarningSince = null;
}
const idleMs = Date.now() - lastCompletionTime;
// Only check for stalls when no jobs are in-flight and it's been a while
if (idleMs > this.opts.stallWarnAfterMs && this.inFlight.size === 0) {
try {
// Filter by registered handler names so a worker that doesn't
// claim a particular job-name doesn't false-positive when those
// jobs accumulate in `waiting`. Only counts work THIS worker would
// actually have claimed.
const handlerNames = this.registeredNames;
const rows = handlerNames.length === 0
? [] as { cnt: string }[]
: await this.engine.executeRaw<{ cnt: string }>(
`SELECT count(*)::text AS cnt FROM minion_jobs
WHERE status = 'waiting'
AND queue = $1
AND name = ANY($2::text[])`,
[this.opts.queue, handlerNames],
);
const waiting = parseInt(rows[0]?.cnt ?? '0', 10);
const idleMinutes = Math.round(idleMs / 60_000);
if (waiting > 0) {
// Two thresholds, both measured from `lastCompletionTime` (NOT
// from when the warning fired). With defaults (warn=5min,
// exit=10min), the first warning fires at idle=5min and the
// unhealthy emit fires at idle=10min — matching the contract
// documented in MinionWorkerOpts.
if (!stallWarningSince) {
stallWarningSince = Date.now();
console.warn(
`[health] Possible stall: ${waiting} waiting job(s) for ` +
`registered handlers, 0 in-flight, ${idleMinutes}m since last completion`,
);
} else if (idleMs > this.opts.stallExitAfterMs) {
console.error(
`[health] Worker stalled for ${Math.round(this.opts.stallExitAfterMs / 60_000)}+ ` +
`minutes with ${waiting} waiting job(s). Emitting 'unhealthy' for process-manager restart.`,
);
healthExited = true;
this.emitUnhealthy({
reason: 'stalled',
waitingCount: waiting,
idleMinutes,
});
}
} else {
stallWarningSince = null; // Queue empty (for our handlers) — not stalled, just idle
}
} catch {
// DB query failed — the liveness probe above will catch persistent failures
}
} else {
stallWarningSince = null;
}
} finally {
healthRunning = false;
if (this.running && !healthExited) {
healthTimer = setTimeout(runHealthCheck, this.opts.healthCheckInterval);
}
}
};
// First tick scheduled after one interval so newly-started workers have
// a chance to do real work before the stall clock starts ticking.
healthTimer = setTimeout(runHealthCheck, this.opts.healthCheckInterval);
}
try {
while (this.running) {
// Promote delayed jobs
@@ -201,6 +408,7 @@ export class MinionWorker {
} finally {
clearInterval(stalledTimer);
if (rssTimer) clearInterval(rssTimer);
if (healthTimer) clearTimeout(healthTimer); // recursive setTimeout pattern
process.removeListener('SIGTERM', shutdown);
process.removeListener('SIGINT', shutdown);
+6 -5
View File
@@ -18,8 +18,8 @@
*/
import { appendFileSync, existsSync, mkdirSync } from 'fs';
import { homedir } from 'os';
import { dirname, join } from 'path';
import { dirname } from 'path';
import { gbrainPath } from '../config.ts';
import type { BrainEngine } from '../engine.ts';
import {
@@ -30,7 +30,7 @@ import {
} from './validators/index.ts';
import type { ValidationFinding, PageValidator } from './writer.ts';
const LINT_LOG_FILE = join(homedir(), '.gbrain', 'validator-lint.jsonl');
const getLintLogFile = () => gbrainPath('validator-lint.jsonl');
const LINT_CONFIG_KEY = 'writer.lint_on_put_page';
export interface PostWriteLintOpts {
@@ -124,7 +124,8 @@ export async function runPostWriteLint(
function writeLocalLintLog(slug: string, findings: ValidationFinding[]): void {
try {
const dir = dirname(LINT_LOG_FILE);
const lintLogFile = getLintLogFile();
const dir = dirname(lintLogFile);
if (!existsSync(dir)) mkdirSync(dir, { recursive: true });
const line = JSON.stringify({
ts: new Date().toISOString(),
@@ -133,7 +134,7 @@ function writeLocalLintLog(slug: string, findings: ValidationFinding[]): void {
warning_count: findings.filter(f => f.severity === 'warning').length,
findings: findings.slice(0, 20), // cap to prevent runaway log size
}) + '\n';
appendFileSync(LINT_LOG_FILE, line, 'utf-8');
appendFileSync(lintLogFile, line, 'utf-8');
} catch {
// Non-fatal; logging failure shouldn't break the main flow.
}
+122
View File
@@ -86,6 +86,16 @@ export class PGLiteEngine implements BrainEngine {
}
async initSchema(): Promise<void> {
// Pre-schema bootstrap: add forward-referenced state the embedded schema
// blob requires but that older brains don't have yet. Without this, a
// pre-v0.18 brain hits `CREATE INDEX idx_pages_source_id ON pages(source_id)`
// (issues #366/#375/#378/#396) or a pre-v0.13 brain hits
// `CREATE INDEX idx_links_source ON links(link_source)` (#266/#357), and
// initSchema crashes before runMigrations gets a chance to apply the
// missing column. Bootstrap is structurally idempotent and a no-op on
// fresh installs and modern brains.
await this.applyForwardReferenceBootstrap();
await this.db.exec(PGLITE_SCHEMA_SQL);
const { applied } = await runMigrations(this);
@@ -94,6 +104,111 @@ export class PGLiteEngine implements BrainEngine {
}
}
/**
* Bootstrap state that PGLITE_SCHEMA_SQL forward-references but that older
* brains don't have yet. Currently covers:
*
* - `sources` table + default seed (FK target of pages.source_id) v0.18
* - `pages.source_id` column (indexed by `idx_pages_source_id`) v0.18
* - `links.link_source` column (indexed by `idx_links_source`) v0.13
* - `links.origin_page_id` column (indexed by `idx_links_origin`) v0.13
* - `content_chunks.symbol_name` column (indexed by `idx_chunks_symbol_name`) v0.19
* - `content_chunks.language` column (indexed by `idx_chunks_language`) v0.19
*
* **Maintenance contract:** when a future migration adds a column-with-index
* or new-table-with-FK referenced by PGLITE_SCHEMA_SQL, extend this method
* AND `test/schema-bootstrap-coverage.test.ts`'s `REQUIRED_BOOTSTRAP_COVERAGE`.
* The coverage test fails loudly if the bootstrap drifts behind the schema.
*/
private async applyForwardReferenceBootstrap(): Promise<void> {
// Single round-trip probe for every forward-reference target.
const { rows } = await this.db.query(`
SELECT
EXISTS (SELECT 1 FROM information_schema.tables
WHERE table_schema='public' AND table_name='pages') AS pages_exists,
EXISTS (SELECT 1 FROM information_schema.columns
WHERE table_schema='public' AND table_name='pages' AND column_name='source_id') AS source_id_exists,
EXISTS (SELECT 1 FROM information_schema.tables
WHERE table_schema='public' AND table_name='links') AS links_exists,
EXISTS (SELECT 1 FROM information_schema.columns
WHERE table_schema='public' AND table_name='links' AND column_name='link_source') AS link_source_exists,
EXISTS (SELECT 1 FROM information_schema.columns
WHERE table_schema='public' AND table_name='links' AND column_name='origin_page_id') AS origin_page_id_exists,
EXISTS (SELECT 1 FROM information_schema.tables
WHERE table_schema='public' AND table_name='content_chunks') AS chunks_exists,
EXISTS (SELECT 1 FROM information_schema.columns
WHERE table_schema='public' AND table_name='content_chunks' AND column_name='symbol_name') AS symbol_name_exists,
EXISTS (SELECT 1 FROM information_schema.columns
WHERE table_schema='public' AND table_name='content_chunks' AND column_name='language') AS language_exists
`);
const probe = rows[0] as {
pages_exists: boolean;
source_id_exists: boolean;
links_exists: boolean;
link_source_exists: boolean;
origin_page_id_exists: boolean;
chunks_exists: boolean;
symbol_name_exists: boolean;
language_exists: boolean;
};
const needsPagesBootstrap = probe.pages_exists && !probe.source_id_exists;
const needsLinksBootstrap = probe.links_exists
&& (!probe.link_source_exists || !probe.origin_page_id_exists);
const needsChunksBootstrap = probe.chunks_exists
&& (!probe.symbol_name_exists || !probe.language_exists);
// Fresh installs (no tables yet) and modern brains both no-op.
if (!needsPagesBootstrap && !needsLinksBootstrap && !needsChunksBootstrap) return;
console.log(' Pre-v0.21 brain detected, applying forward-reference bootstrap');
if (needsPagesBootstrap) {
// Mirror schema-embedded.ts shape for `sources` so the subsequent
// PGLITE_SCHEMA_SQL CREATE TABLE IF NOT EXISTS is a true no-op.
await this.db.exec(`
CREATE TABLE IF NOT EXISTS sources (
id TEXT PRIMARY KEY,
name TEXT NOT NULL UNIQUE,
local_path TEXT,
last_commit TEXT,
last_sync_at TIMESTAMPTZ,
config JSONB NOT NULL DEFAULT '{}'::jsonb,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
INSERT INTO sources (id, name, config)
VALUES ('default', 'default', '{"federated": true}'::jsonb)
ON CONFLICT (id) DO NOTHING;
ALTER TABLE pages ADD COLUMN IF NOT EXISTS source_id TEXT
NOT NULL DEFAULT 'default' REFERENCES sources(id) ON DELETE CASCADE;
`);
}
if (needsLinksBootstrap) {
// v11 (links_provenance_columns) is responsible for the CHECK constraint
// and backfill. The bootstrap only adds enough state for SCHEMA_SQL's
// `CREATE INDEX idx_links_source/origin` not to crash. v11 runs later
// via runMigrations and is idempotent (`IF NOT EXISTS` everywhere).
await this.db.exec(`
ALTER TABLE links ADD COLUMN IF NOT EXISTS link_source TEXT;
ALTER TABLE links ADD COLUMN IF NOT EXISTS origin_page_id INTEGER
REFERENCES pages(id) ON DELETE SET NULL;
`);
}
if (needsChunksBootstrap) {
// v26 (content_chunks_code_metadata) adds the full code-chunk metadata
// surface (language, symbol_name, symbol_type, start_line, end_line).
// The bootstrap only adds the two columns the schema blob's partial
// indexes reference (idx_chunks_symbol_name, idx_chunks_language).
// v26 runs later via runMigrations and adds the rest idempotently.
await this.db.exec(`
ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS language TEXT;
ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS symbol_name TEXT;
`);
}
}
async withReservedConnection<T>(fn: (conn: ReservedConnection) => Promise<T>): Promise<T> {
// PGLite has no connection pool. The single backing connection is
// always effectively reserved — pass it through.
@@ -179,6 +294,13 @@ export class PGLiteEngine implements BrainEngine {
params.push(filters.updated_after);
where.push(`p.updated_at > $${params.length}::timestamptz`);
}
// slugPrefix uses the (source_id, slug) UNIQUE btree for index range scans.
// Escape LIKE metacharacters so the user prefix is treated as a literal.
if (filters?.slugPrefix) {
const escaped = filters.slugPrefix.replace(/[\\%_]/g, (c) => '\\' + c) + '%';
params.push(escaped);
where.push(`p.slug LIKE $${params.length} ESCAPE '\\'`);
}
const whereSql = where.length > 0 ? `WHERE ${where.join(' AND ')}` : '';
params.push(limit, offset);
+141 -2
View File
@@ -3,6 +3,7 @@ import type { BrainEngine, LinkBatchInput, TimelineBatchInput, ReservedConnectio
import { MAX_SEARCH_LIMIT, clampSearchLimit } from './engine.ts';
import { runMigrations } from './migrate.ts';
import { SCHEMA_SQL } from './schema-embedded.ts';
import { verifySchema } from './schema-verify.ts';
import type {
Page, PageInput, PageFilters, PageType,
Chunk, ChunkInput, StaleChunkRow,
@@ -98,9 +99,26 @@ export class PostgresEngine implements BrainEngine {
async initSchema(): Promise<void> {
const conn = this.sql;
// Advisory lock prevents concurrent initSchema() calls from deadlocking
// on DDL statements (DROP TRIGGER + CREATE TRIGGER acquire AccessExclusiveLock)
// on DDL statements (DROP TRIGGER + CREATE TRIGGER acquire AccessExclusiveLock).
//
// Honest limitation: pg_advisory_lock(42) is session-scoped to this pooled
// connection. runMigrations() below uses engine.transaction() and
// withReservedConnection() which may hop to a different backend in the
// pool. Cross-process serialization of initSchema is best-effort, not a
// correctness guarantee. Pre-existing concern; the bootstrap doesn't
// change it.
await conn`SELECT pg_advisory_lock(42)`;
try {
// Pre-schema bootstrap: add forward-referenced state the embedded schema
// blob requires but that older brains don't have yet. Without this, a
// pre-v0.18 brain hits `CREATE INDEX idx_pages_source_id ON pages(source_id)`
// (issues #366/#375/#378/#396), or a pre-v0.13 brain hits
// `CREATE INDEX idx_links_source ON links(link_source)` (#266/#357), and
// SCHEMA_SQL crashes before runMigrations gets a chance to apply the
// missing column. Bootstrap is structurally idempotent and a no-op on
// fresh installs and modern brains.
await this.applyForwardReferenceBootstrap();
await conn.unsafe(SCHEMA_SQL);
// Run any pending migrations automatically
@@ -108,11 +126,126 @@ export class PostgresEngine implements BrainEngine {
if (applied > 0) {
console.log(` ${applied} migration(s) applied`);
}
// Post-migration schema verification: catches columns that migrations
// defined but PgBouncer transaction-mode silently failed to create.
// Self-heals missing columns via ALTER TABLE ADD COLUMN IF NOT EXISTS.
const verify = await verifySchema(this);
if (verify.healed.length > 0) {
console.log(` Schema verify: self-healed ${verify.healed.length} missing column(s)`);
}
} finally {
await conn`SELECT pg_advisory_unlock(42)`;
}
}
/**
* Bootstrap state that SCHEMA_SQL forward-references but that older brains
* don't have yet. Mirror of `PGLiteEngine#applyForwardReferenceBootstrap`
* in shape and intent. Currently covers:
*
* - `sources` table + default seed (FK target of pages.source_id) v0.18
* - `pages.source_id` column (indexed by `idx_pages_source_id`) v0.18
* - `links.link_source` column (indexed by `idx_links_source`) v0.13
* - `links.origin_page_id` column (indexed by `idx_links_origin`) v0.13
* - `content_chunks.symbol_name` column (indexed by `idx_chunks_symbol_name`) v0.19
* - `content_chunks.language` column (indexed by `idx_chunks_language`) v0.19
*
* Keep this in sync with the PGLite version; covered by
* `test/schema-bootstrap-coverage.test.ts` (PGLite side) and
* `test/e2e/postgres-bootstrap.test.ts` (Postgres side).
*/
private async applyForwardReferenceBootstrap(): Promise<void> {
const conn = this.sql;
// Single round-trip probe for every forward-reference target.
// current_schema() resolves to whatever search_path the connection uses,
// which matches schema-embedded.ts's `public.` references.
const probeRows = await conn<{
pages_exists: boolean;
source_id_exists: boolean;
links_exists: boolean;
link_source_exists: boolean;
origin_page_id_exists: boolean;
chunks_exists: boolean;
symbol_name_exists: boolean;
language_exists: boolean;
}[]>`
SELECT
EXISTS (SELECT 1 FROM information_schema.tables
WHERE table_schema = current_schema() AND table_name = 'pages') AS pages_exists,
EXISTS (SELECT 1 FROM information_schema.columns
WHERE table_schema = current_schema() AND table_name = 'pages' AND column_name = 'source_id') AS source_id_exists,
EXISTS (SELECT 1 FROM information_schema.tables
WHERE table_schema = current_schema() AND table_name = 'links') AS links_exists,
EXISTS (SELECT 1 FROM information_schema.columns
WHERE table_schema = current_schema() AND table_name = 'links' AND column_name = 'link_source') AS link_source_exists,
EXISTS (SELECT 1 FROM information_schema.columns
WHERE table_schema = current_schema() AND table_name = 'links' AND column_name = 'origin_page_id') AS origin_page_id_exists,
EXISTS (SELECT 1 FROM information_schema.tables
WHERE table_schema = current_schema() AND table_name = 'content_chunks') AS chunks_exists,
EXISTS (SELECT 1 FROM information_schema.columns
WHERE table_schema = current_schema() AND table_name = 'content_chunks' AND column_name = 'symbol_name') AS symbol_name_exists,
EXISTS (SELECT 1 FROM information_schema.columns
WHERE table_schema = current_schema() AND table_name = 'content_chunks' AND column_name = 'language') AS language_exists
`;
const probe = probeRows[0]!;
const needsPagesBootstrap = probe.pages_exists && !probe.source_id_exists;
const needsLinksBootstrap = probe.links_exists
&& (!probe.link_source_exists || !probe.origin_page_id_exists);
const needsChunksBootstrap = probe.chunks_exists
&& (!probe.symbol_name_exists || !probe.language_exists);
if (!needsPagesBootstrap && !needsLinksBootstrap && !needsChunksBootstrap) return;
console.log(' Pre-v0.21 brain detected, applying forward-reference bootstrap');
if (needsPagesBootstrap) {
// Mirror schema-embedded.ts's `sources` shape so the subsequent
// SCHEMA_SQL CREATE TABLE IF NOT EXISTS is a true no-op.
await conn.unsafe(`
CREATE TABLE IF NOT EXISTS sources (
id TEXT PRIMARY KEY,
name TEXT NOT NULL UNIQUE,
local_path TEXT,
last_commit TEXT,
last_sync_at TIMESTAMPTZ,
config JSONB NOT NULL DEFAULT '{}'::jsonb,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
INSERT INTO sources (id, name, config)
VALUES ('default', 'default', '{"federated": true}'::jsonb)
ON CONFLICT (id) DO NOTHING;
ALTER TABLE pages ADD COLUMN IF NOT EXISTS source_id TEXT
NOT NULL DEFAULT 'default' REFERENCES sources(id) ON DELETE CASCADE;
`);
}
if (needsLinksBootstrap) {
// v11 (links_provenance_columns) handles the CHECK constraint, the
// UNIQUE swap, and the backfill. The bootstrap only adds enough state
// for SCHEMA_SQL's `CREATE INDEX idx_links_source/origin` not to crash.
// v11 runs later via runMigrations and is idempotent.
await conn.unsafe(`
ALTER TABLE links ADD COLUMN IF NOT EXISTS link_source TEXT;
ALTER TABLE links ADD COLUMN IF NOT EXISTS origin_page_id INTEGER
REFERENCES pages(id) ON DELETE SET NULL;
`);
}
if (needsChunksBootstrap) {
// v26 (content_chunks_code_metadata) adds the full code-chunk metadata
// surface. The bootstrap only adds the two columns the schema blob's
// partial indexes reference (idx_chunks_symbol_name, idx_chunks_language).
// v26 runs later via runMigrations and adds the rest idempotently.
await conn.unsafe(`
ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS language TEXT;
ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS symbol_name TEXT;
`);
}
}
async transaction<T>(fn: (engine: BrainEngine) => Promise<T>): Promise<T> {
const conn = this._sql || db.getConnection();
return conn.begin(async (tx) => {
@@ -201,11 +334,17 @@ export class PostgresEngine implements BrainEngine {
const tagJoin = filters?.tag ? sql`JOIN tags t ON t.page_id = p.id` : sql``;
const tagCondition = filters?.tag ? sql`AND t.tag = ${filters.tag}` : sql``;
const updatedCondition = updatedAfter ? sql`AND p.updated_at > ${updatedAfter}::timestamptz` : sql``;
// slugPrefix uses the (source_id, slug) UNIQUE btree index for range scans.
// Escape LIKE metacharacters so the user prefix is treated as a literal.
const slugPrefix = filters?.slugPrefix;
const slugCondition = slugPrefix
? sql`AND p.slug LIKE ${slugPrefix.replace(/[\\%_]/g, (c) => '\\' + c) + '%'} ESCAPE '\\'`
: sql``;
const rows = await sql`
SELECT p.* FROM pages p
${tagJoin}
WHERE 1=1 ${typeCondition} ${tagCondition} ${updatedCondition}
WHERE 1=1 ${typeCondition} ${tagCondition} ${updatedCondition} ${slugCondition}
ORDER BY p.updated_at DESC LIMIT ${limit} OFFSET ${offset}
`;
+282
View File
@@ -0,0 +1,282 @@
/**
* Post-migration schema verification with self-healing.
*
* PgBouncer transaction-mode poolers can silently swallow ALTER TABLE
* statements: the SQL doesn't error, but the column never gets created.
* The migration system increments the schema version counter anyway, so
* gbrain thinks it's on v29 but the actual table is missing columns.
*
* This module parses the canonical CREATE TABLE definitions in
* schema-embedded.ts and diffs them against information_schema.columns.
* Missing columns are self-healed via ALTER TABLE ADD COLUMN IF NOT EXISTS.
*
* Called at the end of initSchema(), after all migrations complete.
*/
import { SCHEMA_SQL } from './schema-embedded.ts';
import type { BrainEngine } from './engine.ts';
/** A column expected to exist in the database. */
export interface ExpectedColumn {
table: string;
column: string;
/** The full column definition (type + constraints) from the CREATE TABLE. */
definition: string;
}
/**
* Parse CREATE TABLE statements from SCHEMA_SQL to extract expected columns.
*
* This is a best-effort parser that handles the gbrain schema conventions:
* - Standard column definitions with types and constraints
* - Skips CONSTRAINT lines, CHECK lines, and UNIQUE lines
* - Handles multi-line definitions
*
* Returns only tables and columns not constraints, indexes, or triggers.
*/
export function parseExpectedColumns(): ExpectedColumn[] {
const results: ExpectedColumn[] = [];
// Match CREATE TABLE IF NOT EXISTS <name> ( ... );
const tableRegex = /CREATE\s+TABLE\s+IF\s+NOT\s+EXISTS\s+(\w+)\s*\(([\s\S]*?)\);/gi;
const SQL_KEYWORDS = new Set(['constraint', 'unique', 'check', 'primary', 'foreign', 'exclude']);
function processLine(tableName: string, line: string) {
line = line.trim().replace(/,\s*$/, '');
if (!line) return;
// Skip CONSTRAINT, UNIQUE, CHECK, PRIMARY KEY lines
if (/^\s*(CONSTRAINT|UNIQUE|CHECK|PRIMARY\s+KEY)/i.test(line)) return;
const colMatch = line.match(/^\s*(\w+)\s+(.+)$/);
if (colMatch) {
const colName = colMatch[1].toLowerCase();
if (SQL_KEYWORDS.has(colName)) return;
results.push({
table: tableName,
column: colName,
definition: colMatch[2].trim(),
});
}
}
let match: RegExpExecArray | null;
while ((match = tableRegex.exec(SCHEMA_SQL)) !== null) {
const tableName = match[1];
const body = match[2];
const lines = body.split('\n');
let currentLine = '';
for (const rawLine of lines) {
const trimmed = rawLine.trim();
// Skip empty lines and comments
if (!trimmed || trimmed.startsWith('--')) {
// If we have accumulated content and hit a blank/comment line,
// the accumulated content is a complete line
if (currentLine.trim()) {
processLine(tableName, currentLine);
currentLine = '';
}
continue;
}
currentLine += ' ' + trimmed;
// If line ends with comma, it's a complete column definition
if (trimmed.endsWith(',')) {
processLine(tableName, currentLine);
currentLine = '';
}
}
// Handle any remaining accumulated line (last column before closing paren)
if (currentLine.trim()) {
processLine(tableName, currentLine);
}
}
// Also parse ALTER TABLE ... ADD COLUMN IF NOT EXISTS statements.
// These are used for columns added outside CREATE TABLE blocks
// (e.g., pages.search_vector, files.source_id).
const alterRegex = /ALTER\s+TABLE\s+(\w+)\s+ADD\s+COLUMN\s+IF\s+NOT\s+EXISTS\s+(\w+)\s+([^;,]+)/gi;
let alterMatch: RegExpExecArray | null;
const seen = new Set(results.map(r => `${r.table}.${r.column}`));
while ((alterMatch = alterRegex.exec(SCHEMA_SQL)) !== null) {
const table = alterMatch[1];
const column = alterMatch[2].toLowerCase();
const definition = alterMatch[3].trim().replace(/,\s*$/, '');
const key = `${table}.${column}`;
if (!seen.has(key)) {
seen.add(key);
results.push({ table, column, definition });
}
}
return results;
}
/**
* Build a simplified type expression suitable for ALTER TABLE ADD COLUMN.
*
* Strips inline REFERENCES, CHECK, UNIQUE, and complex constraints that
* can't be used in ADD COLUMN IF NOT EXISTS. Preserves NOT NULL, DEFAULT,
* and the base type.
*/
export function simplifyColumnDef(definition: string): string {
let def = definition;
// Remove REFERENCES ... (with optional ON DELETE/UPDATE clauses)
def = def.replace(/REFERENCES\s+\w+\([^)]*\)(\s+ON\s+(DELETE|UPDATE)\s+\w+(\s+\w+)?)*\s*/gi, '');
// Remove CHECK constraints (handle nested parens)
def = def.replace(/CHECK\s*\((?:[^()]*|\([^()]*\))*\)/gi, '');
// Remove inline UNIQUE
def = def.replace(/\bUNIQUE\b/gi, '');
// Remove trailing commas and whitespace
def = def.replace(/,\s*$/, '').trim();
// Collapse multiple spaces
def = def.replace(/\s+/g, ' ').trim();
return def;
}
/**
* Query the database for actual columns in the public schema.
* Returns a Set of "table.column" strings for fast lookup.
*/
async function getActualColumns(engine: BrainEngine): Promise<Set<string>> {
const rows = await engine.executeRaw<{ table_name: string; column_name: string }>(
`SELECT table_name, column_name
FROM information_schema.columns
WHERE table_schema = 'public'`
);
const set = new Set<string>();
for (const row of rows) {
set.add(`${row.table_name}.${row.column_name}`);
}
return set;
}
/**
* Get the set of tables that actually exist in the database.
*/
async function getActualTables(engine: BrainEngine): Promise<Set<string>> {
const rows = await engine.executeRaw<{ table_name: string }>(
`SELECT table_name
FROM information_schema.tables
WHERE table_schema = 'public' AND table_type = 'BASE TABLE'`
);
return new Set(rows.map(r => r.table_name));
}
export interface VerifyResult {
/** Total columns checked */
checked: number;
/** Columns that were missing */
missing: Array<{ table: string; column: string }>;
/** Columns successfully self-healed */
healed: Array<{ table: string; column: string }>;
/** Columns that failed to self-heal */
failed: Array<{ table: string; column: string; error: string }>;
}
/**
* Verify that every column defined in schema-embedded.ts actually exists
* in the database. Self-heals missing columns via ALTER TABLE ADD COLUMN.
*
* Should be called after initSchema() + runMigrations() complete.
*
* @returns VerifyResult with details of what was checked and fixed.
* @throws Error if any columns could not be healed (after attempting all).
*/
export async function verifySchema(engine: BrainEngine): Promise<VerifyResult> {
const expected = parseExpectedColumns();
const actualColumns = await getActualColumns(engine);
const actualTables = await getActualTables(engine);
const result: VerifyResult = {
checked: 0,
missing: [],
healed: [],
failed: [],
};
// Group expected columns by table for cleaner logging
for (const col of expected) {
// Skip tables that don't exist yet — they'll be created by schema.sql
// on the next initSchema() call. We only verify columns on tables that
// DO exist (the failure mode is: table exists, migration ran, but ALTER
// TABLE silently failed).
if (!actualTables.has(col.table)) {
continue;
}
result.checked++;
const key = `${col.table}.${col.column}`;
if (!actualColumns.has(key)) {
result.missing.push({ table: col.table, column: col.column });
}
}
if (result.missing.length === 0) {
return result;
}
// Log missing columns
console.warn(`\n⚠️ Schema verification found ${result.missing.length} missing column(s):`);
for (const m of result.missing) {
console.warn(` ${m.table}.${m.column}`);
}
console.warn(' Attempting self-heal via ALTER TABLE ADD COLUMN...\n');
// Build a map from table.column -> definition for self-healing
const defMap = new Map<string, string>();
for (const col of expected) {
defMap.set(`${col.table}.${col.column}`, col.definition);
}
// Attempt to add each missing column
for (const m of result.missing) {
const rawDef = defMap.get(`${m.table}.${m.column}`);
if (!rawDef) {
result.failed.push({ ...m, error: 'No definition found in schema' });
continue;
}
const simpleDef = simplifyColumnDef(rawDef);
try {
const sql = `ALTER TABLE ${m.table} ADD COLUMN IF NOT EXISTS ${m.column} ${simpleDef}`;
await engine.runMigration(0, sql);
result.healed.push({ table: m.table, column: m.column });
console.log(` ✓ Added ${m.table}.${m.column}`);
} catch (err: unknown) {
const msg = err instanceof Error ? err.message : String(err);
result.failed.push({ ...m, error: msg });
console.error(` ✗ Failed to add ${m.table}.${m.column}: ${msg}`);
}
}
if (result.healed.length > 0) {
console.log(`\n Schema self-heal: ${result.healed.length}/${result.missing.length} column(s) recovered.`);
}
if (result.failed.length > 0) {
const failList = result.failed.map(f => `${f.table}.${f.column}: ${f.error}`).join('\n ');
throw new Error(
`Schema verification failed: ${result.failed.length} column(s) could not be added:\n ${failList}\n` +
'This usually means PgBouncer transaction-mode silently dropped ALTER TABLE statements.\n' +
'Fix: connect directly to Postgres (not through PgBouncer) and run: gbrain apply-migrations --yes'
);
}
return result;
}
+5 -1
View File
@@ -183,7 +183,11 @@ function acquireLock(workspace: string, opts: InstallOptions): void {
const existing = readLock(workspace);
const staleMs = opts.lockStaleMs ?? DEFAULT_LOCK_STALE_MS;
if (existing) {
const age = Date.now() - existing.mtimeMs;
// Clamp to 0. On Linux ext4, statSync().mtimeMs has sub-ms precision;
// Date.now() is integer ms. A file written microseconds ago can report
// a negative age here, which would break the staleMs:0 "any age is stale"
// contract the force-unlock path relies on (CI passes, local macOS masks it).
const age = Math.max(0, Date.now() - existing.mtimeMs);
// `staleMs: 0` in tests means "any age counts as stale". Use >=
// so a just-written lock qualifies when the threshold is 0.
// Negative age (mtime in the future) happens on fast CI filesystems
+36
View File
@@ -132,6 +132,42 @@ async function assertSourceExists(engine: BrainEngine, id: string): Promise<void
}
}
/**
* Get the local_path of the resolved source (per the resolveSourceId chain).
*
* Returns the on-disk brain repo path for the source the user is currently
* operating against. Used by `gbrain storage status` and `gbrain export
* --restore-only` to find the brain repo without raw SQL or bare try/catch.
*
* Resolution order:
* 1. `sources.local_path` for the resolved source id (multi-source v0.18+ path)
* 2. Legacy global `sync.repo_path` config key (pre-v0.18 default-source brains)
* 3. null
*
* @returns local_path string, or null if no path is configured anywhere.
* @throws If DB error occurs (does NOT silently swallow). Callers handle
* the null case to provide their own fallback (typically a hard error
* telling the user to pass --repo).
*/
export async function getDefaultSourcePath(
engine: BrainEngine,
cwd: string = process.cwd(),
): Promise<string | null> {
const sourceId = await resolveSourceId(engine, null, cwd);
const rows = await engine.executeRaw<{ local_path: string | null }>(
`SELECT local_path FROM sources WHERE id = $1`,
[sourceId],
);
if (rows[0]?.local_path) return rows[0].local_path;
// Legacy fallback: pre-v0.18 brains stored the repo path in the global
// config table under sync.repo_path. The sources table exists but its
// local_path is NULL for the seeded 'default' row. Fall back so storage
// tiering works without forcing a `gbrain sources add . --path .` migration.
const legacyPath = await engine.getConfig('sync.repo_path');
return legacyPath ?? null;
}
/** Exposed for tests. */
export const __testing = {
readDotfileWalk,
+377
View File
@@ -0,0 +1,377 @@
import { readFileSync, existsSync } from 'fs';
import { join } from 'path';
/**
* Storage tier configuration loaded from gbrain.yml.
*
* The canonical key names are `db_tracked` and `db_only` (engine-agnostic).
* The deprecated keys `git_tracked` and `supabase_only` are still read for
* backward compatibility but emit a once-per-process deprecation warning.
* Sunset: future release will reject the deprecated names.
*/
export interface StorageConfig {
db_tracked: string[];
db_only: string[];
}
export type StorageTier = 'db_tracked' | 'db_only' | 'unspecified';
/** Recognized YAML keys (canonical and deprecated). */
const STORAGE_KEYS = new Set([
'db_tracked', 'db_only',
'git_tracked', 'supabase_only', // deprecated aliases
]);
/**
* Parse the gbrain.yml shape: a top-level `storage:` section with up to four
* array-valued nested keys (canonical `db_tracked` / `db_only` plus the
* deprecated aliases `git_tracked` / `supabase_only`).
*
* Intentionally narrow. Does NOT handle the full YAML spec only the file
* shape gbrain controls. Trades expressiveness for zero-dep parsing and
* predictable behavior. Returns null if the file has no `storage:` section
* (so callers can distinguish "no config" from "empty config").
*
* Replaces gray-matter, which silently returned `{data: {}}` on
* delimiter-less YAML and broke the entire feature on every install.
* The defect that prompted this rewrite: storage-config.ts:24 in the
* pre-v0.22.3 implementation.
*
* Returns the raw key map. The caller (loadStorageConfig) is responsible
* for normalizing deprecated keys canonical, emitting deprecation
* warnings, and merging if both old and new keys appear.
*/
type RawStorage = {
db_tracked?: string[];
db_only?: string[];
git_tracked?: string[];
supabase_only?: string[];
};
function parseStorageYaml(content: string): RawStorage | null {
const lines = content.split('\n').map((line) => line.replace(/\r$/, ''));
let inStorage = false;
let currentList: keyof RawStorage | null = null;
const raw: RawStorage = {};
let sawStorage = false;
for (const line of lines) {
// Strip comments. Conservative: drop trailing `# ...` and full-line `#`.
const noComment = line.replace(/\s+#.*$/, '').replace(/^#.*$/, '');
if (noComment.trim() === '') continue;
// Top-level key (no leading whitespace).
if (!noComment.startsWith(' ') && !noComment.startsWith('\t')) {
const colon = noComment.indexOf(':');
if (colon === -1) continue;
const key = noComment.slice(0, colon).trim();
if (key === 'storage') {
inStorage = true;
sawStorage = true;
currentList = null;
continue;
}
inStorage = false;
currentList = null;
continue;
}
if (!inStorage) continue;
const indented = noComment.replace(/^\s+/, '');
if (indented.startsWith('-')) {
if (!currentList) continue;
const value = indented.slice(1).trim().replace(/^["']|["']$/g, '');
if (value) {
if (!raw[currentList]) raw[currentList] = [];
raw[currentList]!.push(value);
}
continue;
}
const colon = indented.indexOf(':');
if (colon === -1) continue;
const key = indented.slice(0, colon).trim();
if (STORAGE_KEYS.has(key)) {
currentList = key as keyof RawStorage;
// Inline empty list: `db_only: []`.
const remainder = indented.slice(colon + 1).trim();
if (remainder === '[]' && !raw[currentList]) {
raw[currentList] = [];
}
continue;
}
currentList = null;
}
if (!sawStorage) return null;
return raw;
}
/**
* Normalize raw parsed keys into canonical StorageConfig shape.
*
* Resolution order (per plan eng-review pass 2 finding #2):
* 1. If canonical keys present, use them.
* 2. Else if deprecated keys present, map to canonical AND emit a
* once-per-process deprecation warning suggesting `gbrain doctor --fix`.
* 3. If both are present, canonical wins. Deprecated keys are ignored
* with a stronger warning (the user is mid-migration).
*
* Validation (validateStorageConfig) always runs against the canonical
* shape, so error messages reference `db_only` / `db_tracked` regardless
* of which keys the user wrote.
*/
let _deprecationWarned = false;
function normalizeStorageConfig(raw: RawStorage): StorageConfig {
const hasCanonical = Boolean(raw.db_tracked || raw.db_only);
const hasDeprecated = Boolean(raw.git_tracked || raw.supabase_only);
if (hasDeprecated && !_deprecationWarned) {
_deprecationWarned = true;
const which = [
raw.git_tracked ? '`git_tracked`' : null,
raw.supabase_only ? '`supabase_only`' : null,
].filter(Boolean).join(' and ');
if (hasCanonical) {
console.warn(
`Warning: ${which} in gbrain.yml is deprecated and ignored ` +
`(canonical keys db_tracked/db_only are present). ` +
`Remove the deprecated keys, or run \`gbrain doctor --fix\`.`,
);
} else {
console.warn(
`Warning: ${which} in gbrain.yml is deprecated. ` +
`Rename to db_tracked / db_only — see docs/storage-tiering.md. ` +
`Run \`gbrain doctor --fix\` for an automated rename.`,
);
}
}
if (hasCanonical) {
return {
db_tracked: raw.db_tracked ?? [],
db_only: raw.db_only ?? [],
};
}
return {
db_tracked: raw.git_tracked ?? [],
db_only: raw.supabase_only ?? [],
};
}
/**
* Load gbrain.yml configuration from the brain repository root.
*
* Returns null when:
* - repoPath is null/undefined
* - gbrain.yml doesn't exist at the repo root
* - gbrain.yml exists but has no `storage:` section (with sanity warning)
*
* Throws when:
* - gbrain.yml exists but is unreadable (permission denied, etc.) D36 lock:
* fail loud rather than silently disable the feature.
*
* Logs a console.warn (once per process) when:
* - File parses but `storage:` section is empty or missing Issue #1 lock:
* surface "your config didn't take" rather than silently no-op.
*/
let _missingStorageWarned = false;
export function loadStorageConfig(repoPath?: string | null): StorageConfig | null {
if (!repoPath) return null;
const yamlPath = join(repoPath, 'gbrain.yml');
if (!existsSync(yamlPath)) return null;
// Read failure is a real error (not a "feature not configured" signal).
// Throwing here lets the caller decide whether to crash or fall back.
const content = readFileSync(yamlPath, 'utf-8');
let raw: RawStorage | null;
try {
raw = parseStorageYaml(content);
} catch (error) {
console.warn(
`Warning: Failed to parse gbrain.yml: ${error instanceof Error ? error.message : String(error)}`,
);
return null;
}
// No storage section at all → null (with sanity warning).
if (raw === null) {
if (!_missingStorageWarned) {
_missingStorageWarned = true;
console.warn(
`Warning: ${yamlPath} exists but has no storage configuration. ` +
`Add a "storage:" section with db_tracked / db_only arrays, ` +
`or remove gbrain.yml to suppress this warning.`,
);
}
return null;
}
const merged = normalizeStorageConfig(raw);
// Empty storage section → return as-is but warn.
if (merged.db_tracked.length === 0 && merged.db_only.length === 0) {
if (!_missingStorageWarned) {
_missingStorageWarned = true;
console.warn(
`Warning: ${yamlPath} exists but has no storage configuration. ` +
`Add a "storage:" section with db_tracked / db_only arrays, ` +
`or remove gbrain.yml to suppress this warning.`,
);
}
return merged;
}
// Normalize cosmetic issues + throw on semantic overlap (D7).
// Throws StorageConfigError on overlap — propagates to the caller.
return normalizeAndValidateStorageConfig(merged);
}
export class StorageConfigError extends Error {
constructor(message: string) {
super(message);
this.name = 'StorageConfigError';
}
}
/**
* Validate storage configuration for conflicts and issues.
* Returns warning strings; callers decide how to surface them.
*
* Always runs against the canonical (db_tracked / db_only) shape error
* messages reference canonical names regardless of which keys the user
* wrote in gbrain.yml.
*
* Pure: does not mutate. For the auto-normalize behavior (D7), see
* `normalizeAndValidateStorageConfig` below.
*/
export function validateStorageConfig(config: StorageConfig): string[] {
const warnings: string[] = [];
const trackedSet = new Set(config.db_tracked);
for (const path of config.db_only) {
if (trackedSet.has(path)) {
warnings.push(`Directory "${path}" appears in both db_tracked and db_only`);
}
}
const allPaths = [...config.db_tracked, ...config.db_only];
for (const path of allPaths) {
if (!path.endsWith('/')) {
warnings.push(`Directory path "${path}" should end with "/" for consistency`);
}
}
return warnings;
}
/**
* Auto-normalize and strict-validate per D7+D8.
*
* 1. Cosmetic fixups are applied silently with a one-time info message
* naming what changed:
* - missing trailing `/` is added
* The message helps the user learn the canonical form without nagging.
* 2. Semantic problems THROW (don't return warnings):
* - same directory in both tiers (ambiguous routing)
*
* Caller passes a fresh raw config; this returns the normalized shape that
* the rest of the code (matcher, sync, etc.) sees.
*/
let _normalizationInfoEmitted = false;
export function normalizeAndValidateStorageConfig(input: StorageConfig): StorageConfig {
const normalize = (paths: string[]): { normalized: string[]; changed: string[] } => {
const normalized: string[] = [];
const changed: string[] = [];
for (const p of paths) {
if (p.endsWith('/')) {
normalized.push(p);
} else {
normalized.push(p + '/');
changed.push(`"${p}" → "${p}/"`);
}
}
return { normalized, changed };
};
const tracked = normalize(input.db_tracked);
const dbonly = normalize(input.db_only);
const allChanged = [...tracked.changed, ...dbonly.changed];
if (allChanged.length > 0 && !_normalizationInfoEmitted) {
_normalizationInfoEmitted = true;
console.warn(
`Note: normalized ${allChanged.length} storage path(s) in gbrain.yml — ` +
`${allChanged.join(', ')}. Add trailing "/" to suppress this note.`,
);
}
// Semantic check: overlap between tiers throws. Ambiguous routing.
const trackedSet = new Set(tracked.normalized);
for (const path of dbonly.normalized) {
if (trackedSet.has(path)) {
throw new StorageConfigError(
`gbrain.yml: directory "${path}" appears in both db_tracked and db_only — ` +
`pick one tier. Edit gbrain.yml to remove the overlap.`,
);
}
}
return { db_tracked: tracked.normalized, db_only: dbonly.normalized };
}
/**
* Path-segment match: a slug belongs to a tier directory iff the directory
* is a complete path-segment ancestor of the slug. `media/x/` matches
* `media/x/foo` but NOT `media/xerox/foo` eliminates the prefix-collision
* class of bug (Issue #5 of the eng review, D6 lock).
*
* Strict: requires the configured directory to end with `/`. The validator
* (per D7+D8) auto-normalizes input so the matcher only ever sees canonical
* trailing-`/` directories.
*/
function matchesTierDir(slug: string, dir: string): boolean {
if (!dir.endsWith('/')) return false; // not normalized — matcher refuses
// slug must equal dir's bare prefix OR start with the trailing-slash form.
// Example: dir = 'media/x/' matches 'media/x/anything' but not 'media/x'
// or 'media/xerox'. (A slug that exactly equals 'media/x' is a directory-
// level entry the brain doesn't write.)
return slug.startsWith(dir);
}
export function isDbTracked(slug: string, config: StorageConfig): boolean {
return config.db_tracked.some((dir) => matchesTierDir(slug, dir));
}
export function isDbOnly(slug: string, config: StorageConfig): boolean {
return config.db_only.some((dir) => matchesTierDir(slug, dir));
}
export function getStorageTier(slug: string, config: StorageConfig): StorageTier {
if (isDbTracked(slug, config)) return 'db_tracked';
if (isDbOnly(slug, config)) return 'db_only';
return 'unspecified';
}
// ── Deprecated aliases — to be removed in a future release ────────
// Kept so existing callers (storage.ts, export.ts) compile during the
// step-by-step refactor. Will be deleted once those call sites migrate
// to the canonical names.
export const isGitTracked = isDbTracked;
export const isSupabaseOnly = isDbOnly;
/** Reset once-per-process warning flags. Test-only. */
export function __resetMissingStorageWarning(): void {
_missingStorageWarned = false;
_deprecationWarned = false;
_normalizationInfoEmitted = false;
}
+101
View File
@@ -0,0 +1,101 @@
/**
* Shared concurrency policy for sync + import + jobs paths.
*
* Three callers used to embed three different policies:
* - performSync (incremental): >100 files 4 workers
* - performFullSync: Postgres 4 workers
* - jobs.ts sync handler: hardcoded 4
*
* They drift over time and confuse users ("why does my sync not parallelize?"
* is a different answer in each path). This module is one source of truth.
*
* v0.22.13 extracted as part of the parallel-sync hardening (PR #490).
*/
import type { BrainEngine } from './engine.ts';
/** Threshold above which auto-concurrency fires for incremental sync paths. */
export const AUTO_CONCURRENCY_FILE_THRESHOLD = 100;
/** Minimum file count below which the parallel branch is skipped even when
* auto-concurrency would otherwise fire. Prevents spawning workers for trivial
* diffs where setup cost exceeds parallelism gains. Only consulted on the
* auto path; explicit `--workers N` bypasses this. */
export const PARALLEL_FILE_FLOOR = 50;
/** Default worker count when auto-concurrency fires. */
export const DEFAULT_PARALLEL_WORKERS = 4;
/**
* Resolve effective worker count for a sync/import operation.
*
* Inputs:
* - engine.kind: 'pglite' always returns 1 (single-connection)
* - override: caller's explicit --workers / opts.concurrency value
* - fileCount: size of the work batch
*
* Rules:
* - PGLite always 1 (the engine is single-connection regardless)
* - explicit override respect it (clamped to >=1)
* - auto path DEFAULT_PARALLEL_WORKERS when fileCount > AUTO_CONCURRENCY_FILE_THRESHOLD, else 1
*
* Note: this function does NOT consult PARALLEL_FILE_FLOOR. The floor is a
* caller-side gate that decides whether to take the parallel code path even
* when the worker count is > 1. It only applies to the auto path; explicit
* --workers bypasses the floor entirely (per Q1 in PR #490).
*/
export function autoConcurrency(
engine: BrainEngine,
fileCount: number,
override?: number,
): number {
if (engine.kind === 'pglite') return 1;
if (override !== undefined) return Math.max(1, override);
return fileCount > AUTO_CONCURRENCY_FILE_THRESHOLD
? DEFAULT_PARALLEL_WORKERS
: 1;
}
/**
* Decide whether the parallel code path should run.
*
* - workers <= 1 never parallel
* - workers > 1 + explicit override always parallel (user opted in,
* respect them even on small diffs Q1 in PR #490)
* - workers > 1 + auto path parallel only when fileCount > PARALLEL_FILE_FLOOR
*/
export function shouldRunParallel(
workers: number,
fileCount: number,
explicit: boolean,
): boolean {
if (workers <= 1) return false;
if (explicit) return true;
return fileCount > PARALLEL_FILE_FLOOR;
}
/**
* Parse a `--workers N` / `--concurrency N` CLI argument value.
*
* Returns:
* - undefined when the flag was not provided
* - a positive integer when the flag was provided with a valid value
*
* Throws on:
* - non-integer ("foo", "1.5", "")
* - zero or negative ("0", "-3")
* - NaN / Infinity
*
* Q2 in PR #490: the prior parseInt-with-no-validation accepted `--workers 0`
* and silently fell through to auto-concurrency (4 workers), the opposite of
* what the user typed. Fail loud instead.
*/
export function parseWorkers(s: string | undefined): number | undefined {
if (s === undefined) return undefined;
const n = parseInt(s, 10);
if (!Number.isFinite(n) || n < 1 || String(n) !== s.trim()) {
throw new Error(
`--workers must be a positive integer, got: ${JSON.stringify(s)}`,
);
}
return n;
}
+110 -8
View File
@@ -301,12 +301,14 @@ export function resolveSlugForPath(filePath: string, repoPrefix?: string): strin
import { existsSync as _existsSync, readFileSync as _readFileSync, appendFileSync as _appendFileSync, mkdirSync as _mkdirSync } from 'fs';
import { join as _joinPath } from 'path';
import { homedir as _homedir } from 'os';
import { gbrainPath as _gbrainPath } from './config.ts';
import { createHash as _createHash } from 'crypto';
export interface SyncFailure {
path: string;
error: string;
/** Structured error code extracted from the error message. */
code?: string;
commit: string;
line?: number;
ts: string;
@@ -314,8 +316,93 @@ export interface SyncFailure {
acknowledged_at?: string;
}
/**
* Best-effort extraction of a structured error code from a sync failure
* message. Matches known ParseValidationCode patterns (SLUG_MISMATCH,
* YAML_PARSE, etc.) and common DB / timeout errors. Returns 'UNKNOWN'
* when no pattern matches.
*
* Order matters: DB-layer errors are checked BEFORE YAML-layer ones so
* Postgres `duplicate key value violates unique constraint` doesn't get
* mislabeled as a YAML duplicate-key. Frontmatter patterns key off the
* canonical messages emitted by `collectValidationErrors()` in markdown.ts.
*/
export function classifyErrorCode(errorMsg: string): string {
// SLUG_MISMATCH: thrown by importFromFile() at src/core/import-file.ts:374.
if (/slug.*does not match|SLUG_MISMATCH/i.test(errorMsg)) return 'SLUG_MISMATCH';
// DB-layer errors come BEFORE the YAML duplicate-key check. Postgres unique-
// constraint violations contain "duplicate key" but are not a YAML problem.
if (/duplicate key value violates unique constraint|DB_DUPLICATE_KEY/i.test(errorMsg)) {
return 'DB_DUPLICATE_KEY';
}
if (/canceling statement due to statement timeout|STATEMENT_TIMEOUT/i.test(errorMsg)) {
return 'STATEMENT_TIMEOUT';
}
// YAML / frontmatter patterns. These match either the canonical message
// strings in src/core/markdown.ts (collectValidationErrors) or the literal
// ParseValidationCode token, so they fire whether the caller stores the
// message or just the code.
if (/YAML parse failed|YAML_PARSE/i.test(errorMsg)) return 'YAML_PARSE';
if (/YAMLException|duplicated mapping key|YAML_DUPLICATE_KEY/i.test(errorMsg)) {
return 'YAML_DUPLICATE_KEY';
}
if (/File is empty or whitespace-only|Frontmatter must start with ---|MISSING_OPEN/i.test(errorMsg)) {
return 'MISSING_OPEN';
}
if (/No closing --- delimiter|Heading at line .* found inside frontmatter|MISSING_CLOSE/i.test(errorMsg)) {
return 'MISSING_CLOSE';
}
if (/Frontmatter block is empty|EMPTY_FRONTMATTER/i.test(errorMsg)) return 'EMPTY_FRONTMATTER';
if (/Content contains null bytes|NULL_BYTES|null byte/i.test(errorMsg)) return 'NULL_BYTES';
if (/Nested double quotes|NESTED_QUOTES/i.test(errorMsg)) return 'NESTED_QUOTES';
// Generic fallbacks.
if (/invalid UTF-?8|INVALID_UTF8/i.test(errorMsg)) return 'INVALID_UTF8';
// v0.22.12 additions: covers the four real production sites in src/core/import-file.ts
// (lines 199, 347, 352, 401) that previously bucketed to UNKNOWN.
if (/file too large|content too large|FILE_TOO_LARGE/i.test(errorMsg)) return 'FILE_TOO_LARGE';
if (/skipping symlink|symlink|SYMLINK_NOT_ALLOWED/i.test(errorMsg)) return 'SYMLINK_NOT_ALLOWED';
return 'UNKNOWN';
}
/** Group failures by error code and return a sorted summary. */
export function summarizeFailuresByCode(
failures: Array<{ error: string; code?: string }>,
): Array<{ code: string; count: number }> {
const counts: Record<string, number> = {};
for (const f of failures) {
const code = f.code ?? classifyErrorCode(f.error);
counts[code] = (counts[code] ?? 0) + 1;
}
return Object.entries(counts)
.sort(([, a], [, b]) => b - a)
.map(([code, count]) => ({ code, count }));
}
/**
* Format a code-grouped summary as a human-readable multi-line string for
* stderr / doctor output. Accepts either raw failures (which are summarized
* internally) or an already-summarized `{code, count}[]` shape (the return
* value of `summarizeFailuresByCode` or `AcknowledgeResult.summary`).
* Returns an empty string when the input is empty.
*/
export function formatCodeBreakdown(
input: Array<{ error: string; code?: string }> | Array<{ code: string; count: number }>,
): string {
// Distinguish by shape: summary entries have a numeric `count`. Empty array
// returns '' from either branch — both paths produce a 0-length join.
const summary =
input.length > 0 && typeof (input[0] as { count?: unknown }).count === 'number'
? (input as Array<{ code: string; count: number }>)
: summarizeFailuresByCode(input as Array<{ error: string; code?: string }>);
return summary.map(s => ` ${s.code}: ${s.count}`).join('\n');
}
function _failuresDir(): string {
return _joinPath(_homedir(), '.gbrain');
return _gbrainPath();
}
export function syncFailuresPath(): string {
@@ -370,6 +457,7 @@ export function recordSyncFailures(
const entry: SyncFailure = {
path: f.path,
error: f.error,
code: classifyErrorCode(f.error),
commit,
line: f.line,
ts: now,
@@ -380,28 +468,42 @@ export function recordSyncFailures(
}
}
export interface AcknowledgeResult {
count: number;
summary: Array<{ code: string; count: number }>;
}
/**
* Mark all unacknowledged failures as acknowledged. Used by
* `gbrain sync --skip-failed`. Returns the number newly acknowledged.
* `gbrain sync --skip-failed`. Returns count and a structured summary
* grouped by error code so the operator can see *why* files were skipped.
*
* We do not delete acknowledged entries stay as historical record so
* doctor can still show them under a "previously skipped" bucket.
*/
export function acknowledgeSyncFailures(): number {
export function acknowledgeSyncFailures(): AcknowledgeResult {
const entries = loadSyncFailures();
if (entries.length === 0) return 0;
if (entries.length === 0) return { count: 0, summary: [] };
const now = new Date().toISOString();
let changed = 0;
const newlyAcked: SyncFailure[] = [];
const updated = entries.map(e => {
if (e.acknowledged) return e;
changed++;
return { ...e, acknowledged: true, acknowledged_at: now };
// Backfill code for entries that predate the code field.
const code = e.code ?? classifyErrorCode(e.error);
const acked = { ...e, code, acknowledged: true, acknowledged_at: now };
newlyAcked.push(acked);
return acked;
});
if (changed === 0) return 0;
if (changed === 0) return { count: 0, summary: [] };
_mkdirSync(_failuresDir(), { recursive: true });
const fd = require('fs').writeFileSync;
fd(syncFailuresPath(), updated.map(e => JSON.stringify(e)).join('\n') + '\n');
return changed;
return {
count: changed,
summary: summarizeFailuresByCode(newlyAcked),
};
}
/** Return only unacknowledged failures. */
+8
View File
@@ -45,6 +45,14 @@ export interface PageFilters {
offset?: number;
/** ISO date string (YYYY-MM-DD or full ISO timestamp). Filter to pages updated_at > value. */
updated_after?: string;
/**
* Prefix-match filter on slug. Implemented as `WHERE slug LIKE prefix || '%'`
* in both engines so it uses the (source_id, slug) UNIQUE constraint's btree
* index for efficient range scans on large brains. Used by storage-tiering
* commands (gbrain storage status, gbrain export --restore-only) to scope
* queries to a tier directory without loading every page into memory.
*/
slugPrefix?: string;
}
// Chunks
+103
View File
@@ -0,0 +1,103 @@
/**
* Shared MCP tool-call dispatch single source of truth for stdio + HTTP transports.
*
* Both transports validate the same params, build the same OperationContext shape,
* and serialize errors identically. Drift between transports caused PR #483's reversed-args
* + missing-context bugs; this module exists to prevent that recurring.
*/
import type { BrainEngine } from '../core/engine.ts';
import { operations, OperationError } from '../core/operations.ts';
import type { Operation, OperationContext } from '../core/operations.ts';
import { loadConfig } from '../core/config.ts';
export interface ToolResult {
content: { type: 'text'; text: string }[];
isError?: boolean;
}
export interface DispatchOpts {
/** Defaults to true (remote/untrusted). Local CLI callers (`gbrain call`) pass false. */
remote?: boolean;
/** Override the default stderr logger (e.g. CLI uses console.* directly). */
logger?: OperationContext['logger'];
}
/** Validate required params exist and have the expected type. Returns null on success, error message on failure. */
export function validateParams(op: Operation, params: Record<string, unknown>): string | null {
for (const [key, def] of Object.entries(op.params)) {
if (def.required && (params[key] === undefined || params[key] === null)) {
return `Missing required parameter: ${key}`;
}
if (params[key] !== undefined && params[key] !== null) {
const val = params[key];
const expected = def.type;
if (expected === 'string' && typeof val !== 'string') return `Parameter "${key}" must be a string`;
if (expected === 'number' && typeof val !== 'number') return `Parameter "${key}" must be a number`;
if (expected === 'boolean' && typeof val !== 'boolean') return `Parameter "${key}" must be a boolean`;
if (expected === 'object' && (typeof val !== 'object' || Array.isArray(val))) return `Parameter "${key}" must be an object`;
if (expected === 'array' && !Array.isArray(val)) return `Parameter "${key}" must be an array`;
}
}
return null;
}
const stderrLogger: OperationContext['logger'] = {
info: (msg: string) => process.stderr.write(`[info] ${msg}\n`),
warn: (msg: string) => process.stderr.write(`[warn] ${msg}\n`),
error: (msg: string) => process.stderr.write(`[error] ${msg}\n`),
};
export function buildOperationContext(
engine: BrainEngine,
params: Record<string, unknown>,
opts: DispatchOpts = {},
): OperationContext {
return {
engine,
config: loadConfig() || { engine: 'postgres' },
logger: opts.logger || stderrLogger,
dryRun: !!params.dry_run,
remote: opts.remote ?? true,
};
}
/**
* Resolve operation, validate params, build context, invoke handler, format result.
*
* Returns a `ToolResult` with the same shape both MCP transports need:
* `{ content: [{ type: 'text', text }], isError?: boolean }`.
*/
export async function dispatchToolCall(
engine: BrainEngine,
name: string,
params: Record<string, unknown> | undefined,
opts: DispatchOpts = {},
): Promise<ToolResult> {
const op = operations.find(o => o.name === name);
if (!op) {
return { content: [{ type: 'text', text: `Error: Unknown tool: ${name}` }], isError: true };
}
const safeParams = params || {};
const validationError = validateParams(op, safeParams);
if (validationError) {
return {
content: [{ type: 'text', text: JSON.stringify({ error: 'invalid_params', message: validationError }, null, 2) }],
isError: true,
};
}
const ctx = buildOperationContext(engine, safeParams, opts);
try {
const result = await op.handler(ctx, safeParams);
return { content: [{ type: 'text', text: JSON.stringify(result, null, 2) }] };
} catch (e: unknown) {
if (e instanceof OperationError) {
return { content: [{ type: 'text', text: JSON.stringify(e.toJSON(), null, 2) }], isError: true };
}
const msg = e instanceof Error ? e.message : String(e);
return { content: [{ type: 'text', text: `Error: ${msg}` }], isError: true };
}
}
+354
View File
@@ -0,0 +1,354 @@
/**
* HTTP transport for `gbrain serve --http`.
*
* Postgres-only. PGLite users get a clear fail-fast at startup (the access_tokens
* table doesn't exist on PGLite per pglite-schema.ts).
*
* Security model:
* - Every request must include `Authorization: Bearer <token>` (except /health)
* - Tokens are validated against SHA-256 hashes in the access_tokens table
* - Create/manage tokens with auth.ts (gbrain auth create/list/revoke)
* - No open OAuth, no client_credentials, no self-service tokens
*
* Hardening:
* - CORS default-deny: allowlist via GBRAIN_HTTP_CORS_ORIGIN (comma-separated)
* - Rate limit: per-IP pre-auth (protects DB from brute-force load) + per-token-id post-auth
* (limits runaway clients). Default 30 req/min per IP, 60 req/min per token. Bounded LRU
* so attacker-controlled keys can't grow memory unbounded.
* - Body cap: 1 MiB default (GBRAIN_HTTP_MAX_BODY_BYTES). Stream-counted, not buffered
* chunked transfers without Content-Length are still capped.
* - last_used_at debounce: only one UPDATE per token per 60s (SQL-level WHERE clause).
* - mcp_request_log: one row per request with token_name + operation + status + latency.
*
* Replaces the standalone HTTP+OAuth wrapper that was vulnerable to unauthenticated
* client registration (see SECURITY.md).
*/
import { createHash } from 'crypto';
import type { BrainEngine } from '../core/engine.ts';
import { buildToolDefs } from './tool-defs.ts';
import { operations } from '../core/operations.ts';
import { VERSION } from '../version.ts';
import { dispatchToolCall } from './dispatch.ts';
import { buildDefaultLimiters, type RateLimiter } from './rate-limit.ts';
const DEFAULT_BODY_CAP = 1024 * 1024; // 1 MiB
function hashToken(token: string): string {
return createHash('sha256').update(token).digest('hex');
}
function envInt(name: string, fallback: number): number {
const v = process.env[name];
if (!v) return fallback;
const n = parseInt(v, 10);
return Number.isFinite(n) && n > 0 ? n : fallback;
}
function parseCorsAllowlist(): Set<string> | null {
const v = process.env.GBRAIN_HTTP_CORS_ORIGIN;
if (!v) return null;
return new Set(v.split(',').map(s => s.trim()).filter(Boolean));
}
interface HttpTransportOptions {
port: number;
engine: BrainEngine;
/** Override limiters (for tests). Defaults to env-driven buildDefaultLimiters. */
limiters?: { ip: RateLimiter; token: RateLimiter };
}
interface AuthResult {
ok: boolean;
tokenId?: string;
tokenName?: string;
}
/** Read up to `cap` bytes off req.body. Returns null if cap exceeded. */
async function readBodyWithCap(req: Request, cap: number): Promise<string | null> {
const cl = req.headers.get('content-length');
if (cl) {
const n = parseInt(cl, 10);
if (Number.isFinite(n) && n > cap) return null;
}
const reader = req.body?.getReader();
if (!reader) return '';
const chunks: Uint8Array[] = [];
let total = 0;
while (true) {
const { value, done } = await reader.read();
if (done) break;
if (!value) continue;
total += value.byteLength;
if (total > cap) {
try { await reader.cancel(); } catch { /* noop */ }
return null;
}
chunks.push(value);
}
// Concatenate without Buffer to keep this Node-vs-Bun-portable.
const merged = new Uint8Array(total);
let offset = 0;
for (const c of chunks) {
merged.set(c, offset);
offset += c.byteLength;
}
return new TextDecoder().decode(merged);
}
/** Resolve client IP. Honors X-Forwarded-For only when GBRAIN_HTTP_TRUST_PROXY=1. */
function resolveClientIp(req: Request, server: { requestIP: (r: Request) => { address: string } | null }): string {
if (process.env.GBRAIN_HTTP_TRUST_PROXY === '1') {
const xff = req.headers.get('x-forwarded-for');
if (xff) {
const first = xff.split(',')[0]?.trim();
if (first) return first;
}
const xRealIp = req.headers.get('x-real-ip');
if (xRealIp) return xRealIp.trim();
}
const sock = server.requestIP(req);
return sock?.address || 'unknown';
}
export async function startHttpTransport(opts: HttpTransportOptions) {
const { port, engine } = opts;
// Fail-fast: HTTP transport requires Postgres because access_tokens / mcp_request_log
// only exist in the Postgres schema (see src/core/pglite-schema.ts:5-6).
if ((engine as { kind?: string }).kind !== 'postgres') {
console.error('Error: gbrain serve --http requires a Postgres engine for remote auth tokens.');
console.error('PGLite is local-only by design (access_tokens table is Postgres-only).');
console.error('Either:');
console.error(' - Use stdio: gbrain serve');
console.error(' - Migrate to Postgres: gbrain migrate --to supabase');
process.exit(1);
}
const sql = (engine as unknown as { sql: any }).sql;
if (!sql) {
console.error('Error: Postgres engine has no .sql client. Engine may not be connected.');
process.exit(1);
}
const limiters = opts.limiters || buildDefaultLimiters();
const bodyCap = envInt('GBRAIN_HTTP_MAX_BODY_BYTES', DEFAULT_BODY_CAP);
const corsAllowlist = parseCorsAllowlist();
const tools = buildToolDefs(operations);
function corsHeaders(origin: string | null, extra: Record<string, string> = {}): Record<string, string> {
const headers: Record<string, string> = { ...extra };
if (corsAllowlist && origin && corsAllowlist.has(origin)) {
headers['Access-Control-Allow-Origin'] = origin;
headers['Vary'] = 'Origin';
}
return headers;
}
function corsPreflightHeaders(origin: string | null): Record<string, string> {
const headers: Record<string, string> = {
'Access-Control-Allow-Methods': 'POST, GET, OPTIONS',
'Access-Control-Allow-Headers': 'Content-Type, Authorization, Accept',
};
if (corsAllowlist && origin && corsAllowlist.has(origin)) {
headers['Access-Control-Allow-Origin'] = origin;
headers['Vary'] = 'Origin';
}
return headers;
}
async function validateToken(authHeader: string | null): Promise<AuthResult> {
if (!authHeader?.startsWith('Bearer ')) return { ok: false };
const token = authHeader.slice(7);
const hash = hashToken(token);
try {
const [row] = await sql`
SELECT id, name FROM access_tokens
WHERE token_hash = ${hash} AND revoked_at IS NULL
`;
if (!row) return { ok: false };
// Debounced last_used_at update — only writes once per token per 60s.
// SQL-level WHERE clause keeps this race-tolerant even under concurrent requests.
sql`UPDATE access_tokens
SET last_used_at = now()
WHERE id = ${row.id}
AND (last_used_at IS NULL OR last_used_at < now() - interval '60 seconds')`
.catch(() => { /* fire-and-forget */ });
return { ok: true, tokenId: row.id, tokenName: row.name };
} catch {
return { ok: false };
}
}
function logRequest(tokenName: string | null, operation: string, status: string, latencyMs: number) {
sql`INSERT INTO mcp_request_log (token_name, operation, latency_ms, status)
VALUES (${tokenName}, ${operation}, ${latencyMs}, ${status})`
.catch(() => { /* best-effort */ });
}
const server = Bun.serve({
port,
async fetch(req, server) {
const startedMs = Date.now();
const url = new URL(req.url);
const path = url.pathname;
const origin = req.headers.get('origin');
// CORS preflight
if (req.method === 'OPTIONS') {
return new Response(null, { headers: corsPreflightHeaders(origin) });
}
// Health check — no auth, no rate limit. Probes the DB so orchestration
// doesn't see "ok" while clients are getting misleading 401s during a DB outage.
if (path === '/health') {
try {
await sql`SELECT 1`;
return Response.json(
{ status: 'ok', version: VERSION, transport: 'http', db: 'ok' },
{ headers: corsHeaders(origin) },
);
} catch (e: any) {
return Response.json(
{ status: 'unhealthy', version: VERSION, transport: 'http', db: 'unreachable', error: e?.message ?? 'unknown' },
{ status: 503, headers: corsHeaders(origin) },
);
}
}
if (path !== '/mcp') {
return Response.json({ error: 'not_found' }, { status: 404, headers: corsHeaders(origin) });
}
if (req.method !== 'POST') {
return Response.json({ error: 'method_not_allowed' }, { status: 405, headers: corsHeaders(origin) });
}
const ip = resolveClientIp(req, server);
// Pre-auth IP rate limit. Fires BEFORE the DB lookup so we actually limit brute-force load.
const ipCheck = limiters.ip.check(ip);
if (!ipCheck.allowed) {
logRequest(null, 'unknown', 'rate_limited', Date.now() - startedMs);
return Response.json(
{ error: 'rate_limited', message: 'Too many requests' },
{
status: 429,
headers: corsHeaders(origin, { 'Retry-After': String(ipCheck.retryAfter ?? 60) }),
},
);
}
// Body cap (stream-counted; chunked transfers caught here, not at req.json).
const bodyText = await readBodyWithCap(req, bodyCap);
if (bodyText === null) {
logRequest(null, 'unknown', 'body_too_large', Date.now() - startedMs);
return Response.json(
{ error: 'payload_too_large', message: `Request body exceeds ${bodyCap} bytes` },
{ status: 413, headers: corsHeaders(origin) },
);
}
// Auth.
const auth = await validateToken(req.headers.get('Authorization'));
if (!auth.ok) {
logRequest(null, 'unknown', 'auth_failed', Date.now() - startedMs);
return Response.json(
{ error: 'invalid_token', message: 'Bearer token required. Create one: gbrain auth create <name>' },
{ status: 401, headers: corsHeaders(origin) },
);
}
// Post-auth token-id rate limit. Limits runaway authed clients.
const tokCheck = limiters.token.check(auth.tokenId!);
if (!tokCheck.allowed) {
logRequest(auth.tokenName!, 'unknown', 'rate_limited', Date.now() - startedMs);
return Response.json(
{ error: 'rate_limited', message: 'Too many requests for this token' },
{
status: 429,
headers: corsHeaders(origin, { 'Retry-After': String(tokCheck.retryAfter ?? 60) }),
},
);
}
// Parse JSON-RPC body.
let body: { method?: string; params?: any; id?: any };
try {
body = JSON.parse(bodyText);
} catch (e: any) {
logRequest(auth.tokenName!, 'unknown', 'parse_error', Date.now() - startedMs);
return Response.json(
{ error: 'parse_error', message: e?.message ?? 'invalid JSON' },
{ status: 400, headers: corsHeaders(origin) },
);
}
const { method, params, id } = body;
// initialize
if (method === 'initialize') {
logRequest(auth.tokenName!, 'initialize', 'success', Date.now() - startedMs);
return Response.json(
{
result: {
protocolVersion: '2025-03-26',
serverInfo: { name: 'gbrain', version: VERSION },
capabilities: { tools: {} },
},
jsonrpc: '2.0',
id,
},
{ headers: corsHeaders(origin) },
);
}
// notifications/initialized — acknowledge with 204
if (method === 'notifications/initialized') {
return new Response(null, { status: 204, headers: corsHeaders(origin) });
}
// tools/list
if (method === 'tools/list') {
logRequest(auth.tokenName!, 'tools/list', 'success', Date.now() - startedMs);
return Response.json(
{ result: { tools }, jsonrpc: '2.0', id },
{ headers: corsHeaders(origin) },
);
}
// tools/call — dispatch through shared dispatch.ts (parity with stdio)
if (method === 'tools/call') {
const toolName: string = params?.name ?? 'unknown';
const args: Record<string, unknown> = params?.arguments ?? {};
const result = await dispatchToolCall(engine, toolName, args, { remote: true });
const status = result.isError ? 'error' : 'success';
logRequest(auth.tokenName!, `tools/call:${toolName}`, status, Date.now() - startedMs);
return Response.json(
{ result, jsonrpc: '2.0', id },
{ headers: corsHeaders(origin) },
);
}
logRequest(auth.tokenName!, method ?? 'unknown', 'unknown_method', Date.now() - startedMs);
return Response.json(
{ error: 'unknown_method', message: `Unknown method: ${method}` },
{ status: 400, headers: corsHeaders(origin) },
);
},
});
console.error(`GBrain HTTP MCP server running on port ${port}`);
console.error(` Health: http://localhost:${port}/health`);
console.error(` MCP: http://localhost:${port}/mcp`);
console.error(` Auth: Bearer token required (create with: gbrain auth create <name>)`);
if (!corsAllowlist) {
console.error(' CORS: default-deny. Set GBRAIN_HTTP_CORS_ORIGIN=https://your.app to allow browser clients.');
} else {
console.error(` CORS: allowlist = ${[...corsAllowlist].join(', ')}`);
}
console.error('');
console.error('⚠️ Do NOT use open OAuth registration for remote MCP access.');
console.error(' Tokens are managed via: gbrain auth create/list/revoke');
return server;
}
+142
View File
@@ -0,0 +1,142 @@
/**
* Rate limiter for `gbrain serve --http`.
*
* Token-bucket per key, stored in a bounded LRU map so attacker-controlled keys
* can't grow memory unbounded. TTL prune on every access (entries older than
* 2× window are evicted) so abandoned keys don't sit around forever.
*
* Two buckets in the request pipeline (see http-transport.ts):
* 1. Pre-auth IP bucket fires BEFORE the DB lookup so we actually limit
* brute-force load against access_tokens, not just response codes.
* 2. Post-auth token-id bucket fires after auth so legitimate-but-runaway
* clients get throttled at the right principal.
*
* Both buckets behave identically; only the key differs.
*/
export interface RateLimitOpts {
/** Maximum requests in the window. */
limit: number;
/** Window length in milliseconds. */
windowMs: number;
/** LRU cap on distinct keys. Evicts least-recently-used on overflow. */
lruCap: number;
}
export interface RateLimitResult {
allowed: boolean;
/** Seconds until next request would be allowed (only set when !allowed). */
retryAfter?: number;
/** Tokens remaining in the bucket after this check. */
remaining: number;
}
interface Bucket {
tokens: number;
/** Used for refill math: tokens accrue based on elapsed time since this. */
lastRefillMs: number;
/** Used for TTL eviction: time of last check, regardless of refill. Prevents bucket-reset attack
* where an exhausted key would otherwise get TTL-evicted and recreated fresh. */
lastTouchedMs: number;
}
/** Clock function — defaults to Date.now, overridable for tests. */
type Clock = () => number;
export class RateLimiter {
readonly opts: RateLimitOpts;
private readonly buckets: Map<string, Bucket> = new Map();
private readonly clock: Clock;
constructor(opts: RateLimitOpts, clock: Clock = Date.now) {
if (opts.limit <= 0) throw new Error('RateLimiter: limit must be > 0');
if (opts.windowMs <= 0) throw new Error('RateLimiter: windowMs must be > 0');
if (opts.lruCap <= 0) throw new Error('RateLimiter: lruCap must be > 0');
this.opts = opts;
this.clock = clock;
}
check(key: string): RateLimitResult {
const now = this.clock();
this.prune(now);
let bucket = this.buckets.get(key);
if (!bucket) {
bucket = { tokens: this.opts.limit, lastRefillMs: now, lastTouchedMs: now };
} else {
// Refill: tokens accrue continuously over the window. limit/windowMs tokens per ms.
const elapsed = now - bucket.lastRefillMs;
const refilled = Math.floor((elapsed * this.opts.limit) / this.opts.windowMs);
if (refilled > 0) {
bucket.tokens = Math.min(this.opts.limit, bucket.tokens + refilled);
bucket.lastRefillMs = now;
}
bucket.lastTouchedMs = now;
// LRU bookkeeping: re-insert to move to end (Map iteration order = insertion).
this.buckets.delete(key);
}
if (bucket.tokens > 0) {
bucket.tokens -= 1;
this.buckets.set(key, bucket);
this.evictIfOver();
return { allowed: true, remaining: bucket.tokens };
}
// No tokens. Compute Retry-After from when the next token will accrue.
const msPerToken = this.opts.windowMs / this.opts.limit;
const msUntilNext = msPerToken - (now - bucket.lastRefillMs);
const retryAfter = Math.max(1, Math.ceil(msUntilNext / 1000));
this.buckets.set(key, bucket);
this.evictIfOver();
return { allowed: false, retryAfter, remaining: 0 };
}
/** Evict TTL-expired entries (older than 2× window since last touch). Cheap: O(n) but n is bounded by lruCap.
* Uses lastTouchedMs (not lastRefillMs) so an attacker can't reset their bucket by hammering an exhausted key
* past the TTL every check updates lastTouchedMs even when refill produces 0 tokens. */
private prune(now: number): void {
const ttl = this.opts.windowMs * 2;
for (const [key, bucket] of this.buckets) {
if (now - bucket.lastTouchedMs > ttl) {
this.buckets.delete(key);
} else {
// Map iteration is in insertion order; once we hit a fresh entry, the rest are also fresh
// ONLY if we maintain insertion-order = recency. That holds because check() does delete+set on every call.
break;
}
}
}
private evictIfOver(): void {
while (this.buckets.size > this.opts.lruCap) {
// Map iteration starts at oldest (first-inserted). Delete it.
const oldestKey = this.buckets.keys().next().value;
if (oldestKey === undefined) break;
this.buckets.delete(oldestKey);
}
}
/** Test helper: current key count. */
get size(): number {
return this.buckets.size;
}
}
/** Parse a positive integer env var, falling back to default. */
function envInt(name: string, fallback: number): number {
const v = process.env[name];
if (!v) return fallback;
const n = parseInt(v, 10);
return Number.isFinite(n) && n > 0 ? n : fallback;
}
/** Build limiters from env. Keep this lazy — tests can construct RateLimiter directly. */
export function buildDefaultLimiters(clock: Clock = Date.now): { ip: RateLimiter; token: RateLimiter } {
const lruCap = envInt('GBRAIN_HTTP_RATE_LIMIT_LRU', 10000);
const windowMs = 60_000;
return {
ip: new RateLimiter({ limit: envInt('GBRAIN_HTTP_RATE_LIMIT_IP', 30), windowMs, lruCap }, clock),
token: new RateLimiter({ limit: envInt('GBRAIN_HTTP_RATE_LIMIT_TOKEN', 60), windowMs, lruCap }, clock),
};
}
+13 -66
View File
@@ -2,30 +2,10 @@ import { Server } from '@modelcontextprotocol/sdk/server/index.js';
import { StdioServerTransport } from '@modelcontextprotocol/sdk/server/stdio.js';
import { ListToolsRequestSchema, CallToolRequestSchema } from '@modelcontextprotocol/sdk/types.js';
import type { BrainEngine } from '../core/engine.ts';
import { operations, OperationError } from '../core/operations.ts';
import type { Operation, OperationContext } from '../core/operations.ts';
import { loadConfig } from '../core/config.ts';
import { operations } from '../core/operations.ts';
import { VERSION } from '../version.ts';
import { buildToolDefs } from './tool-defs.ts';
/** Validate required params exist and have the expected type */
function validateParams(op: Operation, params: Record<string, unknown>): string | null {
for (const [key, def] of Object.entries(op.params)) {
if (def.required && (params[key] === undefined || params[key] === null)) {
return `Missing required parameter: ${key}`;
}
if (params[key] !== undefined && params[key] !== null) {
const val = params[key];
const expected = def.type;
if (expected === 'string' && typeof val !== 'string') return `Parameter "${key}" must be a string`;
if (expected === 'number' && typeof val !== 'number') return `Parameter "${key}" must be a number`;
if (expected === 'boolean' && typeof val !== 'boolean') return `Parameter "${key}" must be a boolean`;
if (expected === 'object' && (typeof val !== 'object' || Array.isArray(val))) return `Parameter "${key}" must be an object`;
if (expected === 'array' && !Array.isArray(val)) return `Parameter "${key}" must be an array`;
}
}
return null;
}
import { dispatchToolCall, validateParams, buildOperationContext } from './dispatch.ts';
export async function startMcpServer(engine: BrainEngine) {
const server = new Server(
@@ -40,50 +20,21 @@ export async function startMcpServer(engine: BrainEngine) {
tools: buildToolDefs(operations),
}));
// Dispatch tool calls to operation handlers
server.setRequestHandler(CallToolRequestSchema, async (request: any) => {
// Dispatch tool calls via shared dispatch.ts (parity with HTTP transport).
// MCP stdio callers are remote/untrusted; dispatch defaults remote=true.
// The MCP SDK's response type widened in 1.29 to allow a managed-task wrapper;
// gbrain ops are synchronous, so we return the legacy `{ content, isError? }`
// shape and cast through `any` (the SDK accepts it via the ServerResult union).
server.setRequestHandler(CallToolRequestSchema, async (request: any): Promise<any> => {
const { name, arguments: params } = request.params;
const op = operations.find(o => o.name === name);
if (!op) {
return { content: [{ type: 'text', text: `Error: Unknown tool: ${name}` }], isError: true };
}
const ctx: OperationContext = {
engine,
config: loadConfig() || { engine: 'postgres' },
logger: {
info: (msg: string) => process.stderr.write(`[info] ${msg}\n`),
warn: (msg: string) => process.stderr.write(`[warn] ${msg}\n`),
error: (msg: string) => process.stderr.write(`[error] ${msg}\n`),
},
dryRun: !!(params?.dry_run),
// MCP stdio callers are remote/untrusted; enforce strict file confinement.
remote: true,
};
const safeParams = params || {};
const validationError = validateParams(op, safeParams);
if (validationError) {
return { content: [{ type: 'text', text: JSON.stringify({ error: 'invalid_params', message: validationError }, null, 2) }], isError: true };
}
try {
const result = await op.handler(ctx, safeParams);
return { content: [{ type: 'text', text: JSON.stringify(result, null, 2) }] };
} catch (e: unknown) {
if (e instanceof OperationError) {
return { content: [{ type: 'text', text: JSON.stringify(e.toJSON(), null, 2) }], isError: true };
}
const msg = e instanceof Error ? e.message : String(e);
return { content: [{ type: 'text', text: `Error: ${msg}` }], isError: true };
}
return dispatchToolCall(engine, name, params, { remote: true });
});
const transport = new StdioServerTransport();
await server.connect(transport);
}
// Backward compat: used by `gbrain call` command
// Backward compat: used by `gbrain call` command (trusted local path).
export async function handleToolCall(
engine: BrainEngine,
tool: string,
@@ -95,14 +46,10 @@ export async function handleToolCall(
const validationError = validateParams(op, params);
if (validationError) throw new Error(validationError);
const ctx: OperationContext = {
engine,
config: loadConfig() || { engine: 'postgres' },
logger: { info: console.log, warn: console.warn, error: console.error },
dryRun: !!(params?.dry_run),
// Backing path for `gbrain call` CLI command — trusted local invocation.
const ctx = buildOperationContext(engine, params, {
remote: false,
};
logger: { info: console.log, warn: console.warn, error: console.error },
});
return op.handler(ctx, params);
}
+97
View File
@@ -0,0 +1,97 @@
/**
* AgentRunner registry + selection tests. Proves the harness contract is
* truly agent-agnostic via a fake-runner integration.
*/
import { describe, test, expect, beforeEach } from 'bun:test';
import {
registerAgentRunner, resolveAgentRunner, listRegisteredAgents,
_resetRegistryForTests,
type AgentRunner, type DetectResult, type InvokeOpts, type InvokeResult, type TranscriptSink,
} from '../src/core/claw-test/agent-runner.ts';
class FakeRunner implements AgentRunner {
readonly name: string;
invocations = 0;
detected: DetectResult = { available: true, binPath: '/usr/bin/fake-agent' };
constructor(name: string) { this.name = name; }
async detect(): Promise<DetectResult> { return this.detected; }
async invoke(_opts: InvokeOpts): Promise<InvokeResult> {
this.invocations++;
return { exitCode: 0, durationMs: 1 };
}
}
beforeEach(() => {
_resetRegistryForTests();
});
describe('registry', () => {
test('register + resolve roundtrips', () => {
registerAgentRunner('fake', () => new FakeRunner('fake'));
const r = resolveAgentRunner('fake');
expect(r.name).toBe('fake');
});
test('resolve unknown agent throws with helpful list', () => {
registerAgentRunner('alpha', () => new FakeRunner('alpha'));
registerAgentRunner('beta', () => new FakeRunner('beta'));
expect(() => resolveAgentRunner('gamma')).toThrow(/registered: alpha, beta/);
});
test('listRegisteredAgents returns sorted names', () => {
registerAgentRunner('zeta', () => new FakeRunner('zeta'));
registerAgentRunner('alpha', () => new FakeRunner('alpha'));
expect(listRegisteredAgents()).toEqual(['alpha', 'zeta']);
});
test('factory pattern produces independent instances', () => {
registerAgentRunner('fake', () => new FakeRunner('fake'));
const a = resolveAgentRunner('fake') as FakeRunner;
const b = resolveAgentRunner('fake') as FakeRunner;
expect(a).not.toBe(b);
});
});
describe('agent-agnosticism guard', () => {
test('a fake runner can satisfy the AgentRunner contract end-to-end', async () => {
registerAgentRunner('fake', () => new FakeRunner('fake'));
const runner = resolveAgentRunner('fake');
// The harness contract: detect → invoke. Nothing else.
const detected = await runner.detect();
expect(detected.available).toBe(true);
expect(detected.binPath).toBe('/usr/bin/fake-agent');
let written = 0;
const sink: TranscriptSink = {
write: () => { written++; },
nextOffset: () => 0,
close: async () => { /* noop */ },
};
const result = await runner.invoke({
cwd: '/tmp',
brief: 'hello',
env: {},
timeoutMs: 1000,
transcriptSink: sink,
});
expect(result.exitCode).toBe(0);
});
test('a runner reporting unavailable still satisfies the contract', async () => {
class UnavailableRunner implements AgentRunner {
name = 'gone';
async detect() { return { available: false, reason: 'not installed' } as DetectResult; }
async invoke(): Promise<InvokeResult> { throw new Error('should not be called'); }
}
registerAgentRunner('gone', () => new UnavailableRunner());
const r = resolveAgentRunner('gone');
const d = await r.detect();
expect(d.available).toBe(false);
expect(d.reason).toBe('not installed');
});
});
+192
View File
@@ -0,0 +1,192 @@
/**
* PGLite forward-reference bootstrap tests.
*
* Validates the contract of `PGLiteEngine#applyForwardReferenceBootstrap`:
* given a brain that lacks the schema-blob's forward-referenced state, the
* bootstrap adds enough state for PGLITE_SCHEMA_SQL to replay safely.
*
* The bootstrap covers the wedge incidents from issues
* #239/#266/#357/#366/#374/#375/#378/#396 every gbrain release that added
* a column-with-index in the schema blob without a corresponding bootstrap
* triggered the same wedge family.
*
* Honest limitation: test 4 simulates a v20 brain by dropping known forward
* state from a fresh-LATEST instance. This is the same down-mutation pattern
* codex flagged as "weak simulation" it can't simulate every possible
* historical state. Acceptable here because the bootstrap's contract is
* narrow ("given a brain that lacks the specific forward-references,
* initSchema produces a brain at LATEST"), and that contract is exactly
* what this test exercises.
*/
import { describe, test, expect } from 'bun:test';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { LATEST_VERSION } from '../src/core/migrate.ts';
describe('PGLiteEngine#applyForwardReferenceBootstrap', () => {
test('no-op on fresh install (no pages or links table)', async () => {
const engine = new PGLiteEngine();
await engine.connect({});
try {
// Don't call initSchema — verify bootstrap alone does nothing on empty DB
await (engine as any).applyForwardReferenceBootstrap();
const { rows } = await (engine as any).db.query(`
SELECT COUNT(*)::int AS c FROM information_schema.tables
WHERE table_schema = 'public'
`);
expect(rows[0].c).toBe(0);
} finally {
await engine.disconnect();
}
}, 30000);
test('idempotent: calling twice produces same result', async () => {
const engine = new PGLiteEngine();
await engine.connect({});
try {
await engine.initSchema();
const db = (engine as any).db;
// Mutate to pre-v0.18 shape: drop source_id and the sources FK target
await db.exec(`
ALTER TABLE pages DROP CONSTRAINT IF EXISTS pages_source_slug_key;
ALTER TABLE pages ADD CONSTRAINT pages_slug_key UNIQUE (slug);
DROP INDEX IF EXISTS idx_pages_source_id;
ALTER TABLE pages DROP COLUMN IF EXISTS source_id;
DROP TABLE IF EXISTS sources CASCADE;
`);
// First call: applies bootstrap
await (engine as any).applyForwardReferenceBootstrap();
// Second call: must not error, must not duplicate state
await (engine as any).applyForwardReferenceBootstrap();
const { rows: cols } = await db.query(`
SELECT column_name FROM information_schema.columns
WHERE table_name = 'pages' AND column_name = 'source_id'
`);
expect(cols).toHaveLength(1);
const { rows: src } = await db.query(`SELECT COUNT(*)::int AS c FROM sources`);
expect(src[0].c).toBe(1); // 'default' seed not duplicated
} finally {
await engine.disconnect();
}
}, 30000);
test('no-op on modern brain (source_id and links provenance already present)', async () => {
const engine = new PGLiteEngine();
await engine.connect({});
try {
await engine.initSchema();
const db = (engine as any).db;
const before = await db.query(`SELECT COUNT(*)::int AS c FROM sources`);
await (engine as any).applyForwardReferenceBootstrap();
const after = await db.query(`SELECT COUNT(*)::int AS c FROM sources`);
// Bootstrap probe should detect the brain is modern and skip the seed insert
expect(after.rows[0].c).toBe(before.rows[0].c);
} finally {
await engine.disconnect();
}
}, 30000);
test('full path: pre-v0.18 brain reaches LATEST_VERSION via initSchema', async () => {
const engine = new PGLiteEngine();
await engine.connect({});
try {
await engine.initSchema();
const db = (engine as any).db;
// Mutate to pre-v0.18 shape: strip the forward-referenced state.
// Match the shape from #399's regression fixture; constraints first
// (so dropping columns succeeds).
await db.exec(`
ALTER TABLE pages DROP CONSTRAINT IF EXISTS pages_source_slug_key;
ALTER TABLE pages ADD CONSTRAINT pages_slug_key UNIQUE (slug);
DROP INDEX IF EXISTS idx_pages_source_id;
ALTER TABLE pages DROP COLUMN IF EXISTS source_id;
DROP TABLE IF EXISTS sources CASCADE;
ALTER TABLE links DROP CONSTRAINT IF EXISTS links_resolution_type_check;
ALTER TABLE links DROP COLUMN IF EXISTS resolution_type;
`);
await engine.setConfig('version', '20');
// Path under test: bootstrap → SCHEMA_SQL → runMigrations
await engine.initSchema();
expect(await engine.getConfig('version')).toBe(String(LATEST_VERSION));
const { rows: srcCol } = await db.query(`
SELECT column_name FROM information_schema.columns
WHERE table_name = 'pages' AND column_name = 'source_id'
`);
expect(srcCol).toHaveLength(1);
const { rows: defaultSrc } = await db.query(`SELECT id FROM sources WHERE id = 'default'`);
expect(defaultSrc).toHaveLength(1);
} finally {
await engine.disconnect();
}
}, 30000);
test('fresh install regression: initSchema on empty DB produces LATEST', async () => {
// The bootstrap's table-existence probe must not mis-classify "no table"
// as "pre-v0.18 brain." Without the table-existence guard, the bootstrap
// would call runMigrations against an empty DB and crash on
// `relation "config" does not exist`. Regression test for that path.
const engine = new PGLiteEngine();
await engine.connect({});
try {
await engine.initSchema();
expect(await engine.getConfig('version')).toBe(String(LATEST_VERSION));
const db = (engine as any).db;
const pages = await db.query(`SELECT 1 FROM pages LIMIT 0`);
const sources = await db.query(`SELECT 1 FROM sources LIMIT 0`);
const config = await db.query(`SELECT 1 FROM config LIMIT 0`);
expect(pages).toBeDefined();
expect(sources).toBeDefined();
expect(config).toBeDefined();
} finally {
await engine.disconnect();
}
}, 30000);
test('pre-v0.13 links shape: bootstrap adds link_source + origin_page_id', async () => {
// Issues #266 / #357 — pre-v0.13 brains had `links` without
// `link_source` / `origin_page_id`. Schema blob's
// `CREATE INDEX idx_links_source` would crash before v11 ran.
const engine = new PGLiteEngine();
await engine.connect({});
try {
await engine.initSchema();
const db = (engine as any).db;
await db.exec(`
DROP INDEX IF EXISTS idx_links_source;
DROP INDEX IF EXISTS idx_links_origin;
ALTER TABLE links DROP CONSTRAINT IF EXISTS links_from_to_type_source_origin_unique;
ALTER TABLE links DROP COLUMN IF EXISTS link_source;
ALTER TABLE links DROP COLUMN IF EXISTS origin_page_id;
`);
await (engine as any).applyForwardReferenceBootstrap();
const { rows: lsCol } = await db.query(`
SELECT column_name FROM information_schema.columns
WHERE table_name = 'links' AND column_name = 'link_source'
`);
expect(lsCol).toHaveLength(1);
const { rows: opCol } = await db.query(`
SELECT column_name FROM information_schema.columns
WHERE table_name = 'links' AND column_name = 'origin_page_id'
`);
expect(opCol).toHaveLength(1);
} finally {
await engine.disconnect();
}
}, 30000);
});
+14 -4
View File
@@ -1,4 +1,4 @@
import { describe, expect, test, beforeEach, afterEach } from 'bun:test';
import { describe, expect, test, beforeAll, afterAll, beforeEach, afterEach } from 'bun:test';
import { mkdtempSync, rmSync, writeFileSync, readFileSync, existsSync, mkdirSync, symlinkSync } from 'fs';
import { join } from 'path';
import { tmpdir } from 'os';
@@ -9,6 +9,7 @@ import {
BrainWriterError,
} from '../src/core/brain-writer.ts';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { resetPgliteState } from './helpers/reset-pglite.ts';
const fence = '---';
@@ -115,15 +116,24 @@ describe('scanBrainSources (PGLite)', () => {
let tmp: string;
let engine: PGLiteEngine;
beforeEach(async () => {
tmp = mkdtempSync(join(tmpdir(), 'brain-writer-scan-'));
// One PGLite per file — beforeEach wipes data only. PGLite cold-start is
// ~20s on CI; sharing one engine across 6 tests in this block saves ~2 min.
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
});
afterEach(async () => {
afterAll(async () => {
await engine.disconnect();
});
beforeEach(async () => {
await resetPgliteState(engine);
tmp = mkdtempSync(join(tmpdir(), 'brain-writer-scan-'));
});
afterEach(() => {
rmSync(tmp, { recursive: true, force: true });
});
+165
View File
@@ -0,0 +1,165 @@
/**
* gbrain claw-test CLI dispatch tests.
*
* These tests exercise the harness's argument parsing, scenario loading,
* agent registry resolution, and friction-report path. They do NOT spawn
* real gbrain commands (no built binary in CI yet); the canonical scripted
* E2E that walks `gbrain init → import → query → extract → verify` lives
* in test/e2e/claw-test.test.ts and gates on a built binary.
*/
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
import { mkdtempSync, rmSync, existsSync, readFileSync } from 'fs';
import { join } from 'path';
import { tmpdir } from 'os';
import { runFriction } from '../src/commands/friction.ts';
import { listScenarios, loadScenario } from '../src/core/claw-test/scenarios.ts';
import {
registerAgentRunner, resolveAgentRunner, listRegisteredAgents,
_resetRegistryForTests,
type AgentRunner, type DetectResult, type InvokeOpts, type InvokeResult,
} from '../src/core/claw-test/agent-runner.ts';
let tmp: string;
const ORIG_HOME = process.env.GBRAIN_HOME;
beforeEach(() => {
tmp = mkdtempSync(join(tmpdir(), 'claw-test-cli-'));
process.env.GBRAIN_HOME = tmp;
_resetRegistryForTests();
});
afterEach(() => {
process.env.GBRAIN_HOME = ORIG_HOME;
rmSync(tmp, { recursive: true, force: true });
});
describe('shipped scenarios are loadable', () => {
test('default fixtures root contains both v1 scenarios', () => {
delete process.env.GBRAIN_CLAW_SCENARIOS_DIR;
const names = listScenarios();
expect(names).toContain('fresh-install');
expect(names).toContain('upgrade-from-v0.18');
});
test('fresh-install has expected_phases', () => {
delete process.env.GBRAIN_CLAW_SCENARIOS_DIR;
const cfg = loadScenario('fresh-install');
expect(cfg.expectedPhases).toContain('import.files');
expect(cfg.expectedPhases).toContain('extract.links_fs');
expect(cfg.expectedPhases).toContain('doctor.db_checks');
});
test('upgrade-from-v0.18 declares from_version', () => {
delete process.env.GBRAIN_CLAW_SCENARIOS_DIR;
const cfg = loadScenario('upgrade-from-v0.18');
expect(cfg.kind).toBe('upgrade');
expect(cfg.fromVersion).toBe('0.18.0');
expect(cfg.seedRelative).toBe('seed');
});
});
describe('agent registry — fake-runner integration', () => {
test('a fake runner can be registered, resolved, and detect/invoke called', async () => {
let invokeCount = 0;
class FakeRunner implements AgentRunner {
readonly name = 'fake';
async detect(): Promise<DetectResult> { return { available: true, binPath: '/usr/bin/fake' }; }
async invoke(_opts: InvokeOpts): Promise<InvokeResult> {
invokeCount++;
return { exitCode: 0, durationMs: 1 };
}
}
registerAgentRunner('fake', () => new FakeRunner());
expect(listRegisteredAgents()).toContain('fake');
const r = resolveAgentRunner('fake');
const detected = await r.detect();
expect(detected.available).toBe(true);
const result = await r.invoke({
cwd: tmp,
brief: 'test',
env: {},
timeoutMs: 1000,
transcriptSink: { write: () => {}, nextOffset: () => 0, close: async () => {} },
});
expect(result.exitCode).toBe(0);
expect(invokeCount).toBe(1);
});
test('resolveAgentRunner with unknown name throws with registered list', () => {
registerAgentRunner('alpha', () => ({} as AgentRunner));
expect(() => resolveAgentRunner('unknown')).toThrow(/registered: alpha/);
});
});
describe('friction CLI integrates with harness run-id env', () => {
test('GBRAIN_FRICTION_RUN_ID populates harness-style run-ids', () => {
process.env.GBRAIN_FRICTION_RUN_ID = 'claw-test-20260428-fake-abcd1234';
try {
const code = runFriction(['log', '--phase', 'install', '--message', 'simulated harness write']);
expect(code).toBe(0);
const expectedFile = join(tmp, '.gbrain', 'friction', 'claw-test-20260428-fake-abcd1234.jsonl');
expect(existsSync(expectedFile)).toBe(true);
const raw = readFileSync(expectedFile, 'utf-8');
const entry = JSON.parse(raw.split('\n')[0]);
expect(entry.run_id).toBe('claw-test-20260428-fake-abcd1234');
expect(entry.message).toBe('simulated harness write');
} finally {
delete process.env.GBRAIN_FRICTION_RUN_ID;
}
});
});
describe('OpenClawRunner detection (reliable on box without openclaw)', () => {
test('detect returns unavailable when OPENCLAW_BIN missing', async () => {
const orig = process.env.OPENCLAW_BIN;
delete process.env.OPENCLAW_BIN;
try {
const { OpenClawRunner } = await import('../src/core/claw-test/runners/openclaw.ts');
const r = new OpenClawRunner();
const d = await r.detect();
// Either unavailable, or available if openclaw IS on PATH for the dev — both states are valid.
// We only assert the contract shape.
expect(typeof d.available).toBe('boolean');
if (!d.available) {
expect(typeof d.reason).toBe('string');
} else {
expect(d.binPath?.startsWith('/')).toBe(true);
}
} finally {
if (orig !== undefined) process.env.OPENCLAW_BIN = orig;
}
});
test('detect rejects relative OPENCLAW_BIN', async () => {
const orig = process.env.OPENCLAW_BIN;
process.env.OPENCLAW_BIN = 'relative/openclaw';
try {
const { OpenClawRunner } = await import('../src/core/claw-test/runners/openclaw.ts');
const r = new OpenClawRunner();
const d = await r.detect();
expect(d.available).toBe(false);
expect(d.reason).toMatch(/absolute/);
} finally {
if (orig !== undefined) process.env.OPENCLAW_BIN = orig;
else delete process.env.OPENCLAW_BIN;
}
});
test("detect rejects '..' segments in OPENCLAW_BIN", async () => {
const orig = process.env.OPENCLAW_BIN;
process.env.OPENCLAW_BIN = '/tmp/foo/../bar';
try {
const { OpenClawRunner } = await import('../src/core/claw-test/runners/openclaw.ts');
const r = new OpenClawRunner();
const d = await r.detect();
expect(d.available).toBe(false);
expect(d.reason).toMatch(/'\.\.' segments/);
} finally {
if (orig !== undefined) process.env.OPENCLAW_BIN = orig;
else delete process.env.OPENCLAW_BIN;
}
});
});
+1 -1
View File
@@ -108,7 +108,7 @@ describe('autopilot-cycle handler contract (v0.20.5)', () => {
// This is a source-level regression guard
const handlerBlock = jobsSource.slice(
jobsSource.indexOf("worker.register('autopilot-cycle'"),
jobsSource.indexOf("worker.register('autopilot-cycle'") + 500,
jobsSource.indexOf("worker.register('autopilot-cycle'") + 2000,
);
expect(handlerBlock).toContain('signal: job.signal');
+92
View File
@@ -0,0 +1,92 @@
/**
* Tests for src/core/disk-walk.ts single-walk filesystem scan.
*
* Replaces the per-page existsSync+statSync syscall storm in storage.ts
* (Issue #14 of the v0.22.3 eng review).
*/
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
import { mkdtempSync, mkdirSync, rmSync, writeFileSync } from 'fs';
import { join } from 'path';
import { tmpdir } from 'os';
import { walkBrainRepo } from '../src/core/disk-walk.ts';
let tmp: string;
beforeEach(() => {
tmp = mkdtempSync(join(tmpdir(), 'gbrain-walk-test-'));
});
afterEach(() => {
rmSync(tmp, { recursive: true, force: true });
});
function write(relPath: string, content: string): void {
const full = join(tmp, relPath);
mkdirSync(join(full, '..'), { recursive: true });
writeFileSync(full, content);
}
describe('walkBrainRepo', () => {
test('returns empty map for empty directory', () => {
expect(walkBrainRepo(tmp).size).toBe(0);
});
test('returns empty map for nonexistent directory', () => {
expect(walkBrainRepo(join(tmp, 'does-not-exist')).size).toBe(0);
});
test('finds top-level .md files keyed by slug (no .md suffix)', () => {
write('alice.md', '# Alice');
const result = walkBrainRepo(tmp);
expect(result.has('alice')).toBe(true);
expect(result.get('alice')!.size).toBeGreaterThan(0);
});
test('walks nested directories and produces slash-joined slugs', () => {
write('people/alice.md', '# Alice');
write('media/x/tweet-1.md', 'tweet');
write('media/articles/post-1.md', 'post');
const result = walkBrainRepo(tmp);
expect(new Set(result.keys())).toEqual(
new Set(['people/alice', 'media/x/tweet-1', 'media/articles/post-1']),
);
});
test('skips dot-directories (.git, .gbrain, .vscode)', () => {
write('.git/HEAD', 'ref: refs/heads/main');
write('.gbrain/config.json', '{}');
write('.vscode/settings.json', '{}');
write('people/alice.md', '# Alice');
const result = walkBrainRepo(tmp);
expect(new Set(result.keys())).toEqual(new Set(['people/alice']));
});
test('skips node_modules', () => {
write('node_modules/foo/bar.md', 'noise');
write('people/alice.md', '# Alice');
const result = walkBrainRepo(tmp);
expect(new Set(result.keys())).toEqual(new Set(['people/alice']));
});
test('ignores non-.md files', () => {
write('people/alice.md', '# Alice');
write('people/alice.json', '{}');
write('people/photo.png', 'binary');
const result = walkBrainRepo(tmp);
expect(new Set(result.keys())).toEqual(new Set(['people/alice']));
});
test('captures size from stat', () => {
const content = '# Alice\n'.repeat(100);
write('people/alice.md', content);
const result = walkBrainRepo(tmp);
expect(result.get('people/alice')!.size).toBe(content.length);
});
test('captures mtimeMs', () => {
write('people/alice.md', '# Alice');
const result = walkBrainRepo(tmp);
expect(result.get('people/alice')!.mtimeMs).toBeGreaterThan(0);
});
});
+134
View File
@@ -0,0 +1,134 @@
/**
* gbrain claw-test scripted-mode E2E.
*
* Invokes the harness via `bun run src/cli.ts` (NOT a compiled binary
* `bun build --compile` doesn't bundle PGLite's runtime assets like
* pglite.data, so a compiled gbrain can't init a fresh PGLite brain).
* Uses a tiny shim script that the harness can spawn as if it were the
* gbrain binary.
*
* Asserts:
* - exit code 0 on a clean tree
* - the friction JSONL has zero error/blocker entries
* - the harness recorded progress events for the expected phases
*
* Tagged-skip env: CLAW_TEST_SKIP_E2E=1 to opt out (e.g. when PGLite
* WASM is broken on the host the macOS 26.3 #223 bug class).
*/
import { describe, test, expect, beforeAll } from 'bun:test';
import { execFileSync, spawnSync } from 'child_process';
import { mkdirSync, existsSync, mkdtempSync, rmSync, readFileSync, readdirSync, writeFileSync, chmodSync } from 'fs';
import { tmpdir } from 'os';
import { join, resolve } from 'path';
const REPO_ROOT = resolve(import.meta.dir, '..', '..');
const BIN_CACHE = join(REPO_ROOT, 'test', '.cache');
const BIN_PATH = join(BIN_CACHE, 'gbrain.sh');
const SCENARIOS_DIR = join(REPO_ROOT, 'test', 'fixtures', 'claw-test-scenarios');
beforeAll(() => {
if (!existsSync(BIN_CACHE)) mkdirSync(BIN_CACHE, { recursive: true });
// Shim that delegates to `bun run src/cli.ts` so PGLite assets resolve from
// the source tree (bun --compile doesn't bundle them). Marked executable so
// child_process.spawn can run it directly.
const shim = `#!/bin/sh\nexec bun run "${join(REPO_ROOT, 'src', 'cli.ts')}" "$@"\n`;
writeFileSync(BIN_PATH, shim, 'utf-8');
chmodSync(BIN_PATH, 0o755);
}, 30_000);
describe('gbrain claw-test --scenario fresh-install (scripted)', () => {
test('runs end-to-end clean and produces zero error/blocker friction', () => {
const tmp = mkdtempSync(join(tmpdir(), 'claw-test-e2e-fresh-'));
try {
const result = spawnSync(BIN_PATH, ['claw-test', '--scenario', 'fresh-install', '--keep-tempdir'], {
cwd: REPO_ROOT,
env: {
...process.env,
GBRAIN_HOME: tmp,
GBRAIN_BIN_OVERRIDE: BIN_PATH,
GBRAIN_CLAW_SCENARIOS_DIR: join(REPO_ROOT, 'test', 'fixtures', 'claw-test-scenarios'),
},
encoding: 'utf-8',
timeout: 120_000,
});
if (result.status !== 0) {
console.error('STDOUT:', result.stdout);
console.error('STDERR:', result.stderr);
}
expect(result.status).toBe(0);
// Inspect the friction JSONL the harness wrote.
const frictionDir = join(tmp, '.gbrain', 'friction');
expect(existsSync(frictionDir)).toBe(true);
const files = readdirSync(frictionDir).filter(f => f.endsWith('.jsonl'));
expect(files.length).toBeGreaterThan(0);
const runFile = join(frictionDir, files[0]);
const lines = readFileSync(runFile, 'utf-8').split('\n').filter(l => l.trim());
const entries = lines.map(l => JSON.parse(l));
const blockers = entries.filter(e => e.kind === 'friction' && (e.severity === 'error' || e.severity === 'blocker'));
if (blockers.length > 0) {
console.error('unexpected friction entries:', blockers);
}
expect(blockers.length).toBe(0);
} finally {
rmSync(tmp, { recursive: true, force: true });
}
}, 180_000);
test('break path: an invented command produces an error friction entry and exits non-zero', () => {
// We do this by setting GBRAIN_BIN_OVERRIDE to a script that pretends to be gbrain
// and rejects the `import` subcommand specifically.
const tmp = mkdtempSync(join(tmpdir(), 'claw-test-e2e-break-'));
const fakeBin = join(tmp, 'fake-gbrain');
try {
// Write a shim that delegates to real gbrain but rejects 'import' to simulate breakage.
const shimContent = `#!/bin/sh\nif [ "$1" = "import" ]; then echo "fake import error" >&2; exit 17; fi\nexec "${BIN_PATH}" "$@"\n`;
const { writeFileSync, chmodSync } = require('fs');
writeFileSync(fakeBin, shimContent, 'utf-8');
chmodSync(fakeBin, 0o755);
const result = spawnSync(BIN_PATH, ['claw-test', '--scenario', 'fresh-install', '--keep-tempdir'], {
cwd: REPO_ROOT,
env: {
...process.env,
GBRAIN_HOME: tmp,
GBRAIN_BIN_OVERRIDE: fakeBin,
GBRAIN_CLAW_SCENARIOS_DIR: join(REPO_ROOT, 'test', 'fixtures', 'claw-test-scenarios'),
},
encoding: 'utf-8',
timeout: 60_000,
});
expect(result.status).not.toBe(0);
// The friction log should have an error-severity entry for the 'import' phase.
const frictionDir = join(tmp, '.gbrain', 'friction');
const files = readdirSync(frictionDir).filter(f => f.endsWith('.jsonl'));
const lines = readFileSync(join(frictionDir, files[0]), 'utf-8').split('\n').filter(l => l.trim());
const entries = lines.map(l => JSON.parse(l));
const importErrors = entries.filter(e => e.phase === 'import' && e.severity === 'error');
expect(importErrors.length).toBeGreaterThan(0);
} finally {
rmSync(tmp, { recursive: true, force: true });
}
}, 90_000);
});
describe('gbrain friction render integration', () => {
test('render produces a markdown report with the redact placeholder', () => {
const tmp = mkdtempSync(join(tmpdir(), 'claw-test-e2e-render-'));
try {
// Log a friction entry with $HOME embedded, then render --redact md
const home = process.env.HOME ?? '/tmp';
const env = { ...process.env, GBRAIN_HOME: tmp, GBRAIN_FRICTION_RUN_ID: 'render-e2e' };
execFileSync(BIN_PATH, ['friction', 'log', '--phase', 'p', '--message', `error at ${home}/.gbrain/x`], { env, encoding: 'utf-8' });
const out = execFileSync(BIN_PATH, ['friction', 'render', '--run-id', 'render-e2e'], { env, encoding: 'utf-8' });
expect(out).toContain('# Friction report');
expect(out).toContain('<HOME>');
// --redact is the default for md, so home itself should not appear.
expect(out).not.toContain(home + '/.gbrain');
} finally {
rmSync(tmp, { recursive: true, force: true });
}
}, 30_000);
});
+233
View File
@@ -0,0 +1,233 @@
/**
* E2E tests for src/mcp/http-transport.ts against real Postgres.
*
* Catches schema drift (column-name typos that would slip past the unit suite's
* stubbed engine.sql) and proves the F1+F2+F3 dispatch pipeline works against a
* real handler doing real DB work. Also exercises the SQL-level last_used_at
* debounce against real Postgres semantics.
*
* Run: DATABASE_URL=... bun test test/e2e/http-transport.test.ts
*/
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import { createHash, randomBytes } from 'crypto';
import { startHttpTransport } from '../../src/mcp/http-transport.ts';
import { hasDatabase, setupDB, teardownDB, getEngine, getConn } from './helpers.ts';
const skip = !hasDatabase();
const describeE2E = skip ? describe.skip : describe;
if (skip) {
console.log('Skipping E2E http-transport tests (DATABASE_URL not set)');
}
interface ServerHandle {
port: number;
stop: () => Promise<void>;
}
function generateToken(): string {
return 'gbrain_test_' + randomBytes(16).toString('hex');
}
function hashToken(token: string): string {
return createHash('sha256').update(token).digest('hex');
}
async function startServer(): Promise<ServerHandle> {
const engine = getEngine();
const server = await startHttpTransport({ port: 0, engine: engine as any });
return {
port: (server as any).port,
stop: async () => { (server as any).stop(true); },
};
}
function rpc(method: string, params?: unknown, id: number = 1) {
return JSON.stringify({ jsonrpc: '2.0', id, method, ...(params !== undefined ? { params } : {}) });
}
describeE2E('http-transport E2E (real Postgres)', () => {
let srv: ServerHandle;
let validToken: string;
let revokedToken: string;
let validTokenName: string;
beforeAll(async () => {
await setupDB();
const conn = getConn();
// Seed a valid + revoked token directly via SQL (mirrors auth.ts's create path).
validToken = generateToken();
validTokenName = 'e2e-valid-' + randomBytes(4).toString('hex');
await conn.unsafe(
'INSERT INTO access_tokens (name, token_hash) VALUES ($1, $2)',
[validTokenName, hashToken(validToken)],
);
revokedToken = generateToken();
await conn.unsafe(
'INSERT INTO access_tokens (name, token_hash, revoked_at) VALUES ($1, $2, now())',
['e2e-revoked-' + randomBytes(4).toString('hex'), hashToken(revokedToken)],
);
srv = await startServer();
});
afterAll(async () => {
if (srv) await srv.stop();
await teardownDB();
});
test('1. /health → 200 with expected JSON shape', async () => {
const r = await fetch(`http://localhost:${srv.port}/health`);
expect(r.status).toBe(200);
const body = await r.json();
expect(body.status).toBe('ok');
expect(body.transport).toBe('http');
expect(body.version).toBeString();
});
test('2. /mcp tools/list with valid Bearer → 200 + ops list', async () => {
const r = await fetch(`http://localhost:${srv.port}/mcp`, {
method: 'POST',
headers: { 'Authorization': `Bearer ${validToken}`, 'Content-Type': 'application/json' },
body: rpc('tools/list'),
});
expect(r.status).toBe(200);
const body = await r.json();
expect(body.result.tools).toBeArray();
expect(body.result.tools.length).toBeGreaterThan(5);
expect(r.headers.get('content-type')).toContain('application/json');
});
test('3. /mcp tools/call (real op: list_pages) round-trips successfully — F1+F2+F3 guard', async () => {
const r = await fetch(`http://localhost:${srv.port}/mcp`, {
method: 'POST',
headers: { 'Authorization': `Bearer ${validToken}`, 'Content-Type': 'application/json' },
body: rpc('tools/call', { name: 'list_pages', arguments: { limit: 5 } }),
});
expect(r.status).toBe(200);
const body = await r.json();
expect(body.jsonrpc).toBe('2.0');
expect(body.result.content).toBeArray();
// Should NOT be an error — handler ran successfully against the real engine.
expect(body.result.isError).toBeUndefined();
// Result text should parse as JSON (list_pages returns an object/array)
const resultText = body.result.content[0].text;
const parsed = JSON.parse(resultText);
expect(parsed).toBeDefined();
});
test('4. revoked token → 401', async () => {
const r = await fetch(`http://localhost:${srv.port}/mcp`, {
method: 'POST',
headers: { 'Authorization': `Bearer ${revokedToken}`, 'Content-Type': 'application/json' },
body: rpc('tools/list'),
});
expect(r.status).toBe(401);
});
test('5. last_used_at debounce: two consecutive valid calls → only one UPDATE within 60s', async () => {
const conn = getConn();
// Reset last_used_at to NULL so the first call definitely updates
await conn.unsafe('UPDATE access_tokens SET last_used_at = NULL WHERE name = $1', [validTokenName]);
// First request — should update last_used_at
await fetch(`http://localhost:${srv.port}/mcp`, {
method: 'POST',
headers: { 'Authorization': `Bearer ${validToken}`, 'Content-Type': 'application/json' },
body: rpc('tools/list'),
});
// Give the fire-and-forget UPDATE a moment to land
await new Promise(r => setTimeout(r, 50));
const [row1] = await conn.unsafe(
'SELECT last_used_at FROM access_tokens WHERE name = $1',
[validTokenName],
) as { last_used_at: Date | null }[];
expect(row1.last_used_at).not.toBeNull();
const firstUpdate = row1.last_used_at;
// Second request immediately — should NOT trigger another UPDATE (debounced by SQL WHERE)
await fetch(`http://localhost:${srv.port}/mcp`, {
method: 'POST',
headers: { 'Authorization': `Bearer ${validToken}`, 'Content-Type': 'application/json' },
body: rpc('tools/list'),
});
await new Promise(r => setTimeout(r, 50));
const [row2] = await conn.unsafe(
'SELECT last_used_at FROM access_tokens WHERE name = $1',
[validTokenName],
) as { last_used_at: Date | null }[];
// Same timestamp = same UPDATE = debounce held
expect(row2.last_used_at?.getTime()).toBe(firstUpdate?.getTime());
});
test('6. last_used_at debounce: simulating 65s gap → second request DOES update', async () => {
const conn = getConn();
// Set last_used_at to 65 seconds ago — simulates the time gap without waiting in real time
await conn.unsafe(
`UPDATE access_tokens SET last_used_at = now() - interval '65 seconds' WHERE name = $1`,
[validTokenName],
);
const [before] = await conn.unsafe(
'SELECT last_used_at FROM access_tokens WHERE name = $1',
[validTokenName],
) as { last_used_at: Date | null }[];
await fetch(`http://localhost:${srv.port}/mcp`, {
method: 'POST',
headers: { 'Authorization': `Bearer ${validToken}`, 'Content-Type': 'application/json' },
body: rpc('tools/list'),
});
await new Promise(r => setTimeout(r, 50));
const [after] = await conn.unsafe(
'SELECT last_used_at FROM access_tokens WHERE name = $1',
[validTokenName],
) as { last_used_at: Date | null }[];
expect(after.last_used_at?.getTime()).toBeGreaterThan(before.last_used_at!.getTime());
});
test('7. mcp_request_log gets a row per request', async () => {
const conn = getConn();
const beforeRows = await conn.unsafe('SELECT count(*)::int AS n FROM mcp_request_log') as { n: number }[];
const beforeN = beforeRows[0].n;
await fetch(`http://localhost:${srv.port}/mcp`, {
method: 'POST',
headers: { 'Authorization': `Bearer ${validToken}`, 'Content-Type': 'application/json' },
body: rpc('tools/list'),
});
// Fire-and-forget audit insert — give it a tick
await new Promise(r => setTimeout(r, 100));
const afterRows = await conn.unsafe('SELECT count(*)::int AS n FROM mcp_request_log') as { n: number }[];
expect(afterRows[0].n).toBeGreaterThan(beforeN);
const [row] = await conn.unsafe(
`SELECT token_name, operation, status, latency_ms FROM mcp_request_log
WHERE token_name = $1 ORDER BY created_at DESC LIMIT 1`,
[validTokenName],
) as { token_name: string; operation: string; status: string; latency_ms: number }[];
expect(row.token_name).toBe(validTokenName);
expect(row.operation).toBe('tools/list');
expect(row.status).toBe('success');
expect(row.latency_ms).toBeGreaterThanOrEqual(0);
});
test('8. tools/call with malformed params → isError result with invalid_params', async () => {
const r = await fetch(`http://localhost:${srv.port}/mcp`, {
method: 'POST',
headers: { 'Authorization': `Bearer ${validToken}`, 'Content-Type': 'application/json' },
body: rpc('tools/call', { name: 'get_page', arguments: { slug: 42 } }),
});
expect(r.status).toBe(200);
const body = await r.json();
expect(body.result.isError).toBe(true);
expect(body.result.content[0].text).toContain('invalid_params');
});
});
+168
View File
@@ -0,0 +1,168 @@
/**
* E2E parity tests scanIntegrity batch path vs sequential path.
*
* The batch path (Postgres-only fast path added in v0.20.x) and the sequential
* path (engine.getAllSlugs + getPage loop) MUST return the same result for
* every supported case, otherwise gbrain doctor reports different numbers
* depending on engine type or whether batch was attempted.
*
* Codex review of the original perf commit caught a multi-source dedup
* regression: the batch SQL scanned raw (source_id, slug) rows while
* sequential's getAllSlugs() returned a Set<string>. v0.22.7 adds
* SELECT DISTINCT ON (slug) to the batch SQL; these tests prove parity.
*
* Run: DATABASE_URL=... bun test test/e2e/integrity-batch.test.ts
*/
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
import { hasDatabase, setupDB, teardownDB, getEngine, getConn } from './helpers.ts';
import { scanIntegrity } from '../../src/commands/integrity.ts';
const skip = !hasDatabase();
const describeE2E = skip ? describe.skip : describe;
if (skip) {
console.log('Skipping E2E integrity batch parity tests (DATABASE_URL not set)');
}
describeE2E('scanIntegrity batch parity (E2E, Postgres-only)', () => {
beforeAll(async () => {
await setupDB();
});
afterAll(async () => {
await teardownDB();
});
beforeEach(async () => {
// Clean slate per case so fixtures don't leak across describes.
const conn = getConn();
await conn.unsafe(`TRUNCATE pages CASCADE`);
});
describe('dedup', () => {
test('multi-source duplicate slugs scan once, not once-per-source', async () => {
const engine = getEngine();
const conn = getConn();
// Seed default-source page via the engine.
await engine.putPage('people/alice', {
type: 'person',
title: 'Alice',
compiled_truth: 'Alice writes about AI safety.',
timeline: '',
frontmatter: {},
});
// Seed alt-source row via raw SQL — engine.putPage doesn't take a source_id,
// and we specifically need to test that DISTINCT ON (slug) collapses
// the multi-source rows into one scan.
await conn.unsafe(`
INSERT INTO sources (id, name) VALUES ('test-source-2', 'test-source-2')
ON CONFLICT DO NOTHING
`);
await conn.unsafe(`
INSERT INTO pages (source_id, slug, type, title, compiled_truth, timeline, frontmatter)
VALUES ('test-source-2', 'people/alice', 'person', 'Alice (alt source)',
'Alice from another source.', '', '{}'::jsonb)
`);
const batchResult = await scanIntegrity(engine, { limit: 100, batchLoad: true });
const seqResult = await scanIntegrity(engine, { limit: 100, batchLoad: false });
// Both paths must report the same number of distinct slugs scanned.
// Pre-fix: batch reported 2 (one per source row), sequential reported 1.
expect(batchResult.pagesScanned).toBe(seqResult.pagesScanned);
expect(batchResult.pagesScanned).toBe(1);
});
});
describe('hits', () => {
test('bareHits and externalHits arrays match between paths', async () => {
const engine = getEngine();
await engine.putPage('people/alice', {
type: 'person',
title: 'Alice',
compiled_truth: 'Alice tweeted about AI safety last week.',
timeline: '',
frontmatter: {},
});
await engine.putPage('people/bob', {
type: 'person',
title: 'Bob',
compiled_truth: 'Bob wrote at [example](https://example.com/bob).',
timeline: '',
frontmatter: {},
});
const batchResult = await scanIntegrity(engine, { limit: 100, batchLoad: true });
const seqResult = await scanIntegrity(engine, { limit: 100, batchLoad: false });
expect(batchResult.bareHits.length).toBe(seqResult.bareHits.length);
expect(batchResult.externalHits.length).toBe(seqResult.externalHits.length);
expect(batchResult.bareHits.map(h => h.slug).sort()).toEqual(
seqResult.bareHits.map(h => h.slug).sort(),
);
expect(batchResult.externalHits.map(h => h.slug).sort()).toEqual(
seqResult.externalHits.map(h => h.slug).sort(),
);
});
});
describe('validate', () => {
test('validate:false (boolean) page is skipped on both paths', async () => {
const engine = getEngine();
await engine.putPage('people/alice', {
type: 'person',
title: 'Alice',
compiled_truth: 'Alice tweeted about something.',
timeline: '',
frontmatter: {},
});
await engine.putPage('people/legacy', {
type: 'person',
title: 'Legacy',
compiled_truth: 'Legacy tweeted about old stuff.',
timeline: '',
frontmatter: { validate: false },
});
const batchResult = await scanIntegrity(engine, { limit: 100, batchLoad: true });
const seqResult = await scanIntegrity(engine, { limit: 100, batchLoad: false });
expect(batchResult.pagesScanned).toBe(seqResult.pagesScanned);
expect(batchResult.pagesScanned).toBe(1);
expect(batchResult.bareHits.map(h => h.slug)).not.toContain('people/legacy');
expect(seqResult.bareHits.map(h => h.slug)).not.toContain('people/legacy');
});
});
describe('topPages', () => {
test('topPages ordering matches between paths', async () => {
const engine = getEngine();
// Alice has 2 bare-tweet hits; Bob has 1.
await engine.putPage('people/alice', {
type: 'person',
title: 'Alice',
compiled_truth: 'Alice tweeted today. Alice tweeted yesterday too.',
timeline: '',
frontmatter: {},
});
await engine.putPage('people/bob', {
type: 'person',
title: 'Bob',
compiled_truth: 'Bob tweeted once.',
timeline: '',
frontmatter: {},
});
const batchResult = await scanIntegrity(engine, { limit: 100, batchLoad: true });
const seqResult = await scanIntegrity(engine, { limit: 100, batchLoad: false });
expect(batchResult.topPages).toEqual(seqResult.topPages);
});
});
});
+1 -1
View File
@@ -43,7 +43,7 @@ beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({}); // in-memory PGLite
await engine.initSchema(); // installs pages, minion_jobs, config, etc.
});
}, 30000);
afterAll(async () => {
await engine.disconnect();
+93
View File
@@ -0,0 +1,93 @@
/**
* E2E test for PostgresEngine forward-reference bootstrap.
*
* Codex caught that `test/e2e/helpers.ts:74` uses the standalone
* `db.initSchema()` from `src/core/db.ts`, which only runs SCHEMA_SQL and
* never calls runMigrations(). A test using that helper would NOT exercise
* `PostgresEngine.initSchema()`'s reordered path, producing false-positive
* coverage. This test deliberately bypasses the standard helper and
* instantiates `PostgresEngine` directly, calling `engine.initSchema()` so
* the bootstrap SCHEMA_SQL runMigrations sequence runs end-to-end.
*
* Covers issues #366, #375, #378 Postgres-side wedges where pre-v0.18
* brains crashed on `column "source_id" does not exist`.
*
* NOTE: snapshot-based historical state simulation is out of scope for this
* wave (would require maintaining historical schema dumps). The test
* mutates a fresh-LATEST brain to a pre-v0.18 shape; codex flagged this as
* approximate. Acceptable here because the bootstrap's contract is narrow:
* "given a brain that lacks the specific forward-references, initSchema
* produces a brain at LATEST." The test exercises exactly that contract.
*
* Run: DATABASE_URL=postgresql://... bun run test:e2e test/e2e/postgres-bootstrap.test.ts
*/
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import { PostgresEngine } from '../../src/core/postgres-engine.ts';
import { LATEST_VERSION } from '../../src/core/migrate.ts';
const DATABASE_URL = process.env.DATABASE_URL;
const skip = !DATABASE_URL;
describe.skipIf(skip)('PostgresEngine forward-reference bootstrap (E2E)', () => {
let engine: PostgresEngine;
beforeAll(async () => {
engine = new PostgresEngine();
await engine.connect({ database_url: DATABASE_URL! });
});
afterAll(async () => {
await engine.disconnect();
});
test('PostgresEngine.initSchema applies bootstrap → SCHEMA_SQL → migrations on pre-v0.18 brain', async () => {
// First call: bring the test DB to LATEST shape so we have something to mutate.
await engine.initSchema();
// Clear data from prior tests in the suite. Adding a UNIQUE(slug)
// constraint below would fail if multi-source fixtures left rows with
// duplicate slugs across sources (which is valid under the composite
// UNIQUE this test is undoing).
const conn = (engine as any).sql;
await conn.unsafe(`TRUNCATE pages, content_chunks, links, tags, raw_data, timeline_entries, page_versions, ingest_log RESTART IDENTITY CASCADE`);
// Mutate to pre-v0.18 shape: drop source_id and the sources table.
// The advisory lock is released between initSchema calls, so this
// direct DDL won't deadlock.
await conn.unsafe(`
ALTER TABLE pages DROP CONSTRAINT IF EXISTS pages_source_slug_key;
ALTER TABLE pages ADD CONSTRAINT pages_slug_key UNIQUE (slug);
DROP INDEX IF EXISTS idx_pages_source_id;
ALTER TABLE pages DROP COLUMN IF EXISTS source_id CASCADE;
DROP TABLE IF EXISTS sources CASCADE;
`);
await engine.setConfig('version', '20');
// The path under test: full PostgresEngine.initSchema() including the
// bootstrap call, SCHEMA_SQL replay, and runMigrations chain.
await engine.initSchema();
expect(await engine.getConfig('version')).toBe(String(LATEST_VERSION));
// Verify the forward-referenced column exists after upgrade.
const colCheck = await conn`
SELECT column_name FROM information_schema.columns
WHERE table_schema = current_schema()
AND table_name = 'pages'
AND column_name = 'source_id'
`;
expect(colCheck).toHaveLength(1);
// Verify the default source row was seeded.
const srcCheck = await conn`SELECT id FROM sources WHERE id = 'default'`;
expect(srcCheck).toHaveLength(1);
});
test('PostgresEngine.initSchema is idempotent on a brain already at LATEST', async () => {
// Fresh-LATEST brain. Calling initSchema again must not error and must
// not regress the version.
await engine.initSchema();
expect(await engine.getConfig('version')).toBe(String(LATEST_VERSION));
});
});
+273
View File
@@ -0,0 +1,273 @@
/**
* E2E test for storage tiering Postgres-only.
*
* Per the v0.23.0 plan: full lifecycle. Container restart simulation:
* write pages via Postgres, delete files from disk, run gbrain export
* --restore-only, assert files restored. Real .gitignore round-trip.
* Real source-resolver path through getDefaultSourcePath().
*
* Skips gracefully when DATABASE_URL is unset (per CLAUDE.md E2E pattern).
*/
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
import { mkdtempSync, rmSync, writeFileSync, readFileSync, existsSync, mkdirSync } from 'fs';
import { join } from 'path';
import { tmpdir } from 'os';
import { setupDB, teardownDB, getEngine, hasDatabase, getConn } from './helpers.ts';
import {
getStorageStatus,
formatStorageStatusHuman,
__resetPGLiteWarn,
} from '../../src/commands/storage.ts';
import { manageGitignore, __resetPGLiteTierWarn } from '../../src/commands/sync.ts';
import { getDefaultSourcePath } from '../../src/core/source-resolver.ts';
import { __resetMissingStorageWarning } from '../../src/core/storage-config.ts';
if (!hasDatabase()) {
describe('storage-tiering E2E', () => {
test.skip('DATABASE_URL not set — skipping E2E', () => {});
});
} else {
describe('storage-tiering E2E (Postgres lifecycle)', () => {
let tmp: string;
beforeAll(async () => {
await setupDB();
});
afterAll(async () => {
await teardownDB();
});
beforeEach(() => {
tmp = mkdtempSync(join(tmpdir(), 'gbrain-e2e-storage-'));
__resetMissingStorageWarning();
__resetPGLiteWarn();
__resetPGLiteTierWarn();
});
function cleanup(): void {
rmSync(tmp, { recursive: true, force: true });
}
function writeGbrainYml(): void {
writeFileSync(
join(tmp, 'gbrain.yml'),
`storage:
db_tracked:
- people/
db_only:
- media/x/
- media/articles/
`,
);
}
test('engine.kind is postgres', () => {
try {
expect(getEngine().kind).toBe('postgres');
} finally {
cleanup();
}
});
test('full lifecycle: write pages → status reports tiers → manage .gitignore → restore-only path', async () => {
try {
const engine = getEngine();
// Truncate sources + pages so this test has a clean slate.
const conn = getConn();
await conn.unsafe(`TRUNCATE pages, content_chunks, sources CASCADE`);
await conn.unsafe(
`INSERT INTO sources (id, name, local_path) VALUES ('default', 'Default', $1)`,
[tmp],
);
writeGbrainYml();
// Seed 4 pages: 1 db_tracked, 2 db_only, 1 unspecified.
await engine.putPage('people/alice', {
type: 'person',
title: 'Alice',
compiled_truth: 'Alice is a founder.',
timeline: '',
});
await engine.putPage('media/x/tweet-1', {
type: 'media',
title: 'Tweet 1',
compiled_truth: 'tweet body',
timeline: '',
});
await engine.putPage('media/x/tweet-2', {
type: 'media',
title: 'Tweet 2',
compiled_truth: 'tweet body 2',
timeline: '',
});
await engine.putPage('random/note', {
type: 'note',
title: 'Random',
compiled_truth: 'random',
timeline: '',
});
// Storage status reports tier counts correctly.
const status = await getStorageStatus(engine, tmp);
expect(status.totalPages).toBe(4);
expect(status.pagesByTier.db_tracked).toBe(1);
expect(status.pagesByTier.db_only).toBe(2);
expect(status.pagesByTier.unspecified).toBe(1);
// Human formatter renders without errors.
const out = formatStorageStatusHuman(status);
expect(out).toContain('DB tracked: 1 pages');
expect(out).toContain('DB only: 2 pages');
// .gitignore management: empty .gitignore → managed block written.
manageGitignore(tmp, 'postgres');
const gitignore = readFileSync(join(tmp, '.gitignore'), 'utf-8');
expect(gitignore).toContain('# Auto-managed by gbrain');
expect(gitignore).toContain('media/x/');
expect(gitignore).toContain('media/articles/');
// Idempotency: second run adds nothing new.
manageGitignore(tmp, 'postgres');
const gitignore2 = readFileSync(join(tmp, '.gitignore'), 'utf-8');
const xCount = (gitignore2.match(/^media\/x\/$/gm) || []).length;
expect(xCount).toBe(1);
// Source resolution finds the local_path we registered.
const resolvedPath = await getDefaultSourcePath(engine);
expect(resolvedPath).toBe(tmp);
} finally {
cleanup();
}
});
test('container restart simulation: db_only files missing on disk are restorable from DB', async () => {
try {
const engine = getEngine();
const conn = getConn();
// Fresh slate.
await conn.unsafe(`TRUNCATE pages, content_chunks, sources CASCADE`);
await conn.unsafe(
`INSERT INTO sources (id, name, local_path) VALUES ('default', 'Default', $1)`,
[tmp],
);
writeGbrainYml();
// Write some db_only pages to the database.
await engine.putPage('media/x/tweet-1', {
type: 'media',
title: 'Tweet 1',
compiled_truth: 'tweet body 1',
timeline: '',
});
await engine.putPage('media/x/tweet-2', {
type: 'media',
title: 'Tweet 2',
compiled_truth: 'tweet body 2',
timeline: '',
});
// Simulate "files were on disk, but the container restarted."
// Storage status: missingFiles should list them.
const status = await getStorageStatus(engine, tmp);
expect(status.pagesByTier.db_only).toBe(2);
expect(status.missingFiles.length).toBe(2);
// Verify slugPrefix engine filter (Issue #13) works on Postgres for
// the prefix that --restore-only would use.
const tierPages = await engine.listPages({ slugPrefix: 'media/x/', limit: 100 });
expect(tierPages.map((p) => p.slug).sort()).toEqual(['media/x/tweet-1', 'media/x/tweet-2']);
// Source-default path resolution returns the configured local_path
// (the typed accessor that replaces the original raw-SQL try/catch
// in storage.ts:38).
const path = await getDefaultSourcePath(engine);
expect(path).toBe(tmp);
} finally {
cleanup();
}
});
test('slugPrefix filter on Postgres uses index-based range scan (regression for Issue #13)', async () => {
try {
const engine = getEngine();
const conn = getConn();
await conn.unsafe(`TRUNCATE pages, content_chunks, sources CASCADE`);
await conn.unsafe(`INSERT INTO sources (id, name) VALUES ('default', 'Default')`);
// Seed enough data to make a difference between scan types.
for (let i = 0; i < 50; i++) {
await engine.putPage(`media/x/item-${i}`, {
type: 'media',
title: `Item ${i}`,
compiled_truth: 'x',
timeline: '',
});
}
for (let i = 0; i < 50; i++) {
await engine.putPage(`people/p-${i}`, {
type: 'person',
title: `Person ${i}`,
compiled_truth: 'x',
timeline: '',
});
}
// Prefix query should return exactly 50 (people not included).
const xResults = await engine.listPages({ slugPrefix: 'media/x/', limit: 200 });
expect(xResults.length).toBe(50);
for (const p of xResults) {
expect(p.slug.startsWith('media/x/')).toBe(true);
}
// Path-segment risk: slugPrefix 'media/x' (no /) would match
// 'media/xerox' if any existed. The engine treats slugPrefix as a
// literal string prefix; trailing-/ semantics are the matcher's
// responsibility (storage-config.ts).
const looseResults = await engine.listPages({ slugPrefix: 'media/x', limit: 200 });
expect(looseResults.length).toBe(50); // no media/xerox/* exists yet
} finally {
cleanup();
}
});
test('hard-error path: storage status without local_path or --repo gets null repoPath', async () => {
try {
const engine = getEngine();
const conn = getConn();
await conn.unsafe(`TRUNCATE sources CASCADE`);
// Default source with NO local_path.
await conn.unsafe(
`INSERT INTO sources (id, name, local_path) VALUES ('default', 'Default', NULL)`,
);
const path = await getDefaultSourcePath(engine);
expect(path).toBeNull();
} finally {
cleanup();
}
});
test('manageGitignore on Postgres engine does NOT emit PGLite warning', async () => {
try {
writeGbrainYml();
const warnings: string[] = [];
const orig = console.warn;
console.warn = (...a: unknown[]) => warnings.push(a.map(String).join(' '));
try {
manageGitignore(tmp, 'postgres');
} finally {
console.warn = orig;
}
expect(warnings.filter((w) => /limited effect on PGLite/.test(w))).toEqual([]);
} finally {
cleanup();
}
});
});
}
+167
View File
@@ -0,0 +1,167 @@
/**
* E2E test for parallel sync against real Postgres.
*
* T2 happy path: 60-file sync at concurrency=4 against PostgresEngine
* actually constructs N worker engines, imports correctly, and does
* not leak connections (probe pg_stat_activity before/after).
* P4 benchmark: serial vs concurrency=4 timing on the same fixture so
* the v0.22.13 CHANGELOG can quote a real number instead of "~4×".
*
* Gated on DATABASE_URL. Run via:
* docker run -d --name gbrain-test-pg -e POSTGRES_USER=postgres \
* -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=gbrain_test \
* -p 5435:5432 pgvector/pgvector:pg16
* DATABASE_URL=postgresql://postgres:postgres@localhost:5435/gbrain_test \
* bun test test/e2e/sync-parallel.test.ts
*/
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import { mkdtempSync, rmSync, writeFileSync, mkdirSync } from 'fs';
import { join } from 'path';
import { tmpdir } from 'os';
import { execSync } from 'child_process';
import { hasDatabase, setupDB, teardownDB, getEngine, getConn } from './helpers.ts';
const skip = !hasDatabase();
const describeE2E = skip ? describe.skip : describe;
if (skip) {
console.log('Skipping E2E sync-parallel tests (DATABASE_URL not set)');
}
function seedRepo(repoPath: string, fileCount: number): string {
execSync('git init', { cwd: repoPath, stdio: 'pipe' });
execSync('git config user.email "test@test.com"', { cwd: repoPath, stdio: 'pipe' });
execSync('git config user.name "Test"', { cwd: repoPath, stdio: 'pipe' });
mkdirSync(join(repoPath, 'people'), { recursive: true });
for (let i = 0; i < fileCount; i++) {
writeFileSync(join(repoPath, `people/p${i}.md`), [
'---',
'type: person',
`title: Person ${i}`,
'---',
'',
`Person ${i} body — some text long enough to chunk.`,
`Iteration index ${i}, generated by sync-parallel E2E.`,
].join('\n'));
}
execSync('git add -A && git commit -m "initial"', { cwd: repoPath, stdio: 'pipe' });
return execSync('git rev-parse HEAD', { cwd: repoPath, encoding: 'utf-8' }).trim();
}
async function activeConnections(): Promise<number> {
const conn = getConn();
const rows = await conn.unsafe(`
SELECT count(*) AS n FROM pg_stat_activity
WHERE datname = current_database()
AND state IS NOT NULL
`) as Array<{ n: string }>;
return parseInt(rows[0]?.n ?? '0', 10);
}
describeE2E('E2E sync-parallel: T2 happy path + leak probe', () => {
let repoPath: string;
beforeAll(async () => {
await setupDB();
});
afterAll(async () => {
if (repoPath) rmSync(repoPath, { recursive: true, force: true });
await teardownDB();
});
test('60-file Postgres sync at concurrency=4 imports all + no connection leak', async () => {
repoPath = mkdtempSync(join(tmpdir(), 'gbrain-e2e-par-'));
seedRepo(repoPath, 60);
const before = await activeConnections();
const { performSync } = await import('../../src/commands/sync.ts');
const engine = getEngine();
const result = await performSync(engine, {
repoPath,
noPull: true,
noEmbed: true,
concurrency: 4,
});
// First sync routes through performFullSync (delegates to runImport which
// also accepts --workers); status is 'first_sync'.
expect(result.status).toBe('first_sync');
const after = await activeConnections();
// Allow some slack — the helper engine + sync's normal pool stay open.
// Worker engines (4 × 2 = 8 connections) MUST have closed; if they
// hadn't, after - before would be at least 8.
expect(after - before).toBeLessThan(4);
// Verify pages are actually in the DB (via raw SQL — engine API also works).
const conn = getConn();
const pageRows = await conn.unsafe(
`SELECT count(*) AS n FROM pages WHERE slug LIKE 'people/p%'`,
) as Array<{ n: string }>;
const count = parseInt(pageRows[0]?.n ?? '0', 10);
expect(count).toBe(60);
}, 60_000);
});
describeE2E('E2E sync-parallel: P4 benchmark serial vs concurrency=4', () => {
let repoSerial: string;
let repoParallel: string;
beforeAll(async () => {
await setupDB();
});
afterAll(async () => {
if (repoSerial) rmSync(repoSerial, { recursive: true, force: true });
if (repoParallel) rmSync(repoParallel, { recursive: true, force: true });
await teardownDB();
});
test('120-file benchmark: report serial and parallel wall-clock', async () => {
// Two separate repos so neither sync's chunks bleed into the other.
repoSerial = mkdtempSync(join(tmpdir(), 'gbrain-bench-serial-'));
repoParallel = mkdtempSync(join(tmpdir(), 'gbrain-bench-parallel-'));
seedRepo(repoSerial, 120);
seedRepo(repoParallel, 120);
const { performSync } = await import('../../src/commands/sync.ts');
const engine = getEngine();
// Truncate between runs to keep the benchmark honest.
const conn = getConn();
const t1 = Date.now();
await performSync(engine, {
repoPath: repoSerial,
noPull: true,
noEmbed: true,
concurrency: 1,
});
const serialMs = Date.now() - t1;
// Wipe pages before second run so neither one is "incremental".
await conn.unsafe(`TRUNCATE pages CASCADE`);
await conn.unsafe(`TRUNCATE config CASCADE`);
const t2 = Date.now();
await performSync(engine, {
repoPath: repoParallel,
noPull: true,
noEmbed: true,
concurrency: 4,
});
const parallelMs = Date.now() - t2;
const speedup = (serialMs / parallelMs).toFixed(2);
// Emit as a single line stdout consumers can grep for.
console.log(`SYNC_PARALLEL_BENCH 120 files | serial=${serialMs}ms | parallel(4)=${parallelMs}ms | speedup=${speedup}x`);
// Soft assertion: parallel must not be slower than serial. The actual
// speedup ratio depends heavily on Postgres latency profile and is what
// the CHANGELOG quotes — don't gate the test on a specific multiplier.
expect(parallelMs).toBeLessThanOrEqual(serialMs * 1.5); // +50% slack for noisy CI
}, 120_000);
});
+162 -2
View File
@@ -10,10 +10,10 @@
*/
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import { mkdtempSync, writeFileSync, rmSync, mkdirSync, unlinkSync } from 'fs';
import { mkdtempSync, writeFileSync, rmSync, mkdirSync, unlinkSync, existsSync, readFileSync } from 'fs';
import { join } from 'path';
import { execSync } from 'child_process';
import { tmpdir } from 'os';
import { tmpdir, homedir } from 'os';
import {
hasDatabase, setupDB, teardownDB, getEngine,
} from './helpers.ts';
@@ -394,3 +394,163 @@ describeE2E('E2E: Git-to-DB Sync Pipeline', () => {
expect(page!.title).toBe('Draft Meeting Notes');
});
});
/**
* E2E: --skip-failed loop with structured error code summary.
*
* Closes the v0.22.12 ship-blocker gap from issue #500 the whole code path
* (record classify block skip doctor render second cycle) had only
* mocked-JSONL unit coverage. This is the integration test that proves the
* chain holds together with a real Postgres engine, real git history, and
* real frontmatter validation.
*
* Owns its own repo + sync-failures.jsonl lifecycle so it can't leak state
* into the shared describeE2E above. Saves and restores the user's real
* ~/.gbrain/sync-failures.jsonl so running E2E on a developer machine
* doesn't trash their local sync state.
*/
describeE2E('E2E: sync --skip-failed structured summary loop (v0.22.12, issue #500)', () => {
let repoPath: string;
const realFailuresPath = join(homedir(), '.gbrain', 'sync-failures.jsonl');
let savedFailuresContent: string | null = null;
beforeAll(async () => {
await setupDB();
// Save+clear the real ~/.gbrain/sync-failures.jsonl so the test starts from
// a known-empty state. Restored in afterAll. This file is per-machine, NOT
// per-repo, so we have to be defensive about a developer running this
// suite on their actual brain machine.
if (existsSync(realFailuresPath)) {
savedFailuresContent = readFileSync(realFailuresPath, 'utf-8');
unlinkSync(realFailuresPath);
}
// Fresh git repo with one valid file. Mirrors createTestRepo above but
// scoped to this describe block.
repoPath = mkdtempSync(join(tmpdir(), 'gbrain-skipfailed-e2e-'));
execSync('git init', { cwd: repoPath, stdio: 'pipe' });
execSync('git config user.email "test@test.com"', { cwd: repoPath, stdio: 'pipe' });
execSync('git config user.name "Test"', { cwd: repoPath, stdio: 'pipe' });
mkdirSync(join(repoPath, 'people'), { recursive: true });
writeFileSync(join(repoPath, 'people/alice.md'), [
'---', 'type: person', 'title: Alice', '---', '', 'Body.',
].join('\n'));
execSync('git add -A && git commit -m "initial"', { cwd: repoPath, stdio: 'pipe' });
});
afterAll(async () => {
await teardownDB();
if (repoPath) rmSync(repoPath, { recursive: true, force: true });
// Restore the user's real sync-failures.jsonl, if any.
if (savedFailuresContent !== null) {
mkdirSync(join(homedir(), '.gbrain'), { recursive: true });
writeFileSync(realFailuresPath, savedFailuresContent);
} else if (existsSync(realFailuresPath)) {
// Test wrote one but there was none before. Clean up.
unlinkSync(realFailuresPath);
}
});
test('full --skip-failed loop: blocks on bad file, skip advances bookmark, doctor shows code breakdown', async () => {
const { performSync } = await import('../../src/commands/sync.ts');
const { loadSyncFailures, summarizeFailuresByCode } = await import('../../src/core/sync.ts');
const engine = getEngine();
// Step 1: First sync of the clean repo — should succeed.
let result = await performSync(engine, { repoPath, noPull: true, noEmbed: true });
expect(result.status).toBe('first_sync');
const firstCommit = await engine.getConfig('sync.last_commit');
expect(firstCommit).toBeTruthy();
// Step 2: Add a broken file — frontmatter slug doesn't match path-derived slug.
// The file path is people/bob.md so the path-derived slug is "people/bob",
// but we declare slug: "wrong-slug" in frontmatter. import-file.ts:368-377
// raises "Frontmatter slug ... does not match path-derived slug ..." which
// classifier hits as SLUG_MISMATCH.
writeFileSync(join(repoPath, 'people/bob.md'), [
'---', 'type: person', 'title: Bob', 'slug: wrong-slug', '---', '', 'Body.',
].join('\n'));
execSync('git add -A && git commit -m "add broken bob"', { cwd: repoPath, stdio: 'pipe' });
// Step 3: Sync should block. Bookmark must NOT advance.
result = await performSync(engine, { repoPath, noPull: true, noEmbed: true });
expect(result.status).toBe('blocked_by_failures');
const afterBlockedCommit = await engine.getConfig('sync.last_commit');
expect(afterBlockedCommit).toBe(firstCommit); // bookmark stuck at the pre-broken commit
// JSONL has one unacked entry with code SLUG_MISMATCH.
let failures = loadSyncFailures();
expect(failures.length).toBe(1);
expect(failures[0].code).toBe('SLUG_MISMATCH');
expect(failures[0].acknowledged).toBeFalsy();
// Group summary aggregates correctly across the unacked set.
expect(summarizeFailuresByCode(failures)).toEqual([{ code: 'SLUG_MISMATCH', count: 1 }]);
// Step 4: Run with skipFailed — bookmark advances, entry gets acked.
result = await performSync(engine, { repoPath, noPull: true, noEmbed: true, skipFailed: true });
expect(result.status).toBe('synced');
const afterSkipCommit = await engine.getConfig('sync.last_commit');
expect(afterSkipCommit).not.toBe(firstCommit); // bookmark moved past the broken commit
failures = loadSyncFailures();
expect(failures.length).toBe(1);
expect(failures[0].acknowledged).toBe(true);
expect(typeof failures[0].acknowledged_at).toBe('string');
// Step 5: Verify what doctor would render for the historical entry.
// We call the same primitives doctor's `sync_failures` check uses
// (src/commands/doctor.ts:252-275) — loadSyncFailures + summarizeFailuresByCode —
// and assert the rendering string. Directly invoking runDoctor() here is a CLI
// entrypoint with stdout/exit side effects that would truncate this test mid-flow.
{
const all = loadSyncFailures();
const ackedSummary = summarizeFailuresByCode(all);
const ackedBreakdown = ackedSummary.map(s => `${s.code}=${s.count}`).join(', ');
// This is the literal string interpolation doctor.ts:271-274 produces.
const doctorMessage = `${all.length} historical sync failure(s), all acknowledged [${ackedBreakdown}].`;
expect(doctorMessage).toContain('SLUG_MISMATCH=1');
expect(doctorMessage).toContain('1 historical');
}
// Step 6: Add a second broken file — this one with a different failure code
// (also SLUG_MISMATCH but on a different file) so the JSONL has 2 entries
// with DIFFERENT paths but the same code. This proves both: per-file dedup
// honors path identity, and summary aggregation sums across files.
//
// We'd ideally test a different code class here, but the sync path uses
// parseMarkdown WITHOUT {validate:true}, so the markdown.ts validation
// codes (MISSING_OPEN/CLOSE, NESTED_QUOTES, EMPTY_FRONTMATTER, NULL_BYTES)
// don't naturally surface — they'd need {validate:true} plumbed in. That
// plumbing is the v0.22.13+ follow-up. For v0.22.12, two SLUG_MISMATCH
// entries from different files still proves the dedup + aggregation chain.
writeFileSync(join(repoPath, 'people/carol.md'), [
'---', 'type: person', 'title: Carol', 'slug: also-wrong-slug', '---', '', 'Body.',
].join('\n'));
execSync('git add -A && git commit -m "add carol with bad slug"', { cwd: repoPath, stdio: 'pipe' });
// Step 7: Sync blocks again on the new failure. Old entry stays acked.
result = await performSync(engine, { repoPath, noPull: true, noEmbed: true });
expect(result.status).toBe('blocked_by_failures');
failures = loadSyncFailures();
expect(failures.length).toBe(2);
const acked = failures.filter(f => f.acknowledged);
const unacked = failures.filter(f => !f.acknowledged);
expect(acked.length).toBe(1);
expect(acked[0].code).toBe('SLUG_MISMATCH');
expect(acked[0].path).toContain('bob');
expect(unacked.length).toBe(1);
expect(unacked[0].code).toBe('SLUG_MISMATCH');
expect(unacked[0].path).toContain('carol');
// Step 8: Skip again — both entries acked, summary aggregates the count.
result = await performSync(engine, { repoPath, noPull: true, noEmbed: true, skipFailed: true });
expect(result.status).toBe('synced');
failures = loadSyncFailures();
expect(failures.length).toBe(2);
expect(failures.every(f => f.acknowledged)).toBe(true);
const finalSummary = summarizeFailuresByCode(failures);
expect(finalSummary).toEqual([{ code: 'SLUG_MISMATCH', count: 2 }]);
});
});
+15 -4
View File
@@ -7,28 +7,39 @@
*
* All tests use PGLite/in-memory no DB connection required.
*/
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
import { describe, test, expect, beforeAll, afterAll, beforeEach, afterEach } from 'bun:test';
import { mkdtempSync, writeFileSync, mkdirSync, rmSync } from 'fs';
import { tmpdir } from 'os';
import { join } from 'path';
import { runExtractCore } from '../src/commands/extract.ts';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import type { BrainEngine } from '../src/core/engine.ts';
import { resetPgliteState } from './helpers/reset-pglite.ts';
// One PGLite per file (beforeAll), wipe data per test (beforeEach).
// PGLite cold-start dominates wall-time; sharing the engine across all tests
// in this file cuts ~22s × 8 tests = ~3 min on CI.
let engine: PGLiteEngine;
let tempDir: string;
beforeEach(async () => {
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({ engine: 'pglite' });
await engine.initSchema();
});
afterAll(async () => {
await engine.disconnect();
});
beforeEach(async () => {
await resetPgliteState(engine);
tempDir = mkdtempSync(join(tmpdir(), 'gbrain-extract-test-'));
mkdirSync(join(tempDir, 'people'), { recursive: true });
mkdirSync(join(tempDir, 'companies'), { recursive: true });
});
afterEach(async () => {
await engine.disconnect();
afterEach(() => {
rmSync(tempDir, { recursive: true, force: true });
});
@@ -0,0 +1,32 @@
# Claw-test brief — fresh-install
You are testing gbrain on a brand-new install. The user just ran `gbrain init` for the first time. Walk through the canonical first-day flow:
1. **Verify install:** confirm `gbrain --version` works and `gbrain doctor --json` returns a valid JSON object with a `status` field.
2. **Install skillpack:** run `gbrain skillpack install --workspace $PWD`. The workspace already has an `AGENTS.md` routing file.
3. **Import the brain:** run `gbrain import ./brain --no-embed --progress-json`. There are 3 small markdown pages already there.
4. **Query the brain:** run `gbrain query "alice"` and verify >0 results.
5. **Extract links:** run `gbrain extract --source fs --progress-json`.
6. **Verify health:** run `gbrain doctor --json`. The `status` field should be `"ok"`.
## Friction protocol
If anything is confusing, missing, surprising, or wrong, run:
```
gbrain friction log --severity {confused|error|blocker|nit} --phase <which-step> --message "<what-happened>" [--hint "<what-could-be-better>"]
```
Severity guide:
- `blocker` — couldn't proceed at all
- `error` — command failed unexpectedly
- `confused` — docs said one thing, the tool did another, or a step felt unclear
- `nit` — minor polish opportunity
If something *just worked* and was nicer than expected, log a delight too:
```
gbrain friction log --kind delight --phase <step> --message "<what-was-nice>"
```
We want to know what didn't work, not just whether commands exited zero. Be specific.
@@ -0,0 +1,15 @@
---
type: company
name: Acme Example
founded: 2024
founders:
- alice-example
---
# Acme Example
Fictional company used for claw-test fixtures. Founded 2024 by [Alice](people/alice-example).
## What they do
Acme builds an agentic-workflow product on top of [retrieval-augmented-generation](concepts/retrieval-augmented-generation). Early traction comes from a developer-tools wedge.
@@ -0,0 +1,10 @@
---
type: concept
name: Agentic Workflows
---
# Agentic Workflows
Workflows where an LLM-driven agent plans, executes, and revises a sequence of steps with minimal human supervision per step. Key constraints: cost, latency, and observability of the loop.
Companies building in this space include [acme-example](companies/acme-example).
@@ -0,0 +1,13 @@
---
type: person
name: Alice Example
x_handle: alice_example
---
# Alice Example
Alice is a fictional founder used for claw-test fixtures. She started [acme-example](companies/acme-example) in 2024.
## Background
Alice has spent 10 years in software and 2 years in AI tooling. She is exploring product-market fit for an [agentic-workflow](concepts/agentic-workflows) tool.
@@ -0,0 +1,6 @@
{
"min_pages_after_import": 3,
"min_query_results": 1,
"min_links_after_extract": 0,
"doctor_status": "ok"
}
@@ -0,0 +1,10 @@
{
"kind": "fresh-install",
"description": "Canonical 5-minute first-day flow: init → import → query → extract → verify",
"expected_phases": [
"import.files",
"extract.links_fs",
"doctor.db_checks"
],
"brain": "brain"
}
@@ -0,0 +1,25 @@
# Claw-test brief — upgrade-from-v0.18
You inherit a gbrain v0.18 brain (the harness has already replayed a seed SQL dump into a PGLite database). Walk through the upgrade path:
1. **Run `gbrain doctor --json`** first. Note any warnings or fix-hints.
2. **Run `gbrain init --pglite`** with the existing database path. The migration chain should detect the old `schema_version` and walk forward to the latest.
3. **Run `gbrain doctor --json` again.** The `status` field should be `"ok"`.
4. **Verify queries still work:** `gbrain query "alice"` should return results from the seeded brain.
## Friction protocol
If anything is confusing, missing, surprising, or wrong (especially around the migration steps — these are the highest-historical-pain regression points), run:
```
gbrain friction log --severity {confused|error|blocker|nit} --phase <which-step> --message "<what-happened>" [--hint "<what-could-be-better>"]
```
Common upgrade-flow friction patterns to watch for:
- The migration chain failed at a specific schema version (capture the version + error)
- Doctor flagged an issue but the fix-hint wasn't actionable
- `gbrain init --pglite` didn't recognize the existing brain
- Manual SQL was needed to unblock something
If something just worked, log a delight. We're tuning the upgrade flow toward zero-friction.
@@ -0,0 +1,8 @@
---
type: person
name: Alice Example
---
# Alice Example
Same brain content as the fresh-install scenario; this scenario tests upgrade flow rather than ingest. After the migration chain walks forward, agents query and the page must be findable.
@@ -0,0 +1,4 @@
{
"min_pages_after_migration": 1,
"doctor_status": "ok"
}

Some files were not shown because too many files have changed in this diff Show More