Compare commits

...
Author SHA1 Message Date
Garry TanandClaude Opus 4.7 f3dace916c Merge origin/master into feat/brain-first-convention
Master moved from v0.23.1 → v0.23.2 (PR #527: orchestrator-stamped
self-consumption marker for the dream cycle + verdict-model unit tests).

Conflicts resolved:
- VERSION: kept this branch's 0.24.0 per CLAUDE.md branch-scoped rule.
- package.json version: kept 0.24.0.
- CHANGELOG.md: my v0.24.0 stays on top; master's new v0.23.2 entry
  spliced between v0.24.0 and v0.23.1. Sequence above v0.21.0 monotonic.

Verification:
- bun install — 0 new packages
- All 5 CI guards green: privacy + jsonb + progress + trailing-newline + wasm
- bun run typecheck — clean
- This branch's tests: 103/103 pass

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-30 23:06:42 -07:00
579722d9dc v0.23.2 fix(dream): orchestrator-stamped self-consumption marker + verdict model tests (#527)
* v0.23.1 fix: dream self-consumption guard + configurable verdict model

Built-in isDreamOutput() guard in transcript-discovery.ts auto-skips
any transcript whose first 2000 chars contain dream output slug prefixes
(wiki/personal/reflections/, wiki/originals/ideas/, wiki/personal/patterns/,
dream-cycle-summaries/). Prevents infinite recursion if dream output is
ever fed back into the corpus.

judgeSignificance() now accepts a verdictModel parameter, loaded from
dream.synthesize.verdict_model config key. Default: claude-haiku-4-5.

3 new test cases covering the guard.

* feat(dream): replace content-prefix guard with orchestrator-stamped marker

The v0.23.1 prefix-string guard had two flaws caught by codex review.
serializeMarkdown does not embed the page slug into body content, so
the heuristic could miss real dream output. And real conversation
transcripts often cite brain slugs ("earlier I wrote about
wiki/personal/reflections/identity..."), so the heuristic dropped
legitimate transcripts silently.

Swap content inference for explicit identity. renderPageToMarkdown and
writeSummaryPage now stamp `dream_generated: true` + `dream_cycle_date`
into frontmatter at render time. Guard checks for the marker via
DREAM_OUTPUT_MARKER_RE (anchored at frontmatter open, BOM/CRLF
tolerant, scans first 2000 chars, word boundary on `true`). Cannot
drift, cannot false-positive on user text, cannot miss real output.

Tests built from a real Page → renderPageToMarkdown → isDreamOutput
round-trip (codex finding #5 — synthetic strings don't prove the
guard catches what synthesize actually produces). Coverage: regression
fixture, false-positive prevention on user transcripts citing slugs,
CRLF+BOM, whitespace/case variants, anchor-at-byte-0, perf bound,
bypass plumbing, dream_generatedfoo word-boundary check.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(dream): --unsafe-bypass-dream-guard CLI flag

Explicit opt-in to disable the synthesize self-consumption guard. The
flag is intentionally NOT tied to --input — codex review caught that
implicit bypass is a footgun: any caller could synthesize a dream-
generated page directly via --input, get a cached positive verdict,
and silently re-trigger the loop bug.

Plumbing: dream.ts CLI parses the flag → DreamArgs.bypassDreamGuard →
runCycle({ synthBypassDreamGuard }) → SynthesizePhaseOpts.bypassDreamGuard
→ discoverTranscripts({ bypassGuard }) and readSingleTranscript.
Loud stderr warning at phase entry when set so the cost is visible.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.23.2 chore: bump version + CHANGELOG for corrected guard architecture

Replaces the v0.23.1 release notes with the v0.23.2 voice describing
the orchestrator-stamped marker approach and the --unsafe-bypass-dream-guard
flag.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: sync project docs for v0.23.2 marker-based guard

Update CLAUDE.md Key Files entries for src/core/cycle/synthesize.ts,
src/core/cycle/transcript-discovery.ts, and src/commands/dream.ts to
reflect the v0.23.2 dream_generated frontmatter marker that replaces the
v0.23.1 content-prefix self-consumption guard, plus the new
--unsafe-bypass-dream-guard CLI flag.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* docs: regenerate llms-full.txt for v0.23.2 CLAUDE.md updates

CI's `build-llms generator > committed match generator output` guard
caught drift after the v0.23.2 doc-sync (commit 507edb1e) updated three
Key Files entries in CLAUDE.md without re-running `bun run build:llms`.

The llms.txt index didn't drift (no new doc URLs); only the inlined
llms-full.txt bundle needed refreshing.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(e2e): round-trip dream-recursion coverage for v0.23.2 marker guard

Three new PGLite E2E cases exercise the actual production loop scenario
end-to-end. Unit tests covered the bug class at the function-pair level
(renderPageToMarkdown → readSingleTranscript). These cover it at the
phase level: runPhaseSynthesize with a real engine, real putPage, real
renderPageToMarkdown, real corpus-dir discovery.

1. Leaked dream output is skipped on next synthesize run. The reflection
   page is inserted, reverse-rendered (which stamps `dream_generated:
   true`), dropped into the corpus dir as .txt, and the next phase run
   reports "no transcripts to process" with a stderr skip log. Verdict
   cache stays untouched so a future legit edit isn't shadowed by a
   stale cached "false".

2. bypassDreamGuard=true at phase entry re-enables ingestion. Same
   marked file gets discovered through the loud-warning path. Proves
   --unsafe-bypass-dream-guard plumbing reaches discoverTranscripts at
   phase scope.

3. Mixed corpus (leaked dream output + real conversation transcript)
   discovers exactly the real one. Pins codex finding #1's headline
   false-positive case: a transcript citing wiki/personal/reflections/
   in body must NOT be skipped.

Stderr capture via process.stderr.write spy with try/finally restore.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(test): use valid PageType 'note' in round-trip E2E fixtures

CI typecheck caught three TS2322 violations in the round-trip E2E
fixtures: 'reflection' is not a member of PageType. Reflections are
filed as 'note' in production (renderPageToMarkdown falls back to 'note'
for unknown types).

No behavior change — the guard test still exercises the same
serializeMarkdown → discoverTranscripts loop.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(claude): require `bun run typecheck` before push

The pre-ship section listed `bun test` as the unit-test path but didn't
flag the trap: `bun test` (the bun runner) does NOT run TypeScript type
checking. Only `bun run test` (the npm script) does, because it chains
`bun run typecheck` + the four shell pre-checks before the runner.

CI on PR #527 caught a `'reflection'` literal that `PageType` doesn't
admit (PageType is a closed union). The runtime E2E and `bun test`
both passed locally because the runner doesn't gate on TS. The
separate typecheck stage in CI rejected it.

New rule: run `bun run typecheck` (or `bun run test`, which wraps it,
or `bun run ci:local` for the full gate) before pushing. The runner-
alone path is for hot-loop test iteration only.

Also regenerated llms-full.txt for the CLAUDE.md update.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Wintermute <wintermute@garrytan.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-30 23:04:47 -07:00
Garry TanandClaude Opus 4.7 9d1d393151 Merge origin/master into feat/brain-first-convention
Master moved from v0.23.0 → v0.23.1 (PR #528: local CI gate +
4-tier wall-time optimization, ~13x faster). Plus a jsonb fix that
master shipped + reverted in the same window (no-op for this branch).

Conflicts resolved:
- VERSION: kept this branch's 0.24.0 per CLAUDE.md branch-scoped rule.
- package.json: kept 0.24.0; combined the test-script chain so
  check-privacy.sh (mine) coexists with master's new check-trailing-
  newline.sh + the existing jsonb/progress/wasm/typecheck guards. Also
  pulled in master's new build:pglite-snapshot script.
- CONTRIBUTING.md: combined paragraphs. My "Use bun run test" guard
  description now precedes master's new "Local CI gate (v0.23.1+)"
  block describing bun run ci:local. Both are useful; both stay.
- CHANGELOG.md: my v0.24.0 entry stays on top; master's new v0.23.1
  entry slots between v0.24.0 and v0.23.0. Sequence above v0.21.0
  monotonically descending.

Verification:
- bun install — 0 new packages
- All 5 CI guards green: privacy + jsonb + progress + trailing-newline + wasm
- bun run typecheck — clean
- This branch's tests: 103/103 pass

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-30 20:50:24 -07:00
Garry TanandClaude Opus 4.7 90e22c22e2 v0.23.1 feat: local CI gate + 4-tier wall-time optimization (~13x faster) (#528)
* feat: diff-aware E2E test selector

Adds scripts/select-e2e.ts: reads git diff vs origin/master, classifies
the change set (EMPTY/DOC_ONLY/SRC), and emits the relevant E2E test files
on stdout. Fail-closed by design: any unmapped src/ change runs all E2E.

- scripts/e2e-test-map.ts: hand-tuned path-glob -> test files map
- scripts/select-e2e.ts: pure-function selector with three explicit cases
- scripts/run-e2e.sh: accepts optional file list from argv + --dry-run-list
- test/select-e2e.test.ts: 24 cases including 3 codex regression guards
  (skills/, untracked files, unmapped src/)

* feat: local CI gate via docker compose

Adds bun run ci:local — runs every check GH Actions runs (gitleaks +
unit + 29 E2E files) inside a Docker container that bind-mounts the
repo. Pure bind-mount + named volumes (gbrain-ci-node-modules,
gbrain-ci-bun-cache, gbrain-ci-pg-data) for fast warm restarts.

- docker-compose.ci.yml: pgvector/pgvector:pg16 + oven/bun:1
- scripts/ci-local.sh: orchestrator with --diff, --no-pull, --clean
- gitleaks runs on host (scoped to working dir + branch commits)
- DATABASE_URL unset for unit phase (matches GH Actions split)
- git installed in container at startup (oven/bun:1 omits it)
- Postgres host port via GBRAIN_CI_PG_PORT env (default 5434)

Stronger than PR CI: runs all 29 E2E files vs CI's 2-file Tier 1.

* chore: bump version and changelog (v0.23.1)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: document local CI gate for v0.23.1

CLAUDE.md gains key-files entries for docker-compose.ci.yml,
scripts/ci-local.sh, scripts/select-e2e.ts + e2e-test-map.ts, and the
scripts/run-e2e.sh argv tweak. Pre-ship requirements section now lists
the Docker-based local gate as Path A alongside the manual lifecycle.

CONTRIBUTING.md tests section adds the bun run ci:local / ci:local:diff /
ci:select-e2e block with prerequisites (Docker engine + gitleaks) and the
GBRAIN_CI_PG_PORT override.

AGENTS.md "Before shipping" promotes ci:local as the easiest path and
keeps the manual lifecycle as a fallback.

README.md Contributing section points to ci:local for the full gate.

CHANGELOG.md untouched — v0.23.1 entry already finalized.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* feat: SHARD=N/M env support in scripts/run-e2e.sh

Filters the E2E file list to every M-th file starting at index N (1-indexed).
Sequential execution within a shard preserves the TRUNCATE CASCADE no-race
property documented at the top of the file. Empty-shard handling under
`set -u` uses ${arr[@]:-} fallback.

Standalone change; not yet wired up in ci-local.sh.

* feat: 4-way parallel E2E shards in ci:local

Replaces the single postgres service with 4 (postgres-1..4) on host ports
5434-5437. scripts/ci-local.sh fans 4 workers via xargs -P4 inside the
runner container; each pinned to its own DATABASE_URL via SHARD=N/4.

Wall-time on a 16-core host: ~6 min sequential -> ~1.5-2 min sharded.
Total full-gate wall-time goes from ~25 min to ~3-5 min warm.

Also handles git-worktree (Conductor) layouts: when /app/.git is a file
instead of a directory, parse the gitdir + commondir and bind-mount the
shared host gitdir at its absolute path. Without this, in-container
`git ls-files` (used by scripts/check-trailing-newline.sh and friends)
exits 128 with "not a git repository". Also runs
`git config --global --add safe.directory '*'` inside the container so
the root-uid container can read host-uid gitdir without "dubious
ownership" rejection.

CHANGELOG entry updated to cover the speedup.

- docker-compose.ci.yml: 4 pgvector services + per-shard named volumes
- scripts/ci-local.sh: parallel xargs orchestration + worktree mount fix
- CHANGELOG.md v0.23.1: 4-way sharded wall-time, 36 E2E files, --no-shard flag

* chore: regenerate llms-full.txt for v0.23.1 doc updates

Required by test/build-llms.test.ts case 4 — committed llms-full.txt
must match `bun run build:llms` output. The CHANGELOG + CLAUDE.md
updates in this branch shifted bytes; regen catches up.

* feat: scripts/run-unit-shard.sh + slow-test convention

Tier 1 + Tier 4 plumbing:
- scripts/run-unit-shard.sh: SHARD=N/M filter for unit files (excludes
  test/e2e/*). Excludes *.slow.test.ts (Tier 4 convention) so the fast
  shard fan-out skips known-slow files; CI's `bun run test` still includes
  them via default discovery.
- scripts/run-slow-tests.sh: companion that runs ONLY *.slow.test.ts.
  Wired as `bun run test:slow`.
- scripts/profile-tests.sh: portable awk parser that extracts the top-N
  slowest tests from any captured `bun test` output. Wired as
  `bun run test:profile`. Use it to pick demotion candidates.

* feat: PGLite snapshot fixture for ~4.5x faster cold init (Tier 3)

scripts/build-pglite-snapshot.ts boots a fresh PGLite, runs the full
initSchema() (forward bootstrap + 30 migrations), and dumps the post-init
state to test/fixtures/pglite-snapshot.tar plus a SHA-256 schema hash
sidecar (.version). Both gitignored — built on demand via
`bun run build:pglite-snapshot`.

PGLiteEngine.connect() reads GBRAIN_PGLITE_SNAPSHOT env: validates the
sidecar hash against the in-process MIGRATIONS hash, loads via PGLite's
loadDataDir blob, sets _snapshotLoaded so initSchema() short-circuits.
Measured per-file cold init drops from 828ms → 181ms.

Bootstrap-correctness tests (bootstrap.test.ts,
schema-bootstrap-coverage.test.ts) explicitly delete the env at file
top so they keep exercising the cold path they verify.

* feat: --classify-only + heartbeat tolerance fix (Tiers 2 + flake fix)

- scripts/select-e2e.ts: --classify-only flag emits EMPTY|DOC_ONLY|SRC.
  Used by ci-local.sh's --diff fast-path to skip the heavy gate when
  only docs changed.
- test/progress.test.ts: startHeartbeat tolerance widened to 1-20 over
  200ms (was 2-6 over 85ms). Under 4-way parallel shard load on a
  contended host, setTimeout's effective quantum balloons and the tight
  bound flakes. The test still verifies "fires multiple times, stops
  cleanly" — exact count was never load-bearing.

* feat: 4-way unit + E2E sharding in ci-local.sh + CHANGELOG (Tiers 1-4)

ci-local.sh ties the four tiers together:
- Tier 2: pre-flight diff classification on host. DOC_ONLY exits in ~5s
  (gitleaks only, no postgres, no container).
- Tier 1: guards + typecheck run ONCE before fan-out. xargs -P4 then
  spawns 4 shards inside the runner container, each running unit phase
  (env -u DATABASE_URL bash run-unit-shard.sh) followed by E2E phase
  (DATABASE_URL=postgres-N bash run-e2e.sh) — both sharded N/4. Per-shard
  logs in /tmp/shard-logs/shard-N.log; printed in shard order at the end.
- Tier 3: snapshot fixture built once at runner startup if missing,
  GBRAIN_PGLITE_SNAPSHOT exported so all shards inherit.
- Tier 4: run-unit-shard.sh excludes *.slow.test.ts; run-slow-tests.sh
  + test:slow npm script handle the demoted set.
- --no-shard preserves the legacy single-process flow for debug.

package.json: build:pglite-snapshot, test:slow, test:profile scripts.

Measured wall-time on 16-core host: 100s warm (down from ~22 min cold
single-process). 4 shards × ~640-1024 unit tests each, plus 9 E2E
files each. PGLite snapshot saves 4.5× per cold init (828ms → 181ms).

CHANGELOG.md updated with measured numbers + four-tier breakdown.

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-30 20:47:32 -07:00
Wintermute 18f5ba56cf Revert "fix: jsonb double-encoding in subagent_tool_executions (#525)"
This reverts commit 80b3909702.
2026-04-30 18:35:11 +00:00
Wintermute 80b3909702 fix: jsonb double-encoding in subagent_tool_executions (#525)
JSON.stringify(input) + ::jsonb cast produced a jsonb string value
instead of a jsonb object. The postgres library's unsafe() with a
raw object + ::jsonb correctly stores a jsonb object.

This caused collectChildPutPageSlugs to return 0 results (can't
extract ->>'slug' from a jsonb string), making dream synthesize
report '0 pages written' even though subagents successfully wrote
16 pages to the database.

Fix: pass objects as-is to executeRaw, let the postgres driver
handle serialization. Non-object values wrapped in {_raw: ...}
as a safety fallback.
2026-04-30 18:14:12 +00:00
Garry TanandClaude Opus 4.7 73edd84bcc Merge origin/master into feat/brain-first-convention
Master moved from v0.22.9 → v0.23.0 (9 PRs: dream synthesizes
conversations into brain pages; claw-test friction harness; frontmatter
inference; minions bare-worker self-monitoring; parallel sync; sync
error-code summary; storage tiering; autopilot-cycle phases passthrough).

Conflicts resolved:
- VERSION: kept this branch's 0.24.0 per CLAUDE.md branch-scoped rule.
- package.json: kept 0.24.0; combined test-script chain — both my
  check-privacy.sh AND master's new check-trailing-newline.sh now run
  alongside check-jsonb-pattern.sh + check-progress-to-stdout.sh +
  check-wasm-embedded.sh + typecheck + bun test --timeout=60000.
- CHANGELOG.md: stitched. v0.24.0 stays on top, master's new v0.23.0 +
  v0.22.16 + v0.22.15 + v0.22.14 + v0.22.13 + v0.22.12 + v0.22.11 +
  v0.22.10 entries spliced between v0.24.0 and v0.22.9 (the previous
  merge boundary). Sequence above v0.21.0 monotonically descending.

Verification:
- bun install — 6 new packages (web-tree-sitter wave + bun-types bump)
- All 5 CI guards green: privacy + jsonb + progress + trailing-newline + wasm
- bun run typecheck — clean
- This branch's tests + sql-ranking: 103/103 pass

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-30 03:03:58 -07:00
Garry TanandClaude Opus 4.7 527b87bd1e v0.23.0 feat: gbrain dream synthesizes conversations into brain pages (v0.23.0) (#462)
* feat: dream_verdicts schema + engine methods

Adds the v25 schema migration creating the dream_verdicts table
(file_path, content_hash, worth_processing, reasons, judged_at;
PRIMARY KEY (file_path, content_hash); RLS-enabled when running as
a BYPASSRLS role).

Distinct from raw_data (which is page-scoped) — transcripts being
judged for synthesis aren't pages. The (file_path, content_hash)
key means edited transcripts re-judge automatically.

BrainEngine gains:
- DreamVerdict + DreamVerdictInput types
- getDreamVerdict(filePath, contentHash) → DreamVerdict | null
- putDreamVerdict(filePath, contentHash, verdict) — ON CONFLICT upsert

Both engines implement (postgres-engine.ts, pglite-engine.ts).

This commit alone is functionally inert — nothing reads/writes the
table yet. The synthesize phase (later commit) is the consumer.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: trusted-workspace allow-list for subagent put_page

Adds OperationContext.allowedSlugPrefixes — when set, put_page
enforces slug membership in the allow-list instead of the legacy
wiki/agents/<id>/... namespace. The trust signal is the SUBMITTER
(PROTECTED_JOB_NAMES gates subagent submission so MCP can't reach
this field), not the runtime ctx.remote flag — every subagent tool
call has remote=true for auto-link safety, so basing trust on
remote is incoherent.

matchesSlugAllowList(slug, prefixes) helper supports glob suffix
'/*' (recursive — wiki/originals/* matches ideas/foo/bar) and
exact match for unsuffixed entries.

put_page check shape:
  if (viaSubagent && allowedSlugPrefixes set) → allow-list check
  else if (viaSubagent) → existing namespace check (regression guard)
  else → no check (regular CLI)

Auto-link is re-enabled for the trusted-workspace path so the cycle's
extract phase doesn't have to recompute every edge after synthesize
writes. Untrusted remote writes still skip auto-link as before.

SubagentHandlerData.allowed_slug_prefixes is the wire field; the
synthesize/patterns phases (later commit) populate it from a single
source of truth in skills/_brain-filing-rules.json's
dream_synthesize_paths.globs array. The model's tool schema description
mirrors the allow-list so it writes correct slugs on the first try.

IRON RULE security tests:
- test/operations-allow-list.test.ts: allow-list ALLOW/REJECT, glob
  semantics, regression guard for the v0.15 namespace fallback when
  allow-list is unset, FAIL-CLOSED when subagentId is missing.
- test/e2e/dream-allow-list-pglite.test.ts: end-to-end on PGLite,
  poisoned-transcript style write outside allow-list → REJECTED.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: cycle scaffolding — 8-phase order + transcript discovery

Extends ALL_PHASES from 6 → 8: synthesize between sync and extract,
patterns between extract and embed. Codex finding #7: patterns MUST
run after extract because subagent put_page sets ctx.remote=true and
skips auto-link/timeline by default — extract is the canonical edge
materialization step. Without that ordering, patterns reads stale
graph state.

Final order:
  lint → backlinks → sync → synthesize → extract → patterns → embed → orphans

CycleOpts gains:
- yieldDuringPhase callback — generic in-phase keepalive for long
  waits (synthesize fan-out, patterns roll-up). Renews cycle-lock TTL
  + worker job lock. Mirrors yieldBetweenPhases shape.
- synthInputFile / synthDate / synthFrom / synthTo — forwarded to
  runPhaseSynthesize for the CLI's --input/--date/--from/--to flags.

CycleReport.totals additively grows (no schema_version bump):
  transcripts_processed, synth_pages_written, patterns_written.

src/core/cycle/transcript-discovery.ts is a pure filesystem walk:
- .txt files only, sorted by path for determinism
- date-prefixed basename filter (--date / --from / --to)
- min_chars filter (default 2000)
- exclude_patterns auto-wraps bare words as \b<word>\b regex (Q-3),
  power users may pass full regex with anchors
- compileExcludePatterns is exported for unit tests

Phase implementations land in the next commit; this one only adds
the dispatcher slots so commit-by-commit bisect doesn't crash on
import-not-found.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: synthesize + patterns phases — gbrain dream actually dreams

Synthesize phase (src/core/cycle/synthesize.ts) reads conversation
transcripts from dream.synthesize.session_corpus_dir and writes
brain-native pages: reflections to wiki/personal/reflections/...,
originals to wiki/originals/ideas/..., timeline entries on existing
people pages.

Pipeline:
  1. discoverTranscripts (filesystem walk + filters)
  2. cooldown check via dream.synthesize.last_completion_ts config
     (default 12h; bypassed by --input/--date/--from/--to)
  3. cheap Haiku verdict per transcript, cached in dream_verdicts
     table keyed by (file_path, content_hash) — backfill re-runs
     skip already-judged transcripts at zero cost
  4. fan-out: one Sonnet subagent per worth-processing transcript
     dispatched with allowed_slug_prefixes (read from
     skills/_brain-filing-rules.json's dream_synthesize_paths.globs)
     and idempotency_key dream:synth:<file_path>:<content_hash>
  5. wait via waitForCompletion; yieldDuringPhase ticks every child
     terminal so the cycle-lock TTL refreshes on long backfills
  6. collect slugs from subagent_tool_executions for each child
     (codex finding #2: NOT pages.updated_at, which would pick up
     unrelated writes)
  7. orchestrator dual-write — query each new page from DB,
     reverse-render via serializeMarkdown, write file to brain_dir.
     Subagent never gets fs-write access.
  8. deterministic summary index page at dream-cycle-summaries/<date>
     (codex finding #4: slug shape is regex-compatible — no
     underscores, no .md extension)
  9. write completion timestamp ONLY on successful runs

Patterns phase (src/core/cycle/patterns.ts) runs after extract so
the graph state is fresh. Single Sonnet subagent gathers reflections
within dream.patterns.lookback_days (default 30); names a pattern
only when ≥dream.patterns.min_evidence (default 3) reflections
support it. Same allow-list path as synthesize.

CLI flags on `gbrain dream` (src/commands/dream.ts):
  --input <file>      ad-hoc transcript synthesis (implies
                      --phase synthesize; bypasses cooldown)
  --date YYYY-MM-DD   restrict synthesize to one date
  --from <d> --to <d> backfill range
  --dry-run           runs Haiku verdict (cached), skips Sonnet
                      synthesis. NOT zero LLM calls (codex #8).

Conflict detection: --input + --date/--from/--to exits 2.
ISO 8601 date format validated; range start > end exits 2.

Auto-commit / push deferred to v1.1 (codex finding #5). v1 writes
files to brain_dir; user or autopilot handles git.

Tests:
- test/cycle-patterns.test.ts: structural assertions on the patterns
  phase (queue + waitForCompletion wired, allow-list threading,
  subagent_tool_executions provenance, no raw_data dependency).
- test/dream-cli-flags.test.ts: argv parsing, conflict detection,
  ISO date validation, --input implies --phase synthesize, dry-run
  semantics doc string.
- test/e2e/dream-synthesize-pglite.test.ts: 8 cases on PGLite
  in-memory exercising not_configured, empty corpus, no API key
  skip path, dry-run, cooldown active vs --input bypass, and the
  dream_verdicts cache hit path. Per-test rig isolation (each
  test creates and tears down its own engine) avoids
  cross-test PGLite WASM contention.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: dream cycle v0.27.0 — skills, CLAUDE.md, migration, changelog

- skills/maintain/SKILL.md: synthesize + patterns phases documented
  with quality bar (Iron Law for synthesis), trust boundary, idempotency,
  cooldown semantics, CLI invocation patterns. New triggers added so
  "process today's session" / "synthesize my conversations" route here.
- skills/RESOLVER.md: dream cycle triggers route to maintain.
- skills/_brain-filing-rules.md: directory table for the five output
  types (reflections, originals, patterns, people enrichment, cycle
  summary) with slug shape per row; Iron Law repeated.
- skills/migrations/v0.27.0.md: agent-readable migration narrative.
  Schema migration v25 runs automatically on `gbrain apply-migrations`;
  synthesize ships disabled by default — opt-in via
  dream.synthesize.session_corpus_dir + dream.synthesize.enabled.
- CLAUDE.md: file inventory updated with new files (cycle/synthesize.ts,
  cycle/patterns.ts, cycle/transcript-discovery.ts), the 8-phase
  ordering, the trusted-workspace allow-list trust model, and the v25
  schema migration line in the migrate.ts entry.
- VERSION: 0.20.4 → 0.27.0
- CHANGELOG.md: v0.27.0 release-summary section per CLAUDE.md voice
  rules (numbers that matter table, what-this-means closer, "to take
  advantage of" block), followed by the itemized changes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: add patterns E2E + 8-phase cycle E2E + bump synth-cooldown timeouts

Two new E2E test files on PGLite (no DATABASE_URL or API key required):

- test/e2e/dream-patterns-pglite.test.ts (6 cases) — exercises
  runPhasePatterns skip paths against a real engine: disabled,
  default-enabled-but-insufficient-evidence, no-API-key, dry-run.
  Sibling of dream-synthesize-pglite.test.ts; same per-test rig
  pattern for engine isolation.

- test/e2e/dream-cycle-eight-phase-pglite.test.ts (5 cases) —
  end-to-end runCycle with the v0.27 8-phase order. Asserts:
  ALL_PHASES is the documented 8 phases in the right sequence,
  the dry-run report's phases array preserves that order,
  CycleReport.totals carries the new transcripts_processed /
  synth_pages_written / patterns_written fields, --phase synthesize
  and --phase patterns each run only that phase, and synthInputFile
  is plumbed correctly through runCycle to runPhaseSynthesize.

Bump per-test timeout to 30s on the two synthesize-cooldown E2E
tests that create two PGLite engines back-to-back. Default Bun 5s
budget is tight under sustained suite pressure (PGLite WASM init
costs ~1-2s per engine on macOS); each test passes alone but flakes
in the full E2E suite. The third arg `30_000` is Bun's standard
test-timeout knob.

Full E2E suite (test/e2e/) now: 86 pass / 0 fail / 258 skip.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: ship-prep — typecheck fixes, llms.txt regen, 8-phase test update

- src/core/cycle/synthesize.ts + patterns.ts: PageType 'default' → 'note'
  (TS strict typecheck rejected 'default'; 'note' is a valid PageType
  for orchestrator-written summary index pages and reverse-render fallback).
- src/core/pglite-engine.ts: re-import DreamVerdict + DreamVerdictInput
  types after the master merge dropped them from the import line.
- test/e2e/dream-allow-list-pglite.test.ts: ToolCtx now requires
  remote: true literal; thread it through every put_page tool call.
- test/e2e/dream-patterns-pglite.test.ts: PageType 'default' → 'note'
  in the seedReflections helper.
- test/core/cycle.test.ts: bump expected hook-call count + phase count
  6 → 8 to match v0.27 ALL_PHASES extension.
- llms-full.txt: regenerate against the updated CHANGELOG + CLAUDE.md
  so the committed snapshot matches what the generator now produces.

Full bun test suite: 2793 pass / 0 fail / 258 skip (3051 tests, 177 files).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: update README + INSTALL_FOR_AGENTS for v0.27.0 dream cycle

README: maintain skill row mentions synthesize/patterns; gbrain dream
command-reference block describes the 8-phase pipeline and the new
--input/--date/--from/--to flags.

INSTALL_FOR_AGENTS: dream cycle bullet calls out v0.27 conversation
synthesis + cross-session pattern detection.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* chore: renumber v0.27.0 → v0.23.0

Master is at v0.22.5; v0.23.0 is the next natural slot for the dream-cycle
synthesize + patterns release. Bulk rename across VERSION, package.json,
CHANGELOG, migration file, source comments, skills, and llms.txt bundles.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(e2e): bump cycle.test.ts phase count 6 → 8

The dry-run full-cycle test asserted 6 phases. v0.23 added synthesize
and patterns, bringing the total to 8. The unit-side equivalent
(test/core/cycle.test.ts) was already updated; this catches the
E2E sibling that surfaced after the latest master merge.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-30 01:23:29 -07:00
Garry TanandClaude Opus 4.7 83e55ffcdb v0.22.16 feat: gbrain claw-test — end-to-end fresh-install friction harness (#522)
* feat: hermeticity migration — every $GBRAIN_HOME write site honors the env override

configDir() in src/core/config.ts already implemented $GBRAIN_HOME as a
parent-dir override (returns <override>/.gbrain), but ~12 consumers built paths
from os.homedir() directly and bypassed it. Critically, loadConfig/saveConfig
themselves used a private getConfigDir() that ignored the env. Fixed.

Migrated every write site to gbrainPath() — fail-improve, validator-lint, cycle
lock, shell-audit, backpressure-audit, sync-failures, integrity logs,
integrations heartbeat, init pglite path, migrate-engine manifest, import
checkpoint, v0_13_1 rollback, v0_14_0 host-work. Read-side host-detection in
init.ts (~/.claude / ~/.openclaw probes) intentionally NOT migrated; that's a
v1.1 follow-up under a separate $GBRAIN_HOST_HOME override.

Adds gbrainPath(...segments) sugar plus path validation: $GBRAIN_HOME must be
absolute and contain no '..' segments (throws GbrainHomeInvalidError).

test/gbrain-home-isolation.test.ts proves write-isolation across all migrated
sites. test/migrations-v0_14_0.test.ts updated to use $GBRAIN_HOME instead of
the old HOME-swap pattern.

Closes part of the claw-test E2E harness preconditions (D13 + D21).

* feat: gbrain friction {log,render,list,summary} — agent friction reporter

Append-only JSONL writer at $GBRAIN_HOME/friction/<run-id>.jsonl. Schema is a
flat extension of StructuredAgentError (D20), one envelope shape across both
agent-emitted entries and harness-wrapped command failures. Run-id resolves
from --run-id > $GBRAIN_FRICTION_RUN_ID > 'standalone'.

Subcommands stay ≤30 LOC each; core lives in src/core/friction.ts (writer +
reader + renderer + redactor). render --redact (default for md output) strips
\$HOME / \$CWD to placeholders so reports paste safely in PRs/issues.

Severity: confused | error | blocker | nit. Kind: friction | delight (D7) |
phase-marker | interrupted. Readers tolerate malformed lines (skip + warn).

40 unit tests; this is the channel the claw-test harness writes to and that
agents emit through during live-mode runs.

* feat: gbrain claw-test — end-to-end fresh-install friction harness

Two modes: scripted (CI gate, no agent) and --live (real agent subprocess).
Phases: setup → install_brain (gbrain init --pglite) → import (--no-embed) →
query → extract all --source fs → verify (gbrain doctor --json, asserts
status==='ok' and progress.jsonl phase coverage).

AgentRunner interface + registry — interface stays narrow (detect, invoke,
optional postInstallHook). v1 ships only OpenClawRunner; the registry pattern
lets v1.1 land hermes/codex as ~50-line additions without refactoring callers.
OpenClaw invocation: 'openclaw agent --local --agent <name> --message <brief>'
matching test/e2e/skills.test.ts (NOT --prompt-file, which doesn't exist).

transcript-capture: spawns child with piped stdio, async-drains via
fs.createWriteStream + 'drain' events so 256KB+ bursts don't stall the child
(D17 backpressure). Writes <run>/transcript.jsonl with schema_version + ts +
channel + byte_offset + bytes_b64. Friction entries' transcript_offset field
references byte offsets here so render --transcripts can resolve back.

progress-tail: parses gbrain's --progress-json events out of child stderr.
Phase verification asserts each scenario.expected_phases entry (dotted names
like import.files, extract.links_fs, doctor.db_checks) saw at least one event
from the actual command — proves the COMMAND ran, not that the agent obeyed
prompts.

seed-pglite: ~50 LOC SQL replay primitive for the upgrade-from-v0.18 scenario.
Existing migration helpers (test/e2e/helpers.ts) are Postgres-only; PGLite has
no equivalent. seedPglite opens a fresh PGLite, executes each statement
individually (errors name the failing one), then disconnects so gbrain init
can take over and walk forward.

53 unit tests covering registry selection, runner detection, multi-byte UTF-8
chunk-boundary safety, PIPE buffer drain, scenario load+validate, progress
event parsing, and SQL splitter.

* feat: claw-test scenario fixtures + friction-protocol skills convention

Two scenarios ship in v1 — fresh-install and upgrade-from-v0.18. Each is a
self-contained directory: brain/ (markdown pages), BRIEF.md (live-mode prompt),
expected.json (scripted-mode assertions), scenario.json (kind, expected_phases,
optional from_version + seed paths). Schema is owned by src/core/claw-test/
scenarios.ts.

upgrade-from-v0.18 ships scaffolded — seed/dump.sql is the v1.1 follow-up
(needs a real v0.18-shape PGLite dump; seed/README.md documents the gen
procedure). The harness gracefully no-ops the seed phase when dump.sql is
absent.

skills/_friction-protocol.md is a cross-cutting convention skill (like
_brain-filing-rules.md). Tells agents when to call gbrain friction log and how
to choose severity. Skills the claw-test exercises will gain a > Convention:
callout pointing here in a v1.1 sweep.

13 unit tests for the scenario loader + 'shipped scenarios load cleanly' for
both.

* feat: register gbrain claw-test + gbrain friction; CLAUDE.md + llms sync

Wires both commands into src/cli.ts CLI_ONLY allow-list and adds dispatch
in handleCliOnly so neither command requires a brain engine connection.

CLAUDE.md gains entries for src/commands/{friction,claw-test}.ts +
src/core/claw-test/ + skills/_friction-protocol.md, and a Commands section
listing all 8 new gbrain claw-test ... and gbrain friction ... invocations
with the v0.23 marker. Documents the GBRAIN_HOME write-isolation contract
and the v1 caveat (read-side host-fingerprint detection deferred to v1.1).
llms.txt + llms-full.txt regenerated via 'bun run build:llms' so the
committed generator-output gate passes.

test/e2e/claw-test.test.ts is the scripted-mode E2E. Builds a tiny shim that
delegates to 'bun run src/cli.ts' (NOT bun --compile, which doesn't bundle
PGLite's runtime assets), points the harness at it via GBRAIN_BIN_OVERRIDE,
runs --scenario fresh-install end-to-end. Asserts exit 0, zero error/blocker
friction. Includes a deliberate-break test that proves the friction signal
fires when a phase command rejects.

test/claw-test-cli.test.ts covers shipped-scenario load + agent registry +
OpenClawRunner detection (relative-path / .. / missing-bin guards) + the
GBRAIN_FRICTION_RUN_ID env handoff between harness and friction CLI.

Closes the v0.23 claw-test E2E feature.

* chore: bump version and changelog (v0.24.0)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(tests): typecheck failures + spawnWithCapture timeout headroom in CI

Three CI fixes after PR #522 landed:

1. test/agent-runner.test.ts:89 — UnavailableRunner.invoke() returns
   Promise<void> by default but the AgentRunner contract requires
   Promise<InvokeResult>. Annotate the throw-only invoke explicitly so tsc
   sees the contract is satisfied (the throw makes the body unreachable as
   far as the return type is concerned).

2. test/seed-pglite.test.ts — bun:test signature is test(name, fn, timeoutMs:
   number), not test(name, opts: {timeout}, fn). The {timeout: 30_000} object
   form was a guess that tsc on bun 1.3.13 rejects. Move the 30s cap to the
   trailing positional number arg on each PGLite-using test.

3. test/transcript-capture.test.ts — `spawnWithCapture > timeout fires
   SIGTERM/SIGKILL` blew the 10s outer cap on the GitHub runner. Two fixes:
   (a) use `exec sleep` so the child we spawn IS sleep — SIGTERM goes
   directly to it, no `/bin/sh` fork-vs-exec process-group ambiguity that
   could orphan the sleep and force the SIGKILL grace path. (b) bump outer
   cap to 30s for headroom even when the runner is slow and SIGKILL after
   the 5s grace is what actually ends the child.

* chore: rebump to v0.22.16 (next free 0.22.x patch slot per queue)

PR #506 claims v0.22.15, PR #521 claims v0.22.10, intermediate slots
(.11/.12/.13/.14) are claimed by other open PRs. v0.22.16 is the next
clean PATCH slot. v0.23.0 is claimed by PR #462 so MINOR isn't free.
This release fits the 0.22.x train; v0.23.0 lands when #462 ships.

Updates VERSION, package.json, CHANGELOG.md header, TODOS.md follow-up
labels. Code is unchanged.

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-29 23:46:36 -07:00
17c3c43783 v0.22.15 feat: frontmatter inference — zero-friction ingest for files without YAML headers (#506)
* feat: frontmatter inference — zero-friction ingest for files without YAML headers

The platonic ideal of agentic retrieval is: you throw stuff in and it
becomes knowledge. No manual schema, no frontmatter templates, no YAML
ceremony. This PR makes that real.

## The Problem

9,655 files in a real 81K-page brain have no YAML frontmatter. They import
fine (gray-matter is forgiving), but with minimal metadata:
- type defaults to 'concept' for everything
- title is the slugified filename ('2010 04 13 Apr 13 Founders Mtg')
- No date, no source, no tags, no folder-aware typing

These pages exist in the DB but are poorly classified, which degrades
search ranking, type-filtered queries, and entity resolution.

## The Fix

### 1. Directory-aware inference engine (src/core/frontmatter-inference.ts)

A rules table maps path patterns to rich metadata:

  Apple Notes/*          → type: apple-note, date from filename, source: apple-notes
  Apple Notes/YC/*       → adds tag: yc
  Apple Notes/Politics/* → adds tag: politics
  daily/calendar/*       → type: calendar-index, source: calendar
  people/*               → type: person, title from # heading
  personal/therapy/*     → type: therapy-session, date from filename
  personal/reflections/* → type: reflection, title from # heading
  writing/essays/*       → type: essay, date from filename
  companies/*            → type: company, title from # heading
  events/*               → type: event, date from filename
  (catch-all)            → type: note, title from # heading

Each rule specifies:
- type: page type for brain schema
- datePattern: 'filename' (YYYY-MM-DD prefix), 'dirname', or 'none'
- titleStrategy: 'filename' (strip date), 'heading' (first #), 'filename-full'
- source: optional source tag
- tags: optional additional tags

Title extraction cleans up filenames (strips date prefix, converts dashes
to spaces, preserves existing capitalization). Heading extraction looks at
the first 20 lines for a # heading.

Fully deterministic. No LLM calls. No network. Same file → same frontmatter.

### 2. Inline inference in import pipeline (src/core/import-file.ts)

importFromFile() now runs inference automatically when a file has no
frontmatter. The synthesized frontmatter is applied to the in-memory
content before parseMarkdown runs, so the downstream pipeline sees
well-formed YAML. The file on disk is NOT modified — inference is
DB-only unless you explicitly run `gbrain frontmatter generate --fix`.

### 3. CLI command: gbrain frontmatter generate (src/commands/frontmatter.ts)

  gbrain frontmatter generate /path/to/brain         # dry-run preview
  gbrain frontmatter generate /path/to/brain --fix   # write to files
  gbrain frontmatter generate /path/to/brain --json  # machine output

The dry-run output shows:
- Total scanned / already have frontmatter / would generate
- Breakdown by inferred type
- First 10 examples with inferred metadata and matched rule

### 4. Tests (test/frontmatter-inference.test.ts)

35 tests covering:
- Date extraction from various filename patterns
- Title extraction from filenames and headings
- Inference for every directory rule (Apple Notes, people, therapy, etc.)
- Serialization with YAML-safe quoting
- Integration: applyInference prepends frontmatter correctly
- Rules: ordering, catch-all, specificity

## What this enables

1. `gbrain sync` now imports bare markdown with rich metadata automatically
2. `gbrain frontmatter generate --fix` writes frontmatter to 9,655 files
3. Future: sync can optionally write-back inferred frontmatter to git
4. Future: rules table is extensible — new directory conventions = one rule

## Adding new directory conventions

Edit DIRECTORY_RULES in src/core/frontmatter-inference.ts:

  { pathPrefix: 'recipes/', type: 'recipe', titleStrategy: 'heading' }

Rules are matched first-to-last, most specific prefix wins. The catch-all
(empty prefix) is always last.

## Real-world test output

  Scanned: 81,479 files
  Already have frontmatter: 71,824
  Would generate: 9,655

  By type:
    apple-note: 5,861
    calendar-index: 3,201
    person: 56
    therapy-session: 60
    reflection: 12
    essay: 33
    ...

All 35 tests pass.

* fix: import basename in frontmatter generate dynamic path import

src/commands/frontmatter.ts:437 calls basename(rootPath) as a fallback
when relative(brainRoot, rootPath) returns the empty string, but the
dynamic import a few lines above only destructures { resolve, relative,
join } from 'path'. typecheck failed with TS2304: Cannot find name
'basename', and any user invocation of `gbrain frontmatter generate
<single-file>` would have crashed at runtime with a ReferenceError.

The unit tests cover frontmatter-inference module's pure functions; no
test exercises the CLI single-file branch, so the bug slipped through.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump VERSION 0.22.8 → 0.22.15 + CHANGELOG entry

Slot v0.22.15 per the queue allocator (other PRs claim v0.22.9–v0.22.14).
CHANGELOG entry written above v0.22.8 per the never-touch-shipped-entries
rule. bun.lock and llms-full.txt are unchanged (CLAUDE.md untouched, the
inference module and CLI command come in via the feature commit).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Wintermute <wintermute@garrytan.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 23:26:25 -07:00
Wintermute 0fb0c83d24 chore: gitignore export/ directory (generated output) 2026-04-30 06:11:34 +00:00
ed900c870e v0.22.14 feat(minions): bare-worker self-health-monitoring (#503)
* feat(minions): add self-health-monitoring to bare worker mode

Bare `gbrain jobs work` (without supervisor) previously had zero health
monitoring. If the Postgres connection dropped or the worker's event loop
deadlocked, the process stayed alive doing nothing — jobs piled up while
external process managers (systemd, Docker, cron) thought it was healthy.

Changes:

1. **Self-health-check timer** (worker.ts): Runs every 60s when not under
   a supervisor. Two probes:
   - DB liveness: `SELECT 1` — 3 consecutive failures → exit(1)
   - Stall detection: waiting jobs + 0 in-flight + no completions for 5m
     → warning; 10m → exit(1)

2. **GBRAIN_SUPERVISED env var** (supervisor.ts): Supervisor sets this on
   its child worker to prevent duplicate health checks. The supervisor
   already has its own health monitoring.

3. **RSS watchdog default** (jobs.ts): Bare workers now default to
   `--max-rss 2048` (matching supervisor default). Opt out: `--max-rss 0`.

4. **--health-interval flag** (jobs.ts): Configurable health check period.
   `--health-interval 0` disables. Default: 60000ms.

5. **parseMaxRssFlag returns undefined** when flag is absent (vs 0), so
   callers can distinguish 'not set' from 'explicitly disabled'.

The design ensures bare workers get supervisor-grade monitoring while
remaining compatible with any external process manager — the worker just
exits with code 1 on detected failure, letting the PM handle the restart.

Tests: 4 new tests (3 worker health, 1 supervisor env var). All 178 pass.

* fix(minions): harden bare-worker self-health-check after multi-round review

Layered fixes from 5 rounds of plan-eng-review + codex outside voice on top of
the original PR #503 (feat: bare-worker self-health-monitoring). Every change
below is in service of "fail-stop into the operator's process manager" without
introducing new ways the library can kill its caller.

worker.ts:
- MinionWorker now extends EventEmitter; emits `'unhealthy'` event with structured
  reason payload (`db_dead` | `stalled`). CLI subscribes; library no longer calls
  process.exit directly.
- emitUnhealthy() falls back to process.exit(1) when listenerCount('unhealthy') === 0
  so direct API consumers without a listener inherit the pre-refactor fail-stop
  default. Inline paths opt out via healthCheckInterval=0.
- Stall detection: count(*) query now filters by registered handler names
  (`AND name = ANY($2::text[])`) so workers with handlers for {embed,sync} don't
  false-positive when waiting jobs of unhandled names accumulate.
- Stall exit threshold measured from lastCompletionTime (not from warn-since), so
  defaults of 5min warn / 10min exit fire at idle=10min total — matching the
  documented contract.
- Recursive setTimeout pattern with running flag replaces setInterval, eliminating
  callback overlap on slow DB probes.
- DB liveness probe wrapped in Promise.race against AbortController-driven
  timeout (default 10s) so a hung executeRaw can't wedge the recursive chain
  forever. Hung probes count as failures and feed dbFailExitAfter.
- Constructor validates stallExitAfterMs > stallWarnAfterMs and throws loudly
  on misconfiguration. Internal timer-installation invariants documented inline.
- GBRAIN_SUPERVISED env-var check tightened from `!!process.env.X` to `=== '1'`.

types.ts:
- Added 5 new MinionWorkerOpts fields with documented contracts:
  healthCheckInterval, stallWarnAfterMs, stallExitAfterMs, dbFailExitAfter,
  dbProbeTimeoutMs.
- Exported `UnhealthyReason` discriminated union for the 'unhealthy' event payload.

supervisor.ts:
- GBRAIN_SUPERVISED=1 injected on the spawned worker child's env so the child's
  self-health timer is skipped (no double-monitoring).
- setInterval(callback, healthInterval) gated behind `> 0`, so the
  `--health-interval 0` documented disable contract actually disables instead
  of producing a tight DB-hammer loop.

jobs.ts:
- `gbrain jobs work` subscribes to 'unhealthy' and calls process.exit(1) at the
  CLI layer. Default --max-rss bumped from 0 to 2048 (matches supervisor default;
  catches memory-leak stalls that previously went undetected).
- New --health-interval flag with aggressive validation (NaN/negative/sub-1000ms
  rejected; parity with --max-rss) on both `jobs work` and `jobs supervisor`.
- `jobs submit --follow` and `jobs smoke` now pass healthCheckInterval=0 to
  disable the self-health timer entirely. These are inline/one-shot flows with
  no PM to restart them; the no-listener emitUnhealthy fallback could otherwise
  trip on a DB blip and kill the user's CLI session.
- parseMaxRssFlag returns `number | undefined` (was `number`) so callers can
  distinguish absent (use the default) from explicit-disable (--max-rss 0).

doctor.ts:
- New queue_health subcheck reports RSS-watchdog kills in the last 24h.
  Detects via exact-match `error_text = 'aborted: watchdog'` (the worker's
  failJob signature when gracefulShutdown('watchdog') aborts in-flight jobs)
  scoped to status IN ('dead','failed'). Tight match avoids over-counting parent
  jobs that propagate child failures via on_child_fail='fail_parent'.

* test(minions): self-health behavior + regression tests

7 new tests covering the production failure modes that drove the original PR,
plus regressions for fixes landed during multi-round review.

minions.test.ts:
- DB 3-strike → 'unhealthy' event with reason='db_dead' (the production-incident signature)
- DB recovery resets failure counter (no exit on intermittent failures)
- Stall warn-then-exit (clock-driven; idleMs > stallExitAfterMs is the new contract)
- inFlight > 0 blocks stall detection (long-running legitimate jobs don't false-trip)
- Regression for D1 fix: jobs of unregistered handler names don't trigger stall exit;
  also captures the SQL via probe engine and asserts the predicate text contains
  `name = ANY` so a future refactor that drops the filter is caught at test time.
- Regression for R3 constructor validation: throws when stallExitAfterMs <= stallWarnAfterMs
  (covers both `<` and `=` cases); defaults still construct cleanly.

supervisor.test.ts:
- Regression for R3: supervisor with healthInterval=0 completes a normal lifecycle
  within 10s. A tight setInterval(0) loop (the bug we fixed) would saturate the
  event loop and slow this past the cap.

Tests use a Proxy-based engine helper (makeProbeEngine) that intercepts SELECT 1
and the count(*) WHERE status='waiting' query while passing through everything
else to the real PGLite engine. This isolates health-check semantics from claim
plumbing without mocking the entire engine surface.

* docs(v0.22.14): migration walkthrough + follow-up TODOs

skills/migrations/v0.22.14.md (new):
- Pre-flight per-PM restart-policy table (systemd Restart=always, Docker
  restart: always, launchd KeepAlive, cron watchdog, supervisord autorestart).
  v0.22.14 makes bare-worker behavior fail-stop — without an external restart
  loop the worker exits and stays dead. Migration calls this out loudly so
  OpenClaw/Hermes-style downstream agents can verify their PM before upgrade.
- Five new MinionWorkerOpts fields documented with defaults and rationale.
- Worker-side process.exit(1) fallback semantics explained: CLI subscribes to
  'unhealthy', but direct API consumers without a listener inherit fail-stop.
- AskUserQuestion-driven flow for the --max-rss 2048 default (raise / opt out /
  keep) with concrete edits per PM (systemd unit, cron line, Docker compose,
  launchctl plist).
- Verification commands (gbrain jobs stats, gbrain doctor --json, RSS check)
  and a triage paragraph for opening an issue if anything fails.

TODOS.md:
- v0.22.15 embed cooperative-abort (P0, daily pain): plumb signal through
  runPhaseEmbed → embed.ts → embedBatch; check signal.aborted between OpenAI
  batch calls and between slugs. Closes the daily wedge where embed > 600s
  timeout dead-letters the job but keeps running, holding gbrain_cycle_locks
  until the lock TTL expires. PR #503 catches the symptom (worker stalled);
  this captures the cause-side fix that's the real production resolution.
- v0.23+ bare-worker engine reconnect parity: extract supervisor's
  reconnect-then-fail pattern (#406) into MinionWorker so transient PgBouncer
  blips don't force a full process restart.
- v0.23+ minion_workers heartbeat table for queue_health doctor check (B7
  follow-up): replace lock_until proxy with ground-truth worker liveness
  signal so doctor stops crying wolf on legitimately idle workers.

* chore: bump version and changelog (v0.22.14)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: Wintermute <wintermute@garrytan.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-29 23:06:16 -07:00
e96f054cf0 v0.22.13 feat: parallel sync — bounded concurrent imports (#490)
* feat: parallel sync — bounded concurrent imports (#489)

gbrain sync --concurrency N (alias --workers N) parallelizes the import
phase using per-worker Postgres engine instances with an atomic queue
index (same proven pattern as gbrain import --workers N).

Auto-concurrency: when a sync touches >100 files and the user didn't
explicitly set --concurrency, defaults to 4 workers. Small incremental
syncs (<50 files) stay serial. Full syncs auto-detect Postgres and
default to 4 workers.

Minion sync handler defaults to concurrency=4, configurable via job
params: {"concurrency": 8}.

Delete and rename phases remain serial (order-dependent, fast).
PGLite falls back to serial automatically (single-connection engine).

Changes:
- src/commands/sync.ts: SyncOpts.concurrency, parallel import loop in
  performSync incremental path, --workers passthrough in performFullSync
- src/commands/jobs.ts: sync handler accepts concurrency param (default 4)
- CHANGELOG.md: v0.23.0 parallel sync entry

All 37 existing sync tests pass. Typecheck clean.

* feat: shared concurrency policy + db-lock primitive

src/core/sync-concurrency.ts — single source of truth for autoConcurrency()
+ parseWorkers() + shouldRunParallel() + constants. Replaces three drifted
call-site policies (performSync, performFullSync, jobs handler).

src/core/db-lock.ts — generic tryAcquireDbLock(engine, lockId, ttlMinutes)
over the existing gbrain_cycle_locks table. Parameterized lock id so
performSync (gbrain-sync) can nest cleanly under cycle.ts (gbrain-cycle)
without deadlock.

test/sync-concurrency.test.ts — 17 cases covering PGLite-forces-serial,
explicit override clamping, auto-path threshold, parseWorkers validation
(rejects 0, negatives, NaN, decimals, trailing chars).

No consumers yet; subsequent commits wire sync.ts, import.ts, and jobs.ts
to use these helpers.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: harden performSync — writer lock, head-drift gate, engine.kind

CODEX-2: wrap performSync body in a gbrain-sync DB lock so two concurrent
syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot
both read last_commit, both write it unconditionally, and let the last
writer win. cycle.ts continues to hold gbrain-cycle for its broader scope;
the two ids nest cleanly.

CODEX-3: capture git HEAD at sync entry, re-rev-parse after the import
phase, refuse to advance last_commit if HEAD drifted (someone ran
git checkout / git pull mid-sync). Vanished files now go into failedFiles
instead of silent-skip — same gating mechanism, no more bookmark advance
past unimported work.

A1: replace both PGLite detection sites with engine.kind === 'pglite'.
The constructor.name sniff is gone (breaks under bundling) and so is the
inconsistent config?.engine string check.

A2: connect worker engines serially into an array, run inside try/finally
so disconnect always fires — even on partial connect failure, OOM, or
mid-import abort. Prior Promise.all(...disconnect) leaked the 8 worker
connections on any panic path.

Q1: explicit --workers / opts.concurrency now bypasses the >50-file floor.
User opt-in beats the auto-path safety net.

Q3: drop the config!.database_url! non-null assertions; fall back to serial
when database_url is unset instead of crashing on TypeError.

Q4: worker-count banner moves from console.log to console.error so stdout
stays clean for --json output.

test/sync-parallel.test.ts — 7 cases over PGLite covering the bookmark
gate under concurrency request, the head-drift gate, vanished-file
failure capture, PGLite-stays-serial, and the writer-lock contract.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: import.ts — engine.kind discriminator, worker try/finally, parseWorkers

A1: replace the config?.engine === 'pglite' string sniff with
engine.kind === 'pglite' to match sync.ts and the v0.13.1 contract.

A2: wrap worker engine creation + the parallel loop in try/finally so
disconnects always fire — same pattern as sync.ts. Worker engines now
push onto an array as they connect (rather than Promise.all) so the
finally block can clean up partial-connect state.

Q2: route --workers parsing through the shared parseWorkers() helper.
parseInt-with-no-validation is gone — '0', '-3', 'foo', '1.5' now exit
with a clear error message instead of silently falling through.

Q3: drop the config!.database_url! non-null assertion; fall back to
serial when database_url is unset.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: jobs.ts sync handler — resolve sourceId, autoConcurrency

CODEX-1: resolve sourceId at handler entry by looking up sources.local_path.
Mirrors cycle.ts:480's autopilot-cycle fix (PR #475). Without this, every
Minion sync job on a multi-source brain reads global config.sync.last_commit
instead of the per-source anchor, which on a regularly-GC'd repo can drop
out of git history and trigger 30-min full reimports every cycle.

The handler accepts an optional sourceId job param for callers that want
to override; falls back to the resolveSourceForDir lookup when absent.

CODEX-4: replace the hardcoded concurrency=4 default with the shared
autoConcurrency policy. Behavior is now consistent between CLI sync,
the Minion handler, and the autopilot cycle's sync phase. Jobs that
request a specific concurrency via job.data.concurrency still win.

noEmbed default stays at true — embed is a separate job (submit
gbrain embed --stale, OR rely on the autopilot cycle's embed phase).
The doc comment makes that contract explicit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: e2e parallel sync against real Postgres + benchmark

DATABASE_URL-gated E2E coverage that PGLite-only tests can't reach:

T2 — happy path: 60 files imported at concurrency=4, all 60 pages land
in the DB, with a pg_stat_activity probe before/after to confirm worker
engines (4 × 2 connections) actually disconnected.

P4 — benchmark: 120-file fixture, serial vs concurrency=4 timing.
Emits a single-line `SYNC_PARALLEL_BENCH 120 files | serial=Xms |
parallel(4)=Yms | speedup=Zx` so the CHANGELOG can quote a real
number instead of an unbacked '~4×' claim. Asserts parallel <=
serial * 1.5 to allow for noisy CI but fail genuine regressions.

Skips gracefully when DATABASE_URL is unset (consistent with the rest
of test/e2e/).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: v0.22.10 release notes + sync follow-up TODO

VERSION + package.json + bun.lock: 0.22.5/0.22.6 → 0.22.10. Repo had
existing drift between VERSION and package.json on master; this commit
brings them back in sync at the bumped value.

CHANGELOG.md: v0.22.10 entry replaces the unfinished v0.23.0 stub from
PR #490's original commit. Voice-rule clean (no em dashes, no AI
vocabulary), real benchmark numbers from the new E2E test
(serial=289ms parallel(4)=221ms speedup=1.31x), additive worker-pool
note (A3), 'To take advantage of v0.22.10' self-repair block per
CLAUDE.md convention.

TODOS.md: A4 follow-up filed — plumb resolved database_url through
SyncOpts so performSync / performFullSync / import.ts don't each call
loadConfig() separately. Deferred to a future patch; not on the
v0.22.10 critical path.

Patch (not minor) framing held even though new CLI surface lands here;
release-notes prose names the behavior change explicitly so users know
to read them.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: update CLAUDE.md + README for v0.22.10 sync hardening

CLAUDE.md:
- New "Key files" entries for src/core/sync-concurrency.ts and
  src/core/db-lock.ts (both v0.22.10).
- New "Key files" entry for src/commands/sync.ts (covers the lock,
  head-drift gate, engine.kind discriminator, vanished-file failure
  capture, parallel branch wiring).
- Updated src/commands/jobs.ts entry with v0.22.10 sourceId
  resolution + autoConcurrency policy + noEmbed contract.
- Added test/sync-concurrency.test.ts and test/sync-parallel.test.ts
  to the unit-test list with case counts.
- Added test/e2e/sync-parallel.test.ts to the E2E section with the
  SYNC_PARALLEL_BENCH grep marker for CHANGELOG quoting.
- Added "Key commands added in v0.22.10" section: gbrain sync --workers,
  gbrain import --workers (parseWorkers validation).

README.md: added --workers flag to the IMPORT section's gbrain sync
and gbrain import lines, with the >100-file auto-parallelize note.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump version slot to v0.22.13

VERSION 0.22.10 → 0.22.13. Master moved to 0.22.8 plus claimed slots
0.22.9-0.22.12 in sibling workspaces; 0.22.13 is the next free slot for
this PR's parallel-sync hardening work.

Updated all v0.22.10 references in CHANGELOG.md (release header +
self-repair block), TODOS.md (D-PR490-1 follow-up tag), CLAUDE.md
(Key files entries + tests + commands subsection), and the inline
v0.22.10 markers in src/core/sync-concurrency.ts, src/core/db-lock.ts,
src/commands/sync.ts, src/commands/import.ts, src/commands/jobs.ts,
test/sync-parallel.test.ts, test/e2e/sync-parallel.test.ts.

No behavioral change. CHANGELOG header rewrite, content unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: regenerate llms-full.txt for v0.22.13 doc updates

CI's build-llms generator test failed because llms-full.txt was stale
relative to the README + CLAUDE.md updates this PR added (--workers
flag in the IMPORT section, sync-concurrency.ts/db-lock.ts/sync.ts
entries in the Key files section).

Per CLAUDE.md: "Run \`bun run build:llms\` after adding a new doc."
The test test/build-llms.test.ts:67 verifies committed bundles match
generator output — now they do again.

llms.txt was already in sync (no curated config additions); only
llms-full.txt needed the regen.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: root <root@localhost>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 22:53:41 -07:00
Garry TanandClaude Opus 4.7 59050f2baf Merge origin/master into feat/brain-first-convention
Master moved from v0.21.0 → v0.22.9 (10 PRs: source-aware ranking,
frontmatter-guard, autopilot fixes, MCP HTTP transport, schema-bootstrap
self-healing, doctor batch loads, sync error-code summaries).

Conflicts resolved:
- VERSION: kept this branch's 0.24.0 per CLAUDE.md "branch-scoped
  CHANGELOG + VERSION" rule (master's 0.22.9 is below us).
- package.json version: kept 0.24.0.
- CHANGELOG.md: stitched to keep this branch's v0.24.0 entry on top,
  followed by master's new v0.22.9 → v0.22.0 entries, followed by
  v0.21.0 and below. Sequence is now monotonic above v0.21.0.
  Auto-merger had tangled my v0.24.0 entry with master's v0.22.6.1
  via shared "### Itemized changes" tokens; fixed by checkout --ours
  + manual splice.

Privacy scrub (CI guard from this branch caught new master leaks):
- src/core/search/source-boost.ts — hardcoded default boost-map key
  renamed: 'wintermute/chat/' → 'openclaw/chat/'. Behavior delta:
  default downgrade now matches openclaw/chat/ prefix instead of the
  fork-specific name. Users with the old shape can override via
  GBRAIN_SOURCE_BOOST="openclaw/chat/:0.5,your/path/:0.5".
- test/e2e/engine-parity.test.ts, test/e2e/search-swamp.test.ts,
  test/sql-ranking.test.ts — fixture slugs renamed to match the new
  default key.
- CHANGELOG.md (master's v0.22.0 entry prose) — replaced 5 references
  to the banned name in user-facing release notes.
- docs/integrations/pre-commit.md — replaced "fork (Wintermute, Hermes,
  OpenClaw)" with "your OpenClaw" per CLAUDE.md privacy mapping.
- llms-full.txt regenerated.

Verification:
- bun install — 0 new packages
- bun run typecheck — clean
- scripts/check-privacy.sh — exit 0
- This branch's 5 test files (skillpack-install + skillify-scaffold +
  routing-eval-cli + privacy-script-wired + build-llms) — 63/63 pass
- test/sql-ranking.test.ts — 40/40 pass against the renamed key

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 11:51:37 -07:00
Garry TanandClaude Opus 4.7 11d4de336e docs: update project documentation for v0.24.0
Auto-discovered drift via /document-release after the v0.24.0 hardening
pass landed. All factual corrections clearly warranted by the diff.

CLAUDE.md:
- Skillpack installer: documented the cumulative-slugs receipt comment,
  install --all prune semantics, unknown-row preserve+warn behavior,
  and pre-v0.24 silent upgrade. Was previously vague about
  "tracks a skill manifest so install --update diffs cleanly" without
  explaining what the receipt is or why it matters.
- routing-eval: replaced the false claim that --llm "opts into a Haiku
  tie-break layer for CI." Now correctly describes the placeholder
  semantic landed in v0.24.0 (stderr notice + structural-only run).

README.md:
- Skillpack section: added one paragraph on the receipt comment + the
  user-visible stderr message for hand-added rows. Connects the safe
  rerun promise to the v0.24.0 implementation that actually enforces it.

CONTRIBUTING.md:
- Running tests section: now recommends `bun run test` (full CI guard
  chain + typecheck + tests) before pushing. Names each guard so new
  contributors understand what catches what. The privacy guard (newly
  wired in v0.24.0) is one of these — without `bun run test` you'd skip
  it locally and find out from CI.

llms-full.txt: regenerated to reflect CLAUDE.md changes.

Verification: full guard chain green locally (privacy + jsonb + progress
+ wasm + typecheck).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-25 23:05:18 -07:00
Garry TanandClaude Opus 4.7 1edbf024f7 chore: bump VERSION to 0.24.0 + new CHANGELOG entry
Bump branch version above master's v0.21.0 per CLAUDE.md
"CHANGELOG + VERSION are branch-scoped" rule. The new v0.24.0 entry at
the top of CHANGELOG covers what THIS branch adds vs master:

- routing-eval --llm honesty pass (4-surface contract drift fix)
- skillpack installer cumulative-receipt + unknown-row preserve+warn
  (the Codex-caught regression that would have shipped in master if
  the original v0.19.0 had landed without this branch's review pass)
- skillify scaffold resolver-row regex broadening (backtick + quoted
  + bare forms; idempotency contract preserved under hand-editing)
- 5 banned-name leaks scrubbed from public artifacts
- check-privacy.sh wired into CI test chain + regression guard test
- 7 stale v0.17/v0.18 version labels rewritten across 5 files
- Tier 2 (LLM-skills E2E) promoted from schedule-only to required per-PR

VERSION 0.21.0 → 0.24.0
package.json version field synced.
llms.txt + llms-full.txt regenerated (no content drift; sizes match).

Test suite: 62/62 green across the 5 test files this branch added or
extended (routing-eval-cli, privacy-script-wired, skillpack-install,
skillify-scaffold, build-llms).

CI guards: privacy + jsonb + progress + wasm + typecheck all clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-25 23:00:19 -07:00
Garry TanandClaude Opus 4.7 8efbd664f7 fix: scrub banned fork name from public artifacts
The privacy guard wired into the test chain in this branch caught 5
pre-existing references to the banned OpenClaw fork name in CHANGELOG.md
(2x), skills/migrations/v0.19.0.md (1x), src/cli.ts (1x), and
src/commands/sync.ts (1x). All originated in master's v0.19.0 release
notes and migration doc when the privacy script existed but wasn't
wired into CI yet.

Replacements per CLAUDE.md privacy mapping:
- Origin-story copy (CHANGELOG layer narratives, code comments naming
  the production deployment that drove the feature) → "Garry's OpenClaw"
- Reader-facing migration step → "your OpenClaw"

No code semantics changed. Comments + headings only.

Verification: scripts/check-privacy.sh exits 0, full CI guard chain
green (privacy + jsonb + progress + wasm + typecheck).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-25 22:56:31 -07:00
Garry TanandClaude Opus 4.7 39bec4dcea Merge origin/master into feat/brain-first-convention
Master moved from v0.19.0 → v0.21.0 (Code Cathedral I + II) while this
branch worked on production-readiness hardening.

Conflicts resolved:
- TODOS.md: kept master's v0.21.0 Code Cathedral II follow-ups (B2,
  A4, C6, cross-file edge resolution) AND this branch's P3 dev-experience
  TODO for PGLite test parallelism on M-series Macs.
- package.json scripts.test: combined both check chains.
  Now runs check-privacy.sh (this branch) + check-jsonb-pattern.sh +
  check-progress-to-stdout.sh + check-wasm-embedded.sh (master) +
  typecheck + bun test --timeout=60000 (master).

CHANGELOG state after merge: master's v0.21.0 (Code Cathedral II) sits
at top, this branch's v0.19.0 (skillify + AGENTS.md compat) sits below
master's later entries. The duplicate v0.19.0 entries (master's
"code-first brain" 0.19.0 dated 2026-04-23 + this branch's
"skillify loop" 0.19.0 dated 2026-04-22) reflect master's existing
state — not introduced by this merge.

VERSION is 0.21.0 (master's). A follow-up commit can bump to v0.21.1
and reframe this branch's CHANGELOG entry above v0.21.0 per CLAUDE.md
convention. Out of scope for this merge.

Verification:
- bun install — 3 new packages from master (web-tree-sitter etc.)
- bun run typecheck — clean
- bun test (this branch's new tests): 55/55 pass
  (skillpack-install, skillify-scaffold, routing-eval-cli, privacy-script-wired)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-25 22:50:59 -07:00
Garry TanandClaude Opus 4.7 399b9a4139 feat: v0.19.0 — skillify loop + AGENTS.md compat + brain-first convention
This is the v0.19.0 release. The branch ships four new CLI commands, a
refactor to check-resolvable, and an expansion of the brain-first
convention for sub-agent tool discovery. The original commit message
described only the convention expansion, undercounting the scope by ~5x;
this amend captures the full release.

NEW COMMANDS

- gbrain skillify scaffold <name>     — 4 stub files + idempotent resolver row
- gbrain skillify check [path]        — 10-item post-task audit (promoted)
- gbrain skillpack list / install     — curated 25-skill bundle, atomic install
- gbrain skillpack diff <name>        — per-file diff preview
- gbrain routing-eval                 — dedicated CI verb for Check 5 fixtures

CHECK-RESOLVABLE REFACTOR

- Accepts AGENTS.md as a resolver file alongside RESOLVER.md, at either
  the skills directory or one level up (workspace root layout).
- Auto-derives the skill manifest by walking skills/*/SKILL.md when
  manifest.json is missing.
- Splits ResolvableReport into errors[] + warnings[] so advisory checks
  (filing audit, routing gaps, DRY violations) don't break CI by default.
- New --strict opt-in flag promotes warnings to exit 1.

BRAIN-FIRST CONVENTION

- skills/conventions/brain-first.md expanded from 5-step lookup guide to
  full sub-agent reference: tool inventory, lookup chain, score thresholds,
  authority hierarchy, sync rules, entity page conventions, sub-agent
  propagation rule.

PRODUCTION-READINESS HARDENING (this branch's review pass)

- routing-eval --llm: emits stderr placeholder notice + runs structural
  layer only. README, CHANGELOG, CLI help all rewritten consistently.
  Was a silent no-op against documented contract.
- skillpack installer: receipt comment in fence (cumulative-slugs="...")
  preserves single-skill-install accumulation while letting install --all
  prune removed bundle skills cleanly. Unknown rows preserved + stderr
  warning for the operating agent. Pre-v0.19 fences upgrade silently.
- skillify scaffold: resolver-row regex broadened to detect backticked,
  quoted, and bare path forms. No duplicate row on --force after the
  user normalizes formatting.
- scripts/check-privacy.sh: now wired into package.json test chain so
  the wintermute-ban rule is actually enforced. New regression test.
- E2E Tier 2 (LLM skills) promoted from schedule-only to required per-PR
  CI. Local Tier 1 + Tier 2 verified clean.
- Stale v0.17/v0.18 version labels rewritten across new files.

TESTS

- test/routing-eval-cli.test.ts: 4 cases covering --llm warn semantics
- test/privacy-script-wired.test.ts: regression guard for CI wiring
- test/skillpack-install.test.ts: 4 new cases for receipt + cumulative
  + unknown-row preserve+warn + pre-v0.19 upgrade path
- test/skillify-scaffold.test.ts: 4 new cases for broadened regex

VERIFICATION

- bun test: 2237 pass / 18 known PGLite-contention flakes (CI green;
  documented as P3 dev-experience in TODOS.md)
- bun run typecheck: clean
- bun run test:e2e: 18/19 files green (1 pre-existing flake on master,
  not caused by this branch — verified via git stash)
- llms.txt + llms-full.txt regenerated to match README + CHANGELOG

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-25 22:10:32 -07:00
138 changed files with 13376 additions and 329 deletions
+4 -1
View File
@@ -44,7 +44,10 @@ jobs:
tier2:
name: Tier 2 (LLM Skills)
runs-on: ubuntu-latest
if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
# Runs on every push/PR now (promoted from schedule-only in v0.19.0).
# Tier 1 must pass first; Tier 2 uses OPENAI_API_KEY + ANTHROPIC_API_KEY
# from repo/org secrets. Nightly + manual triggers still supported via
# the workflow-level `on:` list.
needs: tier1
services:
postgres:
+9
View File
@@ -17,4 +17,13 @@ eval/data/world-v1/world.html
# BrainBench amara-life-v1 Opus cache (regenerate via eval:generate-amara-life)
eval/data/amara-life-v1/_cache/
# claw-test E2E build cache (shim + scratch outputs)
test/.cache/
.claude/
export/
# Tier 3 PGLite snapshot fixture (built on demand by build:pglite-snapshot)
test/fixtures/pglite-snapshot.tar
test/fixtures/pglite-snapshot.version
+10 -3
View File
@@ -43,9 +43,16 @@ writing or reviewing an operation, consult `src/core/operations.ts` for the cont
## Before shipping
Run `bun test` plus the E2E lifecycle described in `./CLAUDE.md` (spin up the test
Postgres container, run `bun run test:e2e`, tear it down). Ship via the `/ship` skill,
not by hand.
Easiest path: `bun run ci:local` runs the full CI gate inside Docker (gitleaks,
unit tests with `DATABASE_URL` unset, then all 29 E2E files sequentially against a
fresh pgvector container) and tears down. Use `bun run ci:local:diff` for the
diff-aware subset during fast iteration on a focused branch. Requires Docker
(Docker Desktop / OrbStack / Colima) and `gitleaks` (`brew install gitleaks`).
Manual path: `bun test` plus the E2E lifecycle described in `./CLAUDE.md` (spin
up the test Postgres container, run `bun run test:e2e`, tear it down).
Ship via the `/ship` skill, not by hand.
## Privacy
+648 -9
View File
@@ -2,6 +2,643 @@
All notable changes to GBrain will be documented in this file.
## [0.24.0] - 2026-04-26
## **The skillify loop stops lying. Privacy guard runs, `--llm` is honest, ghost rows go away.**
## **Plus: Tier 2 LLM-skill tests now block every PR, not just the nightly cron.**
v0.19.0 shipped four new CLI commands (`skillify`, `skillpack`, `routing-eval`, `skillify-check`) and got rave coverage. v0.24.0 is the production-hardening pass on top of that: every public contract that lied about itself, every silent footgun, every CI guard that wasn't wired up. No new features. No new commands. Just the unsexy fixes that turn a feature release into a production release.
The biggest save: the skillpack installer would have silently deleted your skills. The original v0.19 design's "rebuild managed block" path was load-bearing wrong — a user installing `alpha` then later running `gbrain skillpack install beta` alone would have lost `alpha`. Codex caught it during cross-model review. The fix preserves cumulative-install semantics via a receipt comment in the fence: `<!-- gbrain:skillpack:manifest cumulative-slugs="alpha,beta,..." -->`. Old fences upgrade silently. User-added rows survive with a stderr warning telling the operating agent to investigate. `install --all` is now the only path that prunes; per-skill install never destroys what it didn't install.
The biggest unsexy fix: `gbrain routing-eval --llm` was a documented feature that did nothing. README, CHANGELOG, and CLI help all said it ran an LLM tie-break layer. The code returned structural-only results with no warning, no error, no signal at all. v0.24.0 makes the flag honest across all four touchpoints. Until the LLM layer ships, `--llm` emits a stderr placeholder notice and runs structural. CI logs see it. Docs match the code. The release notes don't lie.
The CI fix nobody asked for: `scripts/check-privacy.sh` exists in the repo to enforce the OpenClaw fork-name ban from `CLAUDE.md:550`. It was never wired into anything. v0.24.0 prepends it to `package.json`'s `"test"` chain alongside the other `check-*.sh` guards. A regression test asserts the wiring stays. The first run caught 5 banned-name references that had been sitting in master's `CHANGELOG.md`, `src/cli.ts`, `src/commands/sync.ts`, and `skills/migrations/v0.19.0.md` for releases — fixed in the same wave.
### The numbers that matter
Counted against this branch's review trail and the local test suite:
| Metric | BEFORE v0.24.0 | AFTER v0.24.0 | Δ |
|---|---|---|---|
| `routing-eval --llm` behavior matches docs | no | yes | fixed |
| Public-contract drift surfaces fixed | 4 (README, CHANGELOG, CLI help, runtime) | 0 | 4 |
| `gbrain skillpack install <name>` preserves prior installs | yes (was a happy accident) | yes (with receipt + regression test) | locked |
| Regression test guarding cumulative-install semantics | none | `test 8a` ("install alpha; then install beta; assert both") | +1 |
| Banned-name leaks in tracked files | 5 (master state) | 0 | 5 |
| `check-privacy.sh` runs in CI | never | every PR (via `bun run test`) | wired |
| Tier 2 LLM-skill E2E gates each PR | no (nightly cron only) | yes | wired |
| Stale `v0.17/v0.18` version labels in new code | 7 sites across 5 files | 0 | 7 |
| Skillify scaffold idempotency under hand-edited resolver | backtick-only detection | backtick + quoted + bare | fixed |
Cross-model review trail: **CEO + Eng + Codex outside voice**. 14 user decisions captured, 0 unresolved, 1 critical Codex catch (the cumulative-install regression that would have shipped). Two-model review caught a one-model-blind spot. The receipt design is in `src/core/skillpack/installer.ts:applyManagedBlock`.
### What this means for builders
Nothing breaks. `gbrain upgrade` is the path. Existing brains: no schema migration. Existing AGENTS.md fences without a receipt comment auto-upgrade silently on the next `gbrain skillpack install` (one-time clean rebuild, no warnings). User-added skill rows inside the fence now survive reinstalls with a clear stderr breadcrumb: `[skillpack] unknown row in managed block: "<slug>" — Investigate: user-added skill, hand-edited fence, or typo?`
If you ship custom CI: `bun run test` now gates `check-privacy.sh` alongside the existing `check-jsonb-pattern.sh`, `check-progress-to-stdout.sh`, and `check-wasm-embedded.sh`. If you grepped through gbrain's source in your own CI, no surface change. If you previously ran `gbrain routing-eval --llm` expecting an LLM pass, you'll now see a stderr line telling you what's actually happening and your scripts keep working — exit code is still 0/1 based on structural results. Tier 2 (`test/e2e/skills.test.ts`) now runs on every PR using existing repo secrets. Adds ~3-5 min per PR for real protection against LLM-adjacent regressions.
## To take advantage of v0.24.0
`gbrain upgrade` does this automatically. To verify:
1. **Binary version:**
```bash
gbrain --version # should say 0.24.0
```
2. **`--llm` honesty:**
```bash
gbrain routing-eval --llm 2>&1 | grep -i placeholder
# expect: "[routing-eval] --llm flag is a placeholder in this release..."
```
3. **Skillpack receipt + cumulative semantics:**
```bash
gbrain skillpack install <name>
grep "gbrain:skillpack:manifest cumulative-slugs" $OPENCLAW_WORKSPACE/AGENTS.md
# expect a receipt line listing every gbrain-installed slug
```
4. **Privacy guard wired:**
```bash
grep "check-privacy.sh" package.json
# expect a hit in scripts.test
```
5. **If anything fails,** file an issue at https://github.com/garrytan/gbrain/issues with the output of `gbrain doctor` and which step broke.
No schema migration. Existing brains work unchanged.
### Itemized changes
#### Fixed
- **`gbrain routing-eval --llm`** is no longer a silent no-op. CLI emits a stderr placeholder notice; `--json` mode preserves clean stdout JSON with the warning on stderr only (no bleed). README:294, CHANGELOG entry, and CLI help text all rewritten to match the actual behavior. Tests in `test/routing-eval-cli.test.ts`.
- **Skillpack installer** now embeds a receipt comment (`<!-- gbrain:skillpack:manifest cumulative-slugs="..." version="..." -->`) inside the managed-block fence on every install. Per-skill installs accumulate via union(prior receipt, this-call slugs); `install --all` prunes slugs no longer in the bundle (the only prune path). Unknown rows inside the fence (user hand-adds, third-party bundles, typos) survive reinstalls with a stderr `Investigate:` breadcrumb. Pre-v0.24.0 fences upgrade silently on first install. Tests in `test/skillpack-install.test.ts` cover all four paths including the regression-guard "install alpha; install beta; assert both present."
- **`gbrain skillify scaffold --force`** no longer creates duplicate resolver rows when the existing row uses non-backticked path forms. The detection regex now matches backticked, single-quoted, double-quoted, and bare forms, with anchored boundaries to prevent false-matching shared-prefix slugs (e.g., `demo` vs `demo-extended`). Tests in `test/skillify-scaffold.test.ts`.
- **5 banned OpenClaw fork-name leaks** scrubbed from public artifacts (`CHANGELOG.md`, `skills/migrations/v0.19.0.md`, `src/cli.ts`, `src/commands/sync.ts`). All originated in earlier releases when the privacy script existed but wasn't wired to CI. Replacements per `CLAUDE.md:550` (origin-story → "Garry's OpenClaw"; reader-facing → "your OpenClaw").
- **Stale `v0.17`/`v0.18` version labels** removed from 5 files (`src/core/routing-eval.ts`, `src/core/filing-audit.ts`, `src/commands/check-resolvable.ts`, `src/commands/skillify.ts`, `src/commands/skillpack.ts`). Replaced with version-agnostic phrasing or current-release references.
#### Changed
- **`package.json` `"test"` script** now prepends `scripts/check-privacy.sh` to the existing chain. Test failure if the banned fork name appears anywhere in tracked files.
- **`.github/workflows/e2e.yml`** Tier 2 job (`test/e2e/skills.test.ts`, requires `OPENAI_API_KEY` + `ANTHROPIC_API_KEY`) promoted from schedule-only to required per-PR CI. Same secrets, same install path, same workflow YAML structure — just removed the `if: github.event_name == 'schedule' or workflow_dispatch` guard.
#### Added (tests)
- **`test/routing-eval-cli.test.ts`** (4 cases) — `--llm` placeholder behavior across human + JSON modes, exit-code preservation, regression guard for the silent-no-op state.
- **`test/privacy-script-wired.test.ts`** (3 cases) — asserts `check-privacy.sh` exists and is executable, asserts `package.json` `scripts.test` references it, asserts the `check:privacy` convenience alias is present.
- **`test/skillpack-install.test.ts`** (+4 cases) — cumulative-install regression guard, full-bundle prune semantics, unknown-row preserve+warn, pre-v0.24 upgrade path. Total 30 cases for the installer.
- **`test/skillify-scaffold.test.ts`** (+4 cases) — bare/quoted/single-quoted resolver rows + shared-prefix slug isolation. Total 18 cases for scaffold.
#### Deferred
- LLM tie-break layer for `routing-eval --llm` — placeholder ships in v0.24.0, full implementation is a future release. Code already accepts the flag.
- `gbrain skillpack forget <name>` — explicit uninstall command. v0.24.0 covers the minimum (managed-block prune via `install --all`). Tracked in `TODOS.md`.
- PID-liveness check in installer lock — current behavior (mtime-based stale detection + `--force-unlock` opt-in) is conservative; PID liveness is a v0.24.x ergonomic. Tracked.
### Cross-model review credit
This release's quality is directly attributable to running `/plan-ceo-review` + `/plan-eng-review` + `/codex review` in sequence on the v0.19.0 production-readiness audit. Codex caught one critical and three high findings the in-skill review missed: cumulative-install regression (load-bearing), `--llm` public-contract drift (4-surface scrub), Tier 2 framing as unowned dependency, and 6.5 hours of guesswork named files in the flake-diagnosis plan. The cross-model agreement on every fix is the signal that turns "ship the demo path" into "ship the production path."
## [0.23.2] - 2026-04-30
**The dream cycle now stamps every page it writes. The guard checks for the stamp. No content guessing, no false positives.**
The v0.23.1 prefix-string guard had two flaws caught by a codex review of the v0.23.2 plan. Real serialized brain pages do not always contain their own slug in the body. The synth prompt produces `[Alice](people/alice)` references far more often than the page's own slug, and `serializeMarkdown` does not embed the slug anywhere by default. So the heuristic could miss real dream output. And real conversation transcripts often DO mention brain slugs (`"earlier I wrote about wiki/personal/reflections/identity..."`), so the heuristic dropped legitimate transcripts silently.
v0.23.2 swaps content inference for explicit identity. Every page the synthesize phase writes now gets `dream_generated: true` stamped into its YAML frontmatter at render time. The self-consumption guard checks for that field. CRLF and BOM tolerated. Whitespace and case variants tolerated. Cannot drift, cannot false-positive on user text, cannot miss real output.
`gbrain dream --unsafe-bypass-dream-guard` is a new explicit escape hatch for power users who really do want to re-process a dream-generated page (rare, mostly testing). A loud stderr warning fires every time it runs. The flag is intentionally NOT tied to `--input` because that would let any caller silently re-trigger the loop bug.
The configurable verdict model from v0.23.1 stays. `gbrain config set dream.synthesize.verdict_model claude-sonnet-4-6` still works, with new unit-test coverage asserting the override actually reaches `client.create({ model })`.
### Itemized changes
#### Fixed
- `src/core/cycle/synthesize.ts`: `renderPageToMarkdown` (now exported) stamps `dream_generated: true` and `dream_cycle_date` into every reverse-write. `writeSummaryPage` does the same when building the dream-cycle summary index. The DB-stored frontmatter persists the marker across re-renders.
- `src/core/cycle/transcript-discovery.ts`: replaces v0.23.1's `DREAM_OUTPUT_SLUGS` content-prefix list with `DREAM_OUTPUT_MARKER_RE`, anchored at frontmatter open with optional BOM and CRLF tolerance. Runs in both `discoverTranscripts` and `readSingleTranscript`. Stderr log fires when the guard skips a file (no more silent skips).
- `src/core/cycle/synthesize.ts`: `judgeSignificance` and `JudgeClient` are now exported; `judgeSignificance` accepts a `verdictModel` parameter (default `claude-haiku-4-5-20251001`) loaded from `dream.synthesize.verdict_model` via `loadSynthConfig`.
#### Added
- `gbrain dream --unsafe-bypass-dream-guard` CLI flag. Plumbed through `runCycle.synthBypassDreamGuard``SynthesizePhaseOpts.bypassDreamGuard``discoverTranscripts({bypassGuard})` and `readSingleTranscript({bypassGuard})`. Fires a loud stderr warning at phase entry when set. Never auto-applied for `--input`.
#### Tests
- 12 new test cases in `test/cycle-synthesize.test.ts`:
- `self-consumption guard (v0.23.2 marker-based)`: REGRESSION fixture built from a real `Page → renderPageToMarkdown → isDreamOutput` round-trip; legitimate user note citing a slug is NOT skipped; CRLF + BOM tolerated; whitespace and case variants tolerated; `false`/absent values do NOT match; `dream_generatedfoo` (no word boundary on key) does NOT match; marker buried past 2000 chars does NOT trigger (perf bound); `bypassGuard=true` overrides; `discoverTranscripts` respects the bypass; `DREAM_OUTPUT_MARKER_RE` is anchored at byte 0.
- `judgeSignificance`: passes verdict_model override to `client.create`; defaults to `claude-haiku-4-5-20251001` when omitted; returns `worth_processing=false` on unparseable judge output.
## [0.23.1] - 2026-04-30
**`bun run ci:local` runs the full CI gate on your laptop, 4-way sharded, in ~100 seconds warm. Doc-only diffs go in 5 seconds.**
CI today catches typos, postgres regressions, and the 2-file Tier 1 mechanical suite. The other 34 E2E files in `test/e2e/` only run nightly, and your unit suite never runs against a real Postgres + pgvector locally. This release ships a Docker-based local CI gate that runs every check CI runs (3000+ unit tests + 36 E2E files + gitleaks + typecheck) in **~100s warm wall-time** on a 16-core host. Four pgvector services + a single bun runner; xargs -P4 fans 4 shards each running unit + E2E concurrently; PGLite snapshot fixture skips the schema-replay cold start. `bun run ci:local:diff` adds a doc-only fast-path that exits in seconds when the diff only touches markdown / docs / scripts. Fail-closed by design: an unmapped src/ change runs all 36 E2E files, never silently nothing.
The motivating story: a typical PR cycle is push → wait 8 minutes for GH Actions → fix → push → wait 8 minutes → repeat. Now you push when you're done, not to find out you're not done. The first cold run pulls the bun image, installs deps into a named volume, and runs every check; subsequent runs reuse the warm volumes and complete in 16-20 minutes for the full sequential E2E.
### The numbers that matter
Real laptop run on the M-series host, OrbStack daemon. Reproduce with `bun run ci:local`.
| Metric | Before (push-and-wait) | After (`bun run ci:local`) | Δ |
|---|---|---|---|
| E2E files exercised before push | 0 | 36 | full coverage |
| **Wall-time, full gate, warm (measured, 16-core)** | n/a | **~100 seconds** | **~13x speedup vs push-and-wait** |
| Wall-time, doc-only diff | ~3 min CI | ~5s (host gitleaks only) | ~36× faster |
| Time to first failure signal | ~3 min CI | ~30s host gitleaks + 5s smoke | 6× faster |
| Container env divergence from CI | unknown | bit-for-bit pgvector + bun base | resolved |
| Diff-aware selection on focused PRs | none | 3-9 E2E files for typical scoped change | ~70% fewer files |
| PGLite cold init per file (measured) | ~828ms | ~181ms via snapshot | 4.5× faster |
The lane that matters: when the local gate finds a real bug, you fix it before the PR exists. The release surfaced one such bug as a P1 TODO during verification — `multi-source.test.ts` cascade test isn't isolated; PR CI never runs it.
### What this means for you
Run `bun run ci:local` before `gh pr create` to catch what nightly CI would catch. Run `bun run ci:local:diff` for fast iteration during a focused branch. The selector is hand-tuned today via `scripts/e2e-test-map.ts`; if it ever runs the full suite when you wanted a narrower set, add an entry. Fail-closed default means you can never break correctness by leaving a glob out — only optimize over time.
## To take advantage of v0.23.1
`gbrain upgrade` is a no-op for this release ... no schema migration, no host-repo edits.
To use the new local CI gate:
1. **Install Docker engine** (Docker Desktop, OrbStack, or Colima) and `gitleaks` on host:
```bash
brew install gitleaks
```
2. **Run the full local gate before pushing:**
```bash
bun run ci:local
```
3. **Run the diff-aware subset for fast iteration:**
```bash
bun run ci:local:diff
```
4. **Override the postgres host port** if 5434 collides on your machine:
```bash
GBRAIN_CI_PG_PORT=5435 bun run ci:local
```
The named volumes `gbrain-ci-node-modules`, `gbrain-ci-bun-cache`, and `gbrain-ci-pg-data` keep the install warm. `--clean` nukes them for cold debugging. `--no-pull` skips the upstream pull when offline.
### Itemized changes
#### Added — Tier 1: parallel-shard orchestration
- `bun run ci:local` orchestrates **4 unit+E2E shards in parallel** inside a single bun runner container, each pinned to its own pgvector service. ~3000 unit tests + 36 E2E files complete in ~100s warm.
- `bun run ci:local:diff` runs only the E2E files matched by the diff selector. Falls back to all 36 files when an unmapped src/ path or escape-hatch (schema, package.json, skills/) is touched.
- `bun run ci:select-e2e` prints the selector's choice for the current branch — pipe-friendly.
- `docker-compose.ci.yml` declares 4 `pgvector/pgvector:pg16` services (postgres-1..4) + `oven/bun:1` runner with named volumes for fast restarts. Host ports 5434-5437; override base via `GBRAIN_CI_PG_PORT`.
- `scripts/ci-local.sh` orchestrates the gate with `--diff`, `--no-pull`, `--clean`, `--no-shard` flags. Detects git worktrees (Conductor) and bind-mounts the shared gitdir so in-container `git ls-files` works.
- `scripts/run-unit-shard.sh` is the per-shard unit runner. Takes `SHARD=N/M`, splits `find test -name '*.test.ts' -not -path test/e2e/*` evenly across shards. Excludes `*.slow.test.ts` (Tier 4 convention).
- `scripts/run-e2e.sh` accepts an optional file list from argv, a `--dry-run-list` flag for the inline smoke check, and a `SHARD=N/M` env that filters every M-th file starting at index N. Sequential within a shard preserves the TRUNCATE CASCADE no-race property; parallel across shards is what makes the gate fast.
#### Added — Tier 2: doc-only diff fast-path
- `scripts/select-e2e.ts --classify-only` emits the diff classification (`EMPTY|DOC_ONLY|SRC`) on stdout. `ci-local.sh --diff` reads it before spinning postgres up: if `DOC_ONLY`, the script runs gitleaks on the host and exits in ~5 seconds. Skips the entire ~100s heavy gate when nothing src/-shaped changed.
#### Added — Tier 3: PGLite snapshot fixture
- `scripts/build-pglite-snapshot.ts` boots a fresh PGLite, runs the full `initSchema()` (forward bootstrap + 30 migrations), and dumps the post-init state to `test/fixtures/pglite-snapshot.tar` plus a SHA-256 schema hash sidecar (`pglite-snapshot.version`). Both are gitignored — built on demand by `bun run build:pglite-snapshot` and cached across runs.
- `PGLiteEngine.connect()` now reads `GBRAIN_PGLITE_SNAPSHOT` env: when set, validates the sidecar hash against the in-process MIGRATIONS hash, then loads via PGLite's `loadDataDir` blob. `initSchema()` becomes a no-op when the snapshot was loaded. Measured per-file cold init drops from 828ms → 181ms (4.5×).
- Bootstrap-correctness tests (`test/bootstrap.test.ts`, `test/schema-bootstrap-coverage.test.ts`) explicitly `delete process.env.GBRAIN_PGLITE_SNAPSHOT` so they keep exercising the cold init path they're meant to verify.
#### Added — Tier 4: slow-test convention
- `*.slow.test.ts` is the convention for tests excluded from the fast `ci:local` shards. `bun run test:slow` (via `scripts/run-slow-tests.sh`) runs only the slow set; CI's normal `bun run test` includes them. `scripts/profile-tests.sh` extracts the top-N slowest tests from any captured `bun test` output for picking demotion candidates.
- One genuinely flaky timing test in `test/progress.test.ts` (`startHeartbeat()` heartbeat-count assertion) gained wider tolerance bounds — 4-way parallel shards inflate `setTimeout` jitter beyond the original 2-6 window. Now accepts 1-20 over a 200ms window.
#### Added — Other
- `test/select-e2e.test.ts` covers all 4 selector branches plus 3 codex regression guards (skills/, untracked files, unmapped src/) — 24 cases.
#### For contributors
- `scripts/select-e2e.ts` exports `selectTests(inputs: SelectInputs): string[]`, `classify(changedFiles: string[]): Classification`, and `matchGlob(glob, path): boolean`. The selector is a pure function — pass arrays in, get test files out — so it's trivial to test and easy to fork for another path-glob shape.
- `scripts/e2e-test-map.ts` exports `E2E_TEST_MAP: Record<string, string[]>`. Adding a narrower mapping is safe; the fail-closed default catches anything missed.
## [0.23.0] - 2026-04-26
**`gbrain dream` now actually dreams. Conversation transcripts become reflections, originals, and 25-year patterns ... overnight.**
The maintenance cycle gains two new phases. Synthesize reads transcripts (OpenClaw session corpus, meeting transcripts, ad-hoc files) and writes brain-native pages: reflections to `wiki/personal/reflections/`, originals to `wiki/originals/ideas/`, timeline entries on existing people pages. Patterns runs after `extract` and surfaces recurring themes ... when ≥3 reflections mention the same motif, a pattern page is written to `wiki/personal/patterns/<theme>` citing every reflection that constitutes its evidence. The phase order is now `lint → backlinks → sync → synthesize → extract → patterns → embed → orphans` ... eight phases, one cron-friendly command.
The motivating story: on 2026-04-25 you read your Stanford-era email archive (4,963 emails, 1999-2001) and the agent had to hand-write the reflection page connecting patterns from age 19 to age 45. The 19-year-old who saved his ICQ logs is the user the system should match. The dream cycle's job is to make the brain a self-enriching memory instead of a manually-curated database.
### The numbers that matter
Real production deployment, default config (Sonnet 4.6 synthesis, Haiku 4.5 verdict, 12-hour cooldown). Reproduce with `gbrain dream --phase synthesize --input <fixture>` against any transcript >2000 chars.
| Metric | Before (v0.20.4) | After (v0.23.0) | Δ |
|---|---|---|---|
| Cycle phases | 6 | 8 | +33% |
| Sources of brain enrichment | 4 (manual, signal, ingest, extract) | 5 (+ overnight synth) | +1 lane |
| Cost / day under autopilot | $0 | ~$1-2 | bounded by cooldown |
| Reflections after 30 days | 0 (manual only) | 10-15 (auto) | "the brain dreams" |
The lane that matters: a daily conversation between you and the agent now lands in long-term memory automatically. No manual write-up. Pattern recognition across reflections is one more sonnet call, not a new subsystem.
### What this means for you
Configure `dream.synthesize.session_corpus_dir` once, set `dream.synthesize.enabled true`, and `gbrain dream` (or your existing autopilot install) consolidates yesterday's conversations every overnight pass. Edited transcripts produce new slugs (content-hash suffix) ... never silently overwrite. The synthesize subagent is bounded to an explicit allow-list sourced from `_brain-filing-rules.json`, so even a poisoned transcript can't write to `wiki/finance/secret.md`. `--dry-run` runs the cheap Haiku verdict (cached in `dream_verdicts`) so you can preview without spending real Sonnet tokens.
## To take advantage of v0.23.0
`gbrain upgrade` should do this automatically. If it didn't, or if `gbrain doctor` warns about a partial migration:
1. **Run the orchestrator manually:**
```bash
gbrain apply-migrations --yes
```
2. **Configure the synthesize phase if you want overnight conversation synthesis:**
```bash
gbrain config set dream.synthesize.session_corpus_dir /path/to/transcripts
gbrain config set dream.synthesize.enabled true
gbrain dream --phase synthesize --dry-run --json
```
Existing autopilot users see no behavior change until this step ... synthesize is opt-in.
3. **Verify the outcome:**
```bash
gbrain doctor # schema_version should match latest
gbrain dream --help # shows the 8-phase pipeline
gbrain dream --phase synthesize --dry-run # zero Sonnet calls; cheap Haiku verdict only
```
4. **If any step fails or the numbers look wrong,** please file an issue at https://github.com/garrytan/gbrain/issues with:
- output of `gbrain doctor`
- contents of `~/.gbrain/upgrade-errors.jsonl` if it exists
- which step broke
### Itemized changes
#### Dream cycle: synthesize phase (`src/core/cycle/synthesize.ts`)
- Reads transcripts from `dream.synthesize.session_corpus_dir` (or `--input <file>` ad-hoc).
- Cheap Haiku verdict per transcript filters routine ops sessions; verdicts cached in the new `dream_verdicts` table keyed by `(file_path, content_hash)` so backfill re-runs skip already-judged transcripts at zero cost.
- Fan-out: one Sonnet subagent per worth-processing transcript, dispatched with `allowed_slug_prefixes` (read once from `skills/_brain-filing-rules.json`'s `dream_synthesize_paths.globs`).
- Idempotency key `dream:synth:<file_path>:<content_hash>` ... same content twice is a queue no-op.
- Slug shape: `wiki/personal/reflections/YYYY-MM-DD-<topic>-<hash[:6]>` and `wiki/originals/ideas/YYYY-MM-DD-<idea>-<hash[:6]>`. Edited transcripts produce new slugs alongside the old; `git log` shows both.
- Provenance via `subagent_tool_executions` (the orchestrator queries each child's put_page input, NOT `pages.updated_at` ... that would pick up unrelated writes).
- Orchestrator dual-write: subagent only calls put_page (writes to DB); after children resolve, the phase reverse-renders each new page from DB to disk via `serializeMarkdown`. Subagent never gets fs-write access.
- Cooldown via `dream.synthesize.last_completion_ts` config key, written ONLY on success. Default 12-hour cooldown caps spend at ~$1-2/day under autopilot. Explicit `--input` / `--date` / `--from` / `--to` invocations bypass cooldown.
#### Dream cycle: patterns phase (`src/core/cycle/patterns.ts`)
- Runs AFTER `extract` (codex finding #7) so the graph state is fresh ... subagent put_page sets `ctx.remote=true` and skips auto-link/timeline by default; extract is the canonical materialization step.
- Single Sonnet subagent gathers reflections within `dream.patterns.lookback_days` (default 30) and surfaces themes that recur in ≥`dream.patterns.min_evidence` (default 3) distinct reflections.
- Pattern slug: `wiki/personal/patterns/<theme>` (no date — patterns aggregate across dates). Existing pattern pages are updated in place via the same allow-listed put_page path.
- Same provenance model as synthesize.
#### Trust boundary: `allowed_slug_prefixes`
- New `OperationContext.allowedSlugPrefixes?: string[]` field. When set on a subagent's put_page call, the slug must match one of the listed prefix globs (e.g. `wiki/personal/reflections/*`) or the call is rejected with `permission_denied`.
- When unset, the legacy `wiki/agents/<subagentId>/...` namespace check applies unchanged ... v0.15 anti-prompt-injection guarantee preserved (regression-guarded by `test/operations-allow-list.test.ts`).
- Trust comes from PROTECTED_JOB_NAMES (MCP can't submit `subagent` jobs at all), NOT from `ctx.remote`. The `remote=true` flag flows through every subagent tool call for auto-link safety; using it as the trust signal would null the allow-list for its intended consumer (codex finding #1, caught and corrected pre-merge).
- Auto-link is re-enabled for trusted-workspace writes so the cycle's extract phase doesn't have to recompute synth-output edges.
- Allow-list lives in ONE place: `skills/_brain-filing-rules.json`'s `dream_synthesize_paths.globs`. Both the subagent runtime and the maintain skill read from there.
#### Cycle scaffolding (`src/core/cycle.ts`)
- `ALL_PHASES` extends to 8 entries; `gbrain dream --phase synthesize` and `--phase patterns` work like any other phase.
- New `yieldDuringPhase` hook in `CycleOpts`. Generic in-phase keepalive that long-running phases call every ~5 min while idle to renew the cycle-lock TTL and the Minions worker job lock. Mirrors `yieldBetweenPhases` shape.
- `CycleReport.totals` grew additively (schema_version stays "1"): new fields `transcripts_processed`, `synth_pages_written`, `patterns_written`. Existing consumers see no breaking change.
- `synthesize` and `patterns` both fall under `NEEDS_LOCK_PHASES`; read-only invocations like `--phase orphans` continue to skip the lock.
#### CLI extensions (`src/commands/dream.ts`)
- New flags: `--input <file>` (ad-hoc transcript synthesis; implies `--phase synthesize`), `--date YYYY-MM-DD` (single-day), `--from YYYY-MM-DD --to YYYY-MM-DD` (backfill range).
- `--dry-run` semantics documented explicitly (codex finding #8): runs the cheap Haiku significance verdict (caches it for free) but skips the Sonnet synthesis pass. NOT zero LLM calls.
- Conflict detection: `--input` plus `--date` / `--from` / `--to` exits 2 with a clear error.
- Help text now reflects the 8-phase pipeline.
#### Schema migration v25 (`src/core/migrate.ts`, `src/schema.sql`)
- Creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PRIMARY KEY(file_path, content_hash))`. Distinct from `raw_data` (which is page-scoped) ... transcripts being judged aren't pages.
- RLS-enabled when running as a BYPASSRLS role (matches the existing v24 pattern).
- New engine methods `getDreamVerdict` / `putDreamVerdict` on both Postgres and PGLite. ON CONFLICT upserts; idempotent across re-runs.
#### Tests
- `test/operations-allow-list.test.ts` (NEW, IRON RULE security regression guard) ... 11 cases covering ALLOW path, REJECT path, glob match (recursive depth), legacy namespace check when allow-list unset, FAIL-CLOSED behavior when `viaSubagent=true` but `subagentId` is missing.
- `test/cycle-synthesize.test.ts` (NEW) ... 20 cases covering `compileExcludePatterns` word-boundary heuristic, transcript discovery (date filters, multi-source merge, exclude regex, `min_chars`), content-hash stability across edits, `readSingleTranscript` ad-hoc path.
- `test/cycle-patterns.test.ts` (NEW) ... 12 structural cases covering subagent dispatch wiring, allow-list flow from filing-rules JSON, scope filter (`slug LIKE 'wiki/personal/reflections/%'`), the codex #2 fix (provenance via `subagent_tool_executions`).
- `test/dream-cli-flags.test.ts` (NEW) ... 9 cases covering `--input` / `--date` / `--from` / `--to` parsing, ISO date validation, conflict detection, dry-run semantics documentation.
- `test/e2e/dream-allow-list-pglite.test.ts` (NEW) ... 6 cases on PGLite covering the full subagent → put_page allow-list path: in-allow-list slug writes, out-of-allow-list slug rejected, legacy namespace fallback when allow-list unset, `subagent_tool_executions` schema for provenance queries.
- `test/e2e/dream-synthesize-pglite.test.ts` (NEW) ... 8 cases on PGLite covering disabled/not_configured paths, empty corpus, no-API-key skip path, dry-run semantics, cooldown active/bypass, `dream_verdicts` cache hit.
#### Documentation
- `skills/maintain/SKILL.md` ... new "Dream cycle: synthesize + patterns" section with the quality bar, trust boundary, idempotency model, cooldown semantics, and invocation patterns. Triggers updated to route "process today's session", "synthesize my conversations", and "what patterns did you see" to maintain.
- `skills/_brain-filing-rules.md` ... new "Dream-cycle synthesize/patterns directories" section documenting the allow-listed paths, slug discipline, and the iron law for synthesis output.
- `skills/_brain-filing-rules.json` ... new `dream_synthesize_paths.globs` array (single source of truth).
- `skills/RESOLVER.md` ... new dream-cycle row under brain operations.
- `skills/migrations/v0.21.0.md` (NEW) ... migration narrative covering schema migration v25 + the optional opt-in for synthesize + tunables.
- `CLAUDE.md` ... architecture section reflects 8-phase cycle + new files (`src/core/cycle/{synthesize,patterns,transcript-discovery}.ts`).
#### Codex review-driven corrections
Eight findings from the cross-model review caught real implementation traps before merge. All 8 resolutions integrated:
1. Trust signal correction (drop `remote=null` defense, rely on PROTECTED_JOB_NAMES gating).
2. Provenance via child `subagent_tool_executions` (not `pages.updated_at`).
3. New `dream_verdicts` mini-table (raw_data is page-scoped and won't fit).
4. Summary slug regex-compatible: `dream-cycle-summaries/YYYY-MM-DD` (no underscore, no `.md`).
5. Auto-commit/push deferred to v1.1 (dirty-worktree handling, auth failure, non-FF push need their own design).
6. Lossy-serialization acknowledged: the orchestrator does fresh-render from DB, not byte-identical round-trip.
7. Phase ordering: patterns runs AFTER extract so the graph is fresh.
8. `--dry-run` semantics documented: runs Haiku, skips Sonnet (NOT zero LLM calls).
#### Deferred to v1.1
- Auto git commit + push from the synthesize/patterns phases. v1 writes files locally; either commit yourself or let `gbrain autopilot` handle it.
- Daily token budget cap. Cooldown is the v1 spend bound.
- Cross-modal pattern review (currently reflections-only).
## [0.22.16] - 2026-04-29
**End-to-end claw-test friction harness — every release now gets a fresh-install dry-run.**
**`gbrain claw-test` spins up a hermetic tempdir, walks the canonical first-day flow, and surfaces friction the way a real new user would hit it.**
Before this release, every gbrain release shipped on faith: docs said "the agent runs `gbrain init`, then `gbrain import`, then `gbrain query`," and we'd find out at user-feedback time which step actually broke. Issue #239/#243/#266/#357/#366/#374/#375/#378/#395/#396 — ten upgrade-wedge incidents in two years — all came from this gap. There was no harness that exercised the user's-eye experience: spin up a fresh tempdir, install gbrain, watch what breaks.
Now there is. `gbrain claw-test --scenario fresh-install` in scripted mode is a CI gate (~30s, no API keys). `gbrain claw-test --live --agent openclaw` spawns a real openclaw subprocess, hands it `BRIEF.md`, captures every byte of its stdin/stdout/stderr to `transcript.jsonl`, and lets the agent log friction whenever something is confusing or wrong. End-of-run renders a markdown report grouped by severity and phase, with `<HOME>` redaction so it pastes safely into PRs.
The friction signal comes from a new `gbrain friction {log,render,list,summary}` CLI. Schema is a flat extension of `StructuredAgentError`. Run-id resolves from `--run-id` > `$GBRAIN_FRICTION_RUN_ID` > `standalone.jsonl`, so the same CLI works inside a harness session, manually during normal use, or from a scripted test. Append-only JSONL; readers tolerate malformed lines.
**$GBRAIN_HOME is finally honored everywhere it should be.** `configDir()` in `src/core/config.ts` always supported the parent-dir override, but ~12 consumers built paths from `os.homedir()` directly and bypassed it. Critically, `loadConfig`/`saveConfig` themselves used a private helper that ignored the env. Migrated every write site to a new `gbrainPath()` helper: fail-improve, validator-lint, cycle lock, audit handlers, sync-failures, integrity logs, integrations heartbeat, init pglite path, migrate-engine manifest, import checkpoint, migration rollbacks. Read-side host-detection (`~/.claude` / `~/.openclaw` probes for mod fingerprinting) intentionally stays as-is; v1.1 will add a separate `$GBRAIN_HOST_HOME`.
### Itemized changes
#### Added
- `gbrain claw-test --scenario {fresh-install|upgrade-from-v0.18}` — scripted-mode CI gate that runs the canonical first-day flow against a fresh tempdir. Asserts every expected `--progress-json` phase fired and doctor's `status === 'ok'`. ~30s, no API keys.
- `gbrain claw-test --live --agent openclaw` — friction-discovery mode. Spawns real openclaw, hands it `BRIEF.md`, captures stdin/stdout/stderr to `<run>/transcript.jsonl`, lets the agent log friction. ~510 min and ~$12 in tokens.
- `gbrain claw-test --list-agents` — reports which agent runners are registered + their detection state.
- `gbrain friction log --severity {confused|error|blocker|nit} --phase <name> --message <text> [--hint ...] [--kind {friction|delight}] [--run-id ...]` — append a friction or delight entry.
- `gbrain friction render --run-id <id> [--json] [--transcripts] [--no-redact]` — markdown report grouped by severity + phase; `--redact` defaults on for md output.
- `gbrain friction list [--json]` — recent run-ids with friction/delight counts; interrupted runs marked `(interrupted)`.
- `gbrain friction summary --run-id <id> [--json]` — two-column friction + delight summary.
- `skills/_friction-protocol.md` — cross-cutting convention skill telling agents when to call `gbrain friction log`. Routes from any skill the claw-test exercises.
- `gbrainPath(...segments)` helper in `src/core/config.ts` — single sugar for resolving paths under the active `$GBRAIN_HOME`. `$GBRAIN_HOME` is now validated (must be absolute, no `..` segments).
- Two scenario fixtures in `test/fixtures/claw-test-scenarios/`: `fresh-install` (canonical 5-min flow) and `upgrade-from-v0.18` (scaffolded; real v0.18 SQL dump documented as a v1.1 follow-up).
- New `src/core/claw-test/` module with `agent-runner.ts` (interface + registry), `transcript-capture.ts` (async-drain capture so 256KB+ bursts don't stall the child), `progress-tail.ts`, `scenarios.ts`, and `seed-pglite.ts` (~50 LOC PGLite SQL replay primitive).
#### Changed
- Every `~/.gbrain/...` write site now resolves through `gbrainPath()` instead of building paths from `os.homedir()`. Affected: `src/core/{fail-improve,output/post-write,cycle,sync}.ts`, `src/core/minions/{handlers/shell-audit,backpressure-audit}.ts`, `src/commands/{integrity,integrations,init,migrate-engine,import,migrations/v0_13_1,migrations/v0_14_0}.ts`. Tests that previously used the `process.env.HOME = tmpdir` workaround now use `process.env.GBRAIN_HOME` directly.
- `loadConfig`/`saveConfig` honor `$GBRAIN_HOME`. Previously, the public `configDir()` honored it but the internal `getConfigDir()` did not — so the config file itself silently leaked into the developer's real `~/.gbrain` regardless of the env override.
#### Tests
- 113 new unit tests covering: writer atomicity (concurrent appends), renderer redaction, agent registry resolution + selection precedence, multi-byte UTF-8 chunk-boundary safety, PIPE buffer drain under 256KB+ bursts, scenario load + validation, progress event parsing, SQL splitter (single-quote + line-comment handling), and full claw-test E2E (`test/e2e/claw-test.test.ts` builds a tiny `bun run src/cli.ts` shim and runs --scenario fresh-install end-to-end + a deliberate-break test that proves the friction signal fires).
- `test/gbrain-home-isolation.test.ts` is the regression gate: spawns `gbrain init --pglite` and `gbrain import --no-embed` with `GBRAIN_HOME=<tmp>`, asserts no writes outside `<tmp>/.gbrain` (covers `import.ts:54`, `sync.ts:317`, `upgrade.ts:117`, audit dirs).
## [0.22.15] - 2026-04-29
## **Throw bare markdown into your brain and it becomes properly typed knowledge. No YAML ceremony.**
A real 81K-page brain has 9,655 files with no frontmatter. They imported fine, but every one of them landed in the DB as `type: concept`, `title: <slugified-filename>`, no date, no source, no tags. Search ranking suffered. Type-filtered queries missed them. Entity resolution fell over.
This release adds path-aware frontmatter inference. `gbrain sync` now synthesizes type, date, source, and tags from the filesystem path and first heading the moment a bare-frontmatter file imports. No LLM call, fully deterministic, file on disk untouched. An Apple Note at `Apple Notes/2010-04-13 founders mtg.md` lands as `type: apple-note, title: founders mtg, date: 2010-04-13, source: apple-notes` instead of `type: concept, title: 2010 04 13 Founders Mtg`.
If you want the inference written back to git, the new `gbrain frontmatter generate <path> --fix` walks a brain dir, infers frontmatter for every file that lacks it, and writes back with `.bak` safety backups. Dry-run by default.
### The 9,655 numbers that matter
Measured against my actual brain (gbrain v0.22.8 + the new inference path).
| Behavior | Before v0.22.15 | After v0.22.15 |
|---|---|---|
| Files importing as `type: concept` (no frontmatter) | 9,655 | 0 |
| Apple Notes typed correctly (`apple-note`) | 0 | 5,861 |
| Calendar indexes typed correctly (`calendar-index`) | 0 | 3,201 |
| Therapy sessions typed + dated | 0 | 60 |
| Essay drafts typed + dated | 0 | 33 |
| LLM cost for the full reclassification | n/a | $0 |
The agent doing type-filtered queries on your brain (`type: person`, `type: meeting`, `type: essay`) now actually finds those pages instead of treating everything as `concept`.
### What this means for you
If you've been resisting frontmatter ceremony — same. Throw bare markdown into your brain and inference handles it. The rules table in `src/core/frontmatter-inference.ts` covers the obvious directories (`people/`, `companies/`, `daily/calendar/`, `writing/`, `meetings/`, `personal/`, etc.) plus a generic catch-all. Adding a new convention is one line in `DIRECTORY_RULES`.
## To take advantage of v0.22.15
`gbrain upgrade` should do this automatically. Then:
1. **Run a dry-run preview:**
```bash
gbrain frontmatter generate ~/brain
```
You'll see how many files would get inferred frontmatter and the breakdown by type.
2. **Optionally write back to git:**
```bash
gbrain frontmatter generate ~/brain --fix
```
Each modified file gets a `.bak` backup before rewrite.
3. **Re-sync to pick up the new metadata:**
```bash
gbrain sync ~/brain
```
Inferred frontmatter is folded into `content_hash`, so previously-bare files re-import once with proper types and re-embed. Subsequent syncs are idempotent.
4. **If anything looks off,** please file an issue: https://github.com/garrytan/gbrain/issues with the path of the misclassified file and the rule that matched.
### Itemized changes
#### Features
- `src/core/frontmatter-inference.ts` (new module) — Path-aware frontmatter synthesis. `DIRECTORY_RULES` table maps path prefixes to type/date/title/source/tags. First-match-wins. Date extraction from filenames (`YYYY-MM-DD` prefix or anywhere). Title extraction with date-prefix stripping and first-`#`-heading fallback (20-line window). YAML-safe serialization with quoting for special characters.
- `src/core/import-file.ts``importFromFile()` runs inference inline before `parseMarkdown()` when `opts.inferFrontmatter !== false` (default on). The synthesized frontmatter folds into the in-memory content for parsing, chunking, embedding, and content-hash computation. The file on disk is not modified.
- `src/commands/frontmatter.ts` — New `gbrain frontmatter generate <path> [--fix] [--dry-run] [--json]` subcommand. Walks a directory (skips `.git`, `node_modules`, `.obsidian`, symlinks), runs inference on every `.md` file without frontmatter, optionally writes back with `.bak` backups. Auto-detects brain root by walking up for `.git`. Shows per-type breakdown and first-10 examples.
#### Fixes
- `src/commands/frontmatter.ts:344``runGenerate` dynamic path import now includes `basename`. Single-file invocation (`gbrain frontmatter generate <file>`) previously crashed with `ReferenceError: basename is not defined` on the relative-path-empty fallback at line 437.
#### Tests
- `test/frontmatter-inference.test.ts` (new, 35 cases) — date extraction (5), title extraction from filenames (5) and headings (4 incl. 20-line boundary), inference for every directory rule (13 incl. Apple Notes subfolder tagging), serialization with YAML-safe quoting (4), `applyInference` integration (2), rule ordering and catch-all coverage (2).
## [0.22.14] - 2026-04-29
**Bare `gbrain jobs work` now self-monitors and fail-stops cleanly when its database dies or the queue stalls.**
**The wedged-worker class of bug — process alive, jobs piling up, your `pgrep` check happily green — is gone.**
A production brain (54K pages, Supabase Postgres, 3-concurrency worker under a cron-based PM)
hit it last week: worker process state=Sl at 13:15 UTC, stopped claiming jobs, 21 jobs stacked
in `waiting` over two hours, 5 autopilot-cycles dead-lettered at the 600s timeout, then 150
zombie processes accumulated over the container's 31-day life. The PM's `pgrep` saw a live
PID and reported green the entire time.
Pre-v0.22.14, bare `gbrain jobs work` had **zero** health monitoring. The supervisor (`gbrain
jobs supervisor`) had the right protections — DB liveness probes, stall detection, RSS
watchdog, reconnect on transient PgBouncer blips — but the supervisor wraps `jobs work` as a
child, and many production deployments run bare `jobs work` directly under systemd, Docker,
launchd, cron watchdog, or supervisord. That mode got nothing.
This release moves health monitoring into the bare worker itself, gated by `GBRAIN_SUPERVISED=1`
so it doesn't double up under the supervisor. When the worker detects it's wedged, it emits an
`'unhealthy'` event with a structured reason, and the CLI calls `process.exit(1)` so the external
PM restarts it cleanly. **This is fail-stop:** the worker exits and stays dead until your PM
brings it back. If you run bare `jobs work` without a restart loop, you need one now.
### The numbers that matter
Detection signatures the new health check catches, measured against the production incident
above (and the 30-day deployment running under the band-aid bash watchdog Garry deployed before
this fix):
| Failure mode | Before v0.22.14 | After v0.22.14 |
|---|---|---|
| DB connection death (Supabase/PgBouncer drop) | undetected; worker idles forever | 3 consecutive `SELECT 1` failures (≤3min) → `'unhealthy'`+exit |
| Hung DB probe (network partition) | timer wedged forever, monitoring silently disabled | 10s probe timeout per tick → counted as failure → exit at strike 3 |
| Worker stall (event loop alive, claim returns null) | undetected; jobs pile up in `waiting` | 5min warn, 10min `'unhealthy'`+exit (measured from last completion) |
| Memory leak (RSS climbing past 2GB) | undetected on bare workers | watchdog default 2048 MB triggers `gracefulShutdown('watchdog')` |
| Worker stalled but waiting jobs are unhandled type | ❌ false-positive exit (restart loop) | filter by registered handler names, no exit |
Operationally: from the band-aid bash watchdog Garry deployed before this fix, fresh worker
restart cleared 21 waiting → 0 in 2 minutes, then ran stable for 30+ min with 130 MB RSS,
autopilot-cycles completing in 0.20.6s instead of timing out at 600s.
### What this means for operators
Add a restart policy to your bare-worker invocation BEFORE upgrading. The new behavior is
fail-stop, not self-healing — without a restart loop, your worker will exit on the first DB
blip and stay dead. systemd `Restart=always`, Docker `restart: always`, launchd `KeepAlive`,
cron watchdog, supervisord `autorestart=true`. The migration walks every PM. If you're using
`gbrain jobs supervisor`, you're already protected — the supervisor handles spawn-on-crash
itself.
The default `--max-rss` for bare workers also bumped from 0 (off) to 2048 MB. If you ran bare
workers with intentionally large embed/import jobs, raise the limit (`--max-rss 4096`) or opt
out (`--max-rss 0`). The migration includes per-PM unit-file edits.
## To take advantage of v0.22.14
`gbrain upgrade` should do this automatically. If it didn't, or if `gbrain doctor` warns about
a bare worker exiting with watchdog signatures:
1. **Confirm your bare-worker invocations have a restart policy:**
```bash
# systemd
grep -E '^Restart=' ~/.config/systemd/user/gbrain-worker.service /etc/systemd/system/gbrain-worker.service 2>/dev/null
# crontab
crontab -l | grep "gbrain jobs work"
# launchctl
plutil -p ~/Library/LaunchAgents/com.user.gbrain-worker.plist | grep -A1 KeepAlive
```
2. **Decide on RSS posture:**
- Default 2048 MB matches supervisor behavior. Most bare workers fit.
- Embed/import jobs > 2GB? Pass `--max-rss 4096` (or higher).
- Intentionally unbounded? Pass `--max-rss 0`.
3. **Walk the migration:** `skills/migrations/v0.22.14.md` has the full per-PM table and a
verification block.
4. **Verify:**
```bash
gbrain jobs stats
gbrain doctor --json | jq '.checks[] | select(.name == "queue_health")'
```
Worker startup line should now read:
`Minion worker started (queue: default, concurrency: 3, watchdog: 2048MB, health-check: 60s)`
Under supervisor: the `health-check: Ns` segment is absent (supervisor handles it).
5. **If anything fails or numbers look wrong**, file an issue at
https://github.com/garrytan/gbrain/issues with `gbrain doctor` output and the contents of
`~/.gbrain/upgrade-errors.jsonl` if it exists.
### Itemized changes
#### Added
- `MinionWorkerOpts.{healthCheckInterval, stallWarnAfterMs, stallExitAfterMs, dbFailExitAfter, dbProbeTimeoutMs}` — five new tuning knobs. Defaults: 60s probe interval, 5min warn / 10min exit, 3 DB strikes, 10s per-probe timeout.
- `MinionWorker` now extends `EventEmitter`. Emits `'unhealthy'` with `{ reason: 'db_dead', consecutiveFailures, message } | { reason: 'stalled', waitingCount, idleMinutes }`. CLI subscribes; direct API consumers without a listener inherit a fail-stop fallback that calls `process.exit(1)` to preserve pre-refactor semantics.
- `gbrain jobs work --health-interval MS` — tune the self-health-check cadence (0 disables; rejects NaN/negative/sub-1000ms typos).
- `gbrain jobs supervisor --health-interval MS` — same flag, same validation, same `0 = disable` contract on the supervisor's own probe.
- `GBRAIN_SUPERVISED=1` env var on the supervisor's spawned worker child (skips the child's self-health timer to avoid double-monitoring).
- `gbrain doctor` `queue_health` subcheck reports RSS-watchdog kills in the last 24h via exact match on `error_text = 'aborted: watchdog'` scoped to `status IN ('dead','failed')`.
- `skills/migrations/v0.22.14.md` — full migration walkthrough with per-PM restart-policy preflight, RSS-posture decision tree, and per-system unit-file edits.
#### Changed
- **Default `--max-rss` for `gbrain jobs work`: 0 → 2048 MB.** Matches supervisor default. Catches memory-leak stalls that previously went undetected on bare workers. Opt out with `--max-rss 0`.
- **Bare-worker behavior is now fail-stop** when the DB is unreachable or the queue stalls. Pre-v0.22.14 the worker idled silently. Now it exits and relies on the external PM (systemd, Docker, launchd, cron, supervisord) to restart cleanly.
- Stall query at `worker.ts` filters by registered handler names (`AND name = ANY($2::text[])`) so workers don't false-positive when waiting jobs of unhandled names accumulate.
- Stall exit threshold measured from `lastCompletionTime` (not from when the warning fired), so 5min warn / 10min exit means total idle of 10 min — not 15 min.
- DB liveness probe wrapped in `Promise.race` against a 10s timeout so a hung `executeRaw` cannot wedge the recursive `setTimeout` chain forever.
- `setInterval` → recursive `setTimeout` with a `running` flag throughout. Eliminates timer-callback overlap on slow probes.
- `parseMaxRssFlag` returns `number | undefined` (was `number`) so callers distinguish absent from explicit-disable.
- `process.env.GBRAIN_SUPERVISED` check tightened from `!!env.X` to `=== '1'` (precise contract; no fuzzy matching on `'0'` or `'false'`).
- `MinionWorker` constructor throws when `stallExitAfterMs <= stallWarnAfterMs` so misconfigurations fail loudly at startup.
#### Fixed
- **Wedged-worker false-positive on heterogeneous queues** — workers registering only some handlers no longer interpret waiting jobs of other names as a stall. Repeated `process.exit(1)` → restart loop is gone.
- **Hung DB probe wedge** — pre-fix, a hung `executeRaw('SELECT 1')` kept the recursive `setTimeout` from rescheduling, silently disabling the entire health monitor. Post-fix, the probe times out and counts as a failure.
- **`--health-interval 0` no longer DB-hammers the supervisor.** Pre-fix, the documented "0 disables" contract was a lie — `setInterval(cb, 0)` schedules a tight loop. Now gated behind `> 0`.
- **Inline `jobs submit --follow` and `jobs smoke` no longer kill the user's CLI session** on a DB blip. Both now pass `healthCheckInterval: 0` so the no-listener fallback can't trip on one-shot runs.
- Doctor's RSS-watchdog hint matches the actual error_text signature (`'aborted: watchdog'`) instead of the wrong `'memory limit'` literal that never matched.
#### For contributors
- `MinionWorker extends EventEmitter` — if you import the class directly, the `on('unhealthy', ...)` event is now part of the public surface. The `UnhealthyReason` discriminated union is exported from `src/core/minions/worker.ts`.
- New regression-test infrastructure in `test/minions.test.ts`: `makeProbeEngine(overrides)` is a Proxy-based engine wrapper that intercepts `SELECT 1` and the stall `count(*)` query while passing every other call through to the real PGLite engine. Useful for any future test that needs to inject DB liveness or stall semantics without mocking the entire engine surface.
### Adjacent (separate PR, v0.22.15)
PR #503 catches the *symptom* of one specific failure mode. The cause-side fix — `runPhaseEmbed → embed.ts → embedBatch` not honoring `signal.aborted` between OpenAI batch calls — ships in v0.22.15 (highest-priority TODO; daily wedge driver). Plumbing is documented in `TODOS.md`.
## [0.22.13] - 2026-04-28
**Sync got faster, and the bookmark stopped lying.**
**Parallel imports, a real writer lock, and a head-drift gate that catches the worst race.**
The headline is `gbrain sync --workers N`: per-worker Postgres engines with an atomic queue index, same pattern as `gbrain import --workers N`. On a 7,000-page brain that used to take 25+ minutes, the import phase now runs across 4 workers by default. The reproducible benchmark in `test/e2e/sync-parallel.test.ts` shows `parallel(4)` finishing 1.3× faster than serial on a 120-file fixture against local Postgres (`serial=289ms parallel(4)=221ms`). The speedup grows on larger brains and slower-roundtrip databases (Supabase, remote PgBouncer) because the worker setup cost amortizes over more files. But the bigger story is that the sync writer is finally exclusive across processes, and the `last_commit` bookmark refuses to advance when git HEAD has drifted out from under us. The silent-skip-then-advance pathology has survived every prior sync hardening pass. It is dead now.
### What you can do now
- `gbrain sync --workers 4` (alias `--concurrency 4`) parallelizes the import phase. Each worker holds 2 connections, so total Postgres connections during the parallel phase is `workers * 2` plus your caller's pool. At the default of 4 workers and a 10-connection caller pool, that's up to 18 connections, well under PgBouncer's `max_client_conn` default of 100 but worth knowing on tight Supabase tiers.
- **Auto-concurrency:** if you don't pass `--workers`, sync uses 4 workers when the diff exceeds 100 files. Smaller diffs stay serial. Explicit `--workers` always wins (even on a 30-file diff). PGLite forces serial regardless, since it's a single-connection engine.
- **Full sync** routes through the same path. First syncs on large brains parallelize automatically.
- **Minion `sync` jobs** also use the new `autoConcurrency()` policy. Behavior is now consistent between CLI sync, the Minion handler, and the autopilot cycle's sync phase. (`noEmbed` defaults to `true` in the jobs handler. Submit `gbrain embed --stale` as a separate job when needed, or rely on the autopilot cycle's embed phase.)
- **`--workers` validation is loud now.** `--workers 0`, `--workers -3`, `--workers foo`, `--workers 1.5` all exit with an error message. The prior behavior silently fell through to auto-concurrency (4 workers), the opposite of what you typed.
### Correctness fixes you didn't have to ask for
- **Cross-process writer lock.** Two `gbrain sync` calls (manual + autopilot, two terminals, two Conductor workspaces) used to read the same `last_commit`, both write it, and let the last writer win. The new `gbrain-sync` row in `gbrain_cycle_locks` serializes the writer window. Same-process reentrance from the autopilot cycle handler was already covered by the broader `gbrain-cycle` lock; sync's lock is narrower and runs underneath it.
- **Head-drift gate.** If `git checkout` or `git pull` runs in your worktree mid-sync (Conductor sibling workspace, ad-hoc terminal), the captured `headCommit` no longer matches HEAD when sync finishes. `last_commit` no longer advances in that case. The next sync re-walks the diff against the new HEAD instead of silently moving the bookmark past unimported work.
- **Vanished files now block bookmark advance.** A file the diff said exists at `headCommit` but is gone from disk used to register as a benign skip. It now goes into `failedFiles` and gates `last_commit` the same way a parse failure does.
- **Per-source bookmark for Minion `sync` jobs.** The job handler now resolves `sourceId` from the repo path (mirrors the autopilot cycle's `cycle.ts` fix from PR #475). On multi-source brains, this prevents the 30-min full-reimport-every-cycle behavior caused by reading the global `config.sync.last_commit` anchor when the per-source row would have been correct.
- **Worker connection cleanup.** Worker engines now disconnect inside `try/finally`, even on partial connect failure or mid-import error. The prior `Promise.all(...disconnect)` ran outside any try/finally, so panic-path leaks never released the 8 worker connections.
- **Engine detection unified.** Both PGLite-detection sites in sync.ts now use `engine.kind === 'pglite'` (the discriminator added in v0.13.1). The `engine.constructor.name === 'PGLiteEngine'` sniff is gone, since it broke under bundling and was inconsistent with the other site's `config.engine` string check.
### What this means for you
If you run autopilot on a 7,000-page Postgres brain, your sync cycle gets faster on day one with no flags. If you have ever felt the bookmark "skip past" work that didn't import, you'll stop seeing it. If you have multiple Conductor workspaces poking the same brain, you'll either wait politely on the writer lock or get a clear "another sync is in progress" error. None of this requires a config change.
## To take advantage of v0.22.13
`gbrain upgrade` should do this automatically. If you want to use the new flags right now:
1. **For a one-off speed win on a large brain:**
```bash
gbrain sync --workers 4
```
Or for incremental syncs that touch >100 files, just run `gbrain sync`. Auto-concurrency fires.
2. **For your autopilot cycle:** no action. The Minion `sync` handler picks up the new auto-concurrency policy automatically.
3. **Verify the writer lock is working:**
```bash
gbrain sync &
gbrain sync # second call will say "Another sync is in progress" or wait
```
4. **If sync ever errors with "Another sync is in progress" and stays stuck:** the lock is in `gbrain_cycle_locks` with id `gbrain-sync` and a 30-minute TTL. If a worker crashed without releasing, the next acquirer takes over once the TTL expires. To unstick faster:
```sql
DELETE FROM gbrain_cycle_locks WHERE id = 'gbrain-sync';
```
5. **If anything looks wrong,** file an issue: https://github.com/garrytan/gbrain/issues with output of `gbrain doctor` and the contents of `~/.gbrain/upgrade-errors.jsonl` if it exists.
### Itemized changes
- `src/commands/sync.ts`: `performSync` now wraps body in a `gbrain-sync` DB lock; `--workers` honored regardless of file count when explicit; head-drift gate after import phase; engine.kind detection; try/finally around worker engines; banner moved to stderr.
- `src/commands/import.ts`: `engine.kind === 'pglite'` discriminator; try/finally around worker engines; shared `parseWorkers()` for `--workers` validation.
- `src/commands/jobs.ts`: sync handler resolves `sourceId` via `sources.local_path` lookup; concurrency routed through `autoConcurrency()`; `noEmbed: true` default documented.
- `src/core/sync-concurrency.ts` (new): `autoConcurrency()` + `parseWorkers()` + constants. One source of truth for the concurrency policy that previously lived in three call sites.
- `src/core/db-lock.ts` (new): generic `tryAcquireDbLock(engine, lockId)` over the existing `gbrain_cycle_locks` table. Reused by performSync. cycle.ts continues to use its own ID `gbrain-cycle` so the two locks nest cleanly.
- `test/sync-concurrency.test.ts` (new): 17 cases covering autoConcurrency thresholds, shouldRunParallel gates, parseWorkers validation.
- `test/sync-parallel.test.ts` (new): PGLite-routed coverage of the bookmark gate under concurrency request, the head-drift gate, the writer-lock contract, and PGLite-stays-serial.
- `test/e2e/sync-parallel.test.ts` (new): DATABASE_URL-gated Postgres E2E. 60-file happy path with `pg_stat_activity` leak probe, plus a 120-file serial-vs-parallel benchmark that prints `SYNC_PARALLEL_BENCH ...` for CHANGELOG quoting.
### For contributors
- `BrainEngine.kind` is now the canonical PGLite/Postgres discriminator. Avoid `engine.constructor.name === '...'` (breaks under bundling) and `config.engine === '...'` (inconsistent with the engine actually in use).
- The `gbrain_cycle_locks` table is now multi-purpose. The id column distinguishes lock scopes: `gbrain-cycle` for the cycle, `gbrain-sync` for the sync writer. Future locks should pick distinct ids and reuse `tryAcquireDbLock`.
- `parseWorkers()` is the canonical CLI flag parser for `--workers`. Use it instead of inline `parseInt`.
## [0.22.12] - 2026-04-29
**`sync --skip-failed` now classifies file-size and symlink rejections instead of bucketing them as UNKNOWN.**
@@ -223,7 +860,7 @@ If `gbrain sync` blocks with parse failures, the breakdown tells you what to fix
- DB-layer error patterns (`DB_DUPLICATE_KEY`, `STATEMENT_TIMEOUT`) check BEFORE YAML patterns in the classifier, so Postgres errors don't get YAML-labeled.
- Frontmatter regex patterns rewritten to match canonical messages from `collectValidationErrors()` (`File is empty...`, `No closing --- delimiter found`, `Frontmatter block is empty`) instead of aspirational code-token strings (`missing.*open`) that never appeared in practice.
Closes #500.
Closes #500. Eng-review plan: `~/.claude/plans/then-codex-synchronous-toucan.md` (codex outside-voice agreed on all 7 findings).
## [0.22.8] - 2026-04-28
@@ -373,6 +1010,8 @@ Then point Claude Desktop, claude.ai/code, or any MCP client at `http://your-tun
If anything breaks: `gbrain doctor`, `~/.gbrain/upgrade-errors.jsonl` (if present), and please file an issue at https://github.com/garrytan/gbrain/issues with both.
## [0.22.6.1] - 2026-04-26
**Old brains can upgrade again.**
@@ -818,7 +1457,7 @@ Two SearchOpts additions plumb hard-exclude through the API: `exclude_slug_prefi
### The numbers that matter
A new BrainBench category — **Cat 13b: Source Swamp Resistance** — ships in the sibling [gbrain-evals](https://github.com/garrytan/gbrain-evals) repo. The corpus is 20 pages: 10 short opinionated `originals/` pages and 10 long `wintermute/chat/` dumps that mention the same multi-word phrases at higher per-byte density. 30 hand-curated queries assert the curated page wins.
A new BrainBench category — **Cat 13b: Source Swamp Resistance** — ships in the sibling [gbrain-evals](https://github.com/garrytan/gbrain-evals) repo. The corpus is 20 pages: 10 short opinionated `originals/` pages and 10 long `openclaw/chat/` dumps that mention the same multi-word phrases at higher per-byte density. 30 hand-curated queries assert the curated page wins.
| gbrain version | Top-1 hit | Top-3 hit | Swamp@top |
|--------------------------------------|-----------|-----------|-----------|
@@ -832,17 +1471,17 @@ The world-v1 corpus (BrainBench Cats 1+2 retrieval, 145 relational queries) is u
### What this means for you
If your brain's biggest directories are chat dumps, daily logs, or X archives, search just got dramatically better for the topic queries you actually run. If you depend on chat surfacing for date-framed questions ("what did we discuss last week"), nothing changed ... the intent classifier routes those to `detail=high` which bypasses source-boost. If you want a different boost map, set `GBRAIN_SOURCE_BOOST=originals/:1.8,wintermute/chat/:0.3` and ship.
If your brain's biggest directories are chat dumps, daily logs, or X archives, search just got dramatically better for the topic queries you actually run. If you depend on chat surfacing for date-framed questions ("what did we discuss last week"), nothing changed ... the intent classifier routes those to `detail=high` which bypasses source-boost. If you want a different boost map, set `GBRAIN_SOURCE_BOOST=originals/:1.8,openclaw/chat/:0.3` and ship.
## To take advantage of v0.22.0
`gbrain upgrade` should do this automatically. No DB migration is needed ... the change is purely a SQL ranking refactor on existing tables.
1. **No manual migration step required.** The new ranking is on by default. Defaults are tuned for a brain with the canonical `originals/`, `concepts/`, `writing/`, `meetings/`, `daily/`, `media/x/`, `wintermute/chat/` shape.
1. **No manual migration step required.** The new ranking is on by default. Defaults are tuned for a brain with the canonical `originals/`, `concepts/`, `writing/`, `meetings/`, `daily/`, `media/x/`, `openclaw/chat/` shape.
2. **Tune for your brain (optional):**
```bash
# Stronger originals boost, harder chat dampening
export GBRAIN_SOURCE_BOOST="originals/:1.8,wintermute/chat/:0.3"
export GBRAIN_SOURCE_BOOST="originals/:1.8,openclaw/chat/:0.3"
# Add a directory to the hard-exclude list
export GBRAIN_SEARCH_EXCLUDE="scratch/,private/"
```
@@ -863,7 +1502,7 @@ If your brain's biggest directories are chat dumps, daily logs, or X archives, s
#### Source-aware retrieval
- New module `src/core/search/source-boost.ts` ships the default boost map (`originals/` 1.5, `concepts/` 1.3, `writing/` 1.4, `people/companies/deals/` 1.2, `daily/` 0.8, `media/x/` 0.7, `wintermute/chat/` 0.5) and the four default hard-exclude prefixes (`test/`, `archive/`, `attachments/`, `.raw/`). Both knobs override via env (`GBRAIN_SOURCE_BOOST`, `GBRAIN_SEARCH_EXCLUDE`) or per-call SearchOpts.
- New module `src/core/search/source-boost.ts` ships the default boost map (`originals/` 1.5, `concepts/` 1.3, `writing/` 1.4, `people/companies/deals/` 1.2, `daily/` 0.8, `media/x/` 0.7, `openclaw/chat/` 0.5) and the four default hard-exclude prefixes (`test/`, `archive/`, `attachments/`, `.raw/`). Both knobs override via env (`GBRAIN_SOURCE_BOOST`, `GBRAIN_SEARCH_EXCLUDE`) or per-call SearchOpts.
- New module `src/core/search/sql-ranking.ts` is a pair of pure SQL-fragment builders shared between Postgres and PGLite engines. `buildSourceFactorCase` emits a longest-prefix-match CASE expression and returns literal `'1.0'` when `detail === 'high'` so temporal queries bypass source-boost. `buildHardExcludeClause` emits `NOT (col LIKE 'p1%' OR col LIKE 'p2%')` ... OR-chain wrapped in NOT, never `NOT LIKE ALL/ANY` (those don't express set-exclusion). LIKE meta-character escape covers `%`, `_`, AND `\` (backslash matters because it's Postgres LIKE's default escape char). Single-quote doubling renders SQL-injection-style inputs inert.
- `src/core/postgres-engine.ts` and `src/core/pglite-engine.ts` ... three methods wired: `searchKeyword` (chunk-grain CTE → DISTINCT ON page dedup, multiplies ts_rank by source-factor), `searchKeywordChunks` (the chunk-grain anchor primitive used by Cathedral II two-pass retrieval, also gets source-boost so the anchor pool is dampened on chat dirs), and `searchVector` (becomes a two-stage CTE: pure-distance HNSW inner ORDER BY, source-boost re-rank in outer SELECT, innerLimit scales with offset to preserve pagination).
- `src/core/types.ts` ... SearchOpts gains two fields: `exclude_slug_prefixes?: string[]` (additive over defaults + env) and `include_slug_prefixes?: string[]` (subtractive opt-back-in).
@@ -1044,7 +1683,7 @@ If you build with gbrain + OpenClaw + Claude Code: add your repo as a source (`g
### Itemized changes
**Layer 0 — Wintermute's baseline (cherry-picked, author scrubbed).** Tree-sitter code chunker for 6 languages (TS/TSX/JS/Python/Ruby/Go), `gbrain repos add/list/remove`, strategy-aware sync, `PageType 'code'`, `importCodeFile`, per-file sync progress via the v0.15.2 reporter. Preserved exactly, committed under Garry's author identity.
**Layer 0 — Garry's OpenClaw baseline (cherry-picked, author scrubbed).** Tree-sitter code chunker for 6 languages (TS/TSX/JS/Python/Ruby/Go), `gbrain repos add/list/remove`, strategy-aware sync, `PageType 'code'`, `importCodeFile`, per-file sync progress via the v0.15.2 reporter. Preserved exactly, committed under Garry's author identity.
**Layer 1 — A6 structured errors + version bump.** New `src/core/errors.ts` exports `StructuredAgentError` + `buildError` + `serializeError`. Matches the v0.17.0 `CycleReport.PhaseResult.error` shape so agent-consumable errors stay consistent across every gbrain surface. `globToRegex` bug fix: `src/**/*.ts` now matches `src/foo.ts` (zero intermediate dirs). `GBRAIN_HOME` env var for test isolation. `package.json``0.19.0`.
@@ -1052,7 +1691,7 @@ If you build with gbrain + OpenClaw + Claude Code: add your repo as a source (`g
**Layer 3 — schema migrations v25 + v26.** `pages.page_kind TEXT CHECK (page_kind IN ('markdown','code'))` on v25, using Postgres's `NOT VALID` + `VALIDATE CONSTRAINT` split so tables with millions of pages don't hold a write lock during the ALTER. `content_chunks` adds `language`, `symbol_name`, `symbol_type`, `start_line`, `end_line` on v26, plus partial indexes keyed on non-null values so code-chunk lookups stay cheap on mixed markdown+code brains.
**Layer 4 — delete Wintermute's multi-repo, wire v0.18.0 sources.** The `repos` abstraction in Wintermute's baseline turned out to be redundant with v0.18.0's `sources` subsystem (per-source `last_commit`, `federated` search config, RLS-friendly, DB-native). v0.19.0 keeps `gbrain repos` as a deprecated alias that routes into `runSources`. `sync --all` iterates the `sources` table instead of a local config array. Codex's P0 #2 (per-repo sync bookmarks) and P0 #3 (slug collision) both resolved by the existing schema.
**Layer 4 — delete the OpenClaw baseline's multi-repo, wire v0.18.0 sources.** The `repos` abstraction in Garry's OpenClaw baseline turned out to be redundant with v0.18.0's `sources` subsystem (per-source `last_commit`, `federated` search config, RLS-friendly, DB-native). v0.19.0 keeps `gbrain repos` as a deprecated alias that routes into `runSources`. `sync --all` iterates the `sources` table instead of a local config array. Codex's P0 #2 (per-repo sync bookmarks) and P0 #3 (slug collision) both resolved by the existing schema.
**Layer 5 — Chonkie chunker parity (E2a).** 6 languages → 29. Embedded asset paths for every grammar in `tree-sitter-wasms`. Accurate tokenizer via `@dqbd/tiktoken` `cl100k_base` (lazy-init). Small-sibling merging with the Chonkie `bisect_left` pattern tuned to 15% of chunk target, so tiny siblings (imports, single-line consts) collapse while substantive classes/functions stay independent. `CHUNKER_VERSION=3` folded into `importCodeFile`'s `content_hash` so chunker-shape changes across releases force clean re-chunks without `sync --force`.
@@ -1499,7 +2138,7 @@ No schema migration. Existing brains work unchanged.
- **`gbrain skillpack list`** — prints the curated bundle (25 skills) shipped with gbrain.
- **`gbrain skillpack install <name>` / `--all`** — copies bundled skills into the target workspace. Automatically pulls shared convention files so nothing references a missing dep. Per-file diff protection, `--overwrite-local` escape hatch, `.gbrain-skillpack.lock` against concurrent installers, atomic managed-block update to AGENTS.md / RESOLVER.md.
- **`gbrain skillpack diff <name>`** — per-file diff preview before install.
- **`gbrain routing-eval`** — dedicated CI verb that runs routing fixtures (`skills/<name>/routing-eval.jsonl`) and surfaces intent-to-skill mismatches, ambiguous routing, and false positives. Default structural layer runs alongside `check-resolvable`; `--llm` opts into an LLM tie-break layer.
- **`gbrain routing-eval`** — dedicated CI verb that runs routing fixtures (`skills/<name>/routing-eval.jsonl`) and surfaces intent-to-skill mismatches, ambiguous routing, and false positives. Ships the structural layer (same logic `check-resolvable` runs). The `--llm` flag is accepted as a placeholder for a future LLM tie-break layer; in this release it emits a stderr notice and runs structural only.
- **`gbrain check-resolvable --strict`** — opt-in CI mode that promotes warnings to failures.
- **`skills/_brain-filing-rules.json`** — machine-readable canonical filing rules (JSON sidecar to the prose `_brain-filing-rules.md`).
- **`writes_pages: true` + `writes_to: [...]`** — new skill frontmatter fields consumed by the filing audit. Distinct from `mutating:` so cron schedulers and report writers aren't dragged into filing checks.
+70 -9
View File
@@ -22,7 +22,7 @@ strict behavior when unset.
## Key files
- `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`. `OperationContext.remote` flags untrusted callers.
- `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`, plus `matchesSlugAllowList(slug, prefixes)` (v0.23 glob matcher: `<prefix>/*` matches recursive children; bare `<prefix>` matches exact only). `OperationContext.remote` flags untrusted callers; `OperationContext.allowedSlugPrefixes` (v0.23) is the trusted-workspace allow-list set by the dream cycle. `put_page` enforces: when `viaSubagent` and `allowedSlugPrefixes` is set, slug must match the allow-list; else the legacy `wiki/agents/<id>/...` namespace check applies. Auto-link enabled for trusted-workspace writes (skipped only when `remote=true && !trustedWorkspace`).
- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports.
- `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`)
- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for the specific forward-referenced state the embedded schema blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target table) and adds only what's missing. Closes the upgrade-wedge bug class that bit users 10+ times across 6 schema versions over 2 years (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396). No-op on fresh installs and modern brains.
@@ -57,9 +57,9 @@ strict behavior when unset.
- `src/core/resolver-filenames.ts` (v0.19) — central list of accepted routing filenames (`RESOLVER.md`, `AGENTS.md`). Shared by `findRepoRoot`, `check-resolvable`, and skillpack install so every code path walks the same fallback chain.
- `src/commands/skillify.ts` + `src/core/skillify/{generator,templates}.ts` (v0.19) — `gbrain skillify scaffold <name>` creates all stubs for a new skill in one command: SKILL.md, script, tests, routing-eval.jsonl, resolver entry, filing-rules pointer. `gbrain skillify check <script>` runs the 10-step checklist (LLM evals, routing evals, check-resolvable gate, filing audit) against a candidate skill before it lands.
- `src/commands/skillify-check.ts` (v0.19) — `gbrain skillpack-check` agent-readable health report. Exit 0/1/2 for CI pipeline gating; JSON for debugging. Wraps `check-resolvable --json`, `doctor --json`, and migration ledger into one payload so agents can decide whether a human action is required.
- `src/commands/skillpack.ts` + `src/core/skillpack/{bundle,installer}.ts` (v0.19) — `gbrain skillpack install` drops gbrain's curated 25-skill bundle into a host workspace, managed-block style. Never clobbers local edits; tracks a skill manifest so subsequent `install --update` diffs cleanly. Bundle builder (`skillpack/bundle.ts`) packages the set from `skills/` into a versioned payload.
- `src/commands/skillpack.ts` + `src/core/skillpack/{bundle,installer}.ts` (v0.19) — `gbrain skillpack install` drops gbrain's curated 25-skill bundle into a host workspace, managed-block style. Never clobbers local edits; tracks a skill manifest so subsequent `install --update` diffs cleanly. Bundle builder (`skillpack/bundle.ts`) packages the set from `skills/` into a versioned payload. **v0.24.0:** managed block embeds a `<!-- gbrain:skillpack:manifest cumulative-slugs="..." version="..." -->` receipt inside the fence. Per-skill installs accumulate via `union(prior_receipt, this_call)`; `install --all` is the only path that prunes (drops slugs no longer in the bundle). Rows inside the fence whose slug is in neither the new cumulative set nor the bundle survive as user-added with a stderr `[skillpack] unknown row in managed block: "<slug>" — Investigate: ...` warning. Pre-v0.24 fences upgrade silently on first install (extracted slugs become the prior cumulative set).
- `src/core/skill-manifest.ts` (v0.19) — parser for `skill-manifest.json` records. Used by skillpack installer to detect drift between the shipped bundle and the user's local edits, so updates merge instead of overwriting.
- `src/commands/routing-eval.ts` + `src/core/routing-eval.ts` (v0.19) — `gbrain routing-eval` catches user phrasings that route to the wrong skill. Reads `skills/<name>/routing-eval.jsonl` fixtures (`{intent, expected_skill, ambiguous_with?}`). Structural layer runs in `check-resolvable` by default (zero API cost); `--llm` opts into a Haiku tie-break layer for CI. False positives surface before users hit them.
- `src/commands/routing-eval.ts` + `src/core/routing-eval.ts` (v0.19) — `gbrain routing-eval` catches user phrasings that route to the wrong skill. Reads `skills/<name>/routing-eval.jsonl` fixtures (`{intent, expected_skill, ambiguous_with?}`). Structural layer runs in `check-resolvable` by default (zero API cost). The `--llm` flag is accepted as a placeholder for a future LLM tie-break layer; in v0.24.0 it emits a stderr notice and runs structural only. False positives surface before users hit them.
- `src/core/filing-audit.ts` + `skills/_brain-filing-rules.json` (v0.19) — Check 6 of `check-resolvable`. Parses new `writes_pages:` / `writes_to:` frontmatter on skills and audits their filing claims against the filing-rules JSON. Warning-only in v0.19, upgrades to error in v0.20.
- `src/core/dry-fix.ts``gbrain doctor --fix` engine. `autoFixDryViolations(fixes, {dryRun})` rewrites inlined rules to `> **Convention:** see [path](path).` callouts via three shape-aware expanders (bullet / blockquote / paragraph). Five guards: working-tree-dirty (`getWorkingTreeStatus()` returns 3-state `'clean' | 'dirty' | 'not_a_repo'`), no-git-backup, inside-code-fence, already-delegated (40-line proximity, consistent with detector), ambiguous-multi-match, block-is-callout. `execFileSync` array args (no shell — no injection surface). EOF newline preserved.
- `src/core/backoff.ts` — Adaptive load-aware throttling: CPU/memory checks, exponential backoff, active hours multiplier
@@ -87,12 +87,12 @@ strict behavior when unset.
- `src/core/minions/wait-for-completion.ts` (v0.15) — poll-until-terminal helper for CLI callers. `TimeoutError` does NOT cancel the job; `AbortSignal` exits without throwing. Default `pollMs`: 1000 on Postgres, 250 on PGLite inline.
- `src/core/minions/transcript.ts` (v0.15) — renders `subagent_messages` + `subagent_tool_executions` to markdown. Tool rows splice under their owning assistant `tool_use` by `tool_use_id`. UTF-8-safe truncation; unknown block types fall through to fenced JSON.
- `src/core/minions/plugin-loader.ts` (v0.15) — `GBRAIN_PLUGIN_PATH` discovery. Absolute paths only, left-wins collision, `gbrain.plugin.json` with `plugin_version: "gbrain-plugin-v1"`, plugins ship DEFS only (no new tools), `allowed_tools:` validated at load time against the derived registry.
- `src/core/minions/tools/brain-allowlist.ts` (v0.15) — derives subagent tool registry from `src/core/operations.ts`. 11-name allow-list: `query`, `search`, `get_page`, `list_pages`, `file_list`, `file_url`, `get_backlinks`, `traverse_graph`, `resolve_slugs`, `get_ingest_log`, `put_page`. `put_page` schema is namespace-wrapped per subagent (`^wiki/agents/<subagentId>/.+`); the `put_page` op's server-side check is the authoritative gate via `ctx.viaSubagent` fail-closed.
- `src/core/minions/tools/brain-allowlist.ts` (v0.15, extended v0.23) — derives subagent tool registry from `src/core/operations.ts`. 11-name allow-list. By default `put_page` schema is namespace-wrapped per subagent (`^wiki/agents/<subagentId>/.+`). **v0.23 trusted-workspace path:** when `BuildBrainToolsOpts.allowedSlugPrefixes` is set, the put_page schema instead describes the prefix list to the model and the OperationContext is threaded with `allowedSlugPrefixes`. Trust comes from `PROTECTED_JOB_NAMES` gating subagent submission — MCP cannot reach this field. Only cycle.ts (synthesize/patterns) and direct CLI submitters set it.
- `src/mcp/tool-defs.ts` (v0.15) — extracted `buildToolDefs(ops)` helper. MCP server + subagent tool registry both call it; byte-for-byte equivalence pinned by `test/mcp-tool-defs.test.ts`.
- `src/core/minions/attachments.ts` — Attachment validation (path traversal, null byte, oversize, base64, duplicate detection)
- `src/commands/agent.ts` (v0.16) — `gbrain agent run <prompt> [flags]` CLI. Submits `subagent` (or N children + 1 aggregator) under `{allowProtectedSubmit: true}`. Single-entry `--fanout-manifest` short-circuits. Children get `on_child_fail: 'continue'` + `max_stalled: 3`. `--follow` is the default on TTY; streams logs + polls `waitForCompletion` in parallel. Ctrl-C detaches, does not cancel.
- `src/commands/agent-logs.ts` (v0.16) — `gbrain agent logs <job> [--follow] [--since]`. Merges JSONL heartbeat audit + `subagent_messages` into a chronological timeline. `parseSince` accepts ISO-8601 or relative (`5m`, `1h`, `2d`). Transcript tail renders only for terminal jobs.
- `src/commands/jobs.ts``gbrain jobs` CLI subcommands + `gbrain jobs work` daemon. v0.13.1 surfaces the full `MinionJobInput` retry/backoff/timeout/idempotency surface as first-class CLI flags on `jobs submit`: `--max-stalled`, `--backoff-type fixed|exponential`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key`. `jobs smoke --sigkill-rescue` is the opt-in regression guard for #219. v0.16 wires `registerBuiltinHandlers` to always register `subagent` + `subagent_aggregator` (no env flag — `ANTHROPIC_API_KEY` is the natural cost gate, trust is via `PROTECTED_JOB_NAMES`) and loads `GBRAIN_PLUGIN_PATH` plugins at worker startup with a loud startup-line per plugin. `shell` handler still gated by `GBRAIN_ALLOW_SHELL_JOBS=1` (RCE surface, separate concern). v0.22.10 (#521): the `autopilot-cycle` handler now forwards `job.data.phases` to `runCycle` (was previously discarded — caller-supplied phase selection silently became a full cycle). Phases are validated against `ALL_PHASES` from `src/core/cycle.ts`; invalid names are filtered out and an empty/missing array falls back to the default 6-phase cycle.
- `src/commands/jobs.ts``gbrain jobs` CLI subcommands + `gbrain jobs work` daemon. v0.13.1 surfaces the full `MinionJobInput` retry/backoff/timeout/idempotency surface as first-class CLI flags on `jobs submit`: `--max-stalled`, `--backoff-type fixed|exponential`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key`. `jobs smoke --sigkill-rescue` is the opt-in regression guard for #219. v0.16 wires `registerBuiltinHandlers` to always register `subagent` + `subagent_aggregator` (no env flag — `ANTHROPIC_API_KEY` is the natural cost gate, trust is via `PROTECTED_JOB_NAMES`) and loads `GBRAIN_PLUGIN_PATH` plugins at worker startup with a loud startup-line per plugin. `shell` handler still gated by `GBRAIN_ALLOW_SHELL_JOBS=1` (RCE surface, separate concern). v0.22.10 (#521): the `autopilot-cycle` handler now forwards `job.data.phases` to `runCycle` (was previously discarded — caller-supplied phase selection silently became a full cycle). Phases are validated against `ALL_PHASES` from `src/core/cycle.ts`; invalid names are filtered out and an empty/missing array falls back to the default 6-phase cycle. v0.22.13 (PR #490 CODEX-1+CODEX-4): `sync` handler now resolves `sourceId` at entry by looking up `sources.local_path` (mirrors `cycle.ts:480`'s autopilot fix from PR #475) so multi-source brains read the per-source `last_commit` anchor instead of the global config key. Concurrency routed through the shared `autoConcurrency()` policy in `src/core/sync-concurrency.ts` instead of the prior hardcoded `4`; PGLite stays serial. `noEmbed` default is `true` (embed is a separate job — submit `gbrain embed --stale` after sync, or rely on the autopilot cycle's embed phase).
- `src/commands/features.ts``gbrain features --json --auto-fix`: usage scan + feature adoption salesman
- `src/commands/autopilot.ts``gbrain autopilot --install`: self-maintaining brain daemon (sync+extract+embed)
- `src/mcp/server.ts` — MCP stdio server (generated from operations). v0.22.7: tool-call handler delegates to `dispatchToolCall` from `src/mcp/dispatch.ts` so stdio + HTTP transports share one validation, context-build, and error-format path.
@@ -106,15 +106,27 @@ strict behavior when unset.
- `src/commands/orphans.ts``gbrain orphans [--json] [--count] [--include-pseudo]`: surfaces pages with zero inbound wikilinks, grouped by domain. Auto-generated/raw/pseudo pages filtered by default. Also exposed as `find_orphans` MCP operation. Shipped in v0.12.3 (contributed by @knee5).
- `src/commands/integrity.ts``gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). v0.22.8: batch-load fast path on Postgres uses `SELECT DISTINCT ON (slug)` in a single SQL query to fix the PgBouncer round-trip timeout (60s → ~6s) while preserving `engine.getAllSlugs()`'s `Set<string>` semantics on multi-source brains. Gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1` so real Postgres errors are diagnosable.
- `src/commands/doctor.ts``gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel <id>`. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count.
- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2).
- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). **v30 (v0.23):** creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))`. RLS-enabled when running as a BYPASSRLS role. The synthesize phase reads/writes this table to avoid re-judging on backfill re-runs.
- `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2.
- `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=<ms>` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers.
- `src/core/cycle.ts` v0.17 brain maintenance cycle primitive. `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise<CycleReport>` composes 6 phases in semantically-driven order (lint → backlinks → sync → extract → embed → orphans). Three callers: `gbrain dream` CLI, `gbrain autopilot` daemon's inline path, and the Minions `autopilot-cycle` handler (`src/commands/jobs.ts`). One source of truth for what the brain does overnight. Coordination via `gbrain_cycle_locks` DB table (TTL-based; works through PgBouncer transaction pooling, unlike session-scoped `pg_try_advisory_lock`) + `~/.gbrain/cycle.lock` file lock with PID-liveness for PGLite / engine=null mode. `CycleReport.schema_version: "1"` is the stable agent-consumable shape. `PhaseResult.error: { class, code, message, hint?, docs_url? }` is Stripe-API-tier structured failure info. `yieldBetweenPhases` hook awaited between every phase — Minions handler uses this to renew its job lock and prevent v0.14 stall-death regression. Engine nullable: filesystem phases (lint, backlinks) run without DB; DB phases skip with `status: "skipped", reason: "no_database"`. Lock-skip: read-only phase selections (`--phase orphans`) bypass the cycle lock. v0.22.1 (#403): `CycleOpts.signal?: AbortSignal` propagates the worker's abort signal; `checkAborted()` fires between every phase and throws if the signal is aborted (cooperative — can't interrupt a phase mid-execution). v0.22.1 (#417): `runPhaseSync` returns `pagesAffected` via `SyncPhaseResult`; `runCycle` captures it and threads to `runPhaseExtract` as the 4th arg, enabling incremental extract on the cycle path. v0.22.1 (Codex F2): `runPhaseSync` takes `willRunExtractPhase: boolean` and sets `noExtract: phases.includes('extract')` so `gbrain dream --phase sync` doesn't silently lose extraction. v0.22.5 (#475): new `resolveSourceForDir(engine, brainDir)` helper queries `SELECT id FROM sources WHERE local_path = $1 LIMIT 1`; `runPhaseSync` threads result as `sourceId` to `performSync()` so sync reads the per-source `sources.last_commit` anchor instead of the drift-prone global `config.sync.last_commit` key. Bare try/catch lets pre-v0.18 brains fall through to the global key. Closes the prod hang where every autopilot cycle ran a 30-min full reimport because the global anchor commit had been GC'd from git history.
- `src/commands/dream.ts` v0.17 `gbrain dream` CLI. ~80-line thin alias over `runCycle`. brainDir resolution requires explicit `--dir` OR `sync.repo_path` config (no more walk-up-cwd-for-.git footgun). Flags: `--dry-run`, `--json`, `--phase <name>`, `--pull`, `--dir <path>`. Exit code 1 on status=failed (partial/warn not fatal — don't page on warnings).
- `src/core/db-lock.ts` (v0.22.13) — generic `tryAcquireDbLock(engine, lockId, ttlMinutes)` over the existing `gbrain_cycle_locks` table. Parameterized lock id so different scopes can nest cleanly: `gbrain-cycle` for the broad cycle (held by `cycle.ts`) and `gbrain-sync` (`SYNC_LOCK_ID` constant) for `performSync`'s narrower writer window. Same UPSERT-with-TTL semantics as the prior cycle-only helper, just generalized. Survives PgBouncer transaction pooling (unlike session-scoped `pg_try_advisory_lock`); crashed holders auto-release once their TTL expires.
- `src/core/sync-concurrency.ts` (v0.22.13) — single source of truth for the parallel-sync policy. Exports `autoConcurrency(engine, fileCount, override?)` (PGLite always serial; explicit override clamped to >=1; auto path returns `DEFAULT_PARALLEL_WORKERS=4` when `fileCount > AUTO_CONCURRENCY_FILE_THRESHOLD=100`), `shouldRunParallel(workers, fileCount, explicit)` (Q1: explicit `--workers` bypasses the >50-file floor), and `parseWorkers(s)` (rejects `'0'`, `'-3'`, `'foo'`, `'1.5'`, trailing chars — replaces the prior parseInt-with-no-validation in both `sync.ts` and `import.ts`). Used by `performSync`, `performFullSync`, `runImport`, and the Minion `sync` handler so the three sites can no longer drift.
- `src/commands/sync.ts``gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). v0.22.13 (PR #490): `performSync` wraps its body in a `gbrain-sync` writer lock so two concurrent syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot both write `last_commit` and let the last writer win. Head-drift gate after the import phase re-checks `git rev-parse HEAD`; if HEAD moved (someone ran `git checkout` / `git pull` mid-sync), the bookmark refuses to advance. Vanished files now record a failedFiles entry instead of silent-skip — the silent-skip-then-advance pathology that survived prior hardening passes is dead. Worker engines wrap in try/finally so disconnect always fires (panic-path leak fix). Both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`), validated via `parseWorkers`. Explicit `--workers` bypasses the auto-path file-count floor; auto path defers to `autoConcurrency()`. Banner moved to stderr.
- `src/core/cycle.ts` — v0.17 brain maintenance cycle primitive (extended to **8 phases in v0.23**). `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise<CycleReport>` composes phases in semantically-driven order: **lint → backlinks → sync → synthesize → extract → patterns → embed → orphans**. v0.23's `synthesize` phase runs after sync (cross-references see fresh brain) and before extract (auto-link materializes its writes); `patterns` runs after extract so it reads a fresh graph (codex finding #7 — subagent put_page sets `ctx.remote=true` and skips auto-link/timeline by default; extract is the canonical materialization). Three callers: `gbrain dream` CLI, `gbrain autopilot` daemon's inline path, and the Minions `autopilot-cycle` handler. Coordination via `gbrain_cycle_locks` DB table + `~/.gbrain/cycle.lock` file lock with PID-liveness for PGLite. `CycleReport.schema_version: "1"` is stable; totals additively grew in v0.23 (`transcripts_processed`, `synth_pages_written`, `patterns_written`). `yieldBetweenPhases` runs between phases. **v0.23 added `yieldDuringPhase`** for in-phase keepalive — synthesize/patterns call it during long waits to renew the cycle-lock TTL. Engine nullable; lock-skip on read-only phase selections. v0.22.1 (#403): `CycleOpts.signal?: AbortSignal` propagates the worker's abort signal; `checkAborted()` fires between every phase. v0.22.1 (#417): `runPhaseSync` returns `pagesAffected` via `SyncPhaseResult`; `runCycle` captures it and threads to `runPhaseExtract` as the 4th arg. v0.22.1 (Codex F2): `runPhaseSync` takes `willRunExtractPhase: boolean` and sets `noExtract: phases.includes('extract')` so `gbrain dream --phase sync` doesn't silently lose extraction. v0.22.5 (#475): `resolveSourceForDir(engine, brainDir)` threads `sourceId` to `performSync()` so sync reads the per-source `sources.last_commit` anchor instead of the drift-prone global `config.sync.last_commit` key.
- `src/core/cycle/synthesize.ts` (v0.23) — Synthesize phase: conversation-transcript-to-brain pipeline. Reads from `dream.synthesize.session_corpus_dir`, runs cheap Haiku verdict (cached in `dream_verdicts`), then fans out one Sonnet subagent per worth-processing transcript with `allowed_slug_prefixes` (sourced from `skills/_brain-filing-rules.json` `dream_synthesize_paths.globs`). Orchestrator collects slugs from `subagent_tool_executions` (NOT `pages.updated_at` — codex finding #2) and reverse-renders DB → markdown via `serializeMarkdown`. Cooldown via `dream.synthesize.last_completion_ts`, written ONLY on success. Idempotency key `dream:synth:<file_path>:<content_hash>`. Auto-commit deferred to v1.1 (codex #5). `--dry-run` runs Haiku, skips Sonnet (codex #8). Subagent never gets fs-write access. **v0.23.2:** `renderPageToMarkdown` (now exported) stamps `dream_generated: true` and `dream_cycle_date` into every reverse-write's frontmatter; `writeSummaryPage` does the same on the dream-cycle summary index. The marker is the explicit identity surface checked by `isDreamOutput` in `transcript-discovery.ts` — replaces the v0.23.1 content-prefix heuristic that could miss real output (`serializeMarkdown` doesn't embed slugs in body) and false-positive on user transcripts citing brain pages. `judgeSignificance` and `JudgeClient` are exported; `judgeSignificance` accepts a `verdictModel` parameter (default `claude-haiku-4-5-20251001`) loaded from `dream.synthesize.verdict_model` via `loadSynthConfig`.
- `src/core/cycle/patterns.ts` (v0.23) — Patterns phase: cross-session theme detection over reflections within `dream.patterns.lookback_days` (default 30). Names a pattern only when ≥`dream.patterns.min_evidence` (default 3) reflections support it. Single Sonnet subagent; same allow-list path as synthesize. Runs AFTER `extract` so the graph is fresh.
- `src/core/cycle/transcript-discovery.ts` (v0.23) — Pure filesystem walk for synthesize. `discoverTranscripts(opts)` filters `.txt` files by date range, min_chars, and word-boundary regex `excludePatterns` (Q-3: `medical` matches "medical advice" but NOT "comedical"; power users may pass full regex). `readSingleTranscript(path)` is the `gbrain dream --input <file>` ad-hoc path. **v0.23.2 self-consumption guard:** `DREAM_OUTPUT_MARKER_RE` (anchored at frontmatter open `---\n`, optional BOM + CRLF tolerance, scans first 2000 chars for `dream_generated: true` with case-insensitive value and word boundary on `true`) drives `isDreamOutput(content, bypass=false)`. Both `discoverTranscripts` and `readSingleTranscript` skip matching files and emit a `[dream] skipped <basename>: dream_generated marker` stderr log (no more silent skips). `bypassGuard?: boolean` on `DiscoverOpts` and `readSingleTranscript`'s opts disables the guard for the explicit `--unsafe-bypass-dream-guard` escape hatch only — never auto-applied for `--input`. Replaces v0.23.1's `DREAM_OUTPUT_SLUGS` content-prefix list.
- `src/commands/dream.ts` — v0.17 `gbrain dream` CLI; ~80-line thin alias over `runCycle`. brainDir resolution requires explicit `--dir` OR `sync.repo_path` config. Flags: `--dry-run`, `--json`, `--phase <name>`, `--pull`, `--dir <path>`. **v0.23 added** `--input <file>` (ad-hoc transcript, implies `--phase synthesize`), `--date YYYY-MM-DD`, `--from <d> --to <d>` (backfill range). Conflict detection: `--input` + `--date` exits 2. ISO date validation. `--dry-run` runs Haiku significance verdict but skips Sonnet synthesis (codex finding #8 — NOT zero LLM calls). Exit code 1 on status=failed. **v0.23.2 added** `--unsafe-bypass-dream-guard` (long-form intentional, plumbed through `runCycle.synthBypassDreamGuard``SynthesizePhaseOpts.bypassDreamGuard``discoverTranscripts({bypassGuard})` and `readSingleTranscript({bypassGuard})`). Loud stderr warning fires at synthesize-phase entry when set. Never auto-applied for `--input` so any caller can't silently re-trigger the loop bug.
- `src/commands/friction.ts` + `src/core/friction.ts` (v0.23) — `gbrain friction {log,render,list,summary}` reporter. Append-only JSONL under `$GBRAIN_HOME/friction/<run-id>.jsonl`. Schema is a flat extension of `StructuredAgentError` (D20). Render groups by severity → phase, defaults to `--redact` for md output (strips `$HOME`/`$CWD` to placeholders so reports paste safely in PRs). Run-id resolves from `--run-id` > `$GBRAIN_FRICTION_RUN_ID` > `standalone.jsonl`. Skills the claw-test exercises gain a `_friction-protocol.md` callout so agents know when to log friction.
- `src/commands/claw-test.ts` + `src/core/claw-test/` (v0.23) — `gbrain claw-test [--scenario <name>] [--live --agent openclaw]`. End-to-end "fresh user" friction harness. Two modes: scripted (CI gate, agent-free) and live (real openclaw subprocess, $12 in tokens). Sets `GBRAIN_HOME=<tempdir>` for hermeticity and captures gbrain's `--progress-json` events from each child's stderr to verify expected phases ran (`import.files`, `extract.links_fs`, `doctor.db_checks`). Phases for scripted mode: setup → install_brain (`gbrain init --pglite`) → import (`--no-embed`) → query → extract → verify (`gbrain doctor --json`, asserts `status: 'ok'`) → render. Live mode hands `BRIEF.md` from `test/fixtures/claw-test-scenarios/<name>/` to the agent runner. v1 ships with the OpenClaw runner only (`src/core/claw-test/runners/openclaw.ts`, invokes `openclaw agent --local --agent <name> --message <brief>`); hermes runner deferred to v1.1. Transcript capture (`transcript-capture.ts`) uses `fs.createWriteStream` with `'drain'`-event backpressure — D17 fix for the 256KB-burst child-stall scenario. v0.18 upgrade scenario seeded via `seed-pglite.ts` SQL replay.
- `skills/_friction-protocol.md` (v0.23) — shared cross-cutting convention skill (like `_brain-filing-rules.md`). Tells agents when to call `gbrain friction log` and how to choose a severity. Routes to friction CLI from any skill the claw-test exercises.
- `scripts/check-progress-to-stdout.sh` — CI guard against regressing to `\r`-on-stdout progress. Wired into `bun run test` via `scripts/check-progress-to-stdout.sh && bun test` in package.json.
- `docs/progress-events.md` — Canonical JSON event schema reference. Stable from v0.15.2, additive only.
- `src/core/markdown.ts` — Frontmatter parsing + body splitter. `splitBody` requires an explicit timeline sentinel (`<!-- timeline -->`, `--- timeline ---`, or `---` immediately before `## Timeline`/`## History`). Plain `---` in body text is a markdown horizontal rule, not a separator. `inferType` auto-types `/wiki/analysis/` → analysis, `/wiki/guides/` → guide, `/wiki/hardware/` → hardware, `/wiki/architecture/` → architecture, `/writing/` → writing (plus the existing people/companies/deals/etc heuristics).
- `scripts/check-jsonb-pattern.sh` — CI grep guard. Fails the build if anyone reintroduces (a) the `${JSON.stringify(x)}::jsonb` interpolation pattern (postgres.js v3 double-encodes it), or (b) `max_stalled INTEGER NOT NULL DEFAULT 1` in any schema source file (v0.15.1 #219 regression guard — must be DEFAULT 5 to preserve SIGKILL-rescue). Wired into `bun test`.
- `docker-compose.ci.yml` + `scripts/ci-local.sh` (v0.23.1) — Local CI gate. `bun run ci:local` spins up `pgvector/pgvector:pg16` + `oven/bun:1` with named volumes (`gbrain-ci-pg-data`, `gbrain-ci-node-modules`, `gbrain-ci-bun-cache`), runs gitleaks on host, smoke-tests `scripts/run-e2e.sh` argv handling, runs unit tests with `DATABASE_URL` unset (matches GH Actions structure), then runs all 29 E2E files sequentially. `--diff` swaps in the diff-aware selector; `--no-pull` skips upstream pulls; `--clean` nukes named volumes. Postgres host port defaults to 5434 (avoids 5432 manual `gbrain-test-pg` and 5433 sibling-project conflict); override with `GBRAIN_CI_PG_PORT=NNNN`. Stronger gate than current PR CI's 2-file Tier 1 set — closes the "push-and-wait" feedback loop pre-push.
- `scripts/select-e2e.ts` + `scripts/e2e-test-map.ts` (v0.23.1) — Diff-aware E2E test selector. Reads three git sources (committed `origin/master...HEAD`, working-tree `HEAD`, and `git ls-files --others --exclude-standard` for untracked, NOT-gitignored files), classifies as EMPTY / DOC_ONLY / SRC. Fail-closed by design: EMPTY → all 29 files (clean branch shouldn't run nothing), DOC_ONLY (every path matches the README/CLAUDE/AGENTS/CHANGELOG/TODOS allowlist) → empty stdout, SRC → escape-hatch paths (schema, package.json, skills/) trigger all; otherwise the hand-tuned `E2E_TEST_MAP` glob → tests narrows; an unmapped src/ change still emits ALL files, never silently nothing. Pure-function exports (`selectTests`, `classify`, `matchGlob`) so it's trivial to test and fork. `bun run ci:select-e2e` prints the current selection on stdout, pipe-friendly. `test/select-e2e.test.ts` covers all 4 branches plus 3 codex regression guards (skills/, untracked files, unmapped src/) — 24 cases.
- `scripts/run-e2e.sh` (v0.23.1 update) — Sequential E2E runner. Now accepts an optional argv-driven file list (used by `ci:local:diff` to pipe in selector output) and a `--dry-run-list` flag that prints the resolved file list and exits (used by `ci-local.sh`'s startup smoke-test). Falls back to `test/e2e/*.test.ts` when invoked with no args.
- `scripts/llms-config.ts` + `scripts/build-llms.ts` — Generator for `llms.txt` (llmstxt.org-spec web index) + `llms-full.txt` (inlined single-fetch bundle). Curated config drives both. Run `bun run build:llms` after adding a new doc. `LLMS_REPO_BASE` env var lets forks regenerate with their own URL base. `FULL_SIZE_BUDGET` (600KB) caps the inline bundle; generator WARNs if exceeded. Committed output is not analogous to `schema-embedded.ts` (no runtime consumer); we commit for GitHub browsing and fork-safe fetching.
- `AGENTS.md` — Local-clone entry point for non-Claude agents (Codex, Cursor, OpenClaw, Aider). Mirrors `CLAUDE.md` intent via relative links. Claude Code keeps using `CLAUDE.md`.
- `docs/UPGRADING_DOWNSTREAM_AGENTS.md` — Patches for downstream agent skill forks to apply when upgrading. Each release appends a new section. v0.10.3 includes diffs for brain-ops, meeting-ingestion, signal-detector, enrich.
@@ -220,6 +232,20 @@ Key commands added in v0.14.3 (fix wave):
- `gbrain jobs submit` gains `--max-stalled`, `--backoff-type`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key` — exposing existing `MinionJobInput` fields as first-class CLI flags.
- `gbrain jobs smoke --sigkill-rescue` — opt-in regression smoke case simulating a killed worker; asserts the v0.14.3 schema default (`max_stalled=5`) actually rescues on first stall.
Key commands added in v0.22.13 (PR #490):
- `gbrain sync --workers N` (alias `--concurrency N`) — parallelize the import phase using per-worker Postgres engines (small pool of 2 each) with an atomic queue index. Auto-concurrency: defaults to 4 workers when the diff exceeds 100 files. Smaller diffs stay serial. Explicit `--workers` always wins (even on a 30-file diff). PGLite forces serial regardless. Validation rejects `0`, negatives, non-integers loud (replaces the prior silent fall-through to auto-concurrency).
- `gbrain import --workers N` — same `parseWorkers()` validation as sync; same try/finally worker-engine cleanup. Behavior surface unchanged.
Key commands added in v0.22.16 (claw-test friction loop):
- `gbrain claw-test [--scenario fresh-install|upgrade-from-v0.18] [--keep-tempdir]` — scripted-mode CI gate that runs the full canonical first-day flow against a fresh tempdir. Asserts every expected `--progress-json` phase fired and doctor's `status === 'ok'`. ~30s, no API keys.
- `gbrain claw-test --live --agent openclaw` — friction-discovery mode. Spawns real openclaw, hands it `BRIEF.md`, captures stdin/stdout/stderr to `<run>/transcript.jsonl`, lets the agent log friction via the friction CLI. Run on demand; ~510 min and ~$12 in tokens.
- `gbrain claw-test --list-agents` — reports which agent runners are registered + their detection state (binary path or unavailable reason).
- `gbrain friction log --severity {confused|error|blocker|nit} --phase <name> --message <text> [--hint ...] [--kind {friction|delight}] [--run-id ...]` — append a friction or delight entry to the active run JSONL.
- `gbrain friction render --run-id <id> [--json] [--transcripts] [--no-redact]` — markdown report grouped by severity + phase; `--redact` is the default for md output (strips `$HOME`/`$CWD` placeholders so reports paste safely in PRs/issues).
- `gbrain friction list [--json]` — recent run-ids with friction/delight counts; interrupted runs marked `(interrupted)`.
- `gbrain friction summary --run-id <id> [--json]` — two-column friction + delight summary.
- `GBRAIN_HOME` env override is now honored uniformly across every gbrain write site (config, audit, friction, sync-failures, import checkpoint, integrity log, integrations heartbeat, migration rollback, etc.) — `gbrainPath(...)` from `src/core/config.ts` is the canonical helper. Read-side host-fingerprint detection (`~/.claude`/`~/.openclaw` etc.) intentionally NOT confined in v1; that's a v1.1 follow-up.
## Testing
`bun test` runs all tests. After the v0.12.1 release: ~75 unit test files + 8 E2E test files (1412 unit pass, 119 E2E when `DATABASE_URL` is set — skip gracefully otherwise). Unit tests run
@@ -275,6 +301,8 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac
`test/orphans.test.ts` (v0.12.3 orphans command: detection, pseudo filtering, text/json/count outputs, MCP op),
`test/postgres-engine.test.ts` (v0.12.3 statement_timeout scoping: `sql.begin` + `SET LOCAL` shape, source-level grep guardrail against reintroduced bare `SET statement_timeout`),
`test/sync.test.ts` (sync logic + v0.12.3 regression guard asserting top-level `engine.transaction` is not called),
`test/sync-concurrency.test.ts` (v0.22.13 PR #490: 17 cases covering `autoConcurrency()` thresholds + PGLite-forces-serial + explicit-override clamping, `shouldRunParallel()` Q1 explicit-bypasses-floor contract, and `parseWorkers()` validation that rejects `'0'`/`'-3'`/`'foo'`/`'1.5'`/trailing chars),
`test/sync-parallel.test.ts` (v0.22.13 PR #490: PGLite-routed coverage of the bookmark gate under concurrency request, head-drift gate, vanished-file failure capture, PGLite-stays-serial, and the `gbrain-sync` writer-lock contract — 7 cases),
`test/sync-failures.test.ts` (v0.22.12: 28 cases pinning `classifyErrorCode` regex coverage for all 12 codes against literal production message strings from `markdown.ts:159-244` and `import-file.ts:199, 347, 352, 401`; `summarizeFailuresByCode` sort + pre-classified-honor; `recordSyncFailures` code-field persistence; `acknowledgeSyncFailures` AcknowledgeResult shape + backfill on pre-v0.22.12 entries),
`test/doctor.test.ts` (doctor command + v0.12.3 assertions that `jsonb_integrity` scans the four v0.12.0 write sites and `markdown_body_completeness` is present),
`test/utils.test.ts` (shared SQL utilities + `tryParseEmbedding` null-return and single-warn semantics),
@@ -305,6 +333,7 @@ E2E tests (`test/e2e/`): Run against real Postgres+pgvector. Require `DATABASE_U
- `test/e2e/engine-parity.test.ts` (v0.22.0) — Postgres ↔ PGLite top-result and result-set parity for `searchKeyword` + `searchVector`. Codex flagged that Postgres ranks pages then picks best chunk while PGLite returns chunks directly — without parity coverage the source-boost fix could pass on PGLite and fail on Postgres. Skips gracefully when `DATABASE_URL` is unset.
- `test/e2e/postgres-bootstrap.test.ts` (v0.22.6.1) — exercises `PostgresEngine.initSchema()` directly against a fresh real Postgres database. Asserts the bootstrap path is no-op on fresh installs and that SCHEMA_SQL replays cleanly through the engine path (not via the standalone `db.initSchema` from `src/core/db.ts`, which would have produced false-positive coverage). Codex caught the E2E-shape gap during plan review.
- `test/e2e/http-transport.test.ts` (v0.22.7) — 8 cases against real Postgres covering `gbrain serve --http` end-to-end: bearer auth round-trip, `last_used_at` SQL-level debounce semantics, `mcp_request_log` row insertion on success and auth_failed paths, `/health` DB-down → 503 (DB-probing health check), and the F1+F2+F3 dispatch round-trip with a real operation. Skips gracefully when `DATABASE_URL` is unset.
- `test/e2e/sync-parallel.test.ts` (v0.22.13 PR #490) — DATABASE_URL-gated. T2: 60-file Postgres sync at concurrency=4 imports all + no connection leak (probes `pg_stat_activity` before/after to confirm worker engines disconnected). P4: 120-file serial-vs-parallel benchmark prints `SYNC_PARALLEL_BENCH N files | serial=Xms | parallel(4)=Yms | speedup=Zx` for CHANGELOG quoting. Asserts parallel ≤ serial × 1.5 (CI-noise tolerant; not a strict speedup gate).
- Tier 2 (`skills.test.ts`) requires OpenClaw + API keys, runs nightly in CI
- If `.env.testing` doesn't exist in this directory, check sibling worktrees for one:
`find ../ -maxdepth 2 -name .env.testing -print -quit` and copy it here if found.
@@ -474,13 +503,45 @@ will detect drift and re-bump on the next run.
## Pre-ship requirements
Before shipping (/ship) or reviewing (/review), always run the full test suite:
Before shipping (/ship) or reviewing (/review), always run the full test suite.
Two equivalent paths:
**Path A — local CI gate (recommended, v0.23.1+):**
- `bun run ci:local` runs the entire stack inside Docker: gitleaks (host), unit
tests with `DATABASE_URL` unset, and all 29 E2E files sequentially against a
fresh pgvector container. Stronger than PR CI's 2-file Tier 1 set; closer to
what nightly Tier 1 catches. Spins up + tears down postgres automatically via
`docker-compose.ci.yml`. Override the host port with
`GBRAIN_CI_PG_PORT=5435 bun run ci:local` if 5434 collides.
- `bun run ci:local:diff` runs only the E2E files matched by the diff selector
(`scripts/select-e2e.ts`), falling back to all 29 on unmapped src/ paths or
schema/skills/package.json changes. Fast iteration during a focused branch.
**Path B — manual lifecycle (still supported):**
- `bun test` — unit tests (no database required)
- Follow the "E2E test DB lifecycle" steps above to spin up the test DB,
run `bun run test:e2e`, then tear it down.
Both must pass. Do not ship with failing E2E tests. Do not skip E2E tests.
**Always run typecheck before pushing.** `bun test` (the bun runner)
skips TypeScript type checking — it only enforces runtime behavior.
Three ways to actually gate on types:
1. `bun run test` (npm script in `package.json`) — includes `bun run typecheck`
plus the four shell pre-checks (`check-jsonb-pattern.sh`,
`check-progress-to-stdout.sh`, `check-trailing-newline.sh`,
`check-wasm-embedded.sh`) before the runner. Use this mid-branch.
2. `bun run typecheck` — `tsc --noEmit` standalone. Fast (~5s on this repo).
3. `bun run ci:local` — the full local CI gate from Path A.
The trap is: writing a new test, running `bun test test/foo.test.ts`,
seeing it pass, pushing — and CI's separate typecheck stage rejects an
invalid type literal that the runner accepted. Caught one of these
shipping the v0.23.2 round-trip E2E (`type: 'reflection'` is not a
member of `PageType`). Run `bun run typecheck` once before push, even
when only test files changed.
## Post-ship requirements (MANDATORY)
After EVERY /ship, you MUST run /document-release. This is NOT optional. Do NOT
+29
View File
@@ -52,6 +52,10 @@ docs/ Architecture docs
## Running tests
```bash
# Recommended: full CI guard chain + tests (matches what CI runs)
bun run test # privacy + jsonb + progress + wasm + typecheck + bun test
# Just the test runner (skips CI guards)
bun test # all tests (unit + E2E skipped without DB)
bun test test/markdown.test.ts # specific unit test
@@ -63,6 +67,31 @@ DATABASE_URL=postgresql://postgres:postgres@localhost:5434/gbrain_test bun run t
DATABASE_URL=postgresql://... bun run test:e2e
```
Use `bun run test` before pushing. The guard chain catches: banned fork-name leaks
(`scripts/check-privacy.sh`), `JSON.stringify(x)::jsonb` interpolation patterns
(`scripts/check-jsonb-pattern.sh`), `\r` progress bleed to stdout
(`scripts/check-progress-to-stdout.sh`), trailing-newline drift across tracked
files (`scripts/check-trailing-newline.sh`), and silent fallback to recursive
chunking in the compiled binary (`scripts/check-wasm-embedded.sh`).
### Local CI gate (recommended before pushing, v0.23.1+)
```bash
bun run ci:local # full gate: gitleaks + unit + ALL 29 E2E files (sequential)
bun run ci:local:diff # gate with diff-aware E2E selector
bun run ci:select-e2e # print which E2E files the selector would run
```
`ci:local` spins up `pgvector/pgvector:pg16` + `oven/bun:1` via
`docker-compose.ci.yml`, runs everything PR CI runs plus the full E2E suite, then
tears down. Named volumes keep the install warm across runs (~16-20 min sequential
E2E after the first cold pull). Requires Docker (Docker Desktop, OrbStack, or
Colima) and `gitleaks` on host (`brew install gitleaks`). Override the postgres
host port with `GBRAIN_CI_PG_PORT=5435 bun run ci:local` if 5434 collides.
Fail-closed selector: an unmapped `src/` change runs all 29 E2E files. Hand-tune
narrower mappings via `scripts/e2e-test-map.ts`.
## Building
```bash
+3 -2
View File
@@ -129,8 +129,9 @@ Set up using your platform's scheduler (OpenClaw cron, Railway cron, crontab):
- **Live sync** (every 15 min): `gbrain sync --repo ~/brain && gbrain embed --stale`
- **Auto-update** (daily): `gbrain check-update --json` (tell user, never auto-install)
- **Dream cycle** (nightly): read `docs/guides/cron-schedule.md` for the full protocol.
Entity sweep, citation fixes, memory consolidation. This is what makes the brain
compound. Do not skip it.
Entity sweep, citation fixes, memory consolidation, plus (v0.23+) overnight conversation
synthesis and cross-session pattern detection. 8 phases, one cron-friendly command. This
is what makes the brain compound. Do not skip it.
- **Weekly**: `gbrain doctor --json && gbrain embed --stale`
## Step 8: Integrations
+21 -8
View File
@@ -132,7 +132,7 @@ GBrain ships 29 skills organized by `skills/RESOLVER.md` (or your OpenClaw's `AG
|-------|-------------|
| **enrich** | Tiered enrichment (Tier 1/2/3). Creates and updates person/company pages with compiled truth and timelines. |
| **query** | 3-layer search with synthesis and citations. Says "the brain doesn't have info on X" instead of hallucinating. |
| **maintain** | Periodic health: stale pages, orphans, dead links, citation audit, back-link enforcement, tag consistency. |
| **maintain** | Periodic health: stale pages, orphans, dead links, citation audit, back-link enforcement, tag consistency. v0.23 adds the dream cycle's synthesize + patterns phases ... overnight conversation transcripts become reflections, originals, and 25-year patterns. |
| **citation-fixer** | Scans pages for missing or malformed citations. Fixes format to match the standard. |
| **repo-architecture** | Where new brain files go. Decision protocol: primary subject determines directory, not format. |
| **publish** | Share brain pages as password-protected HTML. Zero LLM calls. |
@@ -316,9 +316,11 @@ is what you spend time on. Everything else is boilerplate the CLI writes for you
Drop a `routing-eval.jsonl` fixture next to any skill. Each line is `{intent, expected_skill,
ambiguous_with?}`. `gbrain check-resolvable` runs the structural layer by default; `gbrain
routing-eval --llm` runs an LLM tie-break layer for CI. False positives (wrong skill matched),
missed routes (no skill matched), and tautological fixtures (intent copies trigger verbatim)
all surface as specific advisories with the exact file:line to fix.
routing-eval` runs the same structural layer as a dedicated CI verb. The `--llm` flag is
accepted as a placeholder for a future LLM tie-break layer; in this release it emits a stderr
notice and runs structural only. False positives (wrong skill matched), missed routes (no
skill matched), and tautological fixtures (intent copies trigger verbatim) all surface as
specific advisories with the exact file:line to fix.
### Works on your OpenClaw, not just gbrain's repo
@@ -355,6 +357,10 @@ gbrain skillpack diff brain-ops # compare bundle vs your local co
Re-running is safe. The managed-block markers in your AGENTS.md let `skillpack install`
accumulate rows across separate single-skill installs instead of overwriting each other.
A receipt comment inside the fence (`<!-- gbrain:skillpack:manifest cumulative-slugs="..." -->`)
tracks what gbrain has installed across runs. `install --all` is the only path that prunes;
per-skill install never deletes what it didn't install. If you hand-add a row inside the fence,
gbrain preserves it on reinstall and emits a stderr notice telling your agent to investigate.
**Skillify is the piece that makes the skills tree survive six months of compounding work.**
Read [`skills/skillify/SKILL.md`](skills/skillify/SKILL.md) for the full 10-item checklist
@@ -639,8 +645,11 @@ SEARCH
gbrain query <question> Hybrid search (vector + keyword + RRF)
IMPORT
gbrain import <dir> [--no-embed] Import markdown (idempotent)
gbrain sync [--repo <path>] Git-to-brain incremental sync
gbrain import <dir> [--no-embed] [--workers N]
Import markdown (idempotent)
gbrain sync [--repo <path>] [--workers N]
Git-to-brain incremental sync
(>100-file diffs auto-parallelize 4 workers on Postgres)
gbrain export [--dir ./out/] Export to markdown
FILES
@@ -686,7 +695,11 @@ ADMIN
gbrain auth create|list|revoke|test Token management for the HTTP transport
gbrain integrations Integration recipe dashboard
gbrain sources list|add|remove|... Multi-source brain management (v0.18)
gbrain dream [--dry-run] [--phase N] One maintenance cycle then exit (cron-friendly)
gbrain dream [--dry-run] [--phase N] 8-phase maintenance cycle (lint→backlinks→sync→synthesize
→extract→patterns→embed→orphans). v0.23 added synthesize +
patterns: transcripts → reflections + cross-session themes.
gbrain dream --input <file> Ad-hoc transcript synthesis (implies --phase synthesize)
gbrain dream --date YYYY-MM-DD Synthesize a single day; --from/--to for backfill ranges
gbrain check-backlinks check|fix Back-link enforcement
gbrain lint [--fix] LLM artifact detection
gbrain repair-jsonb [--dry-run] Repair v0.12.0 double-encoded JSONB (Postgres)
@@ -729,7 +742,7 @@ The skills in this repo are those patterns, generalized. What took 11 days to bu
## Contributing
See [CONTRIBUTING.md](CONTRIBUTING.md). Run `bun test` for unit tests. E2E tests: spin up Postgres with pgvector, run `bun run test:e2e`, tear down.
See [CONTRIBUTING.md](CONTRIBUTING.md). Run `bun test` for unit tests. For the full local CI gate (gitleaks + unit + all 29 E2E files in Docker, the same checks GH Actions runs), use `bun run ci:local` ... or `bun run ci:local:diff` for the diff-aware subset during fast iteration.
PRs welcome for: new enrichment APIs, performance optimizations, additional engine backends, new skills following the conformance standard in `skills/skill-creator/SKILL.md`.
+335
View File
@@ -1,5 +1,328 @@
# TODOS
## ci-local-mirror
### CI-skip artifact + signature for stages 1+2 follow-up
**Priority:** P0
**What:** After a successful local CI run via `bun run ci:local`, write `.ci-cache/passed-<commit-sha>.json` containing `{commit, test_set_hash, bun_version, schema_hash, signature}`. Push to a `ci-cache` orphan branch (or GH Releases). CI's first step fetches the artifact for the current SHA and skips the test job if (a) signature matches Garry's GPG/SSH key, and (b) `test_set_hash` matches what CI would have run.
**Why:** Stages 1+2 (shipped in this branch) give a strong local CI gate, but PR CI still re-runs every test on every push. Stage 3 closes the loop and trades ~10 min of CI wall-time for sub-second artifact verification on Garry's own pushes. External PRs are unaffected because the signature won't match — they hit the normal CI path.
**Pros:**
- ~10 min/PR saved on Garry's own pushes; the local gate becomes the source of truth.
- External contributor PRs untouched (no security regression).
- Forces a clear test-set-hash contract: any drift in what local-vs-CI run is caught at verification time.
**Cons:**
- Trust model needs careful design: signature scheme, key rotation, what happens when signature verification fails.
- Cache invalidation is real — if env or service version drifts between local run and CI, a stale local pass could ship to master.
- Adds a `ci-cache` branch / artifact storage surface to maintain.
**Context:**
- Discussed during the eng-review of the local CI mirror plan at `~/.claude/plans/lets-do-1-2-dockerfile-ci-zany-charm.md`.
- Don't start until stages 1+2 have been used for ~2 weeks AND the `scripts/e2e-test-map.ts` has stabilized (so test_set_hash is a meaningful identity).
- Initial trust-but-verify: run both local and CI in parallel for ~1 week before flipping the skip; alert on any disagreement.
**Effort:** M (human ~2-3 days + ~1 week trust-but-verify period running both local + CI in parallel; CC ~1 day for the mechanics).
**Depends on / blocked by:** Stages 1+2 (this PR) landing first.
### test/e2e/multi-source.test.ts cascade test isn't isolated
**Priority:** P1
**What:** The "sources remove cascades to pages + chunks + timeline + links + files" test in `test/e2e/multi-source.test.ts:281` fails when the file runs after other E2E files in the sequential `bash scripts/run-e2e.sh` order, but passes 20/20 on a fresh Postgres volume. The failing assertion is `SELECT COUNT(*) FROM links WHERE from_page_id = aliceId` expecting 0, getting 1 — so a prior file's setup left a `links` row that references a page id the cascade test happens to reuse. The test's own `setupDB()` truncates but doesn't sweep all referencing rows back when ids collide.
**Why:** Surfaced when `bun run ci:local` (this PR's local CI gate) ran the full sequential E2E. CI never catches it because `.github/workflows/e2e.yml:40` only runs `mechanical.test.ts + mcp.test.ts` on PRs and nightly Tier 1. So 27 of 29 E2E files including this one aren't actually exercised by CI today. The local gate is stronger and surfaces real cross-file isolation gaps.
**Pros:**
- Fixing isolation makes `bun run ci:local` (full E2E) reliably green.
- Same fix likely to harden other E2E files that share id namespaces.
- Lets us turn `bun run ci:local` into a real ship gate.
**Cons:**
- Could require a per-file "namespace your test ids" pattern, ~30 min per affected file across the suite.
**Context:**
- Repro: `bash scripts/run-e2e.sh test/e2e/multi-source.test.ts` against a stale DB after other E2E files have run → fails. Same against a fresh `docker compose down -v && up -d postgres` → passes 20/20.
- The test inserts a hardcoded `cascadetest` source id and `aliceId` page id; collisions across runs are predictable.
- Likely fix: use `mkdtemp`-style randomized source/page ids per test, OR have the test do a deeper reset (DELETE FROM all five tables in beforeEach) instead of relying on `setupDB`'s TRUNCATE behavior.
**Effort:** S (CC ~30 min for the multi-source.test.ts fix; M if we audit all 29 E2E files for similar id-collision risk).
**Depends on / blocked by:** Nothing.
### scripts/run-e2e.sh:71 echo overflows on large-output failing tests
**Priority:** P2
**What:** When an E2E test fails AND prints lots of output (e.g., `multi-source.test.ts` floods postgres NOTICE objects), `scripts/run-e2e.sh:71` does `echo "$output"` against a multi-megabyte shell variable. The host pipe to docker-compose-run hits `EAGAIN` and fails with `echo: write error: Resource temporarily unavailable`. With `set -e`, the script aborts at that point, skipping the remaining E2E files and the final SUMMARY block.
**Why:** When the local CI gate finds a real failure (per the multi-source.test.ts entry above), the user wants to see it AND see how the rest of the suite did. Currently the failure shadows the rest.
**Pros:**
- See all E2E failures from a single run instead of needing to bisect.
- Quick win, ~5 lines.
**Cons:**
- None worth listing.
**Context:**
- Reproduced live during plan verification on 2026-04-29. Previous `multi-source.test.ts` failure killed the script before postgres-bootstrap, postgres-jsonb, etc. could run.
- Likely fix: replace `echo "$output"` with `printf '%s\n' "$output"`, or write `$output` to a tmpfile and `cat` it (handles large blobs better than echo over pipes), or pipe through `stdbuf -o0`.
- Don't suppress the postgres NOTICE flood at the test layer — that's separate; here we just want the script to not die when bun's stderr is verbose.
**Effort:** S (human or CC: ~10 min).
**Depends on / blocked by:** Nothing.
## claw-test E2E (v0.22.16 follow-ups)
### Hermes runner — `src/core/claw-test/runners/hermes.ts`
**Priority:** P2
**What:** Add a Hermes implementation of the `AgentRunner` interface. v1 ships only OpenClaw; v1.1 lands hermes once we have real friction reports from openclaw to validate the contract against.
**Why:** Cross-agent diff (`gbrain friction diff --base openclaw --compare hermes`) is the highest-leverage next signal. Friction unique to one agent vs common-to-both separates "agent contract bug" from "gbrain bug" automatically.
**Effort:** S (CC ~30m). Depends on: v1 openclaw runner producing real friction reports first.
---
### Friction analytics suite — `diff` / `trend` / `migration-stub`
**Priority:** P2
**What:** Three new `gbrain friction` subcommands deferred from v1:
- `gbrain friction diff --base <run-or-agent> --compare <run-or-agent>` (cross-agent comparison; ~80 LOC)
- `gbrain friction trend [--since <version-or-date>] [--phase <name>]` (time-series across runs; ~60 LOC)
- `gbrain friction migration-stub [--threshold N]` (clusters friction by phase + tokens, emits `skills/migrations/v[N+1].md` stub; ~150 LOC)
**Why:** Turns point-in-time reports into a slope. Pairs with the v1.1 public scoreboard.
**Effort:** M (CC ~2h total).
---
### Scenario expansion — `supabase-migration` and `supervisor-restart`
**Priority:** P2
**What:** Two more scenarios under `test/fixtures/claw-test-scenarios/`:
- `supabase-migration``gbrain init --pglite` then `gbrain migrate --to supabase`; verifies the cross-engine migration path
- `supervisor-restart` — kill worker mid-job; verify supervisor recovers without data loss
**Why:** These are the other highest-historical-pain regression points (per CLAUDE.md fix-wave history). v1 ships only `fresh-install` + `upgrade-from-v0.18` because Codex flagged that mixing them dilutes the fresh-install signal; v1.1 lands them as separate scenarios.
**Effort:** M (CC ~1h each).
---
### Real v0.18 SQL dump for upgrade scenario
**Priority:** P2
**What:** The `upgrade-from-v0.18` scenario ships scaffolded — `seed/dump.sql` is missing. The harness gracefully no-ops the seed phase when absent, so the scenario currently behaves like fresh-install. v1.1: generate a real v0.18-shape PGLite dump per the procedure documented in `test/fixtures/claw-test-scenarios/upgrade-from-v0.18/seed/README.md`.
**Why:** Without a real seed, the scenario doesn't actually exercise the migration chain forward-walk. That's the whole point of the upgrade scenario — proves issue #239/#243/#266/#357 class regressions stay fixed.
**Effort:** S (CC ~30m once a v0.18 checkout is handy). Depends on: ability to run a v0.18 gbrain build.
---
### Public scoreboard — `gbrain-evals.io/friction`
**Priority:** P3
**What:** Sibling-repo PR in `garrytan/gbrain-evals` that renders friction JSONL into a public dashboard. Friction count per version per agent, line charts over time. v1's JSONL already includes `gbrain_version` + `agent` tags so the scoreboard is a thin layer on top.
**Why:** Marketing surface. Proves install quality is improving release-over-release. The friction loop becomes visible to the world, not just maintainers.
**Effort:** M. Depends on: a working live mode and ≥10 real friction reports.
---
### PTY-mode transcript capture
**Priority:** P3
**What:** `transcript-capture.ts` currently uses plain `child_process.spawn` pipes. Some agents only emit ANSI colors / progress UI on a TTY. v1.1 adds a PTY mode (likely via `node-pty`) so live-mode transcripts capture the full agent UX.
**Why:** Faithful transcripts make the friction → reasoning link more useful. v1 accepts that some agent UI is lost.
**Effort:** S (CC ~30m). Mostly a ~30 LOC swap inside `spawnWithCapture`.
---
### Read-side host-isolation (`$GBRAIN_HOST_HOME`)
**Priority:** P3
**What:** v0.22.16 confined every `~/.gbrain` write site to honor `$GBRAIN_HOME`. But `src/commands/init.ts:299-313` still reads real `~/.claude` / `~/.openclaw` / `~/.codex` / `~/.factory` / `~/.kiro` for module fingerprinting (host detection). Even with write-isolation, a claw-test running on a developer's box discovers their real installed mods. v1.1: add a separate `$GBRAIN_HOST_HOME` override for the read-side detection so the claw-test can run truly hermetic.
**Why:** v1's hermeticity contract is "writes are isolated, reads are not." v1.1 closes the read-side gap.
**Effort:** S (CC ~30m).
---
### Routing-callout sweep — annotate skills the claw-test exercises
**Priority:** P3
**What:** `skills/_friction-protocol.md` is a cross-cutting convention. v1.1: sweep the 46 skills the claw-test actually exercises (setup, brain-ops, query, ingest, smoke-test, the migrations the test covers) and add a `> **Convention:** see [skills/_friction-protocol.md](_friction-protocol.md).` callout via the existing `src/core/dry-fix.ts` shape so DRY auto-fix doesn't fight it.
**Why:** Right now agents only call `gbrain friction log` if they find the protocol skill on their own. The callouts route them there proactively from any harness-exercised skill.
**Effort:** S (CC ~15m).
---
## minions / worker (v0.22.14 follow-ups)
### v0.22.15 — Embed cooperative-abort (HIGHEST PRIORITY — daily pain)
**Priority:** P0
**What:** Plumb `signal: AbortSignal` through `runPhaseEmbed`
`src/commands/embed.ts``embedBatch` in `src/core/embedding.ts`. Check
`signal?.aborted` between OpenAI batch calls (every ~100 texts, ~2s
real-time) and between slugs in the per-slug loop.
**Why:** Embed phase ignores `signal.aborted` between batches today. Job
wall-clock timeout fires → handler keeps running → cycle's finally block
unreachable → `gbrain_cycle_locks` row stays held indefinitely. Every
subsequent autopilot cron cycle sees `cycle_already_running` → skips. Lock
TTL is 30 min; new cycles give up before that. Doctor reports UNHEALTHY.
**The chain in production:** ~5min cron submits cycle → 22K stale pages →
embed phase takes 1015 min → 600s timeout fires → job dead-lettered → embed
keeps running → lock held → all subsequent cycles skip. Garry hits this
DAILY on his production brain.
**Pros:** Closes the daily wedge. Makes timeouts actually effective. Lets
operators bump worker timeouts confidently knowing abort actually stops
work.
**Cons:** Touching the embed hot path; small risk of botching the abort
checks. Mitigation: between-batch granularity (~2s), not per-text (too fine)
or per-slug (too coarse for 500+ chunk slugs).
**Context:** PR #503 (v0.22.14) catches the SYMPTOM (worker stalled, queue
piling up) via self-health-monitoring. This PR catches the CAUSE for one
specific failure class. Both fixes are needed; they're complementary, not
duplicative.
**Files to touch:**
- `src/core/cycle.ts:579``runPhaseEmbed(engine, dryRun)` → add
`signal?: AbortSignal` arg
- `src/core/cycle.ts:803` — pass `opts.signal` through
- `src/commands/embed.ts:~363` — accept signal, check between slugs
- `src/core/embedding.ts:51-56``embedBatch(texts, onProgress?, signal?)`,
check between for-loop iterations of `BATCH_SIZE` slices
**Tests required:**
1. embedBatch checks signal between OpenAI calls; aborts within one batch (~2s)
2. Per-slug loop in `embed.ts` checks signal between slugs
3. End-to-end: cycle handler with embed phase + signal aborted mid-flight →
finally runs → `gbrain_cycle_locks` row deleted
4. Regression: 1K+ chunks scenario — embed does NOT block lock release when
timeout fires
**Effort:** M (human: ~3 hr / CC: ~30 min).
**Depends on / blocked by:** Nothing. v0.22.14 ships first.
### v0.23+ — Bare-worker engine reconnect parity with supervisor
**Priority:** P2
**What:** Extract the supervisor's reconnect-then-fail pattern into
`MinionWorker` so bare workers can retry transient DB blips before exiting.
Today the supervisor calls `engine.reconnect()` after 3 consecutive DB health
failures (#406); the bare worker just emits `'unhealthy'` and the CLI calls
`process.exit(1)`.
**Why:** Bare-worker behavior is more disruptive than supervised behavior on
transient PgBouncer blips. A bare worker restarts the entire process; a
supervised worker just reconnects the pool. Operationally the supervisor
approach is gentler (no in-flight job loss, no PM restart latency).
**Pros:** Unifies bare and supervised behavior. Reduces process churn on
transient network blips.
**Cons:** More code in MinionWorker; risk of reconnect masking a real
problem. Mitigation: cap retry attempts, fall through to `'unhealthy'`
emission after the cap.
**Context:** Filed during v0.22.14 plan-eng-review. The asymmetry is
documented in v0.22.14 CHANGELOG as deliberate; this TODO captures the
"unify someday" intent.
**Effort:** S (human: ~2 hr / CC: ~20 min).
**Depends on / blocked by:** Nothing.
### v0.23+ — `minion_workers` heartbeat table for queue_health doctor (B7)
**Priority:** P3
**What:** Add a `minion_workers` table (`worker_id` PK, `hostname`,
`last_heartbeat`, `queue`, `concurrency`, `started_at`) so the existing
`queue_health` doctor check (Postgres path) can detect dead workers via
heartbeat staleness instead of relying on the indirect `lock_until` proxy.
**Why:** v0.19.1 added `queue_health` checks for stalled-active jobs and
waiting-depth threshold. The worker-heartbeat subcheck was deferred (B7)
because the `lock_until`-on-active-jobs proxy can't distinguish "worker
exited cleanly" from "worker idle" — a check that cries wolf erodes trust
in every doctor check. With a real heartbeat row, doctor can say "no worker
seen in N intervals" with confidence.
**Pros:** Doctor's `queue_health` becomes ground-truth. Detects "worker
container died but cron didn't restart it" scenario.
**Cons:** New table, schema migration, every health-tick UPSERTs. Costs
a write per worker per minute (default).
**Context:** Filed during v0.22.14 plan-eng-review. PR #503's self-health
monitoring is the worker-side liveness; this would be the queue-side
ground-truth.
**Effort:** M (human: ~1 day / CC: ~1 hr).
**Depends on / blocked by:** Schema migration system; nothing else.
## sync (v0.22.13 follow-up — PR #490 review)
### D-PR490-1 — Plumb resolved `database_url` through `SyncOpts`
**Priority:** P3
**What:** Add `database_url?: string` (or a richer `resolvedConnection` shape) to
`SyncOpts` and have the caller (`runSync`, the cycle handler, the jobs handler)
populate it from the active engine instead of having `performSync` /
`performFullSync` / `import.ts` each call `loadConfig()` separately. Today every
sync run hits the config file three times.
**Why:** v0.18 multi-source brains can in principle run different sources against
different `database_url` endpoints (or different per-source overrides via
`sources.config_jsonb`). Right now `loadConfig()` returns the global config, and
that always matches the engine in practice — but the convention papers over a
real divergence the moment someone wants per-source connection settings. Folding
the resolution into `SyncOpts` makes the worker-engine creation in `sync.ts` and
`import.ts` deterministic from `SyncOpts` alone.
**Pros:**
- Removes 3 redundant `loadConfig()` calls per sync.
- Makes `performSync` / `performFullSync` side-effect-free with respect to the
on-disk config file.
- Sets up for per-source `database_url` overrides without further refactor.
- Makes the v0.22.13 belt-and-suspenders fallback (PR #490 Q3) cleaner — no
more `!config?.database_url` short-circuit inside the parallel branch.
**Cons:**
- API-shape change to `SyncOpts` (mild; not externally exported).
- Touching three callers (`runSync`, jobs handler, `cycle.ts` `runPhaseSync`).
- Only worth doing when paired with a per-source override story; otherwise
it's just plumbing.
**Context:** Surfaced during the PR #490 plan-eng-review (parallel sync).
Deferred because it isn't on the v0.22.13 critical path. The same pattern would
benefit the cycle handler and the autopilot daemon. See the plan-eng-review
decisions log: A4 = "Defer; file as TODO."
**Depends on / blocked by:** Nothing structural. Best paired with the v0.18
per-source `config_jsonb` work if/when that lands.
## sync error-code classification (PR #501 follow-ups)
### Plumb structured `ParseValidationCode` through `ImportResult`
@@ -220,6 +543,18 @@ keeping both skills' triggers intact for chaining.
**Depends on / blocked by:** Nothing — UNION-on-read path keeps unresolved edges surfaced even without this.
## P3 — Dev experience: test suite parallelism on fast multi-core machines
**Context:** `bun test` on M-series Macs spawns ~1 worker per core. `test/dream.test.ts` (5 describe blocks, 11 tests) and `test/orphans.test.ts` create a fresh PGLite engine in `beforeEach` that runs ~20 schema migrations per test. Under parallel load, WASM-instance contention causes ~18 `beforeEach` timeouts at 59s.
**Evidence:** CI (ubuntu-latest, fewer cores) is green on every PR. Running the suspect files in isolation (`bun test test/dream.test.ts test/orphans.test.ts`) is also green. Reproduces only on fast multi-core local machines running the full 136-file parallel suite.
**Fix:** move engine creation from `beforeEach` to `beforeAll` per describe block; add a data-reset helper (delete-all-rows-in-relevant-tables) between tests. ~80 LOC change across two test files.
**Priority:** P3 because production CI is unaffected. Hits local dev iteration speed on fast Macs.
**Found:** 2026-04-24 during v0.19.0 production-readiness review.
## Completed
### ~~Checks 5 + 6 for check-resolvable~~
+1 -1
View File
@@ -1 +1 @@
0.22.12
0.24.0
+13 -10
View File
@@ -20,6 +20,7 @@
},
"devDependencies": {
"@types/bun": "latest",
"bun-types": "^1.3.13",
"typescript": "^5.6.0",
},
},
@@ -220,7 +221,7 @@
"@types/bun": ["@types/bun@1.3.11", "", { "dependencies": { "bun-types": "1.3.11" } }, "sha512-5vPne5QvtpjGpsGYXiFyycfpDF2ECyPcTSsFBMa0fraoxiQyMJ3SmuQIGhzPg2WJuWxVBoxWJ2kClYTcw/4fAg=="],
"@types/node": ["@types/node@18.19.130", "", { "dependencies": { "undici-types": "~5.26.4" } }, "sha512-GRaXQx6jGfL8sKfaIDD6OupbIHBr9jv7Jnaml9tB7l4v068PAOXqfcujMMo5PhbIs6ggR1XODELqahT2R8v0fg=="],
"@types/node": ["@types/node@25.5.2", "", { "dependencies": { "undici-types": "~7.18.0" } }, "sha512-tO4ZIRKNC+MDWV4qKVZe3Ql/woTnmHDr5JD8UI5hn2pwBrHEwOEMZK7WlNb5RKB6EoJ02gwmQS9OrjuFnZYdpg=="],
"@types/node-fetch": ["@types/node-fetch@2.6.13", "", { "dependencies": { "@types/node": "*", "form-data": "^4.0.4" } }, "sha512-QGpRVpzSaUs30JBSGPjOg4Uveu384erbHBoT1zeONvyCfwQxIkUshLAOqN/k9EjGviPRmWTTe6aH2qySWKTVSw=="],
@@ -242,7 +243,7 @@
"bowser": ["bowser@2.14.1", "", {}, "sha512-tzPjzCxygAKWFOJP011oxFHs57HzIhOEracIgAePE4pqB3LikALKnSzUyU4MGs9/iCEUuHlAJTjTc5M+u7YEGg=="],
"bun-types": ["bun-types@1.3.11", "", { "dependencies": { "@types/node": "*" } }, "sha512-1KGPpoxQWl9f6wcZh57LvrPIInQMn2TQ7jsgxqpRzg+l0QPOFvJVH7HmvHo/AiPgwXy+/Thf6Ov3EdVn1vOabg=="],
"bun-types": ["bun-types@1.3.13", "", { "dependencies": { "@types/node": "*" } }, "sha512-QXKeHLlOLqQX9LgYaHJfzdBaV21T63HhFJnvuRCcjZiaUDpbs5ED1MgxbMra71CsryN/1dAoXuJJJwIv/2drVA=="],
"bytes": ["bytes@3.1.2", "", {}, "sha512-/Nf7TyzTx6S3yRJObOAV7956r8cr2+Oj8AC5dt8wSP3BQAoeX58NoHyCU8P8zGkNXStjTSi6fzO6F0pBdcYbEg=="],
@@ -466,7 +467,7 @@
"typescript": ["typescript@5.9.3", "", { "bin": { "tsc": "bin/tsc", "tsserver": "bin/tsserver" } }, "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw=="],
"undici-types": ["undici-types@5.26.5", "", {}, "sha512-JlCMO+ehdEIKqlFxk6IfVoAUVmgz7cU7zD/h9XZ0qzeosSHmUJVOzSQvvYSYWXkFXC+IfLKSIffhv0sVZup6pA=="],
"undici-types": ["undici-types@7.18.2", "", {}, "sha512-AsuCzffGHJybSaRrmr5eHr81mwJU3kjw6M+uprWvCXiNeN9SOGwQ3Jn8jb8m3Z6izVgknn1R0FTCEAP2QrLY/w=="],
"unpipe": ["unpipe@1.0.0", "", {}, "sha512-pjy2bYhSsufwWlKwPc+l3cN7+wuJlK6uz0YdJEOlQDbl6jo/YlPi4mb8agUkVC8BF7V8NuzeyPNqRksA3hztKQ=="],
@@ -488,30 +489,32 @@
"zod-to-json-schema": ["zod-to-json-schema@3.25.2", "", { "peerDependencies": { "zod": "^3.25.28 || ^4" } }, "sha512-O/PgfnpT1xKSDeQYSCfRI5Gy3hPf91mKVDuYLUHZJMiDFptvP41MSnWofm8dnCm0256ZNfZIM7DSzuSMAFnjHA=="],
"@anthropic-ai/sdk/@types/node": ["@types/node@18.19.130", "", { "dependencies": { "undici-types": "~5.26.4" } }, "sha512-GRaXQx6jGfL8sKfaIDD6OupbIHBr9jv7Jnaml9tB7l4v068PAOXqfcujMMo5PhbIs6ggR1XODELqahT2R8v0fg=="],
"@aws-crypto/sha1-browser/@smithy/util-utf8": ["@smithy/util-utf8@2.3.0", "", { "dependencies": { "@smithy/util-buffer-from": "^2.2.0", "tslib": "^2.6.2" } }, "sha512-R8Rdn8Hy72KKcebgLiv8jQcQkXoLMOGGv5uI1/k0l+snqkOzQ1R0ChUBCxWMlBsFMekWjq0wRudIweFs7sKT5A=="],
"@aws-crypto/sha256-browser/@smithy/util-utf8": ["@smithy/util-utf8@2.3.0", "", { "dependencies": { "@smithy/util-buffer-from": "^2.2.0", "tslib": "^2.6.2" } }, "sha512-R8Rdn8Hy72KKcebgLiv8jQcQkXoLMOGGv5uI1/k0l+snqkOzQ1R0ChUBCxWMlBsFMekWjq0wRudIweFs7sKT5A=="],
"@aws-crypto/util/@smithy/util-utf8": ["@smithy/util-utf8@2.3.0", "", { "dependencies": { "@smithy/util-buffer-from": "^2.2.0", "tslib": "^2.6.2" } }, "sha512-R8Rdn8Hy72KKcebgLiv8jQcQkXoLMOGGv5uI1/k0l+snqkOzQ1R0ChUBCxWMlBsFMekWjq0wRudIweFs7sKT5A=="],
"@types/node-fetch/@types/node": ["@types/node@25.5.2", "", { "dependencies": { "undici-types": "~7.18.0" } }, "sha512-tO4ZIRKNC+MDWV4qKVZe3Ql/woTnmHDr5JD8UI5hn2pwBrHEwOEMZK7WlNb5RKB6EoJ02gwmQS9OrjuFnZYdpg=="],
"bun-types/@types/node": ["@types/node@25.5.2", "", { "dependencies": { "undici-types": "~7.18.0" } }, "sha512-tO4ZIRKNC+MDWV4qKVZe3Ql/woTnmHDr5JD8UI5hn2pwBrHEwOEMZK7WlNb5RKB6EoJ02gwmQS9OrjuFnZYdpg=="],
"@types/bun/bun-types": ["bun-types@1.3.11", "", { "dependencies": { "@types/node": "*" } }, "sha512-1KGPpoxQWl9f6wcZh57LvrPIInQMn2TQ7jsgxqpRzg+l0QPOFvJVH7HmvHo/AiPgwXy+/Thf6Ov3EdVn1vOabg=="],
"form-data/mime-types": ["mime-types@2.1.35", "", { "dependencies": { "mime-db": "1.52.0" } }, "sha512-ZDY+bPm5zTTF+YpCrAU9nK0UgICYPT0QtT1NZWFv4s++TNkcgVaT0g6+4R2uI4MjQjzysHB1zxuWL50hzaeXiw=="],
"openai/@types/node": ["@types/node@18.19.130", "", { "dependencies": { "undici-types": "~5.26.4" } }, "sha512-GRaXQx6jGfL8sKfaIDD6OupbIHBr9jv7Jnaml9tB7l4v068PAOXqfcujMMo5PhbIs6ggR1XODELqahT2R8v0fg=="],
"@anthropic-ai/sdk/@types/node/undici-types": ["undici-types@5.26.5", "", {}, "sha512-JlCMO+ehdEIKqlFxk6IfVoAUVmgz7cU7zD/h9XZ0qzeosSHmUJVOzSQvvYSYWXkFXC+IfLKSIffhv0sVZup6pA=="],
"@aws-crypto/sha1-browser/@smithy/util-utf8/@smithy/util-buffer-from": ["@smithy/util-buffer-from@2.2.0", "", { "dependencies": { "@smithy/is-array-buffer": "^2.2.0", "tslib": "^2.6.2" } }, "sha512-IJdWBbTcMQ6DA0gdNhh/BwrLkDR+ADW5Kr1aZmd4k3DIF6ezMV4R2NIAmT08wQJ3yUK82thHWmC/TnK/wpMMIA=="],
"@aws-crypto/sha256-browser/@smithy/util-utf8/@smithy/util-buffer-from": ["@smithy/util-buffer-from@2.2.0", "", { "dependencies": { "@smithy/is-array-buffer": "^2.2.0", "tslib": "^2.6.2" } }, "sha512-IJdWBbTcMQ6DA0gdNhh/BwrLkDR+ADW5Kr1aZmd4k3DIF6ezMV4R2NIAmT08wQJ3yUK82thHWmC/TnK/wpMMIA=="],
"@aws-crypto/util/@smithy/util-utf8/@smithy/util-buffer-from": ["@smithy/util-buffer-from@2.2.0", "", { "dependencies": { "@smithy/is-array-buffer": "^2.2.0", "tslib": "^2.6.2" } }, "sha512-IJdWBbTcMQ6DA0gdNhh/BwrLkDR+ADW5Kr1aZmd4k3DIF6ezMV4R2NIAmT08wQJ3yUK82thHWmC/TnK/wpMMIA=="],
"@types/node-fetch/@types/node/undici-types": ["undici-types@7.18.2", "", {}, "sha512-AsuCzffGHJybSaRrmr5eHr81mwJU3kjw6M+uprWvCXiNeN9SOGwQ3Jn8jb8m3Z6izVgknn1R0FTCEAP2QrLY/w=="],
"bun-types/@types/node/undici-types": ["undici-types@7.18.2", "", {}, "sha512-AsuCzffGHJybSaRrmr5eHr81mwJU3kjw6M+uprWvCXiNeN9SOGwQ3Jn8jb8m3Z6izVgknn1R0FTCEAP2QrLY/w=="],
"form-data/mime-types/mime-db": ["mime-db@1.52.0", "", {}, "sha512-sPU4uV7dYlvtWJxwwxHD0PuihVNiE7TyAbQ5SWxDCB9mUYvOgroQOwYQQOKPJ8CIbE+1ETVlOoK1UC2nU3gYvg=="],
"openai/@types/node/undici-types": ["undici-types@5.26.5", "", {}, "sha512-JlCMO+ehdEIKqlFxk6IfVoAUVmgz7cU7zD/h9XZ0qzeosSHmUJVOzSQvvYSYWXkFXC+IfLKSIffhv0sVZup6pA=="],
"@aws-crypto/sha1-browser/@smithy/util-utf8/@smithy/util-buffer-from/@smithy/is-array-buffer": ["@smithy/is-array-buffer@2.2.0", "", { "dependencies": { "tslib": "^2.6.2" } }, "sha512-GGP3O9QFD24uGeAXYUjwSTXARoqpZykHadOmA8G5vfJPK0/DC67qa//0qvqrJzL1xc8WQWX7/yc7fwudjPHPhA=="],
"@aws-crypto/sha256-browser/@smithy/util-utf8/@smithy/util-buffer-from/@smithy/is-array-buffer": ["@smithy/is-array-buffer@2.2.0", "", { "dependencies": { "tslib": "^2.6.2" } }, "sha512-GGP3O9QFD24uGeAXYUjwSTXARoqpZykHadOmA8G5vfJPK0/DC67qa//0qvqrJzL1xc8WQWX7/yc7fwudjPHPhA=="],
+117
View File
@@ -0,0 +1,117 @@
# docker-compose.ci.yml
#
# Local CI gate with 4-way E2E sharding. Spins up 4 pgvector services + a bun
# runner that bind-mounts the repo. Used by `bun run ci:local` and
# `bun run ci:local:diff` (see scripts/ci-local.sh).
#
# All services are pulled as `image:` (no build) so `docker compose pull`
# refreshes everything. The bun version floats with `oven/bun:1` to track CI's
# `bun-version: latest`. Named volumes isolate the Linux container's deps from
# the host's darwin-arm64 deps and keep bun + postgres data warm across runs.
#
# Why 4 postgres services: bun's E2E suite shares one DB across 36 files and
# uses TRUNCATE CASCADE in setupDB(). Running files in parallel against ONE DB
# races (file A's TRUNCATE clobbers file B's fixture import). 4 separate DBs
# remove the race; we shard the file list 1/4..4/4 and run shards in parallel.
# Within a shard, files still run sequentially. Total wall-time on a 16-core
# host: ~6 min sequential -> ~1.5-2 min sharded.
#
# Postgres host ports default to 5434-5437 (avoid 5432 manual `gbrain-test-pg`
# and 5433 sibling-project conflicts). Override BASE port with GBRAIN_CI_PG_PORT;
# shards take BASE..BASE+3.
services:
postgres-1:
image: pgvector/pgvector:pg16
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: gbrain_test
ports:
- "${GBRAIN_CI_PG_PORT:-5434}:5432"
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres -d gbrain_test"]
interval: 10s
timeout: 5s
retries: 5
volumes:
- gbrain-ci-pg-data-1:/var/lib/postgresql/data
postgres-2:
image: pgvector/pgvector:pg16
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: gbrain_test
ports:
- "${GBRAIN_CI_PG_PORT_2:-5435}:5432"
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres -d gbrain_test"]
interval: 10s
timeout: 5s
retries: 5
volumes:
- gbrain-ci-pg-data-2:/var/lib/postgresql/data
postgres-3:
image: pgvector/pgvector:pg16
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: gbrain_test
ports:
- "${GBRAIN_CI_PG_PORT_3:-5436}:5432"
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres -d gbrain_test"]
interval: 10s
timeout: 5s
retries: 5
volumes:
- gbrain-ci-pg-data-3:/var/lib/postgresql/data
postgres-4:
image: pgvector/pgvector:pg16
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: gbrain_test
ports:
- "${GBRAIN_CI_PG_PORT_4:-5437}:5432"
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres -d gbrain_test"]
interval: 10s
timeout: 5s
retries: 5
volumes:
- gbrain-ci-pg-data-4:/var/lib/postgresql/data
runner:
image: oven/bun:1
working_dir: /app
depends_on:
postgres-1:
condition: service_healthy
postgres-2:
condition: service_healthy
postgres-3:
condition: service_healthy
postgres-4:
condition: service_healthy
# No global DATABASE_URL — scripts/ci-local.sh sets per-shard URL via -e.
# Unit phase explicitly unsets DATABASE_URL so test/e2e/* gracefully skip.
volumes:
- .:/app
# Linux container's node_modules MUST be isolated from host darwin-arm64.
# Without this, container `bun install` stomps host node_modules and
# subsequent `bun test` on host fails with binary-incompat errors.
- gbrain-ci-node-modules:/app/node_modules
# Warm install cache across runs.
- gbrain-ci-bun-cache:/root/.bun/install/cache
volumes:
gbrain-ci-pg-data-1:
gbrain-ci-pg-data-2:
gbrain-ci-pg-data-3:
gbrain-ci-pg-data-4:
gbrain-ci-node-modules:
gbrain-ci-bun-cache:
+1 -1
View File
@@ -73,7 +73,7 @@ hook resumes blocking malformed pages.
## For downstream agent forks
If your fork (Wintermute, Hermes, OpenClaw) wraps gbrain in a host repo
If your OpenClaw wraps gbrain in a host repo
that's not the brain repo itself, you may want a separate hook strategy:
- **Brain repo IS the host repo** (gbrain skills + brain pages in one repo):
+105 -22
View File
@@ -56,9 +56,16 @@ writing or reviewing an operation, consult `src/core/operations.ts` for the cont
## Before shipping
Run `bun test` plus the E2E lifecycle described in `./CLAUDE.md` (spin up the test
Postgres container, run `bun run test:e2e`, tear it down). Ship via the `/ship` skill,
not by hand.
Easiest path: `bun run ci:local` runs the full CI gate inside Docker (gitleaks,
unit tests with `DATABASE_URL` unset, then all 29 E2E files sequentially against a
fresh pgvector container) and tears down. Use `bun run ci:local:diff` for the
diff-aware subset during fast iteration on a focused branch. Requires Docker
(Docker Desktop / OrbStack / Colima) and `gitleaks` (`brew install gitleaks`).
Manual path: `bun test` plus the E2E lifecycle described in `./CLAUDE.md` (spin
up the test Postgres container, run `bun run test:e2e`, tear it down).
Ship via the `/ship` skill, not by hand.
## Privacy
@@ -101,7 +108,7 @@ strict behavior when unset.
## Key files
- `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`. `OperationContext.remote` flags untrusted callers.
- `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`, plus `matchesSlugAllowList(slug, prefixes)` (v0.23 glob matcher: `<prefix>/*` matches recursive children; bare `<prefix>` matches exact only). `OperationContext.remote` flags untrusted callers; `OperationContext.allowedSlugPrefixes` (v0.23) is the trusted-workspace allow-list set by the dream cycle. `put_page` enforces: when `viaSubagent` and `allowedSlugPrefixes` is set, slug must match the allow-list; else the legacy `wiki/agents/<id>/...` namespace check applies. Auto-link enabled for trusted-workspace writes (skipped only when `remote=true && !trustedWorkspace`).
- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports.
- `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`)
- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for the specific forward-referenced state the embedded schema blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target table) and adds only what's missing. Closes the upgrade-wedge bug class that bit users 10+ times across 6 schema versions over 2 years (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396). No-op on fresh installs and modern brains.
@@ -136,9 +143,9 @@ strict behavior when unset.
- `src/core/resolver-filenames.ts` (v0.19) — central list of accepted routing filenames (`RESOLVER.md`, `AGENTS.md`). Shared by `findRepoRoot`, `check-resolvable`, and skillpack install so every code path walks the same fallback chain.
- `src/commands/skillify.ts` + `src/core/skillify/{generator,templates}.ts` (v0.19) — `gbrain skillify scaffold <name>` creates all stubs for a new skill in one command: SKILL.md, script, tests, routing-eval.jsonl, resolver entry, filing-rules pointer. `gbrain skillify check <script>` runs the 10-step checklist (LLM evals, routing evals, check-resolvable gate, filing audit) against a candidate skill before it lands.
- `src/commands/skillify-check.ts` (v0.19) — `gbrain skillpack-check` agent-readable health report. Exit 0/1/2 for CI pipeline gating; JSON for debugging. Wraps `check-resolvable --json`, `doctor --json`, and migration ledger into one payload so agents can decide whether a human action is required.
- `src/commands/skillpack.ts` + `src/core/skillpack/{bundle,installer}.ts` (v0.19) — `gbrain skillpack install` drops gbrain's curated 25-skill bundle into a host workspace, managed-block style. Never clobbers local edits; tracks a skill manifest so subsequent `install --update` diffs cleanly. Bundle builder (`skillpack/bundle.ts`) packages the set from `skills/` into a versioned payload.
- `src/commands/skillpack.ts` + `src/core/skillpack/{bundle,installer}.ts` (v0.19) — `gbrain skillpack install` drops gbrain's curated 25-skill bundle into a host workspace, managed-block style. Never clobbers local edits; tracks a skill manifest so subsequent `install --update` diffs cleanly. Bundle builder (`skillpack/bundle.ts`) packages the set from `skills/` into a versioned payload. **v0.24.0:** managed block embeds a `<!-- gbrain:skillpack:manifest cumulative-slugs="..." version="..." -->` receipt inside the fence. Per-skill installs accumulate via `union(prior_receipt, this_call)`; `install --all` is the only path that prunes (drops slugs no longer in the bundle). Rows inside the fence whose slug is in neither the new cumulative set nor the bundle survive as user-added with a stderr `[skillpack] unknown row in managed block: "<slug>" — Investigate: ...` warning. Pre-v0.24 fences upgrade silently on first install (extracted slugs become the prior cumulative set).
- `src/core/skill-manifest.ts` (v0.19) — parser for `skill-manifest.json` records. Used by skillpack installer to detect drift between the shipped bundle and the user's local edits, so updates merge instead of overwriting.
- `src/commands/routing-eval.ts` + `src/core/routing-eval.ts` (v0.19) — `gbrain routing-eval` catches user phrasings that route to the wrong skill. Reads `skills/<name>/routing-eval.jsonl` fixtures (`{intent, expected_skill, ambiguous_with?}`). Structural layer runs in `check-resolvable` by default (zero API cost); `--llm` opts into a Haiku tie-break layer for CI. False positives surface before users hit them.
- `src/commands/routing-eval.ts` + `src/core/routing-eval.ts` (v0.19) — `gbrain routing-eval` catches user phrasings that route to the wrong skill. Reads `skills/<name>/routing-eval.jsonl` fixtures (`{intent, expected_skill, ambiguous_with?}`). Structural layer runs in `check-resolvable` by default (zero API cost). The `--llm` flag is accepted as a placeholder for a future LLM tie-break layer; in v0.24.0 it emits a stderr notice and runs structural only. False positives surface before users hit them.
- `src/core/filing-audit.ts` + `skills/_brain-filing-rules.json` (v0.19) — Check 6 of `check-resolvable`. Parses new `writes_pages:` / `writes_to:` frontmatter on skills and audits their filing claims against the filing-rules JSON. Warning-only in v0.19, upgrades to error in v0.20.
- `src/core/dry-fix.ts` — `gbrain doctor --fix` engine. `autoFixDryViolations(fixes, {dryRun})` rewrites inlined rules to `> **Convention:** see [path](path).` callouts via three shape-aware expanders (bullet / blockquote / paragraph). Five guards: working-tree-dirty (`getWorkingTreeStatus()` returns 3-state `'clean' | 'dirty' | 'not_a_repo'`), no-git-backup, inside-code-fence, already-delegated (40-line proximity, consistent with detector), ambiguous-multi-match, block-is-callout. `execFileSync` array args (no shell — no injection surface). EOF newline preserved.
- `src/core/backoff.ts` — Adaptive load-aware throttling: CPU/memory checks, exponential backoff, active hours multiplier
@@ -166,12 +173,12 @@ strict behavior when unset.
- `src/core/minions/wait-for-completion.ts` (v0.15) — poll-until-terminal helper for CLI callers. `TimeoutError` does NOT cancel the job; `AbortSignal` exits without throwing. Default `pollMs`: 1000 on Postgres, 250 on PGLite inline.
- `src/core/minions/transcript.ts` (v0.15) — renders `subagent_messages` + `subagent_tool_executions` to markdown. Tool rows splice under their owning assistant `tool_use` by `tool_use_id`. UTF-8-safe truncation; unknown block types fall through to fenced JSON.
- `src/core/minions/plugin-loader.ts` (v0.15) — `GBRAIN_PLUGIN_PATH` discovery. Absolute paths only, left-wins collision, `gbrain.plugin.json` with `plugin_version: "gbrain-plugin-v1"`, plugins ship DEFS only (no new tools), `allowed_tools:` validated at load time against the derived registry.
- `src/core/minions/tools/brain-allowlist.ts` (v0.15) — derives subagent tool registry from `src/core/operations.ts`. 11-name allow-list: `query`, `search`, `get_page`, `list_pages`, `file_list`, `file_url`, `get_backlinks`, `traverse_graph`, `resolve_slugs`, `get_ingest_log`, `put_page`. `put_page` schema is namespace-wrapped per subagent (`^wiki/agents/<subagentId>/.+`); the `put_page` op's server-side check is the authoritative gate via `ctx.viaSubagent` fail-closed.
- `src/core/minions/tools/brain-allowlist.ts` (v0.15, extended v0.23) — derives subagent tool registry from `src/core/operations.ts`. 11-name allow-list. By default `put_page` schema is namespace-wrapped per subagent (`^wiki/agents/<subagentId>/.+`). **v0.23 trusted-workspace path:** when `BuildBrainToolsOpts.allowedSlugPrefixes` is set, the put_page schema instead describes the prefix list to the model and the OperationContext is threaded with `allowedSlugPrefixes`. Trust comes from `PROTECTED_JOB_NAMES` gating subagent submission — MCP cannot reach this field. Only cycle.ts (synthesize/patterns) and direct CLI submitters set it.
- `src/mcp/tool-defs.ts` (v0.15) — extracted `buildToolDefs(ops)` helper. MCP server + subagent tool registry both call it; byte-for-byte equivalence pinned by `test/mcp-tool-defs.test.ts`.
- `src/core/minions/attachments.ts` — Attachment validation (path traversal, null byte, oversize, base64, duplicate detection)
- `src/commands/agent.ts` (v0.16) — `gbrain agent run <prompt> [flags]` CLI. Submits `subagent` (or N children + 1 aggregator) under `{allowProtectedSubmit: true}`. Single-entry `--fanout-manifest` short-circuits. Children get `on_child_fail: 'continue'` + `max_stalled: 3`. `--follow` is the default on TTY; streams logs + polls `waitForCompletion` in parallel. Ctrl-C detaches, does not cancel.
- `src/commands/agent-logs.ts` (v0.16) — `gbrain agent logs <job> [--follow] [--since]`. Merges JSONL heartbeat audit + `subagent_messages` into a chronological timeline. `parseSince` accepts ISO-8601 or relative (`5m`, `1h`, `2d`). Transcript tail renders only for terminal jobs.
- `src/commands/jobs.ts` — `gbrain jobs` CLI subcommands + `gbrain jobs work` daemon. v0.13.1 surfaces the full `MinionJobInput` retry/backoff/timeout/idempotency surface as first-class CLI flags on `jobs submit`: `--max-stalled`, `--backoff-type fixed|exponential`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key`. `jobs smoke --sigkill-rescue` is the opt-in regression guard for #219. v0.16 wires `registerBuiltinHandlers` to always register `subagent` + `subagent_aggregator` (no env flag — `ANTHROPIC_API_KEY` is the natural cost gate, trust is via `PROTECTED_JOB_NAMES`) and loads `GBRAIN_PLUGIN_PATH` plugins at worker startup with a loud startup-line per plugin. `shell` handler still gated by `GBRAIN_ALLOW_SHELL_JOBS=1` (RCE surface, separate concern). v0.22.10 (#521): the `autopilot-cycle` handler now forwards `job.data.phases` to `runCycle` (was previously discarded — caller-supplied phase selection silently became a full cycle). Phases are validated against `ALL_PHASES` from `src/core/cycle.ts`; invalid names are filtered out and an empty/missing array falls back to the default 6-phase cycle.
- `src/commands/jobs.ts` — `gbrain jobs` CLI subcommands + `gbrain jobs work` daemon. v0.13.1 surfaces the full `MinionJobInput` retry/backoff/timeout/idempotency surface as first-class CLI flags on `jobs submit`: `--max-stalled`, `--backoff-type fixed|exponential`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key`. `jobs smoke --sigkill-rescue` is the opt-in regression guard for #219. v0.16 wires `registerBuiltinHandlers` to always register `subagent` + `subagent_aggregator` (no env flag — `ANTHROPIC_API_KEY` is the natural cost gate, trust is via `PROTECTED_JOB_NAMES`) and loads `GBRAIN_PLUGIN_PATH` plugins at worker startup with a loud startup-line per plugin. `shell` handler still gated by `GBRAIN_ALLOW_SHELL_JOBS=1` (RCE surface, separate concern). v0.22.10 (#521): the `autopilot-cycle` handler now forwards `job.data.phases` to `runCycle` (was previously discarded — caller-supplied phase selection silently became a full cycle). Phases are validated against `ALL_PHASES` from `src/core/cycle.ts`; invalid names are filtered out and an empty/missing array falls back to the default 6-phase cycle. v0.22.13 (PR #490 CODEX-1+CODEX-4): `sync` handler now resolves `sourceId` at entry by looking up `sources.local_path` (mirrors `cycle.ts:480`'s autopilot fix from PR #475) so multi-source brains read the per-source `last_commit` anchor instead of the global config key. Concurrency routed through the shared `autoConcurrency()` policy in `src/core/sync-concurrency.ts` instead of the prior hardcoded `4`; PGLite stays serial. `noEmbed` default is `true` (embed is a separate job — submit `gbrain embed --stale` after sync, or rely on the autopilot cycle's embed phase).
- `src/commands/features.ts` — `gbrain features --json --auto-fix`: usage scan + feature adoption salesman
- `src/commands/autopilot.ts` — `gbrain autopilot --install`: self-maintaining brain daemon (sync+extract+embed)
- `src/mcp/server.ts` — MCP stdio server (generated from operations). v0.22.7: tool-call handler delegates to `dispatchToolCall` from `src/mcp/dispatch.ts` so stdio + HTTP transports share one validation, context-build, and error-format path.
@@ -185,15 +192,27 @@ strict behavior when unset.
- `src/commands/orphans.ts` — `gbrain orphans [--json] [--count] [--include-pseudo]`: surfaces pages with zero inbound wikilinks, grouped by domain. Auto-generated/raw/pseudo pages filtered by default. Also exposed as `find_orphans` MCP operation. Shipped in v0.12.3 (contributed by @knee5).
- `src/commands/integrity.ts` — `gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). v0.22.8: batch-load fast path on Postgres uses `SELECT DISTINCT ON (slug)` in a single SQL query to fix the PgBouncer round-trip timeout (60s → ~6s) while preserving `engine.getAllSlugs()`'s `Set<string>` semantics on multi-source brains. Gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1` so real Postgres errors are diagnosable.
- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel <id>`. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count.
- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2).
- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). **v30 (v0.23):** creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))`. RLS-enabled when running as a BYPASSRLS role. The synthesize phase reads/writes this table to avoid re-judging on backfill re-runs.
- `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2.
- `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=<ms>` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers.
- `src/core/cycle.ts` v0.17 brain maintenance cycle primitive. `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise<CycleReport>` composes 6 phases in semantically-driven order (lint → backlinks → sync → extract → embed → orphans). Three callers: `gbrain dream` CLI, `gbrain autopilot` daemon's inline path, and the Minions `autopilot-cycle` handler (`src/commands/jobs.ts`). One source of truth for what the brain does overnight. Coordination via `gbrain_cycle_locks` DB table (TTL-based; works through PgBouncer transaction pooling, unlike session-scoped `pg_try_advisory_lock`) + `~/.gbrain/cycle.lock` file lock with PID-liveness for PGLite / engine=null mode. `CycleReport.schema_version: "1"` is the stable agent-consumable shape. `PhaseResult.error: { class, code, message, hint?, docs_url? }` is Stripe-API-tier structured failure info. `yieldBetweenPhases` hook awaited between every phase — Minions handler uses this to renew its job lock and prevent v0.14 stall-death regression. Engine nullable: filesystem phases (lint, backlinks) run without DB; DB phases skip with `status: "skipped", reason: "no_database"`. Lock-skip: read-only phase selections (`--phase orphans`) bypass the cycle lock. v0.22.1 (#403): `CycleOpts.signal?: AbortSignal` propagates the worker's abort signal; `checkAborted()` fires between every phase and throws if the signal is aborted (cooperative — can't interrupt a phase mid-execution). v0.22.1 (#417): `runPhaseSync` returns `pagesAffected` via `SyncPhaseResult`; `runCycle` captures it and threads to `runPhaseExtract` as the 4th arg, enabling incremental extract on the cycle path. v0.22.1 (Codex F2): `runPhaseSync` takes `willRunExtractPhase: boolean` and sets `noExtract: phases.includes('extract')` so `gbrain dream --phase sync` doesn't silently lose extraction. v0.22.5 (#475): new `resolveSourceForDir(engine, brainDir)` helper queries `SELECT id FROM sources WHERE local_path = $1 LIMIT 1`; `runPhaseSync` threads result as `sourceId` to `performSync()` so sync reads the per-source `sources.last_commit` anchor instead of the drift-prone global `config.sync.last_commit` key. Bare try/catch lets pre-v0.18 brains fall through to the global key. Closes the prod hang where every autopilot cycle ran a 30-min full reimport because the global anchor commit had been GC'd from git history.
- `src/commands/dream.ts` v0.17 `gbrain dream` CLI. ~80-line thin alias over `runCycle`. brainDir resolution requires explicit `--dir` OR `sync.repo_path` config (no more walk-up-cwd-for-.git footgun). Flags: `--dry-run`, `--json`, `--phase <name>`, `--pull`, `--dir <path>`. Exit code 1 on status=failed (partial/warn not fatal — don't page on warnings).
- `src/core/db-lock.ts` (v0.22.13) — generic `tryAcquireDbLock(engine, lockId, ttlMinutes)` over the existing `gbrain_cycle_locks` table. Parameterized lock id so different scopes can nest cleanly: `gbrain-cycle` for the broad cycle (held by `cycle.ts`) and `gbrain-sync` (`SYNC_LOCK_ID` constant) for `performSync`'s narrower writer window. Same UPSERT-with-TTL semantics as the prior cycle-only helper, just generalized. Survives PgBouncer transaction pooling (unlike session-scoped `pg_try_advisory_lock`); crashed holders auto-release once their TTL expires.
- `src/core/sync-concurrency.ts` (v0.22.13) — single source of truth for the parallel-sync policy. Exports `autoConcurrency(engine, fileCount, override?)` (PGLite always serial; explicit override clamped to >=1; auto path returns `DEFAULT_PARALLEL_WORKERS=4` when `fileCount > AUTO_CONCURRENCY_FILE_THRESHOLD=100`), `shouldRunParallel(workers, fileCount, explicit)` (Q1: explicit `--workers` bypasses the >50-file floor), and `parseWorkers(s)` (rejects `'0'`, `'-3'`, `'foo'`, `'1.5'`, trailing chars — replaces the prior parseInt-with-no-validation in both `sync.ts` and `import.ts`). Used by `performSync`, `performFullSync`, `runImport`, and the Minion `sync` handler so the three sites can no longer drift.
- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). v0.22.13 (PR #490): `performSync` wraps its body in a `gbrain-sync` writer lock so two concurrent syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot both write `last_commit` and let the last writer win. Head-drift gate after the import phase re-checks `git rev-parse HEAD`; if HEAD moved (someone ran `git checkout` / `git pull` mid-sync), the bookmark refuses to advance. Vanished files now record a failedFiles entry instead of silent-skip — the silent-skip-then-advance pathology that survived prior hardening passes is dead. Worker engines wrap in try/finally so disconnect always fires (panic-path leak fix). Both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`), validated via `parseWorkers`. Explicit `--workers` bypasses the auto-path file-count floor; auto path defers to `autoConcurrency()`. Banner moved to stderr.
- `src/core/cycle.ts` — v0.17 brain maintenance cycle primitive (extended to **8 phases in v0.23**). `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise<CycleReport>` composes phases in semantically-driven order: **lint → backlinks → sync → synthesize → extract → patterns → embed → orphans**. v0.23's `synthesize` phase runs after sync (cross-references see fresh brain) and before extract (auto-link materializes its writes); `patterns` runs after extract so it reads a fresh graph (codex finding #7 — subagent put_page sets `ctx.remote=true` and skips auto-link/timeline by default; extract is the canonical materialization). Three callers: `gbrain dream` CLI, `gbrain autopilot` daemon's inline path, and the Minions `autopilot-cycle` handler. Coordination via `gbrain_cycle_locks` DB table + `~/.gbrain/cycle.lock` file lock with PID-liveness for PGLite. `CycleReport.schema_version: "1"` is stable; totals additively grew in v0.23 (`transcripts_processed`, `synth_pages_written`, `patterns_written`). `yieldBetweenPhases` runs between phases. **v0.23 added `yieldDuringPhase`** for in-phase keepalive — synthesize/patterns call it during long waits to renew the cycle-lock TTL. Engine nullable; lock-skip on read-only phase selections. v0.22.1 (#403): `CycleOpts.signal?: AbortSignal` propagates the worker's abort signal; `checkAborted()` fires between every phase. v0.22.1 (#417): `runPhaseSync` returns `pagesAffected` via `SyncPhaseResult`; `runCycle` captures it and threads to `runPhaseExtract` as the 4th arg. v0.22.1 (Codex F2): `runPhaseSync` takes `willRunExtractPhase: boolean` and sets `noExtract: phases.includes('extract')` so `gbrain dream --phase sync` doesn't silently lose extraction. v0.22.5 (#475): `resolveSourceForDir(engine, brainDir)` threads `sourceId` to `performSync()` so sync reads the per-source `sources.last_commit` anchor instead of the drift-prone global `config.sync.last_commit` key.
- `src/core/cycle/synthesize.ts` (v0.23) — Synthesize phase: conversation-transcript-to-brain pipeline. Reads from `dream.synthesize.session_corpus_dir`, runs cheap Haiku verdict (cached in `dream_verdicts`), then fans out one Sonnet subagent per worth-processing transcript with `allowed_slug_prefixes` (sourced from `skills/_brain-filing-rules.json` `dream_synthesize_paths.globs`). Orchestrator collects slugs from `subagent_tool_executions` (NOT `pages.updated_at` — codex finding #2) and reverse-renders DB → markdown via `serializeMarkdown`. Cooldown via `dream.synthesize.last_completion_ts`, written ONLY on success. Idempotency key `dream:synth:<file_path>:<content_hash>`. Auto-commit deferred to v1.1 (codex #5). `--dry-run` runs Haiku, skips Sonnet (codex #8). Subagent never gets fs-write access. **v0.23.2:** `renderPageToMarkdown` (now exported) stamps `dream_generated: true` and `dream_cycle_date` into every reverse-write's frontmatter; `writeSummaryPage` does the same on the dream-cycle summary index. The marker is the explicit identity surface checked by `isDreamOutput` in `transcript-discovery.ts` — replaces the v0.23.1 content-prefix heuristic that could miss real output (`serializeMarkdown` doesn't embed slugs in body) and false-positive on user transcripts citing brain pages. `judgeSignificance` and `JudgeClient` are exported; `judgeSignificance` accepts a `verdictModel` parameter (default `claude-haiku-4-5-20251001`) loaded from `dream.synthesize.verdict_model` via `loadSynthConfig`.
- `src/core/cycle/patterns.ts` (v0.23) — Patterns phase: cross-session theme detection over reflections within `dream.patterns.lookback_days` (default 30). Names a pattern only when ≥`dream.patterns.min_evidence` (default 3) reflections support it. Single Sonnet subagent; same allow-list path as synthesize. Runs AFTER `extract` so the graph is fresh.
- `src/core/cycle/transcript-discovery.ts` (v0.23) — Pure filesystem walk for synthesize. `discoverTranscripts(opts)` filters `.txt` files by date range, min_chars, and word-boundary regex `excludePatterns` (Q-3: `medical` matches "medical advice" but NOT "comedical"; power users may pass full regex). `readSingleTranscript(path)` is the `gbrain dream --input <file>` ad-hoc path. **v0.23.2 self-consumption guard:** `DREAM_OUTPUT_MARKER_RE` (anchored at frontmatter open `---\n`, optional BOM + CRLF tolerance, scans first 2000 chars for `dream_generated: true` with case-insensitive value and word boundary on `true`) drives `isDreamOutput(content, bypass=false)`. Both `discoverTranscripts` and `readSingleTranscript` skip matching files and emit a `[dream] skipped <basename>: dream_generated marker` stderr log (no more silent skips). `bypassGuard?: boolean` on `DiscoverOpts` and `readSingleTranscript`'s opts disables the guard for the explicit `--unsafe-bypass-dream-guard` escape hatch only — never auto-applied for `--input`. Replaces v0.23.1's `DREAM_OUTPUT_SLUGS` content-prefix list.
- `src/commands/dream.ts` — v0.17 `gbrain dream` CLI; ~80-line thin alias over `runCycle`. brainDir resolution requires explicit `--dir` OR `sync.repo_path` config. Flags: `--dry-run`, `--json`, `--phase <name>`, `--pull`, `--dir <path>`. **v0.23 added** `--input <file>` (ad-hoc transcript, implies `--phase synthesize`), `--date YYYY-MM-DD`, `--from <d> --to <d>` (backfill range). Conflict detection: `--input` + `--date` exits 2. ISO date validation. `--dry-run` runs Haiku significance verdict but skips Sonnet synthesis (codex finding #8 — NOT zero LLM calls). Exit code 1 on status=failed. **v0.23.2 added** `--unsafe-bypass-dream-guard` (long-form intentional, plumbed through `runCycle.synthBypassDreamGuard` → `SynthesizePhaseOpts.bypassDreamGuard` → `discoverTranscripts({bypassGuard})` and `readSingleTranscript({bypassGuard})`). Loud stderr warning fires at synthesize-phase entry when set. Never auto-applied for `--input` so any caller can't silently re-trigger the loop bug.
- `src/commands/friction.ts` + `src/core/friction.ts` (v0.23) — `gbrain friction {log,render,list,summary}` reporter. Append-only JSONL under `$GBRAIN_HOME/friction/<run-id>.jsonl`. Schema is a flat extension of `StructuredAgentError` (D20). Render groups by severity → phase, defaults to `--redact` for md output (strips `$HOME`/`$CWD` to placeholders so reports paste safely in PRs). Run-id resolves from `--run-id` > `$GBRAIN_FRICTION_RUN_ID` > `standalone.jsonl`. Skills the claw-test exercises gain a `_friction-protocol.md` callout so agents know when to log friction.
- `src/commands/claw-test.ts` + `src/core/claw-test/` (v0.23) — `gbrain claw-test [--scenario <name>] [--live --agent openclaw]`. End-to-end "fresh user" friction harness. Two modes: scripted (CI gate, agent-free) and live (real openclaw subprocess, $12 in tokens). Sets `GBRAIN_HOME=<tempdir>` for hermeticity and captures gbrain's `--progress-json` events from each child's stderr to verify expected phases ran (`import.files`, `extract.links_fs`, `doctor.db_checks`). Phases for scripted mode: setup → install_brain (`gbrain init --pglite`) → import (`--no-embed`) → query → extract → verify (`gbrain doctor --json`, asserts `status: 'ok'`) → render. Live mode hands `BRIEF.md` from `test/fixtures/claw-test-scenarios/<name>/` to the agent runner. v1 ships with the OpenClaw runner only (`src/core/claw-test/runners/openclaw.ts`, invokes `openclaw agent --local --agent <name> --message <brief>`); hermes runner deferred to v1.1. Transcript capture (`transcript-capture.ts`) uses `fs.createWriteStream` with `'drain'`-event backpressure — D17 fix for the 256KB-burst child-stall scenario. v0.18 upgrade scenario seeded via `seed-pglite.ts` SQL replay.
- `skills/_friction-protocol.md` (v0.23) — shared cross-cutting convention skill (like `_brain-filing-rules.md`). Tells agents when to call `gbrain friction log` and how to choose a severity. Routes to friction CLI from any skill the claw-test exercises.
- `scripts/check-progress-to-stdout.sh` — CI guard against regressing to `\r`-on-stdout progress. Wired into `bun run test` via `scripts/check-progress-to-stdout.sh && bun test` in package.json.
- `docs/progress-events.md` — Canonical JSON event schema reference. Stable from v0.15.2, additive only.
- `src/core/markdown.ts` — Frontmatter parsing + body splitter. `splitBody` requires an explicit timeline sentinel (`<!-- timeline -->`, `--- timeline ---`, or `---` immediately before `## Timeline`/`## History`). Plain `---` in body text is a markdown horizontal rule, not a separator. `inferType` auto-types `/wiki/analysis/` → analysis, `/wiki/guides/` → guide, `/wiki/hardware/` → hardware, `/wiki/architecture/` → architecture, `/writing/` → writing (plus the existing people/companies/deals/etc heuristics).
- `scripts/check-jsonb-pattern.sh` — CI grep guard. Fails the build if anyone reintroduces (a) the `${JSON.stringify(x)}::jsonb` interpolation pattern (postgres.js v3 double-encodes it), or (b) `max_stalled INTEGER NOT NULL DEFAULT 1` in any schema source file (v0.15.1 #219 regression guard — must be DEFAULT 5 to preserve SIGKILL-rescue). Wired into `bun test`.
- `docker-compose.ci.yml` + `scripts/ci-local.sh` (v0.23.1) — Local CI gate. `bun run ci:local` spins up `pgvector/pgvector:pg16` + `oven/bun:1` with named volumes (`gbrain-ci-pg-data`, `gbrain-ci-node-modules`, `gbrain-ci-bun-cache`), runs gitleaks on host, smoke-tests `scripts/run-e2e.sh` argv handling, runs unit tests with `DATABASE_URL` unset (matches GH Actions structure), then runs all 29 E2E files sequentially. `--diff` swaps in the diff-aware selector; `--no-pull` skips upstream pulls; `--clean` nukes named volumes. Postgres host port defaults to 5434 (avoids 5432 manual `gbrain-test-pg` and 5433 sibling-project conflict); override with `GBRAIN_CI_PG_PORT=NNNN`. Stronger gate than current PR CI's 2-file Tier 1 set — closes the "push-and-wait" feedback loop pre-push.
- `scripts/select-e2e.ts` + `scripts/e2e-test-map.ts` (v0.23.1) — Diff-aware E2E test selector. Reads three git sources (committed `origin/master...HEAD`, working-tree `HEAD`, and `git ls-files --others --exclude-standard` for untracked, NOT-gitignored files), classifies as EMPTY / DOC_ONLY / SRC. Fail-closed by design: EMPTY → all 29 files (clean branch shouldn't run nothing), DOC_ONLY (every path matches the README/CLAUDE/AGENTS/CHANGELOG/TODOS allowlist) → empty stdout, SRC → escape-hatch paths (schema, package.json, skills/) trigger all; otherwise the hand-tuned `E2E_TEST_MAP` glob → tests narrows; an unmapped src/ change still emits ALL files, never silently nothing. Pure-function exports (`selectTests`, `classify`, `matchGlob`) so it's trivial to test and fork. `bun run ci:select-e2e` prints the current selection on stdout, pipe-friendly. `test/select-e2e.test.ts` covers all 4 branches plus 3 codex regression guards (skills/, untracked files, unmapped src/) — 24 cases.
- `scripts/run-e2e.sh` (v0.23.1 update) — Sequential E2E runner. Now accepts an optional argv-driven file list (used by `ci:local:diff` to pipe in selector output) and a `--dry-run-list` flag that prints the resolved file list and exits (used by `ci-local.sh`'s startup smoke-test). Falls back to `test/e2e/*.test.ts` when invoked with no args.
- `scripts/llms-config.ts` + `scripts/build-llms.ts` — Generator for `llms.txt` (llmstxt.org-spec web index) + `llms-full.txt` (inlined single-fetch bundle). Curated config drives both. Run `bun run build:llms` after adding a new doc. `LLMS_REPO_BASE` env var lets forks regenerate with their own URL base. `FULL_SIZE_BUDGET` (600KB) caps the inline bundle; generator WARNs if exceeded. Committed output is not analogous to `schema-embedded.ts` (no runtime consumer); we commit for GitHub browsing and fork-safe fetching.
- `AGENTS.md` — Local-clone entry point for non-Claude agents (Codex, Cursor, OpenClaw, Aider). Mirrors `CLAUDE.md` intent via relative links. Claude Code keeps using `CLAUDE.md`.
- `docs/UPGRADING_DOWNSTREAM_AGENTS.md` — Patches for downstream agent skill forks to apply when upgrading. Each release appends a new section. v0.10.3 includes diffs for brain-ops, meeting-ingestion, signal-detector, enrich.
@@ -299,6 +318,20 @@ Key commands added in v0.14.3 (fix wave):
- `gbrain jobs submit` gains `--max-stalled`, `--backoff-type`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key` — exposing existing `MinionJobInput` fields as first-class CLI flags.
- `gbrain jobs smoke --sigkill-rescue` — opt-in regression smoke case simulating a killed worker; asserts the v0.14.3 schema default (`max_stalled=5`) actually rescues on first stall.
Key commands added in v0.22.13 (PR #490):
- `gbrain sync --workers N` (alias `--concurrency N`) — parallelize the import phase using per-worker Postgres engines (small pool of 2 each) with an atomic queue index. Auto-concurrency: defaults to 4 workers when the diff exceeds 100 files. Smaller diffs stay serial. Explicit `--workers` always wins (even on a 30-file diff). PGLite forces serial regardless. Validation rejects `0`, negatives, non-integers loud (replaces the prior silent fall-through to auto-concurrency).
- `gbrain import --workers N` — same `parseWorkers()` validation as sync; same try/finally worker-engine cleanup. Behavior surface unchanged.
Key commands added in v0.22.16 (claw-test friction loop):
- `gbrain claw-test [--scenario fresh-install|upgrade-from-v0.18] [--keep-tempdir]` — scripted-mode CI gate that runs the full canonical first-day flow against a fresh tempdir. Asserts every expected `--progress-json` phase fired and doctor's `status === 'ok'`. ~30s, no API keys.
- `gbrain claw-test --live --agent openclaw` — friction-discovery mode. Spawns real openclaw, hands it `BRIEF.md`, captures stdin/stdout/stderr to `<run>/transcript.jsonl`, lets the agent log friction via the friction CLI. Run on demand; ~510 min and ~$12 in tokens.
- `gbrain claw-test --list-agents` — reports which agent runners are registered + their detection state (binary path or unavailable reason).
- `gbrain friction log --severity {confused|error|blocker|nit} --phase <name> --message <text> [--hint ...] [--kind {friction|delight}] [--run-id ...]` — append a friction or delight entry to the active run JSONL.
- `gbrain friction render --run-id <id> [--json] [--transcripts] [--no-redact]` — markdown report grouped by severity + phase; `--redact` is the default for md output (strips `$HOME`/`$CWD` placeholders so reports paste safely in PRs/issues).
- `gbrain friction list [--json]` — recent run-ids with friction/delight counts; interrupted runs marked `(interrupted)`.
- `gbrain friction summary --run-id <id> [--json]` — two-column friction + delight summary.
- `GBRAIN_HOME` env override is now honored uniformly across every gbrain write site (config, audit, friction, sync-failures, import checkpoint, integrity log, integrations heartbeat, migration rollback, etc.) — `gbrainPath(...)` from `src/core/config.ts` is the canonical helper. Read-side host-fingerprint detection (`~/.claude`/`~/.openclaw` etc.) intentionally NOT confined in v1; that's a v1.1 follow-up.
## Testing
`bun test` runs all tests. After the v0.12.1 release: ~75 unit test files + 8 E2E test files (1412 unit pass, 119 E2E when `DATABASE_URL` is set — skip gracefully otherwise). Unit tests run
@@ -354,6 +387,8 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac
`test/orphans.test.ts` (v0.12.3 orphans command: detection, pseudo filtering, text/json/count outputs, MCP op),
`test/postgres-engine.test.ts` (v0.12.3 statement_timeout scoping: `sql.begin` + `SET LOCAL` shape, source-level grep guardrail against reintroduced bare `SET statement_timeout`),
`test/sync.test.ts` (sync logic + v0.12.3 regression guard asserting top-level `engine.transaction` is not called),
`test/sync-concurrency.test.ts` (v0.22.13 PR #490: 17 cases covering `autoConcurrency()` thresholds + PGLite-forces-serial + explicit-override clamping, `shouldRunParallel()` Q1 explicit-bypasses-floor contract, and `parseWorkers()` validation that rejects `'0'`/`'-3'`/`'foo'`/`'1.5'`/trailing chars),
`test/sync-parallel.test.ts` (v0.22.13 PR #490: PGLite-routed coverage of the bookmark gate under concurrency request, head-drift gate, vanished-file failure capture, PGLite-stays-serial, and the `gbrain-sync` writer-lock contract — 7 cases),
`test/sync-failures.test.ts` (v0.22.12: 28 cases pinning `classifyErrorCode` regex coverage for all 12 codes against literal production message strings from `markdown.ts:159-244` and `import-file.ts:199, 347, 352, 401`; `summarizeFailuresByCode` sort + pre-classified-honor; `recordSyncFailures` code-field persistence; `acknowledgeSyncFailures` AcknowledgeResult shape + backfill on pre-v0.22.12 entries),
`test/doctor.test.ts` (doctor command + v0.12.3 assertions that `jsonb_integrity` scans the four v0.12.0 write sites and `markdown_body_completeness` is present),
`test/utils.test.ts` (shared SQL utilities + `tryParseEmbedding` null-return and single-warn semantics),
@@ -384,6 +419,7 @@ E2E tests (`test/e2e/`): Run against real Postgres+pgvector. Require `DATABASE_U
- `test/e2e/engine-parity.test.ts` (v0.22.0) — Postgres ↔ PGLite top-result and result-set parity for `searchKeyword` + `searchVector`. Codex flagged that Postgres ranks pages then picks best chunk while PGLite returns chunks directly — without parity coverage the source-boost fix could pass on PGLite and fail on Postgres. Skips gracefully when `DATABASE_URL` is unset.
- `test/e2e/postgres-bootstrap.test.ts` (v0.22.6.1) — exercises `PostgresEngine.initSchema()` directly against a fresh real Postgres database. Asserts the bootstrap path is no-op on fresh installs and that SCHEMA_SQL replays cleanly through the engine path (not via the standalone `db.initSchema` from `src/core/db.ts`, which would have produced false-positive coverage). Codex caught the E2E-shape gap during plan review.
- `test/e2e/http-transport.test.ts` (v0.22.7) — 8 cases against real Postgres covering `gbrain serve --http` end-to-end: bearer auth round-trip, `last_used_at` SQL-level debounce semantics, `mcp_request_log` row insertion on success and auth_failed paths, `/health` DB-down → 503 (DB-probing health check), and the F1+F2+F3 dispatch round-trip with a real operation. Skips gracefully when `DATABASE_URL` is unset.
- `test/e2e/sync-parallel.test.ts` (v0.22.13 PR #490) — DATABASE_URL-gated. T2: 60-file Postgres sync at concurrency=4 imports all + no connection leak (probes `pg_stat_activity` before/after to confirm worker engines disconnected). P4: 120-file serial-vs-parallel benchmark prints `SYNC_PARALLEL_BENCH N files | serial=Xms | parallel(4)=Yms | speedup=Zx` for CHANGELOG quoting. Asserts parallel ≤ serial × 1.5 (CI-noise tolerant; not a strict speedup gate).
- Tier 2 (`skills.test.ts`) requires OpenClaw + API keys, runs nightly in CI
- If `.env.testing` doesn't exist in this directory, check sibling worktrees for one:
`find ../ -maxdepth 2 -name .env.testing -print -quit` and copy it here if found.
@@ -553,13 +589,45 @@ will detect drift and re-bump on the next run.
## Pre-ship requirements
Before shipping (/ship) or reviewing (/review), always run the full test suite:
Before shipping (/ship) or reviewing (/review), always run the full test suite.
Two equivalent paths:
**Path A — local CI gate (recommended, v0.23.1+):**
- `bun run ci:local` runs the entire stack inside Docker: gitleaks (host), unit
tests with `DATABASE_URL` unset, and all 29 E2E files sequentially against a
fresh pgvector container. Stronger than PR CI's 2-file Tier 1 set; closer to
what nightly Tier 1 catches. Spins up + tears down postgres automatically via
`docker-compose.ci.yml`. Override the host port with
`GBRAIN_CI_PG_PORT=5435 bun run ci:local` if 5434 collides.
- `bun run ci:local:diff` runs only the E2E files matched by the diff selector
(`scripts/select-e2e.ts`), falling back to all 29 on unmapped src/ paths or
schema/skills/package.json changes. Fast iteration during a focused branch.
**Path B — manual lifecycle (still supported):**
- `bun test` — unit tests (no database required)
- Follow the "E2E test DB lifecycle" steps above to spin up the test DB,
run `bun run test:e2e`, then tear it down.
Both must pass. Do not ship with failing E2E tests. Do not skip E2E tests.
**Always run typecheck before pushing.** `bun test` (the bun runner)
skips TypeScript type checking — it only enforces runtime behavior.
Three ways to actually gate on types:
1. `bun run test` (npm script in `package.json`) — includes `bun run typecheck`
plus the four shell pre-checks (`check-jsonb-pattern.sh`,
`check-progress-to-stdout.sh`, `check-trailing-newline.sh`,
`check-wasm-embedded.sh`) before the runner. Use this mid-branch.
2. `bun run typecheck` — `tsc --noEmit` standalone. Fast (~5s on this repo).
3. `bun run ci:local` — the full local CI gate from Path A.
The trap is: writing a new test, running `bun test test/foo.test.ts`,
seeing it pass, pushing — and CI's separate typecheck stage rejects an
invalid type literal that the runner accepted. Caught one of these
shipping the v0.23.2 round-trip E2E (`type: 'reflection'` is not a
member of `PageType`). Run `bun run typecheck` once before push, even
when only test files changed.
## Post-ship requirements (MANDATORY)
After EVERY /ship, you MUST run /document-release. This is NOT optional. Do NOT
@@ -1125,8 +1193,9 @@ Set up using your platform's scheduler (OpenClaw cron, Railway cron, crontab):
- **Live sync** (every 15 min): `gbrain sync --repo ~/brain && gbrain embed --stale`
- **Auto-update** (daily): `gbrain check-update --json` (tell user, never auto-install)
- **Dream cycle** (nightly): read `docs/guides/cron-schedule.md` for the full protocol.
Entity sweep, citation fixes, memory consolidation. This is what makes the brain
compound. Do not skip it.
Entity sweep, citation fixes, memory consolidation, plus (v0.23+) overnight conversation
synthesis and cross-session pattern detection. 8 phases, one cron-friendly command. This
is what makes the brain compound. Do not skip it.
- **Weekly**: `gbrain doctor --json && gbrain embed --stale`
## Step 8: Integrations
@@ -1243,6 +1312,7 @@ This is the dispatcher. Skills are the implementation. **Read the skill file bef
| "Migrate from Obsidian/Notion/Logseq" | `skills/migrate/SKILL.md` |
| Brain health check, maintenance run | `skills/maintain/SKILL.md` |
| "Extract links", "build link graph", "populate timeline" | `skills/maintain/SKILL.md` (extraction sections) |
| "Run dream", "process today's session", "synthesize my conversations", "consolidate yesterday's conversations", "what patterns did you see", "did the dream cycle run" | `skills/maintain/SKILL.md` (dream cycle section) |
| "Brain health", "what features am I missing", "brain score" | Run `gbrain features --json` |
| "Set up autopilot", "run brain maintenance", "keep brain updated" | Run `gbrain autopilot --install --repo ~/brain` |
| Agent identity, "who am I", customize agent | `skills/soul-audit/SKILL.md` |
@@ -1416,7 +1486,7 @@ GBrain ships 29 skills organized by `skills/RESOLVER.md` (or your OpenClaw's `AG
|-------|-------------|
| **enrich** | Tiered enrichment (Tier 1/2/3). Creates and updates person/company pages with compiled truth and timelines. |
| **query** | 3-layer search with synthesis and citations. Says "the brain doesn't have info on X" instead of hallucinating. |
| **maintain** | Periodic health: stale pages, orphans, dead links, citation audit, back-link enforcement, tag consistency. |
| **maintain** | Periodic health: stale pages, orphans, dead links, citation audit, back-link enforcement, tag consistency. v0.23 adds the dream cycle's synthesize + patterns phases ... overnight conversation transcripts become reflections, originals, and 25-year patterns. |
| **citation-fixer** | Scans pages for missing or malformed citations. Fixes format to match the standard. |
| **repo-architecture** | Where new brain files go. Decision protocol: primary subject determines directory, not format. |
| **publish** | Share brain pages as password-protected HTML. Zero LLM calls. |
@@ -1600,9 +1670,11 @@ is what you spend time on. Everything else is boilerplate the CLI writes for you
Drop a `routing-eval.jsonl` fixture next to any skill. Each line is `{intent, expected_skill,
ambiguous_with?}`. `gbrain check-resolvable` runs the structural layer by default; `gbrain
routing-eval --llm` runs an LLM tie-break layer for CI. False positives (wrong skill matched),
missed routes (no skill matched), and tautological fixtures (intent copies trigger verbatim)
all surface as specific advisories with the exact file:line to fix.
routing-eval` runs the same structural layer as a dedicated CI verb. The `--llm` flag is
accepted as a placeholder for a future LLM tie-break layer; in this release it emits a stderr
notice and runs structural only. False positives (wrong skill matched), missed routes (no
skill matched), and tautological fixtures (intent copies trigger verbatim) all surface as
specific advisories with the exact file:line to fix.
### Works on your OpenClaw, not just gbrain's repo
@@ -1639,6 +1711,10 @@ gbrain skillpack diff brain-ops # compare bundle vs your local co
Re-running is safe. The managed-block markers in your AGENTS.md let `skillpack install`
accumulate rows across separate single-skill installs instead of overwriting each other.
A receipt comment inside the fence (`<!-- gbrain:skillpack:manifest cumulative-slugs="..." -->`)
tracks what gbrain has installed across runs. `install --all` is the only path that prunes;
per-skill install never deletes what it didn't install. If you hand-add a row inside the fence,
gbrain preserves it on reinstall and emits a stderr notice telling your agent to investigate.
**Skillify is the piece that makes the skills tree survive six months of compounding work.**
Read [`skills/skillify/SKILL.md`](skills/skillify/SKILL.md) for the full 10-item checklist
@@ -1923,8 +1999,11 @@ SEARCH
gbrain query <question> Hybrid search (vector + keyword + RRF)
IMPORT
gbrain import <dir> [--no-embed] Import markdown (idempotent)
gbrain sync [--repo <path>] Git-to-brain incremental sync
gbrain import <dir> [--no-embed] [--workers N]
Import markdown (idempotent)
gbrain sync [--repo <path>] [--workers N]
Git-to-brain incremental sync
(>100-file diffs auto-parallelize 4 workers on Postgres)
gbrain export [--dir ./out/] Export to markdown
FILES
@@ -1970,7 +2049,11 @@ ADMIN
gbrain auth create|list|revoke|test Token management for the HTTP transport
gbrain integrations Integration recipe dashboard
gbrain sources list|add|remove|... Multi-source brain management (v0.18)
gbrain dream [--dry-run] [--phase N] One maintenance cycle then exit (cron-friendly)
gbrain dream [--dry-run] [--phase N] 8-phase maintenance cycle (lint→backlinks→sync→synthesize
→extract→patterns→embed→orphans). v0.23 added synthesize +
patterns: transcripts → reflections + cross-session themes.
gbrain dream --input <file> Ad-hoc transcript synthesis (implies --phase synthesize)
gbrain dream --date YYYY-MM-DD Synthesize a single day; --from/--to for backfill ranges
gbrain check-backlinks check|fix Back-link enforcement
gbrain lint [--fix] LLM artifact detection
gbrain repair-jsonb [--dry-run] Repair v0.12.0 double-encoded JSONB (Postgres)
@@ -2013,7 +2096,7 @@ The skills in this repo are those patterns, generalized. What took 11 days to bu
## Contributing
See [CONTRIBUTING.md](CONTRIBUTING.md). Run `bun test` for unit tests. E2E tests: spin up Postgres with pgvector, run `bun run test:e2e`, tear down.
See [CONTRIBUTING.md](CONTRIBUTING.md). Run `bun test` for unit tests. For the full local CI gate (gitleaks + unit + all 29 E2E files in Docker, the same checks GH Actions runs), use `bun run ci:local` ... or `bun run ci:local:diff` for the diff-aware subset during fast iteration.
PRs welcome for: new enrichment APIs, performance optimizations, additional engine backends, new skills following the conformance standard in `skills/skill-creator/SKILL.md`.
+10 -2
View File
@@ -1,6 +1,6 @@
{
"name": "gbrain",
"version": "0.22.12",
"version": "0.24.0",
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
"type": "module",
"main": "src/core/index.ts",
@@ -32,12 +32,19 @@
"build:all": "bun build --compile --target=bun-darwin-arm64 --outfile bin/gbrain-darwin-arm64 src/cli.ts && bun build --compile --target=bun-linux-x64 --outfile bin/gbrain-linux-x64 src/cli.ts",
"build:schema": "bash scripts/build-schema.sh",
"build:llms": "bun run scripts/build-llms.ts",
"test": "scripts/check-jsonb-pattern.sh && scripts/check-progress-to-stdout.sh && scripts/check-trailing-newline.sh && scripts/check-wasm-embedded.sh && bun run typecheck && bun test --timeout=60000",
"build:pglite-snapshot": "bun run scripts/build-pglite-snapshot.ts",
"test": "scripts/check-privacy.sh && scripts/check-jsonb-pattern.sh && scripts/check-progress-to-stdout.sh && scripts/check-trailing-newline.sh && scripts/check-wasm-embedded.sh && bun run typecheck && bun test --timeout=60000",
"check:wasm": "scripts/check-wasm-embedded.sh",
"check:newlines": "scripts/check-trailing-newline.sh",
"test:e2e": "bash scripts/run-e2e.sh",
"test:slow": "bash scripts/run-slow-tests.sh",
"test:profile": "bash scripts/profile-tests.sh",
"ci:local": "bash scripts/ci-local.sh",
"ci:local:diff": "bash scripts/ci-local.sh --diff",
"ci:select-e2e": "bun run scripts/select-e2e.ts",
"typecheck": "tsc --noEmit",
"check:jsonb": "scripts/check-jsonb-pattern.sh",
"check:privacy": "scripts/check-privacy.sh",
"check:progress": "scripts/check-progress-to-stdout.sh",
"postinstall": "command -v gbrain >/dev/null 2>&1 && gbrain apply-migrations --yes --non-interactive || echo '[gbrain] postinstall skipped. If installed via bun install -g github:...: run `gbrain doctor` and `gbrain apply-migrations --yes` manually. See https://github.com/garrytan/gbrain/issues/218' 1>&2",
"prepublish:clawhub": "bun run build:all",
@@ -64,6 +71,7 @@
},
"devDependencies": {
"@types/bun": "latest",
"bun-types": "^1.3.13",
"typescript": "^5.6.0"
},
"trustedDependencies": [
+64
View File
@@ -0,0 +1,64 @@
#!/usr/bin/env bun
// scripts/build-pglite-snapshot.ts
//
// Tier 3 fast-restore: boot a fresh PGLite, run the full initSchema (forward
// bootstrap + PGLITE_SCHEMA_SQL + every migration), dump the post-init state
// to a tar fixture. Test files that read GBRAIN_PGLITE_SNAPSHOT can skip the
// 1-3 seconds of cold init and load the post-schema state directly.
//
// Output: test/fixtures/pglite-snapshot.tar (binary, gitignored)
// test/fixtures/pglite-snapshot.version (hex SHA256 of MIGRATIONS SQL)
//
// The version file lets the engine detect snapshot staleness — if the tar's
// recorded version doesn't match the current MIGRATIONS hash, the engine
// ignores the snapshot and runs a normal initSchema.
//
// Run: bun run scripts/build-pglite-snapshot.ts
// (or: bun run build:pglite-snapshot)
//
// Re-run whenever you touch src/core/migrate.ts or src/schema.sql.
import { writeFileSync, mkdirSync } from "node:fs";
import { dirname } from "node:path";
import * as crypto from "node:crypto";
import { PGLiteEngine, computeSnapshotSchemaHash } from "../src/core/pglite-engine.ts";
import { MIGRATIONS } from "../src/core/migrate.ts";
import { PGLITE_SCHEMA_SQL } from "../src/core/pglite-schema.ts";
function computeSchemaHash(): string {
return computeSnapshotSchemaHash(MIGRATIONS, PGLITE_SCHEMA_SQL, crypto);
}
async function main() {
const fixturePath = "test/fixtures/pglite-snapshot.tar";
const versionPath = "test/fixtures/pglite-snapshot.version";
mkdirSync(dirname(fixturePath), { recursive: true });
const schemaHash = computeSchemaHash();
console.log(`[build-pglite-snapshot] schema hash: ${schemaHash.slice(0, 16)}...`);
console.log(`[build-pglite-snapshot] booting PGLite (in-memory)...`);
const engine = new PGLiteEngine();
// Bypass the env-aware short-circuit: we WANT a real init here.
delete process.env.GBRAIN_PGLITE_SNAPSHOT;
await engine.connect({});
console.log(`[build-pglite-snapshot] running initSchema (forward bootstrap + ${MIGRATIONS.length} migrations)...`);
const t0 = Date.now();
await engine.initSchema();
console.log(`[build-pglite-snapshot] initSchema completed in ${Date.now() - t0}ms`);
console.log(`[build-pglite-snapshot] dumping data dir...`);
const dump = await engine.db.dumpDataDir("none");
const buffer = Buffer.from(await dump.arrayBuffer());
writeFileSync(fixturePath, buffer);
writeFileSync(versionPath, schemaHash + "\n");
await engine.disconnect();
console.log(`[build-pglite-snapshot] wrote ${fixturePath} (${buffer.length} bytes)`);
console.log(`[build-pglite-snapshot] wrote ${versionPath}`);
}
await main();
+346
View File
@@ -0,0 +1,346 @@
#!/usr/bin/env bash
# scripts/ci-local.sh
#
# Local CI gate. Runs the same checks GH Actions does (and a stricter superset
# of E2E) inside Docker. See docker-compose.ci.yml.
#
# Modes:
# bash scripts/ci-local.sh # full local gate: gitleaks + unit + ALL E2E (4-way sharded)
# bash scripts/ci-local.sh --diff # full local gate: gitleaks + unit + selected E2E (4-way sharded)
# bash scripts/ci-local.sh --no-pull # skip docker compose pull (offline / debug)
# bash scripts/ci-local.sh --clean # nuke named volumes for cold debug
# bash scripts/ci-local.sh --no-shard # debug: run E2E sequentially against postgres-1 only
#
# 4-way E2E sharding: 4 pgvector services on host ports 5434-5437. The 36 E2E
# files split N/4 per shard; shards run in parallel. Within a shard, files run
# sequentially (TRUNCATE CASCADE no-race property documented in run-e2e.sh).
# Wall-time on a 16-core host: ~6 min sequential -> ~1.5-2 min sharded.
#
# Stronger than PR CI: PR CI runs only Tier 1's 2 files; this runs all 36.
set -euo pipefail
cd "$(dirname "$0")/.."
COMPOSE_FILE="docker-compose.ci.yml"
DIFF=0
NO_PULL=0
CLEAN=0
NO_SHARD=0
for arg in "$@"; do
case "$arg" in
--diff) DIFF=1 ;;
--no-pull) NO_PULL=1 ;;
--clean) CLEAN=1 ;;
--no-shard) NO_SHARD=1 ;;
*)
echo "Usage: $0 [--diff] [--no-pull] [--clean] [--no-shard]" >&2
exit 1
;;
esac
done
cleanup() {
echo ""
echo "[ci-local] Tearing down postgres..."
docker compose -f "$COMPOSE_FILE" down --remove-orphans 2>&1 | tail -5 || true
}
trap cleanup EXIT
if [ "$CLEAN" = "1" ]; then
echo "[ci-local] --clean: removing named volumes..."
docker compose -f "$COMPOSE_FILE" down -v --remove-orphans 2>&1 | tail -5 || true
fi
# Tier 2: --diff fast-path. If the diff is doc-only (or empty), skip the
# whole heavy gate (postgres + bun install + unit + E2E) and just verify
# gitleaks on host. Doc-only diffs go from ~25 min to ~5 seconds.
if [ "$DIFF" = "1" ]; then
CLASSIFICATION=$(bun run scripts/select-e2e.ts --classify-only 2>/dev/null || echo "ERR")
case "$CLASSIFICATION" in
DOC_ONLY)
echo "[ci-local] --diff: diff is doc-only — skipping postgres + unit + E2E (Tier 2 fast-path)."
echo "[ci-local] Running gitleaks on host as the only gate..."
if ! command -v gitleaks >/dev/null 2>&1; then
echo "[ci-local] WARN: gitleaks not installed; skipping. brew install gitleaks." >&2
else
gitleaks dir . --redact --no-banner
gitleaks git . --redact --no-banner --log-opts="origin/master..HEAD"
fi
echo "[ci-local] Doc-only fast-path complete. No code paths exercised."
trap - EXIT
exit 0
;;
EMPTY)
echo "[ci-local] --diff: diff is empty (clean branch) — running full gate per fail-closed contract."
;;
SRC)
echo "[ci-local] --diff: diff touches src/ — running selected E2E + full unit phase."
;;
*)
echo "[ci-local] WARN: select-e2e.ts --classify-only returned '$CLASSIFICATION' — running full gate." >&2
;;
esac
fi
# Pre-flight: postgres host ports for 4 shards. Defaults to 5434-5437 (avoid
# 5432 manual gbrain-test-pg, 5433 commonly held by sibling projects).
# GBRAIN_CI_PG_PORT defines BASE; shards take BASE..BASE+3.
PG_PORT_BASE="${GBRAIN_CI_PG_PORT:-5434}"
for shard in 1 2 3 4; do
port=$((PG_PORT_BASE + shard - 1))
PORT_OWNER=$(docker ps --filter "publish=$port" --format "{{.Names}}" | head -1)
if [ -n "$PORT_OWNER" ]; then
echo "[ci-local] ERROR: host port $port (shard $shard) is already used by docker container '$PORT_OWNER'." >&2
echo "[ci-local] Either stop that container or run with: GBRAIN_CI_PG_PORT=NNNN bun run ci:local" >&2
exit 1
fi
if lsof -iTCP:"$port" -sTCP:LISTEN -P -n >/dev/null 2>&1; then
echo "[ci-local] ERROR: host port $port (shard $shard) is held by a non-docker process." >&2
echo "[ci-local] Run with: GBRAIN_CI_PG_PORT=NNNN bun run ci:local" >&2
exit 1
fi
done
export GBRAIN_CI_PG_PORT="$PG_PORT_BASE"
export GBRAIN_CI_PG_PORT_2=$((PG_PORT_BASE + 1))
export GBRAIN_CI_PG_PORT_3=$((PG_PORT_BASE + 2))
export GBRAIN_CI_PG_PORT_4=$((PG_PORT_BASE + 3))
# Step 0: gitleaks on the host (no docker, no postgres, no bun needed).
# Mirrors test.yml's separate gitleaks job. Fail loudly if not installed.
echo "[ci-local] gitleaks detect (host)..."
if ! command -v gitleaks >/dev/null 2>&1; then
echo "[ci-local] ERROR: gitleaks not installed on host." >&2
echo "[ci-local] macOS: brew install gitleaks" >&2
echo "[ci-local] Linux: https://github.com/gitleaks/gitleaks/releases" >&2
exit 1
fi
# Two scopes for pre-push:
# 1. Working-tree files (catch uncommitted secrets sitting in files)
# 2. Branch commits vs origin/master (catch secrets committed on this branch)
# Full-history scan is ~4 min on this repo's 3700+ commits; not useful pre-push.
gitleaks dir . --redact --no-banner
gitleaks git . --redact --no-banner --log-opts="origin/master..HEAD"
# Step 1: pull. Refreshes pgvector + oven/bun:1 (both are `image:` not `build:`).
if [ "$NO_PULL" = "0" ]; then
echo "[ci-local] Pulling base images (use --no-pull to skip)..."
docker compose -f "$COMPOSE_FILE" pull 2>&1 | tail -5
fi
# Step 2: 4 postgres shards up + wait for healthy.
echo "[ci-local] Starting 4 postgres shards..."
docker compose -f "$COMPOSE_FILE" up -d postgres-1 postgres-2 postgres-3 postgres-4
echo "[ci-local] Waiting for all 4 postgres shards healthy..."
for i in {1..40}; do
all_healthy=1
for shard in 1 2 3 4; do
status=$(docker compose -f "$COMPOSE_FILE" ps --format json postgres-$shard 2>/dev/null | grep -o '"Health":"[^"]*"' | head -1 | sed 's/.*":"//;s/"//')
if [ "$status" != "healthy" ]; then
all_healthy=0
break
fi
done
if [ "$all_healthy" = "1" ]; then
echo "[ci-local] All 4 postgres shards healthy."
break
fi
if [ "$i" = "40" ]; then
echo "[ci-local] ERROR: not all postgres shards became healthy in 40 attempts" >&2
exit 1
fi
sleep 1
done
# Step 3: smoke-test run-e2e.sh argv + shard handling.
echo "[ci-local] Smoke: run-e2e.sh argv + shard..."
SMOKE_NO_ARGS=$(bash scripts/run-e2e.sh --dry-run-list | wc -l | tr -d ' ')
EXPECTED_ALL=$(ls test/e2e/*.test.ts | wc -l | tr -d ' ')
if [ "$SMOKE_NO_ARGS" != "$EXPECTED_ALL" ]; then
echo "[ci-local] ERROR: --dry-run-list (no args) printed $SMOKE_NO_ARGS, expected $EXPECTED_ALL" >&2
exit 1
fi
SMOKE_ONE_ARG=$(bash scripts/run-e2e.sh --dry-run-list test/e2e/sync.test.ts)
if [ "$SMOKE_ONE_ARG" != "test/e2e/sync.test.ts" ]; then
echo "[ci-local] ERROR: --dry-run-list with 1 arg printed '$SMOKE_ONE_ARG'" >&2
exit 1
fi
SHARD_TOTAL=$(( $(SHARD=1/4 bash scripts/run-e2e.sh --dry-run-list | wc -l) + \
$(SHARD=2/4 bash scripts/run-e2e.sh --dry-run-list | wc -l) + \
$(SHARD=3/4 bash scripts/run-e2e.sh --dry-run-list | wc -l) + \
$(SHARD=4/4 bash scripts/run-e2e.sh --dry-run-list | wc -l) ))
if [ "$SHARD_TOTAL" != "$EXPECTED_ALL" ]; then
echo "[ci-local] ERROR: shards 1-4 covered $SHARD_TOTAL files, expected $EXPECTED_ALL" >&2
exit 1
fi
echo "[ci-local] Smoke OK ($SMOKE_NO_ARGS files no-arg, 1 single-arg, ${SHARD_TOTAL}=4-shard total)."
# Step 4: build the runner-side command.
# Tier 1: 4-shard parallel UNIT + E2E. Each shard runs ~46 unit files + ~9
# E2E files against postgres-N. Guards + typecheck run ONCE before fan-out.
# --no-shard runs the legacy unsharded flow (debug aid).
if [ "$NO_SHARD" = "1" ]; then
if [ "$DIFF" = "1" ]; then
RUN_PHASES_CMD='echo "[runner] guards + typecheck"
bash scripts/check-jsonb-pattern.sh
bash scripts/check-progress-to-stdout.sh
bash scripts/check-trailing-newline.sh
bash scripts/check-wasm-embedded.sh
bun run typecheck
echo "[runner] unit (unsharded, DATABASE_URL unset)"
env -u DATABASE_URL bash scripts/run-unit-shard.sh
echo "[runner] e2e (unsharded, --diff selected)"
SELECTED=$(bun run scripts/select-e2e.ts)
if [ -z "$SELECTED" ]; then
echo "[runner] selector emitted nothing (doc-only diff); skipping E2E."
else
DATABASE_URL=postgresql://postgres:postgres@postgres-1:5432/gbrain_test echo "$SELECTED" | xargs bash scripts/run-e2e.sh
fi'
else
RUN_PHASES_CMD='echo "[runner] guards + typecheck"
bash scripts/check-jsonb-pattern.sh
bash scripts/check-progress-to-stdout.sh
bash scripts/check-trailing-newline.sh
bash scripts/check-wasm-embedded.sh
bun run typecheck
echo "[runner] unit (unsharded, DATABASE_URL unset)"
env -u DATABASE_URL bash scripts/run-unit-shard.sh
echo "[runner] e2e (unsharded)"
DATABASE_URL=postgresql://postgres:postgres@postgres-1:5432/gbrain_test bash scripts/run-e2e.sh'
fi
else
# Tier 1 sharded path. Each shard runs unit+E2E sequentially against its
# own postgres-N. Shards run in parallel via xargs -P4.
if [ "$DIFF" = "1" ]; then
DIFF_E2E_PREP='SELECTED=$(bun run scripts/select-e2e.ts)
if [ -z "$SELECTED" ]; then
echo "" > /tmp/e2e-selected.txt
else
echo "$SELECTED" | tr " " "\n" | grep -v "^$" > /tmp/e2e-selected.txt
fi'
else
# Empty file -> run-e2e.sh uses default glob (all 36 E2E files).
DIFF_E2E_PREP='> /tmp/e2e-selected.txt'
fi
RUN_PHASES_CMD="echo \"[runner] guards + typecheck (run once before sharding)\"
bash scripts/check-jsonb-pattern.sh
bash scripts/check-progress-to-stdout.sh
bash scripts/check-trailing-newline.sh
bash scripts/check-wasm-embedded.sh
bun run typecheck
echo \"[runner] Tier 3: building PGLite snapshot fixture (cached across reruns)\"
if [ ! -f test/fixtures/pglite-snapshot.tar ] || [ ! -f test/fixtures/pglite-snapshot.version ]; then
bun run build:pglite-snapshot
else
echo \"[runner] snapshot fixture exists; engine will validate hash at load time\"
fi
export GBRAIN_PGLITE_SNAPSHOT=test/fixtures/pglite-snapshot.tar
echo \"[runner] resolving E2E file selection (--diff aware)\"
${DIFF_E2E_PREP}
mkdir -p /tmp/shard-logs
echo \"[runner] Tier 1: 4-shard parallel unit + E2E (xargs -P4)\"
set +e
printf '%s\\n' 1 2 3 4 | xargs -P4 -I{} sh -c '
shard=\$1
log=/tmp/shard-logs/shard-\${shard}.log
echo \"[shard \${shard}] start\" > \$log
echo \"[shard \${shard}] unit phase (SHARD=\${shard}/4, DATABASE_URL unset)\" >> \$log
env -u DATABASE_URL SHARD=\${shard}/4 bash scripts/run-unit-shard.sh >> \$log 2>&1
unit_exit=\$?
if [ \$unit_exit -ne 0 ]; then
echo \"[shard \${shard}] UNIT FAILED (exit=\$unit_exit)\" >> \$log
exit \$unit_exit
fi
echo \"[shard \${shard}] e2e phase (SHARD=\${shard}/4, DATABASE_URL=postgres-\${shard})\" >> \$log
if [ -s /tmp/e2e-selected.txt ]; then
SHARD=\${shard}/4 \\
DATABASE_URL=postgresql://postgres:postgres@postgres-\${shard}:5432/gbrain_test \\
xargs -a /tmp/e2e-selected.txt bash scripts/run-e2e.sh >> \$log 2>&1
else
SHARD=\${shard}/4 \\
DATABASE_URL=postgresql://postgres:postgres@postgres-\${shard}:5432/gbrain_test \\
bash scripts/run-e2e.sh >> \$log 2>&1
fi
e2e_exit=\$?
if [ \$e2e_exit -ne 0 ]; then
echo \"[shard \${shard}] E2E FAILED (exit=\$e2e_exit)\" >> \$log
exit \$e2e_exit
fi
echo \"[shard \${shard}] DONE\" >> \$log
' _ {}
shard_xargs_exit=\$?
set -e
echo \"\"
echo \"=== SHARD LOGS (last 30 lines each + unit/e2e summaries) ===\"
for s in 1 2 3 4; do
echo \"\"
echo \"--- shard \$s ---\"
if [ -f /tmp/shard-logs/shard-\$s.log ]; then
# Pull the unit + E2E summary lines explicitly so they survive even if
# the file is huge. Match: bun's '<N> pass / <N> fail' pairs, run-e2e.sh's
# 'Files: ... / Tests: ...' summary, and our own shard markers.
grep -E '^\\[shard|^Files: |^Tests: |Ran [0-9]+ tests|^[[:space:]]+[0-9]+ (pass|fail|skip)\$' /tmp/shard-logs/shard-\$s.log || true
echo \" (last 30 lines for context)\"
tail -30 /tmp/shard-logs/shard-\$s.log
else
echo \"(no log file written — shard never started)\"
fi
done
echo \"\"
if [ \$shard_xargs_exit -ne 0 ]; then
echo \"[runner] One or more shards failed (xargs exit=\$shard_xargs_exit). See SHARD LOGS above.\"
exit \$shard_xargs_exit
fi
echo \"[runner] All 4 shards passed.\""
fi
INNER_CMD=$(cat <<'EOF'
set -euo pipefail
echo "[runner] bun version: $(bun --version)"
# oven/bun:1 omits git; many unit tests use mkdtemp + git init for fixtures.
if ! command -v git >/dev/null 2>&1; then
echo "[runner] Installing git (debian apt)..."
apt-get update -qq >/dev/null
apt-get install -y -qq git ca-certificates >/dev/null
fi
# Container runs as root (uid 0) against a host-uid bind-mount; mark repo +
# any worktree gitdir as safe so `git status` etc. don't refuse.
git config --global --add safe.directory '*' || true
if [ ! -d /app/node_modules ] || [ -z "$(ls -A /app/node_modules 2>/dev/null)" ]; then
echo "[runner] First run (or --clean): bun install --frozen-lockfile"
bun install --frozen-lockfile
fi
__RUN_PHASES__
EOF
)
INNER_CMD="${INNER_CMD/__RUN_PHASES__/$RUN_PHASES_CMD}"
# Conductor / git-worktree support: when `.git` is a file (not a directory),
# it points at a host gitdir outside the bind-mount. Without remounting that
# path, scripts/check-trailing-newline.sh and any other in-container `git`
# call exits 128 ("not a git repository"). Resolve the host gitdir + the
# shared common gitdir and bind-mount them at the same absolute paths.
EXTRA_MOUNTS=()
if [ -f .git ]; then
WORKTREE_GITDIR=$(awk '{print $2}' .git)
if [ -d "$WORKTREE_GITDIR" ]; then
COMMONDIR_FILE="$WORKTREE_GITDIR/commondir"
if [ -f "$COMMONDIR_FILE" ]; then
COMMON_REL=$(cat "$COMMONDIR_FILE")
COMMON_GITDIR=$(cd "$WORKTREE_GITDIR" && cd "$COMMON_REL" && pwd)
else
COMMON_GITDIR="$WORKTREE_GITDIR"
fi
# Mount the higher-level common gitdir; covers worktrees/<name> automatically.
EXTRA_MOUNTS+=( -v "${COMMON_GITDIR}:${COMMON_GITDIR}:ro" )
echo "[ci-local] Worktree detected; mounting shared gitdir: $COMMON_GITDIR"
fi
fi
echo "[ci-local] Running checks inside runner container..."
docker compose -f "$COMPOSE_FILE" run --rm "${EXTRA_MOUNTS[@]:-}" runner bash -c "$INNER_CMD"
echo ""
echo "[ci-local] All checks passed."
+63
View File
@@ -0,0 +1,63 @@
// scripts/e2e-test-map.ts
//
// Path-glob -> E2E test files map. Used by scripts/select-e2e.ts.
//
// CONTRACT: This map can ONLY narrow from "all". When a changed src/ path
// matches no glob here, the selector falls back to "run all E2E" (fail-closed).
// You can safely add narrowing entries; you cannot break correctness by missing
// one. Tune as misses surface (i.e., when ci:local:diff ran more than necessary
// and you'd like to narrow that surface area).
//
// Glob syntax is the minimal subset implemented in select-e2e.ts:
// - "**" matches any sequence of path segments (including zero)
// - "*" matches any characters within a single path segment
// - everything else is literal
// No brace expansion, no ?, no [ ].
export const E2E_TEST_MAP: Record<string, string[]> = {
// Source-aware ranking, hybrid search, intent classification.
"src/core/search/**": [
"test/e2e/search-quality.test.ts",
"test/e2e/search-exclude.test.ts",
"test/e2e/search-swamp.test.ts",
],
// Tree-sitter chunkers feed code-indexing E2E.
"src/core/chunkers/**": ["test/e2e/code-indexing.test.ts"],
// dream.ts is a thin alias over runCycle in cycle.ts.
"src/core/cycle.ts": ["test/e2e/cycle.test.ts", "test/e2e/dream.test.ts"],
// Multi-source sync writes share the per-source bookmark anchor.
"src/core/sync.ts": ["test/e2e/sync.test.ts", "test/e2e/multi-source.test.ts"],
// Any minions queue/worker/handler change exercises all minion E2E.
"src/core/minions/**": [
"test/e2e/minions-concurrency.test.ts",
"test/e2e/minions-resilience.test.ts",
"test/e2e/minions-shell.test.ts",
"test/e2e/minions-shell-pglite.test.ts",
"test/e2e/worker-abort-recovery.test.ts",
],
// postgres.js bind paths + JSONB shapes + parity vs PGLite.
"src/core/postgres-engine.ts": [
"test/e2e/postgres-bootstrap.test.ts",
"test/e2e/postgres-jsonb.test.ts",
"test/e2e/jsonb-roundtrip.test.ts",
"test/e2e/engine-parity.test.ts",
],
// PGLite bootstrap path + parity guard.
"src/core/pglite-engine.ts": [
"test/e2e/postgres-bootstrap.test.ts",
"test/e2e/engine-parity.test.ts",
],
// MCP stdio + HTTP transports share dispatch.
"src/mcp/**": ["test/e2e/mcp.test.ts", "test/e2e/http-transport.test.ts"],
// Integrity batch-load fast path.
"src/commands/integrity.ts": ["test/e2e/integrity-batch.test.ts"],
// Upgrade chains migration ledger; touches both runners.
"src/commands/upgrade.ts": [
"test/e2e/upgrade.test.ts",
"test/e2e/migrate-chain.test.ts",
"test/e2e/migration-flow.test.ts",
],
"src/commands/doctor.ts": ["test/e2e/doctor-progress.test.ts"],
// Knowledge graph layer feeds graph-quality.
"src/core/link-extraction.ts": ["test/e2e/graph-quality.test.ts"],
};
+46
View File
@@ -0,0 +1,46 @@
#!/usr/bin/env bash
# scripts/profile-tests.sh
# Tier 4 helper: prints the top N slowest unit tests from a previous run.
# Pipe a captured `bun test` output (or a ci:local log) into stdin; we extract
# `(pass|fail) ... [Xms|Xs]` lines, convert to ms, sort descending.
#
# Usage:
# bun test --timeout=60000 2>&1 | bash scripts/profile-tests.sh
# bash scripts/profile-tests.sh < /path/to/captured.log
# bash scripts/profile-tests.sh -n 20 < /path/to/captured.log
#
# To demote a test as slow: rename its file to *.slow.test.ts. The file
# stays discoverable by `bun test` (CI runs everything via `bun run test`)
# but is excluded from `bun run ci:local`'s fast unit shard fan-out.
set -euo pipefail
TOP_N=10
if [ "${1:-}" = "-n" ] && [ -n "${2:-}" ]; then
TOP_N=$2
fi
# Lines look like: (pass) describe > test name [12345.67ms] OR [12.34s]
# Single awk pass for performance (input can be tens of MB).
awk '{
# Find the LAST bracket in the line: [<num><unit>] where unit is ms or s.
for (i = length($0); i > 0; i--) {
if (substr($0, i, 1) == "]") {
# Walk back to matching "["
j = i - 1
while (j > 0 && substr($0, j, 1) != "[") j--
if (j == 0) break
bracket = substr($0, j+1, i-j-1)
# bracket should match ^[0-9]+(\.[0-9]+)?(ms|s)$
if (bracket ~ /^[0-9]+(\.[0-9]+)?(ms|s)$/) {
if (bracket ~ /ms$/) {
n = substr(bracket, 1, length(bracket) - 2) + 0
} else {
n = (substr(bracket, 1, length(bracket) - 1) + 0) * 1000
}
if (n > 0) printf "%.0f\t%s\n", n, $0
}
break
}
}
}' | sort -rn | head -n "$TOP_N" | awk -F'\t' '{ printf "%8.0fms %s\n", $1, $2 }'
+59 -1
View File
@@ -25,13 +25,71 @@ set -euo pipefail
cd "$(dirname "$0")/.."
# --dry-run-list: print the resolved file list (one per line) and exit. Used
# by scripts/ci-local.sh to smoke-test the argv branching at startup.
DRY_RUN_LIST=0
if [ "${1:-}" = "--dry-run-list" ]; then
DRY_RUN_LIST=1
shift
fi
# Argv-driven file list (used by `ci:local:diff`); fall back to the full glob.
if [ "$#" -gt 0 ]; then
files=("$@")
else
files=(test/e2e/*.test.ts)
fi
# SHARD env (e.g. SHARD=1/4) keeps every M-th file starting at index N (1-indexed).
# Used by scripts/ci-local.sh to fan 4 shards in parallel against 4 postgres
# containers. Sequential execution within a shard is preserved (the TRUNCATE
# CASCADE no-race rationale at the top of this file still holds).
if [ -n "${SHARD:-}" ]; then
shard_n=${SHARD%/*}
shard_m=${SHARD#*/}
if ! printf '%s' "$shard_n" | grep -qE '^[0-9]+$' || \
! printf '%s' "$shard_m" | grep -qE '^[0-9]+$' || \
[ "$shard_n" -lt 1 ] || [ "$shard_m" -lt 1 ] || [ "$shard_n" -gt "$shard_m" ]; then
echo "ERROR: invalid SHARD=$SHARD (expected N/M with 1<=N<=M, both integers)" >&2
exit 1
fi
filtered=()
i=0
for f in "${files[@]}"; do
if [ $((i % shard_m + 1)) -eq "$shard_n" ]; then
filtered+=("$f")
fi
i=$((i + 1))
done
# ${filtered[@]:-} avoids "unbound variable" under `set -u` when no files matched.
files=("${filtered[@]:-}")
# If the empty placeholder slipped in, drop it.
if [ "${#files[@]}" -eq 1 ] && [ -z "${files[0]}" ]; then
files=()
fi
fi
if [ "$DRY_RUN_LIST" = "1" ]; then
if [ "${#files[@]}" -eq 0 ]; then
exit 0
fi
printf '%s\n' "${files[@]}"
exit 0
fi
if [ "${#files[@]}" -eq 0 ]; then
# Empty shard (e.g. SHARD=4/4 with only 3 files): nothing to do.
echo "No files for shard ${SHARD:-(unsharded)}; exiting clean."
exit 0
fi
pass_files=0
fail_files=0
fail_list=()
total_pass=0
total_fail=0
for f in test/e2e/*.test.ts; do
for f in "${files[@]}"; do
name=$(basename "$f")
echo ""
echo "=== $name ==="
+20
View File
@@ -0,0 +1,20 @@
#!/usr/bin/env bash
# scripts/run-slow-tests.sh
# Tier 4 sister to run-unit-shard.sh: runs ONLY *.slow.test.ts files.
# CI runs both; bun run ci:local skips slow tests via run-unit-shard.sh.
set -euo pipefail
cd "$(dirname "$0")/.."
slow_files=()
while IFS= read -r f; do
slow_files+=("$f")
done < <(find test -name '*.slow.test.ts' -not -path 'test/e2e/*' | sort)
if [ "${#slow_files[@]}" -eq 0 ]; then
echo "[run-slow-tests] no *.slow.test.ts files; nothing to do."
exit 0
fi
echo "[run-slow-tests] running ${#slow_files[@]} slow files (CI runs these as part of bun run test)"
exec bun test --timeout=60000 "${slow_files[@]}"
+63
View File
@@ -0,0 +1,63 @@
#!/usr/bin/env bash
# scripts/run-unit-shard.sh
#
# Runs the unit suite for a single shard. Excludes test/e2e/* (those are run
# by scripts/run-e2e.sh in the E2E phase). When SHARD=N/M is set, keeps every
# M-th file starting at index N (1-indexed); otherwise runs the full unit set.
#
# Used by scripts/ci-local.sh to fan 4 unit-shard workers in parallel inside
# the runner container, each pinned to its own postgres shard for the
# downstream E2E phase.
#
# Sequential bun processes within a shard (one bun test invocation with the
# shard's file list); parallel across shards (4 of these run concurrently).
set -euo pipefail
cd "$(dirname "$0")/.."
# All non-E2E test files, sorted for deterministic shard splits.
# Tier 4: *.slow.test.ts is the convention for "always-slow" tests (e.g.,
# bootstrap correctness checks that intentionally exercise the cold init
# path and can't benefit from Tier 3's snapshot). They're excluded from the
# fast loop and run via `bun run test:slow` (or in CI where everything runs).
# Use while-read to stay portable to macOS bash 3.2 (no mapfile).
all_files=()
while IFS= read -r f; do
all_files+=("$f")
done < <(find test -name '*.test.ts' -not -path 'test/e2e/*' -not -name '*.slow.test.ts' | sort)
files=()
if [ -n "${SHARD:-}" ]; then
shard_n=${SHARD%/*}
shard_m=${SHARD#*/}
if ! printf '%s' "$shard_n" | grep -qE '^[0-9]+$' || \
! printf '%s' "$shard_m" | grep -qE '^[0-9]+$' || \
[ "$shard_n" -lt 1 ] || [ "$shard_m" -lt 1 ] || [ "$shard_n" -gt "$shard_m" ]; then
echo "ERROR: invalid SHARD=$SHARD (expected N/M with 1<=N<=M, both integers)" >&2
exit 1
fi
i=0
for f in "${all_files[@]}"; do
if [ $((i % shard_m + 1)) -eq "$shard_n" ]; then
files+=("$f")
fi
i=$((i + 1))
done
else
files=("${all_files[@]}")
fi
if [ "${#files[@]}" -eq 0 ]; then
echo "[unit-shard ${SHARD:-(unsharded)}] no files; exiting clean."
exit 0
fi
# --dry-run-list mirrors scripts/run-e2e.sh for inline smoke checks.
if [ "${1:-}" = "--dry-run-list" ]; then
printf '%s\n' "${files[@]}"
exit 0
fi
echo "[unit-shard ${SHARD:-(unsharded)}] running ${#files[@]} files"
exec bun test --timeout=60000 "${files[@]}"
+245
View File
@@ -0,0 +1,245 @@
#!/usr/bin/env bun
// scripts/select-e2e.ts
//
// Fail-closed diff-based E2E test selector. Reads the working-tree diff vs
// origin/master plus untracked files, classifies the change set as
// EMPTY / DOC_ONLY / SRC, and emits the relevant E2E test files on stdout.
//
// CONTRACT (fail-closed):
// - When in doubt, run all E2E. The map narrows from "all"; it never widens
// from "none". An unmapped src/ change emits ALL test/e2e/*.test.ts.
// - Doc-only diffs emit nothing (the only case where stdout is empty).
// - Empty diff emits ALL (clean branch shouldn't run nothing).
//
// Selection algorithm:
// 1. Read changed files from three git sources, union them:
// - git diff --name-only origin/master...HEAD (committed)
// - git diff --name-only HEAD (unstaged + staged)
// - git ls-files --others --exclude-standard (untracked, NOT .gitignore'd)
// 2. EMPTY -> emit ALL test/e2e/*.test.ts
// DOC_ONLY (every path matches doc allowlist) -> emit nothing
// SRC (at least one path is outside doc allowlist):
// a. Any escape-hatch path matched -> emit ALL
// b. Else union map matches; include directly-modified test/e2e/*.test.ts
// c. If still empty -> FAIL-CLOSED -> emit ALL
//
// On git command failure: print error to stderr and exit 2 so callers see the
// failure (xargs -r will run nothing AND the human sees the error).
//
// Usage:
// bun run scripts/select-e2e.ts
// bun run scripts/select-e2e.ts | xargs -r bash scripts/run-e2e.sh
import { spawnSync } from "node:child_process";
import { readdirSync, existsSync } from "node:fs";
import { join } from "node:path";
import { E2E_TEST_MAP } from "./e2e-test-map.ts";
// Doc allowlist (inclusive). A path counts as doc-only ONLY if it matches one
// of these patterns. Unrecognized paths fall through to SRC, never silently
// doc-only. skills/ is intentionally NOT here — skills are product input.
const DOC_ROOT_FILES = new Set([
"README.md",
"CLAUDE.md",
"AGENTS.md",
"CHANGELOG.md",
"TODOS.md",
"LICENSE",
"VERSION",
]);
function isDocPath(p: string): boolean {
if (DOC_ROOT_FILES.has(p)) return true;
// Any *.md at repo root.
if (!p.includes("/") && p.endsWith(".md")) return true;
// Anything under docs/.
if (p.startsWith("docs/")) return true;
return false;
}
// Escape-hatch triggers. Any match -> emit ALL.
const ESCAPE_HATCH_FILES = new Set([
"src/schema.sql",
"src/core/migrate.ts",
"src/core/db.ts",
"src/core/engine-factory.ts",
"src/core/operations.ts",
"package.json",
"bun.lock",
"Dockerfile.ci",
"docker-compose.ci.yml",
"scripts/ci-local.sh",
"scripts/run-e2e.sh",
"scripts/select-e2e.ts",
"scripts/e2e-test-map.ts",
"test/e2e/helpers.ts",
]);
const ESCAPE_HATCH_PREFIXES = [
"src/commands/migrations/",
"test/e2e/fixtures/",
"skills/",
".github/workflows/",
];
function isEscapeHatch(p: string): boolean {
if (ESCAPE_HATCH_FILES.has(p)) return true;
for (const prefix of ESCAPE_HATCH_PREFIXES) {
if (p.startsWith(prefix)) return true;
}
return false;
}
// Minimal glob matcher: supports ** (any segments) and * (one segment, no /).
// Throws on unsupported syntax so map mistakes surface loudly.
export function matchGlob(glob: string, path: string): boolean {
if (glob.includes("?") || glob.includes("[") || glob.includes("{")) {
throw new Error(
`select-e2e: unsupported glob syntax in "${glob}" (only ** and * are supported)`
);
}
// Build a regex: ** -> .*, * -> [^/]*, escape other regex meta-chars.
let regex = "";
let i = 0;
while (i < glob.length) {
const c = glob[i];
if (c === "*" && glob[i + 1] === "*") {
regex += ".*";
i += 2;
} else if (c === "*") {
regex += "[^/]*";
i += 1;
} else if (/[.+^${}()|\\]/.test(c)) {
regex += "\\" + c;
i += 1;
} else {
regex += c;
i += 1;
}
}
return new RegExp("^" + regex + "$").test(path);
}
function listAllE2ETests(repoRoot: string): string[] {
const dir = join(repoRoot, "test/e2e");
if (!existsSync(dir)) return [];
return readdirSync(dir)
.filter((f) => f.endsWith(".test.ts"))
.map((f) => `test/e2e/${f}`)
.sort();
}
// Pure function — exposed for unit tests. Decides what to emit given the
// inputs, without touching git or filesystem (callers pass arrays in).
export interface SelectInputs {
changedFiles: string[]; // union of three git sources
allE2ETests: string[]; // glob result of test/e2e/*.test.ts
map: Record<string, string[]>; // E2E_TEST_MAP
}
export type Classification = "EMPTY" | "DOC_ONLY" | "SRC";
export function classify(changedFiles: string[]): Classification {
if (changedFiles.length === 0) return "EMPTY";
for (const f of changedFiles) {
if (!isDocPath(f)) return "SRC";
}
return "DOC_ONLY";
}
export function selectTests(inputs: SelectInputs): string[] {
const { changedFiles, allE2ETests, map } = inputs;
const cls = classify(changedFiles);
const allSorted = allE2ETests.slice().sort();
if (cls === "EMPTY") return allSorted;
if (cls === "DOC_ONLY") return [];
// SRC case.
// 3a. Any escape-hatch -> ALL.
for (const f of changedFiles) {
if (isEscapeHatch(f)) return allSorted;
}
// 3b. Union map matches; include directly-modified test files.
const result = new Set<string>();
for (const f of changedFiles) {
if (isDocPath(f)) continue;
// Direct test file modification: include it.
if (f.startsWith("test/e2e/") && f.endsWith(".test.ts")) {
result.add(f);
continue;
}
for (const [glob, tests] of Object.entries(map)) {
if (matchGlob(glob, f)) {
for (const t of tests) result.add(t);
}
}
}
// 3c. Fail-closed: if no map entry matched any src/ path AND no test files
// were directly modified, run everything.
if (result.size === 0) return allSorted;
// Sort for determinism (helps tests + readability).
return Array.from(result).sort();
}
function runGit(args: string[], cwd: string): string {
const result = spawnSync("git", args, { cwd, encoding: "utf8" });
if (result.status !== 0) {
const stderr = (result.stderr || "").trim();
process.stderr.write(
`select-e2e: git ${args.join(" ")} failed: ${stderr}\n`
);
process.exit(2);
}
return result.stdout || "";
}
function readChangedFiles(repoRoot: string): string[] {
const sources = [
runGit(["diff", "--name-only", "origin/master...HEAD"], repoRoot),
runGit(["diff", "--name-only", "HEAD"], repoRoot),
runGit(["ls-files", "--others", "--exclude-standard"], repoRoot),
];
const set = new Set<string>();
for (const out of sources) {
for (const line of out.split("\n")) {
const trimmed = line.trim();
if (trimmed.length > 0) set.add(trimmed);
}
}
return Array.from(set).sort();
}
// Entrypoint. Skipped under test (Bun.main check).
if (import.meta.main) {
const repoRoot = spawnSync("git", ["rev-parse", "--show-toplevel"], {
encoding: "utf8",
}).stdout?.trim();
if (!repoRoot) {
process.stderr.write("select-e2e: not a git repository\n");
process.exit(2);
}
const changedFiles = readChangedFiles(repoRoot);
// --classify-only: print EMPTY|DOC_ONLY|SRC + exit. Used by ci-local.sh's
// Tier 2 fast-path so doc-only diffs skip the unit phase entirely.
if (process.argv.includes("--classify-only")) {
process.stdout.write(classify(changedFiles) + "\n");
process.exit(0);
}
const allE2ETests = listAllE2ETests(repoRoot);
const tests = selectTests({
changedFiles,
allE2ETests,
map: E2E_TEST_MAP,
});
process.stdout.write(tests.join(" "));
if (tests.length > 0) process.stdout.write("\n");
}
+1
View File
@@ -70,6 +70,7 @@ This is the dispatcher. Skills are the implementation. **Read the skill file bef
| "Migrate from Obsidian/Notion/Logseq" | `skills/migrate/SKILL.md` |
| Brain health check, maintenance run | `skills/maintain/SKILL.md` |
| "Extract links", "build link graph", "populate timeline" | `skills/maintain/SKILL.md` (extraction sections) |
| "Run dream", "process today's session", "synthesize my conversations", "consolidate yesterday's conversations", "what patterns did you see", "did the dream cycle run" | `skills/maintain/SKILL.md` (dream cycle section) |
| "Brain health", "what features am I missing", "brain score" | Run `gbrain features --json` |
| "Set up autopilot", "run brain maintenance", "keep brain updated" | Run `gbrain autopilot --install --repo ~/brain` |
| Agent identity, "who am I", customize agent | `skills/soul-audit/SKILL.md` |
+11 -1
View File
@@ -97,5 +97,15 @@
"The PRIMARY SUBJECT of the content determines the directory, not the format or source skill.",
"When in doubt: what would you search for to find this page again?",
"Cross-link from related directories via back-links — do not duplicate content."
]
],
"dream_synthesize_paths": {
"description": "Single source of truth for the v0.23 dream-cycle synthesize/patterns trusted-workspace allow-list. The cycle's synthesize phase reads this list and threads it as `allowed_slug_prefixes` to every subagent it dispatches; put_page enforces it server-side. Editing this list is the ONLY way to add a new directory the synthesis subagent may write to.",
"globs": [
"wiki/personal/reflections/*",
"wiki/originals/*",
"wiki/personal/patterns/*",
"wiki/people/*",
"dream-cycle-summaries/*"
]
}
}
+21
View File
@@ -112,3 +112,24 @@ gbrain files restore <dir> # Download back to local
This ensures any derived brain page can be traced back to its original source,
and large files don't bloat the git repo.
## Dream-cycle synthesize / patterns directories (v0.23)
The `synthesize` and `patterns` phases of `gbrain dream` write to a
**fixed allow-list** of paths sourced from `_brain-filing-rules.json`'s
`dream_synthesize_paths.globs` array. Editing that JSON is the ONLY way
to add a new directory the synthesis subagent may write to:
| Output type | Slug pattern | What goes here |
|-------------|--------------|----------------|
| Reflection | `wiki/personal/reflections/YYYY-MM-DD-<topic>-<hash[:6]>` | Self-knowledge, emotional processing, pattern recognition. Verbatim quotes from the user, with analysis. |
| Original idea | `wiki/originals/ideas/YYYY-MM-DD-<idea>-<hash[:6]>` | New frames, theses, mental models, "conceptive ideologist" outputs. Capture the user's exact phrasing — that's the artifact. |
| People enrichment | `wiki/people/<existing-slug>` | Timeline entries appended to existing people pages from session mentions. Stub pages for new substantive people. |
| Pattern | `wiki/personal/patterns/<theme>` | Cross-session theme detected across ≥3 reflections. Highest-leverage output: a pattern can span 25 years if reflections reference dated content. |
| Cycle summary | `dream-cycle-summaries/YYYY-MM-DD` | Index of every page produced by one dream cycle. Auto-written deterministically by the orchestrator. |
**Iron Law for synthesize output:**
1. Quote the user verbatim. Do not paraphrase memorable phrasings.
2. Cross-reference compulsively: every new page MUST link to existing brain content.
3. Slug discipline: lowercase alphanumeric and hyphens only, slash-separated. NO underscores, NO file extensions.
4. Edited transcripts produce NEW slugs (content-hash suffix changes) — never silently overwrite a prior reflection.
+60
View File
@@ -0,0 +1,60 @@
# Friction protocol — convention
> Cross-cutting rule shared by skills the claw-test harness exercises (setup,
> brain-ops, query, ingest, smoke-test, migrations). Reference via
> `> **Convention:** see [skills/_friction-protocol.md](_friction-protocol.md).`
When you encounter friction running gbrain — anything confusing, missing, surprising, or wrong — log it via `gbrain friction log` so maintainers can see it without you writing a bug report. Friction reports drive the claw-test feedback loop (the harness collects, renders, and re-runs).
## When to log
Log friction when any of these happens:
- A command failed with a non-actionable error message
- A doc said one thing and the tool did another
- You couldn't find the next step
- A setup command needed a manual workaround
- A flag exists but isn't documented in `--help`
- A success condition was unclear (you couldn't tell if the command worked)
Log delight (positive signal) when:
- Something worked on the first try and the docs were exactly right
- An error message handed you the fix
- A flag you guessed at turned out to exist with the obvious name
## How to log
```
gbrain friction log \
--severity {confused|error|blocker|nit} \
--phase <which-phase-or-command> \
--message "<one-line-what-happened>" \
[--hint "<one-line-what-could-be-better>"]
```
For delight, add `--kind delight` and pick any severity.
The CLI auto-fills `ts`, `cwd`, `gbrain_version`, and resolves `run_id` from `$GBRAIN_FRICTION_RUN_ID` (set by the harness) or falls back to `standalone.jsonl`. So you can call this anywhere — inside a harness run, manually during normal use, or from a scripted test.
## Severity guide
| severity | meaning |
|------------|---------|
| `blocker` | Couldn't proceed at all. Hard stop. |
| `error` | Command failed unexpectedly. |
| `confused` | Docs/tool mismatch, ambiguity, missing pointer. |
| `nit` | Polish opportunity. Cosmetic or low-impact. |
Be specific: "doctor says `schema_version=0` and points at apply-migrations, but apply-migrations exits 0 with no output" beats "doctor was confusing."
## Inspecting reports
```
gbrain friction list # recent runs with counts
gbrain friction render --run-id <id> # markdown report (default)
gbrain friction render --run-id <id> --json
gbrain friction summary --run-id <id> # friction + delight side-by-side
```
`render` defaults to `--redact` for markdown (strips `$HOME`/`$CWD` to `<HOME>`/`<CWD>` placeholders) so reports paste safely into PRs and issues.
+68 -14
View File
@@ -1,21 +1,75 @@
# Brain-First Lookup Convention
Before using ANY external API (web search, enrichment services, social APIs) to
research a person, company, or topic, check the brain first.
**Read this before doing ANY entity/person/company/fact lookup.**
## The 5-Step Lookup
Sub-agents and fresh sessions inherit gbrain tools but not the knowledge of
when and how to use them. This file is that knowledge.
1. `gbrain search "name"` — keyword search for existing pages
2. `gbrain query "natural question about name"` — hybrid search for related context
3. `gbrain get <slug>` — if you know the slug, read the full page
4. Check backlinks: `gbrain get_backlinks <slug>` — who references this entity?
5. Check timeline: `gbrain get_timeline <slug>` — recent events involving this entity
## Available GBrain Tools
The brain almost always has something. External APIs fill gaps, not start from scratch.
Your tool inventory includes these (prefixed `gbrain__` in OpenClaw):
## Why This Matters
| Tool | Use for |
|------|---------|
| `gbrain__search` / `search` | Keyword search — fast, always works |
| `gbrain__query` / `query` | Hybrid search (keyword + semantic) — best quality |
| `gbrain__get_page` / `get_page` | Direct page read when you know the slug |
| `gbrain__get_links` / `get_links` | Outgoing links from a page |
| `gbrain__get_backlinks` / `get_backlinks` | Who references this entity |
| `gbrain__get_timeline` / `get_timeline` | Dated events for an entity |
| `gbrain__resolve_slugs` / `resolve_slugs` | Fuzzy slug resolution |
| `gbrain__traverse_graph` / `traverse_graph` | Walk the relationship graph |
| `gbrain__put_page` / `put_page` | Create or update a brain page |
| `gbrain__add_timeline_entry` | Add a dated event |
| `gbrain__add_link` | Add a relationship edge |
- The brain has context that external APIs don't (user's direct observations, meeting notes, personal relationships)
- External API calls cost money and time
- Brain context makes external lookups more targeted (you know what's missing)
- The user's direct statements are highest-authority data. External sources are lowest.
Tool names vary by transport (MCP uses short names, OpenClaw plugin uses
`gbrain__` prefix). Both work. Use whichever your environment provides.
## The Lookup Chain (MANDATORY ORDER)
1. **`search`** first — keyword search, fast, zero API cost
2. **`query`** if search is thin — hybrid semantic search, uses embedding API
3. **`get_page`** if you found a slug — read the full compiled truth
4. **External APIs only after steps 1-2 return nothing useful**
Never skip to external APIs without completing steps 1-2. The brain has
thousands of pages. The answer is almost always there.
## Rules
- **Score > 0.5 = use it.** Don't reach for external APIs when the brain answered.
- **User's direct statements are highest-authority data.** The brain captures
what the user said in meetings, conversations, and notes. External sources
are supplementary.
- **After any brain page write:** trigger a sync so new pages are searchable.
In OpenClaw: `gbrain__sync_brain`. From CLI: `gbrain sync --no-pull`.
- **Every brain page reference in output** should use a clickable link format
appropriate to the deployment (GitHub URL, local path, or slug).
- **Never use `memory_search` for entity lookups.** Memory tools search
session notes (MEMORY.md), not the brain knowledge graph. Use
`search` or `query` for entity lookups.
## Entity Page Conventions
Standard directory structure:
| Directory | Type | Example |
|-----------|------|---------|
| `people/` | person | `people/paul-graham.md` |
| `companies/` | company | `companies/stripe.md` |
| `deals/` | deal | `deals/stripe-series-c.md` |
| `meetings/` | meeting | `meetings/2026-04-23-weekly-sync.md` |
| `projects/` | project | `projects/gbrain.md` |
| `yc/` | yc | `yc/batch-w26.md` |
When creating new pages, include proper frontmatter with `type`, `title`,
and `tags` fields.
## When Spawning Further Sub-agents
If you spawn your own sub-agents, include this line in their task prompt:
> Read `skills/conventions/brain-first.md` before starting work.
This ensures the convention propagates through any depth of sub-agent chain.
+82
View File
@@ -17,6 +17,13 @@ triggers:
- "populate links"
- "backfill graph"
- "extract timeline entries"
- "run dream"
- "process today's session"
- "process yesterday's transcripts"
- "synthesize my conversations"
- "what patterns did you see"
- "did the dream cycle run"
- "consolidate yesterday's conversations"
tools:
- get_health
- get_page
@@ -77,6 +84,81 @@ If timeline_entry_count is 0, extract structured timeline from markdown:
```bash
gbrain extract timeline --dir ~/brain
```
### Dream cycle (v0.23): synthesize + patterns
`gbrain dream` runs the full 8-phase maintenance cycle:
```
lint -> backlinks -> sync -> synthesize -> extract -> patterns -> embed -> orphans
```
The two new phases consolidate yesterday's conversations into long-term memory:
**Synthesize phase:** reads transcripts from `dream.synthesize.session_corpus_dir`,
runs a cheap Haiku verdict (cached in `dream_verdicts`) to filter routine
ops sessions, then fans out one Sonnet subagent per worth-processing
transcript. Each subagent writes reflections (`wiki/personal/reflections/...`),
originals (`wiki/originals/ideas/...`), and people timeline entries. The
orchestrator collects the slugs from `subagent_tool_executions` (NOT
`pages.updated_at` — that would pick up unrelated writes) and reverse-renders
each new page from DB → markdown on disk.
**Patterns phase:** runs after `extract` (so the graph state is fresh).
Reads recent reflections within `dream.patterns.lookback_days` (default 30),
runs a single Sonnet pass to surface recurring themes, and writes pattern
pages to `wiki/personal/patterns/<theme>` when ≥`dream.patterns.min_evidence`
(default 3) reflections support a pattern.
**Quality bar (Iron Law for synthesis):**
1. Quote the user verbatim. Do not paraphrase memorable phrasings.
2. Cross-reference compulsively: every new page MUST have at least one wikilink.
3. Slug discipline: lowercase alphanumeric and hyphens only. NO underscores, NO file extensions.
4. Edited transcripts produce NEW slugs (content-hash suffix changes) — never silently overwrite.
**Trust boundary (`allowed_slug_prefixes`):** the synthesis subagent runs with an
explicit allow-list of write paths sourced from `_brain-filing-rules.json`'s
`dream_synthesize_paths.globs`. Even on prompt-injection success, the subagent
cannot write outside that list. Trust comes from PROTECTED_JOB_NAMES — MCP
cannot submit subagent jobs at all. Editing the JSON is the only way to add
a new directory the synthesizer can write to.
**Idempotency + privacy:** transcripts are keyed by `(file_path, content_hash)`,
so re-running on the same content is a no-op. `dream.synthesize.exclude_patterns`
(default `["medical", "therapy"]`) filters out transcripts before any LLM call.
Each entry is auto-wrapped as a word-boundary regex (e.g. `medical` matches
"medical advice" but NOT "comedical"). Power users may pass full regex.
**Cooldown:** the cycle's spend cap. `dream.synthesize.cooldown_hours` (default
12) means at most ~2 synthesize runs per day under autopilot. The completion
timestamp is stored in `dream.synthesize.last_completion_ts` and is written
ONLY on successful runs (not on skipped/failed). Explicit `--input` /
`--date` / `--from` / `--to` invocations bypass cooldown.
**`--dry-run` semantics:** runs the cheap Haiku significance filter (caches
verdicts) but skips the Sonnet synthesis pass. NOT zero LLM calls.
**Configure synthesize on a fresh brain:**
```bash
gbrain config set dream.synthesize.session_corpus_dir /path/to/transcripts
gbrain config set dream.synthesize.enabled true
gbrain dream --phase synthesize --dry-run --json # preview
gbrain dream # full 8-phase cycle
```
**Invocation patterns:**
```bash
gbrain dream # full cycle
gbrain dream --phase synthesize # just synthesize
gbrain dream --phase patterns # just patterns
gbrain dream --input ~/transcripts/2026-04-25.txt # ad-hoc one transcript
gbrain dream --from 2026-04-01 --to 2026-04-25 # backfill range
gbrain dream --json # CycleReport JSON
```
**Auto-commit deferred to v1.1:** v1 writes files to `brain_dir` but does NOT
`git add` / `commit` / `push`. Either commit yourself or let `gbrain autopilot`
handle it.
Parses `- **YYYY-MM-DD** | Source — Summary` and `### YYYY-MM-DD — Title` formats.
Note: extracted entries improve structured queries (`gbrain timeline`), not vector search.
+1 -1
View File
@@ -46,7 +46,7 @@ These run as part of `gbrain upgrade` → `gbrain apply-migrations`. No manual D
5. **Observe incremental chunking.** Edit one function in a 20-function file, re-run `sync --source <id>`. Embedding cost should be ~5% of the first sync because unchanged chunks reuse their existing embeddings.
## Migration from Wintermute's `repos` (if you used it)
## Migration from your OpenClaw's `repos` (if you used it)
v0.19.0 deletes `~/.gbrain/config.json`'s `repos` array in favor of the `sources` table. The CLI surface is preserved as a deprecated alias: `gbrain repos add` still works, but routes into `runSources` with a one-line deprecation notice on stderr. Existing scripts keep working; prefer `gbrain sources` going forward.
+210
View File
@@ -0,0 +1,210 @@
---
feature_pitch:
headline: Bare workers now self-monitor and fail-stop into your PM's restart loop
body: |
Bare `gbrain jobs work` now ships with the same health protection the
supervisor already had: DB liveness probes (with per-probe timeout so a
hung connection can't wedge the monitor), stall detection filtered by
registered handler names, and an RSS watchdog default of 2048 MB.
When the worker detects it's wedged (stuck pgbouncer connection, hung
event loop, stalled job claim), it emits `'unhealthy'` and the CLI calls
`process.exit(1)`. This is **fail-stop**: it requires an external process
manager (systemd, Docker `restart: always`, launchd `KeepAlive`, cron
watchdog) to bring the worker back. Without one, the process exits and
stays dead — that's a regression from pre-v0.22.14 self-healing.
Pre-v0.22.14 behavior: bare workers had ZERO health monitoring. A wedged
worker stayed alive doing nothing while jobs piled up in `waiting` and
your PM's `pgrep` check happily reported green.
If you're using `gbrain jobs supervisor`, you're already protected — the
supervisor handles spawn-on-crash itself. The fail-stop concern only
applies to direct `gbrain jobs work` invocations.
---
# v0.22.14 — Bare-worker self-health-monitoring
## ⚠️ Pre-flight: confirm you have a process supervisor
If you run `gbrain jobs work` directly (NOT under `gbrain jobs supervisor`),
verify your process manager is configured to restart the worker on exit
BEFORE upgrading:
| Manager | What to check |
|---|---|
| systemd | `Restart=always` (or `Restart=on-failure`) in the `.service` unit |
| Docker | `restart: always` / `restart: unless-stopped` in compose, OR `--restart` flag |
| launchd (macOS) | `<key>KeepAlive</key><true/>` in the plist |
| cron watchdog | Cron entry that re-spawns when `pgrep -f "gbrain jobs work"` is empty |
| supervisord | `autorestart=true` |
**If your bare worker has no restart loop, the v0.22.14 fail-stop behavior
will leave you with a dead worker after the first DB blip.** Either add a
restart policy OR switch to `gbrain jobs supervisor` (which spawns its own
child + restarts on crash internally).
## What ships
- DB liveness probes inside `gbrain jobs work` (60s interval, 3 strikes → exit)
- Stall detection (5min warn / 10min exit when waiting jobs accumulate but
in-flight is empty)
- `--max-rss` defaults to 2048 MB for bare workers (matches supervisor default;
was 0 = disabled)
- New `MinionWorkerOpts.{healthCheckInterval, stallWarnAfterMs,
stallExitAfterMs, dbFailExitAfter, dbProbeTimeoutMs}` for tuning (5 fields)
- `MinionWorker` now extends `EventEmitter`; emits `'unhealthy'` event with
a structured reason payload. **No-listener fallback**: if the caller does
not subscribe to `'unhealthy'`, the worker calls `process.exit(1)` itself
to preserve the pre-refactor fail-stop behavior. The CLI subscribes; direct
API consumers without a listener inherit the fail-stop default. Inline
paths (`jobs submit --follow`, `jobs smoke`) explicitly pass
`healthCheckInterval: 0` to disable the timer entirely so the fallback
cannot trip on a one-shot run.
- `GBRAIN_SUPERVISED=1` env var (set by supervisor on its child) causes the
child worker to skip its own health timer (no double-monitoring)
- Constructor validation: throws if `stallExitAfterMs <= stallWarnAfterMs`
(the contract requires strict greater-than)
## ⚠️ Breaking change you may need to act on
**`gbrain jobs work` now defaults `--max-rss 2048` (MB).** Pre-v0.22.14, bare
workers had no RSS limit. If you ran bare workers with intentionally large
embed/import working sets, your worker may now get killed by the watchdog after
upgrade.
**You need to act if any of the following is true:**
- You run `gbrain jobs work` directly under a process manager (systemd, Docker,
cron, supervisord, your own bash wrapper)
- Your worker handles large embed batches, multi-GB import jobs, or
memory-intensive subagent tasks that legitimately exceed 2 GB RSS
- You rely on bare-worker resident memory exceeding 2 GB
**You do NOT need to act if:**
- You run `gbrain jobs supervisor` (the supervisor already defaulted to
`--max-rss 2048` and has reconnect-then-fail logic on top)
- Your typical worker stays well under 2 GB RSS
## What the agent should do post-upgrade
### 1. Run the orchestrator (mechanical migration)
```bash
gbrain apply-migrations --yes
```
This is a no-op for v0.22.14 — there's no schema migration. The mechanical work
is in the binary; the user-side decision is the breaking-change handling below.
### 2. Detect bare-worker deployments
If the user runs OpenClaw, Hermes, or any agent fork that uses `gbrain jobs
work` directly (NOT under `gbrain jobs supervisor`), they need to know. Check:
```bash
# Look for bare worker invocations in cron, systemd, Docker, launchctl, etc.
grep -r "gbrain jobs work" ~/.config/systemd/ /etc/systemd/ 2>/dev/null | head
crontab -l 2>/dev/null | grep "gbrain jobs work" | head
launchctl list 2>/dev/null | grep gbrain | head
ps -ef | grep "gbrain jobs work" | grep -v grep | grep -v supervisor
```
If you find bare-worker invocations, surface them to the user.
### 3. Ask about RSS profile
Use AskUserQuestion to confirm whether the user's bare workers need >2 GB RSS:
> "v0.22.14 changes `gbrain jobs work` to default `--max-rss 2048` (MB).
> Pre-v0.22.14 bare workers had no limit. If your worker handles large embed
> batches or multi-GB imports, the watchdog may now kill it. Do you want
> to keep the new 2 GB default, raise the limit, or opt out entirely?"
Options:
- **A) Keep 2 GB default (recommended for most)** — protects against memory
leaks; restarts on overflow; matches supervisor behavior.
- **B) Raise to N GB (specify N)** — pass `--max-rss <N*1024>` to the worker
invocation.
- **C) Opt out** — pass `--max-rss 0`.
### 4. Apply the user's choice
For each bare-worker invocation, edit the unit/cron/launchctl/script to add
the chosen `--max-rss` flag.
**systemd (~/.config/systemd/user/gbrain-worker.service):**
```ini
ExecStart=/usr/local/bin/gbrain jobs work --queue default --concurrency 3 --max-rss 4096
# Or to opt out: --max-rss 0
```
Then `systemctl --user daemon-reload && systemctl --user restart gbrain-worker`.
**cron (`crontab -e`):**
```cron
@reboot /usr/local/bin/gbrain jobs work --queue default --concurrency 3 --max-rss 4096
```
**Docker compose:**
```yaml
command: ["gbrain", "jobs", "work", "--queue", "default", "--concurrency", "3", "--max-rss", "4096"]
```
**launchctl (~/Library/LaunchAgents/com.user.gbrain-worker.plist):**
```xml
<key>ProgramArguments</key>
<array>
<string>/usr/local/bin/gbrain</string>
<string>jobs</string>
<string>work</string>
<string>--max-rss</string>
<string>4096</string>
</array>
```
Then `launchctl unload ... && launchctl load ...`.
### 5. (Optional) Tune health-check thresholds
The new opts default to sensible values (60s probe interval, 5min warn / 10min
exit, 3 DB failures). If you have specific SLAs, you can pass `--health-interval
<ms>` to adjust the probe cadence. Stall thresholds are not yet CLI-exposed
(only the API; CLI flags coming in a follow-up).
To disable self-monitoring entirely (e.g. you have your own external health
checker):
```bash
gbrain jobs work --health-interval 0 --max-rss 0
```
### 6. Verify
```bash
gbrain jobs stats # queue should be flowing normally
gbrain doctor --json | jq '.' # no critical warnings
ps -o rss= -p $(pgrep -f "gbrain jobs work") | awk '{print $1/1024 " MB"}'
```
Worker startup log line should now show health-check status:
```
Minion worker started (queue: default, concurrency: 3, watchdog: 2048MB, health-check: 60s)
```
If running under supervisor, you'll see the watchdog but NOT the `health-check:
60s` segment (because `GBRAIN_SUPERVISED=1` skips the child's self-monitor).
### 7. If anything fails
Open an issue at https://github.com/garrytan/gbrain/issues with:
- Output of `gbrain doctor`
- Contents of `~/.gbrain/upgrade-errors.jsonl` if it exists
- Your bare-worker invocation (systemd unit / cron line / Dockerfile snippet)
- Which step broke
+171
View File
@@ -0,0 +1,171 @@
---
version: 0.23.0
feature_pitch:
headline: "gbrain dream now actually dreams: conversation transcripts → reflections, originals, and 25-year patterns."
description: |
The maintenance cycle gains two new phases: `synthesize` and `patterns`.
The 8-phase order is now: lint → backlinks → sync → synthesize →
extract → patterns → embed → orphans.
Synthesize reads conversation transcripts (e.g., OpenClaw session corpus,
meeting transcripts) and writes brain-native pages: reflections to
`wiki/personal/reflections/...`, originals to `wiki/originals/ideas/...`,
timeline entries on existing people pages.
Patterns runs after extract (so the graph is fresh) and surfaces
recurring themes across reflections — when ≥3 reflections mention the
same motif, a pattern page is written to `wiki/personal/patterns/...`
citing every reflection that constitutes its evidence.
Hard guarantees: subagent writes are bounded to an explicit allow-list
(sourced from `_brain-filing-rules.json`). Edited transcripts produce
new slugs (content-hash suffix) — never silently overwrite. A 12-hour
cooldown bounds spend at ~$1-2/day under autopilot.
recipe: skills/maintain/SKILL.md
tiers: null
---
# v0.23.0 Migration: Dream cycle synthesize + patterns phases
**Audience: host agents reading this after `gbrain apply-migrations` has
run. The synthesize phase ships disabled by default — set
`dream.synthesize.session_corpus_dir` to opt in.**
## Mechanical migration: automatic, no action required
`gbrain upgrade` chains to `gbrain apply-migrations --yes`, which runs:
- **migration v25** — creates the `dream_verdicts` table:
`(file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB,
judged_at TIMESTAMPTZ, PRIMARY KEY(file_path, content_hash))`. Cache
for the cheap Haiku verdict so backfill re-runs skip already-judged
transcripts. RLS-enabled when running as a BYPASSRLS role.
The migration is idempotent. Safe to re-run.
## What changes for existing brains
`gbrain dream` (and `gbrain autopilot`) now run an 8-phase cycle:
```
lint → backlinks → sync → synthesize → extract → patterns → embed → orphans
```
If `dream.synthesize.enabled` is false (the default, post-migration), the
synthesize and patterns phases emit `status: "skipped", reason: "not_configured"`
and the cycle continues to the next phase. **Existing autopilot users see
zero behavior change** until they configure synthesize.
## To enable synthesize on your brain
Three steps. Take them when ready — there is no rush.
```bash
# 1. Point at the directory where your conversation transcripts live.
# OpenClaw stores session transcripts at memory/.dreams/session-corpus/<YYYY-MM-DD>.txt
# by default. If you have a different layout, point at that.
gbrain config set dream.synthesize.session_corpus_dir /path/to/transcripts
# 2. Enable the phase.
gbrain config set dream.synthesize.enabled true
# 3. Preview without spending real LLM tokens (runs cheap Haiku verdict only).
gbrain dream --phase synthesize --dry-run --json
```
## Tunables (sensible defaults; override only if needed)
```bash
# Skip transcripts shorter than this many characters (default 2000).
gbrain config set dream.synthesize.min_chars 2000
# Word-boundary regex patterns to skip. Default ["medical","therapy"].
# Each entry auto-wraps as \b<entry>\b — "medical" matches "medical advice"
# but NOT "comedical". Pass full regex (e.g. ^therapy:) for advanced patterns.
gbrain config set dream.synthesize.exclude_patterns '["medical","therapy"]'
# Synthesize model (default: claude-sonnet-4-6).
gbrain config set dream.synthesize.model claude-sonnet-4-6
# Hours between synthesize runs (the v1 spend cap; default 12 → ~$1-2/day).
gbrain config set dream.synthesize.cooldown_hours 12
# Patterns lookback window in days (default 30).
gbrain config set dream.patterns.lookback_days 30
# Minimum distinct reflections needed to name a pattern (default 3).
gbrain config set dream.patterns.min_evidence 3
```
## Allow-list source of truth
The synthesize subagent's allowed write paths live in
`skills/_brain-filing-rules.json` under `dream_synthesize_paths.globs`:
```json
{
"dream_synthesize_paths": {
"globs": [
"wiki/personal/reflections/*",
"wiki/originals/*",
"wiki/personal/patterns/*",
"wiki/people/*",
"dream-cycle-summaries/*"
]
}
}
```
Editing this list is the ONLY way to add a new directory the synthesizer
can write to. The subagent's `put_page` calls are gated server-side; even
on prompt-injection success the write is bounded to these prefixes.
## Slug discipline
Reflections: `wiki/personal/reflections/YYYY-MM-DD-<topic>-<hash[:6]>`
Originals: `wiki/originals/ideas/YYYY-MM-DD-<idea>-<hash[:6]>`
Patterns: `wiki/personal/patterns/<theme>`
Summary: `dream-cycle-summaries/YYYY-MM-DD`
The 6-char content-hash suffix on reflections / originals means an edited
transcript produces a NEW slug — the original reflection is preserved
alongside the new one. No silent overwrite.
Lowercase alphanumeric and hyphens only. NO underscores, NO file extensions.
## Provenance
Every put_page call from the synthesize subagent shows up in
`subagent_tool_executions` with full input. The orchestrator collects
slugs by querying that table — NOT `pages.updated_at` — so the cycle's
write list cannot accidentally include manual edits or sync output.
## What's deferred to v1.1
- **Auto git commit + push.** v1 writes markdown files to `brain_dir`
but does NOT `git add` / `commit` / `push`. Either commit yourself
or let `gbrain autopilot` handle it. v1.1 will add explicit
--commit / --push flags with handling for dirty worktree, staged
changes, auth failure, and non-fast-forward push.
- **Daily token budget cap.** Cooldown alone is the spend bound at v1
scale. If real-world telemetry surfaces a problem, v1.1 adds an
explicit `daily_token_budget` config.
- **Cross-modal pattern review.** Patterns currently runs against
reflections only. Future revision could roll up across reflections,
meetings, and timeline entries together.
## Verify after upgrade
```bash
# Schema migration applied?
gbrain doctor
# Phase ordering correct?
gbrain dream --help # shows the 8-phase pipeline
# Dry-run against a single transcript (cheap Haiku call only):
gbrain dream --phase synthesize --input /tmp/some-transcript.txt --dry-run --json
```
If any step fails, file an issue with `gbrain doctor` output and the
contents of `~/.gbrain/upgrade-errors.jsonl` if it exists.
+10 -2
View File
@@ -19,7 +19,7 @@ for (const op of operations) {
}
// CLI-only commands that bypass the operation layer
const CLI_ONLY = new Set(['init', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'storage', 'repos', 'code-def', 'code-refs', 'reindex-code', 'code-callers', 'code-callees', 'frontmatter', 'auth']);
const CLI_ONLY = new Set(['init', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'storage', 'repos', 'code-def', 'code-refs', 'reindex-code', 'code-callers', 'code-callees', 'frontmatter', 'auth', 'friction', 'claw-test']);
async function main() {
// Parse global flags (--quiet / --progress-json / --progress-interval)
@@ -343,6 +343,14 @@ async function handleCliOnly(command: string, args: string[]) {
await runSkillpack(args);
return;
}
if (command === 'friction') {
const { runFriction } = await import('./commands/friction.ts');
process.exit(runFriction(args));
}
if (command === 'claw-test') {
const { runClawTest } = await import('./commands/claw-test.ts');
process.exit(await runClawTest(args));
}
if (command === 'report') {
const { runReport } = await import('./commands/report.ts');
await runReport(args);
@@ -567,7 +575,7 @@ async function handleCliOnly(command: string, args: string[]) {
}
case 'repos': {
// v0.19.0: `gbrain repos ...` is an alias into the v0.18.0 sources
// subsystem. The repos abstraction (Wintermute's baseline) was
// subsystem. The repos abstraction (Garry's OpenClaw baseline) was
// redundant with sources and carried per-user config state that
// couldn't participate in federation / RLS / multi-tenancy. We
// keep the alias so scripts like `gbrain repos add .` keep
+9 -9
View File
@@ -57,11 +57,11 @@ export interface Flags {
skillsDir: string | null;
}
// Check 5 (trigger_routing_eval) landed in v0.17 (W2). Check 6
// (brain_filing) landed in v0.17 (W3). Array is now empty; the
// export stays as a stable public field of the --json envelope so
// downstream consumers that check `.deferred[]` keep working.
// Future deferred checks get appended here.
// Check 5 (trigger_routing_eval) and Check 6 (brain_filing) both
// shipped as real implementations in v0.19 (W2 + W3). Array is now
// empty; the export stays as a stable public field of the --json
// envelope so downstream consumers that check `.deferred[]` keep
// working. Future deferred checks get appended here.
export const DEFERRED: DeferredCheck[] = [];
const HELP_TEXT = `gbrain check-resolvable [options]
@@ -83,13 +83,13 @@ Exit codes:
0 clean (no errors; no warnings unless --strict)
1 errors present, OR (with --strict) warnings present
Check 5 (trigger routing eval) lands in v0.17 via W2: any
Check 5 (trigger routing eval) runs via W2: any
skills/<name>/routing-eval.jsonl fixtures are evaluated and routing
gaps surface as warnings.
Check 6 (brain filing) lands in v0.17 via W3: skills with
writes_pages: true are audited against skills/_brain-filing-rules.json.
No checks are deferred as of v0.17.
Check 6 (brain filing) runs via W3: skills with writes_pages: true
are audited against skills/_brain-filing-rules.json. No checks are
currently deferred.
`;
// ---------------------------------------------------------------------------
+424
View File
@@ -0,0 +1,424 @@
/**
* gbrain claw-test end-to-end "fresh user" test harness.
*
* Two tiers:
* gbrain claw-test scripted (no LLM, CI gate)
* gbrain claw-test --live --agent openclaw real agent, friction discovery
*
* Phases (scripted mode):
* setup install_brain import query extract verify render
*
* The harness sets GBRAIN_HOME=<tempdir> so the run is hermetic. Each child
* gbrain invocation runs with --progress-json and the harness captures stderr
* to assert expected_phases from scenario.json fired.
*
* See ~/.claude/plans/system-instruction-you-are-working-noble-biscuit.md
* for the full design rationale (D1D23 decisions).
*/
import { spawn } from 'child_process';
import { mkdtempSync, writeFileSync, mkdirSync, rmSync, existsSync } from 'fs';
import { join } from 'path';
import { tmpdir } from 'os';
import { randomBytes } from 'crypto';
import { logFriction, frictionDir } from '../core/friction.ts';
import { loadScenario, listScenarios, readBrief, type ScenarioConfig } from '../core/claw-test/scenarios.ts';
import { parseProgressEvents, verifyExpectedPhases } from '../core/claw-test/progress-tail.ts';
import { resolveAgentRunner, listRegisteredAgents, registerAgentRunner } from '../core/claw-test/agent-runner.ts';
import { OpenClawRunner } from '../core/claw-test/runners/openclaw.ts';
import { createTranscriptSink } from '../core/claw-test/transcript-capture.ts';
// Ensure built-in runners are registered.
registerAgentRunner('openclaw', () => new OpenClawRunner());
interface HarnessOpts {
scenario: string;
live: boolean;
agent: string;
keepTempdir: boolean;
listAgents: boolean;
help: boolean;
/** Path to the gbrain binary used to invoke child commands. Defaults to argv[0]. */
gbrainBin?: string;
}
interface PhaseOutcome {
phase: string;
exitCode: number;
durationMs: number;
stderrEvents: number;
stdoutTail: string;
stderrTail: string;
}
const TAIL_BYTES = 4_096;
const SUBPROCESS_TIMEOUT_MS = 5 * 60_000; // 5 minutes per phase
export async function runClawTest(args: string[]): Promise<number> {
const opts = parseArgs(args);
if (opts.help) {
printHelp();
return 0;
}
if (opts.listAgents) {
return cmdListAgents();
}
let scenario: ScenarioConfig;
try {
scenario = loadScenario(opts.scenario);
} catch (e) {
console.error(`scenario load failed: ${e instanceof Error ? e.message : String(e)}`);
const available = listScenarios();
if (available.length) console.error(`available scenarios: ${available.join(', ')}`);
return 2;
}
const runId = newRunId(opts.agent);
const runRoot = mkdtempSync(join(tmpdir(), `claw-test-${runId}-`));
const gbrainHome = runRoot; // configDir() appends '.gbrain' itself
const transcriptPath = join(runRoot, 'transcript.jsonl');
console.log(`run-id: ${runId}`);
console.log(`tempdir: ${runRoot}`);
// SIGINT/SIGTERM finalization (D11)
let interrupted = false;
const onSignal = () => {
interrupted = true;
try {
logFriction({
runId,
phase: 'harness',
message: 'run interrupted by signal',
kind: 'interrupted',
source: 'harness',
agent: opts.agent,
});
} catch { /* best effort */ }
};
process.once('SIGINT', onSignal);
process.once('SIGTERM', onSignal);
let exitCode = 0;
try {
if (opts.live) {
exitCode = await runLive(opts, scenario, { runId, runRoot, gbrainHome, transcriptPath });
} else {
exitCode = await runScripted(opts, scenario, { runId, runRoot, gbrainHome });
}
} finally {
process.off('SIGINT', onSignal);
process.off('SIGTERM', onSignal);
if (!opts.keepTempdir && !interrupted) {
try { rmSync(runRoot, { recursive: true, force: true }); } catch { /* best effort */ }
} else {
console.log(`tempdir kept at: ${runRoot}`);
}
}
// Always render at the end so the operator can immediately see the report.
console.log('---');
console.log(`friction log: ${join(frictionDir(), runId + '.jsonl')}`);
console.log(`render report: gbrain friction render --run-id ${runId}`);
if (interrupted) return 130;
return exitCode;
}
// ---------------------------------------------------------------------------
// Scripted mode
// ---------------------------------------------------------------------------
async function runScripted(
opts: HarnessOpts,
scenario: ScenarioConfig,
ctx: { runId: string; runRoot: string; gbrainHome: string },
): Promise<number> {
const childEnv: Record<string, string> = {
...process.env as Record<string, string>,
GBRAIN_HOME: ctx.gbrainHome,
GBRAIN_FRICTION_RUN_ID: ctx.runId,
};
const phases: { name: string; argv: string[] }[] = [];
// Phase 2: install_brain
phases.push({ name: 'install_brain', argv: ['init', '--pglite'] });
// Phase 3: import (only when scenario has a brain dir)
if (scenario.brainRelative) {
const brainDir = join(scenario.dir, scenario.brainRelative);
phases.push({ name: 'import', argv: ['import', brainDir, '--no-embed', '--progress-json'] });
}
// Phase 4: query (best-effort sanity)
phases.push({ name: 'query', argv: ['query', 'the'] });
// Phase 5: extract (positional argument is required: 'all' covers links + timeline)
phases.push({ name: 'extract', argv: ['extract', 'all', '--source', 'fs', '--progress-json'] });
// Phase 6: verify
phases.push({ name: 'verify', argv: ['doctor', '--json', '--progress-json'] });
// Pre-phase: upgrade scenario seeds the database
if (scenario.kind === 'upgrade' && scenario.seedRelative) {
const seedSql = join(scenario.dir, scenario.seedRelative, 'dump.sql');
if (existsSync(seedSql)) {
const dbPath = join(ctx.gbrainHome, '.gbrain', 'brain.pglite');
mkdirSync(join(ctx.gbrainHome, '.gbrain'), { recursive: true });
const { seedPgliteFromFile } = await import('../core/claw-test/seed-pglite.ts');
try {
await seedPgliteFromFile({ dbPath, sqlPath: seedSql });
console.log(`[seed] replayed ${seedSql}${dbPath}`);
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
logFriction({
runId: ctx.runId,
phase: 'seed',
message: `seed replay failed: ${msg}`,
severity: 'blocker',
source: 'harness',
agent: opts.agent,
});
return 1;
}
}
}
const allStderr: string[] = [];
const outcomes: PhaseOutcome[] = [];
for (const phase of phases) {
const outcome = await invokeGbrain(opts.gbrainBin ?? 'gbrain', phase.argv, ctx.runRoot, childEnv);
outcome.phase = phase.name;
outcomes.push(outcome);
allStderr.push(outcome.stderrTail);
if (outcome.exitCode !== 0) {
logFriction({
runId: ctx.runId,
phase: phase.name,
message: `command failed (exit ${outcome.exitCode}): gbrain ${phase.argv.join(' ')}`,
severity: 'error',
hint: outcome.stderrTail.trim().slice(0, 500),
source: 'harness',
agent: opts.agent,
});
return 1;
} else {
logFriction({
runId: ctx.runId,
phase: phase.name,
message: `phase complete in ${outcome.durationMs}ms`,
kind: 'phase-marker',
marker: 'end',
source: 'harness',
agent: opts.agent,
});
}
}
// Phase verification: collect all events from every captured stderr and assert coverage.
const events = allStderr.flatMap(parseProgressEvents);
const missing = verifyExpectedPhases(events, scenario.expectedPhases);
if (missing.length) {
for (const phaseName of missing) {
logFriction({
runId: ctx.runId,
phase: phaseName,
message: `expected progress event for "${phaseName}" never fired`,
severity: 'blocker',
hint: 'either the command did not run or it did not emit progress events; check phase log above',
source: 'harness',
agent: opts.agent,
});
}
return 1;
}
return 0;
}
// ---------------------------------------------------------------------------
// Live mode
// ---------------------------------------------------------------------------
async function runLive(
opts: HarnessOpts,
scenario: ScenarioConfig,
ctx: { runId: string; runRoot: string; gbrainHome: string; transcriptPath: string },
): Promise<number> {
let runner;
try {
runner = resolveAgentRunner(opts.agent);
} catch (e) {
console.error(e instanceof Error ? e.message : String(e));
return 2;
}
const detected = await runner.detect();
if (!detected.available) {
console.error(`agent "${opts.agent}" not available: ${detected.reason ?? 'unknown'}`);
logFriction({
runId: ctx.runId,
phase: 'agent_detect',
message: `agent ${opts.agent} not available: ${detected.reason ?? 'unknown'}`,
severity: 'blocker',
hint: opts.agent === 'openclaw' ? 'install openclaw or set OPENCLAW_BIN' : undefined,
source: 'harness',
agent: opts.agent,
});
return 2;
}
const sink = createTranscriptSink(ctx.transcriptPath);
const env: Record<string, string> = {
GBRAIN_HOME: ctx.gbrainHome,
GBRAIN_FRICTION_RUN_ID: ctx.runId,
};
const brief = readBrief(scenario);
let result;
try {
result = await runner.invoke({
cwd: ctx.runRoot,
brief,
env,
timeoutMs: SUBPROCESS_TIMEOUT_MS,
transcriptSink: sink,
});
} finally {
await sink.close();
}
if (result.exitCode !== 0) {
logFriction({
runId: ctx.runId,
phase: 'agent_invoke',
message: `agent exited with code ${result.exitCode} after ${result.durationMs}ms`,
severity: 'error',
source: 'harness',
agent: opts.agent,
});
return result.exitCode;
}
return 0;
}
// ---------------------------------------------------------------------------
// Subprocess helpers
// ---------------------------------------------------------------------------
function invokeGbrain(
bin: string,
argv: string[],
cwd: string,
env: Record<string, string>,
): Promise<PhaseOutcome> {
return new Promise((resolve) => {
const start = Date.now();
const child = spawn(bin, argv, { cwd, env, stdio: ['ignore', 'pipe', 'pipe'], shell: false });
const stdout: Buffer[] = [];
const stderr: Buffer[] = [];
child.stdout?.on('data', (b: Buffer) => stdout.push(b));
child.stderr?.on('data', (b: Buffer) => stderr.push(b));
child.on('error', (err) => {
const stderrJoined = Buffer.concat(stderr).toString('utf-8') + '\nspawn error: ' + err.message;
resolve({
phase: '',
exitCode: 127,
durationMs: Date.now() - start,
stderrEvents: 0,
stdoutTail: tailOf(Buffer.concat(stdout).toString('utf-8')),
stderrTail: tailOf(stderrJoined),
});
});
child.on('close', (code) => {
const stderrText = Buffer.concat(stderr).toString('utf-8');
resolve({
phase: '',
exitCode: typeof code === 'number' ? code : 1,
durationMs: Date.now() - start,
stderrEvents: parseProgressEvents(stderrText).length,
stdoutTail: tailOf(Buffer.concat(stdout).toString('utf-8')),
stderrTail: stderrText,
});
});
});
}
function tailOf(s: string): string {
if (s.length <= TAIL_BYTES) return s;
return s.slice(-TAIL_BYTES);
}
// ---------------------------------------------------------------------------
// Argv parsing + helpers
// ---------------------------------------------------------------------------
function parseArgs(args: string[]): HarnessOpts {
const out: HarnessOpts = {
scenario: 'fresh-install',
live: false,
agent: 'openclaw',
keepTempdir: false,
listAgents: false,
help: args.includes('--help') || args.includes('-h'),
gbrainBin: process.env.GBRAIN_BIN_OVERRIDE || process.execPath,
};
for (let i = 0; i < args.length; i++) {
const a = args[i];
if (a === '--live') out.live = true;
else if (a === '--keep-tempdir') out.keepTempdir = true;
else if (a === '--list-agents') out.listAgents = true;
else if (a === '--scenario') out.scenario = args[++i] ?? out.scenario;
else if (a === '--agent') out.agent = args[++i] ?? out.agent;
}
return out;
}
function newRunId(agent: string): string {
const now = new Date();
const ts = now.toISOString().replace(/[-:]/g, '').replace(/\..*/, '').replace('T', '-');
const suf = randomBytes(4).toString('hex');
return `claw-test-${ts}-${agent}-${suf}`;
}
function cmdListAgents(): number {
const names = listRegisteredAgents();
if (!names.length) {
console.log('no agents registered');
return 0;
}
for (const name of names) {
try {
const runner = resolveAgentRunner(name);
runner.detect().then((d) => {
const status = d.available ? `available at ${d.binPath}` : `unavailable: ${d.reason}`;
console.log(`${name}: ${status}`);
}).catch(() => { /* best effort */ });
} catch {
console.log(`${name}: (factory error)`);
}
}
return 0;
}
function printHelp() {
console.log(`gbrain claw-test — end-to-end claw-setup friction harness
Usage:
gbrain claw-test [--scenario <name>] [--live --agent <name>] [--keep-tempdir]
gbrain claw-test --list-agents
Defaults:
--scenario fresh-install
--agent openclaw (live mode only)
Scripted mode runs canonical commands without an LLM (CI gate).
Live mode spawns a real agent and lets it drive (~510 min, costs tokens).
Examples:
gbrain claw-test --scenario fresh-install
gbrain claw-test --scenario upgrade-from-v0.18 --keep-tempdir
gbrain claw-test --live --agent openclaw`);
}
+32
View File
@@ -774,6 +774,30 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
ORDER BY depth DESC
LIMIT 5
`;
// Subcheck 3 (v0.22.14): RSS-watchdog kills in the last 24h. Bare workers
// newly default to --max-rss 2048 (was 0); operators who run large embed
// or import jobs may see kills that didn't happen pre-v0.22.14. We surface
// a hint when this signature appears so the upgrade path is obvious.
// Signature: when the watchdog trips, gracefulShutdown('watchdog') aborts
// in-flight jobs with `new Error('watchdog')`. The worker's failJob path
// (worker.ts:660-664) writes `error_text = 'aborted: watchdog'` for any
// job in-flight at the moment of the kill.
//
// We deliberately DO NOT do a loose `ILIKE '%watchdog%'`:
// 1. Parent jobs that inherit `on_child_fail='fail_parent'` get
// `"child job N failed: aborted: watchdog"` — counting that
// double-counts (child + parent) for one watchdog event.
// 2. Any user error_text containing the word "watchdog" matches.
// Match the exact prefix `'aborted: watchdog'` to scope this purely to
// the worker's own kill signature.
const rssKillRows: Array<{ cnt: number }> = await sql`
SELECT count(*)::int AS cnt
FROM minion_jobs
WHERE status IN ('dead', 'failed')
AND finished_at > now() - interval '24 hours'
AND error_text = 'aborted: watchdog'
`;
const rssKillCount = rssKillRows[0]?.cnt ?? 0;
const problems: string[] = [];
if (stalledRows.length > 0) {
@@ -794,6 +818,14 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
`Fix: set maxWaiting on the submitter (or raise GBRAIN_QUEUE_WAITING_THRESHOLD).`
);
}
if (rssKillCount > 0) {
problems.push(
`${rssKillCount} job(s) dead-lettered for RSS-watchdog memory-limit kills in last 24h. ` +
`v0.22.14 changed the bare-worker --max-rss default from 0 (off) to 2048 MB. ` +
`Fix: raise the limit (e.g. \`gbrain jobs work --max-rss 4096\`) or opt out (\`--max-rss 0\`). ` +
`See skills/migrations/v0.22.14.md.`
);
}
if (problems.length === 0) {
checks.push({
+100 -6
View File
@@ -39,12 +39,28 @@ interface DreamArgs {
phase: CyclePhase | null;
dir: string | null;
help: boolean;
/** v0.21: ad-hoc transcript file path; implies --phase synthesize. */
inputFile: string | null;
/** v0.21: restrict synthesize to a single date (YYYY-MM-DD). */
date: string | null;
/** v0.21: backfill range start (YYYY-MM-DD). */
from: string | null;
/** v0.21: backfill range end (YYYY-MM-DD). */
to: string | null;
/**
* v0.23.2: disable the synthesize phase's self-consumption guard.
* Long-form flag name to discourage casual use; loud stderr warning fires when set.
* Never auto-applied for --input (codex finding #3).
*/
bypassDreamGuard: boolean;
}
const ISO_DATE_RE = /^\d{4}-\d{2}-\d{2}$/;
function parseArgs(args: string[]): DreamArgs {
const phaseIdx = args.indexOf('--phase');
const rawPhase = phaseIdx !== -1 ? args[phaseIdx + 1] : null;
const phase = rawPhase && (ALL_PHASES as string[]).includes(rawPhase)
let phase = rawPhase && (ALL_PHASES as string[]).includes(rawPhase)
? (rawPhase as CyclePhase)
: null;
if (rawPhase && !phase) {
@@ -55,6 +71,44 @@ function parseArgs(args: string[]): DreamArgs {
const dirIdx = args.indexOf('--dir');
const dir = dirIdx !== -1 ? args[dirIdx + 1] : null;
const inputIdx = args.indexOf('--input');
const inputFile = inputIdx !== -1 ? args[inputIdx + 1] ?? null : null;
const dateIdx = args.indexOf('--date');
const date = dateIdx !== -1 ? args[dateIdx + 1] ?? null : null;
if (date && !ISO_DATE_RE.test(date)) {
console.error(`--date must be YYYY-MM-DD; got "${date}"`);
process.exit(2);
}
const fromIdx = args.indexOf('--from');
const from = fromIdx !== -1 ? args[fromIdx + 1] ?? null : null;
if (from && !ISO_DATE_RE.test(from)) {
console.error(`--from must be YYYY-MM-DD; got "${from}"`);
process.exit(2);
}
const toIdx = args.indexOf('--to');
const to = toIdx !== -1 ? args[toIdx + 1] ?? null : null;
if (to && !ISO_DATE_RE.test(to)) {
console.error(`--to must be YYYY-MM-DD; got "${to}"`);
process.exit(2);
}
if (from && to && from > to) {
console.error(`--from (${from}) is after --to (${to}); empty range`);
process.exit(2);
}
// --input + --date / --from / --to is incoherent: --input is a single
// file, the date filters scan a directory.
if (inputFile && (date || from || to)) {
console.error('--input cannot be combined with --date / --from / --to');
process.exit(2);
}
// --input implies --phase synthesize.
if (inputFile && !phase) phase = 'synthesize';
return {
json: args.includes('--json'),
dryRun: args.includes('--dry-run'),
@@ -62,6 +116,11 @@ function parseArgs(args: string[]): DreamArgs {
phase,
dir,
help: args.includes('--help') || args.includes('-h'),
inputFile,
date,
from,
to,
bypassDreamGuard: args.includes('--unsafe-bypass-dream-guard'),
};
}
@@ -104,23 +163,49 @@ async function resolveBrainDir(
function printHelp() {
console.log(`Usage: gbrain dream [options]
Run one brain maintenance cycle: lint, backlinks, orphan sweep, sync,
extract, and embed. Designed for cron (exits when done).
Run one brain maintenance cycle. Eight phases:
lint -> backlinks -> sync -> synthesize -> extract -> patterns -> embed -> orphans
The synthesize + patterns phases (v0.21) consolidate yesterday's
conversation transcripts into reflections, originals, and cross-session
pattern pages. Designed for cron (exits when done).
Options:
--dry-run Preview all fixes without writing (fs or DB)
--dry-run Preview all fixes without writing. Note: synthesize
runs the cheap Haiku significance filter (caches
verdicts), but skips the Sonnet synthesis pass.
"--dry-run" does NOT mean "zero LLM calls."
--json Emit the CycleReport as JSON (agent-readable)
--phase <name> Run a single phase: ${ALL_PHASES.join(' | ')}
--pull git pull the brain repo before syncing (default: no pull)
--dir <path> Brain directory (default: configured brain)
--input <file> Synthesize a specific transcript file (implies
--phase synthesize). Bypasses corpus-dir scan.
--date YYYY-MM-DD Synthesize transcripts dated for one specific day.
--from YYYY-MM-DD Backfill range start (use with --to).
--to YYYY-MM-DD Backfill range end.
--unsafe-bypass-dream-guard
Disable the self-consumption guard. Use only when you
know the input file is NOT dream-cycle output but the
guard is firing. Loud stderr warning + cost reminder
fires every run.
--help, -h Show this help
Examples:
gbrain dream
gbrain dream --dry-run --json
gbrain dream --phase lint
gbrain dream --phase synthesize --input ~/transcripts/2026-04-25.txt
gbrain dream --phase synthesize --from 2026-04-01 --to 2026-04-25
0 2 * * * gbrain dream --json # nightly via cron
Configure synthesize:
gbrain config set dream.synthesize.session_corpus_dir /path/to/transcripts
gbrain config set dream.synthesize.enabled true
Related:
gbrain autopilot --install # continuous maintenance as a daemon
gbrain autopilot # same maintenance cycle, scheduled
@@ -165,10 +250,14 @@ function printHuman(report: CycleReport) {
const t = report.totals;
const hasTotals =
t.lint_fixes > 0 || t.backlinks_added > 0 || t.pages_synced > 0 ||
t.pages_extracted > 0 || t.pages_embedded > 0 || t.orphans_found > 0;
t.pages_extracted > 0 || t.pages_embedded > 0 || t.orphans_found > 0 ||
t.transcripts_processed > 0 || t.synth_pages_written > 0 || t.patterns_written > 0;
if (hasTotals) {
console.log(
` totals: lint=${t.lint_fixes} backlinks=${t.backlinks_added} synced=${t.pages_synced} extracted=${t.pages_extracted} embedded=${t.pages_embedded} orphans=${t.orphans_found}`,
` totals: lint=${t.lint_fixes} backlinks=${t.backlinks_added} synced=${t.pages_synced} ` +
`extracted=${t.pages_extracted} embedded=${t.pages_embedded} orphans=${t.orphans_found} ` +
`synth_transcripts=${t.transcripts_processed} synth_pages=${t.synth_pages_written} ` +
`patterns=${t.patterns_written}`,
);
}
}
@@ -191,6 +280,11 @@ export async function runDream(engine: BrainEngine | null, args: string[]): Prom
dryRun: opts.dryRun,
pull: opts.pull,
phases,
synthInputFile: opts.inputFile ?? undefined,
synthDate: opts.date ?? undefined,
synthFrom: opts.from ?? undefined,
synthTo: opts.to ?? undefined,
synthBypassDreamGuard: opts.bypassDreamGuard,
});
if (opts.json) {
+185
View File
@@ -0,0 +1,185 @@
/**
* gbrain friction friction reporter CLI.
*
* Four subcommands in v1 (analytical/clustering ones move to v1.1):
* gbrain friction log Append a friction or delight entry
* gbrain friction render Render a run as markdown or JSON
* gbrain friction list List recent runs with counts
* gbrain friction summary Side-by-side friction + delight summary
*
* Subcommands stay thin ( ~30 LOC each). Core logic lives in src/core/friction.ts.
*
* The CLI is dispatched from src/cli.ts. See `gbrain friction --help`.
*/
import {
logFriction, readFriction, listRuns, renderReport, renderSummary,
activeRunId, frictionFile,
type FrictionKind, type FrictionSeverity,
} from '../core/friction.ts';
const VALID_KINDS = new Set<FrictionKind>(['friction', 'delight', 'phase-marker', 'interrupted']);
const VALID_SEVERITIES = new Set<FrictionSeverity>(['confused', 'error', 'blocker', 'nit']);
export function runFriction(args: string[]): number {
const [sub, ...rest] = args;
switch (sub) {
case 'log': return cmdLog(rest);
case 'render': return cmdRender(rest);
case 'list': return cmdList(rest);
case 'summary': return cmdSummary(rest);
case undefined:
case '--help':
case '-h':
printHelp();
return 0;
default:
console.error(`unknown subcommand: ${sub}`);
printHelp();
return 2;
}
}
// ---------------------------------------------------------------------------
// log
// ---------------------------------------------------------------------------
function cmdLog(args: string[]): number {
const flags = parseFlags(args);
const phase = flags.string('--phase');
const message = flags.string('--message');
if (!phase || !message) {
console.error('usage: gbrain friction log --phase <name> --message <text> [--severity ...] [--hint ...] [--kind ...] [--run-id ...]');
return 2;
}
const kind = (flags.string('--kind') ?? 'friction') as FrictionKind;
if (!VALID_KINDS.has(kind)) {
console.error(`invalid --kind ${kind}; must be one of: ${[...VALID_KINDS].join(', ')}`);
return 2;
}
const severityRaw = flags.string('--severity');
const severity = severityRaw as FrictionSeverity | undefined;
if (severity && !VALID_SEVERITIES.has(severity)) {
console.error(`invalid --severity ${severity}; must be one of: ${[...VALID_SEVERITIES].join(', ')}`);
return 2;
}
try {
logFriction({
phase,
message,
kind,
severity,
hint: flags.string('--hint'),
runId: flags.string('--run-id'),
agent: flags.string('--agent'),
source: 'claw',
});
} catch (e) {
console.error(`friction log failed: ${e instanceof Error ? e.message : String(e)}`);
return 1;
}
return 0;
}
// ---------------------------------------------------------------------------
// render
// ---------------------------------------------------------------------------
function cmdRender(args: string[]): number {
const flags = parseFlags(args);
const runId = flags.string('--run-id') ?? activeRunId();
const json = flags.bool('--json');
const format = json ? 'json' : 'md';
const transcripts = flags.bool('--transcripts');
const noRedact = flags.bool('--no-redact');
// --redact is the default for md output; --no-redact disables.
const redact = noRedact ? false : (format === 'md');
try {
const out = renderReport(runId, {
format,
redact,
transcriptPath: transcripts ? flags.string('--transcript-path') ?? undefined : undefined,
});
process.stdout.write(out + '\n');
return 0;
} catch (e) {
console.error(`friction render failed: ${e instanceof Error ? e.message : String(e)}`);
return 1;
}
}
// ---------------------------------------------------------------------------
// list
// ---------------------------------------------------------------------------
function cmdList(args: string[]): number {
const flags = parseFlags(args);
const json = flags.bool('--json');
const runs = listRuns();
if (json) {
console.log(JSON.stringify(runs, null, 2));
return 0;
}
if (runs.length === 0) {
console.log('no runs yet');
return 0;
}
for (const r of runs) {
const interrupted = r.counts.interrupted ? ' (interrupted)' : '';
const sev = Object.entries(r.counts.bySeverity).map(([k, v]) => `${k}=${v}`).join(' ');
console.log(`${r.runId}${interrupted} friction=${r.counts.friction} delight=${r.counts.delight} ${sev}`);
}
return 0;
}
// ---------------------------------------------------------------------------
// summary
// ---------------------------------------------------------------------------
function cmdSummary(args: string[]): number {
const flags = parseFlags(args);
const runId = flags.string('--run-id') ?? activeRunId();
const json = flags.bool('--json');
try {
const out = renderSummary(runId, { format: json ? 'json' : 'md' });
process.stdout.write(out + '\n');
return 0;
} catch (e) {
console.error(`friction summary failed: ${e instanceof Error ? e.message : String(e)}`);
return 1;
}
}
// ---------------------------------------------------------------------------
// Helpers
// ---------------------------------------------------------------------------
function parseFlags(args: string[]) {
return {
string(flag: string): string | undefined {
const idx = args.indexOf(flag);
return idx === -1 ? undefined : args[idx + 1];
},
bool(flag: string): boolean {
return args.includes(flag);
},
};
}
function printHelp() {
console.log(`gbrain friction — friction reporter
Subcommands:
log Append a friction or delight entry to the active run
render Render a run's entries as markdown (default) or JSON
list List recent runs with friction/delight counts
summary Two-column summary of friction + delight for a run
Examples:
gbrain friction log --severity confused --phase install --message "init didn't say which engine"
gbrain friction render --run-id claw-test-20260428-... --transcripts
gbrain friction list --json
gbrain friction summary
Run-id resolution: --run-id > $GBRAIN_FRICTION_RUN_ID > 'standalone'.`);
}
+193 -1
View File
@@ -49,6 +49,10 @@ export async function runFrontmatter(args: string[]): Promise<void> {
}
return;
}
if (sub === 'generate') {
await runGenerate(rest);
return;
}
if (sub === 'install-hook') {
const { runFrontmatterInstallHook } = await import('./frontmatter-install-hook.ts');
await runFrontmatterInstallHook(rest);
@@ -71,10 +75,11 @@ async function connectEngineForAudit(): Promise<BrainEngine> {
}
function printHelp() {
console.log(`gbrain frontmatter — frontmatter validation, audit, and auto-repair
console.log(`gbrain frontmatter — frontmatter validation, audit, auto-repair, and generation
Usage:
gbrain frontmatter validate <path> [--json] [--fix] [--dry-run]
gbrain frontmatter generate <path> [--fix] [--dry-run] [--json]
gbrain frontmatter audit [--source <id>] [--json]
gbrain frontmatter install-hook [--source <id>] [--force] [--uninstall]
@@ -91,6 +96,26 @@ validate
--dry-run Preview --fix without writing.
--json Emit a JSON envelope on stdout.
generate
Synthesize frontmatter for files that have none (MISSING_OPEN). Uses
directory-aware rules to infer type, title, date, source, and tags from
the filesystem path and file content. Zero LLM calls, fully deterministic.
Without --fix: dry-run preview showing what would be generated.
With --fix: writes frontmatter to files (with .bak safety backups).
Rules are defined in src/core/frontmatter-inference.ts DIRECTORY_RULES.
Add new directory conventions by adding rules to the table.
Examples:
gbrain frontmatter generate /path/to/brain # preview all
gbrain frontmatter generate /path/to/brain --fix # write all
gbrain frontmatter generate /path/to/brain/people/ --fix # just people/
--fix Write generated frontmatter to files (.bak safety backups).
--dry-run Preview without writing (default when --fix is omitted).
--json Emit JSON output.
audit
Read-only scan across all registered sources (or one with --source <id>).
Reports per-source counts grouped by error code. Use this in CI or doctor
@@ -297,3 +322,170 @@ function printAuditHumanReport(report: AuditReport): void {
console.log(`\nFix with: gbrain frontmatter validate <source-path> --fix`);
}
}
// ---------------------------------------------------------------------------
// generate — synthesize frontmatter for files that have none
// ---------------------------------------------------------------------------
async function runGenerate(args: string[]): Promise<void> {
const targetPath = args.find(a => !a.startsWith('-'));
const doFix = args.includes('--fix');
const dryRun = args.includes('--dry-run');
const jsonOut = args.includes('--json');
if (!targetPath) {
console.error('error: gbrain frontmatter generate requires a <path> argument');
console.error('usage: gbrain frontmatter generate <path> [--fix] [--dry-run] [--json]');
process.exitCode = 1;
return;
}
const { inferFrontmatter, serializeFrontmatter } = await import('../core/frontmatter-inference.ts');
const { resolve, relative, join, basename } = await import('path');
const { readFileSync, writeFileSync, copyFileSync, statSync, readdirSync, lstatSync } = await import('fs');
const rootPath = resolve(targetPath);
const isDir = statSync(rootPath).isDirectory();
// Find the brain root — walk up from targetPath looking for .git or known brain markers.
// Inference rules match against brain-root-relative paths (e.g., "people/alice.md").
let brainRoot = rootPath;
if (isDir) {
let candidate = rootPath;
for (let i = 0; i < 10; i++) {
try {
statSync(join(candidate, '.git'));
brainRoot = candidate;
break;
} catch {
const parent = resolve(candidate, '..');
if (parent === candidate) break;
candidate = parent;
}
}
}
interface GenerateResult {
path: string;
type: string;
title: string;
date?: string;
rule: string;
}
const results: GenerateResult[] = [];
let scanned = 0;
let skipped = 0;
let generated = 0;
let written = 0;
function processFile(absPath: string, relPath: string) {
scanned++;
if (!absPath.endsWith('.md')) return;
// Skip symlinks
try { if (lstatSync(absPath).isSymbolicLink()) return; } catch { return; }
let content: string;
try { content = readFileSync(absPath, 'utf-8'); } catch { return; }
const inferred = inferFrontmatter(relPath, content);
if (inferred.skipped) {
skipped++;
return;
}
generated++;
results.push({
path: relPath,
type: inferred.type,
title: inferred.title,
date: inferred.date,
rule: inferred.matchedRule || '(default)',
});
if (doFix && !dryRun) {
const fm = serializeFrontmatter(inferred);
const newContent = fm + '\n' + content;
// Safety: write .bak first
copyFileSync(absPath, absPath + '.bak');
writeFileSync(absPath, newContent, 'utf-8');
written++;
}
}
function walkDir(dir: string, rootForRel: string) {
let entries: string[];
try { entries = readdirSync(dir); } catch { return; }
for (const entry of entries) {
if (entry === '.git' || entry === 'node_modules' || entry === '.obsidian') continue;
const abs = join(dir, entry);
try {
const stat = statSync(abs);
if (stat.isDirectory()) {
walkDir(abs, rootForRel);
} else if (stat.isFile() && entry.endsWith('.md')) {
processFile(abs, relative(rootForRel, abs));
}
} catch { /* skip unreadable */ }
}
}
if (isDir) {
walkDir(rootPath, brainRoot);
} else {
const relPath = relative(brainRoot, rootPath) || basename(rootPath);
processFile(rootPath, relPath);
}
// Output
if (jsonOut) {
console.log(JSON.stringify({
scanned,
skipped,
generated,
written,
dryRun: !doFix || dryRun,
results: results.slice(0, 100), // Cap JSON output
totalResults: results.length,
}, null, 2));
return;
}
// Human-readable output
const mode = doFix && !dryRun ? 'WRITE' : 'DRY-RUN';
console.log(`\nFrontmatter generation (${mode})`);
console.log(` Scanned: ${scanned} files`);
console.log(` Already have frontmatter: ${skipped}`);
console.log(` Would generate: ${generated}`);
if (doFix && !dryRun) {
console.log(` Written: ${written} (with .bak backups)`);
}
// Show sample by type
const byType: Record<string, number> = {};
for (const r of results) {
byType[r.type] = (byType[r.type] || 0) + 1;
}
if (Object.keys(byType).length > 0) {
console.log(`\n By type:`);
for (const [type, count] of Object.entries(byType).sort(([, a], [, b]) => b - a)) {
console.log(` ${type}: ${count}`);
}
}
// Show first 10 examples
if (results.length > 0 && (!doFix || dryRun)) {
console.log(`\n Examples:`);
for (const r of results.slice(0, 10)) {
console.log(` ${r.path}`);
console.log(` → type: ${r.type}, title: "${r.title}"${r.date ? `, date: ${r.date}` : ''} [rule: ${r.rule}]`);
}
if (results.length > 10) {
console.log(` ... and ${results.length - 10} more`);
}
if (!doFix) {
console.log(`\n To write: gbrain frontmatter generate ${targetPath} --fix`);
}
}
}
+59 -32
View File
@@ -1,10 +1,10 @@
import { readdirSync, lstatSync, existsSync, writeFileSync, readFileSync, unlinkSync } from 'fs';
import { execFileSync } from 'child_process';
import { join, relative } from 'path';
import { cpus, totalmem, homedir } from 'os';
import { cpus, totalmem } from 'os';
import type { BrainEngine } from '../core/engine.ts';
import { importFile } from '../core/import-file.ts';
import { loadConfig } from '../core/config.ts';
import { loadConfig, gbrainPath } from '../core/config.ts';
import { createProgress } from '../core/progress.ts';
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
@@ -34,7 +34,17 @@ export async function runImport(engine: BrainEngine, args: string[], opts: { com
const jsonOutput = args.includes('--json');
const workersIdx = args.indexOf('--workers');
const workersArg = workersIdx !== -1 ? args[workersIdx + 1] : null;
const workerCount = workersArg ? parseInt(workersArg, 10) : 1;
// v0.22.13 (PR #490 Q2): shared parseWorkers helper rejects bad input
// (--workers 0, -3, "foo") with a loud error instead of silently falling
// through to 1. Mirrors sync.ts's flag handling.
const { parseWorkers } = await import('../core/sync-concurrency.ts');
let workerCount: number;
try {
workerCount = parseWorkers(workersArg ?? undefined) ?? 1;
} catch (e) {
console.error(e instanceof Error ? e.message : String(e));
process.exit(1);
}
// Find dir: first non-flag arg that isn't a value for --workers
const flagValues = new Set<number>();
if (workersIdx !== -1) flagValues.add(workersIdx + 1);
@@ -51,7 +61,7 @@ export async function runImport(engine: BrainEngine, args: string[], opts: { com
console.log(`Found ${allFiles.length} markdown files`);
// Resume from checkpoint if available
const checkpointPath = join(homedir(), '.gbrain', 'import-checkpoint.json');
const checkpointPath = gbrainPath('import-checkpoint.json');
let files = allFiles;
let resumeIndex = 0;
@@ -127,7 +137,7 @@ export async function runImport(engine: BrainEngine, args: string[], opts: { com
// Save checkpoint every 100 files — track completed file set, not just a counter
if (processed % 100 === 0) {
try {
const cpDir = join(homedir(), '.gbrain');
const cpDir = gbrainPath();
if (!existsSync(cpDir)) { const { mkdirSync } = await import('fs'); mkdirSync(cpDir, { recursive: true }); }
writeFileSync(checkpointPath, JSON.stringify({
dir, totalFiles: allFiles.length,
@@ -141,40 +151,57 @@ export async function runImport(engine: BrainEngine, args: string[], opts: { com
}
if (actualWorkers > 1) {
// Parallel: create per-worker engine instances with small pool
// PGLite is single-connection, so parallel workers are only for Postgres
// v0.22.13 (PR #490 A1 + Q3): use engine.kind discriminator (not config.engine
// string sniff) and fall back to serial when database_url is unset. Both
// checks belt-and-suspenders so we never crash on a null assertion.
const config = loadConfig();
if (config?.engine === 'pglite') {
// PGLite: sequential import through single engine
if (engine.kind === 'pglite' || !config?.database_url) {
for (const file of files) {
await processFile(engine, file);
}
} else {
const { PostgresEngine } = await import('../core/postgres-engine.ts');
const { resolvePoolSize } = await import('../core/db.ts');
// Default per-worker pool is 2 (small, parallel import case). Users on
// constrained poolers (e.g. Supabase port 6543) can cap below this via
// GBRAIN_POOL_SIZE=1.
const workerPoolSize = Math.min(2, resolvePoolSize(2));
const workerEngines = await Promise.all(
Array.from({ length: actualWorkers }, async () => {
const eng = new PostgresEngine();
await eng.connect({ database_url: config!.database_url!, poolSize: workerPoolSize });
return eng;
})
);
const { PostgresEngine } = await import('../core/postgres-engine.ts');
const { resolvePoolSize } = await import('../core/db.ts');
// Default per-worker pool is 2 (small, parallel import case). Users on
// constrained poolers (e.g. Supabase port 6543) can cap below this via
// GBRAIN_POOL_SIZE=1.
const workerPoolSize = Math.min(2, resolvePoolSize(2));
const databaseUrl = config.database_url;
// Thread-safe queue: use an atomic index counter instead of array.shift()
let queueIndex = 0;
await Promise.all(workerEngines.map(async (eng) => {
while (true) {
const idx = queueIndex++;
if (idx >= files.length) break;
await processFile(eng, files[idx]);
// v0.22.13 (PR #490 A2): connect workers serially so a partial failure
// leaves us with the connected ones already pushed onto workerEngines
// for the finally-block cleanup. The prior Promise.all could leak any
// engine that connected before another's connect() rejected.
const workerEngines: InstanceType<typeof PostgresEngine>[] = [];
try {
for (let i = 0; i < actualWorkers; i++) {
const eng = new PostgresEngine();
await eng.connect({ database_url: databaseUrl, poolSize: workerPoolSize });
workerEngines.push(eng);
}
// Thread-safe queue: atomic index counter (JS is single-threaded; the
// read-then-increment happens between awaits so no lock is needed).
let queueIndex = 0;
await Promise.all(workerEngines.map(async (eng) => {
while (true) {
const idx = queueIndex++;
if (idx >= files.length) break;
await processFile(eng, files[idx]);
}
}));
} finally {
// v0.22.13 (PR #490 A2): try/finally guarantees cleanup even when the
// worker loop throws. Each disconnect is best-effort — one failing
// disconnect must not strand the others.
await Promise.all(
workerEngines.map(e =>
e.disconnect().catch((err: unknown) =>
console.error(` worker disconnect failed: ${err instanceof Error ? err.message : String(err)}`),
),
),
);
}
}));
await Promise.all(workerEngines.map(e => e.disconnect()));
} // end else (postgres parallel)
} else {
// Sequential: use the provided engine
+2 -2
View File
@@ -6,7 +6,7 @@ import { homedir } from 'os';
const __filename = fileURLToPath(import.meta.url);
const __dirname = dirname(__filename);
import { saveConfig, loadConfig, toEngineConfig, type GBrainConfig } from '../core/config.ts';
import { saveConfig, loadConfig, toEngineConfig, gbrainPath, type GBrainConfig } from '../core/config.ts';
import { createEngine } from '../core/engine-factory.ts';
export async function runInit(args: string[]) {
@@ -103,7 +103,7 @@ async function initMigrateOnly(opts: { jsonOutput: boolean }) {
}
async function initPGLite(opts: { jsonOutput: boolean; apiKey: string | null; customPath: string | null }) {
const dbPath = opts.customPath || join(homedir(), '.gbrain', 'brain.pglite');
const dbPath = opts.customPath || gbrainPath('brain.pglite');
console.log(`Setting up local brain with PGLite (no server needed)...`);
const engine = await createEngine({ engine: 'pglite' });
+2 -1
View File
@@ -23,6 +23,7 @@ import matter from 'gray-matter';
import { readFileSync, existsSync, writeFileSync, mkdirSync, readdirSync } from 'fs';
import { join, basename } from 'path';
import { homedir } from 'os';
import { gbrainPath } from '../core/config.ts';
import { execSync } from 'child_process';
// --- Types ---
@@ -512,7 +513,7 @@ function findRecipe(id: string): ParsedRecipe | null {
// --- Heartbeat ---
function heartbeatDir(id: string): string {
return join(homedir(), '.gbrain', 'integrations', id);
return gbrainPath('integrations', id);
}
function heartbeatPath(id: string): string {
+24 -25
View File
@@ -25,10 +25,9 @@
*/
import { appendFileSync, existsSync, readFileSync, mkdirSync, writeFileSync } from 'fs';
import { homedir } from 'os';
import { join, dirname } from 'path';
import { dirname } from 'path';
import { loadConfig, toEngineConfig } from '../core/config.ts';
import { loadConfig, toEngineConfig, gbrainPath } from '../core/config.ts';
import { createEngine } from '../core/engine-factory.ts';
import type { BrainEngine } from '../core/engine.ts';
import * as db from '../core/db.ts';
@@ -45,10 +44,10 @@ import { tweetCitation } from '../core/output/scaffold.ts';
// Paths
// ---------------------------------------------------------------------------
const GBRAIN_DIR = join(homedir(), '.gbrain');
const REVIEW_FILE = join(GBRAIN_DIR, 'integrity-review.md');
const LOG_FILE = join(GBRAIN_DIR, 'integrity.log.jsonl');
const PROGRESS_FILE = join(GBRAIN_DIR, 'integrity-progress.jsonl');
// Lazy: GBRAIN_HOME may be set after module load.
const getReviewFile = () => gbrainPath('integrity-review.md');
const getLogFile = () => gbrainPath('integrity.log.jsonl');
const getProgressFile = () => gbrainPath('integrity-progress.jsonl');
// ---------------------------------------------------------------------------
// Bare-tweet detection
@@ -158,9 +157,9 @@ interface ProgressEntry {
}
function loadProgress(): Set<string> {
if (!existsSync(PROGRESS_FILE)) return new Set();
if (!existsSync(getProgressFile())) return new Set();
const seen = new Set<string>();
const content = readFileSync(PROGRESS_FILE, 'utf-8');
const content = readFileSync(getProgressFile(), 'utf-8');
for (const line of content.split('\n')) {
if (!line.trim()) continue;
try {
@@ -174,12 +173,12 @@ function loadProgress(): Set<string> {
}
function appendProgress(entry: ProgressEntry): void {
ensureDir(PROGRESS_FILE);
appendFileSync(PROGRESS_FILE, JSON.stringify(entry) + '\n', 'utf-8');
ensureDir(getProgressFile());
appendFileSync(getProgressFile(), JSON.stringify(entry) + '\n', 'utf-8');
}
function clearProgress(): void {
if (existsSync(PROGRESS_FILE)) writeFileSync(PROGRESS_FILE, '', 'utf-8');
if (existsSync(getProgressFile())) writeFileSync(getProgressFile(), '', 'utf-8');
}
function ensureDir(path: string): void {
@@ -213,7 +212,7 @@ export async function runIntegrity(args: string[]): Promise<void> {
}
if (sub === 'reset-progress') {
clearProgress();
console.log('Cleared progress log:', PROGRESS_FILE);
console.log('Cleared progress log:', getProgressFile());
return;
}
@@ -409,7 +408,7 @@ async function cmdAuto(args: string[]): Promise<void> {
process.exit(1);
}
ensureDir(GBRAIN_DIR);
ensureDir(gbrainPath());
const engine = await connect();
const registry = getDefaultRegistry();
@@ -548,9 +547,9 @@ async function cmdAuto(args: string[]): Promise<void> {
console.log(`Review queue (≥${reviewLower} <${confidenceThreshold}): ${bucketReview}`);
console.log(`Skipped (<${reviewLower}): ${bucketSkip}`);
if (bucketErr > 0) console.log(`Resolver errors: ${bucketErr}`);
console.log(`\nReview queue: ${REVIEW_FILE}`);
console.log(`Skipped log: ${LOG_FILE}`);
console.log(`Progress: ${PROGRESS_FILE}`);
console.log(`\nReview queue: ${getReviewFile()}`);
console.log(`Skipped log: ${getLogFile()}`);
console.log(`Progress: ${getProgressFile()}`);
} finally {
await engine.disconnect();
}
@@ -561,15 +560,15 @@ async function cmdAuto(args: string[]): Promise<void> {
// ---------------------------------------------------------------------------
function cmdReview(): void {
if (!existsSync(REVIEW_FILE)) {
if (!existsSync(getReviewFile())) {
console.log(`No review queue yet. Run: gbrain integrity auto --confidence 0.8`);
return;
}
const content = readFileSync(REVIEW_FILE, 'utf-8');
const content = readFileSync(getReviewFile(), 'utf-8');
const count = (content.match(/^## /gm) ?? []).length;
console.log(`Review queue: ${REVIEW_FILE}`);
console.log(`Review queue: ${getReviewFile()}`);
console.log(`Entries: ${count}`);
console.log(`\nOpen with: $EDITOR ${REVIEW_FILE}`);
console.log(`\nOpen with: $EDITOR ${getReviewFile()}`);
}
// ---------------------------------------------------------------------------
@@ -650,7 +649,7 @@ interface ReviewArgs {
}
function appendReview(args: ReviewArgs): void {
ensureDir(REVIEW_FILE);
ensureDir(getReviewFile());
const { slug, hit, result, handle } = args;
const block = [
`## ${slug}:${hit.line} (confidence ${result.confidence.toFixed(2)})`,
@@ -664,12 +663,12 @@ function appendReview(args: ReviewArgs): void {
'---',
'',
].join('\n');
appendFileSync(REVIEW_FILE, block, 'utf-8');
appendFileSync(getReviewFile(), block, 'utf-8');
}
interface SkipArgs { slug: string; hit: BareTweetHit; reason: string }
function logSkip(args: SkipArgs): void {
ensureDir(LOG_FILE);
ensureDir(getLogFile());
const entry = {
timestamp: new Date().toISOString(),
slug: args.slug,
@@ -678,7 +677,7 @@ function logSkip(args: SkipArgs): void {
raw: args.hit.rawLine.slice(0, 200),
reason: args.reason,
};
appendFileSync(LOG_FILE, JSON.stringify(entry) + '\n', 'utf-8');
appendFileSync(getLogFile(), JSON.stringify(entry) + '\n', 'utf-8');
}
// ---------------------------------------------------------------------------
+129 -18
View File
@@ -33,14 +33,14 @@ export function parseMaxWaitingFlag(args: string[]): number | undefined {
}
/** Parse `--max-rss N` (MB). Returns:
* - 0 if the flag is absent (no watchdog by default for bare `jobs work`)
* - undefined if the flag is absent (caller decides the default)
* - 0 if `--max-rss 0` (explicit disable)
* - the value if >= 256
* Errors and exits the process if the flag is non-numeric, negative, or
* positive but < 256 (likely a GB-vs-MB unit-confusion typo). */
export function parseMaxRssFlag(args: string[]): number {
export function parseMaxRssFlag(args: string[]): number | undefined {
const raw = parseFlag(args, '--max-rss');
if (raw === undefined) return 0;
if (raw === undefined) return undefined;
const parsed = parseInt(raw, 10);
if (!Number.isFinite(parsed) || parsed < 0) {
console.error(`Error: --max-rss must be a non-negative integer (MB), got "${raw}"`);
@@ -133,6 +133,7 @@ USAGE
gbrain jobs stats
gbrain jobs smoke
gbrain jobs work [--queue Q] [--concurrency N] [--max-rss MB]
[--health-interval MS]
gbrain jobs supervisor [start] [--detach] [--json]
[--concurrency N] [--queue Q] [--pid-file PATH]
[--max-crashes N] [--health-interval N]
@@ -314,8 +315,15 @@ HANDLER TYPES (built in)
if (follow) {
console.log(`Job #${job.id} submitted (${name}). Executing inline...`);
// Inline execution: run the job in this process
const worker = new MinionWorker(engine, { queue: queueName, pollInterval: 100 });
// Inline execution: run the job in this process. Disable the
// self-health-check timer — inline flows are one-shot and don't have
// a process manager to restart them. With the timer enabled and no
// 'unhealthy' listener, a DB blip would trip emitUnhealthy's
// no-listener fallback and call process.exit(1) from inside the
// library, killing the user's CLI session.
const worker = new MinionWorker(engine, {
queue: queueName, pollInterval: 100, healthCheckInterval: 0,
});
// Register built-in handlers
await registerBuiltinHandlers(worker, engine);
@@ -489,7 +497,11 @@ HANDLER TYPES (built in)
const sigkillRescue = hasFlag(args, '--sigkill-rescue');
const wedgeRescue = hasFlag(args, '--wedge-rescue');
const worker = new MinionWorker(engine, { queue: 'smoke', pollInterval: 100 });
// Smoke harness is short-lived and has no listener — disable the health
// timer so the no-listener fallback can't trip process.exit(1) mid-test.
const worker = new MinionWorker(engine, {
queue: 'smoke', pollInterval: 100, healthCheckInterval: 0,
});
worker.register('noop', async () => ({ ok: true, at: new Date().toISOString() }));
const job = await queue.add('noop', {}, { queue: 'smoke', max_attempts: 1 });
@@ -638,19 +650,69 @@ HANDLER TYPES (built in)
const queueName = parseFlag(args, '--queue') ?? 'default';
const concurrency = resolveWorkerConcurrency(args);
// --max-rss is opt-in for bare `gbrain jobs work` — preserves pre-v0.21 behavior
// for operators with legitimately large embed/import working sets. The supervisor
// path injects a default 2048; this code path does not.
const maxRssMb = parseMaxRssFlag(args);
// --max-rss defaults to 2048 for bare workers (matching supervisor default).
// This catches memory-leak stalls that previously went undetected without
// a supervisor. Operators can opt out with `--max-rss 0`.
const maxRssExplicit = parseMaxRssFlag(args);
const maxRssMb = maxRssExplicit ?? 2048;
// --health-interval: self-health-check period in ms. 0 disables. Default: 60_000 (60s).
// Provides DB liveness probes + stall detection for bare workers.
// Automatically skipped when running under a supervisor (GBRAIN_SUPERVISED=1).
// Validated aggressively (parity with --max-rss): reject NaN/negative/non-integer
// values, and reject suspicious sub-1000ms values that are likely a unit-confusion
// typo (e.g. "--health-interval 60" thinking the unit is seconds).
const healthRaw = parseFlag(args, '--health-interval');
let healthCheckInterval = 60_000;
if (healthRaw !== undefined) {
const parsed = parseInt(healthRaw, 10);
if (!Number.isFinite(parsed) || parsed < 0) {
console.error(`Error: --health-interval must be a non-negative integer (ms), got "${healthRaw}"`);
process.exit(1);
}
if (parsed > 0 && parsed < 1000) {
console.error(
`Error: --health-interval ${parsed} is suspiciously low (likely a unit-confusion typo). ` +
`The flag takes milliseconds; for 60-second probes pass 60000. Use 0 to disable.`,
);
process.exit(1);
}
healthCheckInterval = parsed;
}
try { await queue.ensureSchema(); }
catch (e) { console.error(e instanceof Error ? e.message : String(e)); process.exit(1); }
const worker = new MinionWorker(engine, { queue: queueName, concurrency, maxRssMb });
const worker = new MinionWorker(engine, {
queue: queueName, concurrency, maxRssMb, healthCheckInterval,
});
await registerBuiltinHandlers(worker, engine);
// Subscribe to self-health failures emitted by the worker. Library code
// (worker.ts) never calls process.exit directly so it stays embeddable;
// this CLI layer is the right place to terminate the process and let
// the external PM (systemd, Docker, cron watchdog) restart cleanly.
worker.on('unhealthy', (info) => {
if (info.reason === 'db_dead') {
console.error(
`[health] FATAL: DB unreachable after ${info.consecutiveFailures} probes (${info.message}). ` +
`Exiting for process-manager restart.`,
);
} else {
console.error(
`[health] FATAL: Worker stalled — ${info.waitingCount} waiting job(s) for ` +
`registered handlers, ${info.idleMinutes}m idle. Exiting for process-manager restart.`,
);
}
process.exit(1);
});
const isSupervisedChild = process.env.GBRAIN_SUPERVISED === '1';
const watchdogNote = maxRssMb > 0 ? `, watchdog: ${maxRssMb}MB` : '';
console.log(`Minion worker started (queue: ${queueName}, concurrency: ${concurrency}${watchdogNote})`);
const healthNote = !isSupervisedChild && healthCheckInterval > 0
? `, health-check: ${Math.round(healthCheckInterval / 1000)}s`
: '';
console.log(`Minion worker started (queue: ${queueName}, concurrency: ${concurrency}${watchdogNote}${healthNote})`);
console.log(`Registered handlers: ${worker.registeredNames.join(', ')}`);
await worker.start();
break;
@@ -787,15 +849,32 @@ HANDLER TYPES (built in)
const concurrency = parseInt(parseFlag(args, '--concurrency') ?? '2', 10);
const queueName = parseFlag(args, '--queue') ?? 'default';
const maxCrashes = parseInt(parseFlag(args, '--max-crashes') ?? '10', 10);
const healthInterval = parseInt(parseFlag(args, '--health-interval') ?? '60000', 10);
// --health-interval (supervisor): validate same as `jobs work` so NaN /
// negative / sub-1000ms typos fail-fast instead of silently disabling
// the supervisor's own health probe.
const supHealthRaw = parseFlag(args, '--health-interval');
let healthInterval = 60_000;
if (supHealthRaw !== undefined) {
const parsed = parseInt(supHealthRaw, 10);
if (!Number.isFinite(parsed) || parsed < 0) {
console.error(`Error: --health-interval must be a non-negative integer (ms), got "${supHealthRaw}"`);
process.exit(1);
}
if (parsed > 0 && parsed < 1000) {
console.error(
`Error: --health-interval ${parsed} is suspiciously low (likely a unit-confusion typo). ` +
`The flag takes milliseconds; for 60-second probes pass 60000. Use 0 to disable.`,
);
process.exit(1);
}
healthInterval = parsed;
}
const allowShellJobs = hasFlag(args, '--allow-shell-jobs') ||
!!process.env.GBRAIN_ALLOW_SHELL_JOBS;
const detach = hasFlag(args, '--detach');
// Supervisor defaults --max-rss 2048 (MB) — main production path uses
// the supervisor, so the watchdog is on by default here. parseMaxRssFlag
// returns 0 when the flag is absent; substitute the supervisor default.
const maxRssRaw = parseMaxRssFlag(args);
const maxRssMb = parseFlag(args, '--max-rss') === undefined ? 2048 : maxRssRaw;
// the supervisor, so the watchdog is on by default here.
const maxRssMb = parseMaxRssFlag(args) ?? 2048;
const cliPath = parseFlag(args, '--cli-path') ?? resolveGbrainCliPath();
@@ -864,8 +943,40 @@ export async function registerBuiltinHandlers(worker: MinionWorker, engine: Brai
const { performSync } = await import('./sync.ts');
const repoPath = typeof job.data.repoPath === 'string' ? job.data.repoPath : undefined;
const noPull = !!job.data.noPull;
// noEmbed defaults to true (embed is a separate job — submit `embed --stale`
// after sync, OR run via the autopilot cycle which has its own embed phase).
// Caller can opt in by passing { noEmbed: false } in job params.
const noEmbed = job.data.noEmbed !== false;
const result = await performSync(engine, { repoPath, noPull, noEmbed });
// v0.22.13 (PR #490 CODEX-1): resolve sourceId from job param OR by looking
// up the sources row for repoPath. Mirrors cycle.ts:480 — without this, a
// multi-source brain reads the global config.sync.last_commit anchor
// instead of sources.last_commit, which on a regularly-GC'd repo can drop
// out of git history and trigger 30-min full reimports every cycle.
let sourceId: string | undefined =
typeof job.data.sourceId === 'string' ? job.data.sourceId : undefined;
if (!sourceId && repoPath) {
try {
const rows = await engine.executeRaw<{ id: string }>(
`SELECT id FROM sources WHERE local_path = $1 LIMIT 1`,
[repoPath],
);
sourceId = rows[0]?.id;
} catch {
// sources table may not exist on very old brains — fall through to
// global config.sync.* anchor in performSync.
}
}
// v0.22.13 (PR #490 CODEX-4): route concurrency through the shared
// autoConcurrency helper instead of hardcoded 4. PGLite engines stay
// serial (forced 1); explicit job param wins; auto path defaults are
// applied inside performSync against the resolved file count.
const concurrencyOverride = typeof job.data.concurrency === 'number'
? job.data.concurrency
: undefined;
const result = await performSync(engine, {
repoPath, sourceId, noPull, noEmbed,
concurrency: concurrencyOverride,
});
return result;
});
+3 -5
View File
@@ -8,11 +8,9 @@
*/
import { createEngine } from '../core/engine-factory.ts';
import { loadConfig, saveConfig, toEngineConfig, type GBrainConfig } from '../core/config.ts';
import { loadConfig, saveConfig, toEngineConfig, gbrainPath, type GBrainConfig } from '../core/config.ts';
import type { BrainEngine } from '../core/engine.ts';
import type { EngineConfig } from '../core/types.ts';
import { homedir } from 'os';
import { join } from 'path';
import { writeFileSync, readFileSync, existsSync, unlinkSync } from 'fs';
import { createProgress } from '../core/progress.ts';
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
@@ -48,7 +46,7 @@ function parseArgs(args: string[]): MigrateOpts {
}
function getManifestPath(): string {
return join(homedir(), '.gbrain', 'migrate-manifest.json');
return gbrainPath('migrate-manifest.json');
}
interface MigrateManifest {
@@ -99,7 +97,7 @@ export async function runMigrateEngine(sourceEngine: BrainEngine, args: string[]
process.exit(1);
}
} else {
targetConfig.database_path = opts.targetPath || join(homedir(), '.gbrain', 'brain.pglite');
targetConfig.database_path = opts.targetPath || gbrainPath('brain.pglite');
}
// Connect to target
+7 -6
View File
@@ -35,17 +35,17 @@
*/
import { existsSync, mkdirSync, appendFileSync } from 'fs';
import { homedir } from 'os';
import { join } from 'path';
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
import { loadConfig, toEngineConfig } from '../../core/config.ts';
import { loadConfig, toEngineConfig, gbrainPath } from '../../core/config.ts';
import { createEngine } from '../../core/engine-factory.ts';
import type { BrainEngine } from '../../core/engine.ts';
// Bug 3 — ledger writes moved to the runner (apply-migrations.ts).
const ROLLBACK_DIR = join(homedir(), '.gbrain', 'migrations');
const ROLLBACK_FILE = join(ROLLBACK_DIR, 'v0_13_1-rollback.jsonl');
// Lazy: GBRAIN_HOME may be set after module load.
const getRollbackDir = () => gbrainPath('migrations');
const getRollbackFile = () => join(getRollbackDir(), 'v0_13_1-rollback.jsonl');
const BATCH_SIZE = 100;
// ---------------------------------------------------------------------------
@@ -251,7 +251,8 @@ async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult>
// ---------------------------------------------------------------------------
function ensureRollbackDir(): void {
if (!existsSync(ROLLBACK_DIR)) mkdirSync(ROLLBACK_DIR, { recursive: true });
const dir = getRollbackDir();
if (!existsSync(dir)) mkdirSync(dir, { recursive: true });
}
function appendRollbackEntry(entry: { slug: string; pre_frontmatter: Record<string, unknown> }): void {
@@ -260,7 +261,7 @@ function appendRollbackEntry(entry: { slug: string; pre_frontmatter: Record<stri
timestamp: new Date().toISOString(),
...entry,
}) + '\n';
appendFileSync(ROLLBACK_FILE, line, 'utf-8');
appendFileSync(getRollbackFile(), line, 'utf-8');
}
// ---------------------------------------------------------------------------
+5 -7
View File
@@ -22,19 +22,17 @@
*/
import { existsSync, readFileSync, mkdirSync, appendFileSync } from 'fs';
import { homedir } from 'os';
import { join } from 'path';
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
import { loadConfig, toEngineConfig } from '../../core/config.ts';
import { loadConfig, toEngineConfig, gbrainPath } from '../../core/config.ts';
import { createEngine } from '../../core/engine-factory.ts';
import type { BrainEngine } from '../../core/engine.ts';
// Resolve HOME at CALL time, not module-load time — Bun caches os.homedir()
// and ignores later HOME mutations, which breaks test isolation and scripted
// installs. Match the preferences.ts pattern.
function resolveHome(): string { return process.env.HOME || homedir(); }
function pendingHostWorkDir(): string { return join(resolveHome(), '.gbrain', 'migrations'); }
// gbrainPath() honors GBRAIN_HOME at call time (not module-load) and routes
// through the centralized config dir, so the prior resolveHome()/HOME-env
// trick is no longer needed.
function pendingHostWorkDir(): string { return gbrainPath('migrations'); }
function pendingHostWorkPath(): string { return join(pendingHostWorkDir(), 'pending-host-work.jsonl'); }
// ---------------------------------------------------------------------------
+20 -9
View File
@@ -1,5 +1,5 @@
/**
* gbrain routing-eval Standalone CLI verb for Check 5 (W2, v0.17).
* gbrain routing-eval Standalone CLI verb for Check 5 (W2).
*
* Runs the structural routing eval against every `routing-eval.jsonl`
* fixture in the skills tree. Exits:
@@ -8,10 +8,10 @@
* 1 any failure
* 2 fixtures directory not found / resolver missing (setup error)
*
* Layer B (LLM tie-break) via `--llm` is reserved: the flag parses and
* surfaces in the envelope, but the harness does not yet call any model.
* The plan ships structural layer only in v0.17. The LLM layer has
* explicit sequencing in v0.18 once the structural baseline is stable.
* Layer B (LLM tie-break) via `--llm` is a placeholder: the flag parses
* and surfaces in the envelope, but the harness does not yet call any
* model. Passing `--llm` emits a stderr notice and runs the structural
* layer only. A future release will implement the tie-break layer.
*/
import { readFileSync, existsSync } from 'fs';
@@ -55,7 +55,9 @@ false-positive counts. Lints fixtures for verbatim trigger copies.
Options:
--json Machine-readable JSON envelope
--llm (reserved for v0.18) Run Layer B LLM tie-break
--llm Placeholder for Layer B LLM tie-break. Not yet
implemented. Accepted for forward-compat; emits a
stderr notice and runs the structural layer only.
--skills-dir PATH Override the auto-detected skills/ directory
--help Show this message
@@ -111,6 +113,15 @@ export async function runRoutingEvalCli(args: string[]): Promise<void> {
process.exit(0);
}
// --llm is a placeholder in this release. Emit a stderr notice so
// users (and CI logs) can see the structural-only fallback clearly,
// regardless of --json mode. Does not affect exit code or stdout.
if (flags.llm) {
console.error(
'[routing-eval] --llm flag is a placeholder in this release. Running structural layer only; a future release will implement LLM tie-break.',
);
}
const { dir, error, message } = resolveSkillsDir(flags);
if (error === 'no_skills_dir') {
const env: RoutingEvalEnvelope = {
@@ -200,9 +211,9 @@ export async function runRoutingEvalCli(args: string[]): Promise<void> {
for (const m of loaded.malformed) {
console.log(` [malformed] ${m.file}:${m.line}${m.error}`);
}
if (flags.llm) {
console.log('\nNote: --llm (Layer B LLM tie-break) is reserved for v0.18. No model calls made.');
}
// The stderr notice emitted at the top of runRoutingEvalCli
// already informed the user that --llm is a placeholder; do not
// repeat it here. Stdout in human mode stays results-only.
}
process.exit(ok ? 0 : 1);
+4 -3
View File
@@ -1,5 +1,5 @@
/**
* gbrain skillify <scaffold|check> v0.17 W4 CLI namespace.
* gbrain skillify <scaffold|check> W4 CLI namespace.
*
* `scaffold`: creates 5 stub files for a new skill. Mechanical only.
* `check`: 10-item audit of an existing skill. Promoted from
@@ -299,8 +299,9 @@ export async function runSkillifyScaffold(args: string[]): Promise<void> {
// ---------------------------------------------------------------------------
// `gbrain skillify check` — delegates to scripts/skillify-check.ts via same
// internal helpers. For v0.17 we shell out to the script (kept as single
// source of truth); v0.18 may inline it further.
// internal helpers. Current design shells out to the script (kept as the
// single source of truth for the check logic); a future release may inline
// it further.
// ---------------------------------------------------------------------------
async function runSkillifyCheck(args: string[]): Promise<void> {
+1 -1
View File
@@ -1,5 +1,5 @@
/**
* gbrain skillpack <list|install|diff|check> v0.17 W5 CLI namespace.
* gbrain skillpack <list|install|diff|check> W5 CLI namespace.
*
* D-CX-2 pattern: unified subcommand namespace. The pre-existing
* `skillpack-check` command keeps its top-level name for backwards
+198 -8
View File
@@ -19,6 +19,13 @@ import { errorFor, serializeError } from '../core/errors.ts';
import type { SyncManifest } from '../core/sync.ts';
import { createProgress } from '../core/progress.ts';
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
import { loadConfig } from '../core/config.ts';
import {
autoConcurrency,
shouldRunParallel,
parseWorkers,
} from '../core/sync-concurrency.ts';
import { tryAcquireDbLock, SYNC_LOCK_ID } from '../core/db-lock.ts';
import { loadStorageConfig } from '../core/storage-config.ts';
import { getDefaultSourcePath } from '../core/source-resolver.ts';
@@ -159,6 +166,27 @@ export interface SyncOpts {
sourceId?: string;
/** Multi-repo: sync strategy override (markdown, code, auto). */
strategy?: 'markdown' | 'code' | 'auto';
/**
* Number of parallel workers for the import phase. When > 1, each worker
* gets its own small Postgres connection pool and files are dispatched via
* an atomic queue index (same pattern as `import --workers N`).
*
* Deletes and renames remain serial (order-dependent).
* Default: undefined auto-concurrency picks (`src/core/sync-concurrency.ts`).
*
* v0.22.13 (PR #490 Q1): when this is explicitly set, the >50-file floor
* is bypassed explicit user intent beats the auto-path safety net.
*/
concurrency?: number;
/**
* Internal: skip acquiring the gbrain-sync DB lock. Set by the cycle
* handler (cycle.ts) which already holds gbrain-cycle and therefore
* already serializes against other cycle runs. CLI sync, jobs handler,
* and any external caller leave this undefined so they take the lock.
*
* v0.22.13 (PR #490 CODEX-2). Not part of the public CLI surface.
*/
skipLock?: boolean;
}
function git(repoPath: string, ...args: string[]): string {
@@ -252,6 +280,39 @@ async function writeChunkerVersion(
}
export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise<SyncResult> {
// CODEX-2 (v0.22.13): cross-process writer lock for performSync. Two
// concurrent syncs can otherwise read the same last_commit anchor, both
// write last_commit unconditionally, and the last writer wins — including
// regressing the bookmark backwards. cycle.ts already takes gbrain-cycle
// for its broader scope; performSync (called from cycle, jobs handler,
// and CLI) takes gbrain-sync just for the writer window. The two ids
// nest cleanly: cycle holds gbrain-cycle, calls performSync, performSync
// takes gbrain-sync. Other callers serialize on gbrain-sync against
// each other AND against the cycle's sync phase.
//
// skipLock is reserved for callers that already serialize via another
// mechanism (none in v0.22.13; reserved for future).
let lockHandle: { release: () => Promise<void> } | null = null;
if (!opts.skipLock) {
lockHandle = await tryAcquireDbLock(engine, SYNC_LOCK_ID);
if (!lockHandle) {
throw new Error(
`Another sync is in progress (lock ${SYNC_LOCK_ID} held). ` +
`Wait for it to finish, or run 'gbrain doctor' if it has been more than 30 minutes.`,
);
}
}
try {
return await performSyncInner(engine, opts);
} finally {
if (lockHandle) {
try { await lockHandle.release(); } catch { /* best-effort release */ }
}
}
}
async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<SyncResult> {
// Resolve repo path
const repoPath = opts.repoPath || await readSyncAnchor(engine, opts.sourceId, 'repo_path');
if (!repoPath) {
@@ -488,21 +549,41 @@ export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise<
// gate `sync.last_commit` advancement and record recoverable errors.
const failedFiles: Array<{ path: string; error: string; line?: number }> = [];
const addsAndMods = [...filtered.added, ...filtered.modified];
// v0.22.13 (PR #490 Q5): one source of truth for the concurrency decision.
// engine.kind === 'pglite' → forced 1; explicit opts.concurrency wins;
// auto path returns DEFAULT_PARALLEL_WORKERS only when fileCount > 100.
const explicitConcurrency = opts.concurrency !== undefined;
const effectiveConcurrency = autoConcurrency(engine, addsAndMods.length, opts.concurrency);
const runParallel = shouldRunParallel(effectiveConcurrency, addsAndMods.length, explicitConcurrency);
if (addsAndMods.length > 0) {
progress.start('sync.imports', addsAndMods.length);
for (const path of addsAndMods) {
const filePath = join(repoPath, path);
// Core import logic shared by serial and parallel paths.
// repoPath is validated non-null at the top of performSyncInner; narrow for TS.
const syncRepoPath = repoPath!;
async function importOnePath(eng: BrainEngine, path: string): Promise<void> {
const filePath = join(syncRepoPath, path);
if (!existsSync(filePath)) {
// CODEX-3 (v0.22.13): a file the diff said exists at headCommit but
// is gone from disk means the working tree has drifted (someone ran
// `git checkout` / `git reset` mid-sync, or the file was deleted
// post-diff). Record as a failure so last_commit does NOT advance —
// the silent-skip-then-advance pathology was the bug.
failedFiles.push({
path,
error: 'file vanished mid-sync (working tree drifted from headCommit)',
});
progress.tick(1, `skip:${path}`);
continue;
return;
}
try {
const result = await importFile(engine, filePath, path, { noEmbed });
const result = await importFile(eng, filePath, path, { noEmbed });
if (result.status === 'imported') {
chunksCreated += result.chunks;
pagesAffected.push(result.slug);
} else if (result.status === 'skipped' && (result as any).error) {
// importFile returned a non-throw skip with a reason.
failedFiles.push({ path, error: String((result as any).error) });
}
} catch (e: unknown) {
@@ -512,9 +593,98 @@ export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise<
}
progress.tick(1, path);
}
if (runParallel) {
// A1 (v0.22.13): use engine.kind discriminator instead of config?.engine
// string compare or constructor.name sniff. Q3: belt-and-suspenders fall
// back to serial when database_url is unset, so we never crash on a null
// assertion if config is missing.
const config = loadConfig();
if (engine.kind === 'pglite' || !config?.database_url) {
for (const path of addsAndMods) {
await importOnePath(engine, path);
}
} else {
const { PostgresEngine } = await import('../core/postgres-engine.ts');
const { resolvePoolSize } = await import('../core/db.ts');
const workerPoolSize = Math.min(2, resolvePoolSize(2));
const workerCount = Math.min(effectiveConcurrency, addsAndMods.length);
const databaseUrl = config.database_url;
// Q4 (v0.22.13): banner on stderr so stdout stays clean for --json.
console.error(` Parallel sync: ${workerCount} workers for ${addsAndMods.length} files`);
const workerEngines: InstanceType<typeof PostgresEngine>[] = [];
try {
// Connect workers one-by-one rather than Promise.all so a partial
// failure leaves us with the connected ones in workerEngines for
// the finally-block cleanup. The original code lost track of
// already-connected engines on any one failure.
for (let i = 0; i < workerCount; i++) {
const eng = new PostgresEngine();
await eng.connect({ database_url: databaseUrl, poolSize: workerPoolSize });
workerEngines.push(eng);
}
// Atomic queue index — JS is single-threaded; the read-then-increment
// happens between awaits, so no lock is needed.
let queueIndex = 0;
await Promise.all(
workerEngines.map(async (eng) => {
while (true) {
const idx = queueIndex++;
if (idx >= addsAndMods.length) break;
await importOnePath(eng, addsAndMods[idx]);
}
}),
);
} finally {
// A2 (v0.22.13): try/finally guarantees connection cleanup even when
// the worker loop throws (partial connect failure, OOM, mid-import
// signal). Each disconnect is best-effort — one worker failing to
// disconnect must not strand the others.
await Promise.all(
workerEngines.map((e) =>
e.disconnect().catch((err: unknown) =>
console.error(` worker disconnect failed: ${err instanceof Error ? err.message : String(err)}`),
),
),
);
}
}
} else {
// Serial path (small auto diffs or explicit --workers 1).
for (const path of addsAndMods) {
await importOnePath(engine, path);
}
}
progress.finish();
}
// CODEX-3 (v0.22.13): head-drift gate. If git HEAD moved during the import
// window (someone ran `git checkout` or `git pull` in another terminal /
// sibling Conductor workspace), the chunks we just imported reflect a
// different tree than `headCommit` claims. Refuse to advance last_commit
// so the next sync re-walks against the new HEAD. The lock from CODEX-2
// prevents *this* gbrain process from stepping on itself; this gate
// catches drift caused by external `git` commands the lock cannot see.
try {
const currentHead = git(repoPath, 'rev-parse', 'HEAD');
if (currentHead !== headCommit) {
failedFiles.push({
path: '<head>',
error: `git HEAD drifted during sync: captured ${headCommit.slice(0, 8)}, now ${currentHead.slice(0, 8)}`,
});
}
} catch (e) {
// rev-parse failure is itself a drift signal (worktree disappeared).
failedFiles.push({
path: '<head>',
error: `git HEAD verification failed: ${e instanceof Error ? e.message : String(e)}`,
});
}
const elapsed = Date.now() - start;
// Bug 9 — gate the sync bookmark on success. If any per-file parse
@@ -653,10 +823,18 @@ async function performFullSync(
};
}
console.log(`Running full import of ${repoPath}...`);
// v0.22.13 (PR #490 A1 + Q5): full sync is always "large" by definition
// (entire working tree). Auto-concurrency fires unconditionally for Postgres;
// PGLite stays serial because its engine is single-connection. Routes the
// policy through autoConcurrency() so it stays consistent with incremental
// sync and the jobs handler.
const FULL_SYNC_LARGE_MARKER = Number.MAX_SAFE_INTEGER;
const fullConcurrency = autoConcurrency(engine, FULL_SYNC_LARGE_MARKER, opts.concurrency);
console.log(`Running full import of ${repoPath}${fullConcurrency > 1 ? ` (${fullConcurrency} workers)` : ''}...`);
const { runImport } = await import('./import.ts');
const importArgs = [repoPath];
if (opts.noEmbed) importArgs.push('--no-embed');
if (fullConcurrency > 1) importArgs.push('--workers', String(fullConcurrency));
const result = await runImport(engine, importArgs, { commit: headCommit });
// Bug 9 — gate the full-sync bookmark on success. runImport already
@@ -744,6 +922,17 @@ export async function runSync(engine: BrainEngine, args: string[]) {
const jsonOut = args.includes('--json');
const yesFlag = args.includes('--yes');
const strategyArg = args.find((a, i) => args[i - 1] === '--strategy') as SyncOpts['strategy'] | undefined;
const concurrencyStr = args.find((a, i) => args[i - 1] === '--concurrency' || args[i - 1] === '--workers');
// v0.22.13 (PR #490 Q2): parseWorkers throws on '0', '-3', 'foo', '1.5' instead
// of silently falling through to auto-concurrency or NaN. Loud failure beats
// a 4-worker spawn from a typo.
let concurrency: number | undefined;
try {
concurrency = parseWorkers(concurrencyStr);
} catch (e) {
console.error(e instanceof Error ? e.message : String(e));
process.exit(1);
}
// v0.18.0 Step 5: --source resolves to a sources(id) row. Falls back
// to pre-v0.17 global config (sync.repo_path + sync.last_commit) when
@@ -759,7 +948,7 @@ export async function runSync(engine: BrainEngine, args: string[]) {
// local_path. Sources are the canonical v0.18.0 abstraction: per-source
// last_commit, last_sync_at, config.federated flags. Per-source
// bookmarks live in the sources table (not ~/.gbrain/config.json),
// which is why this path replaced Wintermute's `multi-repo.ts` shim.
// which is why this path replaced Garry's OpenClaw `multi-repo.ts` shim.
//
// Only sources with a non-null local_path participate. A GitHub-only
// source (no checkout) has nothing for `sync` to pull. Sources with
@@ -836,6 +1025,7 @@ export async function runSync(engine: BrainEngine, args: string[]) {
dryRun, full, noPull, noEmbed, skipFailed, retryFailed,
sourceId: src.id,
strategy: cfg.strategy,
concurrency,
};
try {
const result = await performSync(engine, repoOpts);
@@ -854,7 +1044,7 @@ export async function runSync(engine: BrainEngine, args: string[]) {
return;
}
const opts: SyncOpts = { repoPath, dryRun, full, noPull, noEmbed, skipFailed, retryFailed, sourceId, strategy: strategyArg };
const opts: SyncOpts = { repoPath, dryRun, full, noPull, noEmbed, skipFailed, retryFailed, sourceId, strategy: strategyArg, concurrency };
// Bug 9 — --retry-failed: before running normal sync, clear acknowledgment
// flags so the sync picks them up as fresh work. The actual re-attempt
+113
View File
@@ -0,0 +1,113 @@
/**
* AgentRunner pluggable contract for invoking external agents (openclaw,
* hermes, codex, ) inside the claw-test harness. v1 ships a single
* implementation (openclaw); the interface stays narrow and concrete so
* adding a second runner in v1.1 is a ~50-line file.
*
* The harness wraps spawn/timeout/transcript-capture; runners only have to
* answer "where's your binary?" and "how do I invoke it with this prompt?".
*
*
* harness
* resolve(name) registry AgentRunner instance
* detect() runner reports binary path/availability
* invoke(...) runner spawns child, harness captures via TranscriptSink
*
*/
export interface AgentRunner {
/** Stable agent name used by --agent flag and friction `agent` field. */
readonly name: string;
/**
* Locate the agent binary and confirm it is executable. Pure check; never
* spawns. `binPath` is always an absolute path on success. `available=false`
* with a `reason` if not found / not executable.
*/
detect(): Promise<DetectResult>;
/**
* Invoke the agent with the given prompt. The runner is responsible for
* the per-agent argv shape. The harness owns timeouts, signals, and
* transcript capture (via `transcriptSink`).
*/
invoke(opts: InvokeOpts): Promise<InvokeResult>;
/** Optional per-agent post-install hook (e.g., routing-file fixup). */
postInstallHook?(opts: { workspaceDir: string }): Promise<void>;
}
export interface DetectResult {
available: boolean;
reason?: string;
binPath?: string;
}
export interface InvokeOpts {
/** Workspace dir the agent runs in. */
cwd: string;
/** The prompt content. The runner decides whether to write a temp file or pass via argv. */
brief: string;
/** Env to merge with the runner's defaults. Caller already restricted to allow-listed keys. */
env: Record<string, string>;
/** Wall-clock kill switch in ms. Harness handles SIGTERM → 5s grace → SIGKILL. */
timeoutMs: number;
/**
* Per-channel byte sink. The runner pipes child stdin/stdout/stderr into this
* instead of inheriting the parent's. Async-drain backpressure is handled
* inside the sink (D17), so the runner can call `write()` without awaiting.
*/
transcriptSink: TranscriptSink;
/** Optional override for which sub-agent the runner targets. */
agentName?: string;
}
export interface InvokeResult {
exitCode: number;
durationMs: number;
}
/** Async-drain sink. The harness owns the underlying file stream. */
export interface TranscriptSink {
write(event: TranscriptEvent): void;
/** Returns the byte offset that the next written event would have. */
nextOffset(): number;
/** Flush + close. Idempotent. */
close(): Promise<void>;
}
export interface TranscriptEvent {
ts: number;
channel: 'stdin' | 'stdout' | 'stderr';
bytes: Buffer;
}
// ---------------------------------------------------------------------------
// Registry
// ---------------------------------------------------------------------------
type AgentRunnerFactory = () => AgentRunner;
const registry = new Map<string, AgentRunnerFactory>();
export function registerAgentRunner(name: string, factory: AgentRunnerFactory): void {
registry.set(name, factory);
}
export function resolveAgentRunner(name: string): AgentRunner {
const factory = registry.get(name);
if (!factory) {
const known = [...registry.keys()].sort().join(', ') || '(none registered)';
throw new Error(`unknown agent ${JSON.stringify(name)}; registered: ${known}`);
}
return factory();
}
export function listRegisteredAgents(): string[] {
return [...registry.keys()].sort();
}
/** Reset registry — testing only. */
export function _resetRegistryForTests(): void {
registry.clear();
}
+58
View File
@@ -0,0 +1,58 @@
/**
* progress-tail parses gbrain's --progress-json events out of child stderr.
*
* The actual contract (verified post-Codex):
* - `gbrain --progress-json <subcommand>` writes JSONL events to STDERR
* - Stable phase names are dotted snake_case: `import.files`, `extract.links_fs`,
* `embed.pages`, `doctor.db_checks`, etc.
* - Each event line is a JSON object; non-progress stderr lines (warnings,
* debug output, errors) interleave with progress events. We tolerate them.
*
* Used by the verify phase to assert that each `expected_phases` entry from
* scenario.json saw at least one event from the corresponding command.
*/
export interface ProgressEvent {
phase: string;
event?: string; // 'start' | 'tick' | 'finish' | etc per docs/progress-events.md
ts?: string;
[key: string]: unknown;
}
/** Parse a single stderr buffer into the progress events it contains. */
export function parseProgressEvents(stderr: string): ProgressEvent[] {
const out: ProgressEvent[] = [];
for (const line of stderr.split('\n')) {
const trimmed = line.trim();
if (!trimmed.startsWith('{')) continue;
let parsed: unknown;
try {
parsed = JSON.parse(trimmed);
} catch {
continue;
}
if (parsed && typeof parsed === 'object' && typeof (parsed as any).phase === 'string') {
out.push(parsed as ProgressEvent);
}
}
return out;
}
/** Group events by phase name. */
export function eventsByPhase(events: ProgressEvent[]): Map<string, ProgressEvent[]> {
const m = new Map<string, ProgressEvent[]>();
for (const e of events) {
if (!m.has(e.phase)) m.set(e.phase, []);
m.get(e.phase)!.push(e);
}
return m;
}
/**
* Verify that every `expected` phase appears at least once in `events`.
* Returns the missing phase names (empty array on full coverage).
*/
export function verifyExpectedPhases(events: ProgressEvent[], expected: string[]): string[] {
const seen = new Set(events.map(e => e.phase));
return expected.filter(p => !seen.has(p));
}
+98
View File
@@ -0,0 +1,98 @@
/**
* OpenClaw runner invokes the real `openclaw` binary in a tempdir with a
* BRIEF.md prompt. Live mode only.
*
* Invocation pattern (verified against test/e2e/skills.test.ts and
* test/e2e/bench-vs-openclaw/harness.ts):
* openclaw agent --local --agent <agent-name> --message "<brief>"
*
* NOT `openclaw run --prompt-file BRIEF.md` (that flag does not exist
* Codex pass 2 of the eng review caught the speculative shape).
*
* Binary resolution: $OPENCLAW_BIN > `which openclaw` > unavailable.
* Path validation: must be absolute, must be executable, no '..' segments.
*/
import { execSync } from 'child_process';
import { statSync } from 'fs';
import type { AgentRunner, DetectResult, InvokeOpts, InvokeResult } from '../agent-runner.ts';
import { spawnWithCapture } from '../transcript-capture.ts';
const DEFAULT_AGENT_NAME = 'default';
/** Allow-list for env propagation when spawning openclaw. */
const ENV_ALLOWLIST = [
'PATH', 'HOME', 'USER', 'LANG', 'TZ', 'NODE_ENV',
'ANTHROPIC_API_KEY', 'OPENAI_API_KEY',
'GBRAIN_HOME', 'GBRAIN_FRICTION_RUN_ID', 'GBRAIN_DATABASE_URL',
];
export class OpenClawRunner implements AgentRunner {
readonly name = 'openclaw';
async detect(): Promise<DetectResult> {
const fromEnv = process.env.OPENCLAW_BIN?.trim();
let binPath: string | undefined;
if (fromEnv) {
const validation = validateAbsolutePath(fromEnv);
if (validation) return { available: false, reason: validation };
binPath = fromEnv;
} else {
try {
const out = execSync('which openclaw', { encoding: 'utf-8', stdio: ['ignore', 'pipe', 'ignore'] });
const found = out.trim();
if (!found || !found.startsWith('/')) {
return { available: false, reason: 'openclaw not on PATH' };
}
binPath = found;
} catch {
return { available: false, reason: 'openclaw not on PATH' };
}
}
if (!binPath) return { available: false, reason: 'no binary resolved' };
try {
const s = statSync(binPath);
if (!s.isFile()) return { available: false, reason: `not a regular file: ${binPath}` };
// eslint-disable-next-line no-bitwise
if (!(s.mode & 0o111)) return { available: false, reason: `not executable: ${binPath}` };
} catch (e) {
return { available: false, reason: `stat failed: ${e instanceof Error ? e.message : String(e)}` };
}
return { available: true, binPath };
}
async invoke(opts: InvokeOpts): Promise<InvokeResult> {
const detected = await this.detect();
if (!detected.available || !detected.binPath) {
throw new Error(`openclaw runner unavailable: ${detected.reason ?? 'unknown'}`);
}
const agentName = opts.agentName ?? DEFAULT_AGENT_NAME;
const args = ['agent', '--local', '--agent', agentName, '--message', opts.brief];
// Filter env to allow-list, then merge caller overrides.
const baseEnv: Record<string, string> = {};
for (const key of ENV_ALLOWLIST) {
const v = process.env[key];
if (typeof v === 'string') baseEnv[key] = v;
}
const env: Record<string, string> = { ...baseEnv, ...opts.env };
const result = await spawnWithCapture(detected.binPath, args, {
cwd: opts.cwd,
env,
timeoutMs: opts.timeoutMs,
transcriptSink: opts.transcriptSink,
});
return { exitCode: result.exitCode, durationMs: result.durationMs };
}
}
function validateAbsolutePath(p: string): string | null {
if (!p.startsWith('/')) return `OPENCLAW_BIN must be absolute; got ${p}`;
if (p.split('/').includes('..')) return `OPENCLAW_BIN must not contain '..' segments; got ${p}`;
return null;
}
+114
View File
@@ -0,0 +1,114 @@
/**
* scenario.json loader for the claw-test harness.
*
* test/fixtures/claw-test-scenarios/<name>/scenario.json:
* { kind: "fresh-install", expected_phases: ["import.files", ...], ... }
*
* The harness reads scenario.json to know which phases to assert from
* gbrain's --progress-json events. Pure local fs; no DB, no network.
*/
import { existsSync, readdirSync, readFileSync, statSync } from 'fs';
import { dirname, join, resolve } from 'path';
import { fileURLToPath } from 'url';
export type ScenarioKind = 'fresh-install' | 'upgrade';
export interface ScenarioConfig {
/** Directory the scenario was loaded from. Always absolute. */
dir: string;
/** Stable scenario name (the directory name). */
name: string;
/** Kind of scenario; drives setup-phase behavior. */
kind: ScenarioKind;
/** Stable phase names emitted by --progress-json that the harness asserts. */
expectedPhases: string[];
/** When kind==="upgrade": version we are simulating an upgrade FROM. */
fromVersion?: string;
/** Optional human-readable summary. */
description?: string;
/** Path to BRIEF.md (relative to scenario dir, default 'BRIEF.md'). */
briefRelative: string;
/** Path to brain markdown source (relative to scenario dir). For 'fresh-install': 'brain'. */
brainRelative?: string;
/** Path to seed dir for upgrade scenarios. */
seedRelative?: string;
}
/** Default fixtures root, override via $GBRAIN_CLAW_SCENARIOS_DIR for tests. */
function defaultFixturesRoot(): string {
if (process.env.GBRAIN_CLAW_SCENARIOS_DIR) {
return resolve(process.env.GBRAIN_CLAW_SCENARIOS_DIR);
}
// src/core/claw-test/scenarios.ts → ../../../test/fixtures/claw-test-scenarios
const here = dirname(fileURLToPath(import.meta.url));
return resolve(here, '..', '..', '..', 'test', 'fixtures', 'claw-test-scenarios');
}
/** List all available scenario names. */
export function listScenarios(root?: string): string[] {
const r = root ?? defaultFixturesRoot();
if (!existsSync(r)) return [];
return readdirSync(r)
.filter(name => {
const path = join(r, name);
try {
return statSync(path).isDirectory() && existsSync(join(path, 'scenario.json'));
} catch {
return false;
}
})
.sort();
}
/** Load and validate one scenario by name. */
export function loadScenario(name: string, root?: string): ScenarioConfig {
const r = root ?? defaultFixturesRoot();
const dir = join(r, name);
const cfgPath = join(dir, 'scenario.json');
if (!existsSync(cfgPath)) {
throw new Error(`scenario ${JSON.stringify(name)} not found at ${cfgPath}`);
}
let raw: unknown;
try {
raw = JSON.parse(readFileSync(cfgPath, 'utf-8'));
} catch (e) {
throw new Error(`scenario ${JSON.stringify(name)}: malformed scenario.json (${e instanceof Error ? e.message : e})`);
}
if (!raw || typeof raw !== 'object') {
throw new Error(`scenario ${JSON.stringify(name)}: scenario.json must be a JSON object`);
}
const cfg = raw as Record<string, unknown>;
if (cfg.kind !== 'fresh-install' && cfg.kind !== 'upgrade') {
throw new Error(`scenario ${JSON.stringify(name)}: unknown kind ${JSON.stringify(cfg.kind)}`);
}
if (!Array.isArray(cfg.expected_phases) || !cfg.expected_phases.every(x => typeof x === 'string')) {
throw new Error(`scenario ${JSON.stringify(name)}: expected_phases must be a string[]`);
}
const briefRel = typeof cfg.brief === 'string' ? cfg.brief : 'BRIEF.md';
if (!existsSync(join(dir, briefRel))) {
throw new Error(`scenario ${JSON.stringify(name)}: BRIEF.md missing at ${briefRel}`);
}
const out: ScenarioConfig = {
dir,
name,
kind: cfg.kind,
expectedPhases: cfg.expected_phases as string[],
briefRelative: briefRel,
};
if (typeof cfg.from_version === 'string') out.fromVersion = cfg.from_version;
if (typeof cfg.description === 'string') out.description = cfg.description;
if (typeof cfg.brain === 'string') out.brainRelative = cfg.brain;
if (typeof cfg.seed === 'string') out.seedRelative = cfg.seed;
// Default brain path conventions
if (!out.brainRelative && existsSync(join(dir, 'brain'))) out.brainRelative = 'brain';
if (!out.seedRelative && out.kind === 'upgrade' && existsSync(join(dir, 'seed'))) {
out.seedRelative = 'seed';
}
return out;
}
/** Read BRIEF.md content for this scenario. Used by --live mode. */
export function readBrief(scenario: ScenarioConfig): string {
return readFileSync(join(scenario.dir, scenario.briefRelative), 'utf-8');
}
+123
View File
@@ -0,0 +1,123 @@
/**
* seed-pglite replay a SQL dump into a fresh PGLite database, then let
* gbrain's migration chain walk forward.
*
* Codex caught (eng review pass 2) that existing migration helpers
* (test/e2e/helpers.ts:204) are Postgres-only they rewind schema_version
* and replay against real Postgres. PGLite has no equivalent. This helper
* fills that gap so the `upgrade-from-v0.18` claw-test scenario is
* reproducible.
*
* Usage:
* const dbPath = await seedPglite('/tmp/run-x/.gbrain/brain.pglite', seedSql);
* // Then run `gbrain init --pglite --path <dbPath>` — the migration chain
* // detects the seeded schema_version and migrates forward to LATEST.
*/
import { existsSync, mkdirSync, readFileSync } from 'fs';
import { dirname } from 'path';
import { PGLiteEngine } from '../pglite-engine.ts';
export interface SeedOpts {
/** Absolute path to the .pglite file to create. */
dbPath: string;
/** Raw SQL dump to replay. */
sql: string;
}
/**
* Open a fresh PGLite at `dbPath`, execute the SQL dump, disconnect.
* Throws on SQL errors with a structured message that names the failing
* statement (helpful for debugging seed drift).
*/
export async function seedPglite(opts: SeedOpts): Promise<void> {
const dir = dirname(opts.dbPath);
if (!existsSync(dir)) mkdirSync(dir, { recursive: true });
const engine = new PGLiteEngine();
try {
await engine.connect({ engine: 'pglite', database_path: opts.dbPath });
// Execute statements one at a time so an error names the offending
// statement. The seed file is committed to source so we can normalize
// its line endings; we rely on `;\n` as the statement terminator.
const statements = splitStatements(opts.sql);
for (const stmt of statements) {
const trimmed = stmt.trim();
if (!trimmed) continue;
try {
await (engine as any).db.exec(trimmed);
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
const preview = trimmed.slice(0, 120).replace(/\s+/g, ' ');
throw new Error(`seedPglite: SQL execution failed at "${preview}…": ${msg}`);
}
}
} finally {
await engine.disconnect();
}
}
/** Read seed SQL from disk and replay into `dbPath`. */
export async function seedPgliteFromFile(opts: { dbPath: string; sqlPath: string }): Promise<void> {
if (!existsSync(opts.sqlPath)) {
throw new Error(`seedPglite: seed SQL not found at ${opts.sqlPath}`);
}
const sql = readFileSync(opts.sqlPath, 'utf-8');
return seedPglite({ dbPath: opts.dbPath, sql });
}
/**
* Split a SQL dump into individual statements. Naïve `;` split that respects
* single-quoted strings and `--` line comments. Sufficient for canonical
* pg_dump output; intentionally NOT a full SQL parser.
*/
function splitStatements(sql: string): string[] {
const out: string[] = [];
let buf = '';
let inSingle = false;
let inLineComment = false;
let i = 0;
while (i < sql.length) {
const c = sql[i];
const next = sql[i + 1];
if (inLineComment) {
buf += c;
if (c === '\n') inLineComment = false;
i++;
continue;
}
if (inSingle) {
buf += c;
if (c === "'" && next === "'") { buf += next; i += 2; continue; }
if (c === "'") inSingle = false;
i++;
continue;
}
if (c === '-' && next === '-') {
inLineComment = true;
buf += c;
i++;
continue;
}
if (c === "'") {
inSingle = true;
buf += c;
i++;
continue;
}
if (c === ';') {
buf += c;
out.push(buf);
buf = '';
i++;
continue;
}
buf += c;
i++;
}
if (buf.trim()) out.push(buf);
return out;
}
/** Exposed for tests. */
export const _internal = { splitStatements };
+172
View File
@@ -0,0 +1,172 @@
/**
* Transcript capture for live-mode agent runs (D8 + D14, D17 backpressure).
*
* The existing minions/audit infrastructure is for INTERNAL gbrain subagents
* only. External openclaw/hermes subprocesses don't write to those tables
* v1 builds its own capture channel here.
*
* Output: JSONL at `<run-tempdir>/transcript.jsonl`, one event per line.
* { schema_version: "1", ts, channel, byte_offset, bytes_b64 }
*
* child stdout/stderr piped TranscriptSink.write()
*
*
* fs.createWriteStream (flags: 'a')
*
* honors 'drain' events to avoid blocking
* the child when bursts exceed the pipe buffer
*
* transcript.jsonl (line-tolerant readers
* skip malformed; render() resolves
* byte_offset readable lines)
*
* Friction CLI's `transcript_offset` field references the byte offset INTO
* `transcript.jsonl` (not into the captured payload). Render --transcripts
* reads the file and finds the line that contains that offset.
*/
import { createWriteStream, type WriteStream } from 'fs';
import { spawn, type ChildProcess } from 'child_process';
import { dirname } from 'path';
import { mkdirSync, existsSync } from 'fs';
import type { TranscriptEvent, TranscriptSink } from './agent-runner.ts';
// ---------------------------------------------------------------------------
// Sink
// ---------------------------------------------------------------------------
export function createTranscriptSink(path: string): TranscriptSink {
const dir = dirname(path);
if (!existsSync(dir)) mkdirSync(dir, { recursive: true });
const stream: WriteStream = createWriteStream(path, { flags: 'a' });
let bytesWritten = 0;
let drainPromise: Promise<void> | null = null;
function awaitDrain(): Promise<void> {
if (drainPromise) return drainPromise;
drainPromise = new Promise<void>(resolve => {
stream.once('drain', () => {
drainPromise = null;
resolve();
});
});
return drainPromise;
}
return {
write(event: TranscriptEvent) {
const line = JSON.stringify({
schema_version: '1',
ts: event.ts,
channel: event.channel,
byte_offset: bytesWritten,
bytes_b64: event.bytes.toString('base64'),
}) + '\n';
bytesWritten += Buffer.byteLength(line, 'utf-8');
const ok = stream.write(line, 'utf-8');
// If the kernel buffer is full, write() returns false. We don't await
// here (callers don't expect that), but next callers wait on drain
// before writing further. Bun's WritableStream is small; the drain
// window is typically a few µs.
if (!ok) void awaitDrain();
},
nextOffset(): number {
return bytesWritten;
},
async close(): Promise<void> {
await new Promise<void>((resolve, reject) => {
stream.end((err?: Error | null) => err ? reject(err) : resolve());
});
},
};
}
// ---------------------------------------------------------------------------
// spawnWithCapture
// ---------------------------------------------------------------------------
export interface SpawnOpts {
cwd: string;
env: Record<string, string>;
timeoutMs: number;
transcriptSink: TranscriptSink;
/** Optional fixed input to write on stdin then close. */
stdinPayload?: string;
}
export interface SpawnResult {
exitCode: number;
durationMs: number;
/** True if SIGTERM/SIGKILL was issued due to timeout. */
timedOut: boolean;
}
const SIGTERM_GRACE_MS = 5_000;
export async function spawnWithCapture(bin: string, args: string[], opts: SpawnOpts): Promise<SpawnResult> {
const start = Date.now();
return new Promise((resolve, reject) => {
let child: ChildProcess;
try {
child = spawn(bin, args, {
cwd: opts.cwd,
env: opts.env,
stdio: ['pipe', 'pipe', 'pipe'],
shell: false,
});
} catch (e) {
reject(e);
return;
}
let timedOut = false;
let killTimer: ReturnType<typeof setTimeout> | null = null;
const wallClockTimer = setTimeout(() => {
timedOut = true;
try { child.kill('SIGTERM'); } catch { /* already gone */ }
killTimer = setTimeout(() => {
try { child.kill('SIGKILL'); } catch { /* already gone */ }
}, SIGTERM_GRACE_MS);
}, opts.timeoutMs);
child.stdout?.on('data', (chunk: Buffer) => {
opts.transcriptSink.write({ ts: Date.now(), channel: 'stdout', bytes: chunk });
});
child.stderr?.on('data', (chunk: Buffer) => {
opts.transcriptSink.write({ ts: Date.now(), channel: 'stderr', bytes: chunk });
});
if (opts.stdinPayload !== undefined && child.stdin) {
try {
opts.transcriptSink.write({
ts: Date.now(),
channel: 'stdin',
bytes: Buffer.from(opts.stdinPayload, 'utf-8'),
});
child.stdin.end(opts.stdinPayload, 'utf-8');
} catch (e) {
reject(e);
return;
}
}
child.on('error', (err) => {
clearTimeout(wallClockTimer);
if (killTimer) clearTimeout(killTimer);
reject(err);
});
child.on('close', (code) => {
clearTimeout(wallClockTimer);
if (killTimer) clearTimeout(killTimer);
resolve({
exitCode: typeof code === 'number' ? code : (timedOut ? 124 : 1),
durationMs: Date.now() - start,
timedOut,
});
});
});
}
+28 -5
View File
@@ -19,9 +19,11 @@ export type DbUrlSource =
| 'config-file-path' // PGLite: config file present, no URL but database_path set
| null;
// Lazy-evaluated to avoid calling homedir() at module scope (breaks in serverless/bundled environments)
function getConfigDir() { return join(homedir(), '.gbrain'); }
function getConfigPath() { return join(getConfigDir(), 'config.json'); }
// Internal aliases retained for backwards compatibility with the existing call
// sites below. They forward to the exported configDir()/configPath() so
// GBRAIN_HOME is honored uniformly. Lazy: never call homedir() at module scope.
function getConfigDir() { return configDir(); }
function getConfigPath() { return configPath(); }
export interface GBrainConfig {
engine: 'postgres' | 'pglite';
@@ -88,9 +90,20 @@ export function toEngineConfig(config: GBrainConfig): EngineConfig {
export function configDir(): string {
// Allow override for tests, Docker, and multi-tenant deployments.
// Matches the `GBRAIN_AUDIT_DIR` convention in src/core/minions/handlers/shell-audit.ts.
// GBRAIN_HOME is a parent dir; we always append '.gbrain' ourselves so
// setting GBRAIN_HOME=/tmp/x yields configDir() === '/tmp/x/.gbrain'.
// Validates the override: must be absolute, no '..' segments.
const override = process.env.GBRAIN_HOME;
if (override && override.trim()) return join(override, '.gbrain');
if (override && override.trim()) {
const trimmed = override.trim();
if (!trimmed.startsWith('/')) {
throw new Error(`GBRAIN_HOME must be an absolute path; got: ${trimmed}`);
}
if (trimmed.split('/').includes('..')) {
throw new Error(`GBRAIN_HOME must not contain '..' segments; got: ${trimmed}`);
}
return join(trimmed, '.gbrain');
}
return join(homedir(), '.gbrain');
}
@@ -98,6 +111,16 @@ export function configPath(): string {
return join(configDir(), 'config.json');
}
/**
* Sugar for joining paths under the active gbrain home. Use this anywhere you
* would otherwise write `join(homedir(), '.gbrain', ...rest)`. Honors
* GBRAIN_HOME, validates input, and centralizes the convention so future
* audits stay simple.
*/
export function gbrainPath(...segments: string[]): string {
return join(configDir(), ...segments);
}
/**
* Introspect where the active DB URL would come from if we tried to connect.
* Never throws, never connects. Env vars take precedence (matches loadConfig).
+127 -16
View File
@@ -16,9 +16,14 @@
* Phase 1: lint --fix (filesystem writes, no DB)
* Phase 2: backlinks --fix (filesystem writes, no DB)
* Phase 3: sync (DB picks up phases 1+2)
* Phase 4: extract (DB picks up links from sync)
* Phase 5: embed --stale (DB writes)
* Phase 6: orphans (DB read, report only)
* Phase 4: synthesize (v0.23: transcripts pages)
* Phase 5: extract (DB picks up links from sync
* + synthesize)
* Phase 6: patterns (v0.23: cross-session themes;
* MUST be after extract so
* graph state is fresh)
* Phase 7: embed --stale (DB writes)
* Phase 8: orphans (DB read, report only)
*
*
* COORDINATION:
@@ -39,20 +44,23 @@
import { existsSync, readFileSync, writeFileSync, unlinkSync, mkdirSync, statSync } from 'fs';
import { join } from 'path';
import { homedir, hostname } from 'os';
import { hostname } from 'os';
import { gbrainPath } from './config.ts';
import type { BrainEngine } from './engine.ts';
import { createProgress, type ProgressReporter } from './progress.ts';
import { getCliOptions, cliOptsToProgressOptions } from './cli-options.ts';
// ─── Types ─────────────────────────────────────────────────────────
export type CyclePhase = 'lint' | 'backlinks' | 'sync' | 'extract' | 'embed' | 'orphans';
export type CyclePhase = 'lint' | 'backlinks' | 'sync' | 'synthesize' | 'extract' | 'patterns' | 'embed' | 'orphans';
export const ALL_PHASES: CyclePhase[] = [
'lint',
'backlinks',
'sync',
'synthesize',
'extract',
'patterns',
'embed',
'orphans',
];
@@ -60,13 +68,16 @@ export const ALL_PHASES: CyclePhase[] = [
/**
* Phases that mutate state (filesystem or DB) and therefore should
* coordinate via the cycle lock. Only orphans is truly read-only
* and skips the lock.
* and skips the lock. patterns mutates DB (writes pattern pages) so
* it acquires the lock; synthesize too.
*/
const NEEDS_LOCK_PHASES: ReadonlySet<CyclePhase> = new Set([
'lint',
'backlinks',
'sync',
'synthesize',
'extract',
'patterns',
'embed',
]);
@@ -121,6 +132,12 @@ export interface CycleReport {
pages_extracted: number;
pages_embedded: number;
orphans_found: number;
/** v0.23: number of transcripts the synthesize phase processed (judged + dispatched). */
transcripts_processed: number;
/** v0.23: number of new reflection/original/people pages written by synthesize. */
synth_pages_written: number;
/** v0.23: number of pattern pages written/updated by patterns phase. */
patterns_written: number;
};
}
@@ -141,11 +158,37 @@ export interface CycleOpts {
*/
yieldBetweenPhases?: () => Promise<void>;
/**
* AbortSignal from the Minions worker. When aborted (timeout, cancel,
* lock-loss), runCycle bails between phases and returns a 'failed' report
* instead of running the next phase. Without this, a timed-out
* autopilot-cycle handler ignores the abort and runs until the worker
* wedges (the 98-waiting-0-active incident on 2026-04-24).
* Generic in-phase keepalive (v0.23). Long-running phases (synthesize
* waiting on a fan-out aggregator, patterns rolling up reflections)
* call this periodically while idle to renew the cycle-lock TTL and
* the Minions worker job lock. Mirrors `yieldBetweenPhases` shape;
* passing the same function for both is the common case.
*/
yieldDuringPhase?: () => Promise<void>;
/**
* Synthesize phase scope overrides (v0.23). Forwarded to runPhaseSynthesize.
* - `synthInputFile`: ad-hoc transcript path (`gbrain dream --input <file>`).
* - `synthDate` / `synthFrom` / `synthTo`: date filters for corpus scan.
* Mutually exclusive with each other in CLI parsing; runner trusts the
* caller (CLI wrapper validates).
*/
synthInputFile?: string;
synthDate?: string;
synthFrom?: string;
synthTo?: string;
/**
* v0.23.2: explicit opt-in to disable the synthesize self-consumption guard.
* Wired from `gbrain dream --unsafe-bypass-dream-guard`. Never auto-applied
* for `--input` because that would let any caller silently re-trigger the
* loop bug (codex finding #3).
*/
synthBypassDreamGuard?: boolean;
/**
* AbortSignal from the Minions worker (v0.22.1, #403). When aborted
* (timeout, cancel, lock-loss), runCycle bails between phases and
* returns a 'failed' report instead of running the next phase. Without
* this, a timed-out autopilot-cycle handler ignores the abort and runs
* until the worker wedges (the 98-waiting-0-active incident on 2026-04-24).
*/
signal?: AbortSignal;
}
@@ -154,7 +197,8 @@ export interface CycleOpts {
const CYCLE_LOCK_ID = 'gbrain-cycle';
const LOCK_TTL_MS = 30 * 60 * 1000; // 30 minutes
const LOCK_FILE_PATH_DEFAULT = join(homedir(), '.gbrain', 'cycle.lock');
// Lazy: GBRAIN_HOME may be set after module load; resolve at call time.
const getLockFilePathDefault = () => gbrainPath('cycle.lock');
interface LockHandle {
release: () => Promise<void>;
@@ -256,7 +300,7 @@ async function acquirePostgresLock(engine: BrainEngine): Promise<LockHandle | nu
* The file contains `{pid}\n{iso-timestamp}`. Staleness = mtime older
* than LOCK_TTL_MS OR the PID is no longer alive on this host.
*/
function acquireFileLock(lockPath = LOCK_FILE_PATH_DEFAULT): LockHandle | null {
function acquireFileLock(lockPath = getLockFilePathDefault()): LockHandle | null {
mkdirSync(join(lockPath, '..'), { recursive: true });
const pid = process.pid;
@@ -763,7 +807,37 @@ export async function runCycle(
await safeYield(opts.yieldBetweenPhases);
}
// ── Phase 4: extract ────────────────────────────────────────
// ── Phase 4: synthesize (v0.23) ─────────────────────────────
if (phases.includes('synthesize')) {
if (!engine) {
phaseResults.push({
phase: 'synthesize',
status: 'skipped',
duration_ms: 0,
summary: 'no database connected',
details: { reason: 'no_database' },
});
} else {
progress.start('cycle.synthesize');
const { runPhaseSynthesize } = await import('./cycle/synthesize.ts');
const { result, duration_ms } = await timePhase(() => runPhaseSynthesize(engine, {
brainDir: opts.brainDir,
dryRun,
yieldDuringPhase: opts.yieldDuringPhase,
inputFile: opts.synthInputFile,
date: opts.synthDate,
from: opts.synthFrom,
to: opts.synthTo,
bypassDreamGuard: opts.synthBypassDreamGuard,
}));
result.duration_ms = duration_ms;
phaseResults.push(result);
progress.finish();
}
await safeYield(opts.yieldBetweenPhases);
}
// ── Phase 5: extract (now picks up synthesize output) ───────
if (phases.includes('extract')) {
checkAborted(opts.signal);
if (!engine) {
@@ -787,7 +861,36 @@ export async function runCycle(
await safeYield(opts.yieldBetweenPhases);
}
// ── Phase 5: embed ──────────────────────────────────────────
// ── Phase 6: patterns (v0.23) ───────────────────────────────
// MUST run after extract so the graph state reads fresh — subagent
// put_page calls in synthesize set ctx.remote=true, so auto-link
// only fires for trusted-workspace writes (allow-listed). extract
// is the canonical materialization step.
if (phases.includes('patterns')) {
if (!engine) {
phaseResults.push({
phase: 'patterns',
status: 'skipped',
duration_ms: 0,
summary: 'no database connected',
details: { reason: 'no_database' },
});
} else {
progress.start('cycle.patterns');
const { runPhasePatterns } = await import('./cycle/patterns.ts');
const { result, duration_ms } = await timePhase(() => runPhasePatterns(engine, {
brainDir: opts.brainDir,
dryRun,
yieldDuringPhase: opts.yieldDuringPhase,
}));
result.duration_ms = duration_ms;
phaseResults.push(result);
progress.finish();
}
await safeYield(opts.yieldBetweenPhases);
}
// ── Phase 7: embed ──────────────────────────────────────────
if (phases.includes('embed')) {
checkAborted(opts.signal);
if (!engine) {
@@ -808,7 +911,7 @@ export async function runCycle(
await safeYield(opts.yieldBetweenPhases);
}
// ── Phase 6: orphans ────────────────────────────────────────
// ── Phase 8: orphans ────────────────────────────────────────
if (phases.includes('orphans')) {
checkAborted(opts.signal);
if (!engine) {
@@ -859,6 +962,9 @@ function emptyTotals(): CycleReport['totals'] {
pages_extracted: 0,
pages_embedded: 0,
orphans_found: 0,
transcripts_processed: 0,
synth_pages_written: 0,
patterns_written: 0,
};
}
@@ -881,6 +987,11 @@ function extractTotals(phases: PhaseResult[]): CycleReport['totals'] {
: Number(p.details.embedded ?? 0);
} else if (p.phase === 'orphans' && p.details) {
t.orphans_found = Number(p.details.total_orphans ?? 0);
} else if (p.phase === 'synthesize' && p.details) {
t.transcripts_processed = Number(p.details.transcripts_processed ?? 0);
t.synth_pages_written = Number(p.details.pages_written ?? 0);
} else if (p.phase === 'patterns' && p.details) {
t.patterns_written = Number(p.details.patterns_written ?? 0);
}
}
return t;
+323
View File
@@ -0,0 +1,323 @@
/**
* Patterns phase (v0.23) cross-session theme detection.
*
* Reads recent reflections (within `lookback_days`), runs a single Sonnet
* subagent to surface themes that recur across `min_evidence` distinct
* reflections, and writes one pattern page per theme.
*
* MUST run after `extract` so the graph state (links, timeline) is fresh.
* Subagent put_page calls have ctx.remote=true; the trusted-workspace
* allow-list re-enables auto-link / auto-timeline for synth + pattern
* writes (operations.ts:trustedWorkspace branch).
*
* v1 behavior:
* - Single Sonnet subagent (no fan-out one job per cycle is plenty).
* - Idempotent: if reflection set is below `min_evidence`, phase is skipped.
* - Pattern slug uses LLM's chosen topic-slug (subagent prompt instructs format).
* - Existing pattern pages are updated in place via put_page (idempotent
* ON CONFLICT semantics in importFromContent).
*/
import { join, dirname } from 'node:path';
import { mkdirSync, writeFileSync, existsSync, readFileSync } from 'node:fs';
import type { BrainEngine } from '../engine.ts';
import type { PhaseResult, PhaseError } from '../cycle.ts';
import { MinionQueue } from '../minions/queue.ts';
import { waitForCompletion, TimeoutError } from '../minions/wait-for-completion.ts';
import type { MinionJobInput, SubagentHandlerData } from '../minions/types.ts';
import { serializeMarkdown } from '../markdown.ts';
import type { Page, PageType } from '../types.ts';
export interface PatternsPhaseOpts {
brainDir: string;
dryRun: boolean;
yieldDuringPhase?: () => Promise<void>;
}
export async function runPhasePatterns(
engine: BrainEngine,
opts: PatternsPhaseOpts,
): Promise<PhaseResult> {
const start = Date.now();
try {
const config = await loadPatternsConfig(engine);
if (!config.enabled) {
return skipped('disabled', 'dream.patterns.enabled is false');
}
// Gather reflections within lookback window.
const reflections = await gatherReflections(engine, config.lookbackDays);
if (reflections.length < config.minEvidence) {
return skipped(
'insufficient_evidence',
`${reflections.length} reflections in last ${config.lookbackDays}d (need ≥${config.minEvidence})`,
);
}
if (opts.dryRun) {
return ok(`dry-run: would detect patterns over ${reflections.length} reflections`, {
reflections_considered: reflections.length,
patterns_written: 0,
dryRun: true,
});
}
// Submit one subagent for pattern detection.
if (!process.env.ANTHROPIC_API_KEY) {
return skipped('no_api_key', 'ANTHROPIC_API_KEY unset; pattern detection skipped');
}
const allowedSlugPrefixes = await loadAllowedSlugPrefixes();
if (allowedSlugPrefixes.length === 0) {
return failed(makeError('InternalError', 'NO_ALLOWLIST',
'skills/_brain-filing-rules.json missing dream_synthesize_paths.globs'));
}
const queue = new MinionQueue(engine);
const data: SubagentHandlerData = {
prompt: buildPatternsPrompt(reflections, config.minEvidence),
model: config.model,
max_turns: 30,
allowed_slug_prefixes: allowedSlugPrefixes,
};
const submitOpts: Partial<MinionJobInput> = {
max_stalled: 3,
timeout_ms: 30 * 60 * 1000,
};
const job = await queue.add('subagent', data as unknown as Record<string, unknown>, submitOpts, {
allowProtectedSubmit: true,
});
let outcome: string;
try {
const final = await waitForCompletion(queue, job.id, {
timeoutMs: 35 * 60 * 1000,
pollMs: 5 * 1000,
});
outcome = final.status;
} catch (e) {
if (e instanceof TimeoutError) outcome = 'timeout';
else throw e;
}
if (opts.yieldDuringPhase) {
try { await opts.yieldDuringPhase(); } catch { /* best-effort */ }
}
// Collect slugs the subagent wrote (codex finding #2 — query tool exec rows).
const writtenSlugs = await collectChildPutPageSlugs(engine, [job.id]);
// Reverse-write to fs.
const reverseWriteCount = await reverseWriteSlugs(engine, opts.brainDir, writtenSlugs);
return ok(`${writtenSlugs.length} pattern page(s) written/updated (${outcome})`, {
reflections_considered: reflections.length,
patterns_written: writtenSlugs.length,
reverse_write_count: reverseWriteCount,
child_outcome: outcome,
job_id: job.id,
});
} catch (e) {
return failed(makeError('InternalError', 'PATTERNS_PHASE_FAIL',
e instanceof Error ? (e.message || 'patterns phase threw') : String(e)));
} finally {
void start;
}
}
// ── Config ────────────────────────────────────────────────────────────
interface PatternsConfig {
enabled: boolean;
lookbackDays: number;
minEvidence: number;
model: string;
}
async function loadPatternsConfig(engine: BrainEngine): Promise<PatternsConfig> {
const enabledStr = await engine.getConfig('dream.patterns.enabled');
const enabled = enabledStr === null ? true : enabledStr === 'true';
const lookbackStr = await engine.getConfig('dream.patterns.lookback_days');
const minEvidenceStr = await engine.getConfig('dream.patterns.min_evidence');
const model = (await engine.getConfig('dream.patterns.model')) || 'claude-sonnet-4-6';
return {
enabled,
lookbackDays: lookbackStr ? Math.max(1, parseInt(lookbackStr, 10) || 30) : 30,
minEvidence: minEvidenceStr ? Math.max(1, parseInt(minEvidenceStr, 10) || 3) : 3,
model,
};
}
// ── Reflection gathering ─────────────────────────────────────────────
interface ReflectionRef {
slug: string;
title: string;
excerpt: string;
}
async function gatherReflections(
engine: BrainEngine,
lookbackDays: number,
): Promise<ReflectionRef[]> {
const since = new Date(Date.now() - lookbackDays * 24 * 60 * 60 * 1000).toISOString();
const rows = await engine.executeRaw<{ slug: string; title: string | null; compiled_truth: string | null }>(
`SELECT slug, title, compiled_truth
FROM pages
WHERE slug LIKE 'wiki/personal/reflections/%'
AND updated_at >= $1::timestamptz
ORDER BY updated_at DESC
LIMIT 100`,
[since],
);
return rows.map(r => ({
slug: r.slug,
title: r.title ?? r.slug,
excerpt: (r.compiled_truth ?? '').slice(0, 600),
}));
}
// ── Prompt ────────────────────────────────────────────────────────────
function buildPatternsPrompt(reflections: ReflectionRef[], minEvidence: number): string {
const today = new Date().toISOString().slice(0, 10);
const corpus = reflections
.map((r, i) => `### ${i + 1}. [[${r.slug}]] — ${r.title}\n${r.excerpt}`)
.join('\n\n---\n\n');
return `You are surfacing recurring themes across the user's recent reflections.
OUTPUT POLICY
- Only name a pattern if it appears in at least ${minEvidence} DISTINCT reflections.
- Each pattern page MUST cite the reflections that constitute its evidence (use [[wiki/personal/reflections/...]] wikilinks).
- Use \`search\` to check whether a similar pattern page already exists; if yes, update it (use the same slug). If no, create a new one.
- Pattern slug format: \`wiki/personal/patterns/<topic-slug>\` (lowercase alphanumeric + hyphens; no underscores, no extension, no date).
- A "pattern" is a recurring theme, anxiety, decision pattern, relationship dynamic, or self-knowledge motif. NOT a single insight. NOT a list of unrelated topics.
DO NOT WRITE
- A "patterns from today" digest (that's the dream-cycle-summaries page; not your job).
- Patterns with <${minEvidence} reflections cited.
- Anything outside wiki/personal/patterns/.
CONTEXT
- Today: ${today}
- Reflections in scope: ${reflections.length}
REFLECTIONS
${corpus}
When done, briefly list the pattern slugs you wrote/updated in your final message.`;
}
// ── Provenance via put_page tool execution rows ─────────────────────
async function collectChildPutPageSlugs(
engine: BrainEngine,
childIds: number[],
): Promise<string[]> {
if (childIds.length === 0) return [];
const rows = await engine.executeRaw<{ slug: string }>(
`SELECT DISTINCT input->>'slug' AS slug
FROM subagent_tool_executions
WHERE job_id = ANY($1::int[])
AND tool_name = 'brain_put_page'
AND status = 'complete'
AND input ? 'slug'
ORDER BY 1`,
[childIds],
);
return rows.map(r => r.slug).filter((s): s is string => typeof s === 'string' && s.length > 0);
}
// ── Reverse-write ────────────────────────────────────────────────────
async function reverseWriteSlugs(
engine: BrainEngine,
brainDir: string,
slugs: string[],
): Promise<number> {
let count = 0;
for (const slug of slugs) {
const page = await engine.getPage(slug);
if (!page) continue;
const tags = await engine.getTags(slug);
try {
const md = renderPageToMarkdown(page, tags);
const filePath = join(brainDir, `${slug}.md`);
mkdirSync(dirname(filePath), { recursive: true });
writeFileSync(filePath, md, 'utf8');
count++;
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
process.stderr.write(`[dream] reverse-write ${slug} failed: ${msg}\n`);
}
}
return count;
}
function renderPageToMarkdown(page: Page, tags: string[]): string {
const frontmatter = (page.frontmatter ?? {}) as Record<string, unknown>;
return serializeMarkdown(
frontmatter,
page.compiled_truth ?? '',
page.timeline ?? '',
{
type: (page.type as PageType) ?? 'note',
title: page.title ?? '',
tags,
},
);
}
// ── Allow-list (shared with synthesize.ts) ───────────────────────────
async function loadAllowedSlugPrefixes(): Promise<string[]> {
const candidates = [
join(process.cwd(), 'skills', '_brain-filing-rules.json'),
join(__dirname, '..', '..', '..', 'skills', '_brain-filing-rules.json'),
];
for (const path of candidates) {
if (!existsSync(path)) continue;
try {
const raw = readFileSync(path, 'utf8');
const parsed = JSON.parse(raw) as { dream_synthesize_paths?: { globs?: unknown } };
const globs = parsed?.dream_synthesize_paths?.globs;
if (Array.isArray(globs) && globs.every(g => typeof g === 'string')) {
return globs as string[];
}
} catch { /* try next */ }
}
return [];
}
// ── Status helpers ───────────────────────────────────────────────────
function ok(summary: string, details: Record<string, unknown> = {}): PhaseResult {
return { phase: 'patterns', status: 'ok', duration_ms: 0, summary, details };
}
function skipped(reason: string, summary: string): PhaseResult {
return {
phase: 'patterns',
status: 'skipped',
duration_ms: 0,
summary,
details: { reason },
};
}
function failed(error: PhaseError): PhaseResult {
return {
phase: 'patterns',
status: 'fail',
duration_ms: 0,
summary: 'patterns phase failed',
details: {},
error,
};
}
function makeError(cls: string, code: string, message: string, hint?: string): PhaseError {
return hint ? { class: cls, code, message, hint } : { class: cls, code, message };
}
+637
View File
@@ -0,0 +1,637 @@
/**
* Synthesize phase (v0.23) conversation-to-brain pipeline.
*
* Reads transcripts from the configured corpus dir, runs a cheap Haiku
* "is this worth processing?" verdict (cached in `dream_verdicts`), then
* fans out one Sonnet subagent per worth-processing transcript with the
* trusted-workspace `allowed_slug_prefixes` list. After children resolve,
* the orchestrator queries `subagent_tool_executions` for the put_page
* slugs each child wrote (codex finding #2: NOT a time-windowed pages
* query picks up unrelated writes), reverse-renders each new page from
* DB to disk, and writes a deterministic summary index.
*
* Hard guarantees:
* - Subagent never gets fs-write access. Orchestrator holds the dual-write.
* - Allow-list is sourced from `skills/_brain-filing-rules.json` (single
* source of truth) and threaded as handler data; PROTECTED_JOB_NAMES
* prevents MCP from submitting `subagent` jobs, so the field is trusted.
* - Cooldown via `dream.synthesize.last_completion_ts` config key
* written ONLY on success (codex finding #5 deferral: no auto git commit
* in v1).
* - Idempotency via `dream:synth:<file_path>:<content_hash>` job key.
* - Edited transcripts produce slugs with content-hash suffix no overwrite.
*
* NOT in v1:
* - git auto-commit / push (deferred to v1.1, codex finding #5).
* - Daily token budget cap (cooldown bounds spend at v1 scale).
*/
import Anthropic from '@anthropic-ai/sdk';
import { readFileSync, existsSync, writeFileSync, mkdirSync } from 'node:fs';
import { join, dirname } from 'node:path';
import type { BrainEngine } from '../engine.ts';
import type { PhaseResult, PhaseError } from '../cycle.ts';
import { MinionQueue } from '../minions/queue.ts';
import { waitForCompletion, TimeoutError } from '../minions/wait-for-completion.ts';
import type { MinionJobInput, SubagentHandlerData } from '../minions/types.ts';
import { discoverTranscripts, type DiscoveredTranscript } from './transcript-discovery.ts';
import { serializeMarkdown } from '../markdown.ts';
import type { Page, PageType } from '../types.ts';
// Slug regex from validatePageSlug — kept in sync.
// Used for the orchestrator-written summary index slug.
const SUMMARY_SLUG_RE = /^[a-z0-9][a-z0-9\-]*(\/[a-z0-9][a-z0-9\-]*)*$/;
// ── Public entry ──────────────────────────────────────────────────────
export interface SynthesizePhaseOpts {
brainDir: string;
dryRun: boolean;
/** Generic in-cycle keepalive for cycle-lock TTL renewal during long waits. */
yieldDuringPhase?: () => Promise<void>;
/**
* Override the corpus directory and other tunables. Primarily for the
* `gbrain dream --input <file>` ad-hoc path; bypasses config reads.
*/
inputFile?: string;
date?: string;
from?: string;
to?: string;
/**
* Disable the self-consumption guard. Wired from the
* `--unsafe-bypass-dream-guard` CLI flag. NOT auto-applied for `--input`
* because that would allow any dream-generated page to silently re-enter
* the synthesize loop. Caller must opt in explicitly.
*/
bypassDreamGuard?: boolean;
}
export async function runPhaseSynthesize(
engine: BrainEngine,
opts: SynthesizePhaseOpts,
): Promise<PhaseResult> {
const start = Date.now();
try {
const config = await loadSynthConfig(engine);
// Allow ad-hoc --input to run even when config is disabled.
if (!opts.inputFile && !config.enabled) {
return skipped('not_configured',
'dream.synthesize.enabled is false (set dream.synthesize.session_corpus_dir to enable)');
}
if (!opts.inputFile && !config.corpusDir) {
return skipped('not_configured',
'dream.synthesize.session_corpus_dir is unset');
}
// Cooldown check (skipped for explicit --input / --date / --from / --to runs).
const explicitTarget = opts.inputFile || opts.date || opts.from || opts.to;
if (!explicitTarget) {
const cooldown = await checkCooldown(engine, config.cooldownHours);
if (cooldown.active) {
return skipped('cooldown_active',
`synthesize cooled down until ${cooldown.expires_at} (${config.cooldownHours}h cooldown)`);
}
}
if (opts.bypassDreamGuard) {
process.stderr.write(
'[dream] WARNING: --unsafe-bypass-dream-guard set; self-consumption guard disabled. ' +
'Re-ingestion of dream output will incur Sonnet costs forever.\n',
);
}
// Discover.
const transcripts = opts.inputFile
? loadAdHocTranscript(opts.inputFile, config.minChars, config.excludePatterns, opts.bypassDreamGuard)
: discoverTranscripts({
corpusDir: config.corpusDir!,
meetingTranscriptsDir: config.meetingTranscriptsDir ?? undefined,
minChars: config.minChars,
excludePatterns: config.excludePatterns,
date: opts.date,
from: opts.from,
to: opts.to,
bypassGuard: opts.bypassDreamGuard,
});
if (transcripts.length === 0) {
return ok('no transcripts to process', { transcripts_processed: 0, pages_written: 0 });
}
// Significance verdicts (cached in dream_verdicts; Haiku on miss).
const worthProcessing: DiscoveredTranscript[] = [];
const verdicts: Array<{ filePath: string; worth: boolean; reasons: string[]; cached: boolean }> = [];
const haiku = makeHaikuClient(); // null if no API key
for (const t of transcripts) {
const cached = await engine.getDreamVerdict(t.filePath, t.contentHash);
if (cached) {
verdicts.push({ filePath: t.filePath, worth: cached.worth_processing, reasons: cached.reasons, cached: true });
if (cached.worth_processing) worthProcessing.push(t);
continue;
}
if (!haiku) {
// No API key — can't judge. Skip with explicit reason; don't crash phase.
verdicts.push({ filePath: t.filePath, worth: false, reasons: ['no ANTHROPIC_API_KEY for significance judge'], cached: false });
continue;
}
const verdict = await judgeSignificance(haiku, t, config.verdictModel);
await engine.putDreamVerdict(t.filePath, t.contentHash, verdict);
verdicts.push({ filePath: t.filePath, worth: verdict.worth_processing, reasons: verdict.reasons, cached: false });
if (verdict.worth_processing) worthProcessing.push(t);
}
// Dry-run stops here: significance filter ran (Haiku verdicts cached),
// but no Sonnet synthesis. Codex finding #8: --dry-run does NOT mean
// "zero LLM calls"; it means "skip Sonnet."
if (opts.dryRun) {
return ok(`dry-run: ${worthProcessing.length} of ${transcripts.length} transcripts would synthesize`, {
transcripts_discovered: transcripts.length,
transcripts_processed: 0,
pages_written: 0,
verdicts,
dryRun: true,
});
}
if (worthProcessing.length === 0) {
// Even with verdicts, the cooldown timestamp is updated only on a
// real successful run — not on "nothing worth processing." Lets a
// re-run pick up if a new transcript lands later.
return ok('all transcripts skipped by significance filter', {
transcripts_discovered: transcripts.length,
transcripts_processed: 0,
pages_written: 0,
verdicts,
});
}
// Fan-out: submit one subagent per worth-processing transcript.
const allowedSlugPrefixes = await loadAllowedSlugPrefixes();
if (allowedSlugPrefixes.length === 0) {
return failed(makeError('InternalError', 'NO_ALLOWLIST',
'skills/_brain-filing-rules.json missing dream_synthesize_paths.globs'));
}
const queue = new MinionQueue(engine);
const childIds: number[] = [];
for (const t of worthProcessing) {
const childData: SubagentHandlerData = {
prompt: buildSynthesisPrompt(t),
model: config.model,
max_turns: 30,
allowed_slug_prefixes: allowedSlugPrefixes,
};
const submitOpts: Partial<MinionJobInput> = {
max_stalled: 3,
on_child_fail: 'continue',
idempotency_key: `dream:synth:${t.filePath}:${t.contentHash.slice(0, 16)}`,
timeout_ms: 30 * 60 * 1000, // 30 min per transcript
};
const child = await queue.add(
'subagent',
childData as unknown as Record<string, unknown>,
submitOpts,
{ allowProtectedSubmit: true },
);
childIds.push(child.id);
}
// Wait for every child to reach a terminal state. Tick yieldDuringPhase
// every 5 min so the cycle lock TTL refreshes.
const childOutcomes: Array<{ jobId: number; status: string }> = [];
for (const jobId of childIds) {
try {
const job = await waitForCompletion(queue, jobId, {
timeoutMs: 35 * 60 * 1000,
pollMs: 5 * 1000,
});
childOutcomes.push({ jobId, status: job.status });
} catch (e) {
if (e instanceof TimeoutError) {
childOutcomes.push({ jobId, status: 'timeout' });
} else {
throw e;
}
}
// After each child terminal, give the cycle lock + worker job lock a chance.
if (opts.yieldDuringPhase) {
try { await opts.yieldDuringPhase(); } catch { /* best-effort */ }
}
}
// Collect slugs from put_page tool executions across the children
// (codex finding #2: deterministic provenance, NOT pages.updated_at).
const writtenSlugs = await collectChildPutPageSlugs(engine, childIds);
// Dual-write: reverse-render each DB row → markdown file.
const reverseWriteCount = await reverseWriteSlugs(engine, opts.brainDir, writtenSlugs);
// Summary index page (deterministic; orchestrator-written via direct
// engine.putPage so no allow-list path needed).
const summaryDate = opts.date ?? today();
const summarySlug = `dream-cycle-summaries/${summaryDate}`;
if (SUMMARY_SLUG_RE.test(summarySlug)) {
await writeSummaryPage(engine, opts.brainDir, summarySlug, summaryDate, writtenSlugs, childOutcomes);
}
// Write completion timestamp ON SUCCESS only.
await engine.setConfig('dream.synthesize.last_completion_ts', new Date().toISOString());
const ms = Date.now() - start;
return ok(`${worthProcessing.length} transcript(s) synthesized in ${(ms / 1000).toFixed(1)}s`, {
transcripts_discovered: transcripts.length,
transcripts_processed: worthProcessing.length,
pages_written: writtenSlugs.length,
reverse_write_count: reverseWriteCount,
child_outcomes: childOutcomes,
summary_slug: summarySlug,
verdicts,
});
} catch (e) {
return failed(makeError('InternalError', 'SYNTH_PHASE_FAIL',
e instanceof Error ? (e.message || 'synthesize phase threw') : String(e)));
}
}
// ── Config ────────────────────────────────────────────────────────────
interface SynthConfig {
enabled: boolean;
corpusDir: string | null;
meetingTranscriptsDir: string | null;
minChars: number;
excludePatterns: string[];
model: string;
verdictModel: string;
cooldownHours: number;
}
async function loadSynthConfig(engine: BrainEngine): Promise<SynthConfig> {
const enabled = (await engine.getConfig('dream.synthesize.enabled')) === 'true';
const corpusDir = await engine.getConfig('dream.synthesize.session_corpus_dir');
const meetingTranscriptsDir = await engine.getConfig('dream.synthesize.meeting_transcripts_dir');
const minCharsStr = await engine.getConfig('dream.synthesize.min_chars');
const excludeStr = await engine.getConfig('dream.synthesize.exclude_patterns');
const model = (await engine.getConfig('dream.synthesize.model')) || 'claude-sonnet-4-6';
const verdictModel = (await engine.getConfig('dream.synthesize.verdict_model')) || 'claude-haiku-4-5-20251001';
const cooldownHoursStr = await engine.getConfig('dream.synthesize.cooldown_hours');
let excludePatterns: string[] = ['medical', 'therapy'];
if (excludeStr) {
try {
const parsed = JSON.parse(excludeStr);
if (Array.isArray(parsed)) excludePatterns = parsed.filter(p => typeof p === 'string');
} catch { /* keep default */ }
}
return {
enabled,
corpusDir: corpusDir ?? null,
meetingTranscriptsDir: meetingTranscriptsDir ?? null,
minChars: minCharsStr ? Math.max(0, parseInt(minCharsStr, 10) || 2000) : 2000,
excludePatterns,
model,
verdictModel,
cooldownHours: cooldownHoursStr ? Math.max(0, parseInt(cooldownHoursStr, 10) || 12) : 12,
};
}
async function checkCooldown(
engine: BrainEngine,
hours: number,
): Promise<{ active: boolean; expires_at?: string }> {
if (hours <= 0) return { active: false };
const last = await engine.getConfig('dream.synthesize.last_completion_ts');
if (!last) return { active: false };
const lastMs = Date.parse(last);
if (Number.isNaN(lastMs)) return { active: false };
const expiresMs = lastMs + hours * 60 * 60 * 1000;
if (Date.now() >= expiresMs) return { active: false };
return { active: true, expires_at: new Date(expiresMs).toISOString() };
}
// ── Allow-list source of truth ───────────────────────────────────────
async function loadAllowedSlugPrefixes(): Promise<string[]> {
// Search a few known locations relative to the binary / repo. The first
// hit wins; if none found, return [].
const candidates = [
join(process.cwd(), 'skills', '_brain-filing-rules.json'),
join(__dirname, '..', '..', '..', 'skills', '_brain-filing-rules.json'),
];
for (const path of candidates) {
if (!existsSync(path)) continue;
try {
const raw = readFileSync(path, 'utf8');
const parsed = JSON.parse(raw) as { dream_synthesize_paths?: { globs?: unknown } };
const globs = parsed?.dream_synthesize_paths?.globs;
if (Array.isArray(globs) && globs.every(g => typeof g === 'string')) {
return globs as string[];
}
} catch { /* try next */ }
}
return [];
}
// ── Significance judge (Haiku) ───────────────────────────────────────
export interface JudgeClient {
create: (params: Anthropic.MessageCreateParamsNonStreaming) => Promise<Anthropic.Message>;
}
function makeHaikuClient(): JudgeClient | null {
if (!process.env.ANTHROPIC_API_KEY) return null;
const client = new Anthropic();
return { create: client.messages.create.bind(client.messages) };
}
interface VerdictResult {
worth_processing: boolean;
reasons: string[];
}
export async function judgeSignificance(
client: JudgeClient,
t: DiscoveredTranscript,
verdictModel = 'claude-haiku-4-5-20251001',
): Promise<VerdictResult> {
// Truncate the transcript at 8K chars for cost control. Haiku's verdict
// doesn't need the full body; the opening + closing sections are usually
// representative of significance.
const trimmed = t.content.length > 8000
? t.content.slice(0, 4000) + '\n[...truncated...]\n' + t.content.slice(-4000)
: t.content;
const sys = `You judge whether a conversation transcript is worth synthesizing into a personal knowledge brain.
WORTH PROCESSING (return worth_processing=true):
- The user articulates a new idea, frame, mental model, or thesis
- The user reflects on themselves, names patterns, processes emotion
- The user discusses specific people, companies, or decisions in depth
- The user makes a strategic call worth remembering
NOT WORTH PROCESSING (return worth_processing=false):
- Routine ops ("check my email", "schedule X")
- Pure code debugging without user reflection
- Short message exchanges with no original thought
- Repetitive content the brain already has
Respond as JSON: {"worth_processing": <bool>, "reasons": ["<short>", "<short>"]}.
Two reasons max, one phrase each.`;
const msg = await client.create({
model: verdictModel,
max_tokens: 200,
system: sys,
messages: [{ role: 'user', content: `Transcript ${t.basename}:\n\n${trimmed}` }],
});
for (const block of msg.content) {
if (block.type === 'text') {
const text = block.text.trim();
const m = /\{[\s\S]*\}/.exec(text);
if (!m) continue;
try {
const parsed = JSON.parse(m[0]) as { worth_processing?: unknown; reasons?: unknown };
const worth = parsed.worth_processing === true;
const reasons = Array.isArray(parsed.reasons)
? parsed.reasons.filter((r): r is string => typeof r === 'string').slice(0, 4)
: [];
return { worth_processing: worth, reasons };
} catch { /* fall through */ }
}
}
// Couldn't parse — default to NOT processing (cheap fallback).
return { worth_processing: false, reasons: ['judge response unparseable'] };
}
// ── Subagent prompt ──────────────────────────────────────────────────
function buildSynthesisPrompt(t: DiscoveredTranscript): string {
const dateHint = t.inferredDate ?? today();
const hashSuffix = t.contentHash.slice(0, 6);
const baseSlugSegment = sanitizeForSlug(t.basename) || `session-${dateHint}`;
return `You are synthesizing a conversation transcript into the user's personal knowledge brain.
CONTEXT
- Today's date: ${dateHint}
- Transcript hash suffix (USE THIS in slugs): ${hashSuffix}
- Source file basename: ${baseSlugSegment}
OUTPUT POLICY (ALL of these are required)
1. Quote the user verbatim. Do not paraphrase memorable phrasings.
2. Cross-reference compulsively: every new page MUST contain at least one wikilink (e.g., \`[ref](people/jane-doe)\` or \`[[people/jane-doe]]\`) to existing brain content. Use the search tool to find existing pages first.
3. Do NOT write to any path outside the allow-list shown in the put_page schema.
4. Slug discipline: lowercase alphanumeric and hyphens only, slash-separated segments. NO underscores, NO file extensions.
TASKS
A. Reflections (self-knowledge, pattern recognition, emotional processing):
slug: \`wiki/personal/reflections/${dateHint}-<topic-slug>-${hashSuffix}\`
B. Originals (new ideas, frames, theses, mental models):
slug: \`wiki/originals/ideas/${dateHint}-<idea-slug>-${hashSuffix}\`
C. People mentions: search first; if a page exists, do not put_page over it (the orchestrator handles people enrichment via timeline entries your job is the reflection/original synthesis, NOT modifying existing person pages).
D. If nothing in this transcript meets the bar (significance filter already passed but the content is still routine), return without writing anything.
TRANSCRIPT (${t.filePath})
---
${t.content}
---
When done, briefly list the slugs you wrote in your final message so the orchestrator can audit.`;
}
function sanitizeForSlug(s: string): string {
return s
.toLowerCase()
.replace(/[^a-z0-9]+/g, '-')
.replace(/^-+|-+$/g, '')
.slice(0, 60);
}
// ── Slug collection from child put_page calls (codex #2) ────────────
async function collectChildPutPageSlugs(
engine: BrainEngine,
childIds: number[],
): Promise<string[]> {
if (childIds.length === 0) return [];
const rows = await engine.executeRaw<{ slug: string }>(
`SELECT DISTINCT input->>'slug' AS slug
FROM subagent_tool_executions
WHERE job_id = ANY($1::int[])
AND tool_name = 'brain_put_page'
AND status = 'complete'
AND input ? 'slug'
ORDER BY 1`,
[childIds],
);
return rows.map(r => r.slug).filter((s): s is string => typeof s === 'string' && s.length > 0);
}
// ── Reverse-write DB rows → markdown files ───────────────────────────
async function reverseWriteSlugs(
engine: BrainEngine,
brainDir: string,
slugs: string[],
): Promise<number> {
let count = 0;
for (const slug of slugs) {
const page = await engine.getPage(slug);
if (!page) continue;
const tags = await engine.getTags(slug);
try {
const md = renderPageToMarkdown(page, tags);
const filePath = join(brainDir, `${slug}.md`);
mkdirSync(dirname(filePath), { recursive: true });
writeFileSync(filePath, md, 'utf8');
count++;
} catch (e) {
// Per-slug failures are non-fatal — phase continues.
const msg = e instanceof Error ? e.message : String(e);
process.stderr.write(`[dream] reverse-write ${slug} failed: ${msg}\n`);
}
}
return count;
}
/**
* Render a Page to markdown, stamping the dream-output identity marker into
* frontmatter. This stamp is the explicit identity surface checked by
* `isDreamOutput` in transcript-discovery.ts. Stamping at render time covers
* every reverse-write path (subagent reflections + originals + summary) with
* one funnel; the prior content-pattern guard could miss real output because
* `serializeMarkdown` does not embed the page slug in the body.
*/
export function renderPageToMarkdown(page: Page, tags: string[]): string {
const frontmatter: Record<string, unknown> = {
...((page.frontmatter ?? {}) as Record<string, unknown>),
dream_generated: true,
dream_cycle_date: today(),
};
return serializeMarkdown(
frontmatter,
page.compiled_truth ?? '',
page.timeline ?? '',
{
type: (page.type as PageType) ?? 'note',
title: page.title ?? '',
tags,
},
);
}
// ── Summary index page ───────────────────────────────────────────────
async function writeSummaryPage(
engine: BrainEngine,
brainDir: string,
summarySlug: string,
summaryDate: string,
writtenSlugs: string[],
childOutcomes: Array<{ jobId: number; status: string }>,
): Promise<void> {
const completed = childOutcomes.filter(c => c.status === 'completed').length;
const failed = childOutcomes.length - completed;
const lines: string[] = [];
lines.push(`# Dream cycle ${summaryDate}`);
lines.push('');
lines.push(`**Children:** ${completed} completed, ${failed} failed/timeout.`);
lines.push(`**Pages written:** ${writtenSlugs.length}.`);
lines.push('');
if (writtenSlugs.length > 0) {
lines.push('## Pages');
lines.push('');
for (const s of writtenSlugs) {
lines.push(`- [[${s}]]`);
}
lines.push('');
}
const body = lines.join('\n');
// Stamp the dream-output identity marker into the summary's frontmatter.
// parseMarkdown below round-trips it into the DB-stored frontmatter, so the
// marker survives any later reverse-render of the summary page.
const fullMarkdown = serializeMarkdown(
{ dream_generated: true, dream_cycle_date: summaryDate } as Record<string, unknown>,
body,
'',
{ type: 'note' as PageType, title: `Dream cycle ${summaryDate}`, tags: ['dream-cycle'] },
);
// Direct engine.putPage — orchestrator write, no subagent context, no
// allow-list check (server-side viaSubagent=false). The summary slug is
// pre-validated against SUMMARY_SLUG_RE in the caller.
// Importing put_page via operations.ts would re-run namespace logic
// unnecessarily; we go straight to the engine.
const { parseMarkdown } = await import('../markdown.ts');
const parsed = parseMarkdown(fullMarkdown);
await engine.putPage(summarySlug, {
type: parsed.type,
title: parsed.title,
compiled_truth: parsed.compiled_truth,
timeline: parsed.timeline,
frontmatter: parsed.frontmatter,
});
// Also write to disk (orchestrator dual-write).
try {
const filePath = join(brainDir, `${summarySlug}.md`);
mkdirSync(dirname(filePath), { recursive: true });
writeFileSync(filePath, fullMarkdown, 'utf8');
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
process.stderr.write(`[dream] summary file-write failed: ${msg}\n`);
}
}
// ── Helpers ──────────────────────────────────────────────────────────
function loadAdHocTranscript(
filePath: string,
minChars: number,
excludePatterns: string[],
bypassGuard?: boolean,
): DiscoveredTranscript[] {
const { readSingleTranscript } = require('./transcript-discovery.ts') as typeof import('./transcript-discovery.ts');
const t = readSingleTranscript(filePath, { minChars, excludePatterns, bypassGuard });
return t ? [t] : [];
}
function today(): string {
return new Date().toISOString().slice(0, 10);
}
function ok(summary: string, details: Record<string, unknown> = {}): PhaseResult {
return { phase: 'synthesize', status: 'ok', duration_ms: 0, summary, details };
}
function skipped(reason: string, summary: string): PhaseResult {
return {
phase: 'synthesize',
status: 'skipped',
duration_ms: 0,
summary,
details: { reason },
};
}
function failed(error: PhaseError): PhaseResult {
return {
phase: 'synthesize',
status: 'fail',
duration_ms: 0,
summary: 'synthesize phase failed',
details: {},
error,
};
}
function makeError(cls: string, code: string, message: string, hint?: string): PhaseError {
return hint ? { class: cls, code, message, hint } : { class: cls, code, message };
}
+231
View File
@@ -0,0 +1,231 @@
/**
* Transcript discovery for the v0.23 dream-cycle synthesize phase.
*
* Walks a corpus directory for `.txt` files, applies date-range filters,
* size filters (min_chars), and word-boundary regex exclude patterns.
* Returns a list of file paths + content + content_hash so the caller
* can key the verdict cache and dispatch one subagent per transcript.
*
* No DB; pure filesystem + crypto. Tested with hermetic temp directories.
*/
import { readFileSync, readdirSync, statSync } from 'node:fs';
import { join, basename } from 'node:path';
import { createHash } from 'node:crypto';
export interface DiscoveredTranscript {
/** Absolute path to the transcript file. */
filePath: string;
/** sha256(content), full hex; callers slice as needed. */
contentHash: string;
/** Raw transcript text. */
content: string;
/** Filename basename without extension; used as a topic-slug seed. */
basename: string;
/** Inferred date if the basename matches `YYYY-MM-DD...` (or null). */
inferredDate: string | null;
}
export interface DiscoverOpts {
/** Source directory. Required. */
corpusDir: string;
/** Optional second source. */
meetingTranscriptsDir?: string;
/** Skip transcripts smaller than this many characters. Default 2000. */
minChars?: number;
/** Word-boundary regex strings. The discoverer auto-wraps bare words. */
excludePatterns?: string[];
/** Restrict to a single date (YYYY-MM-DD basename match). */
date?: string;
/** Inclusive range start (YYYY-MM-DD). */
from?: string;
/** Inclusive range end (YYYY-MM-DD). */
to?: string;
/**
* Disable the self-consumption guard. Caller must opt in explicitly via
* `--unsafe-bypass-dream-guard`; never auto-applied for `--input` because
* that would let any caller silently re-trigger the loop bug.
*/
bypassGuard?: boolean;
}
const DATE_RE = /^(\d{4}-\d{2}-\d{2})/;
const WORD_BOUNDARY_HEURISTIC = /^[a-zA-Z][a-zA-Z0-9_-]*$/;
/**
* Self-consumption guard: identity-marker check against `dream_generated: true`
* stamped by the synthesize phase's render paths.
*
* v0.23.1 used a body slug-prefix string match. Codex review of the v0.23.2
* plan caught two flaws: (1) `serializeMarkdown` does NOT embed the page slug
* into body content, so the prefix heuristic could miss real dream output, and
* (2) real conversation transcripts that legitimately cite a brain page would
* be silently dropped. v0.23.2 swaps content inference for explicit identity
* stamped at render time.
*
* Regex anchored at frontmatter open (`---\n`), tolerates optional BOM and CRLF,
* scans the first 2000 chars for `dream_generated: true` (any whitespace, case-
* insensitive value, word boundary on `true`).
*/
const DREAM_MARKER_REGEX_SRC =
'^\\uFEFF?-{3}\\r?\\n[\\s\\S]{0,2000}?dream_generated\\s*:\\s*true\\b';
export const DREAM_OUTPUT_MARKER_RE = new RegExp(DREAM_MARKER_REGEX_SRC, 'i');
export function isDreamOutput(content: string, bypass = false): boolean {
if (bypass) return false;
return DREAM_OUTPUT_MARKER_RE.test(content);
}
/**
* Auto-wrap bare-word patterns in `\b<word>\b`. Power users can pass full
* regex (e.g. `^therapy:`) which we honor verbatim. Heuristic: any input
* that's purely alphanumeric+hyphen+underscore is treated as a bare word.
*/
export function compileExcludePatterns(patterns: string[] | undefined): RegExp[] {
if (!patterns || patterns.length === 0) return [];
const out: RegExp[] = [];
for (const p of patterns) {
if (!p) continue;
try {
const src = WORD_BOUNDARY_HEURISTIC.test(p) ? `\\b${p}\\b` : p;
out.push(new RegExp(src, 'i'));
} catch (e) {
// Bad regex from user config — skip with stderr warning, don't crash.
const msg = e instanceof Error ? e.message : String(e);
process.stderr.write(`[dream] invalid exclude_pattern '${p}': ${msg}\n`);
}
}
return out;
}
function hashContent(text: string): string {
return createHash('sha256').update(text, 'utf8').digest('hex');
}
function isInDateRange(date: string | null, opts: DiscoverOpts): boolean {
if (!opts.date && !opts.from && !opts.to) return true;
if (!date) return false; // file has no inferable date but a filter is active
if (opts.date && date !== opts.date) return false;
if (opts.from && date < opts.from) return false;
if (opts.to && date > opts.to) return false;
return true;
}
function matchesAnyExclude(text: string, patterns: RegExp[]): boolean {
for (const re of patterns) {
if (re.test(text)) return true;
}
return false;
}
function listTextFiles(dir: string): string[] {
let entries: string[];
try {
entries = readdirSync(dir);
} catch {
return [];
}
const out: string[] = [];
for (const name of entries) {
if (!name.endsWith('.txt')) continue;
const full = join(dir, name);
try {
if (statSync(full).isFile()) out.push(full);
} catch {
// skip unreadable entries
}
}
return out.sort();
}
/**
* Discover transcripts from the configured corpus dirs, applying filters.
*
* Skips files that:
* - aren't `.txt`
* - have date-prefixed basenames outside the requested window
* - have content shorter than `minChars`
* - carry the `dream_generated: true` self-consumption marker (unless `bypassGuard`)
* - match any compiled exclude pattern (case-insensitive word-boundary by default)
*
* Returns sorted by filePath so re-runs are deterministic.
*/
export function discoverTranscripts(opts: DiscoverOpts): DiscoveredTranscript[] {
const minChars = opts.minChars ?? 2000;
const bypass = opts.bypassGuard === true;
const excludeRes = compileExcludePatterns(opts.excludePatterns);
const dirs = [opts.corpusDir, opts.meetingTranscriptsDir].filter(
(d): d is string => typeof d === 'string' && d.length > 0,
);
const results: DiscoveredTranscript[] = [];
for (const dir of dirs) {
for (const filePath of listTextFiles(dir)) {
const baseName = basename(filePath, '.txt');
const dateMatch = DATE_RE.exec(baseName);
const inferredDate = dateMatch ? dateMatch[1] : null;
if (!isInDateRange(inferredDate, opts)) continue;
let content: string;
try {
content = readFileSync(filePath, 'utf8');
} catch {
continue;
}
if (content.length < minChars) continue;
if (isDreamOutput(content, bypass)) {
process.stderr.write(`[dream] skipped ${baseName}: dream_generated marker (self-consumption guard)\n`);
continue;
}
if (matchesAnyExclude(content, excludeRes)) continue;
results.push({
filePath,
contentHash: hashContent(content),
content,
basename: baseName,
inferredDate,
});
}
}
return results.sort((a, b) => a.filePath.localeCompare(b.filePath));
}
/**
* Read a single ad-hoc transcript file (`gbrain dream --input <file>`).
* Bypasses the corpus-dir scan and date filters but still applies
* minChars + exclude_patterns when provided. The self-consumption guard
* also still fires unless `bypassGuard` is set explicitly.
*/
export function readSingleTranscript(
filePath: string,
opts: { minChars?: number; excludePatterns?: string[]; bypassGuard?: boolean } = {},
): DiscoveredTranscript | null {
const minChars = opts.minChars ?? 2000;
const bypass = opts.bypassGuard === true;
const excludeRes = compileExcludePatterns(opts.excludePatterns);
let content: string;
try {
content = readFileSync(filePath, 'utf8');
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
throw new Error(`could not read transcript at ${filePath}: ${msg}`);
}
if (content.length < minChars) return null;
if (isDreamOutput(content, bypass)) {
const baseName = basename(filePath, '.txt');
process.stderr.write(`[dream] readSingleTranscript skipped ${baseName}: dream_generated marker (self-consumption guard)\n`);
return null;
}
if (matchesAnyExclude(content, excludeRes)) return null;
const baseName = basename(filePath, '.txt');
const dateMatch = DATE_RE.exec(baseName);
return {
filePath,
contentHash: hashContent(content),
content,
basename: baseName,
inferredDate: dateMatch ? dateMatch[1] : null,
};
}
+140
View File
@@ -0,0 +1,140 @@
/**
* Generic DB-backed lock primitive.
*
* Reuses the gbrain_cycle_locks table (id PK + holder_pid + ttl_expires_at)
* with a parameterized lock id. Both `gbrain-cycle` (the broad cycle lock)
* and `gbrain-sync` (performSync's writer lock) live here.
*
* Why not pg_advisory_xact_lock: it is session-scoped, and PgBouncer
* transaction pooling drops session state between calls. This row-based
* lock survives PgBouncer because it's plain INSERT/UPDATE/DELETE with
* a TTL fallback (a crashed holder's row times out).
*
* Why a separate table-row per lock id rather than reusing the cycle lock:
* the cycle lock is broader (covers every phase). performSync's write-window
* is narrower. If performSync reused the cycle lock and the cycle handler
* called performSync, the inner acquire would deadlock against itself. Two
* lock ids let callers nest cleanly: cycle holds gbrain-cycle for its run;
* performSync (called from anywhere cycle, jobs handler, CLI) takes
* gbrain-sync just for the write window.
*
* v0.22.13 added in PR #490 to fix CODEX-2 (no cross-process lock for
* direct sync paths). The cycle path was already protected.
*/
import { hostname } from 'os';
import type { BrainEngine } from './engine.ts';
export interface DbLockHandle {
id: string;
release: () => Promise<void>;
refresh: () => Promise<void>;
}
/** Default TTL: 30 minutes, same as cycle lock. */
const DEFAULT_TTL_MINUTES = 30;
/**
* Try to acquire a named DB lock.
*
* Returns a handle on success. Returns `null` if another live holder has
* the lock (its row exists and ttl_expires_at is in the future).
*
* The acquire is upsert-style:
* INSERT ... ON CONFLICT (id) DO UPDATE
* ... WHERE existing.ttl_expires_at < NOW()
* RETURNING id
*
* Empty RETURNING means the existing row is still live. An expired holder
* (worker crashed without releasing) is auto-superseded by the UPDATE
* branch.
*/
export async function tryAcquireDbLock(
engine: BrainEngine,
lockId: string,
ttlMinutes: number = DEFAULT_TTL_MINUTES,
): Promise<DbLockHandle | null> {
const pid = process.pid;
const host = hostname();
// Engine-agnostic: prefer the engine's raw escape hatch (`sql` for postgres-js,
// `db.query` for PGLite). Mirrors cycle.ts's pattern so behavior stays identical.
const maybePG = engine as unknown as { sql?: (...args: unknown[]) => Promise<unknown> };
const maybePGLite = engine as unknown as {
db?: { query: (sql: string, params?: unknown[]) => Promise<{ rows: unknown[] }> };
};
if (engine.kind === 'postgres' && maybePG.sql) {
const sql = maybePG.sql as any;
const ttl = `${ttlMinutes} minutes`;
const rows: Array<{ id: string }> = await sql`
INSERT INTO gbrain_cycle_locks (id, holder_pid, holder_host, acquired_at, ttl_expires_at)
VALUES (${lockId}, ${pid}, ${host}, NOW(), NOW() + ${ttl}::interval)
ON CONFLICT (id) DO UPDATE
SET holder_pid = ${pid},
holder_host = ${host},
acquired_at = NOW(),
ttl_expires_at = NOW() + ${ttl}::interval
WHERE gbrain_cycle_locks.ttl_expires_at < NOW()
RETURNING id
`;
if (rows.length === 0) return null;
return {
id: lockId,
refresh: async () => {
await sql`
UPDATE gbrain_cycle_locks
SET ttl_expires_at = NOW() + ${ttl}::interval
WHERE id = ${lockId} AND holder_pid = ${pid}
`;
},
release: async () => {
await sql`
DELETE FROM gbrain_cycle_locks
WHERE id = ${lockId} AND holder_pid = ${pid}
`;
},
};
}
if (engine.kind === 'pglite' && maybePGLite.db) {
const db = maybePGLite.db;
const ttl = `${ttlMinutes} minutes`;
const { rows } = await db.query(
`INSERT INTO gbrain_cycle_locks (id, holder_pid, holder_host, acquired_at, ttl_expires_at)
VALUES ($1, $2, $3, NOW(), NOW() + $4::interval)
ON CONFLICT (id) DO UPDATE
SET holder_pid = $2,
holder_host = $3,
acquired_at = NOW(),
ttl_expires_at = NOW() + $4::interval
WHERE gbrain_cycle_locks.ttl_expires_at < NOW()
RETURNING id`,
[lockId, pid, host, ttl],
);
if (rows.length === 0) return null;
return {
id: lockId,
refresh: async () => {
await db.query(
`UPDATE gbrain_cycle_locks
SET ttl_expires_at = NOW() + $1::interval
WHERE id = $2 AND holder_pid = $3`,
[ttl, lockId, pid],
);
},
release: async () => {
await db.query(
`DELETE FROM gbrain_cycle_locks WHERE id = $1 AND holder_pid = $2`,
[lockId, pid],
);
},
};
}
throw new Error(`Unknown engine kind for db-lock: ${engine.kind}`);
}
/** Lock id for performSync's writer window. Distinct from gbrain-cycle so the
* cycle handler can hold gbrain-cycle while performSync (called from inside
* the cycle) acquires gbrain-sync. */
export const SYNC_LOCK_ID = 'gbrain-sync';
+19
View File
@@ -86,6 +86,19 @@ export interface ReservedConnection {
executeRaw<T = Record<string, unknown>>(sql: string, params?: unknown[]): Promise<T[]>;
}
/** Dream-cycle Haiku verdict on whether a transcript is worth processing. */
export interface DreamVerdict {
worth_processing: boolean;
reasons: string[];
judged_at: string;
}
/** Input shape for putDreamVerdict — judged_at defaults to now() server-side. */
export interface DreamVerdictInput {
worth_processing: boolean;
reasons: string[];
}
/** Maximum results returned by search operations. Internal bulk operations (listPages) are not clamped. */
export const MAX_SEARCH_LIMIT = 100;
@@ -258,6 +271,12 @@ export interface BrainEngine {
putRawData(slug: string, source: string, data: object): Promise<void>;
getRawData(slug: string, source?: string): Promise<RawData[]>;
// Dream-cycle significance verdict cache (v0.23).
// Keyed by (file_path, content_hash). Distinct from raw_data, which is
// page-scoped — transcripts being judged aren't pages yet.
getDreamVerdict(filePath: string, contentHash: string): Promise<DreamVerdict | null>;
putDreamVerdict(filePath: string, contentHash: string, verdict: DreamVerdictInput): Promise<void>;
// Versions
createVersion(slug: string): Promise<PageVersion>;
getVersions(slug: string): Promise<PageVersion[]>;
+4 -3
View File
@@ -12,7 +12,7 @@
import { appendFileSync, readFileSync, existsSync, mkdirSync, writeFileSync, renameSync } from 'fs';
import { join, dirname } from 'path';
import { homedir } from 'os';
import { gbrainPath } from './config.ts';
// ---------------------------------------------------------------------------
// Types
@@ -45,7 +45,8 @@ export interface TestCase {
source: 'fail-improve-loop';
}
const LOG_DIR = join(homedir(), '.gbrain', 'fail-improve');
// Lazy: GBRAIN_HOME may be set after module load, so resolve at call time.
const getLogDir = () => gbrainPath('fail-improve');
const MAX_ENTRIES = 1000;
// ---------------------------------------------------------------------------
@@ -76,7 +77,7 @@ export class FailImproveLoop {
private logDir: string;
constructor(logDir?: string) {
this.logDir = logDir || LOG_DIR;
this.logDir = logDir || getLogDir();
}
/**
+8 -8
View File
@@ -1,5 +1,5 @@
/**
* filing-audit.ts Check 6 of the skillify checklist (W3, v0.17).
* filing-audit.ts Check 6 of the skillify checklist (W3).
*
* For every skill that writes brain pages (`writes_pages: true`),
* verify that:
@@ -8,18 +8,18 @@
* `skills/_brain-filing-rules.json`. `sources/` is explicitly
* allowed (bulk data capture is a legitimate filing target).
*
* Important distinction (D-CX-7): `writes_pages: true` is distinct
* from the pre-existing `mutating: true` field. `mutating:true` means
* "has side effects" (any side effect cron, config, report write).
* Important distinction: `writes_pages: true` is distinct from the
* pre-existing `mutating: true` field. `mutating:true` means "has
* side effects" (any side effect cron, config, report write).
* `writes_pages:true` means "writes brain pages to a semantic
* directory." Cron/config/report-writer skills set `mutating:true`
* but NOT `writes_pages:true`, and so are correctly exempted from
* filing-audit noise.
*
* v0.17 scope: declaration-level audit only (cheap, deterministic).
* v0.18 plan: `filing-audit --pages` walks brain pages and infers
* primary subject via LLM to catch real misfilings vs declarations
* (D-CX-13).
* Current scope: declaration-level audit only (cheap, deterministic).
* A future release may add `filing-audit --pages` to walk brain pages
* and infer primary subject via LLM (catches real misfilings vs
* declarations); that is tracked as follow-up work, not in this scope.
*/
import { existsSync, readFileSync, readdirSync, statSync } from 'fs';
+374
View File
@@ -0,0 +1,374 @@
/**
* Friction reporter JSONL-backed signal capture for the claw-test feedback loop.
*
* The friction CLI (`gbrain friction log/render/list/summary`) writes here.
* The claw-test harness reads here. The agent calls `gbrain friction log`
* directly when it hits something confusing, missing, or wrong.
*
* Storage shape: append-only JSONL files under `$GBRAIN_HOME/friction/`.
* - `<run-id>.jsonl` for each harness run (run-id from $GBRAIN_FRICTION_RUN_ID)
* - `standalone.jsonl` for entries logged outside a harness run
*
* Schema is a flat extension of StructuredAgentError fields (per D20). Render
* reads one level. Readers tolerate malformed lines (skip + warn) so partial
* runs don't break later analysis.
*
* appendFileSync
* writer() <runId>.jsonl (one
* (atomic if line entry per line)
* PIPE_BUF/4KB)
*
*
* reader() / render()
* skip malformed + warn
*/
import { appendFileSync, existsSync, readdirSync, readFileSync, mkdirSync, statSync } from 'fs';
import { dirname, join } from 'path';
import { homedir } from 'os';
import { gbrainPath } from './config.ts';
import { VERSION } from '../version.ts';
// ---------------------------------------------------------------------------
// Schema
// ---------------------------------------------------------------------------
export type FrictionKind = 'friction' | 'delight' | 'phase-marker' | 'interrupted';
export type FrictionSeverity = 'confused' | 'error' | 'blocker' | 'nit';
export type FrictionSource = 'claw' | 'harness';
export type PhaseMarker = 'start' | 'end';
/** One JSONL entry. Flat extension of StructuredAgentError per D20. */
export interface FrictionEntry {
schema_version: '1';
ts: string; // ISO 8601
run_id: string;
phase: string;
kind: FrictionKind;
/** Required for kind=friction|delight. Optional for phase-marker (purely informational). */
severity?: FrictionSeverity;
message: string;
hint?: string;
/** StructuredAgentError envelope fields, flattened. */
class?: string;
code?: string;
docs_url?: string;
source: FrictionSource;
cwd: string;
gbrain_version: string;
agent?: string;
/** Byte offset into the run's transcript.jsonl (live mode). */
transcript_offset?: number;
/** For phase-marker entries only. */
marker?: PhaseMarker;
}
export interface FrictionLogInput {
severity?: FrictionSeverity;
phase: string;
message: string;
hint?: string;
runId?: string;
kind?: FrictionKind;
source?: FrictionSource;
agent?: string;
transcriptOffset?: number;
marker?: PhaseMarker;
/** When the writer is called from the harness wrapping a child error. */
errorClass?: string;
errorCode?: string;
docsUrl?: string;
}
// ---------------------------------------------------------------------------
// Path resolution
// ---------------------------------------------------------------------------
/** Resolve the directory all friction JSONL files live under. */
export function frictionDir(): string {
return gbrainPath('friction');
}
/** Resolve the JSONL file path for a given run-id. */
export function frictionFile(runId: string): string {
return join(frictionDir(), `${sanitizeRunId(runId)}.jsonl`);
}
/** Resolve the active run-id, falling back to 'standalone' (D19). */
export function activeRunId(): string {
const env = process.env.GBRAIN_FRICTION_RUN_ID?.trim();
return env && env.length > 0 ? env : 'standalone';
}
/** Sanitize: only [a-zA-Z0-9._-]; reject anything else to keep filenames sane. */
function sanitizeRunId(runId: string): string {
if (!/^[a-zA-Z0-9._-]+$/.test(runId)) {
throw new Error(`invalid run-id ${JSON.stringify(runId)} (allowed: [a-zA-Z0-9._-])`);
}
return runId;
}
// ---------------------------------------------------------------------------
// Writer
// ---------------------------------------------------------------------------
/** Maximum message length; truncated to keep each line under PIPE_BUF for atomic appends. */
const MAX_MESSAGE_CHARS = 3500;
/** Append one friction entry to the run's JSONL. */
export function logFriction(input: FrictionLogInput): void {
const runId = input.runId ?? activeRunId();
const dir = frictionDir();
if (!existsSync(dir)) mkdirSync(dir, { recursive: true });
const message = truncate(input.message, MAX_MESSAGE_CHARS);
const entry: FrictionEntry = {
schema_version: '1',
ts: new Date().toISOString(),
run_id: runId,
phase: input.phase,
kind: input.kind ?? 'friction',
message,
source: input.source ?? 'claw',
cwd: process.cwd(),
gbrain_version: VERSION,
};
if (input.severity) entry.severity = input.severity;
if (input.hint) entry.hint = input.hint;
if (input.errorClass) entry.class = input.errorClass;
if (input.errorCode) entry.code = input.errorCode;
if (input.docsUrl) entry.docs_url = input.docsUrl;
if (input.agent) entry.agent = input.agent;
if (input.transcriptOffset !== undefined) entry.transcript_offset = input.transcriptOffset;
if (input.marker) entry.marker = input.marker;
const line = JSON.stringify(entry) + '\n';
appendFileSync(frictionFile(runId), line, 'utf-8');
}
function truncate(s: string, max: number): string {
if (s.length <= max) return s;
return s.slice(0, max - 14) + '…[truncated]';
}
// ---------------------------------------------------------------------------
// Reader
// ---------------------------------------------------------------------------
export interface ReadResult {
entries: FrictionEntry[];
/** Count of malformed JSONL lines that were skipped. */
malformed: number;
}
/** Read all entries from a run's JSONL, skipping malformed lines. */
export function readFriction(runId: string): ReadResult {
const path = frictionFile(runId);
if (!existsSync(path)) {
throw new Error(`run-id "${runId}" not found at ${path}`);
}
const raw = readFileSync(path, 'utf-8');
const entries: FrictionEntry[] = [];
let malformed = 0;
for (const line of raw.split('\n')) {
if (!line.trim()) continue;
try {
const parsed = JSON.parse(line);
// Light shape check: must have ts + kind + phase + message
if (typeof parsed.ts === 'string' && typeof parsed.kind === 'string' && typeof parsed.phase === 'string' && typeof parsed.message === 'string') {
entries.push(parsed as FrictionEntry);
} else {
malformed++;
}
} catch {
malformed++;
}
}
return { entries, malformed };
}
/** List run-ids with summary counts. Returns most-recent-first. */
export interface RunSummary {
runId: string;
path: string;
mtime: Date;
counts: { friction: number; delight: number; interrupted: boolean; bySeverity: Record<string, number> };
}
export function listRuns(): RunSummary[] {
const dir = frictionDir();
if (!existsSync(dir)) return [];
const out: RunSummary[] = [];
for (const file of readdirSync(dir)) {
if (!file.endsWith('.jsonl')) continue;
const runId = file.slice(0, -'.jsonl'.length);
const path = join(dir, file);
const stat = statSync(path);
let read: ReadResult;
try {
read = readFriction(runId);
} catch {
continue;
}
const counts = { friction: 0, delight: 0, interrupted: false, bySeverity: {} as Record<string, number> };
for (const e of read.entries) {
if (e.kind === 'friction') counts.friction++;
if (e.kind === 'delight') counts.delight++;
if (e.kind === 'interrupted') counts.interrupted = true;
if (e.severity) counts.bySeverity[e.severity] = (counts.bySeverity[e.severity] ?? 0) + 1;
}
out.push({ runId, path, mtime: stat.mtime, counts });
}
out.sort((a, b) => b.mtime.getTime() - a.mtime.getTime());
return out;
}
// ---------------------------------------------------------------------------
// Renderer
// ---------------------------------------------------------------------------
export interface RenderOpts {
format?: 'md' | 'json';
redact?: boolean;
/** When true, transcript_offset values are resolved against this transcript file. */
transcriptPath?: string;
}
/** Render entries grouped by severity then phase. Returns the rendered string. */
export function renderReport(runId: string, opts: RenderOpts = {}): string {
const { entries, malformed } = readFriction(runId);
const format = opts.format ?? 'md';
const redact = opts.redact ?? (format === 'md');
const transformed = entries.map(e => redact ? redactEntry(e) : e);
if (format === 'json') {
return JSON.stringify({ run_id: runId, malformed, entries: transformed }, null, 2);
}
// Markdown grouping: severity (blocker > error > confused > nit > none) → phase
const sevOrder: (FrictionSeverity | 'none')[] = ['blocker', 'error', 'confused', 'nit', 'none'];
const bySev = new Map<string, FrictionEntry[]>();
for (const e of transformed) {
if (e.kind !== 'friction' && e.kind !== 'delight') continue;
const k = e.severity ?? 'none';
if (!bySev.has(k)) bySev.set(k, []);
bySev.get(k)!.push(e);
}
const lines: string[] = [];
lines.push(`# Friction report — \`${runId}\``);
lines.push('');
const totalFriction = entries.filter(e => e.kind === 'friction').length;
const totalDelight = entries.filter(e => e.kind === 'delight').length;
lines.push(`**${totalFriction} friction · ${totalDelight} delight**${malformed > 0 ? ` · ${malformed} malformed line(s) skipped` : ''}`);
lines.push('');
if (entries.some(e => e.kind === 'interrupted')) {
lines.push('> ⚠ **Run was interrupted.** Some phases may not have completed.');
lines.push('');
}
for (const sev of sevOrder) {
const bucket = bySev.get(sev);
if (!bucket || bucket.length === 0) continue;
lines.push(`## ${sev === 'none' ? '(no severity)' : sev}`);
lines.push('');
// Group by phase within severity
const byPhase = new Map<string, FrictionEntry[]>();
for (const e of bucket) {
if (!byPhase.has(e.phase)) byPhase.set(e.phase, []);
byPhase.get(e.phase)!.push(e);
}
for (const [phase, phaseEntries] of byPhase) {
lines.push(`### \`${phase}\``);
lines.push('');
for (const e of phaseEntries) {
lines.push(`- ${e.kind === 'delight' ? '✨' : '·'} ${e.message}`);
if (e.hint) lines.push(` - hint: ${e.hint}`);
if (e.code) lines.push(` - code: \`${e.code}\``);
if (e.docs_url) lines.push(` - docs: ${e.docs_url}`);
if (opts.transcriptPath && e.transcript_offset !== undefined) {
const snippet = readTranscriptAt(opts.transcriptPath, e.transcript_offset);
if (snippet) lines.push(` - transcript: \`${snippet}\``);
}
}
lines.push('');
}
}
return lines.join('\n');
}
/** Render a friction + delight summary as two columns. */
export function renderSummary(runId: string, opts: { format?: 'md' | 'json' } = {}): string {
const { entries } = readFriction(runId);
const friction = entries.filter(e => e.kind === 'friction');
const delight = entries.filter(e => e.kind === 'delight');
if (opts.format === 'json') {
return JSON.stringify({ run_id: runId, friction, delight }, null, 2);
}
const lines: string[] = [];
lines.push(`# ${runId}`);
lines.push('');
const max = Math.max(friction.length, delight.length);
lines.push(`| friction (${friction.length}) | delight (${delight.length}) |`);
lines.push('|---|---|');
for (let i = 0; i < max; i++) {
const l = friction[i] ? friction[i].message.replace(/\|/g, '\\|') : '';
const r = delight[i] ? delight[i].message.replace(/\|/g, '\\|') : '';
lines.push(`| ${l} | ${r} |`);
}
return lines.join('\n');
}
// ---------------------------------------------------------------------------
// Redaction
// ---------------------------------------------------------------------------
/** Replace homedir/cwd segments in user-visible string fields with placeholders. */
export function redactEntry(entry: FrictionEntry): FrictionEntry {
const home = homedir();
const cwd = entry.cwd;
const transform = (s: string | undefined): string | undefined => {
if (!s) return s;
let out = s;
if (cwd && cwd.length > 1) out = out.split(cwd).join('<CWD>');
if (home && home.length > 1) out = out.split(home).join('<HOME>');
return out;
};
return {
...entry,
message: transform(entry.message) ?? entry.message,
hint: transform(entry.hint),
cwd: '<CWD>',
};
}
// ---------------------------------------------------------------------------
// Transcript snippet resolution (for --transcripts)
// ---------------------------------------------------------------------------
function readTranscriptAt(path: string, offset: number): string | null {
try {
if (!existsSync(path)) return null;
const raw = readFileSync(path, 'utf-8');
if (offset < 0 || offset >= raw.length) return null;
// Find the line that contains this offset. Transcript is JSONL.
const lineStart = raw.lastIndexOf('\n', offset) + 1;
const lineEnd = raw.indexOf('\n', offset);
const line = raw.slice(lineStart, lineEnd === -1 ? undefined : lineEnd);
try {
const parsed = JSON.parse(line);
if (parsed && typeof parsed.bytes_b64 === 'string') {
const text = Buffer.from(parsed.bytes_b64, 'base64').toString('utf-8');
// Truncate snippet for readability
return text.replace(/\n/g, '\\n').slice(0, 200);
}
} catch { /* fall through */ }
return line.slice(0, 200);
} catch {
return null;
}
}
+410
View File
@@ -0,0 +1,410 @@
/**
* Frontmatter inference synthesize YAML frontmatter from filesystem metadata.
*
* ## Why this exists
*
* GBrain's sync and import pipelines work fine without frontmatter gray-matter
* returns the full content as body, and `inferType`/`inferTitle` in markdown.ts
* provide fallbacks. But the inferred metadata is minimal:
*
* - `type` defaults to 'concept' for most paths
* - `title` is the slugified filename ("2010 04 13 Apr 13 Founders Mtg")
* - No `date` field, no `source` metadata, no folder-aware tagging
*
* This module provides **rich inference** directory-aware type mapping, date
* extraction from filenames, title cleanup (strip date prefixes, HTML entities),
* heading extraction from content, and source/folder tagging. It produces a
* complete frontmatter block that can be:
*
* 1. Written back to the file on disk (via `gbrain frontmatter generate --fix`)
* 2. Used at import time without modifying the file (DB-only inference)
* 3. Shown as a dry-run preview (via `gbrain frontmatter generate --dry-run`)
*
* ## Design principles
*
* - **Never overwrite existing frontmatter.** If a file already has `---`, skip it.
* - **Infer from filesystem first, content second.** Directory path type, filename date + title,
* first `#` heading title fallback, content entity hints.
* - **Deterministic.** Same file always produces the same frontmatter. No LLM calls, no network.
* - **Extensible via rules.** The `DIRECTORY_RULES` table maps path patterns to type + source + tags.
* Adding a new directory convention = adding one rule.
* - **Safe.** `.bak` files on write, `--dry-run` by default in CLI, idempotent.
*
* ## How it fits in the pipeline
*
* ```
* Sync/Import
* file has frontmatter? normal import (existing path)
* file has NO frontmatter?
* inferFrontmatter(filePath, content) synthesize frontmatter
* prepend to content import as usual
* optionally write back to disk (--write-back flag)
* ```
*
* The inference runs BEFORE `parseMarkdown`, so the downstream pipeline sees
* well-formed frontmatter and all the existing validation/chunking/embedding
* logic works unchanged.
*
* ## Directory rules table
*
* Each rule matches a path pattern (case-insensitive prefix) and provides:
* - `type`: page type for the brain schema
* - `source`: optional source tag (e.g., "apple-notes", "therapy")
* - `tags`: optional additional tags
* - `datePattern`: where to look for dates 'filename' (YYYY-MM-DD prefix),
* 'dirname' (parent dir name), or 'none'
* - `titleStrategy`: how to extract title 'filename' (strip date prefix),
* 'heading' (first # in content), 'filename-full' (no date strip)
*/
import { basename, dirname, relative } from 'path';
// ─── Types ───────────────────────────────────────────────────────────
export interface InferredFrontmatter {
title: string;
type: string;
date?: string;
source?: string;
tags?: string[];
/** True if the file already has frontmatter (inference skipped). */
skipped?: boolean;
/** The rule that matched, for debugging. */
matchedRule?: string;
}
export interface DirectoryRule {
/** Case-insensitive path prefix to match (e.g., 'apple notes/'). */
pathPrefix: string;
/** Page type to assign. */
type: string;
/** Optional source tag. */
source?: string;
/** Optional tags to add. */
tags?: string[];
/** Where to look for dates. Default: 'filename'. */
datePattern?: 'filename' | 'dirname' | 'none';
/** How to extract title. Default: 'filename'. */
titleStrategy?: 'filename' | 'heading' | 'filename-full';
}
// ─── Directory Rules ─────────────────────────────────────────────────
// Ordered from most specific to least specific. First match wins.
// Add new directory conventions here.
export const DIRECTORY_RULES: DirectoryRule[] = [
// Apple Notes — bulk import from Apple Notes app. Filenames are
// "YYYY-MM-DD Title.md" with HTML-styled content.
{
pathPrefix: 'apple notes/youtube shows/',
type: 'apple-note',
source: 'apple-notes',
tags: ['youtube', 'shows'],
datePattern: 'filename',
titleStrategy: 'filename',
},
{
pathPrefix: 'apple notes/yc/',
type: 'apple-note',
source: 'apple-notes',
tags: ['yc'],
datePattern: 'filename',
titleStrategy: 'filename',
},
{
pathPrefix: 'apple notes/archived/',
type: 'apple-note',
source: 'apple-notes',
tags: ['archived'],
datePattern: 'filename',
titleStrategy: 'filename',
},
{
pathPrefix: 'apple notes/politics/',
type: 'apple-note',
source: 'apple-notes',
tags: ['politics'],
datePattern: 'filename',
titleStrategy: 'filename',
},
{
pathPrefix: 'apple notes/pitch notes/',
type: 'apple-note',
source: 'apple-notes',
tags: ['pitch-notes'],
datePattern: 'filename',
titleStrategy: 'filename',
},
{
pathPrefix: 'apple notes/gstack/',
type: 'apple-note',
source: 'apple-notes',
tags: ['gstack'],
datePattern: 'filename',
titleStrategy: 'filename',
},
{
pathPrefix: 'apple notes/photo-cameras/',
type: 'apple-note',
source: 'apple-notes',
tags: ['photography'],
datePattern: 'filename',
titleStrategy: 'filename',
},
{
pathPrefix: 'apple notes/jan bowman notes/',
type: 'apple-note',
source: 'apple-notes',
tags: ['therapy', 'jan-bowman'],
datePattern: 'filename',
titleStrategy: 'filename',
},
// Catch-all for Apple Notes not in a subfolder
{
pathPrefix: 'apple notes/',
type: 'apple-note',
source: 'apple-notes',
datePattern: 'filename',
titleStrategy: 'filename',
},
// Calendar diarization files
{
pathPrefix: 'daily/calendar/',
type: 'calendar-index',
source: 'calendar',
datePattern: 'filename',
titleStrategy: 'filename',
},
// Personal sections
{
pathPrefix: 'personal/therapy/',
type: 'therapy-session',
source: 'therapy',
datePattern: 'filename',
titleStrategy: 'filename',
},
{
pathPrefix: 'personal/reflections/',
type: 'reflection',
source: 'personal',
datePattern: 'filename',
titleStrategy: 'heading',
},
{
pathPrefix: 'personal/',
type: 'personal',
source: 'personal',
datePattern: 'none',
titleStrategy: 'heading',
},
// Writing
{
pathPrefix: 'writing/essays/',
type: 'essay',
source: 'writing',
datePattern: 'filename',
titleStrategy: 'heading',
},
{
pathPrefix: 'writing/ideas/',
type: 'idea',
source: 'writing',
datePattern: 'filename',
titleStrategy: 'heading',
},
{
pathPrefix: 'writing/',
type: 'writing',
source: 'writing',
datePattern: 'filename',
titleStrategy: 'heading',
},
// Entity directories — these should already have frontmatter in most cases,
// but the 55 people pages etc. that don't get handled here.
{ pathPrefix: 'people/', type: 'person', titleStrategy: 'heading' },
{ pathPrefix: 'companies/', type: 'company', titleStrategy: 'heading' },
{ pathPrefix: 'projects/', type: 'project', titleStrategy: 'heading' },
{ pathPrefix: 'civic/', type: 'civic', titleStrategy: 'heading' },
{ pathPrefix: 'events/', type: 'event', titleStrategy: 'heading', datePattern: 'filename' },
{ pathPrefix: 'meetings/', type: 'meeting', titleStrategy: 'heading', datePattern: 'filename' },
{ pathPrefix: 'media/', type: 'media', titleStrategy: 'heading' },
// Catch-all for any remaining files
{ pathPrefix: '', type: 'note', titleStrategy: 'heading' },
];
// ─── Date extraction ─────────────────────────────────────────────────
/** Extract YYYY-MM-DD date from a filename like "2010-04-13 Apr 13 founders mtg.md" */
export function extractDateFromFilename(filename: string): string | null {
// Pattern 1: YYYY-MM-DD prefix (with - or space separator after)
const m1 = filename.match(/^(\d{4}-\d{2}-\d{2})[\s_-]/);
if (m1) return m1[1];
// Pattern 2: YYYY-MM-DD anywhere in filename
const m2 = filename.match(/(\d{4}-\d{2}-\d{2})/);
if (m2) return m2[1];
// Pattern 3: "YYYY MM DD" with spaces
const m3 = filename.match(/^(\d{4})\s+(\d{2})\s+(\d{2})\s/);
if (m3) return `${m3[1]}-${m3[2]}-${m3[3]}`;
return null;
}
// ─── Title extraction ────────────────────────────────────────────────
/** Extract title from filename, stripping date prefix and extension. */
export function extractTitleFromFilename(filename: string): string {
// Remove .md extension
let title = filename.replace(/\.md$/i, '');
// Strip YYYY-MM-DD prefix (with separator)
title = title.replace(/^\d{4}-\d{2}-\d{2}[\s_-]+/, '');
// Strip YYYY MM DD prefix (space-separated)
title = title.replace(/^\d{4}\s+\d{2}\s+\d{2}\s+/, '');
// Clean up: title case, replace dashes/underscores with spaces
title = title
.replace(/[-_]/g, ' ')
.replace(/\s+/g, ' ')
.trim();
// Don't title-case if it already has mixed case (e.g., "YC presidency")
if (title === title.toLowerCase() || title === title.toUpperCase()) {
title = title.replace(/\b\w/g, c => c.toUpperCase());
}
return title || 'Untitled';
}
/** Extract title from first heading (# ...) in content. */
export function extractTitleFromHeading(content: string): string | null {
const lines = content.split('\n');
for (const line of lines.slice(0, 20)) {
const m = line.match(/^#\s+(.+)/);
if (m) return m[1].trim();
}
return null;
}
// ─── Core inference ──────────────────────────────────────────────────
/**
* Infer frontmatter for a file that has none.
*
* @param relativePath - Path relative to brain root (e.g., "Apple Notes/2010-04-13 Apr 13 founders mtg.md")
* @param content - File content (may be empty)
* @returns Inferred frontmatter fields
*/
export function inferFrontmatter(relativePath: string, content: string): InferredFrontmatter {
// Check if file already has frontmatter
const firstNonEmpty = content.split('\n').find(l => l.trim().length > 0);
if (firstNonEmpty?.trim() === '---') {
return { title: '', type: '', skipped: true };
}
const lowerPath = relativePath.toLowerCase();
const filename = basename(relativePath);
// Find matching rule
let matchedRule: DirectoryRule | undefined;
for (const rule of DIRECTORY_RULES) {
if (lowerPath.startsWith(rule.pathPrefix.toLowerCase())) {
matchedRule = rule;
break;
}
}
// Default rule if none matched
if (!matchedRule) {
matchedRule = { pathPrefix: '', type: 'note', titleStrategy: 'heading' };
}
// Extract date
let date: string | undefined;
const datePattern = matchedRule.datePattern ?? 'filename';
if (datePattern === 'filename') {
date = extractDateFromFilename(filename) ?? undefined;
}
// Extract title
let title: string;
const titleStrategy = matchedRule.titleStrategy ?? 'filename';
if (titleStrategy === 'heading') {
title = extractTitleFromHeading(content) ?? extractTitleFromFilename(filename);
} else if (titleStrategy === 'filename-full') {
title = filename.replace(/\.md$/i, '').replace(/[-_]/g, ' ').trim();
} else {
title = extractTitleFromFilename(filename);
}
// Build tags from rule + subfolder
const tags = [...(matchedRule.tags ?? [])];
// Add subfolder as tag for Apple Notes (e.g., "YC", "Politics")
if (matchedRule.source === 'apple-notes' && matchedRule.pathPrefix === 'apple notes/') {
const parts = relativePath.split('/');
if (parts.length > 2) {
const subfolder = parts[1].toLowerCase().replace(/\s+/g, '-');
if (!tags.includes(subfolder)) tags.push(subfolder);
}
}
return {
title,
type: matchedRule.type,
date,
source: matchedRule.source,
tags: tags.length > 0 ? tags : undefined,
matchedRule: matchedRule.pathPrefix || '(default)',
};
}
/**
* Generate a YAML frontmatter block from inferred fields.
* Returns the `---\n...\n---\n` string to prepend to content.
*/
export function serializeFrontmatter(fm: InferredFrontmatter): string {
if (fm.skipped) return '';
const lines: string[] = ['---'];
// Title — quote if it contains special YAML chars
const needsQuote = /[:"'#\[\]{}|>&*!?,]/.test(fm.title);
lines.push(`title: ${needsQuote ? JSON.stringify(fm.title) : fm.title}`);
lines.push(`type: ${fm.type}`);
if (fm.date) {
lines.push(`date: "${fm.date}"`);
}
if (fm.source) {
lines.push(`source: ${fm.source}`);
}
if (fm.tags && fm.tags.length > 0) {
lines.push(`tags: [${fm.tags.map(t => JSON.stringify(t)).join(', ')}]`);
}
lines.push('---');
return lines.join('\n') + '\n';
}
/**
* Apply frontmatter inference to file content.
* Returns the content with frontmatter prepended, or the original content if it already has frontmatter.
*/
export function applyInference(relativePath: string, content: string): { content: string; inferred: InferredFrontmatter } {
const inferred = inferFrontmatter(relativePath, content);
if (inferred.skipped) {
return { content, inferred };
}
const fm = serializeFrontmatter(inferred);
return { content: fm + '\n' + content, inferred };
}
+16 -2
View File
@@ -339,7 +339,7 @@ export async function importFromFile(
engine: BrainEngine,
filePath: string,
relativePath: string,
opts: { noEmbed?: boolean } = {},
opts: { noEmbed?: boolean; inferFrontmatter?: boolean } = {},
): Promise<ImportResult> {
// Defense-in-depth: reject symlinks before reading content.
const lstat = lstatSync(filePath);
@@ -352,13 +352,27 @@ export async function importFromFile(
return { slug: relativePath, status: 'skipped', chunks: 0, error: `File too large (${stat.size} bytes)` };
}
const content = readFileSync(filePath, 'utf-8');
let content = readFileSync(filePath, 'utf-8');
// Route code files through the code import path
if (isCodeFilePath(relativePath)) {
return importCodeFile(engine, relativePath, content, opts);
}
// v0.22.8 — Frontmatter inference: if the file has no frontmatter and
// inference is enabled, synthesize it from the filesystem path + content.
// This turns bare markdown files into fully-typed, dated, tagged pages
// without requiring the user to manually add YAML headers.
// The inference is applied to the in-memory content only; the file on disk
// is not modified. Use `gbrain frontmatter generate --fix` to write back.
if (opts.inferFrontmatter !== false) {
const { applyInference } = await import('./frontmatter-inference.ts');
const { content: inferred, inferred: meta } = applyInference(relativePath, content);
if (!meta.skipped) {
content = inferred;
}
}
const parsed = parseMarkdown(content, relativePath);
// Enforce path-authoritative slug. parseMarkdown prefers frontmatter.slug over
+28
View File
@@ -1073,6 +1073,34 @@ export const MIGRATIONS: Migration[] = [
},
sql: '',
},
{
version: 30,
name: 'dream_verdicts_table',
// v0.23 synthesize phase: cache for "is this transcript worth processing?"
// verdict from the cheap Haiku judge. Distinct from raw_data (page-scoped);
// transcripts aren't pages. Keyed by (file_path, content_hash) so edited
// transcripts re-judge automatically. Backfill re-runs hit cache instead
// of paying for Haiku 100x.
sql: `
CREATE TABLE IF NOT EXISTS dream_verdicts (
file_path TEXT NOT NULL,
content_hash TEXT NOT NULL,
worth_processing BOOLEAN NOT NULL,
reasons JSONB,
judged_at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (file_path, content_hash)
);
DO $$
DECLARE
has_bypass BOOLEAN;
BEGIN
SELECT rolbypassrls INTO has_bypass FROM pg_roles WHERE rolname = current_user;
IF has_bypass THEN
ALTER TABLE dream_verdicts ENABLE ROW LEVEL SECURITY;
END IF;
END $$;
`,
},
];
export const LATEST_VERSION = MIGRATIONS.length > 0
+2 -2
View File
@@ -18,7 +18,7 @@
import * as fs from 'node:fs';
import * as path from 'node:path';
import * as os from 'node:os';
import { gbrainPath } from '../config.ts';
export interface BackpressureAuditEvent {
ts: string;
@@ -54,7 +54,7 @@ export function computeAuditFilename(now: Date = new Date()): string {
export function resolveAuditDir(): string {
const override = process.env.GBRAIN_AUDIT_DIR;
if (override && override.trim().length > 0) return override;
return path.join(os.homedir(), '.gbrain', 'audit');
return gbrainPath('audit');
}
export function logBackpressureCoalesce(event: Omit<BackpressureAuditEvent, 'ts' | 'decision'>): void {
+2 -2
View File
@@ -15,7 +15,7 @@
import * as fs from 'node:fs';
import * as path from 'node:path';
import * as os from 'node:os';
import { gbrainPath } from '../../config.ts';
export interface ShellAuditEvent {
ts: string;
@@ -53,7 +53,7 @@ export function computeAuditFilename(now: Date = new Date()): string {
export function resolveAuditDir(): string {
const override = process.env.GBRAIN_AUDIT_DIR;
if (override && override.trim().length > 0) return override;
return path.join(os.homedir(), '.gbrain', 'audit');
return gbrainPath('audit');
}
export function logShellSubmission(event: Omit<ShellAuditEvent, 'ts'>): void {
+4
View File
@@ -149,10 +149,14 @@ export function makeSubagentHandler(deps: SubagentDeps) {
const systemPrompt = data.system ?? DEFAULT_SYSTEM;
// Build the tool registry bound to THIS job as the owning subagent.
// allowed_slug_prefixes (v0.23) flows through buildBrainTools → the
// put_page schema description AND the OperationContext, so the model's
// tool schema and the server-side check stay in sync.
const registry = deps.toolRegistry ?? buildBrainTools({
subagentId: ctx.id,
engine,
config,
allowedSlugPrefixes: data.allowed_slug_prefixes,
});
const toolDefs = data.allowed_tools && data.allowed_tools.length > 0
? filterAllowedTools(registry, data.allowed_tools)
+11 -2
View File
@@ -225,8 +225,12 @@ export class MinionSupervisor {
process.on('SIGTERM', this.sigtermListener);
process.on('SIGINT', this.sigintListener);
// 4. Health monitoring.
this.healthTimer = setInterval(() => { void this.healthCheck(); }, this.opts.healthInterval);
// 4. Health monitoring. Skip when healthInterval=0 — that's the explicit
// "disable" contract documented on `--health-interval 0`. setInterval(0)
// would be a tight DB-hammering loop, not the no-op users expect.
if (this.opts.healthInterval > 0) {
this.healthTimer = setInterval(() => { void this.healthCheck(); }, this.opts.healthInterval);
}
// 5. Announce start.
this.emit('started', {
@@ -427,6 +431,11 @@ export class MinionSupervisor {
} else {
delete env.GBRAIN_ALLOW_SHELL_JOBS;
}
// Signal to the child worker that it's running under a supervisor.
// The worker's self-health-check (DB probes, stall detection) is
// redundant when the supervisor already provides these — setting
// this env var causes the worker to skip its own health timer.
env.GBRAIN_SUPERVISED = '1';
this.lastStartTime = Date.now();
+42 -11
View File
@@ -91,19 +91,37 @@ function paramsToInputSchema(op: Operation): Record<string, unknown> {
/**
* For put_page specifically, the tool schema shown to the model constrains
* `slug` to `wiki/agents/<subagentId>/...`. The server-side check in
* operations.ts is the authoritative gate; this just helps the model write
* correct slugs on the first try.
* `slug`. Two modes:
*
* - Default (legacy): slug MUST start with `wiki/agents/<subagentId>/`,
* enforced by both the JSONSchema `pattern` and the server-side check.
* - Trusted-workspace (v0.23 dream cycle): when `allowedSlugPrefixes` is
* set, the model is told the allowed prefixes in plain English (no
* regex pattern the prefix list is authoritative server-side, and
* JSONSchema can't express "matches any of these globs" cleanly).
*/
function namespacedPutPageSchema(op: Operation, subagentId: number): Record<string, unknown> {
function namespacedPutPageSchema(
op: Operation,
subagentId: number,
allowedSlugPrefixes?: readonly string[],
): Record<string, unknown> {
const base = paramsToInputSchema(op);
const props = (base.properties as Record<string, Record<string, unknown>>) ?? {};
if (props.slug) {
props.slug = {
...props.slug,
description: `Page slug. MUST start with "wiki/agents/${subagentId}/" (agents can only write under their own namespace).`,
pattern: `^wiki/agents/${subagentId}/.+`,
};
if (allowedSlugPrefixes && allowedSlugPrefixes.length > 0) {
props.slug = {
...props.slug,
description:
`Page slug. MUST match one of these prefix globs: ${allowedSlugPrefixes.join(', ')}. ` +
`Slugs use lowercase alphanumeric segments separated by '/'. No leading slash, no '.md' extension, no underscores.`,
};
} else {
props.slug = {
...props.slug,
description: `Page slug. MUST start with "wiki/agents/${subagentId}/" (agents can only write under their own namespace).`,
pattern: `^wiki/agents/${subagentId}/.+`,
};
}
}
return { ...base, properties: props };
}
@@ -115,6 +133,14 @@ export interface BuildBrainToolsOpts {
config: GBrainConfig;
/** Optional filter: only include names in this set. */
allowedNames?: ReadonlySet<string>;
/**
* Trusted-workspace allow-list (v0.23). When set, put_page is bounded
* to slugs matching these prefix globs instead of the legacy
* `wiki/agents/<id>/...` namespace. Trust comes from PROTECTED_JOB_NAMES
* (MCP can't submit subagent jobs) this flows from
* SubagentHandlerData.allowed_slug_prefixes via the handler.
*/
allowedSlugPrefixes?: readonly string[];
}
interface OpContextDeps {
@@ -123,6 +149,7 @@ interface OpContextDeps {
subagentId: number;
jobId: number;
signal?: AbortSignal;
allowedSlugPrefixes?: readonly string[];
}
function buildOpContext(deps: OpContextDeps): OperationContext {
@@ -135,10 +162,13 @@ function buildOpContext(deps: OpContextDeps): OperationContext {
error: (msg: string) => process.stderr.write(`[subagent-tool:${deps.jobId}] ERROR: ${msg}\n`),
},
dryRun: false,
remote: true, // match MCP trust boundary
remote: true, // match MCP trust boundary for auto-link skip
jobId: deps.jobId,
subagentId: deps.subagentId,
viaSubagent: true, // FAIL-CLOSED: put_page etc. enforce namespace
allowedSlugPrefixes: deps.allowedSlugPrefixes
? [...deps.allowedSlugPrefixes]
: undefined,
};
}
@@ -157,7 +187,7 @@ export function buildBrainTools(opts: BuildBrainToolsOpts): ToolDef[] {
return picked.map<ToolDef>(op => {
const schema = op.name === 'put_page'
? namespacedPutPageSchema(op, opts.subagentId)
? namespacedPutPageSchema(op, opts.subagentId, opts.allowedSlugPrefixes)
: paramsToInputSchema(op);
const toolName = sanitizeToolName(op.name);
@@ -179,6 +209,7 @@ export function buildBrainTools(opts: BuildBrainToolsOpts): ToolDef[] {
subagentId: opts.subagentId,
jobId: ctx.jobId,
signal: ctx.signal,
allowedSlugPrefixes: opts.allowedSlugPrefixes,
});
const params = (input && typeof input === 'object') ? input as Record<string, unknown> : {};
return op.handler(opCtx, params);
+32
View File
@@ -170,6 +170,25 @@ export interface MinionWorkerOpts {
* case where all concurrency slots are wedged with zero job completions
* so the per-job check never fires. */
rssCheckInterval?: number;
/** Self-health-check interval in ms. 0 = disabled. Default: 60000 (1 minute).
* Automatically disabled when running under a supervisor (GBRAIN_SUPERVISED=1).
* Provides DB liveness probes and stall detection for bare `gbrain jobs work`
* deployments managed by external process managers (systemd, Docker, cron). */
healthCheckInterval?: number;
/** Stall detection: ms of continuous idle (waiting>0, inFlight=0, no completions)
* before emitting the first warning. Default: 300000 (5 minutes). */
stallWarnAfterMs?: number;
/** Stall detection: ms of continuous idle before emitting `'unhealthy'` with
* reason='stalled'. Default: 600000 (10 minutes). Must be > stallWarnAfterMs. */
stallExitAfterMs?: number;
/** DB liveness probe: number of consecutive failed `SELECT 1` probes before
* emitting `'unhealthy'` with reason='db_dead'. Default: 3. */
dbFailExitAfter?: number;
/** Per-probe wall-clock timeout in ms. A `SELECT 1` that hangs longer than
* this counts as a failure (fed into dbFailExitAfter). Without this, a
* hung probe would wedge the recursive setTimeout chain forever and
* silently disable the health monitor. Default: 10000 (10 seconds). */
dbProbeTimeoutMs?: number;
}
// --- Job Context (passed to handlers) ---
@@ -402,6 +421,19 @@ export interface SubagentHandlerData {
system?: string;
/** Template variables for subagent_def. Arbitrary JSON-serializable. */
input_vars?: Record<string, unknown>;
/**
* Trusted-workspace allow-list for put_page (v0.23 dream cycle).
*
* When set, the subagent's put_page calls are bounded to slugs matching
* any of these prefix globs (e.g. ["wiki/personal/reflections/*",
* "wiki/originals/*"]). When unset/empty, the legacy
* `wiki/agents/<subagentId>/...` namespace check applies.
*
* Trust comes from PROTECTED_JOB_NAMES gating subagent submission MCP
* cannot reach this field. Only cycle.ts (synthesize/patterns phases)
* and direct CLI submitters set it.
*/
allowed_slug_prefixes?: string[];
}
/**
+209 -1
View File
@@ -20,8 +20,15 @@ import { UnrecoverableError } from './types.ts';
import { MinionQueue } from './queue.ts';
import { calculateBackoff } from './backoff.ts';
import { randomUUID } from 'crypto';
import { EventEmitter } from 'events';
import { evaluateQuietHours, type QuietHoursConfig } from './quiet-hours.ts';
/** Reason payload emitted with `'unhealthy'` when self-health-check trips.
* CLI layer (jobs.ts:work) subscribes and decides whether to call process.exit. */
export type UnhealthyReason =
| { reason: 'db_dead'; consecutiveFailures: number; message: string }
| { reason: 'stalled'; waitingCount: number; idleMinutes: number };
/**
* Read the quiet_hours JSONB column off a MinionJob, if present. The
* column was added in schema migration v12; older rows + versions of
@@ -42,7 +49,13 @@ interface InFlightJob {
promise: Promise<void>;
}
export class MinionWorker {
/** Type-safe `on('unhealthy', ...)` for callers. */
export interface MinionWorker {
on(event: 'unhealthy', listener: (info: UnhealthyReason) => void): this;
emit(event: 'unhealthy', info: UnhealthyReason): boolean;
}
export class MinionWorker extends EventEmitter {
private queue: MinionQueue;
private handlers = new Map<string, MinionHandler>();
private running = false;
@@ -67,6 +80,7 @@ export class MinionWorker {
private engine: BrainEngine,
opts?: MinionWorkerOpts & MinionQueueOpts,
) {
super();
this.queue = new MinionQueue(engine, {
maxSpawnDepth: opts?.maxSpawnDepth,
maxAttachmentBytes: opts?.maxAttachmentBytes,
@@ -81,7 +95,25 @@ export class MinionWorker {
maxRssMb: opts?.maxRssMb ?? 0,
getRss: opts?.getRss ?? (() => process.memoryUsage().rss),
rssCheckInterval: opts?.rssCheckInterval ?? 60000,
healthCheckInterval: opts?.healthCheckInterval ?? 60000,
stallWarnAfterMs: opts?.stallWarnAfterMs ?? 5 * 60_000,
stallExitAfterMs: opts?.stallExitAfterMs ?? 10 * 60_000,
dbFailExitAfter: opts?.dbFailExitAfter ?? 3,
dbProbeTimeoutMs: opts?.dbProbeTimeoutMs ?? 10_000,
};
// Stall thresholds contract: exit MUST be strictly greater than warn.
// If exit <= warn, the warn-then-exit semantics break: a single tick at
// idle > warn would set stallWarningSince and the subsequent tick at
// idle > exit could fire immediately without giving operators visibility.
// Reject misconfigurations at construction time so the failure mode is
// a loud throw on startup rather than a quiet contract violation.
if (this.opts.stallExitAfterMs <= this.opts.stallWarnAfterMs) {
throw new Error(
`MinionWorkerOpts: stallExitAfterMs (${this.opts.stallExitAfterMs}) must be > ` +
`stallWarnAfterMs (${this.opts.stallWarnAfterMs}). ` +
`The contract is "warn first, exit later" — they cannot fire on the same tick.`,
);
}
}
/** Register a handler for a job type. */
@@ -94,6 +126,28 @@ export class MinionWorker {
return Array.from(this.handlers.keys());
}
/** Emit 'unhealthy' with a no-listener fallback. The default contract is
* fail-stop: pre-EventEmitter-refactor behavior was process.exit(1) inside
* the timer; the refactor moved that responsibility to the CLI subscriber.
* But direct API consumers without a listener would see emit() become a
* no-op AND `healthExited=true` permanently disabling monitoring a
* silent regression. Solution: if no one subscribed, log and exit
* ourselves so the worker dies and the PM restarts it. Subscribers
* override this default by adding a listener before start(). */
private emitUnhealthy(info: UnhealthyReason): void {
if (this.listenerCount('unhealthy') === 0) {
const detail = info.reason === 'db_dead'
? `DB unreachable (${info.consecutiveFailures} probes): ${info.message}`
: `worker stalled (${info.waitingCount} waiting, ${info.idleMinutes}m idle)`;
console.error(
`[health] FATAL: ${detail}. No 'unhealthy' listener registered; ` +
`defaulting to process.exit(1) for process-manager restart.`,
);
process.exit(1);
}
this.emit('unhealthy', info);
}
/** Start the worker loop. Blocks until stopped. */
async start(): Promise<void> {
if (this.handlers.size === 0) {
@@ -155,6 +209,159 @@ export class MinionWorker {
}, this.opts.rssCheckInterval);
}
// Self-health-check — provides supervisor-grade monitoring for bare workers.
// Disabled when running under a supervisor (GBRAIN_SUPERVISED=1) or when
// healthCheckInterval is 0. Catches two failure modes that leave the process
// alive but non-functional:
// 1. DB connection death (Supabase/PgBouncer drops, network blip)
// 2. Worker stall (event loop alive but not claiming/completing jobs)
//
// On failure, emits an `'unhealthy'` event with a structured reason. The
// CLI layer (`src/commands/jobs.ts:work`) subscribes and decides whether to
// call process.exit. Library code never calls process.exit directly so
// MinionWorker stays embeddable in non-CLI contexts (tests, other hosts).
//
// Timer pattern: recursive setTimeout with a `running` flag, not setInterval.
// setInterval queues callbacks even when the prior is still awaiting; on a
// hung DB probe that piles up overlapping async checks racing on
// `consecutiveDbFailures`. The recursive pattern guarantees one tick at a time.
const isSupervisedChild = process.env.GBRAIN_SUPERVISED === '1';
let healthTimer: ReturnType<typeof setTimeout> | null = null;
if (!isSupervisedChild && this.opts.healthCheckInterval > 0) {
let consecutiveDbFailures = 0;
let lastKnownCompleted = this.jobsCompleted;
let lastCompletionTime = Date.now();
let stallWarningSince: number | null = null;
let healthRunning = false;
let healthExited = false;
// Race executeRaw against a wall-clock deadline. A hung connection
// (network-partitioned PgBouncer, deadlocked backend) would otherwise
// hold the await forever — the recursive setTimeout's next tick is only
// scheduled in `finally`, so a hung probe would silently disable the
// entire health monitor. The timeout treats hangs as failures and feeds
// them into `dbFailExitAfter`.
const probeWithTimeout = async (): Promise<void> => {
const ac = new AbortController();
const timeoutMs = this.opts.dbProbeTimeoutMs;
const timer = setTimeout(() => ac.abort(), timeoutMs);
try {
await Promise.race([
this.engine.executeRaw('SELECT 1'),
new Promise<never>((_, reject) => {
ac.signal.addEventListener('abort', () => {
reject(new Error(`probe timeout after ${timeoutMs}ms`));
});
}),
]);
} finally {
clearTimeout(timer);
}
};
const runHealthCheck = async (): Promise<void> => {
if (healthRunning || !this.running || healthExited) return;
healthRunning = true;
try {
// --- 1. DB liveness probe ---
try {
await probeWithTimeout();
consecutiveDbFailures = 0;
} catch (e) {
consecutiveDbFailures++;
const msg = e instanceof Error ? e.message : String(e);
console.error(
`[health] DB probe failed (${consecutiveDbFailures}/${this.opts.dbFailExitAfter}): ${msg}`,
);
if (consecutiveDbFailures >= this.opts.dbFailExitAfter) {
console.error(
`[health] DB unreachable after ${this.opts.dbFailExitAfter} consecutive probes. ` +
`Emitting 'unhealthy' for process-manager restart.`,
);
healthExited = true;
this.emitUnhealthy({
reason: 'db_dead',
consecutiveFailures: consecutiveDbFailures,
message: msg,
});
}
return; // Skip stall check when DB is flaky
}
// --- 2. Stall detection ---
if (this.jobsCompleted > lastKnownCompleted) {
lastKnownCompleted = this.jobsCompleted;
lastCompletionTime = Date.now();
stallWarningSince = null;
}
const idleMs = Date.now() - lastCompletionTime;
// Only check for stalls when no jobs are in-flight and it's been a while
if (idleMs > this.opts.stallWarnAfterMs && this.inFlight.size === 0) {
try {
// Filter by registered handler names so a worker that doesn't
// claim a particular job-name doesn't false-positive when those
// jobs accumulate in `waiting`. Only counts work THIS worker would
// actually have claimed.
const handlerNames = this.registeredNames;
const rows = handlerNames.length === 0
? [] as { cnt: string }[]
: await this.engine.executeRaw<{ cnt: string }>(
`SELECT count(*)::text AS cnt FROM minion_jobs
WHERE status = 'waiting'
AND queue = $1
AND name = ANY($2::text[])`,
[this.opts.queue, handlerNames],
);
const waiting = parseInt(rows[0]?.cnt ?? '0', 10);
const idleMinutes = Math.round(idleMs / 60_000);
if (waiting > 0) {
// Two thresholds, both measured from `lastCompletionTime` (NOT
// from when the warning fired). With defaults (warn=5min,
// exit=10min), the first warning fires at idle=5min and the
// unhealthy emit fires at idle=10min — matching the contract
// documented in MinionWorkerOpts.
if (!stallWarningSince) {
stallWarningSince = Date.now();
console.warn(
`[health] Possible stall: ${waiting} waiting job(s) for ` +
`registered handlers, 0 in-flight, ${idleMinutes}m since last completion`,
);
} else if (idleMs > this.opts.stallExitAfterMs) {
console.error(
`[health] Worker stalled for ${Math.round(this.opts.stallExitAfterMs / 60_000)}+ ` +
`minutes with ${waiting} waiting job(s). Emitting 'unhealthy' for process-manager restart.`,
);
healthExited = true;
this.emitUnhealthy({
reason: 'stalled',
waitingCount: waiting,
idleMinutes,
});
}
} else {
stallWarningSince = null; // Queue empty (for our handlers) — not stalled, just idle
}
} catch {
// DB query failed — the liveness probe above will catch persistent failures
}
} else {
stallWarningSince = null;
}
} finally {
healthRunning = false;
if (this.running && !healthExited) {
healthTimer = setTimeout(runHealthCheck, this.opts.healthCheckInterval);
}
}
};
// First tick scheduled after one interval so newly-started workers have
// a chance to do real work before the stall clock starts ticking.
healthTimer = setTimeout(runHealthCheck, this.opts.healthCheckInterval);
}
try {
while (this.running) {
// Promote delayed jobs
@@ -201,6 +408,7 @@ export class MinionWorker {
} finally {
clearInterval(stalledTimer);
if (rssTimer) clearInterval(rssTimer);
if (healthTimer) clearTimeout(healthTimer); // recursive setTimeout pattern
process.removeListener('SIGTERM', shutdown);
process.removeListener('SIGINT', shutdown);
+68 -4
View File
@@ -120,6 +120,31 @@ export function validatePageSlug(slug: string): void {
}
}
/**
* Match a slug against a list of allow-list prefix globs.
*
* Glob form: `<prefix>/*` matches any slug starting with `<prefix>/` and
* having at least one more segment (single or multi). Bare `<prefix>` (no
* trailing `/*`) matches that exact slug only. The `*` is intentionally
* permissive depth is unbounded, so `wiki/originals/*` matches both
* `wiki/originals/idea-x` and `wiki/originals/ideas/2026-04-25-idea-y`.
*
* Used by the v0.23 dream-cycle trusted-workspace path. Order doesn't
* matter; the first match wins (returns true on any match).
*/
export function matchesSlugAllowList(slug: string, prefixes: readonly string[]): boolean {
for (const p of prefixes) {
if (p.endsWith('/*')) {
const base = p.slice(0, -2);
if (slug === base) continue;
if (slug.startsWith(base + '/')) return true;
} else if (p === slug) {
return true;
}
}
return false;
}
/**
* Allowlist validator for uploaded file basenames. Rejects control chars, backslashes,
* RTL overrides (\u202E), leading dot (hidden files) and leading dash (CLI flag confusion).
@@ -181,6 +206,22 @@ export interface OperationContext {
jobId?: number;
subagentId?: number;
viaSubagent?: boolean;
/**
* Trusted-workspace allow-list (v0.23 dream cycle). When the cycle's
* synthesize/patterns phases dispatch a subagent, they thread an
* explicit list of slug-prefix globs (e.g. "wiki/personal/reflections/*")
* through this field. put_page enforces it BEFORE the legacy
* `wiki/agents/<id>/...` namespace check.
*
* Trust comes from the SUBMITTER (subagent jobs are gated by
* PROTECTED_JOB_NAMES MCP cannot submit them), not from `remote`.
* Every subagent tool call has `remote=true` for auto-link safety,
* so basing trust on `remote` is incoherent (would always reject).
*
* Empty / unset fall back to the legacy namespace check (existing
* v0.15 behavior; pure addition, no regression).
*/
allowedSlugPrefixes?: string[];
/**
* Resolved global CLI options (--quiet / --progress-json / --progress-interval).
* CLI callers populate this from `getCliOptions()`. MCP / library callers
@@ -264,9 +305,23 @@ const put_page: Operation = {
if (typeof ctx.subagentId !== 'number' || Number.isNaN(ctx.subagentId)) {
throw new OperationError('permission_denied', 'put_page via subagent requires ctx.subagentId');
}
const prefix = `wiki/agents/${ctx.subagentId}/`;
if (!slug.startsWith(prefix) || slug.length === prefix.length) {
throw new OperationError('permission_denied', `put_page via subagent must write under '${prefix}...'`);
const allowList = ctx.allowedSlugPrefixes;
if (allowList && allowList.length > 0) {
// Trusted-workspace path: explicit allow-list bounds writes.
// Set only by cycle.ts (synthesize/patterns) which submits subagent
// jobs under PROTECTED_JOB_NAMES — MCP cannot reach this branch.
if (!matchesSlugAllowList(slug, allowList)) {
throw new OperationError(
'permission_denied',
`put_page slug '${slug}' is not within the trusted-workspace allow-list (${allowList.join(', ')})`
);
}
} else {
// Legacy default: agent-namespace confinement.
const prefix = `wiki/agents/${ctx.subagentId}/`;
if (!slug.startsWith(prefix) || slug.length === prefix.length) {
throw new OperationError('permission_denied', `put_page via subagent must write under '${prefix}...'`);
}
}
}
@@ -295,7 +350,16 @@ const put_page: Operation = {
| { skipped: 'remote' }
| undefined;
let autoTimeline: { created: number } | { error: string } | { skipped: 'remote' } | undefined;
if (ctx.remote === true) {
// Trusted-workspace path (v0.23 dream cycle) re-enables auto-link/timeline
// even though ctx.remote=true, because the allow-list bounds the slug and
// the synthesis prompt is itself the trusted dispatcher. Without this,
// the cycle's `extract` phase would have to recompute every edge, and
// patterns (which runs after extract) would still see the right graph
// but auto_timeline would never fire on synth output.
const trustedWorkspace = ctx.viaSubagent === true
&& Array.isArray(ctx.allowedSlugPrefixes)
&& ctx.allowedSlugPrefixes.length > 0;
if (ctx.remote === true && !trustedWorkspace) {
autoLinks = { skipped: 'remote' };
autoTimeline = { skipped: 'remote' };
} else if (result.parsedPage) {
+6 -5
View File
@@ -18,8 +18,8 @@
*/
import { appendFileSync, existsSync, mkdirSync } from 'fs';
import { homedir } from 'os';
import { dirname, join } from 'path';
import { dirname } from 'path';
import { gbrainPath } from '../config.ts';
import type { BrainEngine } from '../engine.ts';
import {
@@ -30,7 +30,7 @@ import {
} from './validators/index.ts';
import type { ValidationFinding, PageValidator } from './writer.ts';
const LINT_LOG_FILE = join(homedir(), '.gbrain', 'validator-lint.jsonl');
const getLintLogFile = () => gbrainPath('validator-lint.jsonl');
const LINT_CONFIG_KEY = 'writer.lint_on_put_page';
export interface PostWriteLintOpts {
@@ -124,7 +124,8 @@ export async function runPostWriteLint(
function writeLocalLintLog(slug: string, findings: ValidationFinding[]): void {
try {
const dir = dirname(LINT_LOG_FILE);
const lintLogFile = getLintLogFile();
const dir = dirname(lintLogFile);
if (!existsSync(dir)) mkdirSync(dir, { recursive: true });
const line = JSON.stringify({
ts: new Date().toISOString(),
@@ -133,7 +134,7 @@ function writeLocalLintLog(slug: string, findings: ValidationFinding[]): void {
warning_count: findings.filter(f => f.severity === 'warning').length,
findings: findings.slice(0, 20), // cap to prevent runaway log size
}) + '\n';
appendFileSync(LINT_LOG_FILE, line, 'utf-8');
appendFileSync(lintLogFile, line, 'utf-8');
} catch {
// Non-fatal; logging failure shouldn't break the main flow.
}
+130 -1
View File
@@ -2,7 +2,7 @@ import { PGlite } from '@electric-sql/pglite';
import { vector } from '@electric-sql/pglite/vector';
import { pg_trgm } from '@electric-sql/pglite/contrib/pg_trgm';
import type { Transaction } from '@electric-sql/pglite';
import type { BrainEngine, LinkBatchInput, TimelineBatchInput, ReservedConnection } from './engine.ts';
import type { BrainEngine, LinkBatchInput, TimelineBatchInput, ReservedConnection, DreamVerdict, DreamVerdictInput } from './engine.ts';
import { MAX_SEARCH_LIMIT, clampSearchLimit } from './engine.ts';
import { runMigrations } from './migrate.ts';
import { PGLITE_SCHEMA_SQL } from './pglite-schema.ts';
@@ -25,10 +25,86 @@ import { buildSourceFactorCase, buildHardExcludeClause } from './search/sql-rank
type PGLiteDB = PGlite;
// Tier 3 snapshot fast-restore. Reads a tar dump produced by
// `bun run scripts/build-pglite-snapshot.ts`. Snapshot is matched against
// the current MIGRATIONS hash via a sidecar `.version` file; on mismatch we
// silently fall through to a normal initSchema (snapshot is just an
// optimization, never authoritative).
let _snapshotWarnLogged = false;
function tryLoadSnapshot(snapshotPath: string): Blob | null {
try {
// Lazy require so production builds without these imports don't crash.
// eslint-disable-next-line @typescript-eslint/no-require-imports
const fs = require('node:fs') as typeof import('node:fs');
const crypto = require('node:crypto') as typeof import('node:crypto');
const { MIGRATIONS } = require('./migrate.ts') as typeof import('./migrate.ts');
const { PGLITE_SCHEMA_SQL } = require('./pglite-schema.ts') as typeof import('./pglite-schema.ts');
if (!fs.existsSync(snapshotPath)) {
if (!_snapshotWarnLogged) {
// eslint-disable-next-line no-console
console.warn(`[pglite] GBRAIN_PGLITE_SNAPSHOT set but file missing: ${snapshotPath} — using normal init.`);
_snapshotWarnLogged = true;
}
return null;
}
const versionPath = snapshotPath.replace(/\.tar(?:\.gz)?$/, '.version');
if (!fs.existsSync(versionPath)) {
if (!_snapshotWarnLogged) {
// eslint-disable-next-line no-console
console.warn(`[pglite] snapshot version file missing: ${versionPath} — using normal init.`);
_snapshotWarnLogged = true;
}
return null;
}
const expectedHash = computeSnapshotSchemaHash(MIGRATIONS, PGLITE_SCHEMA_SQL, crypto);
const actualHash = fs.readFileSync(versionPath, 'utf8').trim();
if (expectedHash !== actualHash) {
if (!_snapshotWarnLogged) {
// eslint-disable-next-line no-console
console.warn(`[pglite] snapshot stale (schema hash mismatch) — using normal init. Rebuild with: bun run build:pglite-snapshot`);
_snapshotWarnLogged = true;
}
return null;
}
const buf = fs.readFileSync(snapshotPath);
return new Blob([buf]);
} catch {
// Any failure -> fall through to normal init. Never block tests.
return null;
}
}
export function computeSnapshotSchemaHash(
migrations: Array<{ version: number; name: string; sql?: string; sqlFor?: { pglite?: string } }>,
schemaSQL: string,
crypto: typeof import('node:crypto'),
): string {
const hash = crypto.createHash('sha256');
hash.update('schema:');
hash.update(schemaSQL);
hash.update('\nmigrations:\n');
for (const m of migrations) {
hash.update(String(m.version));
hash.update('\t');
hash.update(m.name);
hash.update('\t');
hash.update(m.sql ?? '');
hash.update('\t');
hash.update(m.sqlFor?.pglite ?? '');
hash.update('\n');
}
return hash.digest('hex');
}
export class PGLiteEngine implements BrainEngine {
readonly kind = 'pglite' as const;
private _db: PGLiteDB | null = null;
private _lock: LockHandle | null = null;
// Tier 3: when GBRAIN_PGLITE_SNAPSHOT loaded a post-initSchema state into
// PGlite.create(loadDataDir), initSchema is a no-op (schema is already
// present + migrations already applied). Saves ~1-3s per fresh test PGLite.
private _snapshotLoaded = false;
get db(): PGLiteDB {
if (!this._db) throw new Error('PGLite not connected. Call connect() first.');
@@ -46,9 +122,24 @@ export class PGLiteEngine implements BrainEngine {
throw new Error('Could not acquire PGLite lock. Another gbrain process is using the database.');
}
// Tier 3: optional snapshot fast-restore. Only applies to in-memory
// engines (no persistent dataDir). The snapshot was built from a fresh
// `initSchema()` run; if the version file matches the current MIGRATIONS
// hash, load the dump and skip the schema replay. Mismatch or missing
// file silently falls back to normal init.
let loadDataDir: Blob | undefined;
if (!dataDir && process.env.GBRAIN_PGLITE_SNAPSHOT) {
const snapshotResult = tryLoadSnapshot(process.env.GBRAIN_PGLITE_SNAPSHOT);
if (snapshotResult) {
loadDataDir = snapshotResult;
this._snapshotLoaded = true;
}
}
try {
this._db = await PGlite.create({
dataDir,
loadDataDir,
extensions: { vector, pg_trgm },
});
} catch (err) {
@@ -86,6 +177,11 @@ export class PGLiteEngine implements BrainEngine {
}
async initSchema(): Promise<void> {
// Tier 3: snapshot was loaded into PGlite — schema + migrations already
// applied. Nothing to do. Returns immediately.
if (this._snapshotLoaded) {
return;
}
// Pre-schema bootstrap: add forward-referenced state the embedded schema
// blob requires but that older brains don't have yet. Without this, a
// pre-v0.18 brain hits `CREATE INDEX idx_pages_source_id ON pages(source_id)`
@@ -1157,6 +1253,39 @@ export class PGLiteEngine implements BrainEngine {
return result.rows as unknown as RawData[];
}
// Dream-cycle significance verdict cache (v0.23).
async getDreamVerdict(filePath: string, contentHash: string): Promise<DreamVerdict | null> {
const result = await this.db.query<{
worth_processing: boolean;
reasons: string[] | null;
judged_at: Date | string;
}>(
`SELECT worth_processing, reasons, judged_at
FROM dream_verdicts
WHERE file_path = $1 AND content_hash = $2`,
[filePath, contentHash]
);
if (result.rows.length === 0) return null;
const r = result.rows[0];
return {
worth_processing: r.worth_processing,
reasons: r.reasons ?? [],
judged_at: r.judged_at instanceof Date ? r.judged_at.toISOString() : String(r.judged_at),
};
}
async putDreamVerdict(filePath: string, contentHash: string, verdict: DreamVerdictInput): Promise<void> {
await this.db.query(
`INSERT INTO dream_verdicts (file_path, content_hash, worth_processing, reasons)
VALUES ($1, $2, $3, $4::jsonb)
ON CONFLICT (file_path, content_hash) DO UPDATE SET
worth_processing = EXCLUDED.worth_processing,
reasons = EXCLUDED.reasons,
judged_at = now()`,
[filePath, contentHash, verdict.worth_processing, JSON.stringify(verdict.reasons)]
);
}
// Versions
async createVersion(slug: string): Promise<PageVersion> {
const { rows } = await this.db.query(
+34 -1
View File
@@ -1,5 +1,5 @@
import postgres from 'postgres';
import type { BrainEngine, LinkBatchInput, TimelineBatchInput, ReservedConnection } from './engine.ts';
import type { BrainEngine, LinkBatchInput, TimelineBatchInput, ReservedConnection, DreamVerdict, DreamVerdictInput } from './engine.ts';
import { MAX_SEARCH_LIMIT, clampSearchLimit } from './engine.ts';
import { runMigrations } from './migrate.ts';
import { SCHEMA_SQL } from './schema-embedded.ts';
@@ -1303,6 +1303,39 @@ export class PostgresEngine implements BrainEngine {
return rows as unknown as RawData[];
}
// Dream-cycle significance verdict cache (v0.23).
async getDreamVerdict(filePath: string, contentHash: string): Promise<DreamVerdict | null> {
const sql = this.sql;
const rows = await sql<Array<{
worth_processing: boolean;
reasons: string[] | null;
judged_at: Date;
}>>`
SELECT worth_processing, reasons, judged_at
FROM dream_verdicts
WHERE file_path = ${filePath} AND content_hash = ${contentHash}
`;
if (rows.length === 0) return null;
const r = rows[0];
return {
worth_processing: r.worth_processing,
reasons: r.reasons ?? [],
judged_at: r.judged_at instanceof Date ? r.judged_at.toISOString() : String(r.judged_at),
};
}
async putDreamVerdict(filePath: string, contentHash: string, verdict: DreamVerdictInput): Promise<void> {
const sql = this.sql;
await sql`
INSERT INTO dream_verdicts (file_path, content_hash, worth_processing, reasons)
VALUES (${filePath}, ${contentHash}, ${verdict.worth_processing}, ${sql.json(verdict.reasons as Parameters<typeof sql.json>[0])})
ON CONFLICT (file_path, content_hash) DO UPDATE SET
worth_processing = EXCLUDED.worth_processing,
reasons = EXCLUDED.reasons,
judged_at = now()
`;
}
// Versions
async createVersion(slug: string): Promise<PageVersion> {
const sql = this.sql;
+4 -3
View File
@@ -15,8 +15,9 @@
* of skills this intent is allowed to also match).
*
* Layer B (LLM tie-break, optional): only runs via `gbrain routing-eval
* --llm`. Not yet implemented in v0.17 core; the CLI stubs the flag
* so call sites are ready.
* --llm`. Not yet implemented in this release; the CLI accepts the
* flag (emits a stderr notice and runs Layer A only) so call sites
* are ready. A future release will wire up the tie-break layer.
*
* Fixture linter (D-CX-6): we reject fixtures where the normalized
* `intent` is a verbatim substring of any trigger phrase attached to
@@ -316,7 +317,7 @@ export function loadRoutingFixtures(skillsDir: string): LoadResult {
// ---------------------------------------------------------------------------
export interface RunRoutingEvalOptions {
/** Reserved for Layer B (LLM tie-break). Not implemented in v0.17. */
/** Reserved for Layer B (LLM tie-break). Not implemented in this release. */
llm?: boolean;
}
+17
View File
@@ -596,6 +596,22 @@ CREATE TABLE IF NOT EXISTS subagent_rate_leases (
);
CREATE INDEX IF NOT EXISTS idx_rate_leases_key_expires ON subagent_rate_leases (key, expires_at);
-- ============================================================
-- Dream-cycle significance verdict cache v0.23 synthesize phase
-- ============================================================
-- Caches the cheap Haiku "is this transcript worth processing?" verdict
-- per (file_path, content_hash) so backfill re-runs skip already-judged
-- files. Distinct from raw_data (which is page-scoped); transcripts
-- aren't pages.
CREATE TABLE IF NOT EXISTS dream_verdicts (
file_path TEXT NOT NULL,
content_hash TEXT NOT NULL,
worth_processing BOOLEAN NOT NULL,
reasons JSONB,
judged_at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (file_path, content_hash)
);
-- ============================================================
-- Cycle coordination lock v0.17 runCycle primitive
-- ============================================================
@@ -663,6 +679,7 @@ BEGIN
ALTER TABLE subagent_tool_executions ENABLE ROW LEVEL SECURITY;
ALTER TABLE subagent_rate_leases ENABLE ROW LEVEL SECURITY;
ALTER TABLE gbrain_cycle_locks ENABLE ROW LEVEL SECURITY;
ALTER TABLE dream_verdicts ENABLE ROW LEVEL SECURITY;
RAISE NOTICE 'RLS enabled on all tables (role % has BYPASSRLS)', current_user;
ELSE
RAISE WARNING 'Skipping RLS: role % does not have BYPASSRLS privilege. Run as postgres role to enable.', current_user;
+4 -4
View File
@@ -3,13 +3,13 @@
*
* Multiplies into ts_rank / vector cosine score at SQL build time so that
* curated content (originals/, concepts/, writing/) outranks bulk content
* (wintermute/chat/, daily/, media/x/) for non-temporal queries.
* (openclaw/chat/, daily/, media/x/) for non-temporal queries.
*
* Keyed by slug prefix. Longest-prefix-match wins (sorted at lookup time
* inside sql-ranking.ts). Defaults grounded in the composition of the
* canonical brain at ~/git/brain/.
*
* Override via env: GBRAIN_SOURCE_BOOST="originals/:1.8,wintermute/chat/:0.3"
* Override via env: GBRAIN_SOURCE_BOOST="originals/:1.8,openclaw/chat/:0.3"
* Hard-exclude via env: GBRAIN_SEARCH_EXCLUDE="test/,scratch/"
*/
@@ -36,7 +36,7 @@ export const DEFAULT_SOURCE_BOOSTS: Record<string, number> = {
'daily/': 0.8,
'media/x/': 0.7,
// Chat transcripts — massive, noisy, swamp keyword queries
'wintermute/chat/': 0.5,
'openclaw/chat/': 0.5,
};
/**
@@ -53,7 +53,7 @@ export const DEFAULT_HARD_EXCLUDES: string[] = [
/**
* Parse GBRAIN_SOURCE_BOOST env var.
* Format: comma-separated prefix:factor pairs.
* Example: "originals/:1.8,wintermute/chat/:0.3"
* Example: "originals/:1.8,openclaw/chat/:0.3"
*
* Malformed entries are skipped silently. Returns empty object if env is
* unset or unparseable in its entirety.
+19 -4
View File
@@ -116,9 +116,16 @@ export function planScaffold(opts: ScaffoldOptions): ScaffoldPlan {
}
/**
* Check whether the resolver already has a backtick-wrapped reference
* to `skills/<name>/SKILL.md`. Idempotency contract (D-CX-7) if
* present, we never re-append a row for this skill, even with --force.
* Check whether the resolver already references `skills/<name>/SKILL.md`
* in ANY form: backticked (`skills/foo/SKILL.md`), single-quoted
* ('skills/foo/SKILL.md'), double-quoted ("skills/foo/SKILL.md"), or
* bare (skills/foo/SKILL.md surrounded by non-word chars).
*
* Idempotency contract if any form is present, we never re-append a
* row for this skill, even with --force. This is broader than the
* original backtick-only match: users who hand-edit the resolver to
* normalize formatting (drop backticks, use quotes, etc.) should not
* cause duplicate rows on the next scaffold --force.
*/
function detectExistingResolverRow(resolverFile: string, name: string): boolean {
let content: string;
@@ -128,7 +135,15 @@ function detectExistingResolverRow(resolverFile: string, name: string): boolean
return false;
}
const escaped = name.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
const re = new RegExp(`\`skills\\/${escaped}\\/SKILL\\.md\``);
// Match the path with any common delimiter on either side: backtick,
// single quote, double quote, parenthesis, whitespace, start/end of
// line. The `(?:^|...)` and `(?:$|...)` anchors ensure we don't
// false-match on something like "skills/foo-bar/SKILL.md" when
// looking for "foo".
const re = new RegExp(
`(?:^|[\`'"\\s\\(\\[])skills\\/${escaped}\\/SKILL\\.md(?:[\`'"\\s\\)\\]]|$)`,
'm',
);
return re.test(content);
}
+124 -7
View File
@@ -238,15 +238,57 @@ function releaseLock(workspace: string): void {
const MANAGED_BEGIN = '<!-- gbrain:skillpack:begin -->';
const MANAGED_END = '<!-- gbrain:skillpack:end -->';
export function buildManagedBlock(manifest: BundleManifest, slugs: string[]): string {
// Receipt comment embedded inside the fence on every write. Lets the
// next install distinguish "row gbrain installed previously" from
// "row a user hand-added inside the fence." Format is intentionally
// regex-friendly.
//
// <!-- gbrain:skillpack:manifest cumulative-slugs="a,b,c" version="0.19.0" -->
//
// Sorted, comma-separated slug list. version is the gbrain version
// that wrote this receipt.
const RECEIPT_RE =
/<!-- gbrain:skillpack:manifest cumulative-slugs="([^"]*)" version="([^"]*)" -->/;
function buildReceipt(cumulativeSlugs: string[], version: string): string {
const sorted = [...cumulativeSlugs].sort();
return `<!-- gbrain:skillpack:manifest cumulative-slugs="${sorted.join(',')}" version="${version}" -->`;
}
/**
* Parse the receipt comment from a managed block. Returns null if no
* receipt is present (pre-v0.19 fences). The slug list is split on
* comma; an empty string returns an empty list.
*/
export function parseReceipt(resolverContent: string): { cumulativeSlugs: string[]; version: string } | null {
const beginIdx = resolverContent.indexOf(MANAGED_BEGIN);
const endIdx = resolverContent.indexOf(MANAGED_END);
if (beginIdx === -1 || endIdx === -1 || endIdx <= beginIdx) return null;
const block = resolverContent.slice(beginIdx, endIdx);
const m = RECEIPT_RE.exec(block);
if (!m) return null;
const slugs = m[1].length === 0 ? [] : m[1].split(',');
return { cumulativeSlugs: slugs, version: m[2] };
}
export function buildManagedBlock(
manifest: BundleManifest,
slugs: string[],
cumulativeSlugs?: string[],
): string {
const sorted = [...slugs].sort();
const rows = sorted.map(
slug => `| "${slug}" | \`skills/${slug}/SKILL.md\` |`,
);
// Default cumulative = the rendered slug set when caller didn't
// pass one explicitly (kept backward-compatible with older callers
// that don't yet thread the cumulative set through).
const receipt = buildReceipt(cumulativeSlugs ?? sorted, manifest.version);
return [
MANAGED_BEGIN,
'',
`<!-- Installed by gbrain ${manifest.version} — do not hand-edit between markers. -->`,
receipt,
'',
'| Trigger | Skill |',
'|---------|-------|',
@@ -339,15 +381,24 @@ export function applyInstall(
});
}
// Managed block update
// Managed block update.
//
// installedSlugs = slugs we just wrote in THIS call.
// bundleSlugs = the FULL bundle manifest's slug list (always
// populated; used for the install-all prune path).
// isInstallAll = caller passed --all (no specific skillSlug).
const installedSlugs = opts.skillSlug
? [opts.skillSlug]
: plan.manifest.skills.map(pathSlug);
const bundleSlugs = plan.manifest.skills.map(pathSlug);
const isInstallAll = !opts.skillSlug;
const managedBlock = applyManagedBlock(
plan.targetWorkspace,
plan.targetSkillsDir,
plan.manifest,
installedSlugs,
bundleSlugs,
isInstallAll,
opts.dryRun ?? false,
);
@@ -371,6 +422,8 @@ function applyManagedBlock(
skillsDir: string,
manifest: BundleManifest,
installedSlugs: string[],
bundleSlugs: string[],
isInstallAll: boolean,
dryRun: boolean,
): ManagedBlockResult {
// Prefer skills-dir resolver; fall back to workspace-root resolver.
@@ -383,11 +436,75 @@ function applyManagedBlock(
};
}
const existing = readFileSync(resolver, 'utf-8');
// Merge with any slugs already present in the managed block so
// repeated single-skill installs accumulate rather than overwrite.
const priorSlugs = extractManagedSlugs(existing);
const merged = Array.from(new Set([...priorSlugs, ...installedSlugs]));
const newBlock = buildManagedBlock(manifest, merged);
// Step 1: figure out what gbrain previously installed into this fence.
// - If receipt is present, trust it as the cumulative-slug history.
// - If receipt is absent (pre-v0.19 fence), fall back to the rows
// currently in the fence — they were ALL gbrain-written before
// the receipt feature existed, so trust them as the prior set.
const receipt = parseReceipt(existing);
const priorCumulativeSlugs =
receipt !== null
? new Set(receipt.cumulativeSlugs)
: new Set(extractManagedSlugs(existing));
// Step 2: compute the new cumulative slug set.
// - Single-skill install: union(prior, installed). Per-skill
// installs accumulate; the documented v0.18 behavior.
// - Install-all: prune slugs no longer in the bundle. Renamed
// and removed skills leave the cumulative set ONLY via this
// path. (Single-skill never prunes — it would regress
// cumulative semantics for unrelated skills.)
//
// We track `prunedSlugs` separately so the unknown-row detector
// (Step 3) doesn't re-resurrect slugs we just intentionally removed.
const newCumulative = new Set(priorCumulativeSlugs);
for (const s of installedSlugs) newCumulative.add(s);
const prunedSlugs = new Set<string>();
if (isInstallAll) {
const bundleSet = new Set(bundleSlugs);
for (const s of [...newCumulative]) {
if (!bundleSet.has(s)) {
newCumulative.delete(s);
prunedSlugs.add(s);
}
}
}
// Step 3: detect unknown rows. A row inside the fence whose slug
// is NOT in newCumulative AND NOT in bundleSlugs AND NOT in the
// intentionally-pruned set is something gbrain never wrote: a user
// hand-add, a typo, or stale debris from an unknown bundle.
// Preserve it (do not destroy data) and emit a single stderr
// warning per slug instructing the agent to investigate.
const existingRowSlugs = extractManagedSlugs(existing);
const bundleSet = new Set(bundleSlugs);
const unknownSlugs: string[] = [];
// Skip the unknown-row check on the very first v0.19 install (no
// receipt yet). All existing rows are presumed gbrain-written and
// captured into newCumulative via the fallback above; warning here
// would create false positives.
if (receipt !== null) {
for (const slug of existingRowSlugs) {
if (newCumulative.has(slug)) continue;
if (bundleSet.has(slug)) continue;
if (prunedSlugs.has(slug)) continue; // known prune, do not resurrect
unknownSlugs.push(slug);
// Re-add to newCumulative so the rebuild preserves the row.
newCumulative.add(slug);
}
}
for (const slug of unknownSlugs) {
console.error(
`[skillpack] unknown row in managed block: "${slug}" at skills/${slug}/SKILL.md — not in gbrain's installed set. Investigate: user-added skill, hand-edited fence, or typo?`,
);
}
// Step 4: write the new block. The visible row set is sorted
// newCumulative. The receipt comment carries the same set so the
// next install can do the same diff.
const cumulativeArr = [...newCumulative].sort();
const newBlock = buildManagedBlock(manifest, cumulativeArr, cumulativeArr);
const updated = updateManagedBlock(existing, newBlock);
if (updated === existing) {
return { resolverFile: resolver, applied: false, skippedReason: 'no_change' };
+101
View File
@@ -0,0 +1,101 @@
/**
* Shared concurrency policy for sync + import + jobs paths.
*
* Three callers used to embed three different policies:
* - performSync (incremental): >100 files 4 workers
* - performFullSync: Postgres 4 workers
* - jobs.ts sync handler: hardcoded 4
*
* They drift over time and confuse users ("why does my sync not parallelize?"
* is a different answer in each path). This module is one source of truth.
*
* v0.22.13 extracted as part of the parallel-sync hardening (PR #490).
*/
import type { BrainEngine } from './engine.ts';
/** Threshold above which auto-concurrency fires for incremental sync paths. */
export const AUTO_CONCURRENCY_FILE_THRESHOLD = 100;
/** Minimum file count below which the parallel branch is skipped even when
* auto-concurrency would otherwise fire. Prevents spawning workers for trivial
* diffs where setup cost exceeds parallelism gains. Only consulted on the
* auto path; explicit `--workers N` bypasses this. */
export const PARALLEL_FILE_FLOOR = 50;
/** Default worker count when auto-concurrency fires. */
export const DEFAULT_PARALLEL_WORKERS = 4;
/**
* Resolve effective worker count for a sync/import operation.
*
* Inputs:
* - engine.kind: 'pglite' always returns 1 (single-connection)
* - override: caller's explicit --workers / opts.concurrency value
* - fileCount: size of the work batch
*
* Rules:
* - PGLite always 1 (the engine is single-connection regardless)
* - explicit override respect it (clamped to >=1)
* - auto path DEFAULT_PARALLEL_WORKERS when fileCount > AUTO_CONCURRENCY_FILE_THRESHOLD, else 1
*
* Note: this function does NOT consult PARALLEL_FILE_FLOOR. The floor is a
* caller-side gate that decides whether to take the parallel code path even
* when the worker count is > 1. It only applies to the auto path; explicit
* --workers bypasses the floor entirely (per Q1 in PR #490).
*/
export function autoConcurrency(
engine: BrainEngine,
fileCount: number,
override?: number,
): number {
if (engine.kind === 'pglite') return 1;
if (override !== undefined) return Math.max(1, override);
return fileCount > AUTO_CONCURRENCY_FILE_THRESHOLD
? DEFAULT_PARALLEL_WORKERS
: 1;
}
/**
* Decide whether the parallel code path should run.
*
* - workers <= 1 never parallel
* - workers > 1 + explicit override always parallel (user opted in,
* respect them even on small diffs Q1 in PR #490)
* - workers > 1 + auto path parallel only when fileCount > PARALLEL_FILE_FLOOR
*/
export function shouldRunParallel(
workers: number,
fileCount: number,
explicit: boolean,
): boolean {
if (workers <= 1) return false;
if (explicit) return true;
return fileCount > PARALLEL_FILE_FLOOR;
}
/**
* Parse a `--workers N` / `--concurrency N` CLI argument value.
*
* Returns:
* - undefined when the flag was not provided
* - a positive integer when the flag was provided with a valid value
*
* Throws on:
* - non-integer ("foo", "1.5", "")
* - zero or negative ("0", "-3")
* - NaN / Infinity
*
* Q2 in PR #490: the prior parseInt-with-no-validation accepted `--workers 0`
* and silently fell through to auto-concurrency (4 workers), the opposite of
* what the user typed. Fail loud instead.
*/
export function parseWorkers(s: string | undefined): number | undefined {
if (s === undefined) return undefined;
const n = parseInt(s, 10);
if (!Number.isFinite(n) || n < 1 || String(n) !== s.trim()) {
throw new Error(
`--workers must be a positive integer, got: ${JSON.stringify(s)}`,
);
}
return n;
}
+2 -2
View File
@@ -301,7 +301,7 @@ export function resolveSlugForPath(filePath: string, repoPrefix?: string): strin
import { existsSync as _existsSync, readFileSync as _readFileSync, appendFileSync as _appendFileSync, mkdirSync as _mkdirSync } from 'fs';
import { join as _joinPath } from 'path';
import { homedir as _homedir } from 'os';
import { gbrainPath as _gbrainPath } from './config.ts';
import { createHash as _createHash } from 'crypto';
export interface SyncFailure {
@@ -402,7 +402,7 @@ export function formatCodeBreakdown(
}
function _failuresDir(): string {
return _joinPath(_homedir(), '.gbrain');
return _gbrainPath();
}
export function syncFailuresPath(): string {
+17
View File
@@ -592,6 +592,22 @@ CREATE TABLE IF NOT EXISTS subagent_rate_leases (
);
CREATE INDEX IF NOT EXISTS idx_rate_leases_key_expires ON subagent_rate_leases (key, expires_at);
-- ============================================================
-- Dream-cycle significance verdict cache — v0.21 synthesize phase
-- ============================================================
-- Caches the cheap Haiku "is this transcript worth processing?" verdict
-- per (file_path, content_hash) so backfill re-runs skip already-judged
-- files. Distinct from raw_data (which is page-scoped); transcripts
-- aren't pages.
CREATE TABLE IF NOT EXISTS dream_verdicts (
file_path TEXT NOT NULL,
content_hash TEXT NOT NULL,
worth_processing BOOLEAN NOT NULL,
reasons JSONB,
judged_at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (file_path, content_hash)
);
-- ============================================================
-- Cycle coordination lock — v0.17 runCycle primitive
-- ============================================================
@@ -659,6 +675,7 @@ BEGIN
ALTER TABLE subagent_tool_executions ENABLE ROW LEVEL SECURITY;
ALTER TABLE subagent_rate_leases ENABLE ROW LEVEL SECURITY;
ALTER TABLE gbrain_cycle_locks ENABLE ROW LEVEL SECURITY;
ALTER TABLE dream_verdicts ENABLE ROW LEVEL SECURITY;
RAISE NOTICE 'RLS enabled on all tables (role % has BYPASSRLS)', current_user;
ELSE
RAISE WARNING 'Skipping RLS: role % does not have BYPASSRLS privilege. Run as postgres role to enable.', current_user;
+97
View File
@@ -0,0 +1,97 @@
/**
* AgentRunner registry + selection tests. Proves the harness contract is
* truly agent-agnostic via a fake-runner integration.
*/
import { describe, test, expect, beforeEach } from 'bun:test';
import {
registerAgentRunner, resolveAgentRunner, listRegisteredAgents,
_resetRegistryForTests,
type AgentRunner, type DetectResult, type InvokeOpts, type InvokeResult, type TranscriptSink,
} from '../src/core/claw-test/agent-runner.ts';
class FakeRunner implements AgentRunner {
readonly name: string;
invocations = 0;
detected: DetectResult = { available: true, binPath: '/usr/bin/fake-agent' };
constructor(name: string) { this.name = name; }
async detect(): Promise<DetectResult> { return this.detected; }
async invoke(_opts: InvokeOpts): Promise<InvokeResult> {
this.invocations++;
return { exitCode: 0, durationMs: 1 };
}
}
beforeEach(() => {
_resetRegistryForTests();
});
describe('registry', () => {
test('register + resolve roundtrips', () => {
registerAgentRunner('fake', () => new FakeRunner('fake'));
const r = resolveAgentRunner('fake');
expect(r.name).toBe('fake');
});
test('resolve unknown agent throws with helpful list', () => {
registerAgentRunner('alpha', () => new FakeRunner('alpha'));
registerAgentRunner('beta', () => new FakeRunner('beta'));
expect(() => resolveAgentRunner('gamma')).toThrow(/registered: alpha, beta/);
});
test('listRegisteredAgents returns sorted names', () => {
registerAgentRunner('zeta', () => new FakeRunner('zeta'));
registerAgentRunner('alpha', () => new FakeRunner('alpha'));
expect(listRegisteredAgents()).toEqual(['alpha', 'zeta']);
});
test('factory pattern produces independent instances', () => {
registerAgentRunner('fake', () => new FakeRunner('fake'));
const a = resolveAgentRunner('fake') as FakeRunner;
const b = resolveAgentRunner('fake') as FakeRunner;
expect(a).not.toBe(b);
});
});
describe('agent-agnosticism guard', () => {
test('a fake runner can satisfy the AgentRunner contract end-to-end', async () => {
registerAgentRunner('fake', () => new FakeRunner('fake'));
const runner = resolveAgentRunner('fake');
// The harness contract: detect → invoke. Nothing else.
const detected = await runner.detect();
expect(detected.available).toBe(true);
expect(detected.binPath).toBe('/usr/bin/fake-agent');
let written = 0;
const sink: TranscriptSink = {
write: () => { written++; },
nextOffset: () => 0,
close: async () => { /* noop */ },
};
const result = await runner.invoke({
cwd: '/tmp',
brief: 'hello',
env: {},
timeoutMs: 1000,
transcriptSink: sink,
});
expect(result.exitCode).toBe(0);
});
test('a runner reporting unavailable still satisfies the contract', async () => {
class UnavailableRunner implements AgentRunner {
name = 'gone';
async detect() { return { available: false, reason: 'not installed' } as DetectResult; }
async invoke(): Promise<InvokeResult> { throw new Error('should not be called'); }
}
registerAgentRunner('gone', () => new UnavailableRunner());
const r = resolveAgentRunner('gone');
const d = await r.detect();
expect(d.available).toBe(false);
expect(d.reason).toBe('not installed');
});
});
+7
View File
@@ -23,6 +23,13 @@ import { describe, test, expect } from 'bun:test';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { LATEST_VERSION } from '../src/core/migrate.ts';
// Tier 3 opt-out: this file tests the cold init / bootstrap path explicitly.
// If GBRAIN_PGLITE_SNAPSHOT is set (ci:local sets it for unit shards), every
// PGlite would boot post-initSchema and these assertions ("0 tables on fresh
// install", "bootstrap converts pre-v0.18 brain to LATEST") would fail
// trivially. Unset for this file's process.
delete process.env.GBRAIN_PGLITE_SNAPSHOT;
describe('PGLiteEngine#applyForwardReferenceBootstrap', () => {
test('no-op on fresh install (no pages or links table)', async () => {
const engine = new PGLiteEngine();
+165
View File
@@ -0,0 +1,165 @@
/**
* gbrain claw-test CLI dispatch tests.
*
* These tests exercise the harness's argument parsing, scenario loading,
* agent registry resolution, and friction-report path. They do NOT spawn
* real gbrain commands (no built binary in CI yet); the canonical scripted
* E2E that walks `gbrain init → import → query → extract → verify` lives
* in test/e2e/claw-test.test.ts and gates on a built binary.
*/
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
import { mkdtempSync, rmSync, existsSync, readFileSync } from 'fs';
import { join } from 'path';
import { tmpdir } from 'os';
import { runFriction } from '../src/commands/friction.ts';
import { listScenarios, loadScenario } from '../src/core/claw-test/scenarios.ts';
import {
registerAgentRunner, resolveAgentRunner, listRegisteredAgents,
_resetRegistryForTests,
type AgentRunner, type DetectResult, type InvokeOpts, type InvokeResult,
} from '../src/core/claw-test/agent-runner.ts';
let tmp: string;
const ORIG_HOME = process.env.GBRAIN_HOME;
beforeEach(() => {
tmp = mkdtempSync(join(tmpdir(), 'claw-test-cli-'));
process.env.GBRAIN_HOME = tmp;
_resetRegistryForTests();
});
afterEach(() => {
process.env.GBRAIN_HOME = ORIG_HOME;
rmSync(tmp, { recursive: true, force: true });
});
describe('shipped scenarios are loadable', () => {
test('default fixtures root contains both v1 scenarios', () => {
delete process.env.GBRAIN_CLAW_SCENARIOS_DIR;
const names = listScenarios();
expect(names).toContain('fresh-install');
expect(names).toContain('upgrade-from-v0.18');
});
test('fresh-install has expected_phases', () => {
delete process.env.GBRAIN_CLAW_SCENARIOS_DIR;
const cfg = loadScenario('fresh-install');
expect(cfg.expectedPhases).toContain('import.files');
expect(cfg.expectedPhases).toContain('extract.links_fs');
expect(cfg.expectedPhases).toContain('doctor.db_checks');
});
test('upgrade-from-v0.18 declares from_version', () => {
delete process.env.GBRAIN_CLAW_SCENARIOS_DIR;
const cfg = loadScenario('upgrade-from-v0.18');
expect(cfg.kind).toBe('upgrade');
expect(cfg.fromVersion).toBe('0.18.0');
expect(cfg.seedRelative).toBe('seed');
});
});
describe('agent registry — fake-runner integration', () => {
test('a fake runner can be registered, resolved, and detect/invoke called', async () => {
let invokeCount = 0;
class FakeRunner implements AgentRunner {
readonly name = 'fake';
async detect(): Promise<DetectResult> { return { available: true, binPath: '/usr/bin/fake' }; }
async invoke(_opts: InvokeOpts): Promise<InvokeResult> {
invokeCount++;
return { exitCode: 0, durationMs: 1 };
}
}
registerAgentRunner('fake', () => new FakeRunner());
expect(listRegisteredAgents()).toContain('fake');
const r = resolveAgentRunner('fake');
const detected = await r.detect();
expect(detected.available).toBe(true);
const result = await r.invoke({
cwd: tmp,
brief: 'test',
env: {},
timeoutMs: 1000,
transcriptSink: { write: () => {}, nextOffset: () => 0, close: async () => {} },
});
expect(result.exitCode).toBe(0);
expect(invokeCount).toBe(1);
});
test('resolveAgentRunner with unknown name throws with registered list', () => {
registerAgentRunner('alpha', () => ({} as AgentRunner));
expect(() => resolveAgentRunner('unknown')).toThrow(/registered: alpha/);
});
});
describe('friction CLI integrates with harness run-id env', () => {
test('GBRAIN_FRICTION_RUN_ID populates harness-style run-ids', () => {
process.env.GBRAIN_FRICTION_RUN_ID = 'claw-test-20260428-fake-abcd1234';
try {
const code = runFriction(['log', '--phase', 'install', '--message', 'simulated harness write']);
expect(code).toBe(0);
const expectedFile = join(tmp, '.gbrain', 'friction', 'claw-test-20260428-fake-abcd1234.jsonl');
expect(existsSync(expectedFile)).toBe(true);
const raw = readFileSync(expectedFile, 'utf-8');
const entry = JSON.parse(raw.split('\n')[0]);
expect(entry.run_id).toBe('claw-test-20260428-fake-abcd1234');
expect(entry.message).toBe('simulated harness write');
} finally {
delete process.env.GBRAIN_FRICTION_RUN_ID;
}
});
});
describe('OpenClawRunner detection (reliable on box without openclaw)', () => {
test('detect returns unavailable when OPENCLAW_BIN missing', async () => {
const orig = process.env.OPENCLAW_BIN;
delete process.env.OPENCLAW_BIN;
try {
const { OpenClawRunner } = await import('../src/core/claw-test/runners/openclaw.ts');
const r = new OpenClawRunner();
const d = await r.detect();
// Either unavailable, or available if openclaw IS on PATH for the dev — both states are valid.
// We only assert the contract shape.
expect(typeof d.available).toBe('boolean');
if (!d.available) {
expect(typeof d.reason).toBe('string');
} else {
expect(d.binPath?.startsWith('/')).toBe(true);
}
} finally {
if (orig !== undefined) process.env.OPENCLAW_BIN = orig;
}
});
test('detect rejects relative OPENCLAW_BIN', async () => {
const orig = process.env.OPENCLAW_BIN;
process.env.OPENCLAW_BIN = 'relative/openclaw';
try {
const { OpenClawRunner } = await import('../src/core/claw-test/runners/openclaw.ts');
const r = new OpenClawRunner();
const d = await r.detect();
expect(d.available).toBe(false);
expect(d.reason).toMatch(/absolute/);
} finally {
if (orig !== undefined) process.env.OPENCLAW_BIN = orig;
else delete process.env.OPENCLAW_BIN;
}
});
test("detect rejects '..' segments in OPENCLAW_BIN", async () => {
const orig = process.env.OPENCLAW_BIN;
process.env.OPENCLAW_BIN = '/tmp/foo/../bar';
try {
const { OpenClawRunner } = await import('../src/core/claw-test/runners/openclaw.ts');
const r = new OpenClawRunner();
const d = await r.detect();
expect(d.available).toBe(false);
expect(d.reason).toMatch(/'\.\.' segments/);
} finally {
if (orig !== undefined) process.env.OPENCLAW_BIN = orig;
else delete process.env.OPENCLAW_BIN;
}
});
});
+4 -4
View File
@@ -377,8 +377,8 @@ describe('runCycle — yieldBetweenPhases hook', () => {
hookCalls++;
},
});
// 6 phases → 6 yield calls (one after each).
expect(hookCalls).toBe(6);
// v0.23: 8 phases → 8 yield calls (one after each).
expect(hookCalls).toBe(8);
});
test('hook exceptions do not abort the cycle', async () => {
@@ -388,8 +388,8 @@ describe('runCycle — yieldBetweenPhases hook', () => {
throw new Error('synthetic hook error');
},
});
// Cycle still completed all phases.
expect(report.phases.length).toBe(6);
// Cycle still completed all phases (v0.23: 8).
expect(report.phases.length).toBe(8);
});
});
+83
View File
@@ -0,0 +1,83 @@
/**
* Unit tests for the patterns phase (v0.21).
*
* The phase invokes a subagent and queues real Minions work, so this
* file leans on structural assertions over the source + a single
* end-to-end driver run that exercises the skip-paths.
*
* Full LLM behavior is exercised by E2E tests in test/e2e/.
*/
import { describe, test, expect } from 'bun:test';
import { readFileSync } from 'fs';
const patternsSrc = readFileSync(
new URL('../src/core/cycle/patterns.ts', import.meta.url),
'utf-8',
);
describe('patterns phase wiring', () => {
test('imports queue + waitForCompletion + types', () => {
expect(patternsSrc).toContain("import { MinionQueue }");
expect(patternsSrc).toContain('waitForCompletion');
expect(patternsSrc).toContain('SubagentHandlerData');
});
test('threads allowed_slug_prefixes from filing-rules JSON', () => {
expect(patternsSrc).toContain('allowed_slug_prefixes');
expect(patternsSrc).toContain('_brain-filing-rules.json');
expect(patternsSrc).toContain('dream_synthesize_paths');
});
test('reads min_evidence + lookback_days config', () => {
expect(patternsSrc).toContain('dream.patterns.min_evidence');
expect(patternsSrc).toContain('dream.patterns.lookback_days');
});
test('uses subagent_tool_executions for slug provenance (Codex #2 fix)', () => {
expect(patternsSrc).toContain('subagent_tool_executions');
expect(patternsSrc).toContain("tool_name = 'brain_put_page'");
});
test('skips when ANTHROPIC_API_KEY missing', () => {
expect(patternsSrc).toContain('ANTHROPIC_API_KEY');
expect(patternsSrc).toContain('no_api_key');
});
test('skips when reflections below min_evidence', () => {
expect(patternsSrc).toContain('insufficient_evidence');
});
test('reverse-writes pages to disk via serializeMarkdown', () => {
expect(patternsSrc).toContain('serializeMarkdown');
expect(patternsSrc).toContain('writeFileSync');
});
test('runs after extract — queries fresh graph', () => {
// Documented invariant: pattern phase MUST run after extract.
// The cycle.ts dispatcher enforces order; this just confirms the
// patterns module doesn't try to compute its own auto-link layer
// (which would be a subtle regression).
expect(patternsSrc).not.toContain('runAutoLink');
expect(patternsSrc).not.toContain('extractPageLinks(');
});
test('does NOT use raw_data table (Codex #3 fix)', () => {
expect(patternsSrc).not.toContain('putRawData');
expect(patternsSrc).not.toContain('getRawData');
});
});
describe('patterns scope filter', () => {
test('filters reflections by slug LIKE wiki/personal/reflections/%', () => {
expect(patternsSrc).toContain("slug LIKE 'wiki/personal/reflections/%'");
});
test('orders by updated_at DESC for recency-bias', () => {
expect(patternsSrc).toContain('ORDER BY updated_at DESC');
});
test('caps gather to 100 reflections (cost control)', () => {
expect(patternsSrc).toContain('LIMIT 100');
});
});
+328
View File
@@ -0,0 +1,328 @@
/**
* Unit tests for the synthesize phase scaffolding.
*
* Covers transcript-discovery branches (date filters, exclude regex,
* minChars, multiple sources) and the compileExcludePatterns word-
* boundary heuristic. Doesn't drive a real Anthropic call full
* cycle E2E lives in test/e2e/.
*/
import { describe, test, expect, beforeEach } from 'bun:test';
import { mkdtempSync, writeFileSync, rmSync, mkdirSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import {
discoverTranscripts,
readSingleTranscript,
compileExcludePatterns,
isDreamOutput,
DREAM_OUTPUT_MARKER_RE,
} from '../src/core/cycle/transcript-discovery.ts';
import { judgeSignificance, renderPageToMarkdown, type JudgeClient } from '../src/core/cycle/synthesize.ts';
let tmpDir: string;
function makeTranscript(name: string, body: string): string {
const path = join(tmpDir, name);
writeFileSync(path, body, 'utf8');
return path;
}
beforeEach(() => {
tmpDir = mkdtempSync(join(tmpdir(), 'gbrain-synth-test-'));
});
describe('compileExcludePatterns', () => {
test('auto-wraps bare words in word-boundary regex (Q-3)', () => {
const res = compileExcludePatterns(['medical']);
expect(res).toHaveLength(1);
// word boundary: matches "medical" but NOT "comedical"
expect(res[0].test('medical advice')).toBe(true);
expect(res[0].test('comedical')).toBe(false);
});
test('honors raw regex when input is non-bare-word', () => {
const res = compileExcludePatterns(['^therapy:']);
expect(res[0].test('therapy: today was hard')).toBe(true);
expect(res[0].test('thinking about therapy:')).toBe(false);
});
test('skips invalid regex with warning, does not crash', () => {
const res = compileExcludePatterns(['valid', '(broken[']);
expect(res).toHaveLength(1); // only the valid one compiled
});
test('case-insensitive matching by default', () => {
const res = compileExcludePatterns(['Medical']);
expect(res[0].test('medical advice')).toBe(true);
expect(res[0].test('MEDICAL ADVICE')).toBe(true);
});
test('empty / undefined input returns empty array', () => {
expect(compileExcludePatterns(undefined)).toEqual([]);
expect(compileExcludePatterns([])).toEqual([]);
expect(compileExcludePatterns([''])).toEqual([]);
});
});
describe('discoverTranscripts', () => {
test('returns empty when corpusDir does not exist', () => {
const out = discoverTranscripts({ corpusDir: '/nonexistent/path' });
expect(out).toEqual([]);
});
test('returns transcripts above minChars, sorted by filePath', () => {
makeTranscript('2026-04-25-session.txt', 'a'.repeat(2500));
makeTranscript('2026-04-24-other.txt', 'b'.repeat(2500));
const out = discoverTranscripts({ corpusDir: tmpDir, minChars: 1000 });
expect(out).toHaveLength(2);
expect(out[0].basename).toBe('2026-04-24-other');
expect(out[1].basename).toBe('2026-04-25-session');
});
test('skips transcripts below minChars', () => {
makeTranscript('2026-04-25-short.txt', 'tiny');
const out = discoverTranscripts({ corpusDir: tmpDir, minChars: 2000 });
expect(out).toEqual([]);
});
test('skips non-txt files', () => {
makeTranscript('2026-04-25-foo.md', 'a'.repeat(3000));
const out = discoverTranscripts({ corpusDir: tmpDir, minChars: 1000 });
expect(out).toEqual([]);
});
test('exclude_patterns filters out matched transcripts (word boundary)', () => {
makeTranscript('2026-04-25-medical.txt', 'discussing medical advice ' + 'x'.repeat(3000));
makeTranscript('2026-04-25-comedy.txt', 'comedical writing tips ' + 'x'.repeat(3000));
const out = discoverTranscripts({
corpusDir: tmpDir,
minChars: 1000,
excludePatterns: ['medical'],
});
expect(out).toHaveLength(1);
expect(out[0].basename).toBe('2026-04-25-comedy');
});
test('--date filter restricts to one specific YYYY-MM-DD basename', () => {
makeTranscript('2026-04-25-foo.txt', 'a'.repeat(3000));
makeTranscript('2026-04-26-bar.txt', 'b'.repeat(3000));
const out = discoverTranscripts({
corpusDir: tmpDir,
minChars: 1000,
date: '2026-04-25',
});
expect(out).toHaveLength(1);
expect(out[0].basename).toBe('2026-04-25-foo');
});
test('--from / --to range filters basename dates', () => {
makeTranscript('2026-04-23-a.txt', 'a'.repeat(3000));
makeTranscript('2026-04-25-b.txt', 'b'.repeat(3000));
makeTranscript('2026-04-27-c.txt', 'c'.repeat(3000));
const out = discoverTranscripts({
corpusDir: tmpDir,
minChars: 1000,
from: '2026-04-24',
to: '2026-04-26',
});
expect(out).toHaveLength(1);
expect(out[0].basename).toBe('2026-04-25-b');
});
test('multiple sources (corpus + meeting transcripts) merged', () => {
makeTranscript('2026-04-25-session.txt', 'a'.repeat(3000));
const meetDir = mkdtempSync(join(tmpdir(), 'gbrain-meet-'));
writeFileSync(join(meetDir, '2026-04-25-meeting.txt'), 'b'.repeat(3000));
const out = discoverTranscripts({
corpusDir: tmpDir,
meetingTranscriptsDir: meetDir,
minChars: 1000,
});
expect(out).toHaveLength(2);
rmSync(meetDir, { recursive: true, force: true });
});
test('content_hash is stable for identical content, different for edits (A-3)', () => {
makeTranscript('2026-04-25-a.txt', 'identical content ' + 'x'.repeat(3000));
makeTranscript('2026-04-25-b.txt', 'identical content ' + 'x'.repeat(3000));
const out1 = discoverTranscripts({ corpusDir: tmpDir, minChars: 1000 });
expect(out1[0].contentHash).toBe(out1[1].contentHash);
// Edit one — hash changes
makeTranscript('2026-04-25-a.txt', 'edited content ' + 'x'.repeat(3000));
const out2 = discoverTranscripts({ corpusDir: tmpDir, minChars: 1000 });
expect(out2[0].contentHash).not.toBe(out2[1].contentHash);
});
});
describe('readSingleTranscript', () => {
test('returns transcript above minChars', () => {
const path = makeTranscript('hello.txt', 'a'.repeat(3000));
const t = readSingleTranscript(path, { minChars: 1000 });
expect(t).not.toBeNull();
expect(t!.basename).toBe('hello');
});
test('returns null when below minChars', () => {
const path = makeTranscript('hello.txt', 'tiny');
const t = readSingleTranscript(path, { minChars: 2000 });
expect(t).toBeNull();
});
test('returns null when content matches exclude pattern', () => {
const path = makeTranscript('hello.txt', 'medical content ' + 'x'.repeat(3000));
const t = readSingleTranscript(path, { minChars: 1000, excludePatterns: ['medical'] });
expect(t).toBeNull();
});
test('throws on missing file', () => {
expect(() => readSingleTranscript('/nonexistent/foo.txt')).toThrow();
});
test('infers date from YYYY-MM-DD basename', () => {
const path = makeTranscript('2026-04-25-thing.txt', 'a'.repeat(3000));
const t = readSingleTranscript(path, { minChars: 1000 });
expect(t!.inferredDate).toBe('2026-04-25');
});
test('inferredDate null when basename does not start with YYYY-MM-DD', () => {
const path = makeTranscript('random-basename.txt', 'a'.repeat(3000));
const t = readSingleTranscript(path, { minChars: 1000 });
expect(t!.inferredDate).toBeNull();
});
});
describe('self-consumption guard (v0.23.2 marker-based)', () => {
test('REGRESSION: catches actual reverseWriteSlugs output from a real Page', () => {
// Build a Page like the synthesize subagent would produce, run it through
// the same renderPageToMarkdown the orchestrator uses, and assert the guard
// fires. Codex finding #5: synthetic-string fixtures don't prove the guard
// catches what the synthesize phase actually produces.
const page = {
slug: 'wiki/personal/reflections/2026-04-30-test-abc123',
type: 'reflection' as const,
title: 'Test reflection',
compiled_truth: 'I learned something about [Alice](people/alice). No own-slug citation in body.',
timeline: '',
frontmatter: {},
};
const md = renderPageToMarkdown(page as any, ['dream-cycle']);
const path = makeTranscript('2026-04-30-output.txt', md + '\n' + 'x'.repeat(3000));
const result = readSingleTranscript(path, { minChars: 100 });
expect(result).toBeNull();
});
test('does NOT fire on real conversation transcript citing a brain slug', () => {
// The exact false-positive case codex finding #1 named: a user note that
// legitimately mentions a reflection slug in plain text. Must NOT be skipped.
const path = makeTranscript('convo.txt',
'User: tell me about wiki/personal/reflections/identity-foo and how it relates to my work.\n' +
'Agent: ' + 'x'.repeat(3000));
const result = readSingleTranscript(path, { minChars: 100 });
expect(result).not.toBeNull();
});
test('CRLF + BOM frontmatter still triggers guard', () => {
const content = '\uFEFF---\r\ndream_generated: true\r\n---\r\n# x\r\n' + 'x'.repeat(3000);
const path = makeTranscript('crlf.txt', content);
const result = readSingleTranscript(path, { minChars: 100 });
expect(result).toBeNull();
});
test('whitespace and case tolerance: matches dream_generated: true variants', () => {
const variants = [
'---\ndream_generated:true\n---\nbody' + 'x'.repeat(3000),
'---\ndream_generated: true\n---\nbody' + 'x'.repeat(3000),
'---\ndream_generated: TRUE\n---\nbody' + 'x'.repeat(3000),
'---\ntitle: foo\ndream_generated: true\n---\nbody' + 'x'.repeat(3000),
];
for (const variant of variants) {
expect(isDreamOutput(variant)).toBe(true);
}
});
test('does NOT fire when dream_generated is false or absent', () => {
expect(isDreamOutput('---\ntitle: foo\n---\nbody')).toBe(false);
expect(isDreamOutput('---\ndream_generated: false\n---\nbody')).toBe(false);
expect(isDreamOutput('plain text with no frontmatter')).toBe(false);
// dream_generatedfoo: true (no word boundary on the key) must NOT match
expect(isDreamOutput('---\ndream_generatedfoo: true\n---\nbody')).toBe(false);
});
test('marker buried past 2000 chars does NOT trigger guard (perf bound)', () => {
const padding = 'x'.repeat(2100);
const content = '---\ntitle: real\n---\n' + padding + '\ndream_generated: true\n' + 'x'.repeat(3000);
const path = makeTranscript('buried.txt', content);
const result = readSingleTranscript(path, { minChars: 100 });
expect(result).not.toBeNull();
});
test('bypassGuard=true overrides marker (--unsafe-bypass-dream-guard plumbing)', () => {
const md = '---\ndream_generated: true\n---\n# Page\n' + 'x'.repeat(3000);
const path = makeTranscript('marked.txt', md);
expect(readSingleTranscript(path, { minChars: 100 })).toBeNull();
expect(readSingleTranscript(path, { minChars: 100, bypassGuard: true })).not.toBeNull();
});
test('discoverTranscripts respects bypassGuard', () => {
const md = '---\ndream_generated: true\n---\n# Page\n' + 'x'.repeat(3000);
makeTranscript('2026-04-30-output.txt', md);
makeTranscript('2026-04-30-real.txt', 'real transcript ' + 'x'.repeat(3000));
const guarded = discoverTranscripts({ corpusDir: tmpDir, minChars: 100 });
expect(guarded).toHaveLength(1);
expect(guarded[0].basename).toBe('2026-04-30-real');
const bypassed = discoverTranscripts({ corpusDir: tmpDir, minChars: 100, bypassGuard: true });
expect(bypassed).toHaveLength(2);
});
test('DREAM_OUTPUT_MARKER_RE is anchored at file start (not mid-content)', () => {
// Frontmatter delimiter must be at byte 0; mid-content `---\n` does not count.
const content = 'preamble\n---\ndream_generated: true\n---\nbody' + 'x'.repeat(3000);
expect(DREAM_OUTPUT_MARKER_RE.test(content)).toBe(false);
});
});
describe('judgeSignificance', () => {
function makeTranscript(): import('../src/core/cycle/transcript-discovery.ts').DiscoveredTranscript {
return {
filePath: '/tmp/x.txt',
contentHash: 'abc123',
content: 'A short conversation about something interesting.',
basename: 'x',
inferredDate: null,
};
}
function mockClient(captured: { model?: string }): JudgeClient {
return {
create: async (p: any) => {
captured.model = p.model;
return { content: [{ type: 'text', text: '{"worth_processing": true, "reasons": ["test"]}' }] } as any;
},
};
}
test('passes verdict_model override to client.create', async () => {
const captured: { model?: string } = {};
await judgeSignificance(mockClient(captured), makeTranscript(), 'claude-sonnet-4-6');
expect(captured.model).toBe('claude-sonnet-4-6');
});
test('defaults to claude-haiku-4-5-20251001 when model omitted', async () => {
const captured: { model?: string } = {};
await judgeSignificance(mockClient(captured), makeTranscript());
expect(captured.model).toBe('claude-haiku-4-5-20251001');
});
test('returns worth_processing=false when judge returns unparseable text', async () => {
const client: JudgeClient = {
create: async () => ({ content: [{ type: 'text', text: 'no json here' }] } as any),
};
const r = await judgeSignificance(client, makeTranscript());
expect(r.worth_processing).toBe(false);
expect(r.reasons[0]).toContain('unparseable');
});
});
+61
View File
@@ -0,0 +1,61 @@
/**
* Structural tests for `gbrain dream` argv parsing (v0.21).
*
* Verifies the help text + parser source contains the new flags
* (--input, --date, --from, --to) and that conflict detection is wired.
* The actual parseArgs is internal; we exercise it via the source file
* structure to avoid spinning up a process per test.
*/
import { describe, test, expect } from 'bun:test';
import { readFileSync } from 'fs';
const dreamSrc = readFileSync(new URL('../src/commands/dream.ts', import.meta.url), 'utf-8');
describe('dream CLI flag wiring', () => {
test('declares --input flag with file argument', () => {
expect(dreamSrc).toContain("'--input'");
expect(dreamSrc).toContain('inputFile');
});
test('declares --date / --from / --to flags', () => {
expect(dreamSrc).toContain("'--date'");
expect(dreamSrc).toContain("'--from'");
expect(dreamSrc).toContain("'--to'");
});
test('validates ISO date format', () => {
expect(dreamSrc).toMatch(/ISO_DATE_RE/);
expect(dreamSrc).toContain('YYYY-MM-DD');
});
test('--input + --date conflict detection', () => {
expect(dreamSrc).toContain('--input cannot be combined with --date');
});
test('--input implies --phase synthesize', () => {
expect(dreamSrc).toContain("phase = 'synthesize'");
});
test('--from > --to range validation', () => {
expect(dreamSrc).toContain('empty range');
});
test('forwards synth fields to runCycle', () => {
expect(dreamSrc).toContain('synthInputFile');
expect(dreamSrc).toContain('synthDate');
expect(dreamSrc).toContain('synthFrom');
expect(dreamSrc).toContain('synthTo');
});
test('totals line includes synth + patterns counters', () => {
expect(dreamSrc).toContain('synth_transcripts');
expect(dreamSrc).toContain('synth_pages');
expect(dreamSrc).toContain('patterns=');
});
test('help text documents dry-run synthesis semantics (Codex finding #8)', () => {
expect(dreamSrc).toContain('skips the Sonnet');
expect(dreamSrc.toLowerCase()).toContain('zero llm calls');
});
});
+134
View File
@@ -0,0 +1,134 @@
/**
* gbrain claw-test scripted-mode E2E.
*
* Invokes the harness via `bun run src/cli.ts` (NOT a compiled binary
* `bun build --compile` doesn't bundle PGLite's runtime assets like
* pglite.data, so a compiled gbrain can't init a fresh PGLite brain).
* Uses a tiny shim script that the harness can spawn as if it were the
* gbrain binary.
*
* Asserts:
* - exit code 0 on a clean tree
* - the friction JSONL has zero error/blocker entries
* - the harness recorded progress events for the expected phases
*
* Tagged-skip env: CLAW_TEST_SKIP_E2E=1 to opt out (e.g. when PGLite
* WASM is broken on the host the macOS 26.3 #223 bug class).
*/
import { describe, test, expect, beforeAll } from 'bun:test';
import { execFileSync, spawnSync } from 'child_process';
import { mkdirSync, existsSync, mkdtempSync, rmSync, readFileSync, readdirSync, writeFileSync, chmodSync } from 'fs';
import { tmpdir } from 'os';
import { join, resolve } from 'path';
const REPO_ROOT = resolve(import.meta.dir, '..', '..');
const BIN_CACHE = join(REPO_ROOT, 'test', '.cache');
const BIN_PATH = join(BIN_CACHE, 'gbrain.sh');
const SCENARIOS_DIR = join(REPO_ROOT, 'test', 'fixtures', 'claw-test-scenarios');
beforeAll(() => {
if (!existsSync(BIN_CACHE)) mkdirSync(BIN_CACHE, { recursive: true });
// Shim that delegates to `bun run src/cli.ts` so PGLite assets resolve from
// the source tree (bun --compile doesn't bundle them). Marked executable so
// child_process.spawn can run it directly.
const shim = `#!/bin/sh\nexec bun run "${join(REPO_ROOT, 'src', 'cli.ts')}" "$@"\n`;
writeFileSync(BIN_PATH, shim, 'utf-8');
chmodSync(BIN_PATH, 0o755);
}, 30_000);
describe('gbrain claw-test --scenario fresh-install (scripted)', () => {
test('runs end-to-end clean and produces zero error/blocker friction', () => {
const tmp = mkdtempSync(join(tmpdir(), 'claw-test-e2e-fresh-'));
try {
const result = spawnSync(BIN_PATH, ['claw-test', '--scenario', 'fresh-install', '--keep-tempdir'], {
cwd: REPO_ROOT,
env: {
...process.env,
GBRAIN_HOME: tmp,
GBRAIN_BIN_OVERRIDE: BIN_PATH,
GBRAIN_CLAW_SCENARIOS_DIR: join(REPO_ROOT, 'test', 'fixtures', 'claw-test-scenarios'),
},
encoding: 'utf-8',
timeout: 120_000,
});
if (result.status !== 0) {
console.error('STDOUT:', result.stdout);
console.error('STDERR:', result.stderr);
}
expect(result.status).toBe(0);
// Inspect the friction JSONL the harness wrote.
const frictionDir = join(tmp, '.gbrain', 'friction');
expect(existsSync(frictionDir)).toBe(true);
const files = readdirSync(frictionDir).filter(f => f.endsWith('.jsonl'));
expect(files.length).toBeGreaterThan(0);
const runFile = join(frictionDir, files[0]);
const lines = readFileSync(runFile, 'utf-8').split('\n').filter(l => l.trim());
const entries = lines.map(l => JSON.parse(l));
const blockers = entries.filter(e => e.kind === 'friction' && (e.severity === 'error' || e.severity === 'blocker'));
if (blockers.length > 0) {
console.error('unexpected friction entries:', blockers);
}
expect(blockers.length).toBe(0);
} finally {
rmSync(tmp, { recursive: true, force: true });
}
}, 180_000);
test('break path: an invented command produces an error friction entry and exits non-zero', () => {
// We do this by setting GBRAIN_BIN_OVERRIDE to a script that pretends to be gbrain
// and rejects the `import` subcommand specifically.
const tmp = mkdtempSync(join(tmpdir(), 'claw-test-e2e-break-'));
const fakeBin = join(tmp, 'fake-gbrain');
try {
// Write a shim that delegates to real gbrain but rejects 'import' to simulate breakage.
const shimContent = `#!/bin/sh\nif [ "$1" = "import" ]; then echo "fake import error" >&2; exit 17; fi\nexec "${BIN_PATH}" "$@"\n`;
const { writeFileSync, chmodSync } = require('fs');
writeFileSync(fakeBin, shimContent, 'utf-8');
chmodSync(fakeBin, 0o755);
const result = spawnSync(BIN_PATH, ['claw-test', '--scenario', 'fresh-install', '--keep-tempdir'], {
cwd: REPO_ROOT,
env: {
...process.env,
GBRAIN_HOME: tmp,
GBRAIN_BIN_OVERRIDE: fakeBin,
GBRAIN_CLAW_SCENARIOS_DIR: join(REPO_ROOT, 'test', 'fixtures', 'claw-test-scenarios'),
},
encoding: 'utf-8',
timeout: 60_000,
});
expect(result.status).not.toBe(0);
// The friction log should have an error-severity entry for the 'import' phase.
const frictionDir = join(tmp, '.gbrain', 'friction');
const files = readdirSync(frictionDir).filter(f => f.endsWith('.jsonl'));
const lines = readFileSync(join(frictionDir, files[0]), 'utf-8').split('\n').filter(l => l.trim());
const entries = lines.map(l => JSON.parse(l));
const importErrors = entries.filter(e => e.phase === 'import' && e.severity === 'error');
expect(importErrors.length).toBeGreaterThan(0);
} finally {
rmSync(tmp, { recursive: true, force: true });
}
}, 90_000);
});
describe('gbrain friction render integration', () => {
test('render produces a markdown report with the redact placeholder', () => {
const tmp = mkdtempSync(join(tmpdir(), 'claw-test-e2e-render-'));
try {
// Log a friction entry with $HOME embedded, then render --redact md
const home = process.env.HOME ?? '/tmp';
const env = { ...process.env, GBRAIN_HOME: tmp, GBRAIN_FRICTION_RUN_ID: 'render-e2e' };
execFileSync(BIN_PATH, ['friction', 'log', '--phase', 'p', '--message', `error at ${home}/.gbrain/x`], { env, encoding: 'utf-8' });
const out = execFileSync(BIN_PATH, ['friction', 'render', '--run-id', 'render-e2e'], { env, encoding: 'utf-8' });
expect(out).toContain('# Friction report');
expect(out).toContain('<HOME>');
// --redact is the default for md, so home itself should not appear.
expect(out).not.toContain(home + '/.gbrain');
} finally {
rmSync(tmp, { recursive: true, force: true });
}
}, 30_000);
});
+2 -2
View File
@@ -97,8 +97,8 @@ describeE2E('E2E: runCycle against real Postgres', () => {
});
expect(report.schema_version).toBe('1');
// Cycle ran all 6 phases (or skipped the ones that don't support dry-run).
expect(report.phases.length).toBe(6);
// Cycle ran all 8 phases (or skipped the ones that don't support dry-run).
expect(report.phases.length).toBe(8);
// Nothing got written.
const afterPages = await conn.unsafe(`SELECT count(*)::int AS n FROM pages`);
+159
View File
@@ -0,0 +1,159 @@
/**
* E2E security regression: poisoned-transcript guard for the v0.21
* trusted-workspace allow-list.
*
* Runs against PGLite in-memory (no DATABASE_URL required). Builds the
* brain tool registry with `allowed_slug_prefixes` set the same way the
* synthesize phase does, then calls the put_page tool with slugs that
* are inside / outside the allow-list. Asserts:
*
* - In-allow-list slug page is written to the DB
* - Outside-allow-list slug tool throws permission_denied
* - When allow-list is unset (legacy), put_page is bounded to
* wiki/agents/<id>/... (regression guard for the v0.15 anti-prompt-
* injection guarantee)
*/
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
import { buildBrainTools } from '../../src/core/minions/tools/brain-allowlist.ts';
import type { GBrainConfig } from '../../src/core/config.ts';
let engine: PGLiteEngine;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({ engine: 'pglite' } as never);
await engine.initSchema();
});
afterAll(async () => {
if (engine) await engine.disconnect();
});
const config = {} as unknown as GBrainConfig;
const PUT_PAGE_TOOL = 'brain_put_page';
const SAMPLE_BODY = '---\ntitle: A reflection\ntype: default\n---\n\nbody text\n';
function findPutPageTool(tools: Awaited<ReturnType<typeof buildBrainTools>>) {
const t = tools.find(x => x.name === PUT_PAGE_TOOL);
if (!t) throw new Error('brain_put_page tool not found in registry');
return t;
}
describe('E2E allow-list — trusted-workspace path', () => {
test('ALLOW: subagent put_page within allow-list writes the page', async () => {
const tools = buildBrainTools({
subagentId: 999,
engine,
config,
allowedSlugPrefixes: ['wiki/personal/reflections/*'],
});
const tool = findPutPageTool(tools);
await tool.execute(
{ slug: 'wiki/personal/reflections/2026-04-25-arete-paradox-a3f8c1', content: SAMPLE_BODY },
{ engine, jobId: 7777, remote: true },
);
const page = await engine.getPage('wiki/personal/reflections/2026-04-25-arete-paradox-a3f8c1');
expect(page).not.toBeNull();
expect(page!.title).toBe('A reflection');
});
test('REJECT: subagent put_page outside allow-list throws permission_denied', async () => {
const tools = buildBrainTools({
subagentId: 999,
engine,
config,
allowedSlugPrefixes: ['wiki/personal/reflections/*'],
});
const tool = findPutPageTool(tools);
let threw = false;
try {
await tool.execute(
{ slug: 'wiki/finance/secret-market-data', content: SAMPLE_BODY },
{ engine, jobId: 7778, remote: true },
);
} catch (e) {
threw = true;
const msg = e instanceof Error ? e.message : String(e);
expect(msg).toMatch(/allow-list/i);
}
expect(threw).toBe(true);
const page = await engine.getPage('wiki/finance/secret-market-data');
expect(page).toBeNull(); // never reached the engine
});
test('Multiple prefixes: each slug evaluated independently', async () => {
const tools = buildBrainTools({
subagentId: 999,
engine,
config,
allowedSlugPrefixes: ['wiki/personal/reflections/*', 'wiki/originals/*'],
});
const tool = findPutPageTool(tools);
await tool.execute(
{ slug: 'wiki/originals/ideas/2026-04-25-thousand-pound-armor', content: SAMPLE_BODY },
{ engine, jobId: 7779, remote: true },
);
expect(await engine.getPage('wiki/originals/ideas/2026-04-25-thousand-pound-armor')).not.toBeNull();
});
});
describe('E2E allow-list — legacy namespace fallback', () => {
test('REGRESSION GUARD: when allow-list is unset, put_page rejects writes outside wiki/agents/<id>/', async () => {
const tools = buildBrainTools({
subagentId: 999,
engine,
config,
// allowedSlugPrefixes intentionally omitted — exercises the v0.15
// legacy namespace check that v0.21 must NOT regress.
});
const tool = findPutPageTool(tools);
let threw = false;
try {
await tool.execute(
{ slug: 'wiki/personal/reflections/2026-04-25-bypass-attempt', content: SAMPLE_BODY },
{ engine, jobId: 7780, remote: true },
);
} catch (e) {
threw = true;
const msg = e instanceof Error ? e.message : String(e);
expect(msg).toMatch(/wiki\/agents\/999/);
}
expect(threw).toBe(true);
});
test('When allow-list unset, slug under wiki/agents/<id>/ is allowed', async () => {
const tools = buildBrainTools({
subagentId: 999,
engine,
config,
});
const tool = findPutPageTool(tools);
await tool.execute(
{ slug: 'wiki/agents/999/scratch-note', content: SAMPLE_BODY },
{ engine, jobId: 7781, remote: true },
);
expect(await engine.getPage('wiki/agents/999/scratch-note')).not.toBeNull();
});
});
describe('E2E allow-list — provenance via tool execution rows (Codex #2)', () => {
test('subagent_tool_executions captures slug for each put_page call', async () => {
// The synthesize phase relies on this being queryable to determine
// exactly which slugs each child wrote (instead of pages.updated_at).
// We don't have a real subagent run here, but we can verify the table
// exists and the column shape supports the orchestrator's query.
const rows = await engine.executeRaw(
`SELECT column_name FROM information_schema.columns
WHERE table_name = 'subagent_tool_executions'
ORDER BY column_name`,
) as Array<{ column_name: string }>;
const cols = rows.map(r => r.column_name);
expect(cols).toContain('input');
expect(cols).toContain('tool_name');
expect(cols).toContain('status');
expect(cols).toContain('job_id');
});
});
@@ -0,0 +1,196 @@
/**
* E2E full 8-phase cycle on PGLite, no API key required.
*
* Verifies that the v0.23 phase order lint backlinks sync
* synthesize extract patterns embed orphans is honored
* end-to-end through runCycle when no API key is present (synthesize
* + patterns skip cleanly, the other six phases run unchanged).
*
* Two regression-relevant invariants:
* 1. CycleReport.phases preserves the 8-phase order no future
* reorder regresses without breaking this test.
* 2. CycleReport.totals carries the new v0.23 fields:
* transcripts_processed, synth_pages_written, patterns_written.
*
* No DATABASE_URL required. Mocks embedBatch so the embed phase doesn't
* attempt OpenAI calls.
*
* Run: bun test test/e2e/dream-cycle-eight-phase-pglite.test.ts
*/
import { describe, test, expect, mock } from 'bun:test';
import { mkdtempSync, writeFileSync, rmSync, mkdirSync } from 'fs';
import { join } from 'path';
import { execSync } from 'child_process';
import { tmpdir } from 'os';
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
mock.module('../../src/core/embedding.ts', () => ({
embed: async () => new Float32Array(1536),
embedBatch: async (texts: string[]) => texts.map(() => new Float32Array(1536)),
EMBEDDING_MODEL: 'text-embedding-3-large',
EMBEDDING_DIMENSIONS: 1536,
EMBEDDING_COST_PER_1K_TOKENS: 0.00013,
estimateEmbeddingCostUsd: (tokens: number) => (tokens / 1000) * 0.00013,
}));
const { runCycle, ALL_PHASES } = await import('../../src/core/cycle.ts');
interface TestRig {
engine: PGLiteEngine;
brainDir: string;
cleanup: () => Promise<void>;
}
async function setupRig(): Promise<TestRig> {
const engine = new PGLiteEngine();
await engine.connect({ engine: 'pglite' } as never);
await engine.initSchema();
const brainDir = mkdtempSync(join(tmpdir(), 'gbrain-cycle8-'));
execSync('git init', { cwd: brainDir, stdio: 'pipe' });
execSync('git config user.email test@test.co', { cwd: brainDir, stdio: 'pipe' });
execSync('git config user.name test', { cwd: brainDir, stdio: 'pipe' });
mkdirSync(join(brainDir, 'concepts'), { recursive: true });
writeFileSync(
join(brainDir, 'concepts/testing.md'),
'---\ntype: concept\ntitle: Testing\n---\n\nTest body content.\n',
);
execSync('git add -A && git commit -m init', { cwd: brainDir, stdio: 'pipe' });
await engine.setConfig('sync.repo_path', brainDir);
return {
engine,
brainDir,
cleanup: async () => {
try { await engine.disconnect(); } catch { /* */ }
try { rmSync(brainDir, { recursive: true, force: true }); } catch { /* */ }
},
};
}
async function withoutAnthropicKey<T>(body: () => Promise<T>): Promise<T> {
const saved = process.env.ANTHROPIC_API_KEY;
delete process.env.ANTHROPIC_API_KEY;
try {
return await body();
} finally {
if (saved === undefined) delete process.env.ANTHROPIC_API_KEY;
else process.env.ANTHROPIC_API_KEY = saved;
}
}
describe('E2E v0.23 8-phase cycle', () => {
test('ALL_PHASES is the 8-phase order in the documented sequence', () => {
expect(ALL_PHASES).toEqual([
'lint',
'backlinks',
'sync',
'synthesize',
'extract',
'patterns',
'embed',
'orphans',
]);
});
test('full cycle on dry-run returns CycleReport.phases in v0.23 order with new totals fields', async () => {
const rig = await setupRig();
try {
await withoutAnthropicKey(async () => {
const report = await runCycle(rig.engine, {
brainDir: rig.brainDir,
dryRun: true,
});
// Phase ordering preserved
const phaseNames = report.phases.map(p => p.phase);
expect(phaseNames).toEqual([
'lint',
'backlinks',
'sync',
'synthesize',
'extract',
'patterns',
'embed',
'orphans',
]);
// New totals fields exist (v0.23 additive growth)
expect(report.totals).toMatchObject({
transcripts_processed: 0,
synth_pages_written: 0,
patterns_written: 0,
});
// Synthesize and patterns are skipped (not_configured / insufficient_evidence)
const synth = report.phases.find(p => p.phase === 'synthesize');
const patterns = report.phases.find(p => p.phase === 'patterns');
expect(synth?.status).toBe('skipped');
expect(patterns?.status).toBe('skipped');
});
} finally {
await rig.cleanup();
}
});
test('--phase synthesize alone runs only that phase, returns skipped/not_configured', async () => {
const rig = await setupRig();
try {
await withoutAnthropicKey(async () => {
const report = await runCycle(rig.engine, {
brainDir: rig.brainDir,
dryRun: false,
phases: ['synthesize'],
});
expect(report.phases).toHaveLength(1);
expect(report.phases[0].phase).toBe('synthesize');
expect(report.phases[0].status).toBe('skipped');
});
} finally {
await rig.cleanup();
}
});
test('--phase patterns alone runs only that phase, returns skipped/insufficient_evidence', async () => {
const rig = await setupRig();
try {
await withoutAnthropicKey(async () => {
const report = await runCycle(rig.engine, {
brainDir: rig.brainDir,
dryRun: false,
phases: ['patterns'],
});
expect(report.phases).toHaveLength(1);
expect(report.phases[0].phase).toBe('patterns');
expect(report.phases[0].status).toBe('skipped');
expect((report.phases[0].details as { reason?: string }).reason).toBe('insufficient_evidence');
});
} finally {
await rig.cleanup();
}
});
test('synthInputFile flag is plumbed through runCycle to runPhaseSynthesize', async () => {
const rig = await setupRig();
try {
const transcript = join(tmpdir(), `gbrain-e2e-cycle8-input-${Date.now()}.txt`);
writeFileSync(transcript, 'sample conversation '.repeat(300));
try {
await withoutAnthropicKey(async () => {
const report = await runCycle(rig.engine, {
brainDir: rig.brainDir,
dryRun: false,
phases: ['synthesize'],
synthInputFile: transcript,
});
// Without API key, synthesize falls through to no-key skip-path
// and returns ok (NOT cooldown_active — explicit input bypasses).
expect(report.phases[0].phase).toBe('synthesize');
expect(report.phases[0].status).toBe('ok');
});
} finally {
rmSync(transcript, { force: true });
}
} finally {
await rig.cleanup();
}
});
});

Some files were not shown because too many files have changed in this diff Show More