Files
gbrain/test/sources-push-cli.serial.test.ts
T
Garry TanandClaude Fable 5 d35c9c9e44 v0.45.0.0 feat(bootstrap): paste-in personal-agent install for Codex + Claude Code (#3975)
* docs(designs): agent-bootstrap plan + design docs (normative, review-absorbed)

The scrubbed, in-repo sources of truth for the gbrain bootstrap wave:
AGENT_BOOTSTRAP_DESIGN.md (product scope/sequencing) and
AGENT_BOOTSTRAP_PLAN.md (implementation; all review-finding IDs inlined).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(bootstrap): format spec, question bank, identity templates, bundled assets

agent.json manifest (format_version 1, initialized sentinel) + machine-local
install receipt [CX2-1, CX2-12]; 12-question/6-required interview bank with
consent keys and a persist:false sink for the optional provider key [CX2-13];
ten {{TOKEN}} identity templates (generic, adapted to gbrain ops — gates call
recall/query/put_page, write-through-ops rule, keyless agent-authored facts,
silence contract); assets embedded compiled-binary-safe via file-type imports
[ENG-6].

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(bootstrap): runbook, README paste block, bootstrap guide, TODOS entries

BOOTSTRAP_FOR_AGENTS.md (agent-driven install runbook: CLI phase list is the
source of truth, never-invent rules, Codex approvals preflight, keyless posture,
failure-modes table, version stamp for the skew check); README gains the
full-agent paste block pinned to latest-stable inside the Claude Code/Codex
quick start (memory-only tier stays); docs/guides/bootstrap.md carries the full
install/security/consent/degradation/uninstall contract.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(designs): spike instrument for the bootstrap wave (build order 0)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(bootstrap): interview + render engines

Interview gate with read-back confirm-hash (any later answer change clears the
confirmation — the hostile single-batch case is structurally impossible),
per-answer provenance, caps + escaping at set time, config-sink routing for the
provider key; renderer with hard-fail token sweep, subordinate fencing of
principal input, never-clobber + backups, deterministic minimal mode for the
template repo, scaled byte floors. 58 unit tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(bootstrap): private-repo lifecycle — repo create, attach, uninstall, run lock

gh-gated private repo creation with API-verified privacy (rate-limit distinct
from public), refuse-foreign-origin with attach as the sanctioned path, atomic
bootstrap mutex (pid liveness + age + token), receipt-keyed uninstall that
never wholesale-deletes the gbrain home and only offers --delete-brain for a
brain it created; read-only PGLite lock probe (never opens the engine).
54 unit tests, injectable exec seam throughout.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(release): latest-stable ref, template-repo publish job, bootstrap CI guards

release.yml advances the latest-stable tag only after assets publish (the paste
block's permanent ref — copies in the wild never rot) and gains a PAT-gated
publish-template job verified against the vendored tree; two skip-graceful
guards (sanctioned-ref + runbook stamp; template/token bijection + placeholder
assertion + generator byte-diff) wired into verify; README + runbook re-admitted
to the CI cache hash; vendored deterministic template tree generated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(context): IPC v2 turn_context + 8KB assembly + visibility resolver + session identity

Discriminated-union IPC with handler map, protocol echo (stale-serve detection),
shared-secret gate, server-side source binding, per-kind budgets; turn-context
assembly (reflex pointers + volunteered pages + world-only hot facts) under a
data-not-instructions envelope trimmed to the harness's 10KB hook-output cap;
facts.default_visibility resolved through one helper at all four sites (explicit
caller wins, typos fail closed); typed sessionId threads _meta.session_id into
the hot-memory cache key. 50 new tests; 180 adjacent tests confirmed green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(persistence): secret-scan, gbrain sources push, durability unification

Pattern secret scanner (own runtime allowlist, redacted previews, corpus-write
redaction mode); sources push runs the whole scan→stage→commit→pull→push
sequence under one cross-platform lock (mkdir-atomic, pid+age+token) with a
deny-glob backstop, commit-first divergence-safe pull, refuse-unverifiable
visibility, and push-status telemetry; gbrain-home choke point unifies
GBRAIN_HOME semantics with config (0700); durability is parent-repo-aware and
rotates its push log at 0600. 35 new tests; 200 existing green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(sources): harden/pull gates accept sources inside a parent git repo

The bootstrap workspace registers brain/ (a subdirectory) as the source; the
durability core already resolves the repo root, so the command gates now check
inside-a-repo rather than .git-right-here [CX2-3].

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(serve): resident maintenance sweep + keyless capability probe

The lock-owning serve process now closes the persistence loop: startup (3s
post-connect, best-effort, unref'd) and idle (10-min quiet intervals through
the injectable timer seam) sweeps run facts-fence reconciliation, deterministic
link/timeline extraction over recent workspace pages, and spend-gated corpus
ingest (skipped keyless — agent-authored fences cover it). gbrain sweep --once
is the trusted CLI seam bootstrap verify uses. Capability probe renders the
honest keyless/keyed report. Full reuse of the cycle extractor + extract cores;
26 new tests, neighbors green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(hooks): engine-free gbrain hook command, settings writers, transcript parser

Four hook events (session-start digest + crashed-session recovery push,
user-prompt turn-context injection under an 800ms deadline and the 10KB cap,
stop buffers, session-end corpus write with redaction/retention/dedup +
best-effort push); structural JSON settings merger keyed by a _gbrain marker
(foreign hooks and permissions survive); dated host-spec registry; Claude Code
.jsonl parser as a spec-target with a scrubbed 7-shape fixture. Heartbeat is
counters-only by construction. 59 tests; zero engine modules in the import
graph.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(bootstrap): cross-link the full-agent path from the connection docs

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(bootstrap): dispatcher, verify, status — the command assembled

gbrain bootstrap {status,interview,render,repo,hooks,verify,uninstall,attach}:
engine-free except verify (owns its engine, in-process sweep — no live-serve
conflict); phase list is the TS source of truth with install.jsonl telemetry
and the support blob; verify's fail-soft check suite covers the real write path
(put_page → write-through file → sweep → graph floor → recall), passes keyless,
persists snapshots, and ends with the first-run tour. cli.ts wired per the
three-touchpoint rule; doctor gains the bootstrap check group (silent on
machines with no bootstrap state). 28 new tests; 353 adjacent green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: KEY_FILES bootstrap cluster + CLAUDE.md dispatcher row (+ build:llms)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(bootstrap): e2e pins — hook-under-live-serve, attach, degraded modes, compiled binary, Docker harness

The permanent pins: a real serve holds the PGLite lock while the engine-free
hook completes (and a direct engine open provably throws LiveServeLockError);
stale-socket fail-open; machine-2 attach with marker-keyed hook repair;
decline-everything installs verify green with every degradation named; the
compiled binary renders bundled templates in an empty cwd. Offline Docker
harness (networkless, read-only) gated into heavy-tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(bootstrap): register doctor check categories + system-of-record allow comments

The six bootstrap doctor checks join OPS_CHECK_NAMES; the sweep's batch link/
timeline inserts carry the explicit extract-path allow comments (the sweep IS
the extraction path for workspace pages).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(test): shard wedge cap tracks suite growth (1500s -> 1800s)

At ~9000 tests a healthy shard finished at 1466s and two progressing shards
were false-killed at the old cap; 1800s restores ~25% headroom over the
slowest observed healthy shard. Real hangs still hit it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(ci): cache-hash policy — README + runbook edits must invalidate [C2]

The old deny-list assertion predates the paste block; README.md and
BOOTSTRAP_FOR_AGENTS.md are policy-doc re-admissions now, so their edits must
change the hash (a paste-block edit shipping under a cached green was the C2
hole).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(test): classify post-suite exit-hangs as warn-pass; file the leak forensics

A shard killed by the wedge watchdog with every assigned file started and zero
fail markers did all its work and leaked a handle at exit — pre-existing and
master-reproducible (P1 TODO carries the full bisect forensics). Bun's per-test
timeout turns a hung test into a (fail), so the classifier cannot mask one.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(test): shard cap 2400s — the count-balanced heavy shard needs it under contention

Observed: the heavy shard still progressing 22s before an 1800s kill while
siblings finish at 1150-1550s (split balances file count, not weight). Filed
the load-sensitive WAL-repair flake (pre-existing, master's v0.42.75.0 wave).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(bootstrap): quarantine env-mutating suites to the serial lane

check-test-isolation R1: six new files mutate GBRAIN_HOME/env at module scope —
the serial lane (one process per file) is the guard's prescribed home for them.
All 114 tests pass post-rename.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(readme): per-harness install sections — Codex, Claude Code, then OpenClaw/Hermes

Each harness gets its own complete paste-block section (desktop app first,
terminal noted — Claude Code CLI is the identical harness; Codex CLI works
pull-based today); the OpenClaw/Hermes platform path keeps equal weight with
its one-click deploys and INSTALL_FOR_AGENTS block intact; memory-only and
remote-connect tiers consolidated under 'Lighter ways in'. Supersedes the
review's D5 ordering by user direction; stale heading references updated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(readme): Codex as the recommended first step; OpenClaw/Hermes framed as-intended, high-cost

The install section now routes newcomers explicitly: Codex first
(subscription-priced, nothing to deploy), OpenClaw/Hermes as GBrain used the
way it was designed — always on, at real server + API cost.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: doc-audit code chasers — broken recovery hints, stale op description, auto_link key

Four small code fixes surfaced by the markdown accuracy audit:
- doctor's auto-RLS recovery hint pointed at `apply-migrations --force-retry 35`,
  which cannot work (--force-retry targets the vX.Y.Z orchestrator registry, not
  the numeric schema MIGRATIONS array). Hint now points at the recreate SQL in
  docs/guides/rls-and-you.md; test pins against regression.
- v0_11_0 migration printed the same broken-mechanism class of hint
  (`config set minion_mode` writes DB config nothing reads); now names
  `apply-migrations --mode` + preferences.json, the real setter.
- submit_job's op description hardcoded a stale handler list; now points at
  registerBuiltinHandlers as the source plus the --follow discovery trick.
- `auto_link` added to KNOWN_CONFIG_KEYS: read by link-extraction, reconcile-links,
  and sweep, and documented as the off-switch in brain-ops/maintain, but the
  allowlist rejected `config set auto_link false`.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: repo-wide accuracy + MECE reform from the 9-bucket markdown audit

A code-grounded audit of every markdown file (root, architecture, guides,
mcp, tutorials, docs-root, operations/eval/designs, skills, recipes) followed
by a fix wave with per-bucket ownership. Four classes of change:

Accuracy — every documented command/flag verified against src/ before writing:
dead commands replaced with working ones (pages purge-deleted, jobs watch
--follow, gbrain restore, import-based Obsidian flow, space-separated --scopes,
real thin-client recipes, working isolation verification, real supervisor
restart procedure, curl-based ngrok health check, real minion_mode setter);
count drift fixed with rot-proof phrasing (100+ ops, 50+ bundled skills via
skills/manifest.json, 140+ engine methods, KNOBS_HASH_VERSION pointer instead
of hardcoded versions); stale claims corrected (search-mode defaults, RETRIEVAL
pipeline order incl. autocut, sentinel rules, refusal-list mechanism, engine
snapshot, shard cap 2400s + EXIT-HANG classifier in TESTING.md, latest-stable +
publish-template documented in RELEASING.md as release.yml promises).

MECE — one home per concept, pointers elsewhere: test isolation → TESTING.md;
OAuth registration + --bind/--public-url lore → DEPLOY.md; mode bundles →
guides/search-modes.md (the home the CLAUDE.md dispatcher always promised);
merge contract → schema-packs.md; WAL ladder → ENGINES.md; quiet-hours →
quiet-hours.md; capture taxonomy → entity-detection.md; person-page taxonomy →
compiled-truth.md; brain-first protocol → brain-first-lookup.md; refresh
semantics → refresh-algorithm.md; KEY_FILES.md deduplicated (58 extension
entries merged, one entry per file); infra-layer.md rewritten as a pointer page.

Privacy — placeholder sweep across guides, docs, skills, and recipes per the
iron rule; per-release narration stripped from reference docs (current-state
prose only).

Bootstrap coverage — AGENTS.md pointer, RESOLVER routing row, INSTALL.md path,
tutorial cross-links, keyless-mode sections in spend-controls/headless-install.

skills.lock.json regenerated; llms.txt/llms-full.txt rebuilt. Gates: verify
36/36, typecheck clean, doctor 96/96, skills-integrity + resolver + build-llms
+ config-set + migrations all green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(readme): refresh production-brain stats to current brain-repo counts

155,795 pages / 24,589 people / 5,340 companies, counted from the brain
repo's current HEAD; the "100K-page brain" framing moves to 150K to match.
Cron-fleet count unchanged (its store lives on the deployment host, not in
the repos available for verification).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(scan,push): modern OpenAI/Voyage key patterns, scan staged blobs not disk

- secret-scan matches sk-proj-/sk-svcacct-/sk-None- and pa- Voyage keys
  (the bare sk- pattern missed every current OpenAI key format).
- workspacePush stages first, then scans the staged index blobs via
  git cat-file, closing the scan-then-stage TOCTOU where a file changed
  between snapshot and commit shipped unscanned.
- shared binary-sniff helper, memoized glob regexes, atomic push-status
  write, and tests for pull_conflict + gitignored deny-match paths.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ci): regenerate flag registry for new commands, harden shard classifier + release token

- cli-flag-registry.generated.ts regenerated: bootstrap/hook/sweep and
  sources push --message/--allow-unverified-remote were missing, so the
  strict #2185 validator rejected real invocations and skipped the new
  commands entirely.
- EXIT-HANG shard classifier now requires every assigned file to have
  started before warn-passing a watchdog kill (was fail-open).
- release.yml passes TEMPLATE_REPO_PAT via http.extraheader, off the argv.
- compiled-binary e2e fails loud in CI instead of a silent permanent skip.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(hook,sweep,ipc): non-blocking hook pushes, bounded sweep + cache, source-bound resolve

- session-start/session-end no longer run synchronous git + inline push
  inside their self-deadline; a detached child does the push and the hook
  returns immediately (blocked Claude Code startup for minutes on a dirty
  tree before).
- serve sweep drops the unbounded listAllPageRefs, resolves only candidate
  targets, claims corpus files atomically (no double-LLM-spend race), and
  caps the fence LIKE scan; heartbeat writes are O_APPEND with rare compaction.
- hot-memory cache evicts expired entries and bounds entry count (the key is
  caller-controlled via _meta.session_id).
- v1 resolve IPC honors boundSourceId like turn_context; turn-context runs
  its arms concurrently. doctor reads push/heartbeat thresholds from hook.ts.
- new tests: doctor bootstrap checks, hook push-gate + deadline, concurrent
  sweep claims, cache eviction, bound-source resolve.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(bootstrap): origin-ownership gate, world visibility, collision-safe source id, consent + templates

- repo adoption requires an exact receipt repo_url match or authed-owner
  check (undefined repo_url was a wildcard); create verifies privacy BEFORE
  the first push.
- verify sets facts.default_visibility=world if unset, so agent-authored
  facts surface in per-turn context (they defaulted private before).
- source_id derives a path-hash suffix when 'workspace' is taken by another
  checkout; every consumer reads manifest.source_id.
- skipped HOOKS_CONSENT now declines (was falling through to default yes);
  --minimal refuses on an initialized manifest; tilde fences escaped.
- MCP registration pins --surface full; status hard-fails a public origin
  (template door); receipt writers guard against newer/corrupt receipts;
  uninstall only claims brain-deleted after a real rm.
- templates ship jobs disabled + provider-consent + support-relay lines;
  soul-audit re-runs over the shared interview bank. TODOS: 11 follow-ups.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(security): close adversarial-review findings — scan fails closed, whole-PEM redaction, bound repo push

Cross-model adversarial pass (Claude + Codex) on the bootstrap wave:

- secret scan fails CLOSED: an unreadable, oversized, or binary staged blob
  now blocks the push (blocked_unscannable, exit 5) instead of committing
  unscanned; only a confirmed staged deletion is skipped. This was the
  headline "block secrets before they leave the machine" property failing open.
- private-key redaction spans the whole PEM block (header+body+footer), not
  just the header line — the base64 body no longer survives into the corpus
  the sweep sends to an extraction provider.
- bootstrap repo commits the workspace (secret-scan-gated) before the first
  push and verifies the remote actually received it, so a push-fail retry
  can't adopt an empty remote as success.
- privacy verify is re-bound to origin immediately before push (a concurrent
  origin rewrite between verify and push is refused).
- session-end corpus write is atomic and clears the stale ingested/in-progress
  sidecars so a resumed session's appended transcript is re-ingested.
- public-origin refusal enforced at render (not only status); MCP "already
  registered" is verified to target this workspace, not blessed blindly;
  verify probe cleanup scopes deletes to its own slugs, not a token substring;
  allowlist fingerprint floor raised 8→16 hex.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* v0.45.0.0 feat(bootstrap): paste-in personal-agent install for Codex + Claude Code

Turns a Codex or Claude Code session into a persistent personal agent:
interview-rendered identity files, a local PGLite brain, per-turn context via
serve IPC (Claude Code hooks / Codex pull protocol), session-triggered
persistence, and a private GitHub repo as the agent's portable body. Keyless-
first (the harness model is the LLM; one optional key adds embeddings +
extraction). New `gbrain bootstrap` command family + `gbrain hook` + `gbrain
sweep`; doctor bootstrap health checks; latest-stable distribution ref +
template-repo publish job. Opt-in, additive — existing installs untouched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: regenerate flag registry for security-fix flags; drop fabricated gbrain capabilities doc ref

CI caught two real failures under the merged state:
- the flag registry lagged the blocked_unscannable/exit-5 flags the security
  round added, tripping the #2185 freshness guard.
- headless-install.md described the keyless capability report as a
  `gbrain capabilities` command, which the #3502 doc-command resolver
  rejects — reworded to prose (the real surface is bootstrap verify's report).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: sync KEY_FILES + bootstrap plan to security-fix behavior

Cross-referenced the security-fix round against the reference docs and
corrected the drift those commits introduced:

- workspace-push.ts entry: stage-FIRST-then-scan order (the TOCTOU fix),
  fail-closed blocked_unscannable, and the sources-push status -> exit-code map.
- hooks.ts entry: MCP registration pins `serve --surface full`.
- hook.ts entry: session-start/session-end pushes run in a detached child
  (non-blocking); atomic corpus write clears stale sidecars.
- bootstrap.ts entry: render hard-refuses a public origin (template door).
- verify.ts entry: source_id collision resolution (workspace-<path-hash>).
- AGENT_BOOTSTRAP_PLAN as-shipped delta note for the scan/stage reorder.

llms bundle unchanged (KEY_FILES is link-only); build:llms and
test/build-llms.test.ts green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ci): silence SC2016 on the intentional askpass literal in release.yml

The one-shot GIT_ASKPASS script must contain literal $1 and
$TEMPLATE_REPO_PAT so they expand when /bin/sh runs it at git's credential
prompt, not when the outer shell writes the file — single quotes are correct.
Add a scoped shellcheck disable so actionlint passes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(skills): declare 'bootstrap my data' trigger in cold-start frontmatter

The doc reform added 'bootstrap my data' to cold-start's RESOLVER.md row (to
disambiguate data-bootstrap from agent-bootstrap) but not to the skill's own
frontmatter triggers, tripping the RESOLVER↔frontmatter round-trip contract
(resolver.test.ts). Declare it; regenerate skills.lock.json.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(bootstrap): default per-turn hooks + search mode ON without a prompt

Installing gbrain for your coding agent IS the consent for the behaviors that
make it work, so stop re-litigating them with install-time questions whose
"no" defeats the product:

- Per-turn hooks (Claude Code) install ON by default — no prompt. Off-ramps:
  `--no-hooks` at install, `GBRAIN_HOOKS=0` at runtime, `bootstrap uninstall`.
  The "hooks installed" line now surfaces the kill switch so default-on is
  never silent. A persisted HOOKS_CONSENT=no (interview --skip) still declines.
- Search mode defaults to `balanced` silently (nobody knows the modes at
  install; `gbrain search modes` changes it any time).
- MCP scope stays the ONE deliberate prompt — project vs user is a real
  cross-repo privacy choice, not friction.

Marks the two consents `silent: true` in the question bank (new QuestionSpec
field), rewrites the runbook phases so the agent no longer asks them, adds the
`--no-hooks` flag (+ registry regen), and adds default-on / opt-out tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(bootstrap): real end-to-end coverage — cross-session recall, per-turn content, Codex door, realistic corpus

Closes the seven e2e gaps a coverage audit surfaced: the plumbing was
well-unit-tested but the product claims ("Codex works, context shows up every
turn with real content, it remembers across restarts, machine two recovers,
Postgres works") were unproven end to end. Test-only wave — zero src changes.

- Hermetic synthetic corpus (test/fixtures/bootstrap-corpus/ + a loader helper):
  12 interlinked pages (52 edges, timelines), 12 world/private beliefs, 8 gold
  queries — curated from the gbrain-evals synthetic corpora, 100% placeholder
  names, so recall is asserted on a real multi-entity brain instead of a
  2-node self-planted probe.
- GAP1 magic moment: author a fact via the real write path, disconnect the
  engine, reopen against the same DB, recall it — a real session boundary, not
  verify.ts's same-connection SQL read-back. Plus a source-isolation assertion.
- GAP2 per-turn content: hook-under-serve Pin 1 now seeds a known fact and
  asserts its text lands in the injected block AND private beliefs never do
  (was: empty brain, empty_block accepted as a pass).
- GAP3 Codex door: assert the rendered AGENTS.md carries the Gate-3 brain-first
  pull protocol; make the fake codex shim implement `mcp get` so the [FIX7]
  target-verification can actually fail; the Docker cold-machine harness now
  exercises the hooks/MCP registration step instead of skipping it.
- GAP4 corpus recall: turn-context + verify graph-floor/qrels run on the real
  multi-entity brain with real edges.
- GAP5 attach: machine-two now re-ingests the cloned brain/ into a fresh DB and
  recalls a fact authored only on machine one — the multi-device payoff.
- GAP6 keyed + Postgres (env-gated): real embeddings prove semantic recall a
  paraphrase query can reach but keyless BM25 cannot; bootstrap verify drives a
  real Postgres engine (skipIf DATABASE_URL/keys absent).
- GAP7 persistence: session-end runs the REAL push (not the mocked seam) to a
  local bare remote and the remote receives the content; a planted secret is
  blocked at the gate; the 15-min cron installs and fires a scan-gated push.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(bootstrap): real-agent e2e — drive the actual claude + codex binaries end to end

Closes the audit's biggest gap ("no real harness ever drives a turn"). Adapts
gstack's PTY/headless agent harness to prove the bootstrap install + smoke work
against the REAL binaries, not PATH shims. Test/CI/docs only — zero src changes.

- test/helpers/agent-harness.ts: hermetic clean-room child env (ported from
  gstack; drops CONDUCTOR_/CLAUDE_/GSTACK_/MCP_/GBRAIN_, promotes
  GSTACK_ANTHROPIC_API_KEY→ANTHROPIC_API_KEY), real-binary resolvers + auth
  probes, headless `claude -p --output-format stream-json` and `codex exec
  --json` turn runners, a gbrain stdio MCP-config writer, and a keyless brain
  seeder. + a fixture-parse unit test (no binary needed).
- test/e2e/bootstrap-real-claude.serial.test.ts: real `gbrain bootstrap`
  install → REAL `claude mcp add` (verified via `claude mcp get`) → verify
  exit 0 → a real `claude -p --mcp-config --strict-mcp-config` turn that
  invokes mcp__gbrain__search and answers from the brain (proven: toolCalls
  include mcp__gbrain__search, final text carries the seeded fact).
- test/e2e/bootstrap-real-codex.serial.test.ts: same install with REAL `codex
  mcp add` into a real ~/.codex/config.toml + Gate-3 pull-protocol assertion,
  then a real `codex exec --json` turn surfacing the fact (MCP or the pull-
  protocol shell path). Bounded retry absorbs codex's occasional MCP-call
  cancellation without softening the fact-requiring assertion.
- Everything hermetic (temp HOME/CLAUDE_CONFIG_DIR/CODEX_HOME/GBRAIN_HOME;
  real ~/.codex auth copied read-only) and skipIf-gated so it self-skips
  cleanly where the binaries/auth are absent.
- heavy-tests.yml: gated `real-agent-e2e` job (nightly/label, never the PR
  shard; no-op on a runner without authed binaries).
- TODOS: compiled `gbrain` binary can't serve a PGLite brain (bun compile
  omits the WASM/extension payloads); harness falls back to `bun run` serve.

Verified against live claude 4.6 + codex 0.147.0: 15 pass / 0 fail; verify
36/36; typecheck clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ci): real-agent-e2e job — bash array + --timeout (actionlint SC2086 + bun-test-timeout guard)

The real-agent-e2e job's file loop used an unquoted $FILES (SC2086) and ran
`bun test` without --timeout (check-bun-test-timeout guard). Switch to a bash
array and add --timeout=600000 (real-agent turns are slow; the tests self-skip
without authed binaries so it's a no-op elsewhere).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(pglite): embed WASM + extension assets so the compiled binary can serve

A `bun build --compile` gbrain binary could not `serve` a PGLite brain: the
compile bundles JS but not PGLite's runtime payload (pglite.wasm, initdb.wasm,
pglite.data, vector/pg_trgm tarballs), so `serve` on PGLite died with a
bunfs/ENOENT. Now the assets ride inside the binary.

- src/core/pglite-embedded-assets.ts: embeds the five assets via
  `import … with { type: 'file' }` (the ENG-6 idiom) and exposes
  getEmbeddedPgliteOptions() → { pgliteWasmModule, initdbWasmModule, fsBundle,
  extensions:{vector,pg_trgm} }. WASM/fsBundle are consumed as bytes; the two
  extension tarballs are materialized to a content-addressed temp file (atomic,
  size-verified reuse) because PGLite reads them via fs.createReadStream, which
  cannot read a /$bunfs path. Unconditional (works in bun-run and compiled),
  so no fragile mode branch.
- src/core/pglite-engine.ts: static-import getEmbeddedPgliteOptions (engine
  path stays static per the engine-dynamic-import invariant); spread into both
  PGlite.create sites (initial + WAL-repair retry). The bunfs classifier stays
  as a backstop but no longer fires for a correct binary.
- scripts/check-pglite-embedded.sh (+ smoketest): compiles a focused binary and
  asserts it boots PGLite, CREATE EXTENSION vector/pg_trgm, and round-trips a
  page — wired into `bun run verify` (now 37 checks), check:all, and
  check:pglite-embedded. Fail-soft only when compile is unavailable.
- agent-harness.ts probeCompiledPglite now passes → the real-agent e2e uses the
  fast compiled MCP server. TODOS: the P2 "can't serve PGLite" item is closed.

Verified: fresh compiled binary ran `search`/`query` against a PGLite brain and
returned the seeded row (no bunfs/ENOENT); verify 37/37; pglite-engine 120/0
source-mode; typecheck clean; engine-dynamic-import + parity guards pass.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-11 16:42:11 -07:00

311 lines
13 KiB
TypeScript

/**
* `gbrain sources push` CLI wrapper (sources.ts:runPush, via the exported
* runSources dispatcher) — the seams the core workspace-push serial suite
* does NOT cover:
*
* - exit-code contract: 0 pushed / 0 skipped-in-flight, 5 blocked/refused,
* 1 fail (push_failed / no local_path), 2 usage, 4 unknown source
* - `<id>` → sources.local_path resolution (stub engine; no PGLite)
* - --json output (machine-readable result on stdout, human lines absent)
*
* Harness: same real-git fixture pair as test/workspace-push.serial.test.ts
* (bare origin + work clone over the file transport). runPush calls
* process.exit() on the non-zero paths, so the driver patches process.exit
* to throw a sentinel and captures console.log/console.error; everything is
* restored in a finally. Serial file: mutates HOME/GBRAIN_HOME per test
* (Bun snapshots env at startup — spawned git must see the swap, #2943).
*/
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
import { mkdtempSync, rmSync, writeFileSync } from 'node:fs';
import { join } from 'node:path';
import { tmpdir } from 'node:os';
import { execFileSync } from 'node:child_process';
import { runSources } from '../src/commands/sources.ts';
import { acquirePushLock, resolveWorkspaceRoot } from '../src/core/workspace-push.ts';
import { SCAN_ALLOW_FILENAME } from '../src/core/secret-scan.ts';
import type { BrainEngine } from '../src/core/engine.ts';
const T = 60_000; // explicit per-test timeout — bun ignores bunfig.toml's key
const OPENAI = 'sk-' + 'A1b2C3d4E5f6G7h8I9j0K1l2M3n4';
// #2943: env: process.env is REQUIRED — Bun snapshots env at startup, so
// spawned git would otherwise be blind to beforeEach's HOME/GBRAIN_HOME swap.
function git(cwd: string, ...args: string[]): string {
return execFileSync('git', ['-C', cwd, '-c', 'protocol.file.allow=always', ...args], {
stdio: ['ignore', 'pipe', 'pipe'], encoding: 'utf-8', env: process.env,
}).trim();
}
function originHead(bareRepo: string): string {
return git(bareRepo, 'rev-parse', 'refs/heads/main');
}
let root: string;
let work: string;
let bare: string;
let saved: Record<string, string | undefined> = {};
function makePair(): void {
bare = mkdtempSync(join(root, 'origin-')) + '.git';
execFileSync('git', ['init', '-q', '--bare', '-b', 'main', bare], { stdio: 'ignore', env: process.env });
work = mkdtempSync(join(root, 'work-'));
execFileSync('git', ['-c', 'protocol.file.allow=always', 'clone', '-q', bare, work], {
stdio: 'ignore', env: process.env,
});
git(work, 'config', 'user.email', 't@t.t');
git(work, 'config', 'user.name', 'tester');
writeFileSync(join(work, 'README.md'), 'init\n');
git(work, 'add', 'README.md');
git(work, 'commit', '-qm', 'init');
git(work, 'push', '-q', 'origin', 'main');
try { git(work, 'remote', 'set-head', 'origin', 'main'); } catch { /* */ }
}
beforeEach(() => {
root = mkdtempSync(join(tmpdir(), 'spc-'));
saved = {
HOME: process.env.HOME,
GBRAIN_HOME: process.env.GBRAIN_HOME,
GBRAIN_GIT_ALLOW_FILE_TRANSPORT: process.env.GBRAIN_GIT_ALLOW_FILE_TRANSPORT,
};
process.env.HOME = mkdtempSync(join(root, 'home-'));
// CX2-8: GBRAIN_HOME is a PARENT dir → effective home is $HOME/.gbrain.
process.env.GBRAIN_HOME = process.env.HOME;
process.env.GBRAIN_GIT_ALLOW_FILE_TRANSPORT = '1';
makePair();
});
afterEach(() => {
for (const [k, v] of Object.entries(saved)) {
if (v === undefined) delete process.env[k];
else process.env[k] = v;
}
rmSync(root, { recursive: true, force: true });
});
// ── driver: run the wrapper, capture exit code + stdout/stderr ──────────────
class ExitSentinel extends Error {
constructor(public code: number) {
super(`process.exit(${code})`);
}
}
interface CliRun {
exitCode: number;
stdout: string[];
stderr: string[];
}
async function runPushCli(engine: BrainEngine, args: string[]): Promise<CliRun> {
const stdout: string[] = [];
const stderr: string[] = [];
let exitCode = 0;
const origExit = process.exit;
const origLog = console.log;
const origErr = console.error;
(process as { exit: (code?: number) => never }).exit = ((code?: number) => {
throw new ExitSentinel(code ?? 0);
}) as never;
console.log = (...a: unknown[]) => { stdout.push(a.map(String).join(' ')); };
console.error = (...a: unknown[]) => { stderr.push(a.map(String).join(' ')); };
try {
await runSources(engine, ['push', ...args]);
} catch (e) {
if (e instanceof ExitSentinel) exitCode = e.code;
else throw e;
} finally {
process.exit = origExit;
console.log = origLog;
console.error = origErr;
}
return { exitCode, stdout, stderr };
}
/** Engine stub whose executeRaw serves canned source rows (and records params). */
function stubEngine(rows: Array<Record<string, unknown>>, calls: unknown[][] = []): BrainEngine {
return {
executeRaw: async (_sql: string, params?: unknown[]) => {
calls.push(params ?? []);
return rows;
},
} as unknown as BrainEngine;
}
/** Engine that throws on ANY use — proves --path never touches the DB. */
const untouchableEngine = new Proxy({}, {
get(_t, prop) {
return () => { throw new Error(`engine.${String(prop)} must not be called`); };
},
}) as BrainEngine;
// ── exit 2: usage ────────────────────────────────────────────────────────────
describe('usage errors → exit 2', () => {
test('neither <id> nor --path → exit 2 with usage line; engine untouched', async () => {
const r = await runPushCli(untouchableEngine, []);
expect(r.exitCode).toBe(2);
expect(r.stderr.join('\n')).toContain('Usage: gbrain sources push');
}, T);
test('BOTH <id> and --path → exit 2 (mutually exclusive)', async () => {
const r = await runPushCli(untouchableEngine, ['wiki', '--path', work]);
expect(r.exitCode).toBe(2);
expect(r.stderr.join('\n')).toContain('Usage:');
}, T);
test('unknown flag → exit 2 naming the flag', async () => {
const r = await runPushCli(untouchableEngine, ['--bogus']);
expect(r.exitCode).toBe(2);
expect(r.stderr.join('\n')).toContain('Unknown flag: --bogus');
}, T);
});
// ── exit 4 / exit 1: <id> resolution failures ───────────────────────────────
describe('<id> resolution failures', () => {
test('unknown source id → exit 4', async () => {
const r = await runPushCli(stubEngine([]), ['nope']);
expect(r.exitCode).toBe(4);
expect(r.stderr.join('\n')).toContain('Source "nope" not found');
}, T);
test('source with NULL local_path → exit 1 (nothing to push)', async () => {
const r = await runPushCli(stubEngine([{ id: 'wiki', local_path: null }]), ['wiki']);
expect(r.exitCode).toBe(1);
expect(r.stderr.join('\n')).toContain('no local_path');
}, T);
});
// ── <id> → local_path resolution happy path ─────────────────────────────────
describe('<id> → local_path resolution', () => {
test('resolves the source row local_path and pushes THAT dir; exit 0', async () => {
const calls: unknown[][] = [];
const engine = stubEngine([{ id: 'wiki', local_path: work }], calls);
writeFileSync(join(work, 'note.md'), 'remember this\n');
const r = await runPushCli(engine, ['wiki', '--allow-unverified-remote']);
expect(r.exitCode).toBe(0);
expect(r.stdout.join('\n')).toContain(`Pushed`);
expect(originHead(bare)).toBe(git(work, 'rev-parse', 'HEAD')); // the row's dir got pushed
expect(calls[0]).toEqual(['wiki']); // fetched by the given id
}, T);
});
// ── exit 0: pushed / skipped-in-flight ──────────────────────────────────────
describe('exit 0 — pushed and clean skip', () => {
test('--path push succeeds → exit 0, human summary, no DB access', async () => {
writeFileSync(join(work, 'note.md'), 'n\n');
const r = await runPushCli(untouchableEngine, ['--path', work, '--allow-unverified-remote']);
expect(r.exitCode).toBe(0);
expect(r.stdout.join('\n')).toContain('Pushed');
expect(r.stdout.join('\n')).toContain('(new commit)');
expect(originHead(bare)).toBe(git(work, 'rev-parse', 'HEAD'));
}, T);
test('push already in flight → clean skip, exit 0 [G14/A5]', async () => {
// Lock the RESOLVED repo root — workspacePush locks the realpath'd root,
// and mkdtemp dirs on macOS sit behind the /var → /private/var symlink.
const lock = acquirePushLock(resolveWorkspaceRoot(work)!);
expect(lock.acquired).toBe(true);
try {
const r = await runPushCli(untouchableEngine, ['--path', work, '--allow-unverified-remote']);
expect(r.exitCode).toBe(0);
expect(r.stdout.join('\n')).toContain('skipped: push in flight');
expect(r.stdout.join('\n')).toContain(String(process.pid));
} finally {
if (lock.acquired) lock.handle.release();
}
}, T);
});
// ── exit 5: blocked / refused ───────────────────────────────────────────────
describe('exit 5 — blocked and refused', () => {
test('secret-scan findings → exit 5, names file+pattern+allowlist hint', async () => {
writeFileSync(join(work, 'notes.md'), `my key: ${OPENAI}\n`);
const r = await runPushCli(untouchableEngine, ['--path', work, '--allow-unverified-remote']);
expect(r.exitCode).toBe(5);
const err = r.stderr.join('\n');
expect(err).toContain('PUSH BLOCKED');
expect(err).toContain('notes.md');
expect(err).toContain(SCAN_ALLOW_FILENAME);
expect(err.includes(OPENAI)).toBe(false); // the value never surfaces
}, T);
test('tracked deny-glob match → exit 5 with the git rm --cached hint', async () => {
writeFileSync(join(work, '.env'), 'SECRETISH=value\n');
git(work, 'add', '-f', '.env');
git(work, 'commit', '-qm', 'oops tracked env');
const r = await runPushCli(untouchableEngine, ['--path', work, '--allow-unverified-remote']);
expect(r.exitCode).toBe(5);
const err = r.stderr.join('\n');
expect(err).toContain('PUSH BLOCKED');
expect(err).toContain('.env');
expect(err).toContain('git rm --cached');
}, T);
test('unverifiable remote WITHOUT --allow-unverified-remote → exit 5, PUSH REFUSED', async () => {
writeFileSync(join(work, 'note.md'), 'n\n');
const before = originHead(bare);
const r = await runPushCli(untouchableEngine, ['--path', work]);
expect(r.exitCode).toBe(5);
expect(r.stderr.join('\n')).toContain('PUSH REFUSED');
expect(originHead(bare)).toBe(before); // nothing left the machine
}, T);
test('unscannable staged blob (over the scan cap) → exit 5, fail-closed', async () => {
writeFileSync(join(work, 'big.bin'), Buffer.alloc(26 * 1024 * 1024, 0x61));
const before = originHead(bare);
const r = await runPushCli(untouchableEngine, ['--path', work, '--allow-unverified-remote']);
expect(r.exitCode).toBe(5);
const err = r.stderr.join('\n');
expect(err).toContain('PUSH BLOCKED');
expect(err).toContain('big.bin');
expect(originHead(bare)).toBe(before); // nothing left the machine
}, T);
});
// ── exit 1: push failure ────────────────────────────────────────────────────
describe('exit 1 — push failure', () => {
test('unreachable origin → exit 1 with the failure status', async () => {
git(work, 'remote', 'set-url', 'origin', join(root, 'gone.git'));
writeFileSync(join(work, 'note.md'), 'n\n');
const r = await runPushCli(untouchableEngine, ['--path', work, '--allow-unverified-remote']);
expect(r.exitCode).toBe(1);
expect(r.stderr.join('\n')).toContain('push failed (push_failed)');
}, T);
});
// ── --json output ───────────────────────────────────────────────────────────
describe('--json output', () => {
test('pushed → single JSON payload on stdout, human lines suppressed, exit 0', async () => {
writeFileSync(join(work, 'note.md'), 'n\n');
const r = await runPushCli(untouchableEngine, ['--path', work, '--allow-unverified-remote', '--json']);
expect(r.exitCode).toBe(0);
expect(r.stdout.length).toBe(1); // exactly the JSON blob — no "Pushed …" line
const res = JSON.parse(r.stdout[0]);
expect(res.status).toBe('pushed');
expect(res.ok).toBe(true);
expect(res.committed).toBe(true);
expect(res.branch).toBe('main');
}, T);
test('blocked_secrets → exit 5 AND machine-readable findings on stdout', async () => {
writeFileSync(join(work, 'notes.md'), `my key: ${OPENAI}\n`);
const r = await runPushCli(untouchableEngine, ['--path', work, '--allow-unverified-remote', '--json']);
expect(r.exitCode).toBe(5);
expect(r.stdout.length).toBe(1);
const res = JSON.parse(r.stdout[0]);
expect(res.status).toBe('blocked_secrets');
expect(res.findings?.length).toBe(1);
expect(res.findings[0].file).toBe('notes.md');
expect(r.stdout[0].includes(OPENAI)).toBe(false); // redacted in JSON too
// json mode suppresses the wrapper's human finding dump (the core logger
// line still reaches stderr via `[gbrain]` — that one is expected).
expect(r.stderr.join('\n')).not.toContain('allow this finding:');
}, T);
});