mirror of
https://github.com/garrytan/gbrain.git
synced 2026-08-14 08:53:22 +00:00
test(bootstrap): real-agent e2e — drive the actual claude + codex binaries end to end
Closes the audit's biggest gap ("no real harness ever drives a turn"). Adapts
gstack's PTY/headless agent harness to prove the bootstrap install + smoke work
against the REAL binaries, not PATH shims. Test/CI/docs only — zero src changes.
- test/helpers/agent-harness.ts: hermetic clean-room child env (ported from
gstack; drops CONDUCTOR_/CLAUDE_/GSTACK_/MCP_/GBRAIN_, promotes
GSTACK_ANTHROPIC_API_KEY→ANTHROPIC_API_KEY), real-binary resolvers + auth
probes, headless `claude -p --output-format stream-json` and `codex exec
--json` turn runners, a gbrain stdio MCP-config writer, and a keyless brain
seeder. + a fixture-parse unit test (no binary needed).
- test/e2e/bootstrap-real-claude.serial.test.ts: real `gbrain bootstrap`
install → REAL `claude mcp add` (verified via `claude mcp get`) → verify
exit 0 → a real `claude -p --mcp-config --strict-mcp-config` turn that
invokes mcp__gbrain__search and answers from the brain (proven: toolCalls
include mcp__gbrain__search, final text carries the seeded fact).
- test/e2e/bootstrap-real-codex.serial.test.ts: same install with REAL `codex
mcp add` into a real ~/.codex/config.toml + Gate-3 pull-protocol assertion,
then a real `codex exec --json` turn surfacing the fact (MCP or the pull-
protocol shell path). Bounded retry absorbs codex's occasional MCP-call
cancellation without softening the fact-requiring assertion.
- Everything hermetic (temp HOME/CLAUDE_CONFIG_DIR/CODEX_HOME/GBRAIN_HOME;
real ~/.codex auth copied read-only) and skipIf-gated so it self-skips
cleanly where the binaries/auth are absent.
- heavy-tests.yml: gated `real-agent-e2e` job (nightly/label, never the PR
shard; no-op on a runner without authed binaries).
- TODOS: compiled `gbrain` binary can't serve a PGLite brain (bun compile
omits the WASM/extension payloads); harness falls back to `bun run` serve.
Verified against live claude 4.6 + codex 0.147.0: 15 pass / 0 fail; verify
36/36; typecheck clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
co-authored by
Claude Fable 5
Claude Fable 5
parent
99e15f0a10
commit
2f6b817a28
@@ -101,3 +101,44 @@ jobs:
|
||||
path: heavy-artifacts/
|
||||
retention-days: 14
|
||||
if-no-files-found: ignore
|
||||
|
||||
# Real-agent door e2e: drives the ACTUAL `claude` + `codex` binaries (no PATH
|
||||
# shims) against a real gbrain over MCP. These pay real API cost and need the
|
||||
# binaries installed + authed, which a stock GitHub runner does NOT have — so
|
||||
# both tests self-SKIP (describe.skipIf on binary/auth) and the job is a clean
|
||||
# no-op here. It exists so a self-hosted / manually-provisioned runner WITH
|
||||
# authed claude/codex (and ANTHROPIC/OPENAI creds) actually exercises the real
|
||||
# binaries. Heavy cadence only (nightly + `real-agent-e2e` label + dispatch);
|
||||
# NEVER the PR shard matrix.
|
||||
real-agent-e2e:
|
||||
name: Real-agent door e2e (skips without authed binaries)
|
||||
if: |
|
||||
github.event_name != 'pull_request' ||
|
||||
contains(github.event.pull_request.labels.*.name, 'real-agent-e2e') ||
|
||||
contains(github.event.pull_request.labels.*.name, 'heavy-tests')
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 20
|
||||
steps:
|
||||
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
|
||||
with:
|
||||
bun-version: 1.3.13
|
||||
- run: bun install
|
||||
|
||||
# Reference both door tests; run only the ones present (the claude door
|
||||
# may land in a sibling PR). Missing binary/auth → the file self-skips, so
|
||||
# a stock runner reports a green no-op rather than failing.
|
||||
- name: Run real-agent door tests
|
||||
run: |
|
||||
FILES=""
|
||||
for f in \
|
||||
test/e2e/bootstrap-real-claude.serial.test.ts \
|
||||
test/e2e/bootstrap-real-codex.serial.test.ts; do
|
||||
[ -f "$f" ] && FILES="$FILES $f"
|
||||
done
|
||||
if [ -z "$FILES" ]; then
|
||||
echo "SKIP: no real-agent door test files present yet."
|
||||
exit 0
|
||||
fi
|
||||
echo "Running:$FILES"
|
||||
bun test $FILES
|
||||
|
||||
@@ -5018,6 +5018,19 @@ respective shapes. Small, mechanical; pinned by `test/init-embed-check.test.ts`
|
||||
|
||||
## Agent-bootstrap wave follow-ups (filed at build time)
|
||||
|
||||
- [ ] **P2 — compiled `gbrain` binary cannot `serve` a PGLite brain.** `bun
|
||||
build --compile` does not embed PGLite's WASM/extension payloads
|
||||
(`pglite.data`, `vector.tar.gz`, `pg_trgm.tar.gz`) — they're absent from the
|
||||
read-only `/$bunfs` (Bun vfs #1340), so `bin/gbrain serve` fails to open a
|
||||
PGLite data dir. Real installs use `bun install -g` (runs `serve` via bun on
|
||||
source), so normal users are unaffected — but the release also publishes
|
||||
compiled binaries for self-update, and `gbrain serve` from one on PGLite
|
||||
would fail. Fix: embed the WASM/extension assets in the compile step (so the
|
||||
compiled binary is serve-capable and the real-agent e2e MCP server can use
|
||||
it for fast startup), OR document that PGLite `serve` requires the bun-run
|
||||
path. Found building the real-agent e2e (the harness falls back to `bun run
|
||||
src/cli.ts serve`, measured ~300ms tools/list, so speed isn't the blocker —
|
||||
this is purely the compiled-binary capability gap).
|
||||
- [ ] **P1 — `--background --follow` spawns a nonexistent subcommand.**
|
||||
`src/core/cli-options.ts` (~:391) spawns `gbrain jobs follow <id>` after a
|
||||
background submit, but jobs.ts has no `follow` subcommand (`jobs watch
|
||||
@@ -5114,6 +5127,16 @@ respective shapes. Small, mechanical; pinned by `test/init-embed-check.test.ts`
|
||||
install, gh auth, repo create, MCP registration) stays manual. Unblocks after 10
|
||||
consecutive green offline runs in heavy-tests (CEO review D3.3b). Start:
|
||||
tests/docker/ harness + the fake-gh recording shim.
|
||||
- [ ] **P2 — Real-agent door e2e needs a provisioned runner.** The real-binary door
|
||||
tests (`test/e2e/bootstrap-real-{claude,codex}.serial.test.ts`) drive the ACTUAL
|
||||
`claude`/`codex` binaries against a real gbrain and pay API cost. They self-SKIP
|
||||
(`describe.skipIf` on binary/auth) everywhere else, so the `real-agent-e2e` job in
|
||||
`.github/workflows/heavy-tests.yml` is a green no-op on stock GitHub runners. To
|
||||
actually EXERCISE them we need a self-hosted / manually-provisioned runner with
|
||||
authed `claude` + `codex` and provider creds exported
|
||||
(`GSTACK_ANTHROPIC_API_KEY`/`ANTHROPIC_API_KEY`, `VOYAGE_API_KEY`). Until then they
|
||||
run locally on an operator machine only. Start: stand up a labeled runner with the
|
||||
binaries pre-authed, or a scheduled self-hosted lane.
|
||||
|
||||
- [ ] **P1 — unit-shard exit hang: bun test process leaks a ref'd handle and never
|
||||
exits after all tests pass.** Probabilistic (scales with file count/duration),
|
||||
|
||||
@@ -123,3 +123,37 @@ One command: `gbrain doctor`. It covers hook health, push staleness, serve/lock
|
||||
collisions, schema state, and prints fixes. `gbrain bootstrap status --json` emits
|
||||
a support blob (versions, harness, last verify/push, hook failure rate) your agent
|
||||
can relay verbatim when you report a problem.
|
||||
|
||||
## Real-agent e2e
|
||||
|
||||
Most bootstrap tests drive the dispatcher with PATH-shimmed `claude`/`codex`
|
||||
recorders — fast, hermetic, no API cost. Two additional "door" tests drive the
|
||||
ACTUAL binaries end to end so we catch real-world drift (a `codex mcp add` flag
|
||||
that changed shape, a harness that stopped calling our MCP server):
|
||||
|
||||
- `test/e2e/bootstrap-real-claude.serial.test.ts` — real `claude -p` over MCP.
|
||||
- `test/e2e/bootstrap-real-codex.serial.test.ts` — real `codex exec`. It runs the
|
||||
keyless-`init` → interview → render → `gbrain bootstrap hooks --harness codex`
|
||||
path (executing the real `codex mcp add` into a hermetic `~/.codex/config.toml`),
|
||||
asserts the rendered `AGENTS.md` carries the Gate-3 brain-first pull protocol
|
||||
(Codex has no hook system, so the pull protocol is its per-turn seam), then
|
||||
spends one live `codex exec` turn to prove real codex → gbrain MCP → brain →
|
||||
a seeded, brain-only fact (falling back to a shell `gbrain query` if headless
|
||||
stdio-MCP is unavailable).
|
||||
|
||||
These pay real API cost and take 30s–2min per turn, so they are NOT in the PR
|
||||
shard. Everything is hermetic (temp `HOME` / `CODEX_HOME` / `CLAUDE_CONFIG_DIR` /
|
||||
`GBRAIN_HOME` per test — the operator's real `~/.claude`, `~/.gbrain`, `~/.codex`
|
||||
are never touched; auth is copied read-only). Each file self-SKIPS via
|
||||
`describe.skipIf` when its binary or auth is absent, so on a machine without the
|
||||
tool it is a clean no-op that never fails. CI wires them into the `real-agent-e2e`
|
||||
job in `.github/workflows/heavy-tests.yml` (nightly + the `real-agent-e2e` /
|
||||
`heavy-tests` label); on a stock runner they self-skip. To actually exercise the
|
||||
binaries you need a runner with authed `claude`/`codex` and the provider creds
|
||||
(`GSTACK_ANTHROPIC_API_KEY`/`ANTHROPIC_API_KEY`, `VOYAGE_API_KEY`) exported.
|
||||
|
||||
Run locally (where both are installed + authed):
|
||||
|
||||
```bash
|
||||
bun test test/e2e/bootstrap-real-codex.serial.test.ts
|
||||
```
|
||||
|
||||
@@ -0,0 +1,310 @@
|
||||
/**
|
||||
* REAL Claude Code "door" E2E — drives the ACTUAL `claude` binary (no PATH
|
||||
* shims), end to end, against a real keyless PGLite brain. Two proofs:
|
||||
*
|
||||
* (1) SETUP+INSTALL — in a hermetic temp HOME + CLAUDE_CONFIG_DIR +
|
||||
* GBRAIN_HOME + workspace, drive the real `gbrain` bootstrap CLI to an
|
||||
* installed state: `gbrain init --pglite --no-embedding` (keyless),
|
||||
* interview (--set required answers + --confirm), render, then
|
||||
* `gbrain bootstrap hooks --harness claude-code` — which runs the REAL
|
||||
* `claude mcp add` into the temp config (project-scope `.mcp.json` in the
|
||||
* workspace). We then run the REAL `claude mcp get gbrain` and assert it
|
||||
* shows OUR serve command + GBRAIN_SOURCE binding (real-registration
|
||||
* proof), and `gbrain bootstrap verify` exits 0 in this hermetic HOME.
|
||||
*
|
||||
* (2) SMOKE — the real-harness-in-the-loop proof: seed a distinctive,
|
||||
* 100%-synthetic fact into a keyless brain, write a `--mcp-config` that
|
||||
* runs THIS repo's `gbrain serve` over stdio, and drive one headless
|
||||
* `claude -p` turn asking about the seeded entity. Assert a gbrain MCP
|
||||
* tool actually fired (real claude → real gbrain MCP → real brain) AND
|
||||
* the seeded fact surfaced. This is the whole product in one turn.
|
||||
*
|
||||
* Every real-agent turn pays API cost and takes 30s–2min, so prompts are
|
||||
* minimal (one seeded fact, one question) and timeouts are capped. The whole
|
||||
* describe fail-SKIPs (never fail-HARD) on a machine without the `claude`
|
||||
* binary or its auth. Serial: PGLite cold starts + real subprocess spawns
|
||||
* would starve parallel siblings.
|
||||
*/
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync, chmodSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join, resolve } from 'node:path';
|
||||
|
||||
import {
|
||||
resolveClaudeBinary,
|
||||
hasClaudeAuth,
|
||||
hermeticChildEnv,
|
||||
seedBrainForAgent,
|
||||
writeGbrainMcpConfig,
|
||||
claudeHeadlessTurn,
|
||||
resolveGbrainServerCommand,
|
||||
} from '../helpers/agent-harness.ts';
|
||||
import { runBootstrap } from '../../src/commands/bootstrap.ts';
|
||||
import type { ExecRunner, ExecResult } from '../../src/core/bootstrap/repo.ts';
|
||||
import { readBackHash } from '../../src/core/bootstrap/interview.ts';
|
||||
import { createEngine } from '../../src/core/engine-factory.ts';
|
||||
import { addSource } from '../../src/core/sources-ops.ts';
|
||||
|
||||
const REPO_ROOT = resolve(import.meta.dir, '..', '..');
|
||||
const CLAUDE_BIN = resolveClaudeBinary();
|
||||
const CAN_RUN = !!CLAUDE_BIN && hasClaudeAuth();
|
||||
|
||||
/** Same required-answer set the shimmed lifecycle e2e uses (synthetic). */
|
||||
const REQUIRED_ANSWERS: Record<string, string> = {
|
||||
AGENT_NAME: 'Lighthouse',
|
||||
PRINCIPAL_NAME: 'Pat Example',
|
||||
AGENT_PURPOSE: 'Maintain the research corpus and draft the weekly memo without re-briefing.',
|
||||
AGENT_TOP_JOBS: '- corpus upkeep\n- weekly memo\n- meeting prep',
|
||||
PRINCIPAL_CONTEXT: 'Runs a small research group; builds internal tooling; values signal over noise.',
|
||||
VOICE_REGISTER: 'Direct: three options, the second one wins.',
|
||||
};
|
||||
|
||||
// ── Shared hermetic state (built once, torn down after) ─────────────────────
|
||||
let tmpParent: string; // GBRAIN_HOME (parent — config appends `.gbrain`)
|
||||
let home: string; // temp $HOME for the spawned claude
|
||||
let claudeCfg: string; // temp CLAUDE_CONFIG_DIR
|
||||
let ws: string; // the "installed" workspace
|
||||
let binDir: string; // holds the dummy absolute gbrain binary path
|
||||
let gbrainBin: string;
|
||||
|
||||
const SAVED_ENV: Record<string, string | undefined> = {};
|
||||
const ENV_KEYS = ['GBRAIN_HOME', 'GBRAIN_SOURCE', 'GBRAIN_HOOKS', 'DATABASE_URL', 'GBRAIN_DATABASE_URL'];
|
||||
|
||||
/** Capture console.log around an async call (runBootstrap prints via console.log). */
|
||||
async function captureStdout<T>(fn: () => Promise<T>): Promise<{ result: T; out: string }> {
|
||||
const orig = console.log;
|
||||
let out = '';
|
||||
console.log = (...args: unknown[]) => {
|
||||
out += args.map(String).join(' ') + '\n';
|
||||
};
|
||||
try {
|
||||
const result = await fn();
|
||||
return { result, out };
|
||||
} finally {
|
||||
console.log = orig;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* ExecRunner that spawns the REAL `claude` binary with a hermetic env pinned
|
||||
* at the temp HOME + CLAUDE_CONFIG_DIR, cwd=workspace (project-scope MCP config
|
||||
* lives in the workspace `.mcp.json`, and `mcp get`/`list` read it from cwd).
|
||||
* argv[0]==='claude' is rewritten to the resolved absolute binary so PATH quirks
|
||||
* never intercept it. Nothing else on the operator's box is touched.
|
||||
*/
|
||||
const claudeRunner: ExecRunner = async (argv: string[]): Promise<ExecResult> => {
|
||||
const real = argv[0] === 'claude' && CLAUDE_BIN ? [CLAUDE_BIN, ...argv.slice(1)] : argv;
|
||||
try {
|
||||
const proc = Bun.spawn(real, {
|
||||
cwd: ws,
|
||||
env: hermeticChildEnv({ HOME: home, CLAUDE_CONFIG_DIR: claudeCfg }),
|
||||
stdout: 'pipe',
|
||||
stderr: 'pipe',
|
||||
stdin: 'ignore',
|
||||
});
|
||||
const [stdout, stderr, code] = await Promise.all([
|
||||
new Response(proc.stdout).text(),
|
||||
new Response(proc.stderr).text(),
|
||||
proc.exited,
|
||||
]);
|
||||
return { code, stdout, stderr };
|
||||
} catch (e) {
|
||||
return { code: 127, stdout: '', stderr: (e as Error).message };
|
||||
}
|
||||
};
|
||||
|
||||
beforeAll(() => {
|
||||
for (const k of ENV_KEYS) SAVED_ENV[k] = process.env[k];
|
||||
// Ambient DATABASE_URL must not flip the sandboxed brain to Postgres.
|
||||
delete process.env.DATABASE_URL;
|
||||
delete process.env.GBRAIN_DATABASE_URL;
|
||||
delete process.env.GBRAIN_SOURCE;
|
||||
delete process.env.GBRAIN_HOOKS;
|
||||
|
||||
// Short prefix — the IPC socket for the hooks smoke lives under database_path
|
||||
// and unix socket paths cap ~104 bytes on macOS.
|
||||
tmpParent = mkdtempSync(join(tmpdir(), 'gb-rc-'));
|
||||
home = mkdtempSync(join(tmpdir(), 'gb-rc-home-'));
|
||||
claudeCfg = mkdtempSync(join(tmpdir(), 'gb-rc-cfg-'));
|
||||
ws = mkdtempSync(join(tmpdir(), 'gb-rc-ws-'));
|
||||
mkdirSync(join(ws, 'brain'), { recursive: true });
|
||||
|
||||
// A dummy absolute gbrain binary for the MCP registration + hook command
|
||||
// strings (never executed here — the registration only records the path, and
|
||||
// `claude mcp get` reads config without launching the server).
|
||||
binDir = mkdtempSync(join(tmpdir(), 'gb-rc-bin-'));
|
||||
gbrainBin = join(binDir, 'gbrain');
|
||||
writeFileSync(gbrainBin, '#!/bin/sh\nexit 0\n');
|
||||
chmodSync(gbrainBin, 0o755);
|
||||
|
||||
// In-process bootstrap subcommands resolve the home from this env.
|
||||
process.env.GBRAIN_HOME = tmpParent;
|
||||
}, 60_000);
|
||||
|
||||
afterAll(() => {
|
||||
for (const k of ENV_KEYS) {
|
||||
if (SAVED_ENV[k] === undefined) delete process.env[k];
|
||||
else process.env[k] = SAVED_ENV[k];
|
||||
}
|
||||
for (const dir of [tmpParent, home, claudeCfg, ws, binDir]) {
|
||||
try {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
} catch {
|
||||
/* best effort */
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
describe.skipIf(!CAN_RUN)('bootstrap real Claude Code door (serial e2e)', () => {
|
||||
test('SETUP+INSTALL: real `gbrain` bootstrap → real `claude mcp add` → verify exit 0', async () => {
|
||||
// 1. Engine phase: real keyless PGLite init as a subprocess (no API keys).
|
||||
const init = spawnSync(
|
||||
'bun',
|
||||
['run', join(REPO_ROOT, 'src', 'cli.ts'), 'init', '--pglite', '--no-embedding', '--non-interactive'],
|
||||
{
|
||||
cwd: ws,
|
||||
env: { ...process.env, GBRAIN_HOME: tmpParent, HOME: home } as Record<string, string>,
|
||||
encoding: 'utf8',
|
||||
timeout: 120_000,
|
||||
stdio: ['ignore', 'pipe', 'pipe'],
|
||||
},
|
||||
);
|
||||
expect(init.status).toBe(0);
|
||||
const cfgPath = join(tmpParent, '.gbrain', 'config.json');
|
||||
const cfg = JSON.parse(readFileSync(cfgPath, 'utf8')) as { engine: string; database_path: string; embedding_disabled?: boolean };
|
||||
expect(cfg.engine).toBe('pglite');
|
||||
expect(cfg.embedding_disabled).toBe(true);
|
||||
|
||||
// Register the workspace source with its brain/ dir so put_page write-through
|
||||
// (the roundtrip check) lands files in the repo — the installed-state that
|
||||
// `gbrain sources add workspace --path <ws>/brain` produces.
|
||||
const engineConfig = { engine: 'pglite' as const, database_path: cfg.database_path };
|
||||
const engine = await createEngine(engineConfig);
|
||||
await engine.connect(engineConfig);
|
||||
await addSource(engine, { id: 'workspace', localPath: join(ws, 'brain'), force: true });
|
||||
await engine.disconnect();
|
||||
|
||||
// 2. Interview: real CLI, scripted answers, read-back-hash confirm.
|
||||
expect(await runBootstrap(['interview', '--init', '--workspace', ws])).toBe(0);
|
||||
for (const [key, value] of Object.entries(REQUIRED_ANSWERS)) {
|
||||
expect(await runBootstrap(['interview', '--set', key, value, '--workspace', ws])).toBe(0);
|
||||
}
|
||||
expect(await runBootstrap(['interview', '--set', 'MCP_SCOPE', 'project', '--workspace', ws])).toBe(0);
|
||||
expect(await runBootstrap(['interview', '--set', 'HOOKS_CONSENT', 'yes', '--workspace', ws])).toBe(0);
|
||||
const h = readBackHash(ws);
|
||||
if (!h.ok) throw new Error(h.message);
|
||||
expect(await runBootstrap(['interview', '--confirm', h.hash, '--workspace', ws])).toBe(0);
|
||||
|
||||
// 3. Render identity files from the confirmed answers.
|
||||
expect(await runBootstrap(['render', '--workspace', ws])).toBe(0);
|
||||
|
||||
// 4. Hooks: REAL `claude mcp add` (project scope) via the hermetic runner,
|
||||
// plus per-turn hooks written into the workspace settings.
|
||||
const { result: hooksCode, out: hooksOut } = await captureStdout(() =>
|
||||
runBootstrap(['hooks', '--workspace', ws, '--harness', 'claude-code', '--gbrain-bin', gbrainBin], {
|
||||
runner: claudeRunner,
|
||||
}),
|
||||
);
|
||||
expect(hooksCode).toBe(0);
|
||||
// The registration smoke VERIFIED (via real `claude mcp get`) that the
|
||||
// recorded server carries our binary + this workspace's source.
|
||||
expect(hooksOut).toContain('verified targeting this workspace');
|
||||
|
||||
// 5. Real-registration PROOF: query the actual `claude` config back.
|
||||
const got = await claudeRunner(['claude', 'mcp', 'get', 'gbrain']);
|
||||
expect(got.code).toBe(0);
|
||||
const registered = `${got.stdout}\n${got.stderr}`;
|
||||
expect(registered).toContain(gbrainBin); // OUR serve binary, absolute path
|
||||
expect(registered).toContain('serve');
|
||||
expect(registered).toContain('GBRAIN_SOURCE=workspace'); // the source binding
|
||||
|
||||
// 6. `gbrain bootstrap verify` — the whole install contract — exits 0 in
|
||||
// this real hermetic HOME (keyless: magic moment via the zero-LLM fence).
|
||||
const { result: verifyCode, out: verifyOut } = await captureStdout(() =>
|
||||
runBootstrap(['verify', '--workspace', ws, '--json']),
|
||||
);
|
||||
if (verifyCode !== 0) {
|
||||
// Surface the report so a failure is diagnosable, not a bare exit code.
|
||||
throw new Error(`verify failed (exit ${verifyCode}):\n${verifyOut}`);
|
||||
}
|
||||
expect(verifyCode).toBe(0);
|
||||
const payload = JSON.parse(verifyOut) as { ok: boolean; checks: Array<{ id: string; ok: boolean; warn?: boolean }> };
|
||||
expect(payload.ok).toBe(true);
|
||||
const byId = (id: string) => payload.checks.find((c) => c.id === id);
|
||||
expect(byId('roundtrip')?.ok).toBe(true);
|
||||
expect(byId('magic_moment')?.ok).toBe(true);
|
||||
}, 300_000);
|
||||
|
||||
test('SMOKE: real `claude -p` → real gbrain MCP → real brain → surfaced fact', async () => {
|
||||
// Independent hermetic brain for the recall proof (its own home + workspace).
|
||||
const smokeHome = mkdtempSync(join(tmpdir(), 'gb-rc-smoke-'));
|
||||
const smokeWs = mkdtempSync(join(tmpdir(), 'gb-rc-smoke-ws-'));
|
||||
const mcpCfgPath = join(smokeHome, 'gbrain-mcp.json');
|
||||
try {
|
||||
const seeded = await seedBrainForAgent(smokeHome, 'workspace');
|
||||
// Prefer a compiled binary (fast MCP startup) so the child's first tool
|
||||
// call isn't cancelled; fall back to `bun run src/cli.ts serve` with a
|
||||
// wider readiness window if the compile is unavailable in this sandbox.
|
||||
const server = resolveGbrainServerCommand(REPO_ROOT);
|
||||
writeGbrainMcpConfig({
|
||||
path: mcpCfgPath,
|
||||
server,
|
||||
gbrainHome: smokeHome,
|
||||
sourceId: 'workspace',
|
||||
});
|
||||
|
||||
// Bounded retry (max 2) rides out a transient MCP-startup cancellation.
|
||||
// PASS only when an attempt lands BOTH a gbrain tool AND the seeded fact;
|
||||
// never pass on zero tool calls, never soften to "answered something".
|
||||
const perAttemptTimeout = server.kind === 'compiled' ? 150_000 : 190_000;
|
||||
const maxAttempts = 2;
|
||||
let passed = false;
|
||||
let lastEvidence = '';
|
||||
|
||||
for (let attempt = 1; attempt <= maxAttempts && !passed; attempt++) {
|
||||
if (attempt > 1) await new Promise((r) => setTimeout(r, 3_000));
|
||||
const turn = await claudeHeadlessTurn({
|
||||
prompt:
|
||||
`Use your gbrain memory tools to answer: ${seeded.query} ` +
|
||||
`Answer only from the brain, in one short sentence.`,
|
||||
cwd: smokeWs,
|
||||
home,
|
||||
claudeConfigDir: claudeCfg,
|
||||
mcpConfigPath: mcpCfgPath,
|
||||
timeoutMs: perAttemptTimeout,
|
||||
});
|
||||
|
||||
// Real claude actually invoked OUR MCP server over stdio: a gbrain tool
|
||||
// name (e.g. `mcp__gbrain__search`) appears in the tool-call trace.
|
||||
// `toolCalls` holds tool NAMES only, so this can't false-positive on a
|
||||
// stray "gbrain" substring in a file path or the raw event stream.
|
||||
const firedGbrainTool = turn.toolCalls.some((t) => /gbrain/i.test(t));
|
||||
// And the seeded, brain-ONLY fact reached the final answer. Only
|
||||
// "rivermouth" proves recall — the entity "Summit Robotics" is echoed
|
||||
// from the question, so matching it would be a false positive.
|
||||
const gotFact = /rivermouth/i.test(turn.finalText);
|
||||
|
||||
lastEvidence =
|
||||
`[smoke claude attempt ${attempt}/${maxAttempts}] server=${server.kind} ` +
|
||||
`exit=${turn.exitCode} timedOut=${turn.timedOut} firedGbrainTool=${firedGbrainTool} gotFact=${gotFact}\n` +
|
||||
`toolCalls=${JSON.stringify(turn.toolCalls)}\n` +
|
||||
`finalText=${turn.finalText.slice(0, 800)}\n` +
|
||||
`raw tail=${turn.rawLines.slice(-6).join('\n')}`;
|
||||
console.log(lastEvidence);
|
||||
|
||||
if (firedGbrainTool && gotFact) passed = true;
|
||||
}
|
||||
|
||||
expect(passed, `SMOKE failed on all ${maxAttempts} attempts.\n${lastEvidence}`).toBe(true);
|
||||
} finally {
|
||||
for (const dir of [smokeHome, smokeWs]) {
|
||||
try {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
} catch {
|
||||
/* best effort */
|
||||
}
|
||||
}
|
||||
}
|
||||
}, 420_000);
|
||||
});
|
||||
@@ -0,0 +1,357 @@
|
||||
/**
|
||||
* REAL-agent door test — Codex edition. Drives the ACTUAL `codex` binary (no
|
||||
* PATH shim) against a real gbrain, over the two seams that matter for Codex:
|
||||
*
|
||||
* 1. INSTALL — the real `gbrain bootstrap` CLI (keyless `gbrain init`,
|
||||
* interview --set/--confirm, render, then `gbrain bootstrap hooks
|
||||
* --harness codex` executing the REAL `codex mcp add` into a hermetic
|
||||
* ~/.codex/config.toml). Asserts the registration landed (real `codex mcp
|
||||
* get gbrain` + the temp config.toml carry our server + GBRAIN_SOURCE),
|
||||
* `gbrain bootstrap verify` exits 0, and the rendered AGENTS.md carries the
|
||||
* Gate-3 brain-first pull protocol — Codex's ONLY per-turn mechanism (it
|
||||
* has no hook system).
|
||||
*
|
||||
* 2. SMOKE — a live `codex exec` turn. gbrain is registered as a Codex stdio
|
||||
* MCP server (`bun run <repo>/src/cli.ts serve --surface full`) pinned to a
|
||||
* seeded keyless brain. Codex is asked a single question whose answer only
|
||||
* the brain holds; we assert a gbrain tool/command shows up in the turn AND
|
||||
* the seeded fact surfaces in the final text. Proves: real codex → gbrain
|
||||
* MCP → brain → fact. Falls back to the AGENTS.md pull protocol + a direct
|
||||
* shell instruction (`gbrain query`) if headless stdio-MCP is unreliable;
|
||||
* the assertion documents which path proved out.
|
||||
*
|
||||
* EVERYTHING is hermetic (temp HOME / CODEX_HOME / GBRAIN_HOME per test) and the
|
||||
* whole file self-SKIPS via describe.skipIf when the codex binary or its auth is
|
||||
* absent, so it is a clean no-op on a runner without them. Serial: PGLite cold
|
||||
* starts + a real codex spawn would starve parallel siblings; every test carries
|
||||
* an explicit timeout. Real turns cost API + take 30s–2min — prompts are minimal
|
||||
* (one seeded fact, one question) and capped at 240s.
|
||||
*/
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { spawnSync, execFileSync } from 'node:child_process';
|
||||
import {
|
||||
cpSync,
|
||||
existsSync,
|
||||
mkdirSync,
|
||||
mkdtempSync,
|
||||
readFileSync,
|
||||
rmSync,
|
||||
writeFileSync,
|
||||
} from 'node:fs';
|
||||
import { homedir, tmpdir } from 'node:os';
|
||||
import { join, resolve } from 'node:path';
|
||||
|
||||
import {
|
||||
resolveCodexBinary,
|
||||
hasCodexAuth,
|
||||
codexExecTurn,
|
||||
seedBrainForAgent,
|
||||
hermeticChildEnv,
|
||||
resolveGbrainServerCommand,
|
||||
} from '../helpers/agent-harness.ts';
|
||||
import { runBootstrap } from '../../src/commands/bootstrap.ts';
|
||||
import type { ExecRunner } from '../../src/core/bootstrap/repo.ts';
|
||||
import { initState, setAnswer, confirm, readBackHash } from '../../src/core/bootstrap/interview.ts';
|
||||
import { readManifest } from '../../src/core/bootstrap/format.ts';
|
||||
import { createEngine } from '../../src/core/engine-factory.ts';
|
||||
import { addSource } from '../../src/core/sources-ops.ts';
|
||||
|
||||
const REPO_ROOT = resolve(import.meta.dir, '..', '..');
|
||||
const CLI = join(REPO_ROOT, 'src', 'cli.ts');
|
||||
const CODEX_BIN = resolveCodexBinary();
|
||||
const CAN_RUN = !!CODEX_BIN && hasCodexAuth();
|
||||
|
||||
const REQUIRED_ANSWERS: Record<string, string> = {
|
||||
AGENT_NAME: 'Lifeboat',
|
||||
PRINCIPAL_NAME: 'Pat Example',
|
||||
AGENT_PURPOSE: 'Maintain the research corpus and draft the weekly memo without re-briefing.',
|
||||
AGENT_TOP_JOBS: '- corpus upkeep\n- weekly memo\n- meeting prep',
|
||||
PRINCIPAL_CONTEXT: 'Runs a small research group; builds internal tooling; values signal over noise.',
|
||||
VOICE_REGISTER: 'Direct: three options, the second one wins.',
|
||||
};
|
||||
|
||||
const ENV_KEYS = [
|
||||
'GBRAIN_HOME', 'GBRAIN_DATABASE_URL', 'DATABASE_URL', 'GBRAIN_BRAIN_ID',
|
||||
'GBRAIN_SOURCE', 'GBRAIN_HOOKS', 'GBRAIN_BOOTSTRAP_ABORT_AFTER',
|
||||
'CLAUDECODE', 'CLAUDE_CODE_ENTRYPOINT', 'CODEX_HOME', 'CODEX_SANDBOX', 'CODEX_CI',
|
||||
];
|
||||
const SAVED_ENV: Record<string, string | undefined> = {};
|
||||
|
||||
/** Seed a hermetic ~/.codex under `home` with ONLY the operator's real
|
||||
* auth.json (read-only copy) so a spawned codex authenticates without ever
|
||||
* touching the real config dir. Deliberately does NOT copy the operator's
|
||||
* real config.toml: it defines the operator's private remote MCP servers
|
||||
* (which require secrets we don't have and would fail the whole session), and
|
||||
* copying private server names is a privacy smell. `codex mcp add` writes a
|
||||
* fresh, gbrain-only config.toml on top of this. */
|
||||
function seedCodexHome(home: string): string {
|
||||
const codexHome = join(home, '.codex');
|
||||
mkdirSync(codexHome, { recursive: true });
|
||||
const src = join(homedir(), '.codex', 'auth.json');
|
||||
const dst = join(codexHome, 'auth.json');
|
||||
if (existsSync(src) && !existsSync(dst)) {
|
||||
try { cpSync(src, dst); } catch { /* best-effort */ }
|
||||
}
|
||||
// Placeholder global identity files so codexExecTurn's copy-if-missing loop
|
||||
// never pulls the operator's PRIVATE ~/.codex/{AGENTS,SOUL}.md into the turn
|
||||
// (behavior + privacy). The per-turn instruction rides the prompt instead.
|
||||
for (const f of ['AGENTS.md', 'SOUL.md']) {
|
||||
const p = join(codexHome, f);
|
||||
if (!existsSync(p)) {
|
||||
try { writeFileSync(p, '<!-- hermetic test placeholder -->\n'); } catch { /* best-effort */ }
|
||||
}
|
||||
}
|
||||
return codexHome;
|
||||
}
|
||||
|
||||
/** Run the REAL `codex` binary under a hermetic HOME/CODEX_HOME. Used both as
|
||||
* the bootstrap `hooks` exec runner (so `codex mcp add` writes the temp
|
||||
* config.toml) and for direct `codex mcp get`/`list` probes. */
|
||||
function makeCodexRunner(home: string): ExecRunner {
|
||||
const codexHome = join(home, '.codex');
|
||||
return async (argv: string[]) => {
|
||||
try {
|
||||
const proc = Bun.spawn(argv, {
|
||||
env: hermeticChildEnv(
|
||||
{ HOME: home, CODEX_HOME: codexHome },
|
||||
{ extraAllow: ['OPENAI_API_KEY', 'CODEX_*'] },
|
||||
),
|
||||
stdout: 'pipe',
|
||||
stderr: 'pipe',
|
||||
stdin: 'ignore',
|
||||
});
|
||||
const [stdout, stderr, code] = await Promise.all([
|
||||
new Response(proc.stdout).text(),
|
||||
new Response(proc.stderr).text(),
|
||||
proc.exited,
|
||||
]);
|
||||
return { code, stdout, stderr };
|
||||
} catch (e) {
|
||||
return { code: 127, stdout: '', stderr: (e as Error).message };
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
async function captureStdout<T>(fn: () => Promise<T>): Promise<{ result: T; out: string }> {
|
||||
const orig = console.log;
|
||||
let out = '';
|
||||
console.log = (...args: unknown[]) => {
|
||||
out += args.map(String).join(' ') + '\n';
|
||||
};
|
||||
try {
|
||||
const result = await fn();
|
||||
return { result, out };
|
||||
} finally {
|
||||
console.log = orig;
|
||||
}
|
||||
}
|
||||
|
||||
beforeAll(() => {
|
||||
for (const k of ENV_KEYS) SAVED_ENV[k] = process.env[k];
|
||||
// Ambient-state strip: a dev/CI DATABASE_URL must not flip the sandboxed
|
||||
// brain to Postgres; stray GBRAIN_SOURCE/CODEX_* must not leak into a child.
|
||||
for (const k of ENV_KEYS) delete process.env[k];
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
for (const k of ENV_KEYS) {
|
||||
if (SAVED_ENV[k] === undefined) delete process.env[k];
|
||||
else process.env[k] = SAVED_ENV[k];
|
||||
}
|
||||
});
|
||||
|
||||
describe.skipIf(!CAN_RUN)('bootstrap real-codex door (serial e2e)', () => {
|
||||
// ── 1. INSTALL ────────────────────────────────────────────────────────────
|
||||
test('INSTALL: keyless init → interview → render → real `codex mcp add` → verify', async () => {
|
||||
const gbHome = mkdtempSync(join(tmpdir(), 'gb-rc-home-'));
|
||||
const ws = mkdtempSync(join(tmpdir(), 'gb-rc-ws-'));
|
||||
const codexHost = mkdtempSync(join(tmpdir(), 'gb-rc-codex-'));
|
||||
const shimDir = mkdtempSync(join(tmpdir(), 'gb-rc-bin-'));
|
||||
const savedHome = process.env.GBRAIN_HOME;
|
||||
try {
|
||||
// A fresh git workspace (the cwd the human pasted into) with a brain/ dir.
|
||||
execFileSync('git', ['init', '-q', '-b', 'main'], { cwd: ws });
|
||||
execFileSync('git', ['config', 'user.email', 'test@example.com'], { cwd: ws });
|
||||
execFileSync('git', ['config', 'user.name', 'Test'], { cwd: ws });
|
||||
mkdirSync(join(ws, 'brain'), { recursive: true });
|
||||
|
||||
// Hermetic ~/.codex for the real `codex mcp add`.
|
||||
seedCodexHome(codexHost);
|
||||
const codexRunner = makeCodexRunner(codexHost);
|
||||
|
||||
// A dummy absolute gbrain binary the registration records (never executed
|
||||
// by INSTALL — the mcp-add just writes its path into config.toml).
|
||||
const gbrainBin = join(shimDir, 'gbrain');
|
||||
Bun.write(gbrainBin, '#!/bin/sh\nexit 0\n');
|
||||
execFileSync('chmod', ['+x', gbrainBin]);
|
||||
|
||||
process.env.GBRAIN_HOME = gbHome;
|
||||
|
||||
// (a) Keyless init — the REAL CLI, non-interactive, no embedding provider.
|
||||
const init = spawnSync('bun', ['run', CLI, 'init', '--pglite', '--no-embedding', '--non-interactive'], {
|
||||
cwd: ws,
|
||||
env: { ...process.env, GBRAIN_HOME: gbHome },
|
||||
encoding: 'utf8',
|
||||
timeout: 120_000,
|
||||
});
|
||||
expect(init.status).toBe(0);
|
||||
const dbPath = join(gbHome, '.gbrain', 'brain.pglite');
|
||||
expect(existsSync(dbPath)).toBe(true);
|
||||
|
||||
// Register the workspace source the interview's default GBRAIN_SOURCE
|
||||
// routes to (verify's write-through needs a source with a localPath).
|
||||
const engineConfig = { engine: 'pglite' as const, database_path: dbPath };
|
||||
const engine = await createEngine(engineConfig);
|
||||
await engine.connect(engineConfig);
|
||||
await engine.initSchema();
|
||||
await addSource(engine, { id: 'workspace', localPath: join(ws, 'brain'), force: true });
|
||||
await engine.disconnect();
|
||||
|
||||
// (b) Interview — scripted answers, read-back hash confirm.
|
||||
expect(initState(ws).ok).toBe(true);
|
||||
for (const [key, value] of Object.entries(REQUIRED_ANSWERS)) {
|
||||
const r = setAnswer(ws, key, value);
|
||||
if (!r.ok) throw new Error(r.message);
|
||||
}
|
||||
const h = readBackHash(ws);
|
||||
if (!h.ok) throw new Error(h.message);
|
||||
expect(confirm(ws, h.hash).ok).toBe(true);
|
||||
|
||||
// (c) Render — identity files + manifest.
|
||||
expect(await runBootstrap(['render', '--workspace', ws])).toBe(0);
|
||||
expect(readManifest(ws).state).toBe('initialized');
|
||||
|
||||
// AGENTS.md carries the Gate-3 brain-first pull protocol (Codex's only
|
||||
// per-turn mechanism — it has no hook system).
|
||||
const agents = readFileSync(join(ws, 'AGENTS.md'), 'utf8');
|
||||
expect(agents).toContain('Gate 3');
|
||||
expect(agents.toLowerCase()).toContain('brain first');
|
||||
expect(agents).toContain('recall');
|
||||
|
||||
// (d) hooks --harness codex → REAL `codex mcp add` via the runner seam.
|
||||
const { result: hooksCode, out: hooksOut } = await captureStdout(() =>
|
||||
runBootstrap(['hooks', '--workspace', ws, '--harness', 'codex', '--gbrain-bin', gbrainBin], {
|
||||
runner: codexRunner,
|
||||
}),
|
||||
);
|
||||
expect(hooksCode).toBe(0);
|
||||
// Codex has no hooks — the pull protocol is the per-turn seam, stated plainly.
|
||||
expect(hooksOut).toContain('Codex has no hook system');
|
||||
|
||||
// Real `codex mcp get gbrain` shows our server (env values are masked in
|
||||
// the human view, so the source binding is asserted on config.toml below).
|
||||
const get = await codexRunner(['codex', 'mcp', 'get', 'gbrain']);
|
||||
expect(get.code).toBe(0);
|
||||
expect(get.stdout).toContain('gbrain');
|
||||
expect(get.stdout).toContain('serve');
|
||||
expect(get.stdout).toContain('GBRAIN_SOURCE');
|
||||
|
||||
// The hermetic config.toml the real codex wrote carries our server +
|
||||
// the exact GBRAIN_SOURCE binding (the [G1] workspace-source routing).
|
||||
const toml = readFileSync(join(codexHost, '.codex', 'config.toml'), 'utf8');
|
||||
expect(toml).toContain('[mcp_servers.gbrain]');
|
||||
expect(toml).toContain(gbrainBin);
|
||||
expect(toml).toContain('GBRAIN_SOURCE = "workspace"');
|
||||
|
||||
// (e) verify → exit 0 (keyless PGLite; no repo/hooks for the codex path).
|
||||
const { result: verifyCode, out: verifyOut } = await captureStdout(() =>
|
||||
runBootstrap(['verify', '--workspace', ws, '--json']),
|
||||
);
|
||||
expect(verifyCode).toBe(0);
|
||||
const payload = JSON.parse(verifyOut) as { ok: boolean; checks: Array<{ id: string; ok: boolean }> };
|
||||
expect(payload.ok).toBe(true);
|
||||
const roundtrip = payload.checks.find((c) => c.id === 'roundtrip');
|
||||
expect(roundtrip?.ok).toBe(true);
|
||||
} finally {
|
||||
if (savedHome === undefined) delete process.env.GBRAIN_HOME;
|
||||
else process.env.GBRAIN_HOME = savedHome;
|
||||
for (const d of [gbHome, ws, codexHost, shimDir]) {
|
||||
try { rmSync(d, { recursive: true, force: true }); } catch { /* best-effort */ }
|
||||
}
|
||||
}
|
||||
}, 300_000);
|
||||
|
||||
// ── 2. SMOKE ────────────────────────────────────────────────────────────────
|
||||
test('SMOKE: real `codex exec` → gbrain MCP → brain → seeded fact', async () => {
|
||||
const home = mkdtempSync(join(tmpdir(), 'gb-rc-smoke-'));
|
||||
try {
|
||||
// `codex exec` refuses an untrusted cwd ("Not inside a trusted
|
||||
// directory"); a git repo satisfies the check. The turn runs with
|
||||
// cwd=home, so make home a repo (never committed to).
|
||||
execFileSync('git', ['init', '-q', home]);
|
||||
execFileSync('git', ['config', 'user.email', 'test@example.com'], { cwd: home });
|
||||
execFileSync('git', ['config', 'user.name', 'Test'], { cwd: home });
|
||||
|
||||
// Seed a keyless brain the spawned `gbrain serve` will read.
|
||||
const sourceId = 'workspace';
|
||||
const seeded = await seedBrainForAgent(home, sourceId);
|
||||
|
||||
// Hermetic ~/.codex (auth + model settings), then register gbrain as a
|
||||
// stdio MCP server pinned to the seeded brain via the REAL `codex mcp add`
|
||||
// (exact shape from docs/mcp/CODEX.md + registerCodexMcp). Prefer a
|
||||
// compiled binary so MCP startup is fast enough that codex's tool-call
|
||||
// window doesn't close before the server is ready (the "the available
|
||||
// gbrain mcp calls were cancelled" flake); fall back to `bun run
|
||||
// src/cli.ts serve` otherwise.
|
||||
seedCodexHome(home);
|
||||
const runner = makeCodexRunner(home);
|
||||
const server = resolveGbrainServerCommand(REPO_ROOT, ['--surface', 'full']);
|
||||
const add = await runner([
|
||||
'codex', 'mcp', 'add', 'gbrain',
|
||||
'--env', `GBRAIN_HOME=${home}`,
|
||||
'--env', `GBRAIN_SOURCE=${sourceId}`,
|
||||
'--', server.command, ...server.args,
|
||||
]);
|
||||
expect(add.code).toBe(0);
|
||||
|
||||
// The prompt names the shell fallback explicitly: if the MCP tool is
|
||||
// unavailable, run `gbrain query` — which resolves to $HOME/.gbrain
|
||||
// (== the seeded brain, since HOME is the temp dir).
|
||||
const prompt =
|
||||
`You have a gbrain MCP tool connected to a knowledge brain. ` +
|
||||
`Using ONLY that brain (do not guess, do not use general knowledge), answer: ` +
|
||||
`${seeded.query} Report exactly what the brain says. ` +
|
||||
`If no gbrain MCP tool is available, run the shell command ` +
|
||||
`\`bun run ${CLI} query "${seeded.query}"\` and answer from its output.`;
|
||||
|
||||
// Bounded retry (max 2) rides out a transient MCP-startup cancellation.
|
||||
// PASS only when an attempt lands BOTH a gbrain tool/command AND the
|
||||
// seeded fact; never pass on zero tool calls, never soften.
|
||||
const fact = 'rivermouth';
|
||||
const perAttemptTimeout = server.kind === 'compiled' ? 190_000 : 230_000;
|
||||
const maxAttempts = 2;
|
||||
let passed = false;
|
||||
let lastEvidence = '';
|
||||
|
||||
for (let attempt = 1; attempt <= maxAttempts && !passed; attempt++) {
|
||||
if (attempt > 1) await new Promise((r) => setTimeout(r, 3_000));
|
||||
const turn = await codexExecTurn({ prompt, cwd: home, home, timeoutMs: perAttemptTimeout });
|
||||
const raw = turn.rawLines.join('\n');
|
||||
// The harness JSONL parser only captures command_execution/agent_message/
|
||||
// reasoning — a Codex MCP tool call is a distinct `mcp_tool_call` item, so
|
||||
// detect it on the raw stream. server:"gbrain" proves the call hit OUR
|
||||
// registered brain server (not general knowledge).
|
||||
const usedMcp = /"type"\s*:\s*"mcp_tool_call"[^\n]*"server"\s*:\s*"gbrain"/.test(raw);
|
||||
// Shell fallback: codex ran our `gbrain query` command (surfaced in the
|
||||
// parsed command_execution toolCalls). Either path is a valid proof of
|
||||
// real codex → real gbrain → brain.
|
||||
const usedShell = turn.toolCalls.some((c) => /gbrain|cli\.ts\s+query|\bquery\b/i.test(c));
|
||||
const usedGbrain = usedMcp || usedShell;
|
||||
const gotFact = turn.finalText.toLowerCase().includes(fact);
|
||||
|
||||
lastEvidence =
|
||||
`[smoke codex attempt ${attempt}/${maxAttempts}] server=${server.kind} ` +
|
||||
`exit=${turn.exitCode} timedOut=${turn.timedOut} usedMcp=${usedMcp} usedShell=${usedShell} gotFact=${gotFact}\n` +
|
||||
`toolCalls=${JSON.stringify(turn.toolCalls)}\n` +
|
||||
`finalText=${turn.finalText.slice(0, 800)}`;
|
||||
console.log(lastEvidence);
|
||||
|
||||
if (usedGbrain && gotFact) passed = true;
|
||||
}
|
||||
|
||||
expect(passed, `SMOKE failed on all ${maxAttempts} attempts.\n${lastEvidence}`).toBe(true);
|
||||
} finally {
|
||||
try { rmSync(home, { recursive: true, force: true }); } catch { /* best-effort */ }
|
||||
}
|
||||
}, 480_000);
|
||||
});
|
||||
@@ -0,0 +1,743 @@
|
||||
/**
|
||||
* Shared harness for the REAL-agent "door" E2E tests — the ones that drive the
|
||||
* ACTUAL `claude` and `codex` binaries against a live gbrain brain (over MCP
|
||||
* for Claude Code, over CLI for Codex). No PATH shims, no SDK: the door tests
|
||||
* spawn the operator's installed binaries and pay real API cost, so every entry
|
||||
* point here is written to fail-SKIP (never fail-HARD) on a machine that lacks
|
||||
* a binary or its auth.
|
||||
*
|
||||
* Adapted from gstack's test/helpers/{hermetic-env,session-runner,
|
||||
* codex-session-runner,claude-pty-runner}.ts. What changed for gbrain:
|
||||
*
|
||||
* - Hermeticity is UNCONDITIONAL here (no EVALS_HERMETIC escape hatch). A
|
||||
* door test must NEVER see the operator's real ~/.claude, ~/.gbrain, or
|
||||
* ~/.codex. Every spawn gets a scrubbed env pointed at temp HOME /
|
||||
* CLAUDE_CONFIG_DIR / CODEX_HOME / GBRAIN_HOME the test owns.
|
||||
* - promotedEnv is inlined (no dependency on gstack's conductor-env-shim):
|
||||
* GSTACK_ANTHROPIC_API_KEY is promoted to ANTHROPIC_API_KEY when the
|
||||
* canonical key is unset, so a Conductor workspace (which only exports the
|
||||
* GSTACK_ form) still authenticates the child.
|
||||
* - The stream parsers (parseClaudeStream / parseCodexJsonl) are pure and
|
||||
* exported so the companion unit test exercises extraction with ZERO real
|
||||
* binaries.
|
||||
* - seedBrainForAgent + writeGbrainMcpConfig wire a real keyless PGLite brain
|
||||
* to a spawned `gbrain serve` so a Claude Code door test can assert recall.
|
||||
*
|
||||
* The drop-list is the security contract: CONDUCTOR_* / CLAUDE_* / GSTACK_* /
|
||||
* MCP_* / GBRAIN_* never reach a child except via the explicit overrides the
|
||||
* caller passes (which spread LAST and always win).
|
||||
*/
|
||||
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import * as fs from 'node:fs';
|
||||
import * as os from 'node:os';
|
||||
import * as path from 'node:path';
|
||||
|
||||
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
|
||||
import { operations, type OperationContext, type Operation } from '../../src/core/operations.ts';
|
||||
import { saveConfig, gbrainPath, type GBrainConfig } from '../../src/core/config.ts';
|
||||
import { addSource } from '../../src/core/sources-ops.ts';
|
||||
|
||||
// ────────────────────────────────────────────────────────────────────────────
|
||||
// 1. Hermetic child environment
|
||||
// ────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
/** Exact env names a hermetic child keeps. Everything else (unless matched by
|
||||
* a prefix rule or the caller's extraAllow) is dropped. */
|
||||
const ALLOW_EXACT = new Set<string>([
|
||||
// Process basics
|
||||
'PATH', 'HOME', 'TMPDIR', 'TERM', 'COLORTERM', 'LANG', 'LC_ALL', 'SHELL',
|
||||
'USER', 'LOGNAME', 'TZ', 'NODE_ENV', 'CI',
|
||||
// Network reachability — proxied networks can't reach the Anthropic API
|
||||
// without these.
|
||||
'HTTP_PROXY', 'HTTPS_PROXY', 'NO_PROXY',
|
||||
'http_proxy', 'https_proxy', 'no_proxy',
|
||||
'SSL_CERT_FILE', 'SSL_CERT_DIR', 'NODE_EXTRA_CA_CERTS',
|
||||
// Auth — named, NOT the broad ANTHROPIC_* prefix (a prefix rule would smuggle
|
||||
// model/beta/debug knobs that change agent behavior).
|
||||
'ANTHROPIC_API_KEY',
|
||||
'ANTHROPIC_BASE_URL',
|
||||
'ANTHROPIC_AUTH_TOKEN',
|
||||
]);
|
||||
|
||||
/** Prefix rules: eval-harness knobs + CI metadata. Deliberately NOT here:
|
||||
* CONDUCTOR_* / CLAUDE_* / GSTACK_* / MCP_* / GBRAIN_* (session-context
|
||||
* contamination) and operator credentials (GH_TOKEN, OPENAI_API_KEY, …). A
|
||||
* provider runner re-admits its own auth via opts.extraAllow. */
|
||||
const ALLOW_PREFIXES = ['EVALS_', 'GITHUB_'];
|
||||
|
||||
export interface HermeticEnvOpts {
|
||||
/** Additional allowed names (exact) or prefixes (entries ending in '*').
|
||||
* Example: the codex runner passes ['OPENAI_API_KEY', 'CODEX_*']. */
|
||||
extraAllow?: string[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Pure form of the GSTACK_ → canonical promotion. Returns a copy of `base`
|
||||
* with ANTHROPIC_API_KEY / OPENAI_API_KEY filled from their GSTACK_-prefixed
|
||||
* form when the canonical is empty. Never mutates `base`.
|
||||
*/
|
||||
export function promotedEnv(base: NodeJS.ProcessEnv): NodeJS.ProcessEnv {
|
||||
const out: NodeJS.ProcessEnv = { ...base };
|
||||
for (const key of ['ANTHROPIC_API_KEY', 'OPENAI_API_KEY'] as const) {
|
||||
if (!out[key] && out[`GSTACK_${key}`]) out[key] = out[`GSTACK_${key}`];
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a scrubbed child env: promote GSTACK_ keys, keep only the allowlisted
|
||||
* names (+ caller's extraAllow), then spread the caller's overrides LAST so a
|
||||
* per-test HOME / CLAUDE_CONFIG_DIR / CODEX_HOME / GBRAIN_HOME always wins.
|
||||
* Reads process.env at CALL time.
|
||||
*/
|
||||
export function hermeticChildEnv(
|
||||
overrides: Record<string, string | undefined> = {},
|
||||
opts?: HermeticEnvOpts,
|
||||
): NodeJS.ProcessEnv {
|
||||
const promoted = promotedEnv(process.env);
|
||||
|
||||
const extraExact = new Set<string>();
|
||||
const extraPrefixes: string[] = [];
|
||||
for (const entry of opts?.extraAllow ?? []) {
|
||||
if (entry.endsWith('*')) extraPrefixes.push(entry.slice(0, -1));
|
||||
else extraExact.add(entry);
|
||||
}
|
||||
|
||||
const out: NodeJS.ProcessEnv = {};
|
||||
for (const [k, v] of Object.entries(promoted)) {
|
||||
if (v === undefined) continue;
|
||||
const allowed =
|
||||
ALLOW_EXACT.has(k) ||
|
||||
extraExact.has(k) ||
|
||||
ALLOW_PREFIXES.some((p) => k.startsWith(p)) ||
|
||||
extraPrefixes.some((p) => k.startsWith(p));
|
||||
if (allowed) out[k] = v;
|
||||
}
|
||||
if (!out.TERM) out.TERM = 'xterm-256color';
|
||||
for (const [k, v] of Object.entries(overrides)) {
|
||||
if (v !== undefined) out[k] = v;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
// ────────────────────────────────────────────────────────────────────────────
|
||||
// 2. Binary resolution
|
||||
// ────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
function whichBin(name: string): string | null {
|
||||
try {
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
const found = (Bun as any).which?.(name);
|
||||
return found || null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
function firstExecutable(candidates: string[]): string | null {
|
||||
for (const c of candidates) {
|
||||
if (!c) continue;
|
||||
try {
|
||||
fs.accessSync(c, fs.constants.X_OK);
|
||||
return c;
|
||||
} catch {
|
||||
/* keep searching */
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/** Locate the real `claude` binary. Bun.which first, then known install dirs. */
|
||||
export function resolveClaudeBinary(): string | null {
|
||||
const which = whichBin('claude');
|
||||
if (which) return which;
|
||||
const home = process.env.HOME ?? os.homedir();
|
||||
return firstExecutable([
|
||||
'/opt/homebrew/bin/claude',
|
||||
'/usr/local/bin/claude',
|
||||
`${home}/.local/bin/claude`,
|
||||
`${home}/.bun/bin/claude`,
|
||||
`${home}/.npm-global/bin/claude`,
|
||||
]);
|
||||
}
|
||||
|
||||
/** Locate the real `codex` binary. Bun.which first, then known install dirs
|
||||
* (adds ~/.nvm + common node bin dirs where the npm global lands). */
|
||||
export function resolveCodexBinary(): string | null {
|
||||
const which = whichBin('codex');
|
||||
if (which) return which;
|
||||
const home = process.env.HOME ?? os.homedir();
|
||||
const candidates = [
|
||||
'/opt/homebrew/bin/codex',
|
||||
'/usr/local/bin/codex',
|
||||
`${home}/.local/bin/codex`,
|
||||
`${home}/.bun/bin/codex`,
|
||||
`${home}/.npm-global/bin/codex`,
|
||||
`${home}/.cargo/bin/codex`,
|
||||
];
|
||||
// ~/.nvm/versions/node/*/bin/codex and any dir already on PATH.
|
||||
try {
|
||||
const nvmBase = path.join(home, '.nvm', 'versions', 'node');
|
||||
for (const v of fs.readdirSync(nvmBase)) {
|
||||
candidates.push(path.join(nvmBase, v, 'bin', 'codex'));
|
||||
}
|
||||
} catch {
|
||||
/* no nvm */
|
||||
}
|
||||
for (const dir of (process.env.PATH ?? '').split(path.delimiter)) {
|
||||
if (dir) candidates.push(path.join(dir, 'codex'));
|
||||
}
|
||||
return firstExecutable(candidates);
|
||||
}
|
||||
|
||||
// ────────────────────────────────────────────────────────────────────────────
|
||||
// 3. Auth probes (drive skipIf in the door tests)
|
||||
// ────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
/** Claude Code is usable if an Anthropic key is exported (either form) OR the
|
||||
* operator has a real ~/.claude.json (subscription auth). */
|
||||
export function hasClaudeAuth(): boolean {
|
||||
if (process.env.ANTHROPIC_API_KEY || process.env.GSTACK_ANTHROPIC_API_KEY) return true;
|
||||
try {
|
||||
return fs.existsSync(path.join(os.homedir(), '.claude.json'));
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/** Codex is usable if the operator has a real ~/.codex/auth.json. */
|
||||
export function hasCodexAuth(): boolean {
|
||||
try {
|
||||
return fs.existsSync(path.join(os.homedir(), '.codex', 'auth.json'));
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
// ────────────────────────────────────────────────────────────────────────────
|
||||
// 4. Stream parsers (pure — exercised by the unit test with fixtures)
|
||||
// ────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
export interface ParsedClaudeStream {
|
||||
/** The final assistant/result text of the turn. */
|
||||
finalText: string;
|
||||
/** Names of every tool_use block the assistant emitted, in order. */
|
||||
toolCalls: string[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse `claude -p --output-format stream-json` NDJSON. Collects tool_use
|
||||
* names from assistant events and the final answer text. Prefers the terminal
|
||||
* `result` event's `result` field; falls back to the concatenation of the last
|
||||
* assistant message's text blocks. Skips malformed lines.
|
||||
*/
|
||||
export function parseClaudeStream(lines: string[]): ParsedClaudeStream {
|
||||
const toolCalls: string[] = [];
|
||||
let resultText: string | null = null;
|
||||
let lastAssistantText = '';
|
||||
|
||||
for (const line of lines) {
|
||||
if (!line.trim()) continue;
|
||||
let event: any;
|
||||
try {
|
||||
event = JSON.parse(line);
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
if (event.type === 'assistant') {
|
||||
const content = event.message?.content ?? [];
|
||||
const textParts: string[] = [];
|
||||
for (const item of content) {
|
||||
if (item?.type === 'tool_use') toolCalls.push(item.name || 'unknown');
|
||||
else if (item?.type === 'text' && typeof item.text === 'string') textParts.push(item.text);
|
||||
}
|
||||
if (textParts.length > 0) lastAssistantText = textParts.join('');
|
||||
} else if (event.type === 'result') {
|
||||
if (typeof event.result === 'string') resultText = event.result;
|
||||
}
|
||||
}
|
||||
|
||||
return { finalText: (resultText ?? lastAssistantText) || '', toolCalls };
|
||||
}
|
||||
|
||||
export interface ParsedCodexJsonl {
|
||||
/** Concatenated agent_message text. */
|
||||
finalText: string;
|
||||
/** command_execution commands, in order. */
|
||||
toolCalls: string[];
|
||||
/** reasoning item text blocks, in order. */
|
||||
reasoning: string[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse `codex exec --json` JSONL. Extracts agent_message → finalText,
|
||||
* command_execution → toolCalls, reasoning → reasoning. Skips malformed lines.
|
||||
*/
|
||||
export function parseCodexJsonl(lines: string[]): ParsedCodexJsonl {
|
||||
const outputParts: string[] = [];
|
||||
const toolCalls: string[] = [];
|
||||
const reasoning: string[] = [];
|
||||
|
||||
for (const line of lines) {
|
||||
if (!line.trim()) continue;
|
||||
let obj: any;
|
||||
try {
|
||||
obj = JSON.parse(line);
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
if (obj.type === 'item.completed' && obj.item) {
|
||||
const item = obj.item;
|
||||
const text = item.text || '';
|
||||
if (item.type === 'reasoning' && text) reasoning.push(text);
|
||||
else if (item.type === 'agent_message' && text) outputParts.push(text);
|
||||
else if (item.type === 'command_execution' && item.command) toolCalls.push(item.command);
|
||||
}
|
||||
}
|
||||
|
||||
return { finalText: outputParts.join('\n'), toolCalls, reasoning };
|
||||
}
|
||||
|
||||
// ────────────────────────────────────────────────────────────────────────────
|
||||
// 5. Real-binary turns
|
||||
// ────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
/** Read a piped stream to a string of NDJSON/JSONL lines, calling onLine for
|
||||
* each complete line. Returns all collected lines. */
|
||||
async function streamLines(
|
||||
stream: ReadableStream<Uint8Array>,
|
||||
collected: string[],
|
||||
): Promise<void> {
|
||||
const reader = stream.getReader();
|
||||
const decoder = new TextDecoder();
|
||||
let buf = '';
|
||||
try {
|
||||
while (true) {
|
||||
const { done, value } = await reader.read();
|
||||
if (done) break;
|
||||
buf += decoder.decode(value, { stream: true });
|
||||
const parts = buf.split('\n');
|
||||
buf = parts.pop() || '';
|
||||
for (const line of parts) {
|
||||
if (line.trim()) collected.push(line);
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
/* stream cancelled (timeout) or read error — fall through */
|
||||
}
|
||||
if (buf.trim()) collected.push(buf);
|
||||
}
|
||||
|
||||
export interface ClaudeTurnOpts {
|
||||
prompt: string;
|
||||
cwd: string;
|
||||
home: string;
|
||||
claudeConfigDir: string;
|
||||
mcpConfigPath?: string;
|
||||
model?: string;
|
||||
timeoutMs?: number;
|
||||
}
|
||||
|
||||
export interface ClaudeTurnResult {
|
||||
finalText: string;
|
||||
toolCalls: string[];
|
||||
rawLines: string[];
|
||||
exitCode: number | null;
|
||||
timedOut: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* Drive one headless `claude -p` turn against a hermetic HOME/config dir.
|
||||
* Optionally wires an MCP config (with --strict-mcp-config so ONLY that server
|
||||
* is loaded — no operator MCP contamination). Prompt is piped via stdin.
|
||||
*/
|
||||
export async function claudeHeadlessTurn(opts: ClaudeTurnOpts): Promise<ClaudeTurnResult> {
|
||||
const model = opts.model ?? 'claude-sonnet-4-6';
|
||||
const timeoutMs = opts.timeoutMs ?? 180_000;
|
||||
|
||||
const args = [
|
||||
'-p',
|
||||
'--model', model,
|
||||
'--output-format', 'stream-json',
|
||||
'--verbose',
|
||||
'--dangerously-skip-permissions',
|
||||
...(opts.mcpConfigPath ? ['--mcp-config', opts.mcpConfigPath, '--strict-mcp-config'] : []),
|
||||
];
|
||||
|
||||
const proc = Bun.spawn(['claude', ...args], {
|
||||
cwd: opts.cwd,
|
||||
env: hermeticChildEnv({ HOME: opts.home, CLAUDE_CONFIG_DIR: opts.claudeConfigDir }),
|
||||
stdin: 'pipe',
|
||||
stdout: 'pipe',
|
||||
stderr: 'pipe',
|
||||
});
|
||||
|
||||
// Write the prompt then close stdin.
|
||||
proc.stdin.write(opts.prompt);
|
||||
await proc.stdin.end();
|
||||
|
||||
let timedOut = false;
|
||||
const rawLines: string[] = [];
|
||||
const stdoutDone = streamLines(proc.stdout, rawLines);
|
||||
const stderrDone = new Response(proc.stderr).text();
|
||||
|
||||
const timer = setTimeout(() => {
|
||||
timedOut = true;
|
||||
try { proc.kill(); } catch { /* already dead */ }
|
||||
}, timeoutMs);
|
||||
|
||||
await stdoutDone;
|
||||
await stderrDone.catch(() => '');
|
||||
const exitCode = await proc.exited;
|
||||
clearTimeout(timer);
|
||||
|
||||
const parsed = parseClaudeStream(rawLines);
|
||||
return { finalText: parsed.finalText, toolCalls: parsed.toolCalls, rawLines, exitCode, timedOut };
|
||||
}
|
||||
|
||||
export interface CodexTurnOpts {
|
||||
prompt: string;
|
||||
cwd: string;
|
||||
home: string;
|
||||
timeoutMs?: number;
|
||||
sandbox?: string;
|
||||
}
|
||||
|
||||
export interface CodexTurnResult {
|
||||
finalText: string;
|
||||
toolCalls: string[];
|
||||
reasoning: string[];
|
||||
rawLines: string[];
|
||||
exitCode: number | null;
|
||||
timedOut: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* Drive one `codex exec` turn against a hermetic HOME. Copies the operator's
|
||||
* real ~/.codex/* (except skills/) into <home>/.codex so codex authenticates
|
||||
* without touching the real config dir. Parses JSONL output.
|
||||
*/
|
||||
export async function codexExecTurn(opts: CodexTurnOpts): Promise<CodexTurnResult> {
|
||||
const timeoutMs = opts.timeoutMs ?? 240_000;
|
||||
const sandbox = opts.sandbox ?? 'workspace-write';
|
||||
|
||||
// Seed the temp HOME's .codex from the operator's real auth (read-only copy,
|
||||
// skipping skills/ so a downstream test can install its own).
|
||||
const realCodex = path.join(os.homedir(), '.codex');
|
||||
const tempCodex = path.join(opts.home, '.codex');
|
||||
fs.mkdirSync(tempCodex, { recursive: true });
|
||||
if (fs.existsSync(realCodex)) {
|
||||
for (const entry of fs.readdirSync(realCodex)) {
|
||||
if (entry === 'skills') continue;
|
||||
const src = path.join(realCodex, entry);
|
||||
const dst = path.join(tempCodex, entry);
|
||||
if (!fs.existsSync(dst)) {
|
||||
try { fs.cpSync(src, dst, { recursive: true }); } catch { /* best-effort */ }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const proc = Bun.spawn(['codex', 'exec', opts.prompt, '--json', '-s', sandbox], {
|
||||
cwd: opts.cwd,
|
||||
env: hermeticChildEnv({ HOME: opts.home }, { extraAllow: ['OPENAI_API_KEY', 'CODEX_*'] }),
|
||||
stdout: 'pipe',
|
||||
stderr: 'pipe',
|
||||
});
|
||||
|
||||
let timedOut = false;
|
||||
const rawLines: string[] = [];
|
||||
const stdoutDone = streamLines(proc.stdout, rawLines);
|
||||
const stderrDone = new Response(proc.stderr).text();
|
||||
|
||||
const timer = setTimeout(() => {
|
||||
timedOut = true;
|
||||
try { proc.kill(); } catch { /* already dead */ }
|
||||
}, timeoutMs);
|
||||
|
||||
await stdoutDone;
|
||||
await stderrDone.catch(() => '');
|
||||
const exitCode = await proc.exited;
|
||||
clearTimeout(timer);
|
||||
|
||||
const parsed = parseCodexJsonl(rawLines);
|
||||
return {
|
||||
finalText: parsed.finalText,
|
||||
toolCalls: parsed.toolCalls,
|
||||
reasoning: parsed.reasoning,
|
||||
rawLines,
|
||||
exitCode: timedOut ? 124 : exitCode,
|
||||
timedOut,
|
||||
};
|
||||
}
|
||||
|
||||
// ────────────────────────────────────────────────────────────────────────────
|
||||
// 5b. Fast gbrain MCP server command (compiled binary, cached; bun-run fallback)
|
||||
// ────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* A resolved launch spec for the gbrain MCP stdio server. The door tests
|
||||
* register THIS as their MCP server so the child agent's first tool call finds
|
||||
* a server that is already up. A compiled binary starts fast; `bun run
|
||||
* src/cli.ts serve` cold-transpiles the whole CLI on every spawn and can miss
|
||||
* an agent's tool-call window (the source of the codex "the available gbrain
|
||||
* mcp calls were cancelled" flake). `kind` lets the caller widen the turn
|
||||
* timeout when we fell back to the slow path.
|
||||
*/
|
||||
export interface GbrainServerCommand {
|
||||
command: string;
|
||||
/** Full arg vector INCLUDING the `serve` subcommand + any extra flags. */
|
||||
args: string[];
|
||||
kind: 'compiled' | 'bun-run';
|
||||
}
|
||||
|
||||
// Module-level cache: compile the binary at most once per test process.
|
||||
let _compiledBin: string | null = null;
|
||||
let _compileTried = false;
|
||||
let _compileReason = '';
|
||||
let _compileBuildDir: string | null = null;
|
||||
|
||||
/**
|
||||
* Verify a freshly-built `gbrain` binary can actually stand up a PGLite brain.
|
||||
* `bun build --compile` bundles the code into a read-only vfs (`/$bunfs/root`)
|
||||
* but does NOT embed PGLite's WASM runtime + extension tarballs (`pglite.data`,
|
||||
* `vector.tar.gz`, `pg_trgm.tar.gz`), so a compiled `gbrain init --pglite`
|
||||
* fails at `initSchema()` with an ENOENT on those assets (Bun vfs #1340). A
|
||||
* compiled binary that can't open PGLite is useless as our door-test MCP server
|
||||
* (the brain is keyless PGLite), so we probe before trusting it. `init` is the
|
||||
* cheapest exercise of the WASM path that terminates on its own (unlike
|
||||
* `serve`, which either waits on stdin or bails early on a brain-less home).
|
||||
*/
|
||||
function probeCompiledPglite(binPath: string): { ok: boolean; reason: string } {
|
||||
const probeHome = fs.mkdtempSync(path.join(os.tmpdir(), 'gb-agent-probe-'));
|
||||
try {
|
||||
const res = spawnSync(binPath, ['init', '--pglite', '--no-embedding', '--non-interactive'], {
|
||||
env: { ...process.env, HOME: probeHome, GBRAIN_HOME: probeHome, GBRAIN_SKIP_STARTUP_HOOKS: '1' },
|
||||
encoding: 'utf8',
|
||||
timeout: 60_000,
|
||||
stdio: ['ignore', 'pipe', 'pipe'],
|
||||
});
|
||||
const stderr = `${res.stdout ?? ''}\n${res.stderr ?? ''}`;
|
||||
const wasmBroken = /PGLite failed to initialize|Extension bundle not found|\$bunfs|pglite\.data/.test(stderr);
|
||||
if (res.error) return { ok: false, reason: `compiled \`gbrain init --pglite\` probe error: ${res.error.message}` };
|
||||
if (wasmBroken || res.status !== 0) {
|
||||
const lines = stderr.split('\n').map((l) => l.trim()).filter(Boolean);
|
||||
const markerLine =
|
||||
lines.find((l) => /PGLite failed to initialize|Extension bundle not found|\$bunfs|pglite\.data/.test(l)) ??
|
||||
lines[0] ??
|
||||
`exit ${res.status}`;
|
||||
return {
|
||||
ok: false,
|
||||
reason:
|
||||
`compiled \`gbrain\` cannot open a PGLite brain — \`bun build --compile\` does not embed ` +
|
||||
`PGLite's WASM/extension payload (Bun vfs #1340): ${markerLine.slice(0, 160)}`,
|
||||
};
|
||||
}
|
||||
return { ok: true, reason: '' };
|
||||
} catch (e) {
|
||||
return { ok: false, reason: `compiled PGLite probe threw: ${(e as Error).message}` };
|
||||
} finally {
|
||||
try { fs.rmSync(probeHome, { recursive: true, force: true }); } catch { /* best-effort */ }
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a standalone `gbrain` binary once via `bun build --compile` into a
|
||||
* cached temp path AND verify it can serve a PGLite brain. Returns the path (or
|
||||
* null + a reason) — null covers both "compile unavailable in this sandbox" and
|
||||
* "compile succeeds but the binary can't open PGLite" (#1340). Mirrors
|
||||
* test/e2e/bootstrap-compiled-binary.serial.test.ts. Fail-soft: every error is
|
||||
* captured as a reason string, never thrown, so any failure degrades to the
|
||||
* bun-run fallback instead of hard-failing the door suite.
|
||||
*/
|
||||
export function ensureCompiledGbrain(repoRoot: string): { binPath: string | null; reason: string } {
|
||||
if (_compileTried) return { binPath: _compiledBin, reason: _compileReason };
|
||||
_compileTried = true;
|
||||
try {
|
||||
const buildDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gb-agent-bin-'));
|
||||
_compileBuildDir = buildDir;
|
||||
const binPath = path.join(buildDir, 'gbrain');
|
||||
const res = spawnSync('bun', ['build', '--compile', '--outfile', binPath, 'src/cli.ts'], {
|
||||
cwd: repoRoot,
|
||||
encoding: 'utf8',
|
||||
timeout: 240_000,
|
||||
maxBuffer: 64 * 1024 * 1024,
|
||||
});
|
||||
if (res.error) {
|
||||
_compileReason = `bun build --compile unavailable: ${res.error.message}`;
|
||||
} else if (res.status !== 0) {
|
||||
_compileReason = `bun build --compile exited ${res.status}: ${(res.stderr ?? '').slice(-1000)}`;
|
||||
} else if (!fs.existsSync(binPath)) {
|
||||
_compileReason = 'bun build --compile exited 0 but produced no binary';
|
||||
} else {
|
||||
const probe = probeCompiledPglite(binPath);
|
||||
if (probe.ok) _compiledBin = binPath;
|
||||
else _compileReason = probe.reason;
|
||||
}
|
||||
} catch (e) {
|
||||
_compileReason = `bun build --compile threw: ${(e as Error).message}`;
|
||||
}
|
||||
return { binPath: _compiledBin, reason: _compileReason };
|
||||
}
|
||||
|
||||
// Best-effort cleanup of the compiled-binary temp dir at process exit.
|
||||
process.on('exit', () => {
|
||||
if (_compileBuildDir) {
|
||||
try { fs.rmSync(_compileBuildDir, { recursive: true, force: true }); } catch { /* best-effort */ }
|
||||
}
|
||||
});
|
||||
|
||||
/**
|
||||
* Resolve the gbrain MCP server launch spec: a compiled binary
|
||||
* (`{command:<bin>, args:['serve', ...extra]}`) when `bun build --compile`
|
||||
* both succeeds AND can open a PGLite brain, else the `bun run src/cli.ts
|
||||
* serve` fallback. `extraArgs` are appended after `serve` (e.g.
|
||||
* `['--surface','full']` for the codex door). The env
|
||||
* (GBRAIN_HOME/GBRAIN_SOURCE) is layered on by the caller and stays identical
|
||||
* across both kinds.
|
||||
*
|
||||
* NOTE: with the current `bun build --compile` command the compiled binary
|
||||
* canNOT serve a PGLite brain (its WASM/extension payload isn't embedded —
|
||||
* Bun vfs #1340), so this resolves to `bun-run` in practice today. The
|
||||
* bun-run server is measured ready to answer `tools/list` in ~300ms, so the
|
||||
* door tests lean on the bounded SMOKE retry (not server speed) for
|
||||
* robustness. The compiled path stays wired so this self-heals the day the
|
||||
* build embeds those assets.
|
||||
*/
|
||||
export function resolveGbrainServerCommand(repoRoot: string, extraArgs: string[] = []): GbrainServerCommand {
|
||||
const { binPath, reason } = ensureCompiledGbrain(repoRoot);
|
||||
if (binPath) {
|
||||
return { command: binPath, args: ['serve', ...extraArgs], kind: 'compiled' };
|
||||
}
|
||||
console.error(
|
||||
`[agent-harness] compiled gbrain unavailable, falling back to \`bun run src/cli.ts\` ` +
|
||||
`(slower MCP startup, wider readiness window): ${reason}`,
|
||||
);
|
||||
return {
|
||||
command: 'bun',
|
||||
args: ['run', path.join(repoRoot, 'src', 'cli.ts'), 'serve', ...extraArgs],
|
||||
kind: 'bun-run',
|
||||
};
|
||||
}
|
||||
|
||||
// ────────────────────────────────────────────────────────────────────────────
|
||||
// 6. MCP config for a Claude Code door test
|
||||
// ────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
export interface GbrainMcpConfigOpts {
|
||||
path: string;
|
||||
/** Resolved server launch spec (compiled bin or bun-run fallback). */
|
||||
server: { command: string; args: string[] };
|
||||
gbrainHome: string;
|
||||
sourceId: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Write a Claude Code `--mcp-config` JSON that runs THIS repo's gbrain over
|
||||
* stdio, pinned to a hermetic GBRAIN_HOME + source. The server command comes
|
||||
* from `resolveGbrainServerCommand` (compiled binary preferred, `bun run
|
||||
* src/cli.ts serve` fallback) so startup is fast enough that the child agent's
|
||||
* first tool call doesn't get cancelled. Mirrors `claude mcp add gbrain --
|
||||
* gbrain serve` from docs/mcp/CLAUDE_CODE.md.
|
||||
*/
|
||||
export function writeGbrainMcpConfig(opts: GbrainMcpConfigOpts): void {
|
||||
const cfg = {
|
||||
mcpServers: {
|
||||
gbrain: {
|
||||
command: opts.server.command,
|
||||
args: opts.server.args,
|
||||
env: {
|
||||
GBRAIN_HOME: opts.gbrainHome,
|
||||
GBRAIN_SOURCE: opts.sourceId,
|
||||
},
|
||||
},
|
||||
},
|
||||
};
|
||||
fs.mkdirSync(path.dirname(opts.path), { recursive: true });
|
||||
fs.writeFileSync(opts.path, JSON.stringify(cfg, null, 2));
|
||||
}
|
||||
|
||||
// ────────────────────────────────────────────────────────────────────────────
|
||||
// 7. Seed a real keyless PGLite brain the spawned `gbrain serve` will read
|
||||
// ────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
const put_page = operations.find((o) => o.name === 'put_page') as Operation | undefined;
|
||||
|
||||
export interface SeededBrain {
|
||||
fact: string;
|
||||
entity: string;
|
||||
query: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Initialize a keyless PGLite brain at GBRAIN_HOME=<home> and seed ONE page
|
||||
* with a distinctive, 100%-synthetic fact so a door test can assert the agent
|
||||
* recalls it over MCP. Persistent on disk (so the spawned `gbrain serve`
|
||||
* subprocess reads the same brain), keyless (embedding_disabled) so it runs
|
||||
* with no API key, skills published so the verbs surface is available.
|
||||
*
|
||||
* Temporarily pins process.env.GBRAIN_HOME while creating the brain, then
|
||||
* restores it — the door test sets GBRAIN_HOME on the spawned child via the
|
||||
* MCP config's env block, not on this process.
|
||||
*/
|
||||
export async function seedBrainForAgent(home: string, sourceId: string): Promise<SeededBrain> {
|
||||
if (!put_page) throw new Error('seedBrainForAgent: put_page op not registered');
|
||||
|
||||
const entity = 'Summit Robotics';
|
||||
const fact = 'Summit Robotics runs the Rivermouth fulfillment center.';
|
||||
const query = 'Where does Summit Robotics run its fulfillment center?';
|
||||
|
||||
const savedHome = process.env.GBRAIN_HOME;
|
||||
process.env.GBRAIN_HOME = home;
|
||||
try {
|
||||
const dbPath = gbrainPath('brain.pglite'); // <home>/.gbrain/brain.pglite
|
||||
|
||||
const engine = new PGLiteEngine();
|
||||
await engine.connect({ database_path: dbPath, engine: 'pglite' });
|
||||
await engine.initSchema();
|
||||
try {
|
||||
// Register a non-default source so GBRAIN_SOURCE routing has a real row.
|
||||
if (sourceId !== 'default') {
|
||||
const srcDir = path.join(home, `source-${sourceId}`);
|
||||
fs.mkdirSync(srcDir, { recursive: true });
|
||||
try {
|
||||
await addSource(engine, { id: sourceId, localPath: srcDir, force: true });
|
||||
} catch {
|
||||
/* already registered — fine */
|
||||
}
|
||||
}
|
||||
|
||||
const ctx: OperationContext = {
|
||||
engine,
|
||||
config: { engine: 'pglite' } as never,
|
||||
logger: { info: () => {}, warn: () => {}, error: () => {} },
|
||||
dryRun: false,
|
||||
remote: false,
|
||||
sourceId,
|
||||
};
|
||||
await put_page.handler(ctx, {
|
||||
slug: 'companies/summit-robotics',
|
||||
content: `# Summit Robotics\n\n${fact}\n`,
|
||||
});
|
||||
} finally {
|
||||
// Release the PGLite lock so the spawned `gbrain serve` can open the
|
||||
// same data dir.
|
||||
try { await engine.disconnect(); } catch { /* best-effort */ }
|
||||
}
|
||||
|
||||
// Persist a keyless config so the spawned `gbrain serve` reads the same
|
||||
// engine + path and doesn't try to reach an embedding provider.
|
||||
const config: GBrainConfig = {
|
||||
engine: 'pglite',
|
||||
database_path: dbPath,
|
||||
embedding_disabled: true,
|
||||
mcp: { publish_skills: true },
|
||||
} as unknown as GBrainConfig;
|
||||
saveConfig(config);
|
||||
} finally {
|
||||
if (savedHome === undefined) delete process.env.GBRAIN_HOME;
|
||||
else process.env.GBRAIN_HOME = savedHome;
|
||||
}
|
||||
|
||||
return { fact, entity, query };
|
||||
}
|
||||
@@ -0,0 +1,176 @@
|
||||
/**
|
||||
* Unit coverage for the real-agent door harness that needs NO real binary and
|
||||
* pays ZERO API cost. Exercises:
|
||||
* - parseClaudeStream against a captured `claude -p --output-format
|
||||
* stream-json` NDJSON fixture (tool_use names + final result text).
|
||||
* - parseCodexJsonl against a captured `codex exec --json` JSONL fixture
|
||||
* (command_execution → toolCalls, agent_message → finalText, reasoning).
|
||||
* - hermeticChildEnv: drops CONDUCTOR_* / CLAUDE_* / GSTACK_* / MCP_* /
|
||||
* GBRAIN_*, promotes GSTACK_ANTHROPIC_API_KEY, honors extraAllow, and lets
|
||||
* overrides win.
|
||||
* - resolveClaudeBinary / resolveCodexBinary SMOKE (whatever this machine
|
||||
* has — assertion is only that the result is a string-or-null, plus a note
|
||||
* printed when found).
|
||||
*
|
||||
* The env test mutates process.env and restores it in finally so it never
|
||||
* leaks into sibling tests.
|
||||
*/
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import {
|
||||
parseClaudeStream,
|
||||
parseCodexJsonl,
|
||||
hermeticChildEnv,
|
||||
promotedEnv,
|
||||
resolveClaudeBinary,
|
||||
resolveCodexBinary,
|
||||
} from './agent-harness.ts';
|
||||
import { withEnv } from './with-env.ts';
|
||||
|
||||
// A captured claude stream-json turn: a system init line, an assistant text +
|
||||
// tool_use turn, a tool_result user line, a second assistant text turn, and the
|
||||
// terminal result line. Trailing blank + one malformed line prove the parser
|
||||
// tolerates both.
|
||||
const CLAUDE_NDJSON = [
|
||||
'{"type":"system","subtype":"init","session_id":"abc","tools":["mcp__gbrain__recall"]}',
|
||||
'{"type":"assistant","message":{"content":[{"type":"text","text":"Let me search the brain."},{"type":"tool_use","id":"tu_1","name":"mcp__gbrain__recall","input":{"query":"fulfillment center"}}]}}',
|
||||
'{"type":"user","message":{"content":[{"type":"tool_result","tool_use_id":"tu_1","content":"Summit Robotics runs the Rivermouth fulfillment center."}]}}',
|
||||
'{"type":"assistant","message":{"content":[{"type":"text","text":"Summit Robotics runs the Rivermouth fulfillment center."}]}}',
|
||||
'',
|
||||
'this is not json and must be skipped',
|
||||
'{"type":"result","subtype":"success","is_error":false,"result":"Summit Robotics runs the Rivermouth fulfillment center.","num_turns":2,"total_cost_usd":0.01}',
|
||||
];
|
||||
|
||||
// A captured codex exec --json turn: thread.started, a reasoning item, a
|
||||
// command_execution item, an agent_message item, turn.completed, plus a blank
|
||||
// and a malformed line.
|
||||
const CODEX_JSONL = [
|
||||
'{"type":"thread.started","thread_id":"th_123"}',
|
||||
'{"type":"item.completed","item":{"type":"reasoning","text":"I should read the file to answer."}}',
|
||||
'{"type":"item.completed","item":{"type":"command_execution","command":"cat facts.md","exit_code":0}}',
|
||||
'{"type":"item.completed","item":{"type":"agent_message","text":"The fulfillment center is Rivermouth."}}',
|
||||
'',
|
||||
'{oops not json',
|
||||
'{"type":"turn.completed","usage":{"input_tokens":100,"output_tokens":20}}',
|
||||
];
|
||||
|
||||
describe('parseClaudeStream', () => {
|
||||
test('extracts tool_use names and the final result text', () => {
|
||||
const parsed = parseClaudeStream(CLAUDE_NDJSON);
|
||||
expect(parsed.toolCalls).toEqual(['mcp__gbrain__recall']);
|
||||
expect(parsed.finalText).toBe('Summit Robotics runs the Rivermouth fulfillment center.');
|
||||
});
|
||||
|
||||
test('falls back to last assistant text when no result line', () => {
|
||||
const noResult = CLAUDE_NDJSON.filter((l) => !l.includes('"type":"result"'));
|
||||
const parsed = parseClaudeStream(noResult);
|
||||
expect(parsed.finalText).toBe('Summit Robotics runs the Rivermouth fulfillment center.');
|
||||
expect(parsed.toolCalls).toEqual(['mcp__gbrain__recall']);
|
||||
});
|
||||
|
||||
test('empty input yields empty result, never throws', () => {
|
||||
expect(parseClaudeStream([])).toEqual({ finalText: '', toolCalls: [] });
|
||||
});
|
||||
});
|
||||
|
||||
describe('parseCodexJsonl', () => {
|
||||
test('extracts command executions, agent message, and reasoning', () => {
|
||||
const parsed = parseCodexJsonl(CODEX_JSONL);
|
||||
expect(parsed.toolCalls).toEqual(['cat facts.md']);
|
||||
expect(parsed.finalText).toBe('The fulfillment center is Rivermouth.');
|
||||
expect(parsed.reasoning).toEqual(['I should read the file to answer.']);
|
||||
});
|
||||
|
||||
test('empty input yields empty result, never throws', () => {
|
||||
expect(parseCodexJsonl([])).toEqual({ finalText: '', toolCalls: [], reasoning: [] });
|
||||
});
|
||||
});
|
||||
|
||||
describe('promotedEnv', () => {
|
||||
test('promotes GSTACK_ANTHROPIC_API_KEY when canonical is unset', () => {
|
||||
const out = promotedEnv({ GSTACK_ANTHROPIC_API_KEY: 'sk-gstack' } as NodeJS.ProcessEnv);
|
||||
expect(out.ANTHROPIC_API_KEY).toBe('sk-gstack');
|
||||
});
|
||||
|
||||
test('does NOT clobber an existing canonical key', () => {
|
||||
const out = promotedEnv({
|
||||
ANTHROPIC_API_KEY: 'sk-real',
|
||||
GSTACK_ANTHROPIC_API_KEY: 'sk-gstack',
|
||||
} as NodeJS.ProcessEnv);
|
||||
expect(out.ANTHROPIC_API_KEY).toBe('sk-real');
|
||||
});
|
||||
});
|
||||
|
||||
describe('hermeticChildEnv', () => {
|
||||
test('drops CONDUCTOR_*/CLAUDE_*/GSTACK_*/MCP_*/GBRAIN_*, keeps PATH/HOME, promotes GSTACK key', async () => {
|
||||
// Contaminate the process env with everything a real Conductor + Claude
|
||||
// Code session would carry (ANTHROPIC_API_KEY unset so promotion shows).
|
||||
await withEnv(
|
||||
{
|
||||
CONDUCTOR_WORKSPACE_PATH: '/should/drop',
|
||||
CLAUDE_CODE_ENTRYPOINT: 'cli',
|
||||
CLAUDECODE: '1',
|
||||
MCP_SERVER: 'gbrain',
|
||||
GBRAIN_HOME: '/operator/.gbrain',
|
||||
GSTACK_HOME: '/operator/.gstack',
|
||||
ANTHROPIC_API_KEY: undefined,
|
||||
GSTACK_ANTHROPIC_API_KEY: 'sk-promote-me',
|
||||
},
|
||||
() => {
|
||||
const env = hermeticChildEnv({ HOME: '/tmp/hermetic-home', GBRAIN_HOME: '/tmp/hermetic-gbrain' });
|
||||
|
||||
// Dropped.
|
||||
expect(env.CONDUCTOR_WORKSPACE_PATH).toBeUndefined();
|
||||
expect(env.CLAUDE_CODE_ENTRYPOINT).toBeUndefined();
|
||||
expect(env.CLAUDECODE).toBeUndefined();
|
||||
expect(env.MCP_SERVER).toBeUndefined();
|
||||
expect(env.GSTACK_HOME).toBeUndefined();
|
||||
// GSTACK_ANTHROPIC_API_KEY itself is dropped (prefix), but its value was
|
||||
// promoted onto the allowlisted canonical name.
|
||||
expect(env.GSTACK_ANTHROPIC_API_KEY).toBeUndefined();
|
||||
expect(env.ANTHROPIC_API_KEY).toBe('sk-promote-me');
|
||||
|
||||
// Kept.
|
||||
expect(env.PATH).toBe(process.env.PATH);
|
||||
// Overrides win — HOME is the temp one, and a GBRAIN_HOME override is
|
||||
// honored even though the bare GBRAIN_ prefix is dropped.
|
||||
expect(env.HOME).toBe('/tmp/hermetic-home');
|
||||
expect(env.GBRAIN_HOME).toBe('/tmp/hermetic-gbrain');
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
test('extraAllow admits exact names and PREFIX_* forms (codex auth surface)', async () => {
|
||||
await withEnv(
|
||||
{
|
||||
OPENAI_API_KEY: 'sk-openai',
|
||||
CODEX_HOME: '/operator/.codex',
|
||||
CODEX_SANDBOX: 'workspace-write',
|
||||
},
|
||||
() => {
|
||||
const env = hermeticChildEnv({}, { extraAllow: ['OPENAI_API_KEY', 'CODEX_*'] });
|
||||
expect(env.OPENAI_API_KEY).toBe('sk-openai');
|
||||
expect(env.CODEX_HOME).toBe('/operator/.codex');
|
||||
expect(env.CODEX_SANDBOX).toBe('workspace-write');
|
||||
|
||||
// Without extraAllow, the same vars are dropped.
|
||||
const scrubbed = hermeticChildEnv({});
|
||||
expect(scrubbed.OPENAI_API_KEY).toBeUndefined();
|
||||
expect(scrubbed.CODEX_HOME).toBeUndefined();
|
||||
},
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('binary resolution SMOKE', () => {
|
||||
test('resolveClaudeBinary returns a string or null', () => {
|
||||
const bin = resolveClaudeBinary();
|
||||
expect(bin === null || typeof bin === 'string').toBe(true);
|
||||
if (bin) console.log(`[smoke] claude resolved at: ${bin}`);
|
||||
});
|
||||
|
||||
test('resolveCodexBinary returns a string or null', () => {
|
||||
const bin = resolveCodexBinary();
|
||||
expect(bin === null || typeof bin === 'string').toBe(true);
|
||||
if (bin) console.log(`[smoke] codex resolved at: ${bin}`);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user