Compare commits

..
Author SHA1 Message Date
Garry TanandClaude Fable 5 c5765171d3 Merge origin/master (v0.46.3.0) into garrytan/opencode-support
Trio kept at 0.46.4.0 (wave version, highest); CHANGELOG carries both
entries with 0.46.4.0 topmost; template-repo + flag registry regenerated;
llms rebuilt.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 21:29:17 -07:00
Garry TanandClaude Fable 5 9e678f467d Merge origin/master (v0.46.2.0) into garrytan/opencode-support
Trio kept at 0.46.4.0 (wave version, highest); CHANGELOG carries both
entries with 0.46.4.0 topmost; TODOS union-resolved (master's re-filed
pin-privacy candidate dropped — this branch ships it as DONE);
cli-flag-registry + template-repo regenerated; llms rebuilt.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 15:41:54 -07:00
Garry TanandClaude Fable 5 e38e4a845b v0.46.4.0 chore(release): re-bump 0.46.2.0 → 0.46.4.0 (version slots claimed by in-flight PRs)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 15:20:13 -07:00
Garry TanandClaude Fable 5 526595ce01 docs: cross-model doc-review fixes — opencode roster + probe/remote accuracy
Findings from the standard post-ship Codex doc review, verified against
the shipped code: the bootstrap guide's intro, install table, and door
inventory still described a two-client product (opencode added to all
three); INSTALL_FOR_AGENTS' grok section said the personal-agent path is
Claude Code/Codex only; OPENCODE.md called its recipe the bootstrap
"manual equivalent" (bootstrap additionally pins GBRAIN_SOURCE + full
surface), lacked the mcp-list trust caution the pin doc carries, and
never documented the connect --install / --force remote lane; the pin
doc's provisioning bullet now names the pack-verify-install posture the
CI job actually runs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 13:22:41 -07:00
Garry TanandClaude Fable 5 2d8c772171 docs: fold the post-review opencode fix-wave behaviors into KEY_FILES
Three current-state completions the fix agents didn't carry into the
per-file index: connect --agent opencode --install's --force semantics
(maps to the writer's allowReplaceOtherSource — ours-at-old-url
replaceable, foreign still refuses), removeOpencodeMcpEntry's
skipOtherSource option, and bootstrap uninstall's expectation-keyed
opencode sweep (both merged global filenames under the config-dir lock
plus the project file; other-workspace entries skipped with a note).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 13:14:33 -07:00
Garry TanandClaude Fable 5 bdef2dada5 v0.46.2.0 feat(opencode): full-parity client support — bootstrap, harness, connect, claw-test, e2e door
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 12:54:44 -07:00
Garry TanandClaude Fable 5 cc902834d5 chore: regen cli-flag-registry after review-fix waves
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 12:52:40 -07:00
Garry TanandClaude Fable 5 eaca6a94d8 fix: adversarial-review fix wave — cross-model (Claude + Codex) findings
P0: the registration probe no longer executes from the invoking cwd
(mkdtemp cwd for user scope; project scope skips the live probe —
parse-back is authoritative), so a cloned repo's committed opencode.json
can't gain code execution during bootstrap. Probe timeouts now kill the
child (SIGTERM→SIGKILL, bounded drain) instead of abandoning it over the
PGLite lock. Global writes reconcile mcp.<name> across BOTH merged
global filenames. Stale-target cleanup takes the target config-dir lock.
Backups are unique per operation; rollback is content-guarded and
remove-path backups tighten to 0600 when token-bearing. connect --force
now works on the opencode lane with url-appropriate refusal copy.
atomic-write cleans tmp litter and survives the exists/realpath race.
Bun-lane fingerprint is fail-closed on gbrain-less args. Scope answers
trim. bounded() clears its drain-cap timer. CI installs opencode from
byte-verified tarballs. Consent-semantics + hermetic-live-runner
follow-ups filed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 12:51:29 -07:00
Garry TanandClaude Fable 5 7df02405bc fix: pre-landing review fixes — review-army + red-team wave
Security: opencode error-path snippets render <paste-token-here> instead
of the live bearer; the harness opencode catch redacts like the claude
lane; test-harness child envs unconditionally drop GITHUB_TOKEN/ACTIONS_*.
Correctness: bootstrap-lock coverage for every shared-config writer
(hooks/connect/uninstall); uninstall + step-aside gate sweep BOTH global
opencode filenames (merged namespace); uninstall passes a sourceId
expectation and skips other-workspace entries; cross-kind fingerprint
matches classify ours-other-source instead of silently replacing;
dangling-symlink writes preserve the link; failed-smoke rollback restores
atomically; registration probe pins OPENCODE_DISABLE_AUTOUPDATE, a 20s
cap, ANSI-stripped exact-name matching. Guard: check-opencode-pin now
cross-checks per-platform integrities + every OPENCODE_VERSION copy.
Plus deny-path/uninstall/rollback/truth-table/symlink test coverage,
help-prose cosmetics, downgrade doc note, 3 P3 TODOs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 12:02:50 -07:00
Garry TanandClaude Fable 5 bf0e8795f8 fix(bootstrap): close two plan-audit gaps — ACCESS_POLICY opencode scope paragraph + doctor host:opencode pin
Plan-completion audit (ship Step 8) flagged both as PARTIAL: the
ACCESS_POLICY template's MCP-scope section didn't state opencode's
inverted default (user-global; project spawns with NO trust prompt),
and bootstrap_harness_health had no named pin proving an opencode
receipt flows through the host-generic filter. Template-repo regenerated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 11:13:26 -07:00
Garry TanandClaude Fable 5 a534240d2c test: coverage pins for the opencode channel widenings
Ship-audit additions: hook.ts --harness opencode flag-parse attribution
end-to-end, and 'opencode' membership in VOLUNTEER_CHANNELS +
isHarnessChannel (a regression here silently rebadges opencode deliveries
as claude-code).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 10:59:54 -07:00
Garry Tan 6db8dc8a34 Merge remote-tracking branch 'origin/master' into garrytan/opencode-support
# Conflicts:
#	TODOS.md
2026-08-15 10:45:27 -07:00
Garry TanandClaude Fable 5 750275533f chore(todos): file opencode-wave follow-ups + retire fired triggers by title
Door-adapter extraction (test-side) and cadence policy: DONE — the 4th-door
trigger fired. CI-tail composite action re-filed with the sharpened trigger
(first GREEN grok-door AND opencode-door dispatches). hermesChildEnv
GITHUB_* backport: DONE via the shared factory. PIN-doc privacy guard:
DONE (check-pin-doc-privacy.sh in verify). New follow-ups: first-dispatch
watch, plugin/event-system wiring (ambient-recall lane), BrainBench
adapter (with hermes+grok), connect --oauth authorization-code lane,
OPENCODE_CONFIG* re-observation on bumps, opencode-install PTY promotion.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 10:26:44 -07:00
Garry TanandClaude Fable 5 17e36adb57 dx(explore): opencode-install TTY scenario — bootstrap paste block under the real interactive TUI
Unlike grok's brain-only prompt, opencode gets the FULL bootstrap paste
block (it is a bootstrap-supported harness) under a hermetic HOME + both
XDG dirs, the double autoupdate kill (config seed + env), BROWSER=false so
a first-run can never bounce the operator's browser, and auth.json
pre-registered for the secret scrub. Keyless posture INVERTS the grok
scenario: the anonymous free tier means a --keyless run should COMPLETE
the flow — a sign-in wall here is itself a pin-refresh signal, and the
generic early-stop in runInstallSession records it as friction if it ever
appears.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 10:25:52 -07:00
Garry TanandClaude Fable 5 519c3748da docs: opencode across the install/testing/architecture surface
README client roster + remote-connect bullet (with the not-OpenClaw
disambiguation and the bootstrap-supported banner), INSTALL_FOR_AGENTS
'If you are opencode' block (routes bootstrap-capable readers to the
runbook; brain-only registration otherwise), docs/INSTALL per-client list,
MEMORY_VERBS register snippet, bootstrap guide (degradation-matrix row
naming the INVERTED scope default + rationale; harness-mode opencode
bullet; dx-explore scenario line), ambient-recall/push-context harness
mentions, and KEY_FILES current-state entries (opencode-json.ts,
atomic-write.ts, connect/harness/hooks/claw-test entry refreshes).
llms bundles regenerated (build:llms chaser).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 10:24:35 -07:00
Garry TanandClaude Fable 5 fcbf547841 ci(heavy): opencode-door job (day-one full posture, keyless SMOKE) + canary leg + pin guards + hermes installer re-pin
opencode-door takes hermes-door's triggers (nightly + labels + dispatch;
cadence policy: nightly for the NEWEST door agent) with grok-door's
internals — keyless-first ordering, secretless npm provisioning with
wrapper AND per-platform integrity pre-checks, pass-count + paid sentinels,
mid-job version-drift tripwire, evidence scrub RE-KEYED to
ANTHROPIC_API_KEY + auth.json (not XAI/mcp_credentials), unconditional
credential removal. No dedicated dispatch input (any workflow_dispatch
already passes the non-PR arm — an input would be dead yaml). The keyless
tier includes the nonce SMOKE (free tier), so the core door needs NO
secret; the paid anthropic leg rides the secret hermes-door already
consumes. opencode-door-canary lands IN-WAVE (schedule-only,
continue-on-error, unpinned latest): opencode ships near-continuously — a
red canary is a pin-refresh signal, never a gate. real-agent-e2e adds the
opencode door file + env pins.

Guards: check-opencode-pin.sh (stamp↔workflow parity, job-block anchored so
the UNPINNED canary leg cannot satisfy it; fail-closed when the door exists
without the pin doc) and check-pin-doc-privacy.sh (placeholder discipline
for ALL docs/mcp/*-CLI-PIN.md — no operator home paths, no key-shaped
material outside sha512 pins, no non-example emails), both in bun run
verify + guards-manifest, both with fixture-tree bun tests.

Maintenance: hermes-door installer pin refreshed (upstream install.sh
drifted past the prior digest — last two nightlies red; reviewed: the
--commit payload-pin path is intact and the payload pins are unchanged).
docs/TESTING.md gains the opencode door entry + the door cadence policy.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 10:21:12 -07:00
Garry TanandClaude Fable 5 422438bfe7 test(e2e): opencode door — split-gated real-binary e2e on the extracted family (keyless SMOKE included)
The door goes a step beyond grok's split gating: opencode's anonymous free
tier drives MCP tool calls with zero credentials (observed, load-bearing),
so even the nonce SMOKE runs keyless. Tiers: T1 bare-semver version pin
(the SST-vs-claimant discriminator), T2 INSTALL via the documented
'mcp add … -- gbrain serve --surface verbs' shape + the honest 'mcp list'
discriminator (spawns servers; ✓/✗ text asserted — exit code is 0 even on
failure), T2b spawn-gate CANARY (a project-config decoy is spawn-attempted
with no trust prompt — if this ever gates, the bootstrap user-global
default rationale changed: re-observe), T3 writer parity (gbrain's
opencode-json.ts output handshakes through the real binary; cross-tool
preservation both ways incl. the autoupdate seed), T4 keyless SMOKE
(per-run nonce + STRUCTURAL gbrain_* tool_use proof via parseOpencodeJsonl,
list preflight before any turn, 2 attempts), and the paid T5 anthropic leg
(hasOpencodeAuth-gated; self-validating models-gate pins the model id
BEFORE any spend). Hermeticity: HOME + both XDG dirs per child, tmp cwds,
config/credential tripwire over the operator's real opencode state,
checkout guard, --pure on every probe (mcp list autoloads plugins), --pure
placed BEFORE the '--' separator (a trailing append lands inside the server
command — caught live). run-e2e.sh scrubs the OPENCODE_ prefix.

Verified live: 6/6 pass in 35.8s (keyless tier + paid anthropic leg)
against the hermetically pinned opencode-ai@1.18.18.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 10:13:21 -07:00
Garry TanandClaude Fable 5 ccc9f171c0 refactor(test): door-family extraction in agent-harness — shared resolver/childEnv/spawn core; grok+hermes ported; opencode first consumer
The 'P3 — Door-adapter extraction + CI-tail composite action' TODO armed
this at 'the NEXT door agent (4th)' — opencode is the 4th. Test-side only
(the composite CI-tail action stays deferred until the first green
grok-door AND opencode-door dispatches; workflow yaml can't be proven
locally).

- makeBinaryResolver: one shape (fail-closed $*_BIN > which > landing
  spots + nvm/PATH sweeps); claude/codex/hermes/grok resolvers become
  factory products with identical candidate lists.
- makeAgentChildEnv + GITHUB_STEP_META_KEYS: hermeticChildEnv + per-agent
  overrides + key deletion + the step-metadata scrub + binDir prepend.
  hermesChildEnv GAINS the GITHUB_* deletion via the factory (the filed P2
  backport; truth-table extended).
- runOneShotSpawn: shared timeout/kill/kill-9-escalation/bounded-drain core;
  hermesOneShotTurn gains the escalation + bounded drain (strictly safer,
  nothing pinned the old unbounded wait); grokOneShotTurn is now a thin argv
  builder over it.
- 5a-opencode family (first consumer): resolveOpencodeBinary (fail-closed
  OPENCODE_BIN), hasOpencodeAuth (PAID-leg-only gate — the keyless free
  tier carries the core SMOKE), opencodeChildEnv (HOME + BOTH XDG dirs,
  anthropic re-admission, other-provider + OPENCODE_CONFIG* shadow-trio
  deletes), seedOpencodeConfig (config half of the double autoupdate kill),
  opencodeOneShotTurn, and parseOpencodeJsonl (event shapes pinned from the
  live v1.18.18 observation — {type,part} with part.text / part.tool).

EV1 gate (local keyless grok-door): 4 pass / paid-skip in 20.5s BEFORE and
AFTER the port, against a hermetically npm-pinned @xai-official/grok@1.0.4.
The hermes door self-skips without a binary — its port is pinned by the
unit truth-tables (stated honestly, per the plan).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 10:08:18 -07:00
Garry TanandClaude Fable 5 56a272f18d feat(claw-test): opencode runner — detection, pinned one-shot invoke, multi-provider env allowlist
OpencodeRunner (4th AgentRunner): detectBinary('OPENCODE_BIN','opencode');
argv pinned to 'run <brief> --format default' (explicit format so an
upstream default flip cannot silently change the transcript shape; NO
--auto — MCP tool calls fire in run mode without it, verified). Env
allowlist = BASE + an EXPLICIT multi-provider delta (XAI / Google / Gemini
/ OpenRouter keys — BASE carries only Anthropic+OpenAI, and a live-lane
operator on other providers would otherwise see a misleading auth failure)
+ XDG dirs + OPENCODE_CONFIG(_DIR) + OPENCODE_DISABLE_AUTOUPDATE;
OPENCODE_CONFIG_CONTENT (the inline config-shadow channel) deliberately
absent. Bare-semver version preamble (the SST-vs-claimant discriminator)
+ global-config mcp.gbrain contamination tripwire (JSONC-tolerant, checks
BOTH merged filenames). --list-agents pin moves to all four runners with
the openclaw<opencode ordering note.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 10:01:21 -07:00
Garry TanandClaude Fable 5 40c5774fa8 feat(connect): --agent opencode — env-interpolated bearer, direct-writer --install
buildOpencodeMcpAddArgv pins the validated one-liner: the --header value
carries opencode's {env:GBRAIN_REMOTE_TOKEN} interpolation LITERALLY, so the
token never enters argv, the config file, or --json output. The print block
mirrors codexBlock (export line + one-liner + restart note). --install goes
through a new ConnectDeps.writeOpencodeRemoteEntry member (the existing
injectable seam, connect.ts:ConnectDeps) wrapping the JSONC writer in env
token mode — no opencode binary required, idempotent re-runs, foreign
same-name entries refuse with the writer's message (token-redacted), and
the D4 probe smoke-tests the credential end to end.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 09:58:16 -07:00
Garry TanandClaude Fable 5 71fb120514 feat(bootstrap): opencode harness-mode target — managed remote entry with inline bearer, remove/status/rollback
HarnessSelector gains 'opencode' (forced-wire like codex: the JSONC writer
needs no opencode CLI). Wiring is one managed mcp.<name> remote entry with
the inline Authorization bearer in the user-global opencode config, 0600,
under the [X11] lock ordering (config-dir → opencode-dir). Ownership [C8]:
idempotent re-runs match on the serve url; rotation across a url change
recognizes the old entry via the PRIOR receipt's url; anything else under
the name refuses inside the writer. Failed-smoke rollback restores the .bak
or removes a fresh entry, and the fresh mint is revoked (impostor-guard
economics hold). --remove classifies against the receipt url and skips
not-ours entries with a note; --status recovers the bearer from the entry
(url-matched — a foreign entry's credential is never transmitted). Consent
copy: per-host numbered item, joined-list reach statement (a fourth harness
can no longer silently mislabel the ternary tree), opencode off-ramp.

Fixture hygiene: the harness serial fixture now injects opencodeConfig +
detectOpencode — the default path resolution reaches the operator's REAL
~/.config/opencode (the claudeUserSettingsPath lesson, caught live when the
registrar-mode test wrote a fixture token there; cleaned up).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 09:53:45 -07:00
Garry TanandClaude Fable 5 3577432456 feat(bootstrap): opencode workspace lane — hooks --harness opencode, scope-aware direct-writer registration, channels, templates
The Harness union widening is typecheck-SILENT at every existing
'claude-code ? A : B' ternary, so the hot sites now dispatch exhaustively
(HARNESSES satisfies anchor; exec-lane bin map returns null for opencode —
its registrations go through the JSONC writer whose fingerprint IS the
[FIX7] check, never through <host> mcp get).

Scope INVERSION for opencode: default user-global (opencode spawns
project-config-defined MCP servers with NO trust prompt — verified; a
committed project entry would auto-execute on every collaborator machine).
MCP_SCOPE=project is an explicit opt-in that writes the workspace
opencode.json with a PATH-resolved command (committed-candidate file: no
absolute machine paths, no fail-open analog exists) and prints the sharing
warning + enabled:false opt-out. detectHarness probes OPENCODE/OPENCODE_PID
(observed 1.18.18). Ownership: a remote-type mcp.gbrain in the global
config makes the stdio lane step aside (codexBlockOwnsName analog); foreign
entries refuse. Verification: writer post-render parse-back is
authoritative; best-effort 'opencode mcp list --pure' probe (skipped on
plugin-bearing configs — mcp list is a code-execution surface).

Atomic with this commit (each-commit-green): questions.json MCP_SCOPE +
SURFACE_PRIMARY copy, AGENTS/GITHUB template pull-protocol generalization,
BOOTSTRAP_FOR_AGENTS.md scope guidance + opencode wiring bullet,
status.ts interview/wire resume hints, check-bootstrap-templates.sh §(e)
pins (now 'Claude Code and opencode' + the 'NO trust prompt' spawn-gate
rationale pin), guard-test fixtures, the status-test hint pin, the vendored
template-repo regen, the offline docker opencode leg, and uninstall's
receipt-keyed opencode removal. Channels: 'opencode' joins
VOLUNTEER_CHANNELS + HARNESS_CHANNELS (reserved attribution slot, codex
precedent).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 09:45:48 -07:00
Garry TanandClaude Fable 5 39687cb7fe feat(bootstrap): opencode-json managed config writer + opencode-2026-08 host spec
- src/core/bootstrap/opencode-json.ts: comment-preserving JSONC writer
  (jsonc-parser surgical edits — opencode's own mcp add preserves comments,
  the writer matches that bar). Ownership is a 4-state structural
  fingerprint (ours-same-source | ours-other-source | foreign | absent)
  keyed on GBRAIN_SOURCE EQUALITY ([FIX7] parity), never a marker key.
  Distinct read-failure classes (ENOENT create / empty-as-{} / unreadable
  refuse); foreign refusal on write AND remove; post-render validation
  (our entry round-trips, every other key survives) keeps the original on
  failure; 0600 + .bak-0600 only for inline-bearer entries; bearer
  recovery helper for harness --status.
- host-specs.ts: TARGETS['opencode-2026-08'] (verified 2026-08-15 against
  a hermetic opencode-ai@1.18.18) + opencodeConfigDir/GlobalConfigPath/
  ProjectConfigPath (XDG-only — OPENCODE_CONFIG* observed INERT in
  1.18.18, honoring them would be a silent no-op install) +
  OPENCODE_HAS_HOOKS=false.
- atomic-write.ts: rule-of-three extraction of the symlink-resolving,
  mode-inheriting atomic writer; codex-toml.ts + hooks.ts ported onto it
  (behavior pinned by their existing suites).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 09:31:45 -07:00
Garry TanandClaude Fable 5 6d25dd0711 docs(mcp): pin observed opencode CLI behavior (OPENCODE-CLI-PIN.md) + registration guide
Phase-0 hermetic observation of opencode-ai@1.18.18 (npm wrapper + platform
payload integrities pinned). Load-bearing observations: keyless anonymous
free tier answers headless runs AND drives MCP tool calls without --auto
(nonce SMOKE proven end-to-end against a real gbrain serve --surface verbs);
mcp list is the honest discriminator (spawns servers, exit 0 regardless —
parse the text); mcp add takes '-- command' (undocumented in --help) but
always writes user-global opencode.jsonc; project-defined local servers
spawn with NO trust gate (drives the user-global bootstrap default); JSONC
parses in .json-named files and both filenames merge; OPENCODE_CONFIG* env
vars observed inert (docs-contradiction, called out); OPENCODE=1 set in bash
children (detectHarness probe); AGENTS.md loads, CLAUDE.md not double-loaded.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 09:25:00 -07:00
53 changed files with 1002 additions and 3433 deletions
-1
View File
@@ -1 +0,0 @@
{"schema_version":3,"run_id":"f2b40f7ef-retrieval-canary-na-0","ran_at":"2026-08-15T15:37:16.659Z","suite":"retrieval-canary","mode":"n/a","commit":"f2b40f7ef","seed":0,"params":{"qrels":"test/fixtures/eval-baselines/qrels-search.json","embedder":"deterministic","k":10,"metrics":{"mean_recall_at_k":1,"first_relevant_hit_rate":1,"expected_top1_hit_rate":0.8333333333333334,"expected_top1_denominator":12,"queries_run":12,"queries_total":12},"floors":{"recall_at_k":0.7,"first_relevant_hit":0.6,"expected_top1":0.5}},"status":"completed","duration_ms":2290}
-1
View File
@@ -27,4 +27,3 @@
# Markdown it does not own), but pinning this repo's own .md checkout to LF
# removes the whole class for anyone working here.
*.md text eol=lf
/.gbrain-evals/eval-results.jsonl merge=union
+8 -126
View File
@@ -20,44 +20,6 @@ concurrency:
cancel-in-progress: true
jobs:
# ──────────────────────────────────────────────────────────────────────
# e2e-cache-check: same content-hash skip as test.yml's cache-check, in
# its own key namespace (e2e-pass-<hash>). Doc-only pushes previously
# provisioned 3 pgvector services and spent real OpenAI/Anthropic/
# ZeroEntropy tokens in tier2; now they skip. SCHEDULED runs are exempt
# below — the nightly is a drift check against live providers and must
# run even when the tree is unchanged.
# ──────────────────────────────────────────────────────────────────────
e2e-cache-check:
runs-on: ubuntu-latest
timeout-minutes: 10
outputs:
hit: ${{ steps.lookup.outputs.cache-hit }}
hash: ${{ steps.compute.outputs.hash }}
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- name: Compute content hash
id: compute
run: |
HASH=$(bash scripts/ci-cache-hash.sh --verbose 2>/tmp/cache-diag)
cat /tmp/cache-diag
echo "Computed cache hash: $HASH"
echo "hash=$HASH" >> "$GITHUB_OUTPUT"
- name: Lookup actions/cache for e2e-pass-<hash>
id: lookup
uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
key: e2e-pass-${{ steps.compute.outputs.hash }}
path: .e2e-cache-marker
lookup-only: true
- name: Cache status
run: |
if [ "${{ steps.lookup.outputs.cache-hit }}" = "true" ]; then
echo "✓ e2e cache HIT for hash ${{ steps.compute.outputs.hash }} — e2e jobs will skip (unless scheduled)"
else
echo "✗ e2e cache MISS for hash ${{ steps.compute.outputs.hash }} — e2e suite will run"
fi
jsonb-parity:
# Dedicated required guard for the JSONB double-encode bug-class (#2339).
# PGLite parses a double-encoded jsonb string silently, so this assertion can
@@ -66,8 +28,6 @@ jobs:
# Postgres and HARD-FAILS if DATABASE_URL is missing, so the guard can never
# silently skip.
name: JSONB parity (#2339 regression guard)
needs: e2e-cache-check
if: needs.e2e-cache-check.outputs.hit != 'true' || github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
runs-on: ubuntu-latest
timeout-minutes: 15
services:
@@ -89,12 +49,7 @@ jobs:
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
- uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: ~/.bun/install/cache
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
restore-keys: bun-cache-${{ runner.os }}-
- run: bun install --frozen-lockfile
- run: bun install
- name: Require DATABASE_URL (no silent skip)
env:
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/gbrain_test
@@ -118,8 +73,6 @@ jobs:
tier1:
name: Tier 1 (Mechanical)
needs: e2e-cache-check
if: needs.e2e-cache-check.outputs.hit != 'true' || github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
runs-on: ubuntu-latest
timeout-minutes: 20
services:
@@ -141,12 +94,7 @@ jobs:
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
- uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: ~/.bun/install/cache
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
restore-keys: bun-cache-${{ runner.os }}-
- run: bun install --frozen-lockfile
- run: bun install
- name: Run Tier 1 E2E tests
# job-isolation rides tier1 deliberately: e2e.yml runs only explicitly
# NAMED files (no glob) — an unwired e2e file is silent coverage loss.
@@ -158,15 +106,12 @@ jobs:
tier2:
name: Tier 2 (LLM Skills)
# Runs on every push/PR (promoted from schedule-only in v0.19.0), in
# PARALLEL with tier1 (own postgres service — the old `needs: tier1`
# serialized ~2min for no shared state). The jsonb-parity gate (~40s)
# stays in front as the broken-build SPEND gate: this job burns real
# OpenAI/Anthropic/ZeroEntropy tokens and must not fire when the build
# can't even pass the cheapest DB guard.
needs: [e2e-cache-check, jsonb-parity]
if: needs.e2e-cache-check.outputs.hit != 'true' || github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
runs-on: ubuntu-latest
# Runs on every push/PR now (promoted from schedule-only in v0.19.0).
# Tier 1 must pass first; Tier 2 uses OPENAI_API_KEY + ANTHROPIC_API_KEY
# from repo/org secrets. Nightly + manual triggers still supported via
# the workflow-level `on:` list.
needs: tier1
timeout-minutes: 30
services:
postgres:
@@ -187,12 +132,7 @@ jobs:
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
- uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: ~/.bun/install/cache
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
restore-keys: bun-cache-${{ runner.os }}-
- run: bun install --frozen-lockfile
- run: bun install
- name: Install OpenClaw
# Bound + retry the install: a transient npm/registry stall here used to
# hang unbounded and (since the v0.42.50.0 job timeout) burn the entire
@@ -239,61 +179,3 @@ jobs:
# zeroEntropyCompatFetch response-rewriter + URL rewrite + flexible
# dim handling + gateway.rerank against the real provider.
ZEROENTROPY_API_KEY: ${{ secrets.ZEROENTROPY_API_KEY }}
# ──────────────────────────────────────────────────────────────────────
# e2e-cache-write: seals e2e-pass-<hash> only when every gated job
# succeeded (writing earlier would bless states the suite never proved).
# Scheduled runs may also write: a nightly green at an unchanged hash is
# the same proof a push green is.
# ──────────────────────────────────────────────────────────────────────
e2e-cache-write:
needs: [e2e-cache-check, jsonb-parity, tier1, tier2]
if: success() && needs.e2e-cache-check.outputs.hit != 'true'
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: Create cache marker
run: |
mkdir -p .e2e-cache-marker
echo "${{ needs.e2e-cache-check.outputs.hash }}" > .e2e-cache-marker/hash
echo "$GITHUB_SHA" > .e2e-cache-marker/sha
echo "$GITHUB_REF" > .e2e-cache-marker/ref
- uses: actions/cache/save@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
key: e2e-pass-${{ needs.e2e-cache-check.outputs.hash }}
path: .e2e-cache-marker
# ──────────────────────────────────────────────────────────────────────
# e2e-status: the single stable "did E2E pass?" name (mirror of
# test.yml's test-status). Succeeds when the cache hit on a non-scheduled
# run, or when every gated job succeeded.
# ──────────────────────────────────────────────────────────────────────
e2e-status:
needs: [e2e-cache-check, jsonb-parity, tier1, tier2]
if: always()
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: Aggregate result
run: |
HIT="${{ needs.e2e-cache-check.outputs.hit }}"
JSONB="${{ needs.jsonb-parity.result }}"
TIER1="${{ needs.tier1.result }}"
TIER2="${{ needs.tier2.result }}"
EVENT="${{ github.event_name }}"
echo "e2e-cache-check.hit=$HIT event=$EVENT"
echo "jsonb-parity=$JSONB tier1=$TIER1 tier2=$TIER2"
# schedule AND workflow_dispatch always run the real suite — a
# manual dispatch is an explicit ask for a live run, so a cache
# hit must not report green-without-running for either.
if [ "$HIT" = "true" ] && [ "$EVENT" != "schedule" ] && [ "$EVENT" != "workflow_dispatch" ]; then
echo "✓ e2e cache HIT for hash ${{ needs.e2e-cache-check.outputs.hash }} — E2E green"
exit 0
fi
for r in "$JSONB" "$TIER1" "$TIER2"; do
if [ "$r" != "success" ]; then
echo "✗ gated e2e job did not succeed (got $r) — E2E fail"
exit 1
fi
done
echo "✓ all e2e jobs succeeded — E2E green"
+4 -24
View File
@@ -64,12 +64,7 @@ jobs:
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
- uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: ~/.bun/install/cache
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
restore-keys: bun-cache-${{ runner.os }}-
- run: bun install --frozen-lockfile
- run: bun install
- name: Run heavy tests
env:
@@ -153,12 +148,7 @@ jobs:
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
- uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: ~/.bun/install/cache
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
restore-keys: bun-cache-${{ runner.os }}-
- run: bun install --frozen-lockfile
- run: bun install
# Reference the door tests; run only the ones present (a door may land
# in a sibling PR). Missing binary/auth → the file self-skips, so a
@@ -219,12 +209,7 @@ jobs:
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
- uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: ~/.bun/install/cache
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
restore-keys: bun-cache-${{ runner.os }}-
- run: bun install --frozen-lockfile
- run: bun install
# `runner.temp` is not an allowed context in job-level env, so the
# evidence dir is derived here and exported for every later step (the
@@ -426,12 +411,7 @@ jobs:
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
- uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: ~/.bun/install/cache
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
restore-keys: bun-cache-${{ runner.os }}-
- run: bun install --frozen-lockfile
- run: bun install
- name: Prepare evidence dir
run: |
-5
View File
@@ -19,11 +19,6 @@ on:
permissions:
contents: read
# Rapid pushes to the same PR previously queued duplicate scans.
concurrency:
group: osv-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
osv-scan:
permissions:
+2 -17
View File
@@ -35,7 +35,6 @@ concurrency:
jobs:
version:
runs-on: ubuntu-latest
timeout-minutes: 10
outputs:
version: ${{ steps.v.outputs.version }}
exists: ${{ steps.v.outputs.exists }}
@@ -81,7 +80,6 @@ jobs:
target: bun-linux-x64
artifact: gbrain-linux-x64
runs-on: ${{ matrix.os }}
timeout-minutes: 30
permissions:
contents: read
id-token: write # for attest-build-provenance (Sigstore OIDC)
@@ -91,12 +89,7 @@ jobs:
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
- uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: ~/.bun/install/cache
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
restore-keys: bun-cache-${{ runner.os }}-
- run: bun install --frozen-lockfile
- run: bun install
# No test re-run here: the Test workflow already gated this exact SHA at
# merge (10 shards + E2E). Re-running the whole suite serially on the
# release runner is a flakier duplicate gate — it blocked the first
@@ -123,7 +116,6 @@ jobs:
needs: [version, build]
if: needs.version.outputs.exists == 'false'
runs-on: ubuntu-latest
timeout-minutes: 15
permissions:
contents: write # create the tag + release (scoped to this job only)
steps:
@@ -190,7 +182,6 @@ jobs:
needs: [version, release]
if: needs.version.outputs.exists == 'false'
runs-on: ubuntu-latest
timeout-minutes: 15
permissions:
contents: read
env:
@@ -216,13 +207,7 @@ jobs:
with:
bun-version: 1.3.13
- if: steps.gate.outputs.publish == 'true'
uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: ~/.bun/install/cache
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
restore-keys: bun-cache-${{ runner.os }}-
- if: steps.gate.outputs.publish == 'true'
run: bun install --frozen-lockfile
run: bun install
- if: steps.gate.outputs.publish == 'true'
name: Generate template tree and byte-diff against the vendored copy
run: |
+21 -90
View File
@@ -91,24 +91,16 @@ jobs:
# now enforces a paid GITLEAKS_LICENSE (fails the job with "missing
# gitleaks license" for accounts it can't validate). The CLI is free, uses
# the committed .gitleaks.toml allowlist, and scans the same commit range.
- name: Cache gitleaks tarball
uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: /tmp/gitleaks-dl
key: gitleaks-8.30.1-linux-x64
- name: Install gitleaks (pinned + checksum-verified)
run: |
set -euo pipefail
VER=8.30.1
BASE="gitleaks_${VER}_linux_x64.tar.gz"
URL="https://github.com/gitleaks/gitleaks/releases/download/v${VER}"
mkdir -p /tmp/gitleaks-dl
[ -f "/tmp/gitleaks-dl/${BASE}" ] || curl -fsSL -o "/tmp/gitleaks-dl/${BASE}" "${URL}/${BASE}"
# Checksums fetched fresh EVERY run: a cache-restored tarball is
# re-verified against the published digest, never trusted.
curl -fsSL -o "/tmp/${BASE}" "${URL}/${BASE}"
curl -fsSL -o /tmp/gitleaks_checksums.txt "${URL}/gitleaks_${VER}_checksums.txt"
( cd /tmp/gitleaks-dl && grep " ${BASE}\$" /tmp/gitleaks_checksums.txt | sha256sum -c - )
tar -xzf "/tmp/gitleaks-dl/${BASE}" -C /tmp gitleaks
( cd /tmp && grep " ${BASE}\$" gitleaks_checksums.txt | sha256sum -c - )
tar -xzf "/tmp/${BASE}" -C /tmp gitleaks
install /tmp/gitleaks /usr/local/bin/gitleaks
gitleaks version
- name: Scan for secrets (gitleaks CLI, .gitleaks.toml)
@@ -142,25 +134,11 @@ jobs:
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
# This job is the ONE designated saver of the bun cache (the others
# restore-only, so 5 redundant post-job save attempts disappear).
# admin/bun.lock is in the key because verify's check:admin-build
# installs from it.
- uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: ~/.bun/install/cache
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock', 'admin/bun.lock') }}
restore-keys: bun-cache-${{ runner.os }}-
# verify's runner sources test-env.sh and builds the snapshot for its
# PGLite-booting eval checks — restore the cache so it's the ~40ms
# freshness check, not a cold build ahead of all ~47 checks.
- uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: |
test/fixtures/pglite-snapshot.tar
test/fixtures/pglite-snapshot.version
key: pglite-snapshot-${{ runner.os }}-${{ hashFiles('src/core/migrate.ts', 'src/core/pglite-schema.ts', 'test/helpers/legacy-embedding-config.ts', 'scripts/build-pglite-snapshot.ts') }}
- run: bun install --frozen-lockfile
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
- run: bun install
- run: bun run verify
# Guard: no bare `bun test` in workflows/scripts — bun ignores
# bunfig.toml's timeout, and hooks (beforeAll/afterAll) get the 5s
@@ -169,11 +147,10 @@ jobs:
- run: bash scripts/check-bun-test-timeout.sh
serial-tests:
# *.serial.test.ts — one bun process per file (module-registry isolation),
# POOLED across files by scripts/run-serial-tests.sh (was strictly
# sequential: an 8.5-minute job whose serialization the quarantine
# contract never required). Lives in its own runner so the matrix shards
# aren't carrying the serial tail.
# *.serial.test.ts at --max-concurrency=1. Lives in its own runner so
# the matrix shards aren't carrying the serial-pass tail (the old shape
# stuffed this into `test (1)` after the matrix work, which compounded
# shard 1's overload).
needs: cache-check
if: needs.cache-check.outputs.hit != 'true'
runs-on: ubuntu-latest
@@ -183,27 +160,11 @@ jobs:
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
- uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
- uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: ~/.bun/install/cache
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
restore-keys: bun-cache-${{ runner.os }}-
# PGLite schema snapshot (~42MB): the runner builds it when absent or
# stale (its runtime hash is authoritative — a stale restore is rebuilt,
# never trusted). Cached so the build is paid once per schema change,
# not once per job per run. This job SAVES; verify + matrix + slow jobs
# restore-only. The key is an approximation on purpose: it only has to
# be a superset-trigger of real schema changes.
# KEY HAS 5 HOMES in this file (this save + 4 restores: verify, matrix,
# slow-eval, slow-perf) — edit all together, or drift shows up only as
# silent rebuild cost.
- uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: |
test/fixtures/pglite-snapshot.tar
test/fixtures/pglite-snapshot.version
key: pglite-snapshot-${{ runner.os }}-${{ hashFiles('src/core/migrate.ts', 'src/core/pglite-schema.ts', 'test/helpers/legacy-embedding-config.ts', 'scripts/build-pglite-snapshot.ts') }}
- run: bun install --frozen-lockfile
- run: bun install
- run: bun run test:serial
slow-eval-longmemeval:
@@ -224,20 +185,11 @@ jobs:
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
- uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
- uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: ~/.bun/install/cache
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
restore-keys: bun-cache-${{ runner.os }}-
- uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: |
test/fixtures/pglite-snapshot.tar
test/fixtures/pglite-snapshot.version
key: pglite-snapshot-${{ runner.os }}-${{ hashFiles('src/core/migrate.ts', 'src/core/pglite-schema.ts', 'test/helpers/legacy-embedding-config.ts', 'scripts/build-pglite-snapshot.ts') }}
- run: bun install --frozen-lockfile
- name: Ensure PGLite snapshot (build-or-validate, non-fatal)
run: bash -c '. scripts/lib/test-env.sh && ensure_pglite_snapshot slow-eval && echo "GBRAIN_PGLITE_SNAPSHOT=${GBRAIN_PGLITE_SNAPSHOT:-}" >> "$GITHUB_ENV"'
- run: bun install
- run: bun test test/eval-longmemeval-e2e.slow.test.ts --timeout=60000
brainbench:
@@ -254,19 +206,16 @@ jobs:
timeout-minutes: 10 # ~15s hermetic run; matches the per-job-timeout hardening (#2254)
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- name: Fetch origin/master baseline ref (shallow)
# The gate reads ONE file via `git show origin/master:...` — a depth-1
# fetch of the master ref replaces the previous full 3700-commit clone.
run: git fetch --no-tags --depth=1 origin +refs/heads/master:refs/remotes/origin/master
with:
fetch-depth: 0 # the gate needs origin/master's baseline
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
- uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
- uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: ~/.bun/install/cache
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
restore-keys: bun-cache-${{ runner.os }}-
- run: bun install --frozen-lockfile
- run: bun install
- run: bash scripts/ci-brainbench-gate.sh
env:
BRAINBENCH_OUT: ${{ runner.temp }}/brainbench-result.json
@@ -293,20 +242,11 @@ jobs:
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
- uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
- uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: ~/.bun/install/cache
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
restore-keys: bun-cache-${{ runner.os }}-
- uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: |
test/fixtures/pglite-snapshot.tar
test/fixtures/pglite-snapshot.version
key: pglite-snapshot-${{ runner.os }}-${{ hashFiles('src/core/migrate.ts', 'src/core/pglite-schema.ts', 'test/helpers/legacy-embedding-config.ts', 'scripts/build-pglite-snapshot.ts') }}
- run: bun install --frozen-lockfile
- name: Ensure PGLite snapshot (build-or-validate, non-fatal)
run: bash -c '. scripts/lib/test-env.sh && ensure_pglite_snapshot slow-perf && echo "GBRAIN_PGLITE_SNAPSHOT=${GBRAIN_PGLITE_SNAPSHOT:-}" >> "$GITHUB_ENV"'
- run: bun install
- run: bun test test/entity-resolve-perf.slow.test.ts --timeout=300000
# MEMORY_VERBS v1 (Cathedral 1): the entity() p99 < 100ms contract gate
# (20K-page corpus + ratio guard) shares this runner — same perf-job
@@ -359,20 +299,11 @@ jobs:
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
- uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
- uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: ~/.bun/install/cache
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
restore-keys: bun-cache-${{ runner.os }}-
# Restore-only: test-shard.sh validates the snapshot's runtime hash and
# rebuilds when stale (the serial-tests job is the designated saver).
- uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: |
test/fixtures/pglite-snapshot.tar
test/fixtures/pglite-snapshot.version
key: pglite-snapshot-${{ runner.os }}-${{ hashFiles('src/core/migrate.ts', 'src/core/pglite-schema.ts', 'test/helpers/legacy-embedding-config.ts', 'scripts/build-pglite-snapshot.ts') }}
- run: bun install --frozen-lockfile
- run: bun install
- name: Run test shard ${{ matrix.shard }}/10
run: scripts/test-shard.sh ${{ matrix.shard }} 10
+1 -1
View File
@@ -1,4 +1,4 @@
<!-- gbrain-runbook-stamp: 0.46.5.0 -->
<!-- gbrain-runbook-stamp: 0.46.4.0 -->
<!-- This stamp must equal the VERSION file at every release; CI enforces it
(scripts/check-bootstrap-tag.sh). `gbrain bootstrap status` compares it to
the installed binary and warns on skew. -->
-34
View File
@@ -2,40 +2,6 @@
All notable changes to GBrain will be documented in this file.
## [0.46.5.0] - 2026-08-15
**CI in half, evals actually gating.** The Test workflow ran 89.5 minutes on
every push; its long pole was a serial-test job that executed ~140 per-file bun
processes strictly one-at-a-time even though the quarantine only ever required
per-process isolation. This release pools that lane (8.5 min → ~4 min in CI,
~2.5 min locally), wires the PGLite schema-snapshot fast-path into the CI test
runners (it previously existed but only the local loop used it), and rebalances
the 10-shard matrix on freshly mined weights (new files without a mined weight
now fall back to the p75 file weight instead of the median) — a measured branch
run landed the whole workflow at 255s. E2E stops spending real provider tokens on doc-only
pushes (content-hash skip with nightly + manual-dispatch exemptions) and runs
its tiers in parallel behind a fast broken-build spend gate.
Retrieval quality now has a hermetic CLI canary: `gbrain eval gate` accepts a
deterministic embedder option that drives the full hybrid/RRF pipeline with
zero API keys, gated in CI on every run (`check:eval-canary`, alongside the new
`check:eval-chronicle` gate) with its run ledger committed to
`.gbrain-evals/eval-results.jsonl`. Two registered-but-never-executed guards
came alive, a registration⇒execution coverage test closes that class for good,
and 47 orphaned eval-harness tests joined the CI matrix behind a keyless
allowlist. Test reliability hardening rounds it out: externally-killed serial
files get a sequential rescue re-run (never a silent pass), machine-global
files live on a growth-guarded exclusive lane, and the shard-balance test now
asserts the matrix CI actually runs instead of recomputing its own inputs.
**To take advantage of v0.46.5.0:** nothing to configure — CI and the local
loops (`bun run test`, `bun run test:serial`, `bun run verify`) are just
faster. New knobs if you need them: `GBRAIN_SERIAL_POOL=1` restores the old
fully-sequential serial lane, `GBRAIN_VERIFY_MAX_PARALLEL` bounds verify's
worker pool, `GBRAIN_NO_SNAPSHOT=1` opts any runner out of the snapshot
fast-path. Run the retrieval canary yourself with
`bun run scripts/run-eval-canary.ts` (add `--record` to append the committed
ledger).
## [0.46.4.0] - 2026-08-15
**opencode joins the supported-client roster — at full parity from day one.**
+2
View File
@@ -127,6 +127,8 @@ the database name must carry "test" as a word segment (like `gbrain_test`
above) or destructive tests refuse to run — opt a differently-named database
in one-shot with `GBRAIN_E2E_ALLOW_DB=<name>`.
Use `bun run verify` before pushing. It runs 19+ guard checks in parallel
Use `bun run verify` before pushing. It runs 40+ guard checks in parallel
(`scripts/run-verify-parallel.sh`), including: banned fork-name leaks
(`scripts/check-privacy.sh`), `JSON.stringify(x)::jsonb` interpolation
+6 -79
View File
@@ -177,77 +177,14 @@ fix-wave plan; the wave series (W0.5W9, 3.4, 3.6) tracks its own scope there.
- [ ] **Legacy Anthropic-SDK subagent loop deletion.** **Priority: P2.** One
release after W8 flips `agent.use_gateway_loop` default ON (flag stays as
the revert path for that release).
- [x] **Deeper test-suite speedup** beyond the W0 snapshot default-on
LANDED in the test/eval/CI speedup pass (serial pool 8.5min → ~2.5min,
snapshot in every CI runner + memoized loader, verify worker pool,
perf-gate row shrink, chunk-grain engine consolidation). Remaining
long-tail items are filed in "Test/eval/CI speedup pass deferrals" below.
- [ ] **Deeper test-suite speedup** beyond the W0 snapshot default-on (which
already cut the full parallel suite ~4,900s → ~490s). **Priority: P3.**
Revisit with post-W0 timing data; diminishing returns until measured.
- [ ] **PGLite schema build-time derivation** from SCHEMA_SQL via a named
transform list. **Priority: P3.** Only if W3's schema drift TEST proves
annoying in practice — the test alone kills the drift bug class (Codex
D4.8/D5.23: fresh-schema equivalence ≠ upgrade correctness; old-shape
bootstrap fixtures + replay coverage stay regardless).
## Test/eval/CI speedup pass deferrals (filed with the pass; plan: ~/.claude/plans/system-instruction-you-are-working-iterative-hopcroft.md)
Each was explicitly deferred in the pass's CEO/eng/outside-voice reviews.
- [ ] **Sleep-to-poll conversions.** **What:** replace ~49.5s of hard-coded
`setTimeout` waits with event/poll-based waits; no fake timers exist in the
suite. Worst offenders: test/minions.test.ts (12.2s across 43 sites),
test/process-cleanup.test.ts (5.0s), test/worker-lock-renewal-e2e.serial.test.ts
(4.0s), test/e2e/worker-abort-recovery.test.ts (3.6s), test/e2e/zombie-reaping.test.ts
(3.3s). **Why deferred:** careful per-site work against flake-hardened timings;
~50s ceiling. **Effort:** M. **Priority:** P3.
- [ ] **E2E: PGLite-only parallel lane + default SHARD.** **What:** run-e2e.sh runs
181 files sequentially (one bun cold start each); ~42 PGLite-only files need no
Postgres and no TRUNCATE-race protection — run them in a parallel lane; default
the existing SHARD support (only ci-local uses it). Fold into the Postgres
template-database entry below in this file (CREATE DATABASE … TEMPLATE, ~50ms).
**Why deferred:** e2e is off the CI critical path after the workflow restructure;
ci-local + nightly benefit only. **Effort:** M. **Priority:** P2.
- [ ] **Second PGLite snapshot keyed by dims/model.** **What:** ~34 test files
configure zembed/1280 and always cold-init (the snapshot's shape gate correctly
refuses the 1536 fixture). Bake a second snapshot per shape; the version-file
format already carries dims/model. **Why deferred:** moderate effort, small win,
and it interacts with the shape gate the memoized loader deliberately keeps hot.
**Effort:** M. **Priority:** P3.
- [ ] **Persistent-engine snapshot.** **What:** the snapshot fast-path only covers
in-memory engines (`!dataDir` gate at pglite-engine.ts). ~58 files pass
database_path and pay full cold init (~121s weighted). Needs tar-extract-into-
dataDir (or PGlite loadDataDir with a dataDir) design. **Effort:** M. **Priority:** P3.
- [ ] **Engine consolidation audit: doctor/bootstrap/migrations-v0_19_0.** **What:**
33 files construct 95 engines; chunk-grain-fts was consolidated in-pass, but
doctor.test.ts (9 engines), bootstrap.test.ts (9), migrations-v0_19_0.test.ts (7)
need a per-file audit — migration-from-old-schema tests structurally cannot share
a current-schema engine or use the snapshot. **Effort:** M. **Priority:** P3.
- [ ] **Verify per-check double-spawn removal.** **What:** each CHECKS entry costs a
`bun run <key>` startup before its bash script; invoking scripts directly from a
manifest would drop ~47 bun startups. **Why deferred:** micro-win; touches the
package.json-scripts-as-API convention. **Effort:** S. **Priority:** P3.
- [ ] **Snapshot-tar digest verification (defense-in-depth).** **What:** the CI
actions/cache for `test/fixtures/pglite-snapshot.tar` validates only the
schema-hash/dims lines in the sidecar `.version` — which travels in the SAME
cache entry, so both are forgeable together by anyone with cache write access.
Record a sha256 of the tar bytes in the version file at build time and have
`tryLoadSnapshot` verify it (mirror of the gitleaks fetch-fresh-digest
pattern). **Why deferred:** exploitability bounded by GitHub cache scoping
(fork caches isolated; poisoning needs push access) and impact is test-DB
contents only. **Effort:** S. **Priority:** P3.
- [ ] **Redact provider/DB strings in eval ledger writes.** **What:**
`EvalRunRecord.error` (free text) is persisted unredacted by
`persistRunRecord` (eval-run-all) and the canary's record mode into the now-
TRACKED `.gbrain-evals/eval-results.jsonl` — a failed keyed run whose error
embeds a connection string would ride a later commit into the public repo.
Route `record.error` + provider-derived params through
`redactConnectionInfo`/`redactPgUrl` before append; optionally add
`.gbrain-evals/` to the fixture-privacy scan surface. **Effort:** S.
**Priority:** P2.
- [ ] **check-image-decoders-embedded.sh into verify CHECKS.** **What:** the guard
runs its own `bun build --compile` (~60s) — too heavy per-verify. Revisit if the
binary-embed bug class recurs; guards-manifest.tsv carries the exemption note,
and the registration⇒execution coverage test allowlists it explicitly.
**Effort:** S. **Priority:** P3.
## Jobs fix-wave follow-ups (filed v0.45.15.0 — upstream issues #2/#3/#4)
- [ ] **P2 — `jobs submit --max-pending` public flag.** maxPending stays an
@@ -3074,13 +3011,8 @@ outside-voice triage on the reshaped plan.
- [ ] **v0.42+: ship the coordinated `gbrain-evals/baselines/v0.41-launch.baseline.ndjson`
+ `gbrain-evals/qrels/v0.41-launch.qrels.json` (hermetic-synthetic per D9).**
Generate locally via `gbrain bench publish --from <hermetic-test-corpus>` then
commit to the sibling gbrain-evals repo. PARTIALLY SUPERSEDED by the test/eval/CI
speedup pass: an in-repo canonical qrels target now exists (`gbrain eval gate`
with the deterministic embedder option against `test/fixtures/eval-baselines/
qrels-search.json`; runner `scripts/run-eval-canary.ts`, CI-gated via
check:eval-canary, ledger `.gbrain-evals/eval-results.jsonl`). What remains
here is only the sibling-repo REGRESSION baseline (.baseline.ndjson for the
jaccard/top1 gate) — the correctness-gate half is done.
commit to the sibling gbrain-evals repo. Gives `gbrain eval gate` a canonical
baseline target so users don't have to bootstrap their own immediately.
## v0.40.7.0 Schema Cathedral v3 follow-ups (v0.40.7+)
@@ -4156,12 +4088,7 @@ verify Voyage adapter integration in `src/core/ai/recipes/voyage.ts`).
## test infra (v0.26.4 follow-up — intra-file parallelism)
### Sweep cross-file shared-state contention; enable `bun test --concurrent` for another 2-3x speedup
**Priority:** P3 (downgraded from P0 in the test/eval/CI speedup pass — premises stale:
the entry says "~58 PGLiteEngine instantiations", the suite now has 600+; the serial
quarantine grew from 4 files to ~140, and the pass's pooled serial runner + CI snapshot
+ verify pool delivered a comparable multiple for hours of work instead of the 1-2
weeks this sweep estimates. Re-scope against post-pass timing data before spending
anything here; `test.concurrent` adoption remains at zero.)
**Priority:** P0
**Status:** v0.26.7 shipped foundation slice (helpers + lint + mock.module quarantine). v0.26.8 (env sweep) and v0.26.9 (PGLite sweep + codemod + measurement) carry the rest.
**What:** v0.26.4 shipped file-level parallel fan-out (8 shards) and got `bun run test` from 18 minutes to ~85s — a 12x speedup. The next layer is **intra-file** parallelism via Bun's `--concurrent` flag (or per-test `test.concurrent()` markers). This requires every test file to be safe under concurrent execution within the same `bun test` process.
+1 -1
View File
@@ -1 +1 @@
0.46.5.0
0.46.4.0
+10 -16
View File
@@ -11,11 +11,11 @@ Six test command tiers, each with a clear scope:
| Command | What it runs | Wallclock | When to use |
|---|---|---|---|
| `bun run test` | Parallel unit-test fast loop. Sharded fan-out via `scripts/run-unit-parallel.sh` (default 4 shards — CPU-detected, clamped to a max of 8; 4 matches CI's fan-out and avoids PGLite WASM-init contention), then a serial pass over `*.serial.test.ts`. Excludes `*.slow.test.ts` and `test/e2e/*`. No pre-checks, no typecheck. Builds/refreshes the PGLite schema snapshot BEFORE the shard fan-out and exports `GBRAIN_PGLITE_SNAPSHOT` so PGLite-booting files restore a baked schema instead of replaying every migration (~3.5x per booting file; see "PGLite schema snapshot" below). Opt out: `GBRAIN_NO_SNAPSHOT=1`. Memory-safe by default: total concurrency (shards × intra-shard files) is capped to available memory at `GBRAIN_TEST_MEM_PER_FILE_MB` (default 1536 — a PGLite WASM instance) per concurrent file, and two phantom-failure classes are automatically re-run serially (the rescue pass): failures carrying the WASM out-of-memory signature, and shards killed externally (SIGTERM/SIGKILL well before the shard timeout — sibling workspaces' process cleanup, memory jetsam). Phantoms pass serially and the run goes green with an `oom_rescued` note; real failures fail again serially and stay red. Knobs: `GBRAIN_TEST_NO_MEM_ADAPT=1`, `GBRAIN_TEST_NO_OOM_FALLBACK=1`, `GBRAIN_TEST_MAX_CONCURRENCY` (intra-shard, default 4), `GBRAIN_TEST_SHARD_TIMEOUT` / `GBRAIN_TEST_SHARD_KILL_AFTER`, plus `--shards N` / `--max-concurrency N` / `--dry-run` script args. | a few minutes on a Mac dev box | Inner edit loop. Default. |
| `bun run verify` | CI's authoritative pre-test gate set, fanned out by `scripts/run-verify-parallel.sh` through a bounded worker pool (default `detect_cpus`; override `GBRAIN_VERIFY_MAX_PARALLEL`) with the heavy checks ordered first (typecheck, the two compile-embed checks, admin build, fuzz bundles, guard self-tests, the PGLite-booting eval checks, whole-tree greps). The battery includes the deterministic `check:eval-chronicle` and `check:eval-canary` eval gates. The `CHECKS` array in that script is the single source of truth — CI literally calls `bun run verify` in a dedicated job. | ~40s (pool-bounded; longest check dominates) | Before pushing; before `/ship`. |
| `bun run test` | Parallel unit-test fast loop. Sharded fan-out via `scripts/run-unit-parallel.sh` (default 4 shards — CPU-detected, clamped to a max of 8; 4 matches CI's fan-out and avoids PGLite WASM-init contention), then a serial pass over `*.serial.test.ts`. Excludes `*.slow.test.ts` and `test/e2e/*`. No pre-checks, no typecheck. Builds/refreshes the PGLite schema snapshot BEFORE the shard fan-out and exports `GBRAIN_PGLITE_SNAPSHOT` so PGLite-booting files restore a baked schema instead of replaying every migration (~10x wallclock on a full run; see "PGLite schema snapshot" below). Opt out: `GBRAIN_NO_SNAPSHOT=1`. Memory-safe by default: total concurrency (shards × intra-shard files) is capped to available memory at `GBRAIN_TEST_MEM_PER_FILE_MB` (default 1536 — a PGLite WASM instance) per concurrent file, and two phantom-failure classes are automatically re-run serially (the rescue pass): failures carrying the WASM out-of-memory signature, and shards killed externally (SIGTERM/SIGKILL well before the shard timeout — sibling workspaces' process cleanup, memory jetsam). Phantoms pass serially and the run goes green with an `oom_rescued` note; real failures fail again serially and stay red. Knobs: `GBRAIN_TEST_NO_MEM_ADAPT=1`, `GBRAIN_TEST_NO_OOM_FALLBACK=1`, `GBRAIN_TEST_MAX_CONCURRENCY` (intra-shard, default 4), `GBRAIN_TEST_SHARD_TIMEOUT` / `GBRAIN_TEST_SHARD_KILL_AFTER`, plus `--shards N` / `--max-concurrency N` / `--dry-run` script args. | a few minutes on a Mac dev box | Inner edit loop. Default. |
| `bun run verify` | CI's authoritative pre-test gate set, fanned out in parallel by `scripts/run-verify-parallel.sh`: the full `check:*` battery (privacy, jsonb, progress, source-id, test-isolation, wasm, …) plus `bun run typecheck`. The `CHECKS` array in that script is the single source of truth — CI literally calls `bun run verify` in a dedicated job. | ~16s (parallel; typecheck dominates) | Before pushing; before `/ship`. |
| `bun run test:full` | `verify && bun run test && bun run test:slow && [smart e2e]`. The local equivalent of "everything CI runs." Smart e2e: runs e2e only when `DATABASE_URL` is set; else loud skip notice to stderr. | ~3-5min depending on slow + e2e | Pre-merge sanity, before opening a PR. |
| `bun run test:slow` | Just the `*.slow.test.ts` set (intentional cold-path correctness checks). | seconds-to-minutes | When touching slow-path code. |
| `bun run test:serial` | Just the `*.serial.test.ts` set (cross-file-contention quarantine; one bun process per file for true module-registry isolation), run through a POOL of concurrent per-file processes — the isolation is per-process, not per-machine. Pool defaults to `min(detect_cpus, 4)` then memory-adapts (same doctrine as the parallel runner); a small growth-guarded set of files (machine-global state or contention-critical timing — see the justified `EXCLUSIVE_FILES` list in `scripts/run-serial-tests.sh`, capped at 3 by `test/scripts/serial-files.test.ts`) runs on a sequential EXCLUSIVE lane after the pool. Per-test timeout 120s (pooled contention headroom); each pooled file is wall-clock-killed at 300s (`timeout -k`, exit-hang containment). Externally-killed files (exit 143/137 or a missing exit sentinel — sibling-workspace cleanup, memory jetsam) get ONE sequential rescue re-run, mirroring the parallel runner's doctrine: phantoms stay green with a rescue note, real failures stay red. Prints per-file PASS lines plus a top-10 slowest-files list. Knobs: `GBRAIN_SERIAL_POOL=N` (explicit pool width — bypasses the memory clamp; `1` restores fully-sequential), `GBRAIN_SERIAL_FILE_TIMEOUT`. | ~2.5min for all ~140 files at pool=4 (was ~8.5min sequential) | Debugging quarantined files; CI's serial-tests job. |
| `bun run test:serial` | Just the `*.serial.test.ts` set (cross-file-contention quarantine; one bun process per file for true module-registry isolation). | ~1s per quarantined file | Debugging a specific quarantined file. |
| `bun run test:e2e` | Real Postgres E2E. Requires Docker + `DATABASE_URL`. Sequential. | ~5-10min | Pre-ship; nightly. |
There is no `check:all` script anymore — it was a second, hand-synced guard
@@ -32,17 +32,11 @@ self-test" below).
post-`initSchema()` PGLite data dir into `test/fixtures/pglite-snapshot.tar`
plus a version file; `PGLiteEngine.initSchema()` restores the tar instead of
replaying the embedded schema + all migrations when the env var
`GBRAIN_PGLITE_SNAPSHOT` points at it. Runners activate it through the shared
`ensure_pglite_snapshot` helper in `scripts/lib/test-env.sh` (also home of
`detect_cpus` and `detect_available_mem_mb`), sourced by
`run-unit-parallel.sh`, `test-shard.sh`, `run-slow-tests.sh`,
`run-serial-tests.sh`, and `run-verify-parallel.sh`; `scripts/ci-local.sh`
calls the builder directly. The helper builds/refreshes the snapshot and
exports the env var, no-ops on `GBRAIN_NO_SNAPSHOT=1` or an already-inherited
path, and is non-fatal on build failure — tests fall back to cold init, with
a one-line "active" echo so a silent fallback stays visible in CI logs.
Measured effect: ~3.5x per PGLite-booting file (a cold boot replays every
migration, ~3.1s each on a CI shard). Properties:
`GBRAIN_PGLITE_SNAPSHOT` points at it. Both `bun run test`
(`scripts/run-unit-parallel.sh`, before the shard fan-out) and
`scripts/ci-local.sh` call the builder unconditionally and export the env var.
Measured effect: a full parallel suite run drops ~10x (PGLite-booting files go
~1.63s → ~0.91s each). Properties:
- **Idempotent.** A hash short-circuit exits in ~40ms when the snapshot is
fresh, and REBUILDS a stale one. The hash covers `PGLITE_SCHEMA_SQL`, every
@@ -112,7 +106,7 @@ there even though they pass on Linux and macOS.
### CI vs local: intentionally divergent file sets
- **CI matrix** (`.github/workflows/test.yml`) runs `scripts/test-shard.sh` across 10 matrix shards partitioned by weight-aware LPT bin-packing (`scripts/sharding.ts`; files with no mined weight fall back to the p75 file weight so a new unweighted file can't silently unbalance a shard) and INCLUDES `*.slow.test.ts` (the two outlier slow files run as dedicated jobs alongside the matrix) plus `evals/**/*.test.ts` (keyless-allowlist-gated — `test/scripts/evals-collection.test.ts`). Each shard's bun process is bounded by `--max-concurrency` (`GBRAIN_TEST_MAX_CONCURRENCY`, default 4). Every bun-test job — matrix shards, serial-tests, verify, the slow/eval jobs — activates the PGLite schema snapshot (built in-runner via `scripts/lib/test-env.sh`; the brainbench gate brings its own in-memory PGLite and skips it; the ~42MB tar is also cached across jobs via actions/cache, with the runner's own hash check staying authoritative). CI EXCLUDES `*.serial.test.ts` from the shards and runs them in the pooled `serial-tests` job via `bun run test:serial` — one bun process per file preserves the `mock.module` quarantine; the pool runs those processes concurrently. `bun run verify` gets its own job too, as does the BrainBench memory-conformance gate (`brainbench` job → `scripts/ci-brainbench-gate.sh`, hermetic in-memory PGLite, ~15s), which compares HEAD's fresh run against master's committed baseline (`evals/brainbench/baselines/main.json`) — the `test-status` aggregate checks its result explicitly. E2E (`.github/workflows/e2e.yml`) mirrors the content-hash skip in its own `e2e-pass-<hash>` namespace (scheduled nightly runs are exempt and always run), runs tier1 and tier2 in parallel with the jsonb-parity job in front of tier2 as the token-spend gate, and aggregates through `e2e-status`. CI is the ground truth for "did everything pass."
- **CI matrix** (`.github/workflows/test.yml`) runs `scripts/test-shard.sh` across 10 matrix shards partitioned by weight-aware LPT bin-packing (`scripts/sharding.ts`) and INCLUDES `*.slow.test.ts` (the two outlier slow files run as dedicated jobs alongside the matrix). CI EXCLUDES `*.serial.test.ts` from the shards and runs them in a dedicated job via `bun run test:serial`, one bun process per file — keeping serial files out of the shard processes is what preserves the `mock.module` quarantine (a top-level mock in one file leaks into every other file sharing its process). `bun run verify` gets its own job too, as does the BrainBench memory-conformance gate (`brainbench` job → `scripts/ci-brainbench-gate.sh`, hermetic in-memory PGLite, ~15s), which compares HEAD's fresh run against master's committed baseline (`evals/brainbench/baselines/main.json`) — the `test-status` aggregate checks its result explicitly. CI is the ground truth for "did everything pass."
- **Local fast loop** (`scripts/run-unit-shard.sh` via the parallel wrapper) uses round-robin-by-index sharding and EXCLUDES `*.slow.test.ts` AND `*.serial.test.ts`. Local trades coverage for inner-loop speed; CI catches what local skips.
This divergence is intentional. Don't try to make them equal — the two scripts deliberately solve different problems. The regression test at `test/scripts/run-unit-shard.test.ts` pins what the local fast loop should and shouldn't include; `test/scripts/run-unit-parallel.test.ts` pins the wrapper's memory-adaptive concurrency and the OOM/external-kill serial rescue pass.
@@ -137,7 +131,7 @@ Triage rule: a `warn-pass` EXIT-HANG line in `.context/test-summary.txt` is NOT
- `*.test.ts` → fast loop (parallel up-to-4-shard fan-out, memory-adaptive).
- `*.slow.test.ts` → run via `bun run test:slow` only (intentional cold-path tests; would dominate the fast loop's wallclock).
- `*.serial.test.ts` → run via `bun run test:serial` after the parallel pass completes; one bun process per file (`--max-concurrency=1` within a shared process is not enough — the module registry still leaks `mock.module`), with those per-file processes POOLED (per-process isolation never required one-at-a-time execution). Files touching machine-global state (launchd/cron) live on the sequential `EXCLUSIVE_FILES` lane inside `scripts/run-serial-tests.sh` — growth-guarded to ≤3 entries with justification comments. Quarantine for tests that share file-wide state and race when run alongside other files in the same `bun test` process. Several dozen files, discovered by the `*.serial.test.ts` glob — no list to maintain. Typical residents: `mock.module(...)` users (top-level mocks leak across files in a shard process, e.g. `test/embed.serial.test.ts`), env-coupled files (e.g. `test/brain-registry.serial.test.ts`), and process-lifecycle suites that assert on `process.exitCode` (e.g. `test/pglite-engine-disconnect.serial.test.ts`). **Do not put the parallelism back on a serial file unless you've fixed the contention root cause** (it just re-introduces the flake).
- `*.serial.test.ts` → run via `bun run test:serial` after the parallel pass completes; one bun process per file (`--max-concurrency=1` within a shared process is not enough — the module registry still leaks `mock.module`). Quarantine for tests that share file-wide state and race when run alongside other files in the same `bun test` process. Several dozen files, discovered by the `*.serial.test.ts` glob — no list to maintain. Typical residents: `mock.module(...)` users (top-level mocks leak across files in a shard process, e.g. `test/embed.serial.test.ts`), env-coupled files (e.g. `test/brain-registry.serial.test.ts`), and process-lifecycle suites that assert on `process.exitCode` (e.g. `test/pglite-engine-disconnect.serial.test.ts`). **Do not put the parallelism back on a serial file unless you've fixed the contention root cause** (it just re-introduces the flake).
- `test/e2e/*.test.ts` → real-Postgres E2E. Skipped when `DATABASE_URL` is unset. One out-of-directory file rides this lane: `test/phantom-redirect-engine-parity.test.ts` (lives in `test/` for its PGLite arm, but its Postgres arm is only reachable through a DATABASE_URL-bearing lane — the unit wrappers strip the URL per #3485, so `run-e2e.sh`'s no-args list and CI's parity job carry it).
- `tests/heavy/*.sh` → ops-shape shell scripts. Cost minutes per run; NOT in default `bun test`. Run via `bun run test:heavy` or scheduled nightly via `.github/workflows/heavy-tests.yml`. Examples: pg_upgrade matrix (boot legacy brain → walk to head), RSS budget gate (measure peak worker RSS vs committed baseline), read-latency-under-sync (p50/p95/p99 under concurrent writer load), sync lock regression (N concurrent syncs assert 1 winner + N-1 lock-busy + zero leaked `gbrain_cycle_locks` rows). See `tests/heavy/README.md` for when to add a script here vs `*.slow.test.ts`. Files prefixed with `_` (e.g. `tests/heavy/_build_legacy_fixtures.sh`) are helpers/libs invoked by sibling tests — the runner skips them.
- `test/fuzz/*.test.ts` → property-based fuzz harness. Pure-validator targets in `pure-validators.test.ts` are guarded by `scripts/check-fuzz-purity.sh` (in `bun run verify`), which `bun build --target=bun` bundles each target and greps the resulting bundle for banned transitive imports (`node:fs`, `node:child_process`, engine modules). Anything that fails the guard moves to `mixed-validators.test.ts` (still property-tested, but no purity guarantee) or `filesystem-validators.test.ts` (fs-backed, uses temp dirs). Fuzz tests run in the default `bun test` loop because they're fast (~3s for ~12 properties × 1000 runs each).
File diff suppressed because one or more lines are too long
-15
View File
@@ -64,21 +64,6 @@ CI. Production retrieval differs via the query cache, salience freshness,
expansion, etc. The gate measures retrieval quality with a fixed pipeline;
your users may see different results when the cache is warm.
For a fully hermetic run (CI canaries, keyless environments), add
`--embedder deterministic` to the correctness gate: query embeddings come
from the qrels fixture's basis-vector dims (`src/eval/deterministic-embed.ts`)
instead of the gateway, so the gate runs with no API keys and no network.
Correctness-gate-only — it is rejected together with `--baseline` (replay
re-embeds captured queries via the gateway) and requires `--qrels`. Bare
`hybridSearch` never reads or writes the semantic query cache, so a
deterministic run cannot poison cached production results. This is what CI's
`check:eval-canary` gate runs via `scripts/run-eval-canary.ts`: a throwaway
PGLite brain seeded from the qrels fixture, gating the hybrid ranking
pipeline (keyword/title/alias arms + RRF) with synthetic vectors. Honest
scope: semantic-embedding regressions remain the keyed eval suites' job.
Reproduce locally with `bun run scripts/run-eval-canary.ts` (`--record`
appends to the `.gbrain-evals/eval-results.jsonl` ledger).
### `.qrels.json` shape
Two equivalent representations per entry:
+2 -8
View File
@@ -51,14 +51,8 @@ Test infra: PGLite snapshot default-on for `bun run test`. Per-PGLite-file:
Full-suite wall-clock (post-snapshot): recorded in the W0 ship notes — see
the run banner of the W0 PR's `bun run test` evidence.
Retrieval canary: PASS @ f2b40f7ef (hermetic deterministic-embedder CLI run;
recall@10=1.0000 first_relevant=1.0000 expected_top1=0.8333 vs floors
0.70/0.60/0.50; run `bun run scripts/run-eval-canary.ts` to reproduce, ledger:
.gbrain-evals/eval-results.jsonl). Honest scope: the canary gates the hybrid
ranking pipeline (keyword/title/alias arms + RRF against gold qrels) with
synthetic basis vectors — no API keys, no production brain, so the live-serve
lock is moot. Semantic-embedding regressions remain the keyed eval suites'
job. Wired into `bun run verify` as check:eval-canary.
Retrieval canary: NOT RUN at W0 (production brain locked by live serve; W0
touches no search paths). REQUIRED before W1 lands.
Verified-bug status at W0 ship: cycle-lock refresh + fencing (TODO-OPS-2
closed), stall-death parent unblock, started_at ×4, modality carry, import
+1 -1
View File
@@ -1,7 +1,7 @@
{
"id": "gbrain-context-engine",
"name": "gbrain",
"version": "0.46.5.0",
"version": "0.46.4.0",
"description": "Personal knowledge brain with Postgres + pgvector hybrid search",
"family": "bundle-plugin",
"configSchema": {
+1 -5
View File
@@ -94,10 +94,6 @@
"check:operations-filter-bypass": "bash scripts/check-operations-filter-bypass.sh",
"check:fixture-privacy": "bash scripts/check-fixture-privacy.sh",
"check:conversation-parser": "bun src/cli.ts eval conversation-parser test/fixtures/conversation-formats/all.jsonl --no-llm",
"check:eval-chronicle": "bun src/cli.ts eval chronicle",
"check:eval-canary": "bun run scripts/run-eval-canary.ts",
"check:pagetype-exhaustive": "bash scripts/check-pagetype-exhaustive.sh",
"check:pg-url-redaction": "bash scripts/check-pg-url-redaction.sh",
"check:source-scope-onboard": "bash scripts/check-source-scope-onboard.sh",
"postinstall": "bun run scripts/postinstall.ts",
"prepublish:clawhub": "bun run build:all",
@@ -164,7 +160,7 @@
"bun": ">=1.3.10"
},
"license": "MIT",
"version": "0.46.5.0",
"version": "0.46.4.0",
"overrides": {
"@hono/node-server": "^2.0.5",
"fast-uri": "^3.1.5",
+2 -5
View File
@@ -21,10 +21,7 @@ ROOT=$(cd "$(dirname "$0")/.." && pwd)
# - The redactor itself: src/core/url-redact.ts
# - Test fixtures that build redacted strings from full URLs
# - Documentation comments referring to the pattern
# The marker text is the exemption; its comment wrapper is not load-bearing
# (inside a /** block comment a literal `*/` would terminate the comment, so
# block-comment examples carry the bare marker).
ALLOW_REGEX='url-redact\.ts|test/url-redact\.test\.ts|allow-pg-url-literal'
ALLOW_REGEX='url-redact\.ts|test/url-redact\.test\.ts|/\* allow-pg-url-literal \*/'
# The pattern matches an unredacted Postgres URL appearing in a string
# literal, NOT preceded by `redactPgUrl(` or `***@`. We also match any
@@ -50,6 +47,6 @@ echo "ERROR: unredacted postgres:// URL found in source. Use redactPgUrl() befor
echo ""
echo "$FILTERED"
echo ""
echo "Allowed exemption: append an allow-pg-url-literal comment marker on the line"
echo "Allowed exemption: append \"/* allow-pg-url-literal */\" comment on the line"
echo "(only for fixtures and the redactor itself)."
exit 1
+1 -9
View File
@@ -41,14 +41,6 @@ ROOT="$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
cd "$ROOT"
TARGET_DIR="${1:-test}"
# When scanning the default root, also lint evals/**/*.test.ts — those files
# are collected into the CI matrix (scripts/test-shard.sh) and must obey the
# same isolation rules as everything else CI executes. An explicit TARGET_DIR
# argument (guard self-test fixtures) scans only itself.
EXTRA_DIRS=""
if [ "$TARGET_DIR" = "test" ] && [ -d evals ]; then
EXTRA_DIRS="evals"
fi
ALLOWLIST_FILE="$ROOT/scripts/check-test-isolation.allowlist"
# Read allowlist (one filename per line, # comments allowed). Empty file
@@ -80,7 +72,7 @@ is_allowlisted() {
# Find non-serial unit test files (excluding test/e2e). Portable across
# bash 3.2 (macOS default) and bash 4+; no mapfile.
FILE_LIST="$(find "$TARGET_DIR" $EXTRA_DIRS -name '*.test.ts' \
FILE_LIST="$(find "$TARGET_DIR" -name '*.test.ts' \
-not -name '*.serial.test.ts' \
-not -path "*/e2e/*" \
-type f 2>/dev/null | sort)"
+2 -4
View File
@@ -101,12 +101,10 @@ done
IFS='|' eval 'PATTERN="${PATTERN_PARTS[*]}"'
# Find tool.
# evals/ joins the scan: its *.test.ts files are collected into the CI
# matrix (scripts/test-shard.sh) and carry the same privacy bar.
if command -v rg >/dev/null 2>&1; then
matches="$(rg -niH --no-heading -t ts "$PATTERN" test evals 2>/dev/null || true)"
matches="$(rg -niH --no-heading -t ts "$PATTERN" test 2>/dev/null || true)"
elif command -v grep >/dev/null 2>&1; then
matches="$(grep -rniE --include='*.test.ts' "$PATTERN" test evals 2>/dev/null || true)"
matches="$(grep -rniE --include='*.test.ts' "$PATTERN" test 2>/dev/null || true)"
else
echo "check-test-real-names: ERROR: neither rg nor grep available." >&2
exit 2
+4 -4
View File
@@ -11,12 +11,12 @@
# bash scripts/ci-local.sh --clean # nuke named volumes for cold debug
# bash scripts/ci-local.sh --no-shard # debug: run E2E sequentially against postgres-1 only
#
# 4-way E2E sharding: 4 pgvector services on host ports 5434-5437. The test/e2e/ file set splits
# roughly N/4 per shard; shards run in parallel. Within a shard, files run
# 4-way E2E sharding: 4 pgvector services on host ports 5434-5437. The 36 E2E
# files split N/4 per shard; shards run in parallel. Within a shard, files run
# sequentially (TRUNCATE CASCADE no-race property documented in run-e2e.sh).
# Wall-time on a 16-core host: ~6 min sequential -> ~1.5-2 min sharded.
#
# Stronger than PR CI: PR CI runs a handful of named files across its tiers; this runs every test/e2e file.
# Stronger than PR CI: PR CI runs only Tier 1's 2 files; this runs all 36.
set -euo pipefail
@@ -231,7 +231,7 @@ else
echo "$SELECTED" | tr " " "\n" | grep -v "^$" > /tmp/e2e-selected.txt
fi'
else
# Empty file -> run-e2e.sh uses default glob (every test/e2e file).
# Empty file -> run-e2e.sh uses default glob (all 36 E2E files).
DIFF_E2E_PREP='> /tmp/e2e-selected.txt'
fi
RUN_PHASES_CMD="echo \"[runner] guards + typecheck (run once before sharding)\"
+6 -6
View File
@@ -17,15 +17,15 @@
# guard class selftest notes
check-no-double-retry.sh scanner yes regex hole fixed in W0 (could not match `() =>`); perl multi-line pass replaces never-installed pcregrep
check-jsonb-pattern.sh scanner yes nested-paren hole fixed in W0; safe ::text::jsonb spelling stays unflagged
check-jsonb-params.mjs scanner yes positional $N::jsonb AST-lite scanner; argv/env root override; not in verify CHECKS: exercised by its unit test + self-test fixtures
check-jsonb-params.mjs scanner yes positional $N::jsonb AST-lite scanner; argv/env root override
check-batch-audit-site.sh scanner todo
check-bun-test-timeout.sh scanner todo not in verify CHECKS: runs directly as a test.yml verify-job step
check-bun-test-timeout.sh scanner todo
check-fixture-privacy.sh scanner todo
check-no-legacy-getconnection.sh scanner todo was reachable from neither verify nor CI pre-W0 (check:all only)
check-no-pii-in-agent-voice.sh scanner todo
check-operations-filter-bypass.sh scanner todo
check-pagetype-exhaustive.sh scanner todo wired into verify CHECKS (v0.45.x test/eval/CI pass; was registered-but-never-executed)
check-pg-url-redaction.sh scanner todo wired into verify CHECKS (v0.45.x test/eval/CI pass; was registered-but-never-executed)
check-pagetype-exhaustive.sh scanner todo
check-pg-url-redaction.sh scanner todo
check-privacy.sh scanner todo
check-progress-to-stdout.sh scanner todo
check-proposal-pii.sh scanner todo
@@ -47,12 +47,12 @@ check-exports-count.sh scanner todo was reachable from neither verify nor CI pre
check-trailing-newline.sh scanner todo was reachable from neither verify nor CI pre-W0 (check:all only)
check-test-isolation.sh scanner todo allowlist data file: check-test-isolation.allowlist
check-admin-build.sh buildfresh exempt runs the admin build; the build is the test
check-admin-embedded.sh buildfresh exempt embed freshness diff; not in verify CHECKS: duplicates check:admin-build's build
check-admin-embedded.sh buildfresh exempt embed freshness diff
check-admin-scope-drift.sh buildfresh exempt regenerates + diffs
check-bootstrap-templates.sh buildfresh exempt regenerates template tree + diffs
check-eval-glossary-fresh.sh buildfresh exempt regenerates + diffs
check-fuzz-purity.sh buildfresh exempt executes fuzz corpus
check-image-decoders-embedded.sh buildfresh exempt binary embed check; not in verify CHECKS: own bun build --compile too heavy per-verify
check-image-decoders-embedded.sh buildfresh exempt binary embed check
check-pglite-embedded.sh buildfresh exempt binary embed check
check-skills-manifest-fresh.sh buildfresh exempt regenerates + diffs
check-tool-catalog-fresh.sh buildfresh exempt regenerates + diffs
1 # CI guard registry (W0 fix-wave, Tier-1 #11 / D5.14).
17 # guard
18 check-no-double-retry.sh
19 check-jsonb-pattern.sh
20 check-jsonb-params.mjs
21 check-batch-audit-site.sh
22 check-bun-test-timeout.sh
23 check-fixture-privacy.sh
24 check-no-legacy-getconnection.sh
25 check-no-pii-in-agent-voice.sh
26 check-operations-filter-bypass.sh
27 check-pagetype-exhaustive.sh
28 check-pg-url-redaction.sh
29 check-privacy.sh
30 check-progress-to-stdout.sh
31 check-proposal-pii.sh
47 check-trailing-newline.sh
48 check-test-isolation.sh
49 check-admin-build.sh
50 check-admin-embedded.sh
51 check-admin-scope-drift.sh
52 check-bootstrap-templates.sh
53 check-eval-glossary-fresh.sh
54 check-fuzz-purity.sh
55 check-image-decoders-embedded.sh
56 check-pglite-embedded.sh
57 check-skills-manifest-fresh.sh
58 check-tool-catalog-fresh.sh
-79
View File
@@ -1,79 +0,0 @@
# scripts/lib/test-env.sh — shared helpers for the test-runner family
# (test-shard.sh, run-serial-tests.sh, run-slow-tests.sh, run-unit-parallel.sh,
# run-verify-parallel.sh). Source AFTER cd'ing to the repo root:
#
# . scripts/lib/test-env.sh
#
# bash 3.2 compatible (macOS system bash): no mapfile, no wait -n, no ${var^^}.
# Every helper degrades gracefully inside the script-sandbox tests
# (test/scripts/run-unit-parallel.test.ts symlinks a minimal PATH with no
# sysctl/nproc/vm_stat/timeout and no package.json).
# ──────────────────────────────────────────────────────────────────────────
# CPU detection: Apple Silicon perf cores → Mac total physical → nproc → 4.
# Returns a single positive integer.
# ──────────────────────────────────────────────────────────────────────────
detect_cpus() {
local n=""
n=$(sysctl -n hw.perflevel0.physicalcpu 2>/dev/null) && [ -n "$n" ] && [ "$n" -gt 0 ] && echo "$n" && return
n=$(sysctl -n hw.physicalcpu 2>/dev/null) && [ -n "$n" ] && [ "$n" -gt 0 ] && echo "$n" && return
n=$(nproc 2>/dev/null) && [ -n "$n" ] && [ "$n" -gt 0 ] && echo "$n" && return
echo 4
}
# ──────────────────────────────────────────────────────────────────────────
# Available-memory detection (MB). macOS: vm_stat free + inactive +
# speculative + purgeable pages (inactive/purgeable are reclaimable on
# pressure, which is exactly the scenario we size for). Linux: MemAvailable.
# Unknown platform → 0, and the caller skips adaptation entirely.
# ──────────────────────────────────────────────────────────────────────────
detect_available_mem_mb() {
if command -v vm_stat >/dev/null 2>&1; then
vm_stat 2>/dev/null | awk '
/page size of/ { psize = $8 }
/Pages free/ { free = $NF }
/Pages inactive/ { inactive = $NF }
/Pages speculative/ { spec = $NF }
/Pages purgeable/ { purge = $NF }
END {
gsub(/\./, "", free); gsub(/\./, "", inactive)
gsub(/\./, "", spec); gsub(/\./, "", purge)
if (psize == 0) psize = 16384
printf "%d\n", (free + inactive + spec + purge) * psize / 1048576
}'
return
fi
if [ -r /proc/meminfo ]; then
awk '/MemAvailable/ { printf "%d\n", $2 / 1024; found = 1 } END { if (!found) print 0 }' /proc/meminfo
return
fi
echo 0
}
# ──────────────────────────────────────────────────────────────────────────
# PGLite schema snapshot: build (idempotent, ~40ms when fresh; mkdir-lock
# concurrency-safe; hash folds handler-migration source) and export
# GBRAIN_PGLITE_SNAPSHOT for child bun processes. 500+ test files each
# cold-boot PGLite + replay every migration without it (~3.5x per booting
# file — see docs/TESTING.md).
#
# No-op when GBRAIN_NO_SNAPSHOT=1 or when a parent runner already exported
# the path (double-building is harmless but noisy). Non-fatal on build
# failure — tests fall back to cold init. The one-line "active" echo makes
# a silent fall-back-to-cold-init regression visible in CI logs.
# $1: label for log lines (defaults to test-env).
# ──────────────────────────────────────────────────────────────────────────
ensure_pglite_snapshot() {
local label="${1:-test-env}"
[ "${GBRAIN_NO_SNAPSHOT:-0}" = "1" ] && return 0
if [ -n "${GBRAIN_PGLITE_SNAPSHOT:-}" ]; then
echo "[$label] PGLite snapshot active (inherited): $GBRAIN_PGLITE_SNAPSHOT" >&2
return 0
fi
if bun run build:pglite-snapshot >/dev/null 2>&1; then
export GBRAIN_PGLITE_SNAPSHOT=test/fixtures/pglite-snapshot.tar
echo "[$label] PGLite snapshot active: $GBRAIN_PGLITE_SNAPSHOT" >&2
else
echo "[$label] snapshot build failed (non-fatal) — tests run with cold init" >&2
fi
}
-1
View File
@@ -96,7 +96,6 @@ mkdir -p "$E2E_TMP_HOME/.gbrain"
for _e2e_var in $(env | grep -oE '^(CONDUCTOR_|MCP_|OPENCLAW_|HERMES_|GROK_|OPENCODE_|GBRAIN_)[A-Za-z0-9_]*' | sort -u); do
case "$_e2e_var" in
GBRAIN_HOME) ;; # required for HOME isolation (set above) — keep
GBRAIN_PGLITE_SNAPSHOT) ;; # snapshot fast-path fixture (exported by ci-local.sh / runners) — keep
GBRAIN_TEST_ALLOW_DATABASE_URL) ;; # #3485 preload opt-in (set above) — keep
GBRAIN_E2E_ALLOW_DB) ;; # #3485 name-floor opt-in — the guard's own error
# message tells operators to set it; stripping it
-320
View File
@@ -1,320 +0,0 @@
/**
* scripts/run-eval-canary.ts — hermetic CLI retrieval-quality canary.
*
* Boots a throwaway PGLite brain under a temp GBRAIN_HOME, seeds the qrels
* fixture corpus, then spawns the REAL gbrain CLI to run the qrels
* correctness gate with the deterministic embedder (basis-vector query
* embeddings). No API keys, no network, no writes to the personal brain,
* no writes to tracked files in check mode.
*
* Seeding is the "V2" shape (feasibility-spike finding, BINDING): the
* expected-top1 page carries its query text in the `timeline` column too,
* because page-grain FTS (`pages.search_vector`) indexes title(A) +
* timeline(C) ONLY — `compiled_truth` is deliberately unindexed. With
* V1-style seeding (query text in compiled_truth only) the title arm votes
* only for the sibling page and expected_top1 is structurally 0.0.
*
* Modes:
* default check mode (CI): assert exit 0 + floors, print a one-line
* summary, clean up. Writes nothing to tracked files.
* record mode (pass the record flag) everything above PLUS append one
* EvalRunRecord-shaped JSONL line to
* <repo>/.gbrain-evals/eval-results.jsonl (the eval ledger).
*
* Honest scope: this gates the hybrid ranking pipeline (keyword/title/alias
* arms + RRF against gold qrels) with synthetic vectors. Semantic-embedding
* regressions remain the keyed eval suites' job.
*
* Budget: ≤60s under a saturated pool (two PGLite boots). If it breaches
* ~100s under contention, move the verify entry into the serial-tests CI
* job instead (same fallback as the chronicle eval).
*/
import { appendFileSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { dirname, join, resolve } from 'node:path';
import { fileURLToPath } from 'node:url';
import { execSync, spawnSync } from 'node:child_process';
import type { BrainEngine } from '../src/core/engine.ts';
import type { ChunkInput } from '../src/core/types.ts';
import { basisEmbedding, parseLegacyQrels } from '../src/eval/deterministic-embed.ts';
import type { LegacyQrelsQuery } from '../src/eval/deterministic-embed.ts';
const ROOT = resolve(dirname(fileURLToPath(import.meta.url)), '..');
const QRELS_PATH = join(ROOT, 'test', 'fixtures', 'eval-baselines', 'qrels-search.json');
// The embedding space the throwaway brain is pinned to. The gateway must be
// configured with this BEFORE initSchema (the schema's vector(dims) columns
// derive from gateway config at initSchema time — config read only, no key
// needed), and the brain's GBRAIN_HOME config.json pins the same model+dims
// so the CLI subprocess resolves 1536 too.
const EMBEDDING_MODEL = 'openai:text-embedding-3-large';
const EMBEDDING_DIMENSIONS = 1536;
// Env the child must NOT inherit: engine reroutes (a stray DATABASE_URL
// flips the engine to postgres; a brain id reroutes to a mount), embedding
// overrides (would fight the pinned 1536 space), and provider keys (the
// canary must behave identically keyed and keyless — determinism by
// construction, not by the parent's shell profile).
const CHILD_ENV_STRIP = [
'DATABASE_URL',
'GBRAIN_DATABASE_URL',
'GBRAIN_BRAIN_ID',
'GBRAIN_SOURCE',
'GBRAIN_EMBEDDING_MODEL',
'GBRAIN_EMBEDDING_DIMENSIONS',
'OPENAI_API_KEY',
'ANTHROPIC_API_KEY',
'ZEROENTROPY_API_KEY',
'VOYAGE_API_KEY',
'OPENROUTER_API_KEY',
'DASHSCOPE_API_KEY',
'GOOGLE_GENERATIVE_AI_API_KEY',
'GEMINI_API_KEY',
];
// The legacy qrels parser lives with the embedder builder — one parser for
// the shape (re-exported here for the test that drives this runner).
export { parseLegacyQrels };
export type { LegacyQrelsQuery };
/**
* Seed the V2 canary corpus. For each query's relevant slugs:
* - putPage typed by prefix (person/company/note), title = slug tail;
* the expected-top1 page carries the primary text in BOTH
* compiled_truth and timeline (the V2 amendment — timeline is what
* page-grain FTS indexes); siblings carry "Mentioned in context of
* <query>" in timeline.
* - upsertChunks with the same fixture text, basisEmbedding at the
* query's dim, token_count 10, chunk_source compiled_truth/timeline.
*/
export async function seedCanaryCorpus(engine: BrainEngine, queries: LegacyQrelsQuery[]): Promise<void> {
const seenSlugs = new Set<string>();
for (const q of queries) {
for (const slug of q.relevant_slugs) {
if (seenSlugs.has(slug)) continue;
seenSlugs.add(slug);
const isExpected = slug === q.first_relevant_slug;
const primaryText = `Primary content about ${q.query}`;
const mentionText = `Mentioned in context of ${q.query}`;
const type = slug.startsWith('people/')
? 'person'
: slug.startsWith('companies/')
? 'company'
: 'note';
await engine.putPage(slug, {
type,
title: slug.split('/').pop() ?? slug,
compiled_truth: isExpected ? primaryText : '',
// V2: the expected page's query text goes in timeline too — that is
// the page-grain-FTS-indexed column (title A + timeline C).
timeline: isExpected ? primaryText : mentionText,
});
const chunk: ChunkInput = {
chunk_index: 0,
chunk_text: isExpected ? primaryText : mentionText,
chunk_source: isExpected ? 'compiled_truth' : 'timeline',
embedding: basisEmbedding(q.embedding_dim, EMBEDDING_DIMENSIONS),
token_count: 10,
};
await engine.upsertChunks(slug, [chunk]);
}
}
}
interface GateJson {
verdict: 'pass' | 'fail';
correctness_gate: {
ran: boolean;
summary?: {
k: number;
queries_total: number;
queries_run: number;
queries_errored: number;
mean_recall_at_k: number;
first_relevant_hit_rate: number;
expected_top1_hit_rate: number;
expected_top1_denominator: number;
};
thresholds?: {
recall_at_k: number;
first_relevant_hit: number;
expected_top1: number;
};
breaches?: Array<Record<string, unknown>>;
};
}
function shortSha(): string {
try {
return execSync('git rev-parse --short HEAD', { cwd: ROOT, encoding: 'utf-8' }).trim();
} catch {
return 'unknown';
}
}
async function main(): Promise<number> {
const recordMode = process.argv.includes('--record');
const startedAt = Date.now();
const tmpHome = mkdtempSync(join(tmpdir(), 'gbrain-eval-canary-'));
// Keep the RUNNER's own gbrain home inside the sandbox too, so nothing in
// the seeding path can read or write the operator's real ~/.gbrain.
process.env.GBRAIN_HOME = tmpHome;
try {
const gbrainDir = join(tmpHome, '.gbrain');
mkdirSync(gbrainDir, { recursive: true });
const dbPath = join(gbrainDir, 'brain.pglite');
writeFileSync(
join(gbrainDir, 'config.json'),
JSON.stringify(
{
engine: 'pglite',
database_path: dbPath,
embedding_model: EMBEDDING_MODEL,
embedding_dimensions: EMBEDDING_DIMENSIONS,
},
null,
2,
) + '\n',
);
// Gateway config BEFORE initSchema (dims gotcha above). Empty env
// snapshot: no key is consulted, and none is needed for schema sizing.
const { configureGateway } = await import('../src/core/ai/gateway.ts');
configureGateway({
embedding_model: EMBEDDING_MODEL,
embedding_dimensions: EMBEDDING_DIMENSIONS,
env: {},
});
const { PGLiteEngine } = await import('../src/core/pglite-engine.ts');
const engine = new PGLiteEngine();
await engine.connect({ engine: 'pglite', database_path: dbPath });
await engine.initSchema();
const qrelsRaw = readFileSync(QRELS_PATH, 'utf-8');
await seedCanaryCorpus(engine, parseLegacyQrels(qrelsRaw));
// PGLite is single-writer: release the brain before the CLI child opens it.
await engine.disconnect();
const childEnv: Record<string, string | undefined> = { ...process.env };
for (const k of CHILD_ENV_STRIP) delete childEnv[k];
childEnv.GBRAIN_HOME = tmpHome;
// Spawn the REAL CLI. cwd is the temp home (not the repo) so repo-local
// dotfiles and Bun-auto-loaded .env files can't reroute the brain.
const child = spawnSync(
process.execPath,
[join(ROOT, 'src', 'cli.ts'), 'eval', 'gate', '--qrels', QRELS_PATH, '--embedder', 'deterministic', '--json'],
{
cwd: tmpHome,
env: childEnv as NodeJS.ProcessEnv,
encoding: 'utf-8',
timeout: 110_000,
maxBuffer: 32 * 1024 * 1024,
},
);
if (child.error) {
process.stderr.write(`[eval-canary] FAIL: could not spawn the CLI: ${child.error.message}\n`);
return 1;
}
if (child.status !== 0) {
process.stderr.write(`[eval-canary] FAIL: gate exit=${child.status ?? 'null(timeout/signal)'}\n`);
process.stderr.write(`[eval-canary] gate stdout tail:\n${(child.stdout ?? '').slice(-2000)}\n`);
process.stderr.write(`[eval-canary] gate stderr tail:\n${(child.stderr ?? '').slice(-2000)}\n`);
return 1;
}
// Parse the gate's JSON envelope (stdout carries only the envelope; any
// engine warnings go to stderr).
const stdout = child.stdout ?? '';
const jsonStart = stdout.indexOf('{');
if (jsonStart < 0) {
process.stderr.write(`[eval-canary] FAIL: no JSON found on gate stdout:\n${stdout.slice(-2000)}\n`);
return 1;
}
const gate = JSON.parse(stdout.slice(jsonStart)) as GateJson;
const summary = gate.correctness_gate.summary;
const floors = gate.correctness_gate.thresholds;
if (gate.verdict !== 'pass' || !summary || !floors) {
process.stderr.write(`[eval-canary] FAIL: verdict=${gate.verdict} summary=${JSON.stringify(summary)}\n`);
return 1;
}
// Exit 0 already implies floors held; assert explicitly anyway so a
// future exit-code regression in the gate can't silently pass the canary.
const breaches: string[] = [];
if (summary.queries_errored > 0) breaches.push(`queries_errored=${summary.queries_errored}`);
if (summary.mean_recall_at_k < floors.recall_at_k) breaches.push(`recall ${summary.mean_recall_at_k} < ${floors.recall_at_k}`);
if (summary.first_relevant_hit_rate < floors.first_relevant_hit) breaches.push(`first_relevant ${summary.first_relevant_hit_rate} < ${floors.first_relevant_hit}`);
if (summary.expected_top1_denominator > 0 && summary.expected_top1_hit_rate < floors.expected_top1) {
breaches.push(`expected_top1 ${summary.expected_top1_hit_rate} < ${floors.expected_top1}`);
}
if (breaches.length > 0) {
process.stderr.write(`[eval-canary] FAIL: ${breaches.join('; ')}\n`);
return 1;
}
const commit = shortSha();
const durationMs = Date.now() - startedAt;
process.stdout.write(
`[eval-canary] PASS commit=${commit}` +
` mean_recall_at_k=${summary.mean_recall_at_k.toFixed(4)}` +
` first_relevant_hit_rate=${summary.first_relevant_hit_rate.toFixed(4)}` +
` expected_top1_hit_rate=${summary.expected_top1_hit_rate.toFixed(4)}` +
` floors=${floors.recall_at_k}/${floors.first_relevant_hit}/${floors.expected_top1}` +
` k=${summary.k} queries=${summary.queries_run}/${summary.queries_total}` +
` duration_ms=${durationMs}\n`,
);
if (recordMode) {
// EvalRunRecord-shaped ledger line (matches src/commands/eval-run-all.ts;
// suite widened to the canary's own name, mode 'n/a' per the
// search-mode-independent convention).
const record = {
schema_version: 3,
run_id: `${commit}-retrieval-canary-na-0`,
ran_at: new Date().toISOString(),
suite: 'retrieval-canary',
mode: 'n/a',
commit,
seed: 0,
params: {
qrels: 'test/fixtures/eval-baselines/qrels-search.json',
embedder: 'deterministic',
k: summary.k,
metrics: {
mean_recall_at_k: summary.mean_recall_at_k,
first_relevant_hit_rate: summary.first_relevant_hit_rate,
expected_top1_hit_rate: summary.expected_top1_hit_rate,
expected_top1_denominator: summary.expected_top1_denominator,
queries_run: summary.queries_run,
queries_total: summary.queries_total,
},
floors: {
recall_at_k: floors.recall_at_k,
first_relevant_hit: floors.first_relevant_hit,
expected_top1: floors.expected_top1,
},
},
status: 'completed',
duration_ms: durationMs,
};
const ledgerDir = join(ROOT, '.gbrain-evals');
mkdirSync(ledgerDir, { recursive: true });
const ledgerPath = join(ledgerDir, 'eval-results.jsonl');
appendFileSync(ledgerPath, JSON.stringify(record) + '\n', 'utf-8');
process.stdout.write(`[eval-canary] recorded → ${ledgerPath}\n`);
}
return 0;
} catch (err) {
process.stderr.write(`[eval-canary] FAIL: ${(err as Error).stack ?? (err as Error).message}\n`);
return 1;
} finally {
rmSync(tmpHome, { recursive: true, force: true });
}
}
if (import.meta.main) {
process.exit(await main());
}
+15 -272
View File
@@ -1,81 +1,18 @@
#!/usr/bin/env bash
# scripts/run-serial-tests.sh — run *.serial.test.ts files, one bun process per
# file, POOLED across files.
# scripts/run-serial-tests.sh — run *.serial.test.ts files with --max-concurrency=1.
#
# Serial files are tests that share file-wide state (top-level mock.module,
# module-level singletons that intentionally cross test cases) and would race
# under intra-file concurrency. Discovered via filename suffix; no annotation
# inside the file is needed.
#
# Each file gets its OWN bun process. `--max-concurrency=1` alone was not
# enough: files in the same process share the module registry, so a top-level
# `mock.module(...)` in one file leaks into the next file's imports. Per-file
# processes give true isolation — and that isolation is per-PROCESS, not
# per-machine, so separate processes run CONCURRENTLY through the pool below.
# (The previous runner executed the ~140 processes strictly one-at-a-time:
# an 8.5-minute CI job whose serialization was never required by the
# quarantine contract.)
#
# Excluded by run-unit-shard.sh and run-unit-parallel.sh's parallel pass.
# Invoked separately by run-unit-parallel.sh after the parallel pass succeeds.
#
# Knobs:
# GBRAIN_SERIAL_POOL=N pool width (default min(detect_cpus, 4),
# then memory-adapted; 1 restores the old
# fully-sequential behavior)
# GBRAIN_SERIAL_FILE_TIMEOUT=S wall-clock kill per file (default 300;
# needs timeout/gtimeout on PATH, else no wrap)
# GBRAIN_TEST_MEM_PER_FILE_MB per-process memory budget (default 1536)
# GBRAIN_TEST_NO_MEM_ADAPT=1 skip the memory clamp
set -euo pipefail
# #3485: serial tests need no database — strip ambient DB URLs at this
# wrapper boundary (same four-layer guard as run-slow-tests.sh / the
# parallel runner) so the bunfig preload guard passes and nothing can
# reach a real brain.
unset DATABASE_URL GBRAIN_DATABASE_URL
cd "$(dirname "$0")/.."
. scripts/lib/test-env.sh
# ──────────────────────────────────────────────────────────────────────────
# EXCLUSIVE_FILES: files that must never run concurrently with anything else
# (machine-global state or contention-critical timing). They run sequentially
# AFTER the pool drains, without the wall-clock kill (a SIGKILL
# mid-registration could strand a real scheduled job). Growth guard:
# test/scripts/serial-files.test.ts fails when this list grows past 3
# entries — every addition needs a justification comment like the entries
# below.
# ──────────────────────────────────────────────────────────────────────────
EXCLUSIVE_FILES=(
# launchd/cron lifecycle arc: install → self-disable → reinstall →
# uninstall ordering against (PATH-shimmed) launchctl; the arc asserts
# machine-level sequencing and is the flake-class canary.
"test/autopilot-launchd-lifecycle.serial.test.ts"
# hardenBrainRepo({installCron:true}) executes REAL launchctl/crontab
# (src/core/brain-repo-durability.ts) — a concurrent or killed run could
# strand a real scheduled job on the machine.
"test/brain-durability-hook.serial.test.ts"
# hardenBrainRepo's own scaffolding commit fires the just-installed
# post-commit hook (background push) which races the synchronous
# push-probe on the same bare remote ("cannot lock ref" →
# needs_attention non-empty). The race is intra-call; pooled CPU
# contention widens the window past what the assertions tolerate
# (observed on a 4-vCPU CI runner, never locally). Sequential lane
# restores master-era timing until the probe learns to retry ref-lock
# contention.
"test/brain-repo-durability.serial.test.ts"
)
is_exclusive() {
local f="$1" e
for e in "${EXCLUSIVE_FILES[@]}"; do
[ "$f" = "$e" ] && return 0
done
return 1
}
# Use while-read for portability to macOS bash 3.2 (no mapfile).
files=()
while IFS= read -r f; do
@@ -87,229 +24,35 @@ if [ "${#files[@]}" -eq 0 ]; then
exit 0
fi
# --dry-run-list mirrors run-unit-shard.sh for inline checks/tests. Lists
# ALL discovered files, pooled and exclusive alike.
# --dry-run-list mirrors run-unit-shard.sh for inline checks/tests.
if [ "${1:-}" = "--dry-run-list" ]; then
printf '%s\n' "${files[@]}"
exit 0
fi
ensure_pglite_snapshot "serial-tests"
# Partition into pooled vs exclusive (exclusive entries missing from the
# discovered set are simply ignored — the list names repo files, and a
# sandbox copy of this script won't have them).
pool_files=()
exclusive_present=()
for f in "${files[@]}"; do
if is_exclusive "$f"; then
exclusive_present+=("$f")
else
pool_files+=("$f")
fi
done
# ──────────────────────────────────────────────────────────────────────────
# Pool sizing: min(detect_cpus, 4) — each pooled bun process can hold a
# PGLite WASM instance (~1.5GB) — then clamped by available memory (same
# layer-1 doctrine as run-unit-parallel.sh, 4GB OS reserve).
# ──────────────────────────────────────────────────────────────────────────
POOL="${GBRAIN_SERIAL_POOL:-}"
if [ -z "$POOL" ]; then
POOL=$(detect_cpus)
[ "$POOL" -gt 4 ] && POOL=4
if [ "${GBRAIN_TEST_NO_MEM_ADAPT:-0}" != "1" ]; then
MEM_PER_FILE_MB="${GBRAIN_TEST_MEM_PER_FILE_MB:-1536}"
AVAIL_MB=$(detect_available_mem_mb)
if [ "${AVAIL_MB:-0}" -gt 0 ] 2>/dev/null; then
BUDGET_MB=$((AVAIL_MB - 4096))
[ "$BUDGET_MB" -lt "$MEM_PER_FILE_MB" ] && BUDGET_MB="$MEM_PER_FILE_MB"
MAX_POOL=$((BUDGET_MB / MEM_PER_FILE_MB))
[ "$MAX_POOL" -lt 1 ] && MAX_POOL=1
[ "$POOL" -gt "$MAX_POOL" ] && POOL="$MAX_POOL"
fi
fi
fi
if ! printf '%s' "$POOL" | grep -qE '^[0-9]+$' || [ "$POOL" -lt 1 ]; then
echo "[serial-tests] ERROR: invalid pool size: $POOL" >&2
exit 2
fi
# Wall-clock kill per pooled file: contains the exit-hang class (a bun
# process that finishes its tests but never exits). SIGTERM first, SIGKILL
# after a grace period (`timeout -k`). macOS without coreutils has neither
# binary — run unwrapped there (CI is Linux and always wraps).
PER_FILE_TIMEOUT="${GBRAIN_SERIAL_FILE_TIMEOUT:-300}"
TIMEOUT_BIN=""
command -v timeout >/dev/null 2>&1 && TIMEOUT_BIN="timeout"
[ -z "$TIMEOUT_BIN" ] && command -v gtimeout >/dev/null 2>&1 && TIMEOUT_BIN="gtimeout"
LOG_DIR=$(mktemp -d "${TMPDIR:-/tmp}/gbrain-serial.XXXXXX")
trap 'rm -rf "$LOG_DIR"' EXIT
if [ -n "$TIMEOUT_BIN" ]; then
TIMEOUT_DESC="${PER_FILE_TIMEOUT}s via $TIMEOUT_BIN"
else
TIMEOUT_DESC="none (no timeout/gtimeout on PATH)"
fi
echo "[serial-tests] ${#files[@]} file(s): pool=$POOL (${#exclusive_present[@]} exclusive), per-file timeout=$TIMEOUT_DESC"
# Per-test timeout is 120s (not the fast-loop 60s): pooled contention can
# push a 30-50s file past 60s — the same flake class the slow lane hardened
# against in v0.40.10. The literal `bun test --max-concurrency=1` below is
# contract-pinned by test/scripts/serial-files.test.ts.
run_one_file() {
# $1 file, $2 log path, $3 exit-sentinel path, $4 wrap ("wrap"|"nowrap")
local f="$1" log="$2" exitf="$3" wrap="$4" rc=0
if [ "$wrap" = "wrap" ] && [ -n "$TIMEOUT_BIN" ]; then
"$TIMEOUT_BIN" -k 15 "$PER_FILE_TIMEOUT" \
bun test --max-concurrency=1 --timeout=120000 "$f" > "$log" 2>&1 || rc=$?
else
bun test --max-concurrency=1 --timeout=120000 "$f" > "$log" 2>&1 || rc=$?
fi
echo "$rc" > "$exitf"
}
start_epoch=$(date +%s)
idx=0
if [ "${#pool_files[@]}" -gt 0 ]; then
for f in "${pool_files[@]}"; do
while [ "$(jobs -rp | wc -l | tr -d ' ')" -ge "$POOL" ]; do
sleep 0.2
done
(
s=$(date +%s)
run_one_file "$f" "$LOG_DIR/$idx.log" "$LOG_DIR/$idx.exit" "wrap"
e=$(date +%s)
echo "$((e - s))" > "$LOG_DIR/$idx.dur"
) &
idx=$((idx + 1))
done
wait
fi
# Exclusive lane: sequential, unwrapped (see EXCLUSIVE_FILES comment).
if [ "${#exclusive_present[@]}" -gt 0 ]; then
for f in "${exclusive_present[@]}"; do
s=$(date +%s)
run_one_file "$f" "$LOG_DIR/$idx.log" "$LOG_DIR/$idx.exit" "nowrap"
e=$(date +%s)
echo "$((e - s))" > "$LOG_DIR/$idx.dur"
idx=$((idx + 1))
done
fi
# ──────────────────────────────────────────────────────────────────────────
# Aggregate from the exit sentinels.
# exit 0 → pass
# exit 124 → killed by the per-file wall-clock timeout (real
# failure: the exit-hang class this cap exists for)
# exit 143 / 137, or a → EXTERNAL-KILL class: a stray SIGTERM/SIGKILL from
# missing sentinel outside this runner (sibling workspaces' process
# cleanup, memory jetsam — the same class
# run-unit-parallel.sh rescues). Queued for ONE
# sequential rescue re-run below; a rescue that
# fails again is a real failure. Never a silent pass.
# anything else → real failure
# ──────────────────────────────────────────────────────────────────────────
ordered_files=()
if [ "${#pool_files[@]}" -gt 0 ]; then ordered_files+=("${pool_files[@]}"); fi
if [ "${#exclusive_present[@]}" -gt 0 ]; then ordered_files+=("${exclusive_present[@]}"); fi
echo "[serial-tests] running ${#files[@]} file(s), one bun process per file"
# Each serial file gets its OWN bun process. `--max-concurrency=1` was not
# enough: files in the same process share the module registry, so a top-level
# `mock.module(...)` in one file leaks into the next file's imports
# (eval-takes-quality-runner mocks gateway.ts and the next file fails on
# `import { resetGateway }` because the mock factory didn't export it).
# Per-file processes give true isolation; cost is ~100ms startup × N files.
fail_count=0
failed_files=()
rescue_files=()
# Aggregate pass count across pooled files, re-emitted below in bun's own
# " N pass" summary format so run-unit-parallel.sh's headline counter
# (bun_summary_count) still sees the serial suite's tests. Failing files'
# logs are cat'ed raw (their " N pass/fail" lines land in the stream
# directly), so only PASSING files accumulate here — no double counting.
pass_total=0
i=0
for f in "${ordered_files[@]}"; do
dur="?"
[ -f "$LOG_DIR/$i.dur" ] && dur=$(cat "$LOG_DIR/$i.dur")
if [ ! -f "$LOG_DIR/$i.exit" ]; then
echo "[serial-tests] KILLED ${dur}s $f — missing exit sentinel (external kill/OOM) — queued for serial rescue" >&2
rescue_files+=("$f")
else
rc=$(cat "$LOG_DIR/$i.exit")
if [ "$rc" = "0" ]; then
summary=$(grep -E '^ *[0-9]+ pass' "$LOG_DIR/$i.log" | tail -1 | tr -d ' ' || true)
n=$(printf '%s' "$summary" | grep -oE '^[0-9]+' || echo 0)
pass_total=$((pass_total + n))
echo "[serial-tests] PASS ${dur}s $f ${summary:+($summary)}"
elif [ "$rc" = "137" ] && [ "$dur" != "?" ] && [ "$dur" -ge "$PER_FILE_TIMEOUT" ] 2>/dev/null; then
# 137 with full duration = OUR timeout's SIGKILL escalation (a hang
# that ignored SIGTERM), not an external kill — a real failure; a
# rescue re-run would just re-hang for another ~315s.
echo "[serial-tests] FAIL ${dur}s $f — exit 137 (hang survived SIGTERM; killed by ${PER_FILE_TIMEOUT}s per-file timeout)" >&2
cat "$LOG_DIR/$i.log" >&2
fail_count=$((fail_count + 1))
failed_files+=("$f")
elif [ "$rc" = "143" ] || [ "$rc" = "137" ]; then
echo "[serial-tests] KILLED ${dur}s $f — exit $rc (external SIGTERM/SIGKILL) — queued for serial rescue" >&2
rescue_files+=("$f")
else
note=""
[ "$rc" = "124" ] && note=" (killed by ${PER_FILE_TIMEOUT}s per-file timeout)"
echo "[serial-tests] FAIL ${dur}s $f — exit $rc$note" >&2
cat "$LOG_DIR/$i.log" >&2
fail_count=$((fail_count + 1))
failed_files+=("$f")
fi
for f in "${files[@]}"; do
if ! bun test --max-concurrency=1 --timeout=60000 "$f"; then
fail_count=$((fail_count + 1))
failed_files+=("$f")
fi
i=$((i + 1))
done
# Rescue pass: one sequential, unpooled re-run per externally-killed file.
# Phantoms pass here and the run stays green (with a rescue note); real
# failures fail again and go red. Mirrors run-unit-parallel.sh's doctrine.
if [ "${#rescue_files[@]}" -gt 0 ]; then
echo "[serial-tests] rescue pass: ${#rescue_files[@]} externally-killed file(s), re-running serially" >&2
for f in "${rescue_files[@]}"; do
# Exclusive-lane files keep their no-kill contract on rescue too (the
# lane exists because a SIGKILL mid-registration strands real state).
wrap_mode="wrap"
is_exclusive "$f" && wrap_mode="nowrap"
s=$(date +%s)
run_one_file "$f" "$LOG_DIR/$i.log" "$LOG_DIR/$i.exit" "$wrap_mode"
e=$(date +%s)
rc=$(cat "$LOG_DIR/$i.exit" 2>/dev/null || echo 1)
if [ "$rc" = "0" ]; then
summary=$(grep -E '^ *[0-9]+ pass' "$LOG_DIR/$i.log" | tail -1 | tr -d ' ' || true)
n=$(printf '%s' "$summary" | grep -oE '^[0-9]+' || echo 0)
pass_total=$((pass_total + n))
echo "[serial-tests] PASS $((e - s))s $f ${summary:+($summary)} (rescued: external-kill phantom)"
else
echo "[serial-tests] FAIL $((e - s))s $f — exit $rc on rescue re-run" >&2
cat "$LOG_DIR/$i.log" >&2
fail_count=$((fail_count + 1))
failed_files+=("$f")
fi
i=$((i + 1))
done
fi
# Slowest-file table: feeds flake triage + future weight mining.
echo "[serial-tests] slowest files:"
i=0
for f in "${ordered_files[@]}"; do
[ -f "$LOG_DIR/$i.dur" ] && echo "$(cat "$LOG_DIR/$i.dur") $f"
i=$((i + 1))
done | sort -rn | head -10 | sed 's/^/ /'
total_epoch=$(( $(date +%s) - start_epoch ))
if [ "$fail_count" -gt 0 ]; then
echo "" >&2
echo "[serial-tests] $fail_count file(s) failed (${total_epoch}s total):" >&2
echo "[serial-tests] $fail_count file(s) failed:" >&2
for f in "${failed_files[@]}"; do
echo " - $f" >&2
done
exit 1
fi
# bun-summary-format aggregate: run-unit-parallel.sh's headline counter
# (bun_summary_count awk: $1 numeric, $2 == "pass") reads this line — without
# it the serial suite's tests vanish from `bun run test`'s pass=N banner.
echo " $pass_total pass"
echo "[serial-tests] all ${#ordered_files[@]} file(s) passed in ${total_epoch}s (pool=$POOL)"
echo "[serial-tests] all ${#files[@]} file(s) passed"
-3
View File
@@ -11,9 +11,6 @@ set -euo pipefail
unset DATABASE_URL GBRAIN_DATABASE_URL
cd "$(dirname "$0")/.."
. scripts/lib/test-env.sh
ensure_pglite_snapshot "run-slow-tests"
slow_files=()
while IFS= read -r f; do
slow_files+=("$f")
+48 -5
View File
@@ -62,11 +62,54 @@ cd "$(dirname "$0")/.."
# fan-out — shards inherit a finished fixture. Opt out: GBRAIN_NO_SNAPSHOT=1
# (the migration-replay canary tests clear the env themselves regardless).
# ──────────────────────────────────────────────────────────────────────────
# detect_cpus / detect_available_mem_mb / ensure_pglite_snapshot live in the
# shared lib (also sourced by test-shard.sh, run-serial-tests.sh,
# run-slow-tests.sh) — one implementation, no copy drift.
. scripts/lib/test-env.sh
ensure_pglite_snapshot "run-unit-parallel"
if [ "${GBRAIN_NO_SNAPSHOT:-0}" != "1" ]; then
if bun run build:pglite-snapshot >/dev/null 2>&1; then
export GBRAIN_PGLITE_SNAPSHOT=test/fixtures/pglite-snapshot.tar
else
echo "[run-unit-parallel] snapshot build failed (non-fatal) — tests run with cold init" >&2
fi
fi
# ──────────────────────────────────────────────────────────────────────────
# CPU detection: Apple Silicon perf cores → Mac total physical → nproc → 4.
# Returns a single positive integer.
# ──────────────────────────────────────────────────────────────────────────
detect_cpus() {
local n=""
n=$(sysctl -n hw.perflevel0.physicalcpu 2>/dev/null) && [ -n "$n" ] && [ "$n" -gt 0 ] && echo "$n" && return
n=$(sysctl -n hw.physicalcpu 2>/dev/null) && [ -n "$n" ] && [ "$n" -gt 0 ] && echo "$n" && return
n=$(nproc 2>/dev/null) && [ -n "$n" ] && [ "$n" -gt 0 ] && echo "$n" && return
echo 4
}
# ──────────────────────────────────────────────────────────────────────────
# Available-memory detection (MB). macOS: vm_stat free + inactive +
# speculative + purgeable pages (inactive/purgeable are reclaimable on
# pressure, which is exactly the scenario we size for). Linux: MemAvailable.
# Unknown platform → 0, and the caller skips adaptation entirely.
# ──────────────────────────────────────────────────────────────────────────
detect_available_mem_mb() {
if command -v vm_stat >/dev/null 2>&1; then
vm_stat 2>/dev/null | awk '
/page size of/ { psize = $8 }
/Pages free/ { free = $NF }
/Pages inactive/ { inactive = $NF }
/Pages speculative/ { spec = $NF }
/Pages purgeable/ { purge = $NF }
END {
gsub(/\./, "", free); gsub(/\./, "", inactive)
gsub(/\./, "", spec); gsub(/\./, "", purge)
if (psize == 0) psize = 16384
printf "%d\n", (free + inactive + spec + purge) * psize / 1048576
}'
return
fi
if [ -r /proc/meminfo ]; then
awk '/MemAvailable/ { printf "%d\n", $2 / 1024; found = 1 } END { if (!found) print 0 }' /proc/meminfo
return
fi
echo 0
}
# ──────────────────────────────────────────────────────────────────────────
# Argument parsing. --shards N override wins over $SHARDS; both are clamped.
+21 -65
View File
@@ -25,61 +25,28 @@ set -uo pipefail
cd "$(dirname "$0")/.."
# detect_cpus + ensure_pglite_snapshot (the PGLite-booting eval checks use
# the snapshot fast-path when the shape matches).
. scripts/lib/test-env.sh
# ──────────────────────────────────────────────────────────────────────────
# Checks to run. Each entry is a bun-script name (the `package.json`
# "scripts" key), invoked as `bun run <name>`.
# Checks to run. Order is irrelevant (parallel), but keep stable for log
# determinism + grep-ability. Each entry is a bun-script name (the
# `package.json` "scripts" key), invoked as `bun run <name>`.
#
# ORDER MATTERS for wallclock: the spawn loop below is capped at
# GBRAIN_VERIFY_MAX_PARALLEL workers, so the heaviest checks go FIRST
# (LPT-style — makespan ≈ max(longest check, total/POOL)). The heavy block:
# typecheck (tsc), two `cp -R src` + `bun build --compile` binary builds,
# the admin vite+tsc build, the fuzz bundles, guard self-tests, the
# PGLite-booting eval checks, and the whole-tree greps. Everything after is
# sub-second; that tail keeps its historical order for grep-ability.
#
# To add a check: append to the right block. To skip in CI temporarily,
# comment the line — the runner doesn't care about count.
# To add a check: append to this array. To skip in CI temporarily, comment
# the line — the parallel runner doesn't care about count.
# ──────────────────────────────────────────────────────────────────────────
CHECKS=(
# ── heavy (longest-first) ──
"typecheck"
"check:admin-build"
"check:wasm"
"check:pglite-embedded"
"check:fuzz-purity"
# W0 fix-wave (Tier-1 #11): guard self-tests — every scanner guard proves it
# can fail (bad fixture → exit 1) before it counts as coverage. Registry:
# scripts/guards-manifest.tsv (package.json's stale `check:all` copy deleted).
"check:guard-self-test"
# Chronicle eval: $0, deterministic, exit-0-only-on-perfect (6 gold tasks).
# Boots its own PGLite — budget ≤60s under a saturated pool; if it breaches
# ~100s under contention, move it into the serial-tests CI job instead.
"check:eval-chronicle"
# Retrieval canary: $0, hermetic, deterministic-embedder CLI run of the
# qrels correctness gate. Boots two PGLite processes (seed + real CLI) —
# budget ≤60s under a saturated pool; if it breaches ~100s under
# contention, move it into the serial-tests CI job instead (same fallback
# as eval-chronicle above).
"check:eval-canary"
"check:bootstrap-templates"
"check:skill-brain-first"
"check:conversation-parser"
"check:resolver"
"check:privacy"
"check:test-names"
"check:test-isolation"
# ── light tail (sub-second greps; historical order) ──
"check:proposal-pii"
"check:test-names"
"check:jsonb"
"check:search-path"
"check:source-id-projection"
"check:source-config-leak"
"check:progress"
"check:no-tracked-symlinks"
"check:test-isolation"
"check:wasm"
"check:pglite-embedded"
"check:admin-build"
"check:admin-scope-drift"
"check:cli-exec"
"check:system-of-record"
@@ -88,11 +55,15 @@ CHECKS=(
"check:skills-manifest"
"check:no-pii-agent-voice"
"check:synthetic-corpus-privacy"
"check:skill-brain-first"
"check:fuzz-purity"
"check:operations-filter-bypass"
"check:gateway-routed"
"check:worker-pool-atomicity"
"check:doc-history"
"check:fixture-privacy"
"check:conversation-parser"
"check:resolver"
"check:source-scope-onboard"
"check:no-double-retry"
"check:batch-audit-site"
@@ -102,14 +73,17 @@ CHECKS=(
"check:pin-doc-privacy"
"check:worker-lock-renewal-shape"
"check:bootstrap-tag"
"check:bootstrap-templates"
"check:skill-refs"
# W0 fix-wave (Tier-1 #11): guard self-tests — every scanner guard proves it
# can fail (bad fixture → exit 1) before it counts as coverage. Registry:
# scripts/guards-manifest.tsv (package.json's stale `check:all` copy deleted).
"check:guard-self-test"
# Previously reachable ONLY from the deleted check:all (i.e. never run):
"check:newlines"
"check:exports-count"
"check:no-legacy-getconnection"
# Revived registered-but-never-executed guards (this pass):
"check:pagetype-exhaustive"
"check:pg-url-redaction"
"typecheck"
)
if [ "${#CHECKS[@]}" -eq 0 ]; then
@@ -150,22 +124,8 @@ if command -v gtimeout >/dev/null 2>&1; then TIMEOUT_BIN="gtimeout"
elif command -v timeout >/dev/null 2>&1; then TIMEOUT_BIN="timeout"
fi
# Bounded worker pool. Unbounded fan-out ran two `cp -R src` +
# `bun build --compile` builds, the admin vite build, tsc, and ~40 greps
# simultaneously on a 4-vCPU CI runner — pushing slow checks into the
# 120s per-check timeout (the documented flake class on slower hosts).
# Default = detect_cpus so a many-core dev machine keeps its wide fan-out;
# escape hatch: GBRAIN_VERIFY_MAX_PARALLEL=999.
MAX_PAR="${GBRAIN_VERIFY_MAX_PARALLEL:-$(detect_cpus)}"
if ! printf '%s' "$MAX_PAR" | grep -qE '^[0-9]+$' || [ "$MAX_PAR" -lt 1 ]; then
echo "ERROR: invalid GBRAIN_VERIFY_MAX_PARALLEL: $MAX_PAR" >&2
exit 2
fi
ensure_pglite_snapshot "verify-parallel"
START_TS=$(date +%s)
echo "[verify-parallel] running ${#CHECKS[@]} checks (pool=$MAX_PAR, timeout=${TIMEOUT}s, logs=$LOG_DIR)" >&2
echo "[verify-parallel] running ${#CHECKS[@]} checks in parallel (timeout=${TIMEOUT}s, logs=$LOG_DIR)" >&2
# ──────────────────────────────────────────────────────────────────────────
# Spawn one background process per check. Each child captures its own exit
@@ -178,10 +138,6 @@ echo "[verify-parallel] running ${#CHECKS[@]} checks (pool=$MAX_PAR, timeout=${T
PIDS=()
SAFE_NAMES=()
for c in "${CHECKS[@]}"; do
# Throttle to the worker pool (bash 3.2 — no wait -n; jobs -rp reaps).
while [ "$(jobs -rp | wc -l | tr -d ' ')" -ge "$MAX_PAR" ]; do
sleep 0.1
done
safe="${c//:/_}"
SAFE_NAMES+=("$safe")
LOG_FILE="$LOG_DIR/$safe.log"
+3 -21
View File
@@ -96,17 +96,6 @@ export function computeMedian(values: number[]): number {
: sorted[mid]!;
}
/**
* Quantile (nearest-rank) of a list of numbers. Empty input returns 0.
* q in [0, 1]; q=0.75 is the missing-file fallback weight (see partition).
*/
export function computeQuantile(values: number[], q: number): number {
if (values.length === 0) return 0;
const sorted = [...values].sort((a, b) => a - b);
const idx = Math.min(sorted.length - 1, Math.max(0, Math.ceil(q * sorted.length) - 1));
return sorted[idx]!;
}
export interface PartitionOpts {
/**
* Weight to assign files that are absent from the weights map. Defaults
@@ -144,15 +133,8 @@ export function partition(
const shards: string[][] = Array.from({ length: n }, () => []);
if (files.length === 0) return shards;
// Compute fallback weight from the p75 of present weights, unless the
// caller supplied an explicit override. p75, not median: the weight
// distribution is extremely right-skewed (median ~27ms, mean ~800ms —
// most files are trivial greps, the tail boots PGLite), and files
// MISSING from the map skew heavy (new integration tests land unweighted
// more often than new pure-unit tests). A median fallback modeled 45% of
// the corpus at ~30ms and let one shard silently carry the unweighted
// heavies; p75 over-weights small new files slightly (harmless — LPT
// self-corrects on the next mine) instead of under-weighting big ones.
// Compute fallback weight from the median of present weights, unless
// the caller supplied an explicit override.
let fallback: number;
if (opts.fallbackWeight !== undefined) {
if (!Number.isFinite(opts.fallbackWeight) || opts.fallbackWeight < 0) {
@@ -162,7 +144,7 @@ export function partition(
}
fallback = opts.fallbackWeight;
} else {
fallback = computeQuantile(Array.from(weights.values()), 0.75);
fallback = computeMedian(Array.from(weights.values()));
}
// Cold-start guard: if the weights map is empty AND no explicit
// fallback was supplied, every effective weight would be 0 and LPT
+2 -19
View File
@@ -56,8 +56,6 @@ fi
cd "$(dirname "$0")/.."
. scripts/lib/test-env.sh
# Collect non-E2E, non-serial unit test files. Slow files INCLUDED — see
# header comment. Local run-unit-shard.sh excludes slow files (different
# policy by design).
@@ -74,13 +72,7 @@ cd "$(dirname "$0")/.."
# total bounded. With 10 matrix shards the per-shard total drops to ~272s.
# Dedicated jobs run in parallel so total CI wallclock = max(matrix ~4.5min,
# slow-eval ~3.3min, slow-entity-resolve-perf ~2.6min) ≈ 4.5min.
# evals/ is included: its *.test.ts files (eval-harness unit tests) were
# previously collected by NO runner — 45+ real tests never executed anywhere.
# Every collected evals file must be KEYLESS (no API keys, no network) —
# enforced by the allowlist guard in test/scripts/evals-collection.test.ts.
# The local fast loop (run-unit-shard.sh) stays test-only by design (see
# docs/TESTING.md "CI vs local: intentionally divergent file sets").
ALL_FILES=$(find test evals -name '*.test.ts' \
ALL_FILES=$(find test -name '*.test.ts' \
-not -name '*.serial.test.ts' \
-not -name 'eval-longmemeval-e2e.slow.test.ts' \
-not -name 'entity-resolve-perf.slow.test.ts' \
@@ -102,12 +94,6 @@ if [ "$DRY_RUN_LIST" = "1" ]; then
exit 0
fi
# Snapshot fast-path (after the dry-run exit so list-only calls stay
# instant): ~370 PGLite-booting matrix files pay ~3.1s cold init each
# without it. The echo inside makes silent cold-init regressions visible
# in CI logs.
ensure_pglite_snapshot "test-shard"
ALL_COUNT=$(printf '%s\n' "$ALL_FILES" | grep -c '^' || true)
SHARD_COUNT=$(printf '%s\n' "$SHARD_FILES" | grep -c '^' || true)
# grep -c on empty input returns 0 even with trailing newline edge cases
@@ -122,7 +108,4 @@ fi
# Convert newline-separated file list to argv. xargs handles the
# whitespace correctly without word-splitting on spaces in paths.
# --max-concurrency mirrors the local runner: unbounded intra-process
# concurrency under parallel PGLite boots produced real shard deaths (the
# 22-minute matrix timeout in test.yml records 13 of them).
printf '%s\n' "$SHARD_FILES" | xargs bun test --timeout=60000 --max-concurrency="${GBRAIN_TEST_MAX_CONCURRENCY:-4}"
printf '%s\n' "$SHARD_FILES" | xargs bun test --timeout=60000
+717 -1201
View File
File diff suppressed because it is too large Load Diff
+3 -74
View File
@@ -40,7 +40,7 @@ import {
parseQrelsFile,
type QrelsFile,
} from '../core/bench/qrels-file.ts';
import { runCorrectnessGate, type CorrectnessGateOpts, type CorrectnessResult } from '../core/bench/correctness-gate.ts';
import { runCorrectnessGate, type CorrectnessResult } from '../core/bench/correctness-gate.ts';
import { replayCore, type ReplaySummary } from './eval-replay.ts';
interface GateOpts {
@@ -55,18 +55,6 @@ interface GateOpts {
thresholdRecallAtK?: number;
thresholdFirstRelevantHit?: number;
thresholdExpectedTop1?: number;
/**
* Hermetic embedder selector. The only accepted value is 'deterministic':
* query embeddings come from the qrels fixture's basis-vector dims
* (src/eval/deterministic-embed.ts) instead of the gateway, so the
* correctness gate runs with no API keys. Correctness-gate-only; rejected
* when combined with the baseline regression gate (replay re-embeds
* captured queries via the gateway). Cache safety: this path drives bare
* `hybridSearch`, which never reads or writes the semantic query cache
* (both live in `hybridSearchCached`), so deterministic runs cannot
* poison cached production results by construction.
*/
embedder?: string;
}
interface Breach {
@@ -151,10 +139,6 @@ function parseArgs(args: string[]): GateOpts {
opts.thresholdExpectedTop1 = Number(next);
i++;
break;
case '--embedder':
opts.embedder = next;
i++;
break;
default:
break;
}
@@ -185,13 +169,6 @@ Thresholds (override baseline metadata; CLI > embedded > defaults):
--threshold-expected-top1 FLOAT Correctness: expected_top1-hit-rate floor (default ${DEFAULT_QRELS_THRESHOLDS.expected_top1})
-k, --k N Top-K for recall@K (default ${DEFAULT_QRELS_THRESHOLDS.k})
Hermetic mode (correctness gate only):
--embedder deterministic Embed queries as the qrels fixture's basis
vectors instead of calling the gateway —
no API keys, fully reproducible (eval
canaries/CI). Rejected together with the
baseline regression gate.
Output:
--json Print JSON envelope to stdout
-h, --help Show this help
@@ -309,7 +286,6 @@ function runCorrectnessGateDispatch(
qrelsPath: string,
k: number,
cliOverrides: Pick<GateOpts, 'thresholdRecallAtK' | 'thresholdFirstRelevantHit' | 'thresholdExpectedTop1'>,
searchFn?: CorrectnessGateOpts['searchFn'],
): Promise<GateResult['correctness_gate']> {
return (async () => {
let qrelsFile: QrelsFile;
@@ -336,7 +312,7 @@ function runCorrectnessGateDispatch(
let result: CorrectnessResult;
try {
result = await runCorrectnessGate(engine, qrelsFile, { k, ...(searchFn ? { searchFn } : {}) });
result = await runCorrectnessGate(engine, qrelsFile, { k });
} catch (err) {
return {
ran: true,
@@ -472,29 +448,6 @@ export async function runEvalGate(engine: BrainEngine, args: string[]): Promise<
process.exit(2);
}
// Hermetic embedder validation. Only 'deterministic' is supported; the
// regression gate is out of scope (replay re-embeds captured queries via
// the gateway, which needs a provider key — defeating the hermetic point).
if (opts.embedder !== undefined) {
if (opts.embedder !== 'deterministic') {
console.error(
`Error: unsupported embedder "${opts.embedder}" — the only supported value is "deterministic".`,
);
process.exit(2);
}
if (opts.baseline) {
console.error(
'Error: the deterministic embedder cannot be combined with the baseline regression gate ' +
'(replay re-embeds captured queries via the gateway). Use it with the qrels correctness gate only.',
);
process.exit(2);
}
if (!opts.qrels) {
console.error('Error: the deterministic embedder requires a qrels file.');
process.exit(2);
}
}
const result: GateResult = {
schema_version: 1,
verdict: 'pass',
@@ -515,35 +468,11 @@ export async function runEvalGate(engine: BrainEngine, args: string[]): Promise<
if (opts.qrels) {
const k = opts.k ?? DEFAULT_QRELS_THRESHOLDS.k;
// Deterministic embedder: build a searchFn that threads basis-vector
// query embeddings (derived from the qrels fixture itself) into bare
// hybridSearch via the queryEmbedFn seam. The rest of the pipeline
// (keyword/title/alias arms, RRF, boosts) runs exactly as production.
let deterministicSearchFn: CorrectnessGateOpts['searchFn'] | undefined;
if (opts.embedder === 'deterministic') {
let queryEmbedFn: (text: string) => Float32Array;
try {
const { buildQrelsQueryEmbedFn } = await import('../eval/deterministic-embed.ts');
queryEmbedFn = buildQrelsQueryEmbedFn(readFileSync(opts.qrels, 'utf-8'));
} catch (err) {
console.error(
`Error: could not build the deterministic embedder from ${opts.qrels}: ${(err as Error).message}`,
);
process.exit(2);
}
const { hybridSearch } = await import('../core/search/hybrid.ts');
deterministicSearchFn = async (e, q, o) => {
const results = await hybridSearch(e, q, { limit: o.limit, queryEmbedFn });
return results.map(r => ({ source_id: r.source_id, slug: r.slug }));
};
}
result.correctness_gate = await runCorrectnessGateDispatch(engine, opts.qrels, k, {
thresholdRecallAtK: opts.thresholdRecallAtK,
thresholdFirstRelevantHit: opts.thresholdFirstRelevantHit,
thresholdExpectedTop1: opts.thresholdExpectedTop1,
}, deterministicSearchFn);
});
if (result.correctness_gate.breaches && result.correctness_gate.breaches.length > 0) {
result.verdict = 'fail';
}
+2 -2
View File
@@ -8,7 +8,7 @@
* connection string into the error message:
* - `connection to server at "db.example.supabase.com" (1.2.3.4), port 5432 failed: ...`
* - `FATAL: password authentication failed for user "postgres"`
* - `could not connect to server: postgresql://user:pass@host:5432/db` (allow-pg-url-literal)
* - `could not connect to server: postgresql://user:pass@host:5432/db`
*
* If an operator pastes a JSONL audit dump into a GitHub issue or Slack,
* those errors leak credentials. The project's audit-as-debug-tool
@@ -34,7 +34,7 @@ interface RedactPattern {
* occurrences in a single string get redacted.
*/
const PATTERNS: ReadonlyArray<RedactPattern> = [
// postgres:// and postgresql:// URLs. Includes user:pass@host:port/db /* allow-pg-url-literal */
// postgres:// and postgresql:// URLs. Includes user:pass@host:port/db
// shapes plus query-string variants. Terminator is whitespace or
// common JSON/markdown delimiters.
{ kind: 'pg_url', re: /postgres(?:ql)?:\/\/[^\s"'>)]+/gi },
+1 -1
View File
@@ -45,7 +45,7 @@ export const CLI_FLAG_REGISTRY: Record<string, readonly string[]> = {
'edges-backfill': ['--aliases', '--all', '--all-sources', '--brain', '--concurrency', '--federated', '--help', '--include-null-signature', '--json', '--max-age', '--max-chunks', '--max-cost-usd', '--no-extract', '--no-federated', '--older-than', '--path', '--pattern', '--pending', '--repo', '--reset', '--resolve', '--restore-only', '--source', '--stale', '--supersessions', '--thin', '--timeout', '--workers'],
'embed': ['--aliases', '--all', '--background', '--batch-size', '--brain', '--brain-wide-max-cost-usd', '--break-lock', '--catch-up', '--dry-run', '--embedding-dimensions', '--embedding-model', '--explain', '--fast', '--fix', '--follow', '--force', '--force-break-lock', '--from-pages', '--help', '--http', '--include-null-signature', '--json', '--lang', '--markdown', '--max-age', '--max-cost-usd', '--model', '--multimodal', '--name', '--near-symbol', '--no', '--no-embed', '--no-embedding', '--no-extract', '--pace', '--pace-max-concurrency', '--parallel', '--path', '--pattern', '--pending', '--pglite', '--prefix', '--priority', '--progress-interval', '--progress-json', '--quiet', '--remediate', '--reset', '--resolve', '--restore-only', '--serial', '--slugs', '--source', '--stale', '--supabase', '--supersessions', '--surface', '--symbol-kind', '--thin', '--timeout', '--to', '--token-ttl', '--version'],
'enrich': ['--aliases', '--all', '--all-sources', '--allow-empty', '--apply', '--asof', '--auto', '--background', '--bound-max-concurrent', '--bound-slug-prefixes', '--bound-source', '--bound-tools', '--brain', '--brain-wide-max-cost-usd', '--break-lock', '--budget-usd-per-day', '--by-mention', '--clone-dir', '--code', '--concurrency', '--confirm-destructive', '--content', '--date', '--days', '--detail', '--dry-run', '--embedding-dimensions', '--embedding-model', '--entities', '--explain', '--fast', '--federated', '--file', '--fix', '--follow', '--force', '--force-break-lock', '--from-pages', '--help', '--http', '--image', '--include-null-signature', '--json', '--judge-model', '--kind', '--lang', '--limit', '--markdown', '--max-age', '--max-cost', '--max-cost-usd', '--max-runtime', '--max-usd', '--min-context', '--mode', '--model', '--multimodal', '--near-symbol', '--no', '--no-embed', '--no-embedding', '--no-extract', '--offset', '--older-than', '--order', '--path', '--pattern', '--pending', '--progress-interval', '--progress-json', '--quiet', '--reenrich-after', '--remediate', '--reset', '--resolve', '--restore-only', '--resume', '--save', '--session', '--session-id', '--since', '--slug', '--slugs', '--source', '--source-id', '--stale', '--stats', '--supersessions', '--surface', '--symbol-kind', '--thin', '--thin-threshold', '--timeout', '--to', '--token-ttl', '--trusted-extraction', '--types', '--url', '--url-managed', '--version', '--with-db', '--workers', '--yes'],
'eval': ['--ab-relational', '--against', '--aliases', '--all', '--allow-regression', '--background', '--baseline', '--batch', '--brain', '--brain-wide-max-cost-usd', '--budget-usd', '--budget-usd-answer', '--budget-usd-retrieval', '--committed-baseline', '--compare', '--compare-limit', '--concurrent', '--config-a', '--config-b', '--corpus', '--cycles', '--days', '--dedup-cosine', '--dedup-max-per-page', '--dedup-type-ratio', '--dimensions', '--distance-min', '--embedder', '--embedding-dimensions', '--embedding-model', '--expand', '--explain', '--fast', '--fixtures', '--follow', '--force', '--from-capture', '--from-db', '--from-pages', '--gold', '--grounding-min', '--harness', '--help', '--http', '--include-holdout', '--include-null-signature', '--input', '--json', '--judge', '--justification', '--k', '--limit', '--llm', '--max-pair-chars', '--max-tokens', '--max-usd', '--md', '--metric', '--min-recall', '--mode', '--model', '--models', '--modes', '--multimodal', '--name', '--no', '--no-cache', '--no-embed', '--no-embedding', '--no-expand', '--no-extract', '--no-llm', '--older-than', '--out', '--output', '--output-dir', '--parallel', '--pattern', '--pending', '--progress-interval', '--progress-json', '--qrels', '--queries-file', '--query', '--questions', '--quiet', '--receipt-dir', '--refresh-cache', '--remediate', '--reset', '--resolve', '--rrf-k', '--rubric-version', '--runs', '--sampling', '--save', '--seed', '--severity', '--short', '--show-toplevel', '--since', '--skip-replay', '--slot-a-model', '--slot-b-model', '--slot-c-model', '--slug', '--slug-prefix', '--source', '--stale', '--strategy', '--strict', '--suite', '--suites', '--supersessions', '--surface', '--task', '--thin', '--threshold', '--threshold-expected-top1', '--threshold-first-relevant-hit', '--threshold-jaccard', '--threshold-latency-multiplier', '--threshold-recall-at-k', '--threshold-top1', '--timeout', '--to', '--token-ttl', '--tool', '--top-k', '--top-regressions', '--until', '--update-baseline', '--usefulness-min', '--verbose', '--version', '--with-code-intel', '--yes'],
'eval': ['--ab-relational', '--against', '--aliases', '--all', '--allow-regression', '--background', '--baseline', '--batch', '--brain', '--brain-wide-max-cost-usd', '--budget-usd', '--budget-usd-answer', '--budget-usd-retrieval', '--committed-baseline', '--compare', '--compare-limit', '--concurrent', '--config-a', '--config-b', '--corpus', '--cycles', '--days', '--dedup-cosine', '--dedup-max-per-page', '--dedup-type-ratio', '--dimensions', '--distance-min', '--embedding-dimensions', '--embedding-model', '--expand', '--explain', '--fast', '--fixtures', '--follow', '--force', '--from-capture', '--from-db', '--from-pages', '--gold', '--grounding-min', '--harness', '--help', '--http', '--include-holdout', '--include-null-signature', '--input', '--json', '--judge', '--justification', '--k', '--limit', '--llm', '--max-pair-chars', '--max-tokens', '--max-usd', '--md', '--metric', '--min-recall', '--mode', '--model', '--models', '--modes', '--multimodal', '--name', '--no', '--no-cache', '--no-embed', '--no-embedding', '--no-expand', '--no-extract', '--no-llm', '--older-than', '--out', '--output', '--output-dir', '--parallel', '--pattern', '--pending', '--progress-interval', '--progress-json', '--qrels', '--queries-file', '--query', '--questions', '--quiet', '--receipt-dir', '--refresh-cache', '--remediate', '--reset', '--resolve', '--rrf-k', '--rubric-version', '--runs', '--sampling', '--save', '--seed', '--severity', '--short', '--show-toplevel', '--since', '--skip-replay', '--slot-a-model', '--slot-b-model', '--slot-c-model', '--slug', '--slug-prefix', '--source', '--stale', '--strategy', '--strict', '--suite', '--suites', '--supersessions', '--surface', '--task', '--thin', '--threshold', '--threshold-expected-top1', '--threshold-first-relevant-hit', '--threshold-jaccard', '--threshold-latency-multiplier', '--threshold-recall-at-k', '--threshold-top1', '--timeout', '--to', '--token-ttl', '--tool', '--top-k', '--top-regressions', '--until', '--update-baseline', '--usefulness-min', '--verbose', '--version', '--with-code-intel', '--yes'],
'export': ['--aliases', '--all', '--background', '--brain', '--brain-wide-max-cost-usd', '--dir', '--explain', '--federated', '--fix', '--follow', '--help', '--include-null-signature', '--json', '--lang', '--markdown', '--multimodal', '--near-symbol', '--no-extract', '--no-federated', '--path', '--pattern', '--pending', '--progress-interval', '--progress-json', '--quiet', '--remediate', '--repo', '--reset', '--resolve', '--restore-only', '--slug-prefix', '--source', '--stale', '--supersessions', '--symbol-kind', '--thin', '--timeout', '--type'],
'extract': ['--aliases', '--all', '--background', '--brain', '--brain-wide-max-cost-usd', '--by-mention', '--catch-up', '--code', '--concurrency', '--dir', '--dry-run', '--explain', '--federated', '--follow', '--from-meetings', '--help', '--include-frontmatter', '--include-null-signature', '--infer-dates', '--json', '--kind', '--lang', '--markdown', '--max-age', '--max-cost-usd', '--multimodal', '--name-status', '--near-symbol', '--ner', '--no-extract', '--no-federated', '--older-than', '--pack', '--path', '--pattern', '--pending', '--progress-interval', '--progress-json', '--quiet', '--remediate', '--repo', '--reset', '--resolve', '--restore-only', '--run-id', '--since', '--slug', '--source', '--source-id', '--stale', '--strategy', '--supersessions', '--symbol-kind', '--thin', '--timeout', '--type', '--verbose', '--workers', '--yes'],
'extract-conversation-facts': ['--aliases', '--all', '--all-sources', '--background', '--brain', '--brain-wide-max-cost-usd', '--break-lock', '--by-mention', '--clone-dir', '--code', '--concurrency', '--confirm-destructive', '--dry-run', '--embedding-dimensions', '--embedding-model', '--explain', '--fix', '--follow', '--force', '--force-break-lock', '--help', '--include-null-signature', '--json', '--judge-model', '--lang', '--limit', '--markdown', '--max-age', '--max-cost', '--max-cost-usd', '--max-runtime', '--model', '--multimodal', '--near-symbol', '--no', '--no-embed', '--no-embedding', '--no-extract', '--older-than', '--override-disabled', '--path', '--pattern', '--pending', '--pglite', '--progress-interval', '--progress-json', '--quiet', '--remediate', '--reset', '--resolve', '--restore-only', '--segment-limit', '--session', '--since', '--sleep', '--slug', '--source', '--source-id', '--stale', '--supabase', '--supersessions', '--symbol-kind', '--thin', '--timeout', '--to', '--types', '--url', '--url-managed', '--version', '--workers', '--yes'],
+34 -91
View File
@@ -117,85 +117,35 @@ const PGLITE_EDGE_BATCH_MAX_BIND_PARAMS = 30_000;
// silently fall through to a normal initSchema (snapshot is just an
// optimization, never authoritative).
let _snapshotWarnLogged = false;
// Per-process memo. MIGRATIONS + PGLITE_SCHEMA_SQL are static for the life of
// the process, so the schema hash is too; the version file and the ~42MB tar
// are read once per (path, process) instead of once per engine construction
// (a full suite constructs 600+ engines — the un-memoized loader re-read the
// tar and re-hashed 131 migration handler sources every time, ~84MB of
// transient allocation per call). A null entry means the path is terminally
// unusable this process (missing/stale/torn) — no retry per construction.
// The dims/model shape gate is deliberately NOT memoized: tests reconfigure
// the gateway mid-process (zembed/1280) and a mismatched engine must fall
// back to cold init even when an earlier engine loaded this same snapshot.
// Accepted limitation: a snapshot file rewritten mid-process is not observed;
// the only writer (build-pglite-snapshot.ts) runs before test fan-out.
let _snapshotSchemaHashMemo: string | null = null;
// blob stays null until the FIRST caller whose shape gate passes — a process
// whose gateway shape never matches the snapshot (the zembed/1280 test
// files) never pays the 42MB tar read at all.
const _snapshotFileMemo = new Map<string, { versionLines: string[]; blob: Blob | null } | null>();
let _snapshotTarReads = 0;
export function __snapshotMemoStatsForTests(): { tarReads: number; memoEntries: number } {
return { tarReads: _snapshotTarReads, memoEntries: _snapshotFileMemo.size };
}
export function __resetSnapshotMemoForTests(): void {
_snapshotSchemaHashMemo = null;
_snapshotFileMemo.clear();
_snapshotTarReads = 0;
_snapshotWarnLogged = false;
}
export function tryLoadSnapshot(snapshotPath: string): Blob | null {
try {
let entry = _snapshotFileMemo.get(snapshotPath);
if (entry === null) return null; // terminally unusable this process
if (entry === undefined) {
// First touch of this path in this process — do the file work once.
// Lazy require so production builds without these imports don't crash.
// eslint-disable-next-line @typescript-eslint/no-require-imports
const fs = require('node:fs') as typeof import('node:fs'); // engine-dynamic-import-ok
const crypto = require('node:crypto') as typeof import('node:crypto'); // engine-dynamic-import-ok
const { MIGRATIONS } = require('./migrate.ts') as typeof import('./migrate.ts'); // engine-dynamic-import-ok
const { PGLITE_SCHEMA_SQL } = require('./pglite-schema.ts') as typeof import('./pglite-schema.ts'); // engine-dynamic-import-ok
// Lazy require so production builds without these imports don't crash.
// eslint-disable-next-line @typescript-eslint/no-require-imports
const fs = require('node:fs') as typeof import('node:fs'); // engine-dynamic-import-ok
const crypto = require('node:crypto') as typeof import('node:crypto'); // engine-dynamic-import-ok
const { MIGRATIONS } = require('./migrate.ts') as typeof import('./migrate.ts'); // engine-dynamic-import-ok
const { PGLITE_SCHEMA_SQL } = require('./pglite-schema.ts') as typeof import('./pglite-schema.ts'); // engine-dynamic-import-ok
if (!fs.existsSync(snapshotPath)) {
if (!_snapshotWarnLogged) {
// eslint-disable-next-line no-console
console.warn(`[pglite] GBRAIN_PGLITE_SNAPSHOT set but file missing: ${snapshotPath} — using normal init.`);
_snapshotWarnLogged = true;
}
_snapshotFileMemo.set(snapshotPath, null);
return null;
if (!fs.existsSync(snapshotPath)) {
if (!_snapshotWarnLogged) {
// eslint-disable-next-line no-console
console.warn(`[pglite] GBRAIN_PGLITE_SNAPSHOT set but file missing: ${snapshotPath} — using normal init.`);
_snapshotWarnLogged = true;
}
const versionPath = snapshotPath.replace(/\.tar(?:\.gz)?$/, '.version');
if (!fs.existsSync(versionPath)) {
if (!_snapshotWarnLogged) {
// eslint-disable-next-line no-console
console.warn(`[pglite] snapshot version file missing: ${versionPath} — using normal init.`);
_snapshotWarnLogged = true;
}
_snapshotFileMemo.set(snapshotPath, null);
return null;
}
if (_snapshotSchemaHashMemo === null) {
_snapshotSchemaHashMemo = computeSnapshotSchemaHash(MIGRATIONS, PGLITE_SCHEMA_SQL, crypto);
}
const versionLines = fs.readFileSync(versionPath, 'utf8').trim().split('\n');
if (_snapshotSchemaHashMemo !== (versionLines[0] ?? '')) {
if (!_snapshotWarnLogged) {
// eslint-disable-next-line no-console
console.warn(`[pglite] snapshot stale (schema hash mismatch) — using normal init. Rebuild with: bun run build:pglite-snapshot`);
_snapshotWarnLogged = true;
}
_snapshotFileMemo.set(snapshotPath, null);
return null;
}
entry = { versionLines, blob: null };
_snapshotFileMemo.set(snapshotPath, entry);
return null;
}
const versionPath = snapshotPath.replace(/\.tar(?:\.gz)?$/, '.version');
if (!fs.existsSync(versionPath)) {
if (!_snapshotWarnLogged) {
// eslint-disable-next-line no-console
console.warn(`[pglite] snapshot version file missing: ${versionPath} — using normal init.`);
_snapshotWarnLogged = true;
}
return null;
}
const expectedHash = computeSnapshotSchemaHash(MIGRATIONS, PGLITE_SCHEMA_SQL, crypto);
const versionLines = fs.readFileSync(versionPath, 'utf8').trim().split('\n');
const actualHash = versionLines[0] ?? '';
// W0 fix-wave: the version file's dims=/model= lines record the embedding
// shape the snapshot was BAKED with. A snapshot whose vector(dims) columns
@@ -204,8 +154,6 @@ export function tryLoadSnapshot(snapshotPath: string): Blob | null {
// fixture went default-on). Resolve our would-be shape through the same
// gateway-with-default fallback initSchema uses and refuse a mismatch.
// Version files without the shape lines (pre-W0) are treated as stale.
// Re-evaluated on EVERY call against the CURRENT gateway config — never
// memoized (see memo comment above).
let wantDims: number | string = DEFAULT_EMBEDDING_DIMENSIONS;
let wantModel: string = DEFAULT_EMBEDDING_MODEL;
try {
@@ -213,30 +161,25 @@ export function tryLoadSnapshot(snapshotPath: string): Blob | null {
wantDims = gw.getEmbeddingDimensions();
wantModel = gw.getEmbeddingModel();
} catch { /* gateway not configured — defaults, same as initSchema */ }
const shapeOk = entry.versionLines[1] === `dims=${wantDims}` && entry.versionLines[2] === `model=${wantModel}`;
const shapeOk = versionLines[1] === `dims=${wantDims}` && versionLines[2] === `model=${wantModel}`;
if (!shapeOk) {
if (!_snapshotWarnLogged) {
// eslint-disable-next-line no-console
console.warn(`[pglite] snapshot embedding shape mismatch (want dims=${wantDims} model=${wantModel}, have ${entry.versionLines[1] ?? 'none'} ${entry.versionLines[2] ?? ''}) — using normal init. Rebuild with: bun run build:pglite-snapshot`);
console.warn(`[pglite] snapshot embedding shape mismatch (want dims=${wantDims} model=${wantModel}, have ${versionLines[1] ?? 'none'} ${versionLines[2] ?? ''}) — using normal init. Rebuild with: bun run build:pglite-snapshot`);
_snapshotWarnLogged = true;
}
return null;
}
if (entry.blob === null) {
// Tar read deferred until the first shape-matching caller (see memo
// comment above). A torn/unreadable tar is terminal for the process.
try {
// eslint-disable-next-line @typescript-eslint/no-require-imports
const fs = require('node:fs') as typeof import('node:fs'); // engine-dynamic-import-ok
const buf = fs.readFileSync(snapshotPath);
_snapshotTarReads += 1;
entry.blob = new Blob([new Uint8Array(buf.buffer as ArrayBuffer, buf.byteOffset, buf.byteLength)]);
} catch {
_snapshotFileMemo.set(snapshotPath, null);
return null;
if (expectedHash !== actualHash) {
if (!_snapshotWarnLogged) {
// eslint-disable-next-line no-console
console.warn(`[pglite] snapshot stale (schema hash mismatch) — using normal init. Rebuild with: bun run build:pglite-snapshot`);
_snapshotWarnLogged = true;
}
return null;
}
return entry.blob;
const buf = fs.readFileSync(snapshotPath);
return new Blob([buf]);
} catch {
// Any failure -> fall through to normal init. Never block tests.
return null;
+3 -29
View File
@@ -812,21 +812,6 @@ export interface HybridSearchOpts extends SearchOpts {
*/
_queryEmbedDeadline?: QueryEmbedDeadline;
/**
* Hermetic eval canaries/CI non-semantic embeddings. When set, the query
* embedding for the TEXT vector arm comes from this function (e.g. qrels
* basis vectors) INSTEAD of the gateway's query-embed path, and the
* no-embedding-provider keyword-only short-circuit is bypassed so the
* vector arm runs with no provider key configured at all. Never set on
* production paths; when absent, behavior is byte-for-byte unchanged.
*
* Cache note: bare `hybridSearch` neither reads nor writes the semantic
* query cache by construction both the lookup and the store live only in
* `hybridSearchCached` so a deterministic-embedding eval run through this
* seam cannot poison `query_cache` for production queries.
*/
queryEmbedFn?: (text: string) => Float32Array | Promise<Float32Array>;
/**
* INTERNAL cache-consult outcome threaded from `hybridSearchCached` into
* the inner `hybridSearch` so the ONE telemetry record per search (emitted
@@ -1279,10 +1264,7 @@ export async function hybridSearch(
earlyModality === 'both' ||
mayEscalateToMultimodal) &&
isAvailable('embedding', multimodalProviderProbe);
// Hermetic eval canaries/CI: a caller-supplied queryEmbedFn produces the
// vector-arm query embedding without the gateway, so provider
// availability is irrelevant — skip the keyword-only short-circuit.
if (!opts?.queryEmbedFn && !isAvailable('embedding', providerProbe) && !willTryMultimodal) {
if (!isAvailable('embedding', providerProbe) && !willTryMultimodal) {
// v0.43 — fuse the relational arm with keyword so typed-edge answers
// survive on the no-embedding-provider path (the relational win is most
// valuable exactly when vector is unavailable). The title arm fuses here
@@ -1520,20 +1502,12 @@ export async function hybridSearch(
// share one ~6s budget); direct callers get a fresh deadline. On timeout
// the embed rejects → salvage below (or keyword-only when all reject).
const embedDl = opts?._queryEmbedDeadline ?? makeQueryEmbedDeadline();
// Hermetic eval canaries/CI: queryEmbedFn (non-semantic deterministic
// embeddings) replaces the gateway query-embed for the text vector arm.
// No deadline needed — it's a synchronous-ish local computation with no
// network. Absent queryEmbedFn, the bounded gateway path is unchanged.
const embedOneQuery = (q: string): Promise<Float32Array> =>
opts?.queryEmbedFn
? Promise.resolve(opts.queryEmbedFn(q))
: embedQueryBounded(q, embedOpts, embedDl);
if (!searchSalvageEnabled()) {
// ENG-7 kill switch (GBRAIN_SEARCH_SALVAGE=off): pre-wave
// all-or-nothing fan-outs — one variant's failure abandons every
// embedding and falls back to keyword-only.
try {
const embeddings = await Promise.all(queries.map(q => embedOneQuery(q)));
const embeddings = await Promise.all(queries.map(q => embedQueryBounded(q, embedOpts, embedDl)));
queryEmbedding = embeddings[0];
const textLists = await Promise.all(
embeddings.map(emb => engine.searchVector(emb, searchOpts)),
@@ -1563,7 +1537,7 @@ export async function hybridSearch(
// WP2/T3 (ENG-15) salvage fan-outs: allSettled on BOTH the embed
// fan-out and the searchVector fan-out so one variant's failure no
// longer abandons the survivors (the query-vs-search asymmetry fix).
const settled = await Promise.allSettled(queries.map(q => embedOneQuery(q)));
const settled = await Promise.allSettled(queries.map(q => embedQueryBounded(q, embedOpts, embedDl)));
const okEmbeds: Float32Array[] = [];
const embedFailures: unknown[] = [];
for (const s of settled) {
-83
View File
@@ -1,83 +0,0 @@
/**
* Deterministic basis-vector embeddings for hermetic eval canaries/CI.
*
* NON-SEMANTIC embeddings: each query embeds as a unit basis vector at a
* fixed dimension, so retrieval through the full hybrid pipeline (vector +
* keyword/title/alias arms + RRF) is exactly reproducible with no API keys,
* no network, and no provider drift. Used by the qrels correctness gate's
* deterministic embedder path and the retrieval canary runner
* (scripts/run-eval-canary.ts). Mirrors the basis-vector convention in
* test/eval-replay-gate.test.ts and test/fixtures/eval-baselines/
* qrels-search.json (each fixture query carries an `embedding_dim`).
*/
/** Unit basis vector with 1.0 at `idx % dim` and 0.0 elsewhere. */
export function basisEmbedding(idx: number, dim = 1536): Float32Array {
const emb = new Float32Array(dim);
emb[idx % dim] = 1.0;
return emb;
}
/**
* 32-bit FNV-1a hash. Deterministic, dependency-free; used only to derive a
* stable fallback basis dimension for query texts not present in the qrels
* fixture.
*/
export function fnv1a(text: string): number {
let h = 0x811c9dc5;
for (let i = 0; i < text.length; i++) {
h ^= text.charCodeAt(i);
h = Math.imul(h, 0x01000193);
}
return h | 0;
}
export interface LegacyQrelsQuery {
query_id: string;
query: string;
embedding_dim: number;
relevant_slugs: string[];
first_relevant_slug: string;
}
/**
* Parse the legacy qrels fixture shape
* ({queries: [{query, embedding_dim, relevant_slugs, first_relevant_slug}]}).
* THE single parser for this shape the canary runner and the embedder
* builder both consume it. Throws on malformed JSON or a missing `queries`
* array (callers surface a usage error; the gate's own qrels parser reports
* shape problems in detail).
*/
export function parseLegacyQrels(raw: string): LegacyQrelsQuery[] {
const parsed = JSON.parse(raw) as { queries?: unknown };
if (!Array.isArray(parsed.queries)) {
throw new Error('qrels fixture missing "queries" array');
}
return parsed.queries as LegacyQrelsQuery[];
}
/**
* Build a query-embed function from a raw qrels fixture (the legacy shape:
* `{queries: [{query, embedding_dim, ...}]}`). Known query texts map to
* `basisEmbedding(embedding_dim)`; unknown texts fall back to a
* deterministic FNV-1a-derived basis dimension in [100, 1099] outside the
* fixture's low dims, so an unknown query can never accidentally vote for a
* fixture page's basis direction (fixture dims are small integers).
*
* Throws on malformed JSON or a missing `queries` array (caller surfaces a
* usage error; the gate's own qrels parser reports shape problems in detail).
*/
export function buildQrelsQueryEmbedFn(qrelsRaw: string): (text: string) => Float32Array {
const queries = parseLegacyQrels(qrelsRaw);
const dimByQuery = new Map<string, number>();
for (const q of queries) {
if (typeof (q as { query?: unknown })?.query === 'string' && typeof (q as { embedding_dim?: unknown })?.embedding_dim === 'number') {
dimByQuery.set(q.query, q.embedding_dim);
}
}
return (text: string): Float32Array => {
const dim = dimByQuery.get(text);
if (dim !== undefined) return basisEmbedding(dim);
return basisEmbedding(100 + (Math.abs(fnv1a(text)) % 1000));
};
}
+1 -1
View File
@@ -1,6 +1,6 @@
# gbrain agent workspace — template
<!-- gbrain-template-stamp: 0.46.5.0 -->
<!-- gbrain-template-stamp: 0.46.4.0 -->
This repository is the **"Use this template"** distribution artifact for a
[gbrain](https://github.com/garrytan/gbrain) personal-agent workspace — the same
+27 -21
View File
@@ -20,24 +20,6 @@
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { MIGRATIONS } from '../src/core/migrate.ts';
import { resetPgliteState } from './helpers/reset-pglite.ts';
// ONE engine for the whole file (was three describe-scoped engines = three
// full PGLite boots for 11 tests). Each data-bearing describe resets state
// in its own beforeAll and re-seeds — required, not just hygiene: the
// 'refactor' corpus of the searchKeyword describe would otherwise pollute
// the searchKeywordChunks describe's expectations.
let engine: PGLiteEngine;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
});
afterAll(async () => {
await engine.disconnect();
}, 30_000);
describe('Cathedral II v28 migration — search_vector backfill', () => {
test('v28 migration exists in registry', () => {
@@ -62,8 +44,12 @@ describe('Cathedral II v28 migration — search_vector backfill', () => {
});
describe('Cathedral II Layer 3 — searchKeyword external contract', () => {
let engine: PGLiteEngine;
beforeAll(async () => {
await resetPgliteState(engine);
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
// Two pages, each with multiple chunks that match "refactor" so we can
// verify the dedup pass returns one chunk per page. upsertChunks fires
@@ -100,6 +86,10 @@ describe('Cathedral II Layer 3 — searchKeyword external contract', () => {
]);
});
afterAll(async () => {
await engine.disconnect();
}, 30_000);
test('returns one row per matched page (dedup to best chunk per page)', async () => {
const results = await engine.searchKeyword('refactor');
const slugs = results.map(r => r.slug).sort();
@@ -131,8 +121,12 @@ describe('Cathedral II Layer 3 — searchKeyword external contract', () => {
});
describe('Cathedral II Layer 3 — searchKeywordChunks (internal primitive)', () => {
let engine: PGLiteEngine;
beforeAll(async () => {
await resetPgliteState(engine);
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
// Page with multiple matching chunks so chunk-grain results can
// return two chunks from the same page (no dedup).
@@ -149,6 +143,10 @@ describe('Cathedral II Layer 3 — searchKeywordChunks (internal primitive)', ()
]);
});
afterAll(async () => {
await engine.disconnect();
}, 30_000);
test('does not dedup: can return multiple chunks from the same page', async () => {
const results = await engine.searchKeywordChunks('refactor', { limit: 20 });
const slugs = results.map(r => r.slug);
@@ -176,8 +174,12 @@ describe('Cathedral II Layer 3 — searchKeywordChunks (internal primitive)', ()
});
describe('Cathedral II Layer 3 — doc-comment weight precedence (A4 foundation)', () => {
let engine: PGLiteEngine;
beforeAll(async () => {
await resetPgliteState(engine);
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
// Two pages, each with one chunk. Alpha's chunk has the target term
// 'hexagon' in its doc_comment (weight A); Beta's chunk has it in
@@ -215,6 +217,10 @@ describe('Cathedral II Layer 3 — doc-comment weight precedence (A4 foundation)
);
});
afterAll(async () => {
await engine.disconnect();
}, 30_000);
test('doc-comment match outranks body-text match on the same term', async () => {
const results = await engine.searchKeyword('hexagon');
expect(results.length).toBeGreaterThan(0);
-186
View File
@@ -1,186 +0,0 @@
/**
* test/eval-canary.test.ts hermetic retrieval canary (deterministic
* embedder + runner script).
*
* Pins:
* 1. basisEmbedding determinism + dims (src/eval/deterministic-embed.ts).
* 2. buildQrelsQueryEmbedFn maps fixture queries to their basis dims and
* falls back deterministically (FNV-1a-derived dim) for unknown texts.
* 3. scripts/run-eval-canary.ts end-to-end in check mode: real CLI
* subprocess, exit 0, metrics at/above the qrels default floors.
* 4. Determinism: two in-process runs of the correctness gate through the
* queryEmbedFn seam produce identical metrics.
* 5. Check mode writes nothing to tracked files (git status unchanged).
*
* Fully hermetic: no API keys, no network, no DATABASE_URL.
*/
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
import { readFileSync } from 'node:fs';
import { join, resolve } from 'node:path';
import { execSync, spawnSync } from 'node:child_process';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { resetPgliteState } from './helpers/reset-pglite.ts';
import { basisEmbedding, buildQrelsQueryEmbedFn, fnv1a } from '../src/eval/deterministic-embed.ts';
import { parseLegacyQrels, seedCanaryCorpus } from '../scripts/run-eval-canary.ts';
import { runCorrectnessGate, type CorrectnessGateOpts } from '../src/core/bench/correctness-gate.ts';
import { parseQrelsFile, DEFAULT_QRELS_THRESHOLDS } from '../src/core/bench/qrels-file.ts';
import { hybridSearch } from '../src/core/search/hybrid.ts';
const ROOT = resolve(import.meta.dir, '..');
const QRELS_PATH = join(ROOT, 'test', 'fixtures', 'eval-baselines', 'qrels-search.json');
// ---------------------------------------------------------------------------
// Canonical PGLite block (CLAUDE.md R3+R4)
// ---------------------------------------------------------------------------
let engine: PGLiteEngine;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
});
afterAll(async () => {
await engine.disconnect();
});
beforeEach(async () => {
await resetPgliteState(engine);
});
// ---------------------------------------------------------------------------
// 1. basisEmbedding
// ---------------------------------------------------------------------------
describe('basisEmbedding', () => {
test('default dim 1536, 1.0 at idx, 0.0 elsewhere', () => {
const e = basisEmbedding(5);
expect(e.length).toBe(1536);
expect(e[5]).toBe(1.0);
expect(e[0]).toBe(0.0);
expect(e.reduce((s, v) => s + v, 0)).toBe(1.0);
});
test('custom dim + idx wraparound', () => {
const e = basisEmbedding(103, 100);
expect(e.length).toBe(100);
expect(e[3]).toBe(1.0); // 103 % 100
expect(e.reduce((s, v) => s + v, 0)).toBe(1.0);
});
test('deterministic across calls', () => {
expect([...basisEmbedding(42)]).toEqual([...basisEmbedding(42)]);
});
});
// ---------------------------------------------------------------------------
// 2. buildQrelsQueryEmbedFn
// ---------------------------------------------------------------------------
describe('buildQrelsQueryEmbedFn', () => {
const raw = readFileSync(QRELS_PATH, 'utf-8');
test('maps every fixture query to its embedding_dim basis vector', () => {
const fn = buildQrelsQueryEmbedFn(raw);
const fixture = parseLegacyQrels(raw);
expect(fixture.length).toBeGreaterThanOrEqual(10);
for (const q of fixture) {
expect([...fn(q.query)]).toEqual([...basisEmbedding(q.embedding_dim)]);
}
});
test('unknown text falls back to a deterministic FNV-1a-derived dim in [100, 1099]', () => {
const fn = buildQrelsQueryEmbedFn(raw);
const unknown = 'a query text that is definitely not in the fixture';
const a = fn(unknown);
const b = fn(unknown);
expect([...a]).toEqual([...b]);
const idx = a.findIndex(v => v === 1.0);
expect(idx).toBe(100 + (Math.abs(fnv1a(unknown)) % 1000));
expect(idx).toBeGreaterThanOrEqual(100);
expect(idx).toBeLessThanOrEqual(1099);
});
test('throws on malformed input', () => {
expect(() => buildQrelsQueryEmbedFn('not json')).toThrow();
expect(() => buildQrelsQueryEmbedFn('{"no_queries": true}')).toThrow(/queries/);
});
});
// ---------------------------------------------------------------------------
// 4. In-process determinism through the queryEmbedFn seam
// ---------------------------------------------------------------------------
describe('correctness gate through the queryEmbedFn seam', () => {
test('two in-process runs produce identical metrics, at/above the default floors', async () => {
const raw = readFileSync(QRELS_PATH, 'utf-8');
await seedCanaryCorpus(engine, parseLegacyQrels(raw));
const queryEmbedFn = buildQrelsQueryEmbedFn(raw);
const qrels = parseQrelsFile(raw);
const searchFn: NonNullable<CorrectnessGateOpts['searchFn']> = async (e, q, o) => {
const results = await hybridSearch(e, q, { limit: o.limit, queryEmbedFn });
return results.map(r => ({ source_id: r.source_id, slug: r.slug }));
};
const a = await runCorrectnessGate(engine, qrels, { searchFn });
const b = await runCorrectnessGate(engine, qrels, { searchFn });
expect(a.summary).toEqual(b.summary);
expect(a.per_query).toEqual(b.per_query);
expect(a.summary.queries_errored).toBe(0);
expect(a.summary.mean_recall_at_k).toBeGreaterThanOrEqual(DEFAULT_QRELS_THRESHOLDS.recall_at_k);
expect(a.summary.first_relevant_hit_rate).toBeGreaterThanOrEqual(DEFAULT_QRELS_THRESHOLDS.first_relevant_hit);
expect(a.summary.expected_top1_denominator).toBeGreaterThan(0);
expect(a.summary.expected_top1_hit_rate).toBeGreaterThanOrEqual(DEFAULT_QRELS_THRESHOLDS.expected_top1);
}, 60_000);
});
// ---------------------------------------------------------------------------
// 3 + 5. Runner end-to-end (check mode) + tracked-file invariance
// ---------------------------------------------------------------------------
describe('run-eval-canary.ts (check mode)', () => {
// Captured by the end-to-end test; asserted separately below so a
// tracked-file write shows up as its own named failure.
let statusBefore: string | null = null;
let statusAfter: string | null = null;
let runExit: number | null = null;
test('spawns the real CLI gate hermetically and passes the floors', () => {
// Status scoped to the paths check mode could plausibly touch — a
// whole-tree porcelain diff flakes when a concurrent shard sibling (or a
// developer save) creates an unrelated file during the ~30s window.
const STATUS_SCOPE = 'git status --porcelain -- .gbrain-evals test/fixtures docs/eval';
statusBefore = execSync(STATUS_SCOPE, { cwd: ROOT, encoding: 'utf-8' });
const child = spawnSync(
process.execPath,
[join(ROOT, 'scripts', 'run-eval-canary.ts')],
// Outer budget strictly above the runner's inner CLI-child timeout so
// the runner's own diagnostics always win the race.
{ cwd: ROOT, encoding: 'utf-8', timeout: 118_000 },
);
statusAfter = execSync(STATUS_SCOPE, { cwd: ROOT, encoding: 'utf-8' });
runExit = child.status;
const combined = (child.stdout ?? '') + (child.stderr ?? '');
expect(child.status).toBe(0);
const m = combined.match(
/mean_recall_at_k=([\d.]+) first_relevant_hit_rate=([\d.]+) expected_top1_hit_rate=([\d.]+)/,
);
expect(m).not.toBeNull();
expect(Number(m![1])).toBeGreaterThanOrEqual(DEFAULT_QRELS_THRESHOLDS.recall_at_k);
expect(Number(m![2])).toBeGreaterThanOrEqual(DEFAULT_QRELS_THRESHOLDS.first_relevant_hit);
expect(Number(m![3])).toBeGreaterThanOrEqual(DEFAULT_QRELS_THRESHOLDS.expected_top1);
}, 120_000);
test('check mode writes nothing to tracked files (git status unchanged)', () => {
// Guard: the end-to-end test above must have actually run + passed spawn.
expect(runExit).toBe(0);
expect(statusBefore).not.toBeNull();
expect(statusAfter).toBe(statusBefore!);
});
});
-46
View File
@@ -141,52 +141,6 @@ describe('eval gate: usage errors', () => {
});
});
describe('eval gate: embedder flag validation', () => {
// The hermetic-canary embedder option accepts exactly one value and only
// composes with the correctness (qrels) gate. A regression that silently
// accepts a bad value would fall through to the keyed gateway path and
// defeat the hermetic guarantee.
const REAL_QRELS = 'test/fixtures/eval-baselines/qrels-search.json';
test('unsupported embedder value → exit 2', async () => {
const out = await withExitCapture(() =>
runEvalGate(engine, ['--embedder', 'semantic', '--qrels', REAL_QRELS]),
);
expect(out.exitCode).toBe(2);
});
test('deterministic embedder combined with the baseline gate → exit 2', async () => {
const out = await withExitCapture(() =>
runEvalGate(engine, [
'--embedder', 'deterministic',
'--baseline', '/tmp/does-not-exist-12345.ndjson',
'--qrels', REAL_QRELS,
]),
);
expect(out.exitCode).toBe(2);
});
test('deterministic embedder without a qrels file → exit 2', async () => {
const out = await withExitCapture(() =>
runEvalGate(engine, ['--embedder', 'deterministic']),
);
expect(out.exitCode).toBe(2);
});
test('deterministic embedder with a malformed qrels file → exit 2', async () => {
const { mkdtempSync, writeFileSync } = await import('node:fs');
const { tmpdir } = await import('node:os');
const { join } = await import('node:path');
const dir = mkdtempSync(join(tmpdir(), 'gate-embedder-'));
const bad = join(dir, 'malformed.json');
writeFileSync(bad, '{"not_queries": []}');
const out = await withExitCapture(() =>
runEvalGate(engine, ['--embedder', 'deterministic', '--qrels', bad]),
);
expect(out.exitCode).toBe(2);
});
});
describe('eval gate: regression-only path', () => {
test('malformed baseline → surfaces as breach (verdict fail, exit 1)', async () => {
const dir = mkdtempSync(join(tmpdir(), 'eval-gate-test-'));
+6 -4
View File
@@ -26,7 +26,6 @@ import { join } from 'path';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { resetPgliteState } from './helpers/reset-pglite.ts';
import { withEnv } from './helpers/with-env.ts';
import { basisEmbedding } from '../src/eval/deterministic-embed.ts';
import type { ChunkInput } from '../src/core/types.ts';
// ---------------------------------------------------------------------------
@@ -73,9 +72,12 @@ function loadFixture(): QrelFixture {
return fix;
}
// basisEmbedding (1.0 at `idx`, 0.0 elsewhere) is imported from
// src/eval/deterministic-embed.ts — the shared home for hermetic
// basis-vector embeddings (also used by the retrieval canary).
/** Basis vector with 1.0 at `idx` and 0.0 elsewhere. Mirrors search-quality.test.ts. */
function basisEmbedding(idx: number, dim = 1536): Float32Array {
const emb = new Float32Array(dim);
emb[idx % dim] = 1.0;
return emb;
}
/**
* Seed each relevant slug with a chunk whose embedding aligns with the
+11 -19
View File
@@ -569,7 +569,7 @@ describe('migration v35 — auto_rls_event_trigger structural guards', () => {
// 1. Structural — assert the migration SQL literally contains the helper
// CREATE INDEX + DROP INDEX (deterministic, fast, catches the regression
// even at 0-row scale where wall-clock can't distinguish O(n²) from O(1)).
// 2. Behavioral — populate 200 duplicates and assert the migration completes
// 2. Behavioral — populate 1000 duplicates and assert the migration completes
// under the wall-clock cap. Sanity check at small scale; the structural
// assertion is the real guard.
@@ -962,7 +962,7 @@ describe('migrate runner v67 — typed-claim columns materialized on PGLite', ()
});
});
describe('migrate: v8 (links_dedup) regression — must be fast on 200 duplicate rows', () => {
describe('migrate: v8 (links_dedup) regression — must be fast on 1K duplicate rows', () => {
let engine: PGLiteEngine;
beforeAll(async () => {
@@ -975,7 +975,7 @@ describe('migrate: v8 (links_dedup) regression — must be fast on 200 duplicate
await engine.disconnect();
});
test('200 duplicate links dedup completes in <90s and leaves table deduped', async () => {
test('1000 duplicate links dedup completes in <90s and leaves table deduped', async () => {
// Set up: drop BOTH the old (v8) and new (v11) unique constraints so
// duplicates can be inserted, then reset version so v8 + v11 re-run.
// v11 replaces the v8 constraint name; we drop whichever is present.
@@ -989,19 +989,15 @@ describe('migrate: v8 (links_dedup) regression — must be fast on 200 duplicate
const fromId = (await db.query(`SELECT id FROM pages WHERE slug = 'p/from'`)).rows[0].id;
const toId = (await db.query(`SELECT id FROM pages WHERE slug = 'p/to'`)).rows[0].id;
// Insert 200 duplicates of the same (from, to, type) row
// 200 rows, not 1000: the O(n²) shape this gate guards is still
// unmistakable at 200 (minutes vs sub-second dedup) and the insert loop
// stops burning ~15-25s of suite budget per test on row traffic that
// adds no discriminating power.
for (let i = 0; i < 200; i++) {
// Insert 1000 duplicates of the same (from, to, type) row
for (let i = 0; i < 1000; i++) {
await db.query(
`INSERT INTO links (from_page_id, to_page_id, link_type, context) VALUES ($1, $2, $3, $4)`,
[fromId, toId, 'mention', `dup-${i}`]
);
}
const beforeCount = (await db.query(`SELECT COUNT(*)::int AS c FROM links`)).rows[0].c;
expect(beforeCount).toBe(200);
expect(beforeCount).toBe(1000);
// Reset version to 7 so v8 + v9 + v10 + v11 re-run
await engine.setConfig('version', '7');
@@ -1043,7 +1039,7 @@ describe('migrate: v8 (links_dedup) regression — must be fast on 200 duplicate
});
});
describe('migrate: v9 (timeline_dedup_index) regression — must be fast on 200 duplicate rows', () => {
describe('migrate: v9 (timeline_dedup_index) regression — must be fast on 1K duplicate rows', () => {
let engine: PGLiteEngine;
beforeAll(async () => {
@@ -1056,26 +1052,22 @@ describe('migrate: v9 (timeline_dedup_index) regression — must be fast on 200
await engine.disconnect();
});
test('200 duplicate timeline entries dedup completes in <90s and leaves table deduped', async () => {
test('1000 duplicate timeline entries dedup completes in <90s and leaves table deduped', async () => {
const db = (engine as any).db;
await db.exec(`DROP INDEX IF EXISTS idx_timeline_dedup`);
await engine.putPage('p/timeline', { type: 'concept', title: 'TL', compiled_truth: '', timeline: '' });
const pageId = (await db.query(`SELECT id FROM pages WHERE slug = 'p/timeline'`)).rows[0].id;
// Insert 200 duplicates of the same (page_id, date, summary) row
// 200 rows, not 1000: the O(n²) shape this gate guards is still
// unmistakable at 200 (minutes vs sub-second dedup) and the insert loop
// stops burning ~15-25s of suite budget per test on row traffic that
// adds no discriminating power.
for (let i = 0; i < 200; i++) {
// Insert 1000 duplicates of the same (page_id, date, summary) row
for (let i = 0; i < 1000; i++) {
await db.query(
`INSERT INTO timeline_entries (page_id, date, source, summary, detail) VALUES ($1, $2::date, $3, $4, $5)`,
[pageId, '2024-01-15', `src-${i}`, 'Founded NovaMind', `detail-${i}`]
);
}
const beforeCount = (await db.query(`SELECT COUNT(*)::int AS c FROM timeline_entries`)).rows[0].c;
expect(beforeCount).toBe(200);
expect(beforeCount).toBe(1000);
await engine.setConfig('version', '7');
-58
View File
@@ -1,58 +0,0 @@
/**
* Guard for the evals/ CI-matrix collection (scripts/test-shard.sh).
*
* evals/**\/*.test.ts files run in the keyless 10-shard matrix. This repo's
* eval HARNESSES are key-requiring by default (Anthropic/OpenAI), so a new
* test file dropped under evals/ could silently start spending tokens in CI
* or fail keyless. Growth is therefore allowlist-gated: every collected
* evals file must be named here, and adding one asserts you checked it runs
* with NO API keys and NO network (mirror of the serial runner's
* EXCLUSIVE_FILES guard).
*/
import { describe, expect, it } from "bun:test";
import { execFileSync } from "child_process";
import { resolve } from "path";
const REPO_ROOT = resolve(import.meta.dir, "..", "..");
// Keyless-verified evals test files. Verify before adding:
// env -u ANTHROPIC_API_KEY -u OPENAI_API_KEY bun test <file>
const KEYLESS_ALLOWLIST = new Set([
// pure-function scoring/CI-parsing helpers; imports no gateway (verified)
"evals/functional-area-resolver/harness-runner.test.ts",
]);
describe("evals/ collection into the CI matrix", () => {
const collected = (): string[] => {
const lists: string[] = [];
// Union across all shards = the full collected set.
for (let i = 1; i <= 10; i++) {
const out = execFileSync(
"bash",
["scripts/test-shard.sh", "--dry-run-list", String(i), "10"],
{ cwd: REPO_ROOT, encoding: "utf-8" },
);
lists.push(out);
}
return lists
.join("\n")
.split("\n")
.map((s) => s.trim())
.filter((f) => f.startsWith("evals/"));
};
it("every collected evals file is on the keyless allowlist", () => {
const files = collected();
expect(files.length).toBeGreaterThan(0); // the collection itself works
const unlisted = files.filter((f) => !KEYLESS_ALLOWLIST.has(f));
expect(unlisted).toEqual([]);
});
it("every allowlisted file is actually collected (no dead entries)", () => {
const files = new Set(collected());
for (const f of KEYLESS_ALLOWLIST) {
expect(files.has(f)).toBe(true);
}
});
});
-175
View File
@@ -1,175 +0,0 @@
/**
* Behavioral tests for scripts/run-serial-tests.sh's POOLED execution:
*
* 1. All-pass: pooled files run concurrently, one-line PASS summaries,
* exit 0.
* 2. One failing file: exit 1, full log echoed, failed-files summary.
* 3. Hung file: killed by the per-file wall-clock timeout (exit 124/137
* surfaced with a timeout note) the exit-hang class containment.
* Skipped when no timeout/gtimeout binary exists on the host.
*
* The missing-sentinel(=failure) and EXCLUSIVE_FILES growth guards are
* source-pinned in test/scripts/serial-files.test.ts; these tests exercise
* the live pool in a minimal-PATH sandbox (same pattern as
* run-unit-parallel.test.ts).
*/
import { describe, it, expect, beforeAll, afterAll } from 'bun:test';
import { execFileSync } from 'child_process';
import { mkdtempSync, mkdirSync, writeFileSync, rmSync, copyFileSync, chmodSync, symlinkSync } from 'fs';
import { tmpdir } from 'os';
import { dirname, join, resolve } from 'path';
const REPO_ROOT = resolve(import.meta.dir, '..', '..');
let ROOT: string;
let ENV: Record<string, string>;
let hasTimeoutBin = false;
function stageSandbox(): string {
const root = mkdtempSync(join(tmpdir(), 'gbrain-serial-pool-'));
mkdirSync(join(root, 'scripts', 'lib'), { recursive: true });
mkdirSync(join(root, 'test'), { recursive: true });
for (const s of ['run-serial-tests.sh', 'lib/test-env.sh']) {
mkdirSync(dirname(join(root, 'scripts', s)), { recursive: true });
copyFileSync(resolve(REPO_ROOT, 'scripts', s), join(root, 'scripts', s));
}
chmodSync(join(root, 'scripts', 'run-serial-tests.sh'), 0o755);
const bin = join(root, 'bin');
mkdirSync(bin);
const tools = [
'bash', 'sh', 'env', 'dirname', 'basename', 'mktemp', 'date', 'sleep',
'cat', 'tail', 'head', 'rm', 'mkdir', 'grep', 'sed', 'awk', 'wc', 'tr',
'find', 'sort', 'bun', 'timeout', 'gtimeout',
];
for (const tool of tools) {
const p = Bun.which(tool);
if (p) {
symlinkSync(p, join(bin, tool));
if (tool === 'timeout' || tool === 'gtimeout') hasTimeoutBin = true;
}
}
return root;
}
function runScript(extraEnv: Record<string, string> = {}): { code: number; out: string } {
try {
const out = execFileSync('bash', [join(ROOT, 'scripts', 'run-serial-tests.sh')], {
cwd: ROOT,
encoding: 'utf-8',
env: { ...ENV, ...extraEnv },
stdio: ['ignore', 'pipe', 'pipe'],
});
return { code: 0, out };
} catch (err) {
const e = err as { status?: number; stdout?: string; stderr?: string };
return { code: e.status ?? -1, out: `${e.stdout ?? ''}${e.stderr ?? ''}` };
}
}
const PASSING = `import { describe, it, expect } from 'bun:test';
describe('passing', () => { it('works', () => { expect(1 + 1).toBe(2); }); });`;
const FAILING = `import { describe, it, expect } from 'bun:test';
describe('failing', () => { it('POOL_SENTINEL_ASSERTION breaks', () => { expect(1).toBe(2); }); });`;
const HANGING = `import { it } from 'bun:test';
it('hangs forever', async () => { await new Promise(() => {}); });`;
beforeAll(() => {
ROOT = stageSandbox();
ENV = {
PATH: join(ROOT, 'bin'),
HOME: process.env.HOME ?? ROOT,
TMPDIR: process.env.TMPDIR ?? '/tmp',
// Sandbox has no package.json — skip the snapshot build path entirely.
GBRAIN_NO_SNAPSHOT: '1',
GBRAIN_SERIAL_POOL: '2',
};
});
afterAll(() => {
rmSync(ROOT, { recursive: true, force: true });
});
describe('pooled serial runner', () => {
it('runs pooled files and passes with one-line summaries', () => {
writeFileSync(join(ROOT, 'test', 'a-ok.serial.test.ts'), PASSING);
writeFileSync(join(ROOT, 'test', 'b-ok.serial.test.ts'), PASSING);
const r = runScript();
expect(r.code).toBe(0);
expect(r.out).toContain('PASS');
expect(r.out).toContain('test/a-ok.serial.test.ts');
expect(r.out).toContain('test/b-ok.serial.test.ts');
expect(r.out).toContain('all 2 file(s) passed');
expect(r.out).toContain('pool=2');
rmSync(join(ROOT, 'test', 'a-ok.serial.test.ts'));
rmSync(join(ROOT, 'test', 'b-ok.serial.test.ts'));
});
it('a failing file fails the run with its full log and a failed-files summary', () => {
writeFileSync(join(ROOT, 'test', 'a-ok.serial.test.ts'), PASSING);
writeFileSync(join(ROOT, 'test', 'z-bad.serial.test.ts'), FAILING);
const r = runScript();
expect(r.code).toBe(1);
// Full bun log of the failing file is echoed (its assertion name shows).
expect(r.out).toContain('POOL_SENTINEL_ASSERTION');
expect(r.out).toContain('1 file(s) failed');
expect(r.out).toContain('test/z-bad.serial.test.ts');
// The passing sibling still reports PASS (pool completes, no fail-fast).
expect(r.out).toContain('PASS');
rmSync(join(ROOT, 'test', 'a-ok.serial.test.ts'));
rmSync(join(ROOT, 'test', 'z-bad.serial.test.ts'));
});
it('--dry-run-list lists every serial file without running anything', () => {
writeFileSync(join(ROOT, 'test', 'a-ok.serial.test.ts'), PASSING);
const out = execFileSync(
'bash',
[join(ROOT, 'scripts', 'run-serial-tests.sh'), '--dry-run-list'],
{ cwd: ROOT, encoding: 'utf-8', env: ENV },
);
expect(out.trim().split('\n')).toEqual(['test/a-ok.serial.test.ts']);
rmSync(join(ROOT, 'test', 'a-ok.serial.test.ts'));
});
it('an externally-SIGTERMed file is rescued by a sequential re-run (phantom stays green)', () => {
// Self-kills with SIGTERM on first run (exit 143 — the external-kill
// class: sibling-workspace cleanup, memory jetsam), passes on the
// rescue re-run. Mirrors run-unit-parallel's oom-once fixture.
const sentinel = join(ROOT, 'test', 'killed-once.sentinel');
const KILLED_ONCE = `import { it, expect } from 'bun:test';
import { existsSync, writeFileSync } from 'fs';
it('passes after one external SIGTERM', () => {
const sentinel = ${JSON.stringify(sentinel)};
if (!existsSync(sentinel)) {
writeFileSync(sentinel, '1');
process.kill(process.pid, 'SIGTERM');
}
expect(1).toBe(1);
});`;
writeFileSync(join(ROOT, 'test', 'k-killed.serial.test.ts'), KILLED_ONCE);
try {
const r = runScript();
// (The "queued for serial rescue" line goes to stderr, which the
// success path of runScript doesn't capture — the stdout rescue
// marker + exit 0 are the contract.)
expect(r.out).toContain('rescued: external-kill phantom');
expect(r.code).toBe(0);
} finally {
rmSync(join(ROOT, 'test', 'k-killed.serial.test.ts'), { force: true });
rmSync(sentinel, { force: true });
}
}, 60000);
it('a hung file is killed by the per-file wall-clock timeout', () => {
if (!hasTimeoutBin) return; // macOS without coreutils: no wrapper, documented
writeFileSync(join(ROOT, 'test', 'h-hang.serial.test.ts'), HANGING);
const r = runScript({ GBRAIN_SERIAL_FILE_TIMEOUT: '3' });
expect(r.code).toBe(1);
expect(r.out).toContain('per-file timeout');
expect(r.out).toContain('test/h-hang.serial.test.ts');
rmSync(join(ROOT, 'test', 'h-hang.serial.test.ts'));
}, 60000);
});
+4 -10
View File
@@ -23,15 +23,12 @@ import { describe, it, expect, beforeAll, afterAll } from 'bun:test';
import { execFileSync, spawnSync } from 'child_process';
import { mkdtempSync, mkdirSync, writeFileSync, readFileSync, existsSync, rmSync, copyFileSync, chmodSync, symlinkSync } from 'fs';
import { tmpdir } from 'os';
import { dirname, join, resolve } from 'path';
import { join, resolve } from 'path';
const REPO_ROOT = resolve(import.meta.dir, '..', '..');
const PARALLEL_SH_SRC = resolve(REPO_ROOT, 'scripts/run-unit-parallel.sh');
const SHARD_SH_SRC = resolve(REPO_ROOT, 'scripts/run-unit-shard.sh');
const SERIAL_SH_SRC = resolve(REPO_ROOT, 'scripts/run-serial-tests.sh');
// The runners `source scripts/lib/test-env.sh` — every sandbox copy of a
// runner must stage the lib too or the source line fails at startup.
const TESTENV_SH_SRC = resolve(REPO_ROOT, 'scripts/lib/test-env.sh');
let TMPROOT: string;
@@ -40,9 +37,8 @@ beforeAll(() => {
// and 4 fixture test files (3 pass, 1 fail). The wrapper's `find test`
// expression will pick them up via cwd.
TMPROOT = mkdtempSync(join(tmpdir(), 'gbrain-parallel-test-'));
mkdirSync(join(TMPROOT, 'scripts', 'lib'), { recursive: true });
mkdirSync(join(TMPROOT, 'scripts'), { recursive: true });
mkdirSync(join(TMPROOT, 'test'), { recursive: true });
copyFileSync(TESTENV_SH_SRC, join(TMPROOT, 'scripts', 'lib', 'test-env.sh'));
copyFileSync(PARALLEL_SH_SRC, join(TMPROOT, 'scripts', 'run-unit-parallel.sh'));
copyFileSync(SHARD_SH_SRC, join(TMPROOT, 'scripts', 'run-unit-shard.sh'));
@@ -194,8 +190,7 @@ describe('run-unit-parallel.sh no-timeout-binary fallback (rc from shard wait, n
FROOT = mkdtempSync(join(tmpdir(), 'gbrain-parallel-fallback-'));
mkdirSync(join(FROOT, 'scripts'), { recursive: true });
mkdirSync(join(FROOT, 'test'), { recursive: true });
for (const s of ['run-unit-parallel.sh', 'run-unit-shard.sh', 'run-serial-tests.sh', 'lib/test-env.sh']) {
mkdirSync(dirname(join(FROOT, 'scripts', s)), { recursive: true });
for (const s of ['run-unit-parallel.sh', 'run-unit-shard.sh', 'run-serial-tests.sh']) {
copyFileSync(resolve(REPO_ROOT, 'scripts', s), join(FROOT, 'scripts', s));
chmodSync(join(FROOT, 'scripts', s), 0o755);
}
@@ -280,8 +275,7 @@ describe('run-unit-parallel.sh OOM rescue lane', () => {
OROOT = mkdtempSync(join(tmpdir(), 'gbrain-parallel-oom-'));
mkdirSync(join(OROOT, 'scripts'), { recursive: true });
mkdirSync(join(OROOT, 'test'), { recursive: true });
for (const s of ['run-unit-parallel.sh', 'run-unit-shard.sh', 'run-serial-tests.sh', 'lib/test-env.sh']) {
mkdirSync(dirname(join(OROOT, 'scripts', s)), { recursive: true });
for (const s of ['run-unit-parallel.sh', 'run-unit-shard.sh', 'run-serial-tests.sh']) {
copyFileSync(resolve(REPO_ROOT, 'scripts', s), join(OROOT, 'scripts', s));
chmodSync(join(OROOT, 'scripts', s), 0o755);
}
+2 -50
View File
@@ -61,52 +61,6 @@ describe("run-verify-parallel.sh — CLI contract", () => {
});
});
describe("guard registration ⇒ execution coverage", () => {
// guard-self-test.sh enforces that every scripts/check-* guard is
// REGISTERED in guards-manifest.tsv, but nothing enforced that a
// registered guard actually EXECUTES anywhere — five guards sat
// registered-but-dead until the v0.45.x test/eval/CI pass. This closes
// the loop: every manifest guard must be reachable from verify's CHECKS
// (via a package.json script that invokes its file), or be explicitly
// exempted HERE with the reason it runs elsewhere / deliberately not.
const EXECUTION_EXEMPT: Record<string, string> = {
"check-bun-test-timeout.sh":
"runs directly as a test.yml verify-job step (not via CHECKS — avoids a package.json edit)",
"check-jsonb-params.mjs":
"exercised by test/check-jsonb-params.test.ts + guard self-test fixtures",
"check-admin-embedded.sh":
"duplicates check:admin-build's vite+tsc build; embed freshness covered there",
"check-image-decoders-embedded.sh":
"runs its own bun build --compile — too heavy for per-verify cadence",
};
it("every manifest guard is executed by verify or explicitly exempt", () => {
const manifestLines = readFileSync("scripts/guards-manifest.tsv", "utf8")
.split("\n")
.filter((l) => l.trim() && !l.startsWith("#"));
const guards = manifestLines.map((l) => l.split("\t")[0]!).filter(Boolean);
expect(guards.length).toBeGreaterThan(30);
const pkg = JSON.parse(readFileSync("package.json", "utf8")) as {
scripts: Record<string, string>;
};
const dry = spawnSync("bash", [SCRIPT, "--dry-list"], { encoding: "utf8" });
expect(dry.status).toBe(0);
const executed = new Set(dry.stdout.trim().split("\n"));
const missing: string[] = [];
for (const g of guards) {
if (EXECUTION_EXEMPT[g]) continue;
const invokingKeys = Object.entries(pkg.scripts)
.filter(([, cmd]) => cmd.includes(`scripts/${g}`))
.map(([key]) => key);
const covered = invokingKeys.some((k) => executed.has(k));
if (!covered) missing.push(g);
}
expect(missing).toEqual([]);
});
});
describe("run-verify-parallel.sh — failure surfacing (synthetic dispatcher)", () => {
// We can't inject a fake check into the real script without touching the
// CHECKS array. Instead, we write a SMALLER synthetic dispatcher that
@@ -249,15 +203,13 @@ describe("run-verify-parallel.sh — no-timeout-binary fallback rc capture (regr
function makeFallbackHarness(): { root: string; env: Record<string, string> } {
const root = mkdtempSync(join(tmpdir(), "verify-fallback-"));
mkdirSync(join(root, "scripts", "lib"), { recursive: true });
mkdirSync(join(root, "scripts"), { recursive: true });
copyFileSync(SCRIPT, join(root, "scripts", "run-verify-parallel.sh"));
// The dispatcher sources the shared runner lib — stage it too (E3).
copyFileSync("scripts/lib/test-env.sh", join(root, "scripts", "lib", "test-env.sh"));
const bin = join(root, "bin");
mkdirSync(bin);
// Everything the dispatcher and its subshells invoke, minus timeout bins.
for (const tool of ["bash", "sh", "env", "dirname", "mktemp", "date", "sleep", "cat", "tail", "head", "rm", "mkdir", "pkill", "grep", "sed", "awk", "wc", "tr"]) {
for (const tool of ["bash", "sh", "env", "dirname", "mktemp", "date", "sleep", "cat", "tail", "head", "rm", "mkdir", "pkill", "grep", "sed", "awk"]) {
const p = Bun.which(tool);
if (p) symlinkSync(p, join(bin, tool));
}
+1 -29
View File
@@ -18,7 +18,7 @@
import { describe, it, expect } from 'bun:test';
import { execFileSync } from 'child_process';
import { existsSync, readFileSync } from 'fs';
import { readFileSync } from 'fs';
import { resolve } from 'path';
const REPO_ROOT = resolve(import.meta.dir, '..', '..');
@@ -68,31 +68,3 @@ describe('run-serial-tests.sh contract', () => {
expect(overlap).toEqual([]);
});
});
describe('EXCLUSIVE_FILES (pooled-runner opt-out) guards', () => {
const src = () => readFileSync(SERIAL_SH, 'utf-8');
const entries = () =>
[...src().matchAll(/^\s*"(test\/[^"]+\.serial\.test\.ts)"\s*$/gm)].map(m => m[1]);
it('every exclusive entry exists on disk and is discovered by the runner', () => {
const listed = dryRunList(SERIAL_SH);
const e = entries();
expect(e.length).toBeGreaterThan(0);
for (const f of e) {
expect(existsSync(resolve(REPO_ROOT, f))).toBe(true);
expect(listed).toContain(f);
}
});
it('the exclusive list does not silently grow (quarantine-growth guard)', () => {
// Exclusivity re-serializes the runner one file at a time — the exact
// 8.5-minute disease the pool removed. Each entry must carry a
// justification comment; past 3 entries, stop and rethink the design
// (per-file PATH shims are usually the right fix) instead of quarantining.
expect(entries().length).toBeLessThanOrEqual(3);
});
it('a missing exit sentinel is a failure, never a silent pass', () => {
expect(src()).toMatch(/missing exit sentinel/);
});
});
+7 -20
View File
@@ -7,7 +7,6 @@
import { describe, expect, it } from "bun:test";
import {
computeMedian,
computeQuantile,
imbalanceRatio,
loadWeights,
partition,
@@ -84,32 +83,20 @@ describe("partition — happy path", () => {
});
describe("partition — fallback semantics", () => {
it("missing weights default to corpus p75 (missing files skew heavy)", () => {
// weights = {a:100, b:50}, p75 (nearest-rank of [50,100] at 0.75) = 100.
// Files c + d are unknown → 100 each.
it("missing weights default to corpus median", () => {
// weights = {a:100, b:50}, median = 75. Files c + d are unknown → 75 each.
const weights: WeightMap = new Map([
["a", 100],
["b", 50],
]);
const out = partition(["a", "b", "c", "d"], weights, 2);
// Effective weights: a=100, b=50, c=100, d=100. Exact LPT placement of
// ties is implementation-defined — assert the invariant instead: totals
// differ by ≤ the fallback (the LPT bound), and every file landed once.
// Effective weights: a=100, b=50, c=75, d=75. LPT: 100→s0, 75→s1 (c),
// 75→s1 (d, ties broken alpha)... actually: 100→s0=100, 75→s1=75,
// 75→s1 vs s0 → s1=150, 50→s0=150. Balanced 150/150.
const totalsEffective = out.map((s) =>
s.reduce((acc, f) => acc + (weights.get(f) ?? 100), 0),
s.reduce((acc, f) => acc + (weights.get(f) ?? 75), 0),
);
expect(Math.abs(totalsEffective[0]! - totalsEffective[1]!)).toBeLessThanOrEqual(100);
const flat = out.flat().sort();
expect(flat).toEqual(["a", "b", "c", "d"]);
});
it("computeQuantile: nearest-rank p75 on a skewed corpus", () => {
// Right-skewed like the real weights file: p75 lands in the tail's
// foothills, far above the median.
expect(computeQuantile([1, 1, 1, 1000], 0.75)).toBe(1);
expect(computeQuantile([1, 2, 3, 4], 0.75)).toBe(3);
expect(computeQuantile([50, 100], 0.75)).toBe(100);
expect(computeQuantile([], 0.75)).toBe(0);
expect(totalsEffective[0]).toBe(totalsEffective[1]);
});
it("explicit fallback override beats median", () => {
+15 -50
View File
@@ -130,68 +130,33 @@ describe('test-shard.sh — LPT balance contract', () => {
});
function totalsFor(shards: string[][]): number[] {
// p75 fallback mirroring sharding.ts's computeQuantile choice (the
// weight distribution is right-skewed and missing files skew heavy).
const sorted = Array.from(weightsMap.values()).sort((a, b) => a - b);
// Use 30ms as the cold-start fallback (matches mine-shard-weights
// median observation). When weights are loaded, missing files get
// the corpus median anyway via sharding.ts.
const fallback = weightsLoaded
? sorted[Math.min(sorted.length - 1, Math.ceil(0.75 * sorted.length) - 1)] ?? 30
? Array.from(weightsMap.values()).sort((a, b) => a - b)[
Math.floor(weightsMap.size / 2)
] ?? 30
: 1;
return shards.map((s) =>
s.reduce((acc, f) => acc + (weightsMap.get(f) ?? fallback), 0),
);
}
// CI runs THIS many shards (test.yml matrix). The old version of this
// test asserted 4- and 6-shard splits — configurations nothing runs.
const CI_SHARDS = 10;
it('CI_SHARDS matches the test.yml matrix (parsed, not regexed)', () => {
const fs = require('fs');
const yaml = require('js-yaml');
const wf = yaml.load(
fs.readFileSync(resolve(REPO_ROOT, '.github/workflows/test.yml'), 'utf8'),
) as { jobs: { test: { strategy: { matrix: { shard: unknown[] } } } } };
const matrix = wf.jobs.test.strategy.matrix.shard;
expect(Array.isArray(matrix)).toBe(true);
expect(matrix.length).toBe(CI_SHARDS);
});
it(`${CI_SHARDS}-shard wallclock imbalance ratio ≤ 1.5 (the configuration CI actually runs)`, () => {
const shards = Array.from({ length: CI_SHARDS }, (_, i) => dryRunList(i + 1, CI_SHARDS));
it('4-shard wallclock imbalance ratio ≤ 1.5', () => {
const shards = [1, 2, 3, 4].map(s => dryRunList(s, 4));
for (const s of shards) expect(s.length).toBeGreaterThan(0);
const totals = totalsFor(shards);
const ratio = Math.max(...totals) / Math.min(...totals);
expect(ratio).toBeLessThanOrEqual(1.5);
}, 60_000);
});
// The two guards below are what make the ratio assertion above MEAN
// something: recomputing totals with the same weights the partitioner
// used is near-tautological — unless the weights actually cover the
// corpus and refer to real files. Weight rot (files added without a
// re-mine, or renamed away from their entries) used to be invisible:
// 45% of the corpus once rode a 30ms median fallback while really
// averaging ~4s, and the "balanced" partition was balanced on fiction.
it('weights cover ≥70% of matrix-eligible files (anti-rot gate)', () => {
const all = Array.from(
new Set(Array.from({ length: CI_SHARDS }, (_, i) => dryRunList(i + 1, CI_SHARDS)).flat()),
);
expect(all.length).toBeGreaterThan(0);
const covered = all.filter((f) => weightsMap.has(f)).length;
const coverage = covered / all.length;
// Regenerate from the latest green Test run:
// bun run scripts/mine-shard-weights.ts --run <run id>
expect(coverage).toBeGreaterThanOrEqual(0.7);
}, 60_000);
it('no stale weight keys — every entry names a tracked file', () => {
const tracked = new Set(
execFileSync('git', ['ls-files', 'test', 'evals'], { cwd: REPO_ROOT, encoding: 'utf-8' })
.split('\n')
.map((s) => s.trim())
.filter(Boolean),
);
const stale = [...weightsMap.keys()].filter((k) => !tracked.has(k));
expect(stale).toEqual([]);
it('6-shard wallclock imbalance ratio ≤ 1.5', () => {
const shards = [1, 2, 3, 4, 5, 6].map(s => dryRunList(s, 6));
for (const s of shards) expect(s.length).toBeGreaterThan(0);
const totals = totalsFor(shards);
const ratio = Math.max(...totals) / Math.min(...totals);
expect(ratio).toBeLessThanOrEqual(1.5);
});
it('6-shard partition is deterministic across runs', () => {
+1 -42
View File
@@ -12,12 +12,7 @@ import { mkdtempSync, rmSync, writeFileSync } from 'fs';
import { tmpdir } from 'os';
import { join } from 'path';
import * as crypto from 'node:crypto';
import {
tryLoadSnapshot,
computeSnapshotSchemaHash,
__snapshotMemoStatsForTests,
__resetSnapshotMemoForTests,
} from '../src/core/pglite-engine.ts';
import { tryLoadSnapshot, computeSnapshotSchemaHash } from '../src/core/pglite-engine.ts';
import { MIGRATIONS } from '../src/core/migrate.ts';
import { PGLITE_SCHEMA_SQL } from '../src/core/pglite-schema.ts';
import { getEmbeddingDimensions, getEmbeddingModel } from '../src/core/ai/gateway.ts';
@@ -26,7 +21,6 @@ let dir: string;
beforeEach(() => {
dir = mkdtempSync(join(tmpdir(), 'gbrain-snap-guard-'));
__resetSnapshotMemoForTests();
});
afterEach(() => {
@@ -69,41 +63,6 @@ test('matching hash + shape loads the blob', () => {
expect(blob!.size).toBeGreaterThan(0);
});
test('memo: same path is read once per process, blob identical across calls', () => {
const tar = writeFixture(`${currentHash()}\ndims=${getEmbeddingDimensions()}\nmodel=${getEmbeddingModel()}\n`);
const b1 = tryLoadSnapshot(tar);
const afterFirst = __snapshotMemoStatsForTests().tarReads;
const b2 = tryLoadSnapshot(tar);
const afterSecond = __snapshotMemoStatsForTests().tarReads;
expect(b1).not.toBeNull();
expect(b2).toBe(b1); // same Blob instance — the tar was not re-read
expect(afterFirst).toBe(1);
expect(afterSecond).toBe(1);
});
test('memo: shape refusal is per-call, never cached as terminal — and costs zero tar reads', () => {
// Hash matches but dims mismatch: the version entry is memoized yet every
// call re-runs the shape gate against the CURRENT gateway config — an
// engine with a matching config later in the same process could still
// load this snapshot (the zembed/1280 poisoning guard staying hot behind
// the memo). The 42MB tar read is deferred until a shape-MATCHING caller,
// so a process that only ever refuses never reads it at all.
const tar = writeFixture(`${currentHash()}\ndims=99999\nmodel=${getEmbeddingModel()}\n`);
expect(tryLoadSnapshot(tar)).toBeNull();
expect(__snapshotMemoStatsForTests().tarReads).toBe(0);
expect(__snapshotMemoStatsForTests().memoEntries).toBe(1); // entry exists — not terminal
expect(tryLoadSnapshot(tar)).toBeNull();
expect(__snapshotMemoStatsForTests().tarReads).toBe(0);
});
test('memo: stale hash is terminal — tar never read, repeat calls short-circuit', () => {
const tar = writeFixture(`deadbeef\ndims=${getEmbeddingDimensions()}\nmodel=${getEmbeddingModel()}\n`);
expect(tryLoadSnapshot(tar)).toBeNull();
expect(__snapshotMemoStatsForTests().tarReads).toBe(0);
expect(tryLoadSnapshot(tar)).toBeNull();
expect(__snapshotMemoStatsForTests().tarReads).toBe(0);
});
test('D5.13: a migration handler edit changes the hash (sql-only hashing missed 19 handler migrations)', () => {
const base = [
{ version: 1, name: 'a', sql: 'CREATE TABLE t(x int)' },