mirror of
https://github.com/garrytan/gbrain.git
synced 2026-08-14 00:48:18 +00:00
* fix(cli): exit deliberately after bounded teardown instead of riding the 10s backstop (#2084)
Root cause: bounded teardown (endPoolBounded, #2015) RESOLVES, but lingering
sockets — embedding-provider fetch keep-alive, PgBouncer txn-mode sockets the
bound raced past — keep Bun's event loop alive, so every `gbrain query` paid
a flat 10s tax exiting via the hard-deadline force-exit banner.
Three changes, one contract:
- flushStdoutThenExit (cli-force-exit.ts): when main() resolves and the
command is not a daemon, exit deliberately — after stdout AND stderr drain
(writableLength===0, 'drain'-event + poll loop, 2s unref'd guard for a
blocked pipe). Incident #1959 (force-exit truncating piped stdout) is the
regression class; pinned by a 256KB real-pipe subprocess test.
- drainThenDisconnect (cli.ts): ONE owner-disconnect helper at all 8 sites
(op-dispatch, CLI_ONLY fall-through, search dashboard, doctor remediation
x3, ze-switch, dream, read-only timeout path). Drains the background-work
registry, then disconnect (best-effort), bounded by the 10s unref'd
hard-deadline — which is now armed around the TEARDOWN window only, not
before the op handler (the old placement would have force-killed any op
slower than 10s). Closes the filed TODOS P3 drain-hoist: six sites
previously skipped the drain entirely and had no hang timer at all.
- Inner process.exit sweep: mid-handler exits in engine-owning/output-bearing
paths (status, friction, claw-test, smoke-test, eval cross-modal /
takes-quality replay / conversation-parser / whoknows-thin, status-thin)
become process.exitCode + return so they flow through the drains and the
flush-exit. Pre-engine usage/parse/refusal exits stay as-is.
BrainRegistry.disconnectAll deliberately unchanged: zero production callers
in src/, per-engine disconnects already bounded, and the kernel reclaims
sockets on exit (src/core/timeout.ts doctrine).
DAEMON_COMMANDS gains 'watch' ahead of the #2095 push transport.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(e2e): PgBouncer transaction-mode pooler in CI + teardown e2e (#2084)
Three consecutive waves (#1972 → #2015 → #2084) fixed pooler-teardown bugs
verified only against one production deployment — CI had no transaction-mode
pooler and could never see the class. Now it can:
- docker-compose.ci.yml: `pgbouncer` service (transaction pooling) fronting
postgres-1, mirroring the production split-pool topology (direct :5432 +
pooled :6543). AUTH_TYPE=plain (pg16 SCRAM verifiers need the plaintext
password in the userlist) + IGNORE_STARTUP_PARAMETERS for the
statement_timeout/idle_in_transaction_session_timeout startup params
gbrain's client sets (the Supabase pooler whitelists the same).
- test/e2e/pgbouncer-teardown.test.ts: schema + fixture via the DIRECT url
into a dedicated `gbrain_pgbouncer` database (never races shard TRUNCATEs),
then spawns the real CLI against the POOLED url and asserts: exit 0,
stdout intact (the #1959 truncation class), and NO
"did not return within 10000ms — force-exiting" banner (pre-#2084 it
printed on 100% of query-shaped ops on this topology). Class bound, not
exact timing. Skips gracefully without GBRAIN_PGBOUNCER_URL.
- scripts/ci-local.sh: threads GBRAIN_PGBOUNCER_URL +
GBRAIN_PGBOUNCER_DIRECT_URL into all three e2e phases.
Verified live: both tests green against pgbouncer 1.25.2 in transaction mode.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(schema): context_volunteer_events table (v116) — push-context feedback log (#2095)
One row per page the brain volunteers (op / reflex / watch channels).
"Used" is DERIVED, never written: pages.last_retrieved_at > volunteered_at
(the existing bumpLastRetrievedAt write-back is the open/cite signal), so
there is no second tracking path. session_id/turn are nullable
caller-supplied attribution; rationale is a deterministic template string,
never raw conversation text.
- Migration v116 (idempotent) + mirrors in src/schema.sql +
src/core/pglite-schema.ts + regenerated schema-embedded.ts (regen also
folds in pre-existing comment-only drift from the v114 links edits).
- src/core/context/volunteer-events.ts: insertVolunteerEvents (ONE
multi-row parameterized INSERT — never per-row awaited round-trips) +
purgeStaleVolunteerEvents (90-day GC, returns 0 on pre-v116 brains).
- Dream cycle purge phase prunes stale events alongside op_checkpoints /
brainstorm checkpoints / batch-retry audit files.
- RLS on Postgres comes from the v35 auto_rls_on_create_table event
trigger (the same mechanism that covered v110 page_aliases and v115
op_checkpoint_paths); the volunteer Postgres e2e pins it.
- No ::jsonb anywhere; no bootstrap probe needed (nothing references the
table pre-creation; writers guard with try/catch).
Tests: v116 shape + columns + indexes + live insert/purge round-trip on
PGLite (test/migrate.test.ts, 161 pass); schema-bootstrap-coverage green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(context): multi-turn window extraction + confidence-scored volunteer core (#2095)
- entity-salience.ts: extractCandidatesFromWindow(turns) — runs the existing
per-turn extractor across the last N turns (oldest→newest), merges by the
normalizeAlias form with occurrence/newest-turn/user-mention metadata, and
orders by salience (recency > frequency > user-role) so the MAX_CANDIDATES
cap drops stale assistant chatter, not the entity the user just named.
Closes the filed assistant-introduced-entities recall TODO; true pronoun
coreference (never-named antecedents) stays out of scope.
- retrieval-reflex.ts: ReflexPointer gains source_id + arm + confidence +
matchedNorm. ARM_CONFIDENCE (alias 0.9 / title 0.8 / slug-suffix 0.6)
lives next to the arm definitions so identity and score can't drift.
Arm-2 provenance is classified in JS (codex D8 — the combined OR can't
report which predicate matched). Federated sourceIds[] scope (alias arm
loops per source; arm 2 uses source_id = ANY — no engine-interface
change). Suppression gains 'slug-only' mode (codex D7, REQUIRED for
windowing): the legacy title-whole-word rule would suppress every entity
merely MENTIONED in a prior window turn, breaking the feature by
construction — slugs only enter context when a pointer/page was actually
surfaced. Default stays 'slug-and-title' for the window=1 legacy path.
- volunteer.ts (new): parseWindow (lenient user:/assistant: prefixes, CRLF,
unprefixed → one user turn), volunteerContext (zero-LLM: extract →
resolve → +0.05 multi-turn/newest-turn boost → min_confidence 0.7 gate →
cap 3/5; deterministic rationale strings, never raw conversation text),
and volunteerUsageStats (per-arm/channel precision from the
last_retrieved_at join, labeled approximate — 5-min throttle false
negatives, unrelated-read false positives; codex D9).
Tests: 35 green across volunteer-context (window parsing, pronoun follow-up
via assistant-introduced entity, confidence gating, slug-only suppression,
takes-fence privacy, multi-source scope, caps, stats join math) +
retrieval-reflex back-compat + resolve-ipc.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(ops): volunteer_context op — CLI (stdin) + MCP, drained event sink (#2095)
New read-scope op on the contract surface (CLI `gbrain volunteer-context`
with stdin → window, MCP tool for free): takes a rolling conversation
window, returns confidence-gated page pointers with rationales + synopses.
`window` is optional-unless-stats (validated in the handler, codex D9);
`stats: true` returns the volunteered-vs-used precision summary, labeled
APPROXIMATE (the 5-min last-retrieved throttle and unrelated reads both
bias the join). Source scope threads through sourceScopeOpts — federated
grants narrow the volunteer to the granted sources.
Event logging is fire-and-forget through a new `volunteer-events`
background-work sink (volunteer-events.ts, mirrors last-retrieved: tracked
dangling promise set + bounded drain + snapshot-drop on timeout so a
long-lived process never accumulates ghosts). ONE batched INSERT per call,
drained on every exit path by the commit-1 drain hoist; failure never
fails the op (pinned by an injected failing-engine test).
cli formatResult renders both shapes (pointer lines with confidence/arm/
rationale; the stats summary with per-arm precision).
Tests: op contract surface, window-required validation, sink round-trip
with session_id/turn attribution, failing-engine fail-open, federated
grant scoping, stats mode (26 green on PGLite) + a real-Postgres e2e
proving the op + sink + stats join AND that context_volunteer_events has
RLS enabled (keeps the auto-RLS event-trigger mechanism honest for v116).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(context): reflex consumes the rolling window + ambient-channel logging (#2095)
The default-on retrieval reflex now extracts entities from the last N turns
(retrieval_reflex_window_turns, default 4; env
GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS; window=1 reproduces the legacy
current-turn-only behavior exactly). assemble() passes the recent
user/assistant turns (hard cap 12); the reflex slices to the configured
window. Assistant-introduced entities and "what did she invest in?"
follow-ups whose antecedent was NAMED in the window now surface pointers —
the issue's "zero agent-initiated queries" success criterion on the
ambient path.
Under windowing, suppression switches to slug-only (codex D7): the legacy
title-whole-word rule would suppress every entity merely MENTIONED in a
prior window turn, breaking the feature by construction. Slugs only enter
prior context when a pointer/page was actually surfaced, so
already-surfaced pages still suppress. The suppression mode flows through
all three resolver rungs (host opts, serve IPC request, direct Postgres).
Ambient-channel feedback (codex D11): the server-side resolver paths
(serve IPC + direct Postgres) log volunteered pointers with
channel: 'reflex' through the drained volunteer-events sink, so
`gbrain volunteer-context --stats` measures the default-on path where most
volunteering happens. Host-injected resolvers (no gbrain engine) can't
log — documented gap. Precision gates, 1.5s ceiling, fail-open, and the
pointer cap are unchanged.
Tests: prev-assistant-turn entity fires; window=1 legacy parity; slug-only
vs already-surfaced suppression; throwing resolver stays fail-open
(16 green).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(cli): gbrain watch — push transport over stdin (#2095)
The issue's headline: the brain volunteers pages as the conversation flows,
instead of waiting to be asked. `some-transcript-feed | gbrain watch` reads
turns line-by-line ('user:'/'assistant:' prefixes set the role; unprefixed
lines are user turns), keeps a rolling window (--window-turns, default 4),
and streams confidence-gated pointers with rationales to stdout (--json for
JSONL). Session dedupe rides the core's slug-only suppression — a slug is
volunteered at most once per session. Events log on channel 'watch' with
session_id + turn through the drained sink.
Lifecycle: watch BLOCKS in the stdin iteration (like `jobs work`) — an
interactive TTY stays alive until Ctrl-C/Ctrl-D, piped input ends at EOF —
so it is deliberately NOT in DAEMON_COMMANDS (reverts the commit-1
placeholder): when main() resolves the work is over, the CLI_ONLY finally
drains volunteer events via drainThenDisconnect, and the entrypoint
flush-exit ends the process. Keeping it in the daemon set would have made
the piped EOF path hang on lingering sockets — the exact #2084 class.
SIGINT closes the stream and flows through the same drain path instead of
killing mid-write. Per-turn resolution failures are fail-open (the stream
never dies on a transient DB error).
Full wiring (eng-review D12): CLI_ONLY + CLI_ONLY_SELF_HELP (WATCH_HELP) +
THIN_CLIENT_REFUSED_COMMANDS (thin clients use the volunteer_context MCP
op) + main --help entry.
Tests: 18 green — help, per-turn volunteering + clean EOF return, rolling
window via assistant-introduced entity, session dedupe, --json shape with
turn attribution, channel-watch event rows, --min-confidence gate, CRLF/
blank tolerance, daemon-gate semantics. Live smoke: piped `gbrain watch`
on a fresh PGLite brain exits 0 at EOF with no force-exit banner.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: KEY_FILES + push-context guide + TODOS for the #2084/#2095 wave
- docs/architecture/KEY_FILES.md (current-state): context entries gain the
window extractor, arm provenance/confidence, suppression modes, volunteer
+ volunteer-events modules; background-work entry now lists FIVE sinks and
the drainThenDisconnect owner-disconnect contract; new entries for
src/core/cli-force-exit.ts (the exit contract) and src/commands/watch.ts.
- docs/guides/push-context.md (new): the three channels (reflex/op/watch),
the confidence model, CLI usage, config keys, and the approximate-stats
caveat. Linked from CLAUDE.md's reference map.
- CLAUDE.md: ops line mentions volunteer_context + the guide link;
bun run build:llms regenerated in the same commit (freshness test green).
- TODOS.md: #2095 deferrals filed (SSE push channel, policy skill + doctor
check, structured messages[] param); the #1981 entity-detection TODO
narrowed (window extraction covered assistant-introduced entities +
named-antecedent follow-ups); the drain-hoist P3 marked DONE by the
#2084 wave.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(e2e): truncate context_volunteer_events in setupDB (#2095)
The new feedback-log table wasn't in ALL_TABLES, so volunteered-event rows
persisted across e2e runs on a reused database and poisoned count/stats
assertions in volunteer-context-postgres on the second run. No FK to pages
(slug join), so position before pages is for hygiene only.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(cli): own the exit verdict — never trust ambient process.exitCode (#2084)
Caught by the full unit suite: `gbrain apply-migrations` on PGLite started
exiting 99. Root cause: PGLite's Emscripten runtime writes the WASM
backend's proc_exit status into process.exitCode (initdb at create-time,
the postmaster at close-time — `exitCode=status` in pglite's dist), and
the writes land ASYNCHRONOUSLY, outside any snapshot/restore window around
create/close (a guarded attempt verified this). The pre-#2084 success path
never read process.exitCode, so the pollution was invisible; the new
deliberate flush-exit propagated it faithfully.
Fix: gbrain records its own verdict. setCliExitCode(n)/getCliExitCode() in
cli-force-exit.ts — every gbrain-owned exit-code assignment routes through
the setter (still mirrored to process.exitCode for outside readers), and
both exit paths (entrypoint flushStdoutThenExit + the drainThenDisconnect
hard-deadline backstop) read the getter. Swept all assignment sites:
cli.ts (op error, friction, claw-test, smoke-test, eval runners, status,
import errors) + reindex/transcripts/brainstorm/frontmatter/autopilot.
Also updates the v0.42.20 structural pins to the drainThenDisconnect shape
(ordering invariant asserted INSIDE the helper + >=8 helper call sites,
superseding the two-inline-pairs assertion).
Verified: apply-migrations spawn test green; `init --migrate-only` exits 0;
an errored op still exits 1.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test: re-pin the teardown-arming invariant at its post-#2084 home
Master's v0.42.41.0 triage wave and the #2084 wave fixed the same
pre-armed-timer bug independently; the merge keeps #2084's shape (arming
inside the shared drainThenDisconnect helper, covering all 8 exit paths).
The structural pin now asserts the same invariant — no pre-try arming;
gated, unref'd, before-drain, cleared — at the helper.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test: coverage for ambient reflex-channel logging + watch window/cap flags
Ship coverage audit (85%, gate PASS) named five gaps; the two substantive
cheap ones close here: the codex-D11 logChannel='reflex' path now has a
behavioral pin (events land on channel 'reflex' through the drained sink;
no logChannel → no events), and gbrain watch's --window-turns / --max-pages
flags are exercised (turn-1 attribution under window=1; cap to one page).
Remaining flagged-not-blocking: the wallclock-timeout branch (untestable
without >10s real-clock flake — same rationale as the arming pin),
formatResult's volunteer case (module-private), and the cycle purge wiring.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test: close the remaining plan-audit gaps — formatResult rendering + watch SIGINT
formatResult exported for tests (same import-safety contract as cliAliases);
test/cli-format-volunteer.test.ts pins the pointer lines, empty-gate message,
and approximate stats summary. test/watch-command.test.ts gains a real
subprocess SIGINT test: piped stdin that never reaches EOF, SIGINT mid-stream,
assert exit 0 with no force-exit banner — the drain-then-exit lifecycle under
the actual signal, not just the shared exit path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: doctor's FAIL verdict was zeroed by the owned exit — sweep stragglers + class pin
The merged-state suite caught it: doctor --fast --json reported FAIL but
exited 0. Master's v0.42.41.0 brought raw `process.exitCode =` writes
(doctor.ts hasFail ternary, extract.ts) that the #2084 verdict-owning exit
silently zeroes — getCliExitCode() deliberately never reads ambient
process.exitCode (the PGLite-Emscripten pollution defense), so any setter
that bypasses setCliExitCode reports success on failure.
Swept both sites and added the structural class pin: a test greps src/ for
raw `process.exitCode =` outside cli-force-exit.ts, so the next merge that
introduces one fails loudly instead of lying about exit codes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: bump version and changelog (v0.42.43.0)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test: quarantine the watch SIGINT subprocess test to the serial lane
The parallel unit shards flake on concurrent CLI subprocess spawns (failed
at 7ms in-suite, green solo) — same isolation rationale as
apply-migrations-pglite-spawn.serial.test.ts and #2141's R3 quarantine.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: update project documentation for v0.42.43.0
Post-ship doc verification against the release diff (#2095 push-based
context + #2084 superset hardening), with a cross-model doc review:
- push-context.md: version tag corrected to v0.42.43.0; per-call knobs
now cover prior_context/days and watch's flag surface accurately;
feedback-log writes described as best-effort; synopsis fence-strip
described as unconditional.
- CLAUDE.md: stale operation count (~47 -> ~90); volunteer_context
release reference corrected to v0.42.43.0.
- KEY_FILES.md: ci-local entry rewritten to current topology (4-shard
parallel default, four Postgres services, transaction-mode PgBouncer
+ GBRAIN_PGBOUNCER_URL/_DIRECT_URL exports); stale E2E file counts
dropped from the selector entry.
- TESTING.md: inventory entries for the new #2084 structural pins
(cli-exit-verdict-pin, cli-pipe-truncation), the push-context test
suite (volunteer-context, watch-command, watch-sigint.serial,
cli-format-volunteer), migrate v117 coverage, and the two new E2E
files (pgbouncer-teardown env gating, volunteer-context-postgres RLS
pin); check:all row corrected (not a superset of verify).
- AGENTS.md + RELEASING.md: ci:local descriptions updated to the
sharded + pooler topology.
- CHANGELOG (wording only, entry preserved): "retrieved" instead of
"opened" for the used-signal, pooler scoped to the local CI gate,
feedback log labeled best-effort.
- llms-config.ts: index the new push-context guide; bundles
regenerated (build:llms) and freshness test green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(test): correct the v116 reference — the table shipped as migration v117
* fix: pre-landing review hardening — federated alias parallelism, trust-boundary clamps, shared protocol helpers (#2095)
Five specialist reviewers (testing/maintainability/security/performance/
data-migration) on the reconciled diff; every finding applied:
Performance: the alias arm now resolves all granted sources CONCURRENTLY
(a federated caller paid M sequential RTTs per turn — ~355ms at 5 sources
cross-region, inside the reflex's 1.5s budget); watch's session dedupe is
O(1) Set membership instead of a monotonically growing priorContext string
(O(T²) over a long-lived session); getWindowTurns iterates from the tail
(per-turn cost no longer grows with session length); the resolver's
provenance maps fold into the existing candidate pass.
Security: volunteer_context clamps caller-supplied attribution at the trust
boundary — session_id capped at 256 chars (a read-scoped token could bank
~1MiB TEXT per request, retained 90 days), turn logged only when a safe
integer (a non-integer threw inside the batched INSERT and silently dropped
the whole batch). The privacy comments now state precisely what rationale
may contain (the matched entity's surface form — which by construction
resolved to an existing alias/title/slug — never free conversation text).
Maintainability: TURN_PREFIX_RE + formatVolunteeredPage exported from
volunteer.ts and shared by watch/cli (the two surfaces can no longer
drift); volunteerEventRowsFrom is the single VolunteerEventRow assembly
site for all three channels; watch's window default now honors the same
retrieval_reflex_window_turns config knob the reflex reads; the stale
pre-v116 comments swept to pre-v117.
Testing: the two flake-class CRITICALs fixed (pipe test asserts the
backstop banner instead of a cold-CI-hostile 9s wall bound; the SIGINT test
waits on watch's new machine-readable ready line instead of a fixed 15s
sleep — 2.5s and deterministic now); new coverage for the sink's timeout
branch + ghost-reference drop, watch per-turn fail-open, untrusted knob
clamps (min_confidence/max_pages/days), window-cap ordering (newest user
mention survives), serve-IPC suppression passthrough + channel=reflex
logging, windowTurnCount edge semantics, and structural pins for the sink
registration + cycle purge wiring. The exit-verdict pin's grep is now
operator/whitespace-tolerant.
Deferred with TODOs: resolver index shapes for the per-turn query;
batched first-prune after a long dream-cycle gap.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(context): red-team hardening — pre-cap dedupe, delivery-side reflex logging, window clamp
Four red-team findings on the #2095 push-context surface:
- RT1 starvation: watch's session-dedupe Set filtered AFTER volunteerContext's
cap, so a recurring already-pushed entity burned cap slots every turn and
starved fresh pages behind it. VolunteerOpts.excludeSlugs now skips inside
the pointer loop BEFORE the confidence gate and the cap.
- RT3 honest stats: reflex-channel event logging moved from inside the
resolver to the DELIVERY point — serve's resolve-IPC onDelivered hook fires
only after the response write succeeds, and buildReflexAddition logs only
after the per-turn timeout admits the block. A block the client's 250ms
budget abandoned was never injected and no longer counts as volunteered.
(logChannel resolver opt removed; logDeliveredReflexPointers is the seam.)
- RT5 unbounded window: --window-turns is clamped to [1, 64] so a config typo
can't reintroduce the re-scan-everything-per-turn cost class.
- RT2/RT4 documented + filed: PGLite watch connection monopoly (WATCH_HELP,
push-context guide, TODO to route watch via serve IPC); host-resolver
suppression contract at ResolveEntitiesFn (TODO for a capability gate).
Tests: starvation guard (watch + volunteerContext unit), window clamp floor +
ceiling, delivery-side logging (helper writes channel=reflex through the
drained sink; bare resolver writes nothing; empty list no-op), IPC wiring test
rewired to onDelivered. KEY_FILES.md + push-context.md updated; build:llms run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(context): env-plane window knob works config-less; harden two gateway-state-leak victims
Three CI-only check failures, two root causes:
1. windowTurnCount ignored GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS when
loadConfig() returned null (no config file AND no DATABASE_URL — a clean
CI shard with no brain). loadConfig drops its env→config mapping in that
case, so the documented escape hatch silently died and the window fell
back to 4 → windowed extraction widened when the test set window=1 →
prior-turn entity leaked. Fixed: read the env var directly in
windowTurnCount, mirroring reflexEnabled's direct process.env read. This
is a real product bug, not just a test artifact — any config-less host
using the env hatch was affected. Regression test pins it.
2. sync-cost-preview + doctor-federation-health failed only IN-SHARD: a
sibling test configured a non-legacy (ZeroEntropy 1280-d / $0.05) gateway
and never reset it. The legacy-embedding preload only restores the
OpenAI/1536 default when the gateway slot is EMPTY, so a non-empty foreign
config survives into the next file — and a file's beforeAll runs BEFORE
the preload's restoring beforeEach, so federation-health built a
vector(1280) column and its 1536-d fixture hit CheckExpectedDim. My new
test files reshuffled the deterministic file→shard assignment, exposing
this latent ordering bug. Hardened both victims to establish the gateway
state they assert (sync-cost-preview resets to the unconfigured fallback;
federation-health pins legacy 1536 before initSchema) so they're
order-independent. Verified against a simulated leaker run before them.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(context): use withEnv() in the window env-hatch test (test-isolation guard)
The regression test added in 82cc7fff mutated process.env directly, which
check:test-isolation (R1) forbids — use the withEnv() helper that restores on
exit, same as the rest of this file. Behavior identical; guard green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
co-authored by
Claude Fable 5
Claude Fable 5
parent
4ee530f3c5
commit
a81f7e05e8
@@ -104,8 +104,9 @@ writing or reviewing an operation, consult `src/core/operations.ts` for the cont
|
||||
## Before shipping
|
||||
|
||||
Easiest path: `bun run ci:local` runs the full CI gate inside Docker (gitleaks,
|
||||
unit tests with `DATABASE_URL` unset, then all 29 E2E files sequentially against a
|
||||
fresh pgvector container) and tears down. Use `bun run ci:local:diff` for the
|
||||
guards + typecheck, then 4-shard parallel unit + E2E against four pgvector
|
||||
containers plus a transaction-mode PgBouncer; unit phase keeps `DATABASE_URL`
|
||||
unset) and tears down. Use `bun run ci:local:diff` for the
|
||||
diff-aware subset during fast iteration on a focused branch. Requires Docker
|
||||
(Docker Desktop / OrbStack / Colima) and `gitleaks` (`brew install gitleaks`).
|
||||
|
||||
|
||||
@@ -2,6 +2,27 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.42.43.0] - 2026-06-12
|
||||
|
||||
**The brain now volunteers relevant pages instead of waiting to be asked (gbrain#2095).** Retrieval used to be pull-only: a deep session could run for hours with zero brain contributions — not because the brain had nothing, but because nothing prompted the agent to ask, and pages stored under coined names were missed by literal-string queries. Push-based context inverts that, on three channels sharing one zero-LLM, confidence-gated core: the ambient retrieval reflex now reads the last few conversation turns (an entity your assistant introduced two turns ago resolves on the "what did she invest in?" follow-up), a new `volunteer_context` operation gives any agent a per-turn volunteer surface over CLI stdin or MCP, and `gbrain watch` streams volunteered pages as a transcript flows through it.
|
||||
|
||||
Every volunteered page carries an honest confidence (alias match 0.9, exact title 0.8, slug-suffix 0.6, small boosts for repeated or newest-turn mentions; default gate 0.7) and a one-line rationale. A feedback loop closes the tuning circle: volunteered pages are logged, "used" is derived from whether the page actually got retrieved afterwards, and `gbrain volunteer-context --stats` reports per-arm precision (labeled approximate, because the retrieval signal is throttled). Suppression learned the difference between a page that was actually surfaced and one merely mentioned — under windowing only a surfaced page is held back, so prior-turn mentions can't silence themselves.
|
||||
|
||||
This release also lands on top of v0.42.42.0's exit-contract work as a strict superset: the transaction-mode pooler topology behind three consecutive teardown waves is now reproduced in the local CI gate (a real pooler service + an end-to-end teardown test), the exit-verdict sweep is completed across every command surface (notably `gbrain doctor`, whose FAIL verdict could still report exit 0), and a structural guard makes the next raw exit-code write fail in CI instead of silently reporting success on failure.
|
||||
|
||||
### Added
|
||||
- **`volunteer_context` operation** (CLI: `gbrain volunteer-context`, MCP tool) — pipe recent turns in (`user:` / `assistant:` prefixed lines, or plain text), get confidence-gated page pointers with rationales and synopses out. `--stats` returns the volunteered-vs-used precision summary. Per-call knobs: `max_pages`, `min_confidence`, `session_id`/`turn` attribution.
|
||||
- **`gbrain watch`** — the streaming push transport: feed a transcript on stdin, volunteered pages stream out (`--json` for JSONL), each slug at most once per session. Piped input exits cleanly at end-of-input; interactive sessions run until Ctrl-C.
|
||||
- **Rolling-window retrieval reflex** — the ambient channel extracts entities from the last 4 turns (configurable via `retrieval_reflex_window_turns` / `GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS`; 1 restores the previous single-turn behavior). Assistant-introduced entities and named-antecedent follow-ups now surface pointers with zero agent-initiated queries.
|
||||
- **Volunteered-context feedback log** — volunteered pages are recorded best-effort (channel, arm, confidence, optional session/turn) with 90-day retention handled by the nightly cycle; rationales are deterministic templates, never raw conversation text. Synopses always strip the takes/facts privacy fences before reaching a prompt.
|
||||
- **Transaction-mode pooler in the local CI gate** — `bun run ci:local` now runs a real transaction-pooling service in front of Postgres with an end-to-end teardown test, so the bug class behind gbrain#1972/#2015/#2084 is reproducible before it ships, not after.
|
||||
|
||||
### Fixed
|
||||
- **`gbrain doctor` exits 1 on FAIL again on every engine.** Its verdict write predated the v0.42.42.0 exit-verdict channel and was being silently zeroed; swept, plus a structural test that fails CI on the next raw exit-code write anywhere in the CLI.
|
||||
- **Reflex pointer suppression under multi-turn windows** distinguishes "page already surfaced" from "entity merely mentioned earlier" — without this, window extraction would have suppressed every prior-turn entity by construction.
|
||||
|
||||
### To take advantage of v0.42.43.0
|
||||
`gbrain upgrade` (applies the new feedback-log migration automatically). The wider reflex window is on by default for context-engine hosts — set `retrieval_reflex_window_turns: 1` in `~/.gbrain/config.json` to restore single-turn behavior. Agents without the context engine: call `volunteer_context` per turn (window in, pointers out), or pipe a transcript through `gbrain watch --json`. After a few days, `gbrain volunteer-context --stats` shows which resolution arms are earning their keep; raise or lower `min_confidence` accordingly.
|
||||
## [0.42.42.0] - 2026-06-12
|
||||
|
||||
**`gbrain query` no longer pays a flat 10-second exit tax on managed Postgres behind a transaction-mode pooler — and CLI exit codes finally tell the truth on PGLite.** On deployments where the pooler holds sockets open past the bounded pool drain (gbrain#2084, a residual of gbrain#1972), every query printed its results and then sat for 10 seconds until the force-exit banner fired. The cause was two-layered: the hard-deadline timer was armed *before* the operation handler, so a multi-second search on a large brain burned the teardown budget (and any operation slower than 10 seconds was silently killed mid-run with exit 0 and truncated output); and the CLI never exited explicitly on success — it waited for Bun's event loop to drain, which a stuck pooler socket can hold open forever.
|
||||
|
||||
@@ -38,7 +38,7 @@ mount, CEO-class with multiple team brains) and
|
||||
|
||||
## Architecture
|
||||
|
||||
Contract-first: `src/core/operations.ts` defines ~47 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`). CLI and MCP
|
||||
Contract-first: `src/core/operations.ts` defines ~90 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`; v0.42.43.0 adds `volunteer_context` — push-based context, see `docs/guides/push-context.md`). CLI and MCP
|
||||
server are both generated from this single source. Engine factory (`src/core/engine-factory.ts`)
|
||||
dynamically imports the configured engine (`'pglite'` or `'postgres'`). Skills are fat
|
||||
markdown files (tool-agnostic, work with both CLI and plugin contexts).
|
||||
@@ -97,6 +97,7 @@ detail on demand.)
|
||||
| any file in `src/` (what it does + its invariants) | `docs/architecture/KEY_FILES.md` — find the file's entry |
|
||||
| search / ranking / hybrid / retrieval | `docs/architecture/RETRIEVAL.md` + the `search/*` entries in `KEY_FILES.md` |
|
||||
| search modes / cost knobs | `docs/guides/search-modes.md` |
|
||||
| push-based context (volunteer/watch/reflex window) | `docs/guides/push-context.md` |
|
||||
| schema packs / page types / extraction | `docs/architecture/schema-packs.md`, `type-taxonomy.md`, `lens-packs.md` |
|
||||
| thin-client / remote MCP / cross-modal | `docs/architecture/thin-client.md` |
|
||||
| the CLI surface (commands + flags) | `gbrain --help` / `gbrain --tools-json`, plus the relevant `KEY_FILES.md` entry |
|
||||
|
||||
@@ -1,5 +1,65 @@
|
||||
# TODOS
|
||||
|
||||
## gbrain#2095 push-based context follow-ups (v0.43+)
|
||||
|
||||
Filed from the #2095 wave (volunteer_context op + reflex window + `gbrain watch`).
|
||||
Deliberately scoped OUT of v1 per the eng-review scope decision (success criteria
|
||||
are the bar). Plan + GSTACK REVIEW REPORT at
|
||||
`~/.claude/plans/system-instruction-you-are-working-cheerful-elephant.md`.
|
||||
|
||||
- [ ] **P3 — SSE/HTTP push channel via serve-http.** The op + `gbrain watch` cover
|
||||
pull-per-turn and stdin streaming; a serve-http SSE feed would push volunteered
|
||||
pages to remote agents without a local CLI. **Why:** thin-client/remote-MCP
|
||||
deployments get push too. **Cons:** async plumbing + auth scoping; no consumer
|
||||
wired today. **Where:** `src/commands/serve-http.ts` + `src/core/context/volunteer.ts`.
|
||||
**Blocked by:** a real consumer (revisit when one exists).
|
||||
- [ ] **P3 — policy skill + doctor check for push-context.** The ambient reflex
|
||||
needed doctor visibility because silent failure was invisible; volunteer is
|
||||
invoked-on-demand so v1 skipped it. If `volunteer-context --stats` adoption shows
|
||||
agents not discovering the surface, ship a `push-context` recipe (mirror
|
||||
`recipes/retrieval-reflex/`) + a doctor check reading the events table.
|
||||
**Where:** `recipes/`, `src/commands/doctor.ts`.
|
||||
- [ ] **P3 — structured `messages[]` param for volunteer_context.** v1 takes a
|
||||
string window (`user:`/`assistant:` prefixes) to avoid a dual-shape contract.
|
||||
If MCP callers accumulate parsing bugs, add a structured array param beside it.
|
||||
**Where:** `src/core/operations.ts:volunteer_context` + `src/core/context/volunteer.ts:parseWindow`.
|
||||
- [ ] **P3 — index shapes for the per-turn resolver query.** The arm-2 resolver
|
||||
(`retrieval-reflex.ts`: `lower(title) = ANY() OR slug = ANY() OR slug LIKE
|
||||
ANY('%/...')`) predates #2095 but now runs per turn on three channels
|
||||
(reflex window, volunteer_context, watch) federated across sources. Neither
|
||||
the leading-wildcard suffix arm nor `lower(title)` is index-served. If
|
||||
per-turn latency telemetry on large brains comes back hot: add
|
||||
`(source_id, lower(title))` btree + a reverse(slug) text_pattern_ops (or
|
||||
gin_trgm) index, or split the OR into three index-friendly queries.
|
||||
**Where:** `src/core/context/retrieval-reflex.ts`, migration.
|
||||
- [ ] **P3 — batch the volunteer-events pruner's first run after a long gap.**
|
||||
`purgeStaleVolunteerEvents` is one unbatched DELETE with a bare
|
||||
`volunteered_at` predicate (full scan; fine for a TTL-bounded table). Edge:
|
||||
a brain whose dream cycle was off for months could hit the pooler's ~2min
|
||||
statement_timeout on the first prune, get swallowed by the catch, and never
|
||||
make progress. If observed: id-batched chunks (`DELETE ... WHERE id IN
|
||||
(SELECT ... LIMIT 10000)` looped). **Where:**
|
||||
`src/core/context/volunteer-events.ts:purgeStaleVolunteerEvents`.
|
||||
- [ ] **P3 — route `gbrain watch` through the serve resolve-IPC on PGLite.**
|
||||
`watch` connects directly, so on a PGLite brain it monopolizes the single
|
||||
connection for its whole (potentially hours-long) session — a concurrent
|
||||
`gbrain serve` or any write path blocks on the lock until watch exits.
|
||||
WATCH_HELP documents the monopoly; the fix is an IPC rung in watch's
|
||||
resolver (reuse `resolveViaIpc` like the ambient reflex's ladder) so a
|
||||
running serve answers and watch never takes the lock. **Why:** watch +
|
||||
serve concurrently is the natural agent topology. **Where:**
|
||||
`src/commands/watch.ts`, `src/core/context/resolve-ipc.ts` (red-team RT2).
|
||||
- [ ] **P3 — capability/version gate for host-injected reflex resolvers.**
|
||||
Windowing switched the orchestrator's suppression request to 'slug-only';
|
||||
a host resolver built against the pre-window contract that still applies
|
||||
title-whole-word suppression silently self-suppresses every windowed
|
||||
entity. The contract is documented at `ResolveEntitiesFn` (reflex.ts), but
|
||||
nothing detects a stale host. Add a capability handshake (e.g. resolver
|
||||
advertises `supportsSuppressionModes`) and fall back to
|
||||
`window_turns: 1` semantics when absent. **Where:**
|
||||
`src/core/context/reflex.ts:ResolveEntitiesFn` + the OpenClaw plugin
|
||||
contract (red-team RT4).
|
||||
|
||||
## gbrain triage wave follow-ups (filed v0.42.41.0)
|
||||
|
||||
Deferred from the v0.42.41.0 fix wave (eng-reviewed as separate scope, not hotfixes).
|
||||
@@ -32,11 +92,13 @@ Filed from the #1981 ship (v0.42.39.0). Deliberately scoped OUT — the v1 extra
|
||||
is deterministic + precision-biased. See plan + GSTACK REVIEW REPORT at
|
||||
`~/.claude/plans/system-instruction-you-are-working-wild-yeti.md`.
|
||||
|
||||
- [ ] **P3 — broaden entity detection beyond proper-case ASCII.** The v1 extractor
|
||||
(`src/core/context/entity-salience.ts`) misses lowercase names, many non-Latin
|
||||
scripts, pronoun follow-ups ("what about her?"), and assistant-introduced entities.
|
||||
These need conversation state or an LLM pass. **Why:** higher recall on the read
|
||||
side. **Where:** `entity-salience.ts` + the orchestrator's `priorContextText`.
|
||||
- [ ] **P3 — broaden entity detection beyond proper-case ASCII.** The extractor
|
||||
(`src/core/context/entity-salience.ts`) misses lowercase names and many non-Latin
|
||||
scripts; these need an LLM pass or script-aware heuristics. **Why:** higher recall
|
||||
on the read side. **Where:** `entity-salience.ts`. *(Partially done by the #2095
|
||||
wave: `extractCandidatesFromWindow` now covers assistant-introduced entities and
|
||||
pronoun follow-ups whose antecedent was NAMED in the rolling window; true pronoun
|
||||
coreference for never-named antecedents remains with the LLM-pass idea.)*
|
||||
- [ ] **P3 — recall knob: optional fuzzy/prefix-expansion resolution.** The resolver
|
||||
(`src/core/context/retrieval-reflex.ts`) is exact-only (alias + title + slug-suffix)
|
||||
for precision. Revisit adding `resolveEntitySlug`'s trgm-fuzzy / prefix-expansion
|
||||
|
||||
@@ -85,6 +85,40 @@ services:
|
||||
volumes:
|
||||
- gbrain-ci-pg-data-4:/var/lib/postgresql/data
|
||||
|
||||
# v0.43 (#2084 / eng-review TD1): PgBouncer in TRANSACTION pooling mode
|
||||
# fronting postgres-1 — the production topology (Supabase direct :5432 +
|
||||
# pooled :6543) behind three consecutive pooler-teardown waves
|
||||
# (#1972 → #2015 → #2084) that CI could never reproduce.
|
||||
# test/e2e/pgbouncer-teardown.test.ts uses a DEDICATED database
|
||||
# (gbrain_pgbouncer) on postgres-1 so it never races shard 1's
|
||||
# TRUNCATE-based fixtures; pgbouncer's wildcard [databases] section
|
||||
# forwards any dbname to DB_HOST.
|
||||
pgbouncer:
|
||||
image: edoburu/pgbouncer:latest
|
||||
environment:
|
||||
DB_HOST: postgres-1
|
||||
DB_PORT: "5432"
|
||||
DB_USER: postgres
|
||||
DB_PASSWORD: postgres
|
||||
POOL_MODE: transaction
|
||||
# plain (CI-only): pg16 stores SCRAM verifiers, and pgbouncer can only
|
||||
# answer the server's SCRAM challenge when its userlist holds the
|
||||
# PLAINTEXT password — an md5-hashed userlist fails with
|
||||
# "server login failed: wrong password type".
|
||||
AUTH_TYPE: plain
|
||||
MAX_CLIENT_CONN: "200"
|
||||
DEFAULT_POOL_SIZE: "10"
|
||||
# gbrain's client sets statement_timeout + idle_in_transaction_session_timeout
|
||||
# as startup parameters (db.ts buildConnectionParams); the Supabase pooler
|
||||
# whitelists them, so this pooler must too or every connection is refused
|
||||
# before the teardown path is even reached.
|
||||
IGNORE_STARTUP_PARAMETERS: extra_float_digits,statement_timeout,idle_in_transaction_session_timeout,search_path
|
||||
ports:
|
||||
- "${GBRAIN_CI_PGBOUNCER_PORT:-6543}:5432"
|
||||
depends_on:
|
||||
postgres-1:
|
||||
condition: service_healthy
|
||||
|
||||
runner:
|
||||
image: oven/bun:1
|
||||
working_dir: /app
|
||||
@@ -97,6 +131,8 @@ services:
|
||||
condition: service_healthy
|
||||
postgres-4:
|
||||
condition: service_healthy
|
||||
pgbouncer:
|
||||
condition: service_started
|
||||
# No global DATABASE_URL — scripts/ci-local.sh sets per-shard URL via -e.
|
||||
# Unit phase explicitly unsets DATABASE_URL so test/e2e/* gracefully skip.
|
||||
volumes:
|
||||
|
||||
+10
-8
@@ -12,15 +12,17 @@ Before shipping (/ship) or reviewing (/review), always run the full test suite.
|
||||
Two equivalent paths:
|
||||
|
||||
**Path A — local CI gate (recommended, v0.23.1+):**
|
||||
- `bun run ci:local` runs the entire stack inside Docker: gitleaks (host), unit
|
||||
tests with `DATABASE_URL` unset, and all 29 E2E files sequentially against a
|
||||
fresh pgvector container. Stronger than PR CI's 2-file Tier 1 set; closer to
|
||||
what nightly Tier 1 catches. Spins up + tears down postgres automatically via
|
||||
`docker-compose.ci.yml`. Override the host port with
|
||||
`GBRAIN_CI_PG_PORT=5435 bun run ci:local` if 5434 collides.
|
||||
- `bun run ci:local` runs the entire stack inside Docker: gitleaks (host),
|
||||
guards + typecheck, then 4-shard parallel unit + E2E against four pgvector
|
||||
containers plus a transaction-mode PgBouncer service (unit phase keeps
|
||||
`DATABASE_URL` unset; `--no-shard` for the legacy sequential flow). Stronger
|
||||
than PR CI's 2-file Tier 1 set; closer to what nightly Tier 1 catches. Spins
|
||||
up + tears down postgres automatically via `docker-compose.ci.yml`. Override
|
||||
the host port with `GBRAIN_CI_PG_PORT=5435 bun run ci:local` if 5434 collides.
|
||||
- `bun run ci:local:diff` runs only the E2E files matched by the diff selector
|
||||
(`scripts/select-e2e.ts`), falling back to all 29 on unmapped src/ paths or
|
||||
schema/skills/package.json changes. Fast iteration during a focused branch.
|
||||
(`scripts/select-e2e.ts`), falling back to ALL E2E files on unmapped src/
|
||||
paths or schema/skills/package.json changes. Fast iteration during a focused
|
||||
branch.
|
||||
|
||||
**Path B — manual lifecycle (still supported):**
|
||||
- `bun test` — unit tests (no database required)
|
||||
|
||||
+10
-2
@@ -15,7 +15,7 @@ Seven test command tiers, each with a clear scope:
|
||||
| `bun run test:slow` | Just the `*.slow.test.ts` set (intentional cold-path correctness checks). | seconds-to-minutes | When touching slow-path code. |
|
||||
| `bun run test:serial` | Just the `*.serial.test.ts` set (cross-file-contention quarantine; one bun process per file for true module-registry isolation). | ~1s per quarantined file | Debugging a specific quarantined file. |
|
||||
| `bun run test:e2e` | Real Postgres E2E. Requires Docker + `DATABASE_URL`. Sequential. | ~5-10min | Pre-ship; nightly. |
|
||||
| `bun run check:all` | All 7 historical pre-checks (privacy + jsonb + progress + no-legacy-getconnection + trailing-newline + wasm + exports-count). Superset of `verify`. | ~10s | Local-only sweep. The 4 not in `verify` are nice-to-haves. |
|
||||
| `bun run check:all` | The historical pre-check scripts (22, chained sequentially in package.json). Overlaps `verify` heavily but is NOT a superset — `verify`'s `CHECKS` array in `scripts/run-verify-parallel.sh` (~30 entries incl. typecheck) is the authoritative gate; `check:all` keeps a few local-only extras (trailing-newline, exports-count, no-legacy-getconnection). | ~10s | Local-only sweep for the extras. |
|
||||
|
||||
### CI vs local: intentionally divergent file sets
|
||||
|
||||
@@ -126,6 +126,12 @@ Unit tests and what they cover:
|
||||
- `test/cli-finish-teardown.test.ts` — the #2084 teardown contract: `computeTeardownDeadlineMs` formula/floor/live-registry scaling + `GBRAIN_TEARDOWN_DEADLINE_MS` override (garbage/zero/negative values fall back to the formula); `finishCliTeardown` clean path (drain BEFORE disconnect, no exit, no warn), backstop on hung drain or disconnect (honors an errored op's exit code), throwing drain/disconnect warned + swallowed; the gbrain-owned verdict channel is immune to PGLite WASM `process.exitCode` writes; `flushThenExit` unit coverage with mocked streams (exits once after both stream callbacks, non-TTY aliveness grace, blocked-pipe guard, EPIPE-safe, `GBRAIN_FLUSH_GRACE_MS` override).
|
||||
- `test/flush-then-exit-harness.test.ts` — real spawned-Bun pipe semantics for `flushThenExit` (fixture: `test/fixtures/flush-then-exit-harness.ts`): a 4MB piped stdout payload arrives byte-complete with the exit code even with a late reader, small output survives exit with a concurrent reader, and the fence resolves promptly (wall time well under the guard + grace ceiling).
|
||||
- `test/cli-should-force-exit.test.ts` — `shouldForceExitAfterMain` daemon-survival gate: `serve` (stdio and `--http`) never force-exits, including with preceding global flags; op commands / empty / flag-only argv do; the #2084 case that space-separated global-flag VALUES can't fake a command (`--timeout 30s serve` resolves to the `serve` daemon, not a `30s` command).
|
||||
- `test/cli-exit-verdict-pin.test.ts` — #2084 structural class pin: greps `src/` so the NEXT raw `process.exitCode =` write fails CI (a raw write bypasses the gbrain-owned verdict channel and gets silently zeroed by the deliberate flush-exit — the bug that made doctor's FAIL path exit 0). Runtime variants live in `test/cli-finish-teardown.test.ts`; this is the review-time guard.
|
||||
- `test/cli-pipe-truncation.test.ts` — real-CLI pipe completeness (the #1959 incident class), implementation-agnostic: the actual CLI run the way agents run it (piped stdout) produces complete, parseable, byte-stable `--tools-json` output and exits deliberately, well under the teardown backstop. Synthetic flush-mechanism coverage stays in `test/flush-then-exit-harness.test.ts`.
|
||||
- `test/volunteer-context.test.ts` — push-based context core (#2095), hermetic in-memory PGLite: `parseWindow` lenient `user:`/`assistant:` parsing, multi-turn window extraction, confidence-gated volunteering (arm confidences, multi-turn/newest-turn boosts, `min_confidence` gate, max-pages cap), slug-only suppression, privacy (rationales are deterministic templates; synopses pass the takes/facts fence), and the approximate usage-stats join.
|
||||
- `test/watch-command.test.ts` — `gbrain watch` push transport (#2095): streaming loop, rolling window, session dedupe, `--json` JSONL shape, `channel: 'watch'` event logging, clean EOF return. Hermetic PGLite + injected line/write deps (no subprocess, no real stdin).
|
||||
- `test/watch-sigint.serial.test.ts` — `gbrain watch` SIGINT lifecycle against a real spawned CLI subprocess with a tmpdir brain. SERIAL: parallel unit shards flake on concurrent subprocess spawns (same rationale as `apply-migrations-pglite-spawn.serial.test.ts`).
|
||||
- `test/cli-format-volunteer.test.ts` — `formatResult`'s `volunteer_context` human rendering: pointer lines with confidence/arm/rationale, the empty-result message, the approximate stats summary.
|
||||
- `test/config.test.ts` — config redaction.
|
||||
- `test/files.test.ts` — MIME/hash.
|
||||
- `test/import-file.test.ts` — import pipeline.
|
||||
@@ -133,7 +139,7 @@ Unit tests and what they cover:
|
||||
- `test/file-migration.test.ts` — file migration.
|
||||
- `test/file-resolver.test.ts` — file resolution.
|
||||
- `test/import-resume.test.ts` — import checkpoints.
|
||||
- `test/migrate.test.ts` — migration: v8/v9 helper-btree-index SQL structural assertions; 1000-row wall-clock fixtures guarding the O(n²)→O(n log n) fix; v12/v13 SQL shape; `sqlFor` + `transaction:false` runner semantics; the `max_stalled DEFAULT 1` regression guard; v24 `sqlFor.pglite: ''` no-op assertion.
|
||||
- `test/migrate.test.ts` — migration: v8/v9 helper-btree-index SQL structural assertions; 1000-row wall-clock fixtures guarding the O(n²)→O(n log n) fix; v12/v13 SQL shape; `sqlFor` + `transaction:false` runner semantics; the `max_stalled DEFAULT 1` regression guard; v24 `sqlFor.pglite: ''` no-op assertion; v117 `context_volunteer_events` (named + idempotent entry, documented columns + both source-scoped indexes after `initSchema`, insert + 90-day `purgeStaleVolunteerEvents` round-trip).
|
||||
- `test/bootstrap.test.ts` — bootstrap contract: no-op on fresh install, idempotent across two `initSchema()` calls, no-op on modern brain that already has every probed column, full bootstrap path on a simulated legacy brain, fresh-install regression guard, legacy `links` shape coverage.
|
||||
- `test/schema-bootstrap-coverage.test.ts` — CI guard. `REQUIRED_BOOTSTRAP_COVERAGE` lists every forward reference in `PGLITE_SCHEMA_SQL`; the test fails loudly if `applyForwardReferenceBootstrap` skips one (extend both arrays when adding a column-with-index to the embedded schema blob). Also parses `src/core/migrate.ts` source text for every `ALTER TABLE ... ADD COLUMN` (top-level `sql:`, `sqlFor.{postgres,pglite}` overrides, AND handler-body `engine.runMigration(N, \`ALTER TABLE ...\`)`) and asserts each (table, column) pair is covered by the bootstrap OR by the schema blob's CREATE TABLE bodies — catching the column-only forward-reference class (e.g. `sources.archived`, `oauth_clients.source_id`) that a CREATE INDEX parser alone can't see. `parseBaseTableColumns` strips SQL line + block comments before identifying column names so commented-out lines don't hide adjacent columns.
|
||||
- `test/helpers/schema-diff.ts` + `test/helpers/schema-diff.test.ts` + `test/e2e/schema-drift.test.ts` — cross-engine schema parity gate. Helper exports pure `snapshotSchema(query)` / `diffSnapshots(pg, pglite, opts)` / `formatDiffForFailure(diff)` / `isCleanDiff(diff)` over a four-tuple per column (`data_type`, `udt_name`, `is_nullable`, `column_default`). E2E test spins up fresh PGLite + Postgres, runs `engine.initSchema()` on each, snapshots `information_schema.columns`, then diffs. 2-table allowlist (`files`, `file_migration_ledger`) — every other Postgres table must reach PGLite via `PGLITE_SCHEMA_SQL` or a migration's `sqlFor.pglite` branch. Sentinels for `oauth_clients`, `mcp_request_log`, `access_tokens`, `eval_candidates` give tighter blame messages. Skips without `DATABASE_URL`. Wired into `scripts/e2e-test-map.ts` so changes to `src/schema.sql`, `src/core/pglite-schema.ts`, or `src/core/migrate.ts` trigger it. The failure message names every drift with a paste-ready hint pointing at `src/core/pglite-schema.ts`.
|
||||
@@ -223,6 +229,8 @@ E2E tests live in `test/e2e/` and run against real Postgres+pgvector (require `D
|
||||
- `test/e2e/upgrade.test.ts` — check-update against real GitHub API (network required).
|
||||
- `test/e2e/minions-shell-pglite.test.ts` — PGLite `--follow` inline shell-job path (in-memory, no `DATABASE_URL` required) — the path the minion-orchestrator skill documents for dev use.
|
||||
- `test/e2e/pglite-cli-exit.serial.test.ts` — real spawned-CLI exit behavior on PGLite (in-memory, no `DATABASE_URL`): read commands (`search`/`get`/`query`) exit 0 promptly; CLI_ONLY `capture` exits clean and frees the single-writer lock; the `#2084` describes pin every swept disconnect site — a failed op exits 1 with the error on stderr, and the dashboard, read-only-timeout, doctor, and `dream --dry-run` paths all exit with no force-exit banner.
|
||||
- `test/e2e/pgbouncer-teardown.test.ts` — PgBouncer TRANSACTION-mode teardown (#2084 / the #1972→#2015→#2084 class). Pins the bug CLASS, not timings: a CLI op against a txn-mode pooled URL exits 0 with intact stdout and does NOT ride the 10s hard-deadline backstop (the `engine.disconnect() did not return` banner is the smoking gun — pre-#2084 it printed on 100% of query-shaped ops). Gated by `GBRAIN_PGBOUNCER_URL` + `GBRAIN_PGBOUNCER_DIRECT_URL` (NOT `DATABASE_URL`) — set automatically by `bun run ci:local`'s `pgbouncer` compose service; skips gracefully elsewhere. Uses a DEDICATED `gbrain_pgbouncer` database so it never races the `gbrain_test` TRUNCATE fixtures.
|
||||
- `test/e2e/volunteer-context-postgres.test.ts` — `volunteer_context` on REAL Postgres (#2095; engine parity beyond the hermetic PGLite unit suite): resolution arms through the actual op handler, the fire-and-forget volunteer-event sink landing rows, the stats join, and the RLS pin that `context_volunteer_events` has ROW LEVEL SECURITY enabled (keeps the v35 auto-RLS event trigger honest for migration-created tables). `DATABASE_URL`-gated.
|
||||
- `test/e2e/openclaw-reference-compat.test.ts` — `check-resolvable` + `skillpack install` against a minimal AGENTS.md workspace fixture (`test/fixtures/openclaw-reference-minimal/`), regression guard for the OpenClaw deployment shape.
|
||||
- `test/e2e/search-swamp.test.ts` — reproduces the source-swamp case. Seeds a curated `originals/talks/article-outline-fat-code` page against two `<fork>/chat/` pages stuffed with the same multi-word phrase. Asserts the article wins keyword AND vector ranking, that `detail=high` lets the chat swamp re-surface, and that `source_id` passes through the two-stage CTE intact. PGLite in-memory.
|
||||
- `test/e2e/search-exclude.test.ts` — `test/` + `archive/` pages hidden by default, `include_slug_prefixes` opts back in, caller-supplied `exclude_slug_prefixes` adds to defaults. Both keyword and vector search paths.
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,79 @@
|
||||
# Push-based context (#2095, v0.42.43.0)
|
||||
|
||||
Retrieval used to be pull-only: the agent had to *know to ask* before the brain
|
||||
contributed anything. Push-based context inverts that — the brain volunteers
|
||||
relevant pages from the recent conversation, confidence-gated so push noise
|
||||
never becomes worse than pull silence.
|
||||
|
||||
Three channels share one zero-LLM core (`src/core/context/volunteer.ts`):
|
||||
|
||||
| Channel | Surface | When to use |
|
||||
|---|---|---|
|
||||
| `reflex` | automatic, inside the context engine | default-on for plugin hosts; nothing to call |
|
||||
| `op` | `gbrain volunteer-context` / MCP `volunteer_context` | agents without the plugin; one call per turn |
|
||||
| `watch` | `gbrain watch` | stream a transcript in, volunteered pages stream out |
|
||||
|
||||
## How it decides
|
||||
|
||||
1. **Extract** entities across the last N turns (capitalized runs, `@handles`),
|
||||
merged with recency / frequency / user-role salience. Assistant-introduced
|
||||
entities and "what did she invest in?" follow-ups whose antecedent was named
|
||||
in the window now resolve.
|
||||
2. **Resolve** through the alias table, exact titles, and slug suffixes — each
|
||||
arm carries an honest confidence: alias 0.9, exact title 0.8, slug-suffix 0.6,
|
||||
+0.05 when mentioned in ≥2 turns or the newest turn.
|
||||
3. **Gate** at `min_confidence` (default 0.7 — slug-suffix matches need an
|
||||
explicit lower gate), suppress pages already surfaced (slug-presence only),
|
||||
cap at 3 pages (hard cap 5).
|
||||
|
||||
## CLI
|
||||
|
||||
```bash
|
||||
# one-shot: pipe recent turns (oldest → newest)
|
||||
printf 'user: ask alice-example about the deal\nassistant: noted\nuser: what did she say?\n' \
|
||||
| gbrain volunteer-context
|
||||
|
||||
# streaming: volunteered pages print as the transcript flows
|
||||
some-transcript-feed | gbrain watch --json
|
||||
|
||||
# the feedback loop: how often were volunteered pages actually opened?
|
||||
gbrain volunteer-context --stats
|
||||
```
|
||||
|
||||
Stats are **approximate** by design: "used" means `pages.last_retrieved_at >
|
||||
volunteered_at` — the 5-minute last-retrieved throttle causes false negatives
|
||||
and unrelated reads of the same page cause false positives. Use the per-arm
|
||||
precision to tune `min_confidence`, not as an exact metric.
|
||||
|
||||
**PGLite + `gbrain watch`:** PGLite is single-connection, and watch holds its
|
||||
connection for the whole session — a concurrent `gbrain serve` or any write
|
||||
path blocks until watch exits. On a PGLite brain, run watch in bursts (piped
|
||||
input exits at EOF) or use the ambient reflex channel instead, which routes
|
||||
through a running serve's resolve socket rather than taking the lock. Routing
|
||||
watch through that same socket is a filed follow-up (TODOS.md). Postgres
|
||||
brains are unaffected.
|
||||
|
||||
## Config
|
||||
|
||||
| Key | Default | What it does |
|
||||
|---|---|---|
|
||||
| `retrieval_reflex_window_turns` | 4 | turns the ambient reflex extracts from; 1 = legacy current-turn-only (file/env plane: `GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS`) |
|
||||
| `retrieval_reflex` | true | the ambient channel's master switch |
|
||||
| `retrieval_reflex_max_pointers` | 3 | pointer cap per turn |
|
||||
|
||||
Per-call knobs: `max_pages` + `min_confidence` on both the op and `gbrain watch`
|
||||
(`--max-pages` / `--min-confidence`, plus `--window-turns` / `--source` on watch);
|
||||
on the op only: `prior_context` (text whose already-surfaced slugs are suppressed),
|
||||
`session_id` / `turn` attribution params (watch stamps its own per-session id and
|
||||
turn numbers in the feedback log), and `days` to size the `--stats` window.
|
||||
|
||||
## Storage + privacy
|
||||
|
||||
Volunteered pages log to `context_volunteer_events` (migration v117): slug,
|
||||
arm, confidence, channel, optional session/turn — the rationale is a
|
||||
deterministic template string, never raw conversation text. Event writes are
|
||||
best-effort (fire-and-forget, drained at CLI exit) — the log is a tuning signal,
|
||||
not an audit trail. Rows are pruned after 90 days by the dream cycle's purge
|
||||
phase. Synopses always strip the takes/facts fences — the same strip `get_page`
|
||||
applies to untrusted callers, applied unconditionally here so private fence rows
|
||||
never reach a prompt regardless of caller trust.
|
||||
+91
-3
@@ -117,8 +117,9 @@ writing or reviewing an operation, consult `src/core/operations.ts` for the cont
|
||||
## Before shipping
|
||||
|
||||
Easiest path: `bun run ci:local` runs the full CI gate inside Docker (gitleaks,
|
||||
unit tests with `DATABASE_URL` unset, then all 29 E2E files sequentially against a
|
||||
fresh pgvector container) and tears down. Use `bun run ci:local:diff` for the
|
||||
guards + typecheck, then 4-shard parallel unit + E2E against four pgvector
|
||||
containers plus a transaction-mode PgBouncer; unit phase keeps `DATABASE_URL`
|
||||
unset) and tears down. Use `bun run ci:local:diff` for the
|
||||
diff-aware subset during fast iteration on a focused branch. Requires Docker
|
||||
(Docker Desktop / OrbStack / Colima) and `gitleaks` (`brew install gitleaks`).
|
||||
|
||||
@@ -186,7 +187,7 @@ mount, CEO-class with multiple team brains) and
|
||||
|
||||
## Architecture
|
||||
|
||||
Contract-first: `src/core/operations.ts` defines ~47 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`). CLI and MCP
|
||||
Contract-first: `src/core/operations.ts` defines ~90 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`; v0.42.43.0 adds `volunteer_context` — push-based context, see `docs/guides/push-context.md`). CLI and MCP
|
||||
server are both generated from this single source. Engine factory (`src/core/engine-factory.ts`)
|
||||
dynamically imports the configured engine (`'pglite'` or `'postgres'`). Skills are fat
|
||||
markdown files (tool-agnostic, work with both CLI and plugin contexts).
|
||||
@@ -245,6 +246,7 @@ detail on demand.)
|
||||
| any file in `src/` (what it does + its invariants) | `docs/architecture/KEY_FILES.md` — find the file's entry |
|
||||
| search / ranking / hybrid / retrieval | `docs/architecture/RETRIEVAL.md` + the `search/*` entries in `KEY_FILES.md` |
|
||||
| search modes / cost knobs | `docs/guides/search-modes.md` |
|
||||
| push-based context (volunteer/watch/reflex window) | `docs/guides/push-context.md` |
|
||||
| schema packs / page types / extraction | `docs/architecture/schema-packs.md`, `type-taxonomy.md`, `lens-packs.md` |
|
||||
| thin-client / remote MCP / cross-modal | `docs/architecture/thin-client.md` |
|
||||
| the CLI surface (commands + flags) | `gbrain --help` / `gbrain --tools-json`, plus the relevant `KEY_FILES.md` entry |
|
||||
@@ -3372,6 +3374,92 @@ the bundled resolver lives at [`skills/RESOLVER.md`](../../skills/RESOLVER.md).
|
||||
|
||||
---
|
||||
|
||||
## docs/guides/push-context.md
|
||||
|
||||
Source: https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/push-context.md
|
||||
|
||||
# Push-based context (#2095, v0.42.43.0)
|
||||
|
||||
Retrieval used to be pull-only: the agent had to *know to ask* before the brain
|
||||
contributed anything. Push-based context inverts that — the brain volunteers
|
||||
relevant pages from the recent conversation, confidence-gated so push noise
|
||||
never becomes worse than pull silence.
|
||||
|
||||
Three channels share one zero-LLM core (`src/core/context/volunteer.ts`):
|
||||
|
||||
| Channel | Surface | When to use |
|
||||
|---|---|---|
|
||||
| `reflex` | automatic, inside the context engine | default-on for plugin hosts; nothing to call |
|
||||
| `op` | `gbrain volunteer-context` / MCP `volunteer_context` | agents without the plugin; one call per turn |
|
||||
| `watch` | `gbrain watch` | stream a transcript in, volunteered pages stream out |
|
||||
|
||||
## How it decides
|
||||
|
||||
1. **Extract** entities across the last N turns (capitalized runs, `@handles`),
|
||||
merged with recency / frequency / user-role salience. Assistant-introduced
|
||||
entities and "what did she invest in?" follow-ups whose antecedent was named
|
||||
in the window now resolve.
|
||||
2. **Resolve** through the alias table, exact titles, and slug suffixes — each
|
||||
arm carries an honest confidence: alias 0.9, exact title 0.8, slug-suffix 0.6,
|
||||
+0.05 when mentioned in ≥2 turns or the newest turn.
|
||||
3. **Gate** at `min_confidence` (default 0.7 — slug-suffix matches need an
|
||||
explicit lower gate), suppress pages already surfaced (slug-presence only),
|
||||
cap at 3 pages (hard cap 5).
|
||||
|
||||
## CLI
|
||||
|
||||
```bash
|
||||
# one-shot: pipe recent turns (oldest → newest)
|
||||
printf 'user: ask alice-example about the deal\nassistant: noted\nuser: what did she say?\n' \
|
||||
| gbrain volunteer-context
|
||||
|
||||
# streaming: volunteered pages print as the transcript flows
|
||||
some-transcript-feed | gbrain watch --json
|
||||
|
||||
# the feedback loop: how often were volunteered pages actually opened?
|
||||
gbrain volunteer-context --stats
|
||||
```
|
||||
|
||||
Stats are **approximate** by design: "used" means `pages.last_retrieved_at >
|
||||
volunteered_at` — the 5-minute last-retrieved throttle causes false negatives
|
||||
and unrelated reads of the same page cause false positives. Use the per-arm
|
||||
precision to tune `min_confidence`, not as an exact metric.
|
||||
|
||||
**PGLite + `gbrain watch`:** PGLite is single-connection, and watch holds its
|
||||
connection for the whole session — a concurrent `gbrain serve` or any write
|
||||
path blocks until watch exits. On a PGLite brain, run watch in bursts (piped
|
||||
input exits at EOF) or use the ambient reflex channel instead, which routes
|
||||
through a running serve's resolve socket rather than taking the lock. Routing
|
||||
watch through that same socket is a filed follow-up (TODOS.md). Postgres
|
||||
brains are unaffected.
|
||||
|
||||
## Config
|
||||
|
||||
| Key | Default | What it does |
|
||||
|---|---|---|
|
||||
| `retrieval_reflex_window_turns` | 4 | turns the ambient reflex extracts from; 1 = legacy current-turn-only (file/env plane: `GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS`) |
|
||||
| `retrieval_reflex` | true | the ambient channel's master switch |
|
||||
| `retrieval_reflex_max_pointers` | 3 | pointer cap per turn |
|
||||
|
||||
Per-call knobs: `max_pages` + `min_confidence` on both the op and `gbrain watch`
|
||||
(`--max-pages` / `--min-confidence`, plus `--window-turns` / `--source` on watch);
|
||||
on the op only: `prior_context` (text whose already-surfaced slugs are suppressed),
|
||||
`session_id` / `turn` attribution params (watch stamps its own per-session id and
|
||||
turn numbers in the feedback log), and `days` to size the `--stats` window.
|
||||
|
||||
## Storage + privacy
|
||||
|
||||
Volunteered pages log to `context_volunteer_events` (migration v117): slug,
|
||||
arm, confidence, channel, optional session/turn — the rationale is a
|
||||
deterministic template string, never raw conversation text. Event writes are
|
||||
best-effort (fire-and-forget, drained at CLI exit) — the log is a tuning signal,
|
||||
not an audit trail. Rows are pruned after 90 days by the dream cycle's purge
|
||||
phase. Synopses always strip the takes/facts fences — the same strip `get_page`
|
||||
applies to untrusted callers, applied unconditionally here so private fence rows
|
||||
never reach a prompt regardless of caller trust.
|
||||
|
||||
---
|
||||
|
||||
## docs/mcp/DEPLOY.md
|
||||
|
||||
Source: https://raw.githubusercontent.com/garrytan/gbrain/master/docs/mcp/DEPLOY.md
|
||||
|
||||
@@ -25,6 +25,7 @@ Repo: https://github.com/garrytan/gbrain
|
||||
- [docs/guides/minions-deployment.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/minions-deployment.md): Deploying the gbrain jobs worker: crontab + watchdog, inline --follow, systemd/Procfile/fly.toml, upgrade checklist.
|
||||
- [docs/guides/quiet-hours.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/quiet-hours.md): Notification hold + timezone-aware delivery.
|
||||
- [docs/guides/scaling-skills.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/scaling-skills.md): Three-tier architecture for agents with 300+ skills: always-loaded, resolver-routed, and dormant. Per-turn token math, the v0.41.7.0 compact list-format resolver, and the `gbrain doctor` safety net. 306 skills, ~21K tokens freed per turn, zero capability loss.
|
||||
- [docs/guides/push-context.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/push-context.md): Push-based context: the brain volunteers confidence-gated pages from the rolling conversation window. Three channels (ambient reflex, volunteer_context op, gbrain watch), config knobs, and the volunteered-vs-used feedback loop.
|
||||
- [docs/mcp/DEPLOY.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/mcp/DEPLOY.md): MCP server deployment.
|
||||
|
||||
## AI providers
|
||||
|
||||
+1
-1
@@ -143,5 +143,5 @@
|
||||
"bun": ">=1.3.10"
|
||||
},
|
||||
"license": "MIT",
|
||||
"version": "0.42.42.0"
|
||||
"version": "0.42.43.0"
|
||||
}
|
||||
|
||||
+12
-2
@@ -196,7 +196,10 @@ SELECTED=$(bun run scripts/select-e2e.ts)
|
||||
if [ -z "$SELECTED" ]; then
|
||||
echo "[runner] selector emitted nothing (doc-only diff); skipping E2E."
|
||||
else
|
||||
DATABASE_URL=postgresql://postgres:postgres@postgres-1:5432/gbrain_test echo "$SELECTED" | xargs bash scripts/run-e2e.sh
|
||||
DATABASE_URL=postgresql://postgres:postgres@postgres-1:5432/gbrain_test \
|
||||
GBRAIN_PGBOUNCER_URL=postgresql://postgres:postgres@pgbouncer:5432/gbrain_pgbouncer \
|
||||
GBRAIN_PGBOUNCER_DIRECT_URL=postgresql://postgres:postgres@postgres-1:5432/gbrain_test \
|
||||
echo "$SELECTED" | xargs bash scripts/run-e2e.sh
|
||||
fi'
|
||||
else
|
||||
RUN_PHASES_CMD='echo "[runner] guards + typecheck"
|
||||
@@ -208,7 +211,10 @@ bun run typecheck
|
||||
echo "[runner] unit (unsharded, DATABASE_URL unset)"
|
||||
env -u DATABASE_URL bash scripts/run-unit-shard.sh
|
||||
echo "[runner] e2e (unsharded)"
|
||||
DATABASE_URL=postgresql://postgres:postgres@postgres-1:5432/gbrain_test bash scripts/run-e2e.sh'
|
||||
DATABASE_URL=postgresql://postgres:postgres@postgres-1:5432/gbrain_test \
|
||||
GBRAIN_PGBOUNCER_URL=postgresql://postgres:postgres@pgbouncer:5432/gbrain_pgbouncer \
|
||||
GBRAIN_PGBOUNCER_DIRECT_URL=postgresql://postgres:postgres@postgres-1:5432/gbrain_test \
|
||||
bash scripts/run-e2e.sh'
|
||||
fi
|
||||
else
|
||||
# Tier 1 sharded path. Each shard runs unit+E2E sequentially against its
|
||||
@@ -257,10 +263,14 @@ printf '%s\\n' 1 2 3 4 | xargs -P4 -I{} sh -c '
|
||||
if [ -s /tmp/e2e-selected.txt ]; then
|
||||
SHARD=\${shard}/4 \\
|
||||
DATABASE_URL=postgresql://postgres:postgres@postgres-\${shard}:5432/gbrain_test \\
|
||||
GBRAIN_PGBOUNCER_URL=postgresql://postgres:postgres@pgbouncer:5432/gbrain_pgbouncer \\
|
||||
GBRAIN_PGBOUNCER_DIRECT_URL=postgresql://postgres:postgres@postgres-1:5432/gbrain_test \\
|
||||
xargs -a /tmp/e2e-selected.txt bash scripts/run-e2e.sh >> \$log 2>&1
|
||||
else
|
||||
SHARD=\${shard}/4 \\
|
||||
DATABASE_URL=postgresql://postgres:postgres@postgres-\${shard}:5432/gbrain_test \\
|
||||
GBRAIN_PGBOUNCER_URL=postgresql://postgres:postgres@pgbouncer:5432/gbrain_pgbouncer \\
|
||||
GBRAIN_PGBOUNCER_DIRECT_URL=postgresql://postgres:postgres@postgres-1:5432/gbrain_test \\
|
||||
bash scripts/run-e2e.sh >> \$log 2>&1
|
||||
fi
|
||||
e2e_exit=\$?
|
||||
|
||||
@@ -151,6 +151,12 @@ export const SECTIONS: DocSection[] = [
|
||||
"Three-tier architecture for agents with 300+ skills: always-loaded, resolver-routed, and dormant. Per-turn token math, the v0.41.7.0 compact list-format resolver, and the `gbrain doctor` safety net. 306 skills, ~21K tokens freed per turn, zero capability loss.",
|
||||
path: "docs/guides/scaling-skills.md",
|
||||
},
|
||||
{
|
||||
title: "docs/guides/push-context.md",
|
||||
description:
|
||||
"Push-based context: the brain volunteers confidence-gated pages from the rolling conversation window. Three channels (ambient reflex, volunteer_context op, gbrain watch), config knobs, and the volunteered-vs-used feedback loop.",
|
||||
path: "docs/guides/push-context.md",
|
||||
},
|
||||
{
|
||||
title: "docs/mcp/DEPLOY.md",
|
||||
description: "MCP server deployment.",
|
||||
|
||||
+56
-12
@@ -24,6 +24,7 @@ import type { GBrainConfig } from './core/config.ts';
|
||||
import type { AIGatewayConfig } from './core/ai/types.ts';
|
||||
import type { BrainEngine } from './core/engine.ts';
|
||||
import { operations, OperationError } from './core/operations.ts';
|
||||
import { formatVolunteeredPage } from './core/context/volunteer.ts';
|
||||
import type { Operation, OperationContext } from './core/operations.ts';
|
||||
import { shouldForceExitAfterMain, finishCliTeardown, flushThenExit, currentExitCode, setCliExitVerdict } from './core/cli-force-exit.ts';
|
||||
import { serializeMarkdown } from './core/markdown.ts';
|
||||
@@ -43,7 +44,7 @@ for (const op of operations) {
|
||||
}
|
||||
|
||||
// CLI-only commands that bypass the operation layer
|
||||
const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade']);
|
||||
const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'watch']);
|
||||
// CLI-only commands whose handlers print their own --help text. These are
|
||||
// excluded from the generic short-circuit so detailed per-command and
|
||||
// per-subcommand usage stays reachable.
|
||||
@@ -67,6 +68,8 @@ const CLI_ONLY_SELF_HELP = new Set([
|
||||
'capture',
|
||||
// v0.42 self-upgrade ships its own usage (flags + the agent-skill story).
|
||||
'self-upgrade',
|
||||
// v0.43 (#2095): watch ships WATCH_HELP (flags + the stdin-turn protocol).
|
||||
'watch',
|
||||
// v0.37 fix wave (Lane D.4 + CDX2-12): sync's --no-embed flag was
|
||||
// unreachable via help because the dispatcher's generic CLI-only
|
||||
// short-circuit fired before runSync could print its own usage block.
|
||||
@@ -802,8 +805,27 @@ async function makeContext(engine: BrainEngine, params: Record<string, unknown>)
|
||||
};
|
||||
}
|
||||
|
||||
function formatResult(opName: string, result: unknown): string {
|
||||
// Exported for tests (same import-safety contract as cliAliases/printOpHelp).
|
||||
export function formatResult(opName: string, result: unknown): string {
|
||||
switch (opName) {
|
||||
case 'volunteer_context': {
|
||||
const r = result as any;
|
||||
// Stats mode (the feedback loop).
|
||||
if (r && r.approximate === true && Array.isArray(r.by_arm)) {
|
||||
const lines = [
|
||||
`volunteered-context precision — last ${r.days} day(s) (${r.note})`,
|
||||
`total: ${r.total_volunteered} volunteered, ${r.total_used} used`,
|
||||
];
|
||||
for (const a of r.by_arm) {
|
||||
lines.push(` ${a.match_arm}/${a.channel}: ${a.used}/${a.volunteered} used (precision ${a.precision})`);
|
||||
}
|
||||
if (!r.by_arm.length) lines.push(' (no volunteer events in the window)');
|
||||
return lines.join('\n') + '\n';
|
||||
}
|
||||
const pages = (r?.pages ?? []) as any[];
|
||||
if (!pages.length) return 'Nothing volunteered (no entity cleared the confidence gate).\n';
|
||||
return pages.map((p) => formatVolunteeredPage(p)).join('\n') + '\n';
|
||||
}
|
||||
case 'get_page': {
|
||||
const r = result as any;
|
||||
if (r.error === 'ambiguous_slug') {
|
||||
@@ -925,6 +947,9 @@ function formatResult(opName: string, result: unknown): string {
|
||||
const THIN_CLIENT_REFUSED_COMMANDS = new Set([
|
||||
'sync', 'embed', 'extract', 'extract-conversation-facts', 'enrich', 'migrate', 'apply-migrations',
|
||||
'repair-jsonb', 'orphans', 'integrity', 'serve',
|
||||
// v0.43 (#2095): watch streams against a LOCAL engine; thin clients get
|
||||
// the volunteer_context MCP op instead.
|
||||
'watch',
|
||||
// v0.31.1 (CDX-2 op coverage matrix): more local-only commands
|
||||
'dream', 'transcripts', 'storage',
|
||||
// v0.31.1 CDX-2 audit: takes/sources have multiple subcommands; some
|
||||
@@ -1156,11 +1181,14 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
}
|
||||
if (command === 'friction') {
|
||||
const { runFriction } = await import('./commands/friction.ts');
|
||||
process.exit(runFriction(args));
|
||||
// #2084 inner-exit sweep: verdict + return so teardown + the flush seam run.
|
||||
setCliExitVerdict(runFriction(args));
|
||||
return;
|
||||
}
|
||||
if (command === 'claw-test') {
|
||||
const { runClawTest } = await import('./commands/claw-test.ts');
|
||||
process.exit(await runClawTest(args));
|
||||
setCliExitVerdict(await runClawTest(args));
|
||||
return;
|
||||
}
|
||||
if (command === 'report') {
|
||||
const { runReport } = await import('./commands/report.ts');
|
||||
@@ -1269,7 +1297,7 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
execSync(`bash "${scriptPath}"`, { stdio: 'inherit', env: { ...process.env } });
|
||||
} catch (e: any) {
|
||||
// Non-zero exit = some tests failed (exit code = failure count)
|
||||
process.exit(e.status ?? 1);
|
||||
setCliExitVerdict(e.status ?? 1);
|
||||
}
|
||||
return;
|
||||
}
|
||||
@@ -1318,7 +1346,8 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
// The handler self-configures the AI gateway from loadConfig() + process.env.
|
||||
if (command === 'eval' && args[0] === 'cross-modal') {
|
||||
const { runEvalCrossModal } = await import('./commands/eval-cross-modal.ts');
|
||||
process.exit(await runEvalCrossModal(args.slice(1)));
|
||||
setCliExitVerdict(await runEvalCrossModal(args.slice(1)));
|
||||
return;
|
||||
}
|
||||
|
||||
// v0.32 EXP-5 (codex review #10): `eval takes-quality replay <receipt>`
|
||||
@@ -1329,7 +1358,8 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
// engine-required path below.
|
||||
if (command === 'eval' && args[0] === 'takes-quality' && args[1] === 'replay') {
|
||||
const { runReplayNoBrain } = await import('./commands/eval-takes-quality.ts');
|
||||
process.exit(await runReplayNoBrain(args.slice(2)));
|
||||
setCliExitVerdict(await runReplayNoBrain(args.slice(2)));
|
||||
return;
|
||||
}
|
||||
|
||||
// v0.28.8: longmemeval brings its own in-memory PGLite. Bypassing
|
||||
@@ -1361,7 +1391,8 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
// gate runs on machines with no `~/.gbrain/config.json`.
|
||||
if (command === 'eval' && args[0] === 'conversation-parser') {
|
||||
const { runEvalConversationParser } = await import('./commands/eval-conversation-parser.ts');
|
||||
process.exit(await runEvalConversationParser(args.slice(1)));
|
||||
setCliExitVerdict(await runEvalConversationParser(args.slice(1)));
|
||||
return;
|
||||
}
|
||||
|
||||
// v0.41.13.0: `gbrain conversation-parser list-builtins | validate
|
||||
@@ -1389,7 +1420,8 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
const cfgPre = loadConfig();
|
||||
if (isThinClient(cfgPre)) {
|
||||
const { runEvalWhoknows } = await import('./commands/eval-whoknows.ts');
|
||||
process.exit(await runEvalWhoknows(null, args.slice(1)));
|
||||
setCliExitVerdict(await runEvalWhoknows(null, args.slice(1)));
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1403,7 +1435,8 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
if (cfgPre && isThinClient(cfgPre)) {
|
||||
const { runStatus } = await import('./commands/status.ts');
|
||||
const result = await runStatus(null, args);
|
||||
process.exit(result.exitCode);
|
||||
setCliExitVerdict(result.exitCode);
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1716,8 +1749,8 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
case 'status': {
|
||||
const { runStatus } = await import('./commands/status.ts');
|
||||
const result = await runStatus(engine, args);
|
||||
process.exit(result.exitCode);
|
||||
// eslint-disable-next-line no-unreachable
|
||||
// #2084 inner-exit sweep: a mid-switch exit skips the finally teardown.
|
||||
setCliExitVerdict(result.exitCode);
|
||||
break;
|
||||
}
|
||||
// v0.38 — Capture: single human-facing entrypoint for ingestion.
|
||||
@@ -1887,6 +1920,15 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
await runQuarantine(engine, args);
|
||||
break;
|
||||
}
|
||||
case 'watch': {
|
||||
// v0.43 (#2095): push-based context transport. Blocks in the stdin
|
||||
// iteration (interactive stays alive; piped exits at EOF), then the
|
||||
// finally below runs finishCliTeardown (volunteer events drain with
|
||||
// every other sink) and the import.meta.main seam flush-exits.
|
||||
const { runWatch } = await import('./commands/watch.ts');
|
||||
await runWatch(engine, args);
|
||||
break;
|
||||
}
|
||||
case 'storage': {
|
||||
const { runStorage } = await import('./commands/storage.ts');
|
||||
await runStorage(engine, args);
|
||||
@@ -2267,6 +2309,8 @@ ADMIN
|
||||
--public-url URL Public issuer URL (required behind proxy/tunnel)
|
||||
connect <mcp-url> --token <t> Wire Claude Code to a remote gbrain (bearer token)
|
||||
[--install] [--json] Print the paste-ready command, or --install to run it
|
||||
watch [--json] Push-based context: pipe conversation turns in,
|
||||
volunteered brain pages stream out (#2095)
|
||||
call <tool> '<json>' Raw tool invocation
|
||||
version Version info
|
||||
--tools-json Tool discovery (JSON)
|
||||
|
||||
@@ -0,0 +1,190 @@
|
||||
/**
|
||||
* v0.43 (#2095) — `gbrain watch`: the push transport for push-based context.
|
||||
*
|
||||
* Reads conversation turns from stdin AS THEY ARRIVE (plain text = a user
|
||||
* turn; `user:` / `assistant:` prefixed lines set the role), maintains a
|
||||
* rolling window in-process, and volunteers confidence-gated brain pages to
|
||||
* stdout after every turn. The consumer pipes its transcript in and reads
|
||||
* volunteered pointers out — no per-entity CLI round-trips.
|
||||
*
|
||||
* Lifecycle: BLOCKS in the stdin iteration (like `gbrain jobs work`), so an
|
||||
* interactive TTY session stays alive until Ctrl-C / Ctrl-D and piped input
|
||||
* exits at EOF. Either way the handler RETURNS, the CLI_ONLY finally runs
|
||||
* finishCliTeardown (volunteer events bank before teardown), and the
|
||||
* entrypoint flush-exit ends the process deliberately — which is exactly why
|
||||
* `watch` is NOT in DAEMON_COMMANDS: it never returns from main() while work
|
||||
* is still running. SIGINT closes the stream and flows through the same
|
||||
* drain path instead of killing mid-write.
|
||||
*
|
||||
* Session dedupe: a slug is volunteered at most once per watch session —
|
||||
* already-pushed slugs ride VolunteerOpts.excludeSlugs, skipped inside the
|
||||
* core's pointer loop BEFORE the confidence gate and maxPages cap (O(1)
|
||||
* membership; a post-call filter would let a recurring entity starve new
|
||||
* pages out of the cap — red-team finding).
|
||||
*
|
||||
* PGLite note: watch holds the single-writer engine for the whole session,
|
||||
* so on the default engine it cannot run concurrently with `gbrain serve`
|
||||
* (or any other gbrain process) against the same brain — run it against
|
||||
* Postgres, or stop serve first. Routing watch through the serve resolve-IPC
|
||||
* socket (like the ambient reflex) is a filed follow-up.
|
||||
*/
|
||||
|
||||
import { createInterface } from 'node:readline';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import {
|
||||
volunteerContext,
|
||||
formatVolunteeredPage,
|
||||
TURN_PREFIX_RE,
|
||||
VOLUNTEER_DEFAULT_MAX_PAGES,
|
||||
VOLUNTEER_DEFAULT_MIN_CONFIDENCE,
|
||||
} from '../core/context/volunteer.ts';
|
||||
import type { WindowTurn } from '../core/context/entity-salience.ts';
|
||||
import { DEFAULT_WINDOW_TURNS, windowTurnCount } from '../core/context/reflex.ts';
|
||||
import { loadConfig } from '../core/config.ts';
|
||||
import { logVolunteerEventsFireAndForget, volunteerEventRowsFrom } from '../core/context/volunteer-events.ts';
|
||||
|
||||
export const WATCH_HELP = `gbrain watch — push-based context: volunteer brain pages per conversation turn (#2095)
|
||||
|
||||
Reads turns from stdin (one per line; 'user:' / 'assistant:' prefixes set the
|
||||
role, unprefixed lines are user turns) and prints confidence-gated page
|
||||
pointers with rationales after each turn. A slug is volunteered at most once
|
||||
per session. Piped input exits at EOF; interactive sessions exit on Ctrl-C.
|
||||
|
||||
Usage:
|
||||
some-transcript-feed | gbrain watch [--json]
|
||||
gbrain watch # interactive: type turns, Ctrl-C to end
|
||||
|
||||
PGLite brains: watch holds the single-writer engine for the whole session —
|
||||
it cannot run alongside \`gbrain serve\` (or other gbrain processes) against
|
||||
the same brain. Use a Postgres brain for concurrent access.
|
||||
|
||||
Flags:
|
||||
--json JSONL output (one volunteered page per line)
|
||||
--window-turns N rolling extraction window (default ${DEFAULT_WINDOW_TURNS})
|
||||
--max-pages N max pages volunteered per turn (default ${VOLUNTEER_DEFAULT_MAX_PAGES}, cap 5)
|
||||
--min-confidence X confidence gate 0..1 (default ${VOLUNTEER_DEFAULT_MIN_CONFIDENCE})
|
||||
--source <id> source scope (defaults to the canonical 6-tier resolution)
|
||||
--help this text
|
||||
`;
|
||||
|
||||
function numFlag(args: string[], flag: string): number | undefined {
|
||||
const i = args.indexOf(flag);
|
||||
if (i < 0 || i + 1 >= args.length) return undefined;
|
||||
const n = Number(args[i + 1]);
|
||||
return Number.isFinite(n) ? n : undefined;
|
||||
}
|
||||
|
||||
function strFlag(args: string[], flag: string): string | undefined {
|
||||
const i = args.indexOf(flag);
|
||||
return i >= 0 && i + 1 < args.length ? args[i + 1] : undefined;
|
||||
}
|
||||
|
||||
export interface WatchIoDeps {
|
||||
/** Injected line source for tests (defaults to readline over stdin). */
|
||||
lines?: AsyncIterable<string>;
|
||||
write?: (s: string) => void;
|
||||
isTTY?: boolean;
|
||||
}
|
||||
|
||||
export async function runWatch(engine: BrainEngine, args: string[], deps: WatchIoDeps = {}): Promise<void> {
|
||||
if (args.includes('--help') || args.includes('-h')) {
|
||||
(deps.write ?? ((s: string) => process.stdout.write(s)))(WATCH_HELP);
|
||||
return;
|
||||
}
|
||||
|
||||
const json = args.includes('--json');
|
||||
// --window-turns wins; otherwise the same config knob the ambient reflex
|
||||
// honors (retrieval_reflex_window_turns, default 4) applies here too.
|
||||
// Hard cap 64: an unbounded window re-scans every retained turn on every
|
||||
// turn over an hours-long session — the cost class the priorContext fix
|
||||
// removed, reintroduced via a config typo (red-team finding).
|
||||
const windowTurns = Math.min(
|
||||
64,
|
||||
Math.max(1, Math.floor(numFlag(args, '--window-turns') ?? windowTurnCount(loadConfig()))),
|
||||
);
|
||||
const maxPages = numFlag(args, '--max-pages');
|
||||
const minConfidence = numFlag(args, '--min-confidence');
|
||||
const write = deps.write ?? ((s: string) => process.stdout.write(s));
|
||||
const isTTY = deps.isTTY ?? Boolean(process.stdin.isTTY);
|
||||
|
||||
const { resolveSourceId } = await import('../core/source-resolver.ts');
|
||||
const sourceId = await resolveSourceId(engine, strFlag(args, '--source') ?? null, process.cwd());
|
||||
const sourceIds = [sourceId];
|
||||
const sessionId = `watch-${process.pid}-${Date.now().toString(36)}`;
|
||||
|
||||
if (!deps.lines) {
|
||||
// The ready line doubles as a machine-readable readiness signal for
|
||||
// scripted consumers (and the SIGINT lifecycle test): engine + source
|
||||
// resolution are done, the stdin loop starts next.
|
||||
process.stderr.write(
|
||||
isTTY
|
||||
? `[watch] interactive session ${sessionId} ready — type turns ('assistant: ...' to set role), Ctrl-C to end\n`
|
||||
: `[watch] session ${sessionId} ready\n`,
|
||||
);
|
||||
}
|
||||
|
||||
const rl = deps.lines
|
||||
? null
|
||||
: createInterface({ input: process.stdin, crlfDelay: Infinity });
|
||||
const lines: AsyncIterable<string> = deps.lines ?? (rl as AsyncIterable<string>);
|
||||
|
||||
// SIGINT closes the stream so the for-await ends and the normal
|
||||
// drain-then-exit path runs (never a mid-write kill).
|
||||
const onSigint = () => {
|
||||
rl?.close();
|
||||
};
|
||||
process.on('SIGINT', onSigint);
|
||||
|
||||
const window: WindowTurn[] = [];
|
||||
const pushedSlugs = new Set<string>(); // session dedupe (slug-only suppression input)
|
||||
let turnNo = 0;
|
||||
|
||||
try {
|
||||
for await (const rawLine of lines) {
|
||||
const line = rawLine.replace(/\r$/, '');
|
||||
if (!line.trim()) continue;
|
||||
const m = TURN_PREFIX_RE.exec(line);
|
||||
const turn: WindowTurn = m
|
||||
? { role: m[1].toLowerCase() as WindowTurn['role'], text: (m[2] ?? '').trim() }
|
||||
: { role: 'user', text: line.trim() };
|
||||
if (!turn.text) continue;
|
||||
turnNo++;
|
||||
window.push(turn);
|
||||
if (window.length > windowTurns) window.splice(0, window.length - windowTurns);
|
||||
|
||||
let pages;
|
||||
try {
|
||||
pages = await volunteerContext(engine, [...window], {
|
||||
sourceIds,
|
||||
maxPages,
|
||||
minConfidence,
|
||||
// Session dedupe: skipped inside the core BEFORE the gate + cap
|
||||
// (O(1) per pointer) so a recurring slug can't starve new pages.
|
||||
excludeSlugs: pushedSlugs,
|
||||
});
|
||||
} catch {
|
||||
continue; // fail-open per turn: a transient DB error never kills the stream
|
||||
}
|
||||
if (!pages.length) continue;
|
||||
|
||||
for (const p of pages) pushedSlugs.add(p.slug);
|
||||
logVolunteerEventsFireAndForget(
|
||||
engine,
|
||||
volunteerEventRowsFrom(pages, { channel: 'watch', session_id: sessionId, turn: turnNo }),
|
||||
);
|
||||
|
||||
if (json) {
|
||||
for (const p of pages) {
|
||||
write(JSON.stringify({ turn: turnNo, ...p }) + '\n');
|
||||
}
|
||||
} else {
|
||||
for (const p of pages) {
|
||||
write(formatVolunteeredPage(p) + '\n');
|
||||
}
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
process.off('SIGINT', onSigint);
|
||||
rl?.close();
|
||||
}
|
||||
}
|
||||
@@ -169,6 +169,14 @@ export interface GBrainConfig {
|
||||
retrieval_reflex?: boolean;
|
||||
/** Max pointers injected per turn (default 3). File-plane only. */
|
||||
retrieval_reflex_max_pointers?: number;
|
||||
/**
|
||||
* v0.43 (#2095) — how many recent turns the reflex extracts entities from
|
||||
* (default 4). 1 reproduces the legacy current-turn-only behavior (and the
|
||||
* legacy slug+title suppression). File-plane / env
|
||||
* (GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS) only — same plane as the other
|
||||
* reflex knobs.
|
||||
*/
|
||||
retrieval_reflex_window_turns?: number;
|
||||
embedding_image_ocr?: boolean;
|
||||
embedding_image_ocr_model?: string;
|
||||
|
||||
@@ -533,6 +541,10 @@ export function loadConfig(): GBrainConfig | null {
|
||||
...(process.env.GBRAIN_RETRIEVAL_REFLEX
|
||||
? { retrieval_reflex: !(process.env.GBRAIN_RETRIEVAL_REFLEX === 'false' || process.env.GBRAIN_RETRIEVAL_REFLEX === '0') }
|
||||
: {}),
|
||||
...(process.env.GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS &&
|
||||
Number.isFinite(Number(process.env.GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS))
|
||||
? { retrieval_reflex_window_turns: Number(process.env.GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS) }
|
||||
: {}),
|
||||
...(process.env.GBRAIN_REMOTE_CLIENT_SECRET && fileConfig?.remote_mcp
|
||||
? { remote_mcp: { ...fileConfig.remote_mcp, oauth_client_secret: process.env.GBRAIN_REMOTE_CLIENT_SECRET } }
|
||||
: {}),
|
||||
|
||||
@@ -179,6 +179,29 @@ function getLastUserText(messages: AgentMessage[]): string {
|
||||
return '';
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.43 (#2095): the rolling extraction window — the most recent
|
||||
* user/assistant turns (oldest → newest, current turn last), capped here at
|
||||
* a generous max; the reflex slices to its configured
|
||||
* retrieval_reflex_window_turns (default 4).
|
||||
*/
|
||||
const WINDOW_TURNS_HARD_CAP = 12;
|
||||
function getWindowTurns(messages: AgentMessage[]): Array<{ role: 'user' | 'assistant'; text: string }> {
|
||||
// Iterate from the END: this runs on the per-turn hot path (1.5s reflex
|
||||
// budget) and only the last 12 turns matter — flattening every content
|
||||
// block of a multi-hundred-turn session just to slice the tail would make
|
||||
// the cost grow with session length.
|
||||
const out: Array<{ role: 'user' | 'assistant'; text: string }> = [];
|
||||
for (let i = messages.length - 1; i >= 0 && out.length < WINDOW_TURNS_HARD_CAP; i--) {
|
||||
const m = messages[i];
|
||||
if (m?.role !== 'user' && m?.role !== 'assistant') continue;
|
||||
const text = messageText(m.content);
|
||||
if (!text) continue;
|
||||
out.push({ role: m.role, text });
|
||||
}
|
||||
return out.reverse();
|
||||
}
|
||||
|
||||
/**
|
||||
* Joined text of everything the agent has ALREADY seen — every message EXCEPT
|
||||
* the current turn (the last user message). Used for "already in context"
|
||||
@@ -649,6 +672,9 @@ export function createGBrainContextEngine(ctx: {
|
||||
workspaceDir,
|
||||
currentUserText: getLastUserText(messages),
|
||||
priorContextText: getPriorContextText(messages),
|
||||
// v0.43 (#2095): rolling window — assistant-introduced entities and
|
||||
// named-antecedent follow-ups from recent turns now resolve too.
|
||||
windowTurns: getWindowTurns(messages),
|
||||
resolveEntities: ctx.resolveEntities,
|
||||
});
|
||||
|
||||
|
||||
@@ -7,12 +7,15 @@
|
||||
* touching the brain, so it must be fast (one regex pass) and precision-biased:
|
||||
* a false candidate costs a wasted resolve and, worse, a misleading pointer.
|
||||
*
|
||||
* DELIBERATE v1 limits (documented, not bugs — see issue #1981 / eng-review):
|
||||
* DELIBERATE limits (documented, not bugs — see issue #1981 / eng-review):
|
||||
* - Proper-case + ASCII biased. Misses lowercase names ("garry") and many
|
||||
* non-Latin scripts.
|
||||
* - Current-user-message only. No pronoun follow-ups ("what about her?"), no
|
||||
* entities the assistant introduced.
|
||||
* These are TODOs, not v1 scope. Do NOT market this as full "human-like recall".
|
||||
* - extractCandidates is single-turn. The v0.43 (#2095) window layer
|
||||
* (extractCandidatesFromWindow) widens extraction across the last N turns
|
||||
* — assistant-introduced entities and "what about her?" follow-ups whose
|
||||
* antecedent was NAMED in the window now resolve; true pronoun
|
||||
* coreference (never-named antecedents) remains out of scope.
|
||||
* Do NOT market this as full "human-like recall".
|
||||
*
|
||||
* Resolution (alias/slug lookup) lives in retrieval-reflex.ts; this module only
|
||||
* decides WHAT to look up.
|
||||
@@ -172,3 +175,83 @@ export function extractCandidates(text: string): EntityCandidate[] {
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
// ── Rolling-window extraction (v0.43, #2095 push-based context) ──────────
|
||||
|
||||
export interface WindowTurn {
|
||||
role: 'user' | 'assistant';
|
||||
text: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* A window candidate with the salience metadata the volunteer layer's
|
||||
* confidence boost reads: how many turns mentioned it, whether the NEWEST
|
||||
* turn did, and whether the USER (vs only the assistant) ever said it.
|
||||
*/
|
||||
export interface WindowEntityCandidate extends EntityCandidate {
|
||||
/** Number of distinct turns that mentioned this candidate. */
|
||||
occurrences: number;
|
||||
/** Mentioned in the newest (last) turn of the window. */
|
||||
inNewestTurn: boolean;
|
||||
/** Mentioned in at least one USER turn (assistant-only mentions rank lower). */
|
||||
userMention: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract candidates across the last N turns (oldest → newest). Pure,
|
||||
* zero-LLM: runs the per-turn extractor on each turn and merges by the
|
||||
* normalizeAlias form. Ordering is salience-aware — recency of last mention,
|
||||
* cross-turn frequency, and a user-role boost — so when the merged set
|
||||
* exceeds MAX_CANDIDATES, the dropped tail is the stalest assistant-only
|
||||
* chatter, not the entity the user just named.
|
||||
*/
|
||||
export function extractCandidatesFromWindow(turns: WindowTurn[]): WindowEntityCandidate[] {
|
||||
if (!turns?.length) return [];
|
||||
interface WAcc extends WindowEntityCandidate {
|
||||
lastTurnIdx: number;
|
||||
order: number;
|
||||
}
|
||||
const acc = new Map<string, WAcc>();
|
||||
let order = 0;
|
||||
const lastIdx = turns.length - 1;
|
||||
|
||||
for (let i = 0; i < turns.length; i++) {
|
||||
const turn = turns[i];
|
||||
if (!turn?.text) continue;
|
||||
for (const c of extractCandidates(turn.text)) {
|
||||
const norm = normalizeAlias(c.query);
|
||||
if (!norm) continue;
|
||||
const existing = acc.get(norm);
|
||||
if (existing) {
|
||||
existing.occurrences += 1;
|
||||
existing.lastTurnIdx = i;
|
||||
existing.inNewestTurn = existing.inNewestTurn || i === lastIdx;
|
||||
if (turn.role === 'user' && !existing.userMention) {
|
||||
// First USER-said surface form beats an assistant-introduced one
|
||||
// for the display label.
|
||||
existing.display = c.display;
|
||||
existing.userMention = true;
|
||||
}
|
||||
} else {
|
||||
acc.set(norm, {
|
||||
display: c.display,
|
||||
query: c.query,
|
||||
occurrences: 1,
|
||||
lastTurnIdx: i,
|
||||
inNewestTurn: i === lastIdx,
|
||||
userMention: turn.role === 'user',
|
||||
order: order++,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Salience weight: recency dominates, then frequency, then user-role.
|
||||
// Deterministic tie-break on first-seen order.
|
||||
const weight = (c: WAcc) =>
|
||||
(c.lastTurnIdx + 1) / turns.length + Math.min(c.occurrences, 4) * 0.1 + (c.userMention ? 0.15 : 0);
|
||||
return Array.from(acc.values())
|
||||
.sort((a, b) => weight(b) - weight(a) || a.order - b.order)
|
||||
.slice(0, MAX_CANDIDATES)
|
||||
.map(({ lastTurnIdx: _l, order: _o, ...rest }) => rest);
|
||||
}
|
||||
|
||||
@@ -20,7 +20,12 @@ import { join } from 'node:path';
|
||||
import { mkdirSync, appendFileSync } from 'node:fs';
|
||||
import { loadConfig, type GBrainConfig } from '../config.ts';
|
||||
import type { BrainEngine } from '../engine.ts';
|
||||
import { extractCandidates, type EntityCandidate } from './entity-salience.ts';
|
||||
import {
|
||||
extractCandidates,
|
||||
extractCandidatesFromWindow,
|
||||
type EntityCandidate,
|
||||
type WindowTurn,
|
||||
} from './entity-salience.ts';
|
||||
import {
|
||||
resolveEntitiesToPointers,
|
||||
DEFAULT_MAX_POINTERS,
|
||||
@@ -28,22 +33,69 @@ import {
|
||||
} from './retrieval-reflex.ts';
|
||||
import { resolveViaIpc, resolveSocketPath, IPC_UNAVAILABLE } from './resolve-ipc.ts';
|
||||
|
||||
/** Host capability shape (D1=A): candidates in, pointers out. Narrow by design. */
|
||||
/** Per-turn resolver options shared by every rung of the ladder. */
|
||||
export interface ResolveEntitiesOpts {
|
||||
priorContextText?: string;
|
||||
maxPointers?: number;
|
||||
/** v0.43 (#2095): 'slug-only' under windowing — see ResolvePointersOpts. */
|
||||
suppression?: 'slug-and-title' | 'slug-only';
|
||||
}
|
||||
|
||||
/**
|
||||
* Host capability shape (D1=A): candidates in, pointers out. Narrow by design.
|
||||
*
|
||||
* CONTRACT (red-team): a host resolver MUST honor `opts.suppression`. Under
|
||||
* windowing the orchestrator passes 'slug-only' — a resolver that keeps
|
||||
* applying the legacy title-whole-word rule will suppress every entity merely
|
||||
* mentioned in a prior window turn and silently disable the feature. Hosts
|
||||
* built against the pre-window contract should be upgraded or pinned to
|
||||
* `retrieval_reflex_window_turns: 1`. (A capability/version gate so the
|
||||
* orchestrator can detect a stale host is a filed TODO.)
|
||||
*/
|
||||
export type ResolveEntitiesFn = (
|
||||
candidates: EntityCandidate[],
|
||||
opts: { priorContextText?: string; maxPointers?: number },
|
||||
opts: ResolveEntitiesOpts,
|
||||
) => Promise<PointerBlock | null>;
|
||||
|
||||
export interface ReflexParams {
|
||||
workspaceDir: string;
|
||||
/** The current turn's user text (drives extraction). */
|
||||
/** The current turn's user text (drives extraction when no window is given). */
|
||||
currentUserText: string;
|
||||
/** Joined PRIOR turns + loaded page bodies — EXCLUDES the current turn (suppression). */
|
||||
priorContextText: string;
|
||||
/**
|
||||
* v0.43 (#2095): recent turns (oldest → newest, current turn last). When
|
||||
* present and the configured window is > 1, extraction widens to the last
|
||||
* N turns (assistant-introduced entities + named-antecedent follow-ups now
|
||||
* resolve) and suppression switches to slug-only (codex D7 — the title rule
|
||||
* would suppress every entity merely mentioned in a prior window turn).
|
||||
*/
|
||||
windowTurns?: WindowTurn[];
|
||||
/** Host-provided resolver, if the OpenClaw plugin contract supplied one. */
|
||||
resolveEntities?: ResolveEntitiesFn;
|
||||
}
|
||||
|
||||
/** Default extraction window (turns). 1 = legacy current-turn-only. */
|
||||
export const DEFAULT_WINDOW_TURNS = 4;
|
||||
|
||||
export function windowTurnCount(cfg: GBrainConfig | null): number {
|
||||
// Env plane is read DIRECTLY here (mirroring reflexEnabled's direct
|
||||
// process.env read), not just via loadConfig's env→config mapping. When
|
||||
// there's no config file AND no DATABASE_URL, loadConfig() returns null and
|
||||
// drops that mapping entirely — so without this, the documented
|
||||
// GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS escape hatch would be silently
|
||||
// ignored and the window would fall back to the default of 4 (a real
|
||||
// config-less-environment bug, e.g. a clean CI shard with no brain).
|
||||
const env = process.env.GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS;
|
||||
if (env != null && env !== '') {
|
||||
const e = Number(env);
|
||||
if (Number.isFinite(e) && e >= 1) return Math.floor(e);
|
||||
}
|
||||
const n = cfg?.retrieval_reflex_window_turns;
|
||||
if (typeof n === 'number' && Number.isFinite(n) && n >= 1) return Math.floor(n);
|
||||
return DEFAULT_WINDOW_TURNS;
|
||||
}
|
||||
|
||||
const TIMEOUT_MS = 1500; // generous per-turn ceiling; the work is usually <100ms
|
||||
const HEARTBEAT_PATH = join(homedir(), '.gbrain', 'integrations', 'retrieval-reflex', 'heartbeat.jsonl');
|
||||
|
||||
@@ -68,14 +120,37 @@ export async function buildReflexAddition(params: ReflexParams): Promise<string
|
||||
const cfg = loadConfig();
|
||||
if (!reflexEnabled(cfg)) return null;
|
||||
|
||||
// Zero-candidate fast path: one regex pass, no brain touch.
|
||||
const candidates = extractCandidates(params.currentUserText);
|
||||
// v0.43 (#2095): widen extraction across the last N turns when a window
|
||||
// is supplied and configured > 1. Window=1 reproduces the legacy
|
||||
// current-turn-only behavior exactly (including suppression mode).
|
||||
const windowN = windowTurnCount(cfg);
|
||||
const windowed = windowN > 1 && (params.windowTurns?.length ?? 0) > 0;
|
||||
const candidates: EntityCandidate[] = windowed
|
||||
? extractCandidatesFromWindow(params.windowTurns!.slice(-windowN))
|
||||
: extractCandidates(params.currentUserText);
|
||||
// Zero-candidate fast path: regex passes only, no brain touch.
|
||||
if (!candidates.length) return null;
|
||||
|
||||
const opts = { priorContextText: params.priorContextText, maxPointers: maxPointers(cfg) };
|
||||
const opts: ResolveEntitiesOpts = {
|
||||
priorContextText: params.priorContextText,
|
||||
maxPointers: maxPointers(cfg),
|
||||
suppression: windowed ? 'slug-only' : 'slug-and-title',
|
||||
};
|
||||
const block = await withTimeout(resolve(params, cfg, candidates, opts), TIMEOUT_MS);
|
||||
if (!block || !block.pointers.length) return null;
|
||||
|
||||
// Accept-side reflex-channel logging (red-team): the block survived the
|
||||
// per-turn timeout, so these pointers ARE being injected. Only the
|
||||
// direct-Postgres rung has an engine here; the IPC rung logs server-side
|
||||
// at delivery; host-injected resolvers can't log (documented gap).
|
||||
if (!params.resolveEntities && isPostgres(cfg)) {
|
||||
const engine = await getPostgresEngine(cfg);
|
||||
if (engine) {
|
||||
const { logDeliveredReflexPointers } = await import('./retrieval-reflex.ts');
|
||||
logDeliveredReflexPointers(engine, block.pointers);
|
||||
}
|
||||
}
|
||||
|
||||
writeHeartbeat(cfg, block.pointers.length);
|
||||
return block.text;
|
||||
} catch {
|
||||
@@ -87,7 +162,7 @@ async function resolve(
|
||||
params: ReflexParams,
|
||||
cfg: GBrainConfig | null,
|
||||
candidates: EntityCandidate[],
|
||||
opts: { priorContextText?: string; maxPointers?: number },
|
||||
opts: ResolveEntitiesOpts,
|
||||
): Promise<PointerBlock | null> {
|
||||
// 1. Host capability (any engine).
|
||||
if (params.resolveEntities) {
|
||||
|
||||
@@ -35,6 +35,8 @@ export interface ResolveRequest {
|
||||
priorContextText?: string;
|
||||
maxPointers?: number;
|
||||
sourceId?: string;
|
||||
/** v0.43 (#2095, codex D7): suppression mode — 'slug-only' under windowing. */
|
||||
suppression?: 'slug-and-title' | 'slug-only';
|
||||
}
|
||||
|
||||
export type ResolveHandler = (req: ResolveRequest) => Promise<PointerBlock | null>;
|
||||
@@ -98,6 +100,13 @@ export async function resolveViaIpc(
|
||||
export async function startResolveIpcServer(
|
||||
socketPath: string,
|
||||
handler: ResolveHandler,
|
||||
/**
|
||||
* v0.43 (#2095, red-team): fired ONLY after the response was successfully
|
||||
* written to the client — the accept-side seam for reflex-channel feedback
|
||||
* logging. A block the client never received (timeout, dead socket) was
|
||||
* never injected into a prompt and must not count as "volunteered".
|
||||
*/
|
||||
onDelivered?: (block: PointerBlock, req: ResolveRequest) => void,
|
||||
): Promise<net.Server | null> {
|
||||
// Remove a stale socket file if present (a previous serve that didn't clean up).
|
||||
cleanupStaleSocket(socketPath);
|
||||
@@ -113,14 +122,23 @@ export async function startResolveIpcServer(
|
||||
if (nl < 0) return;
|
||||
const line = buf.slice(0, nl);
|
||||
let resp: string;
|
||||
let delivered: { block: PointerBlock; req: ResolveRequest } | null = null;
|
||||
try {
|
||||
const req = JSON.parse(line) as ResolveRequest;
|
||||
const block = await handler(req);
|
||||
resp = JSON.stringify({ ok: true, block });
|
||||
if (block) delivered = { block, req };
|
||||
} catch (e) {
|
||||
resp = JSON.stringify({ ok: false, error: (e as Error).message });
|
||||
}
|
||||
try { conn.write(resp + '\n'); } catch { /* client gone */ }
|
||||
try {
|
||||
conn.write(resp + '\n');
|
||||
// Write accepted — the client (250ms budget) may still have hung
|
||||
// up, but this is the closest observable delivery point.
|
||||
if (delivered && onDelivered) {
|
||||
try { onDelivered(delivered.block, delivered.req); } catch { /* telemetry only */ }
|
||||
}
|
||||
} catch { /* client gone — do NOT log undelivered pointers */ }
|
||||
conn.end();
|
||||
});
|
||||
conn.on('error', () => { try { conn.destroy(); } catch { /* noop */ } });
|
||||
|
||||
@@ -34,10 +34,40 @@ import type { EntityCandidate } from './entity-salience.ts';
|
||||
export const DEFAULT_MAX_POINTERS = 3;
|
||||
const SYNOPSIS_MAX = 160;
|
||||
|
||||
/** Which resolution arm produced a pointer (provenance → honest confidence). */
|
||||
export type ResolveArm = 'alias' | 'title' | 'slug-suffix';
|
||||
|
||||
/**
|
||||
* v0.43 (#2095) — arm → confidence. Lives HERE, next to the arm definitions,
|
||||
* so arm identity and its score can't drift apart (eng-review note). The
|
||||
* volunteer layer imports these; small deterministic boosts (multi-turn /
|
||||
* newest-turn mention) are added on top there.
|
||||
*/
|
||||
export const ARM_CONFIDENCE: Record<ResolveArm, number> = {
|
||||
alias: 0.9,
|
||||
title: 0.8,
|
||||
'slug-suffix': 0.6,
|
||||
};
|
||||
|
||||
export interface ReflexPointer {
|
||||
display: string;
|
||||
slug: string;
|
||||
/** Which brain source the page lives in (federated callers need it for dedup). */
|
||||
source_id: string;
|
||||
synopsis: string;
|
||||
/** Resolution provenance (v0.43 #2095). */
|
||||
arm: ResolveArm;
|
||||
/** Base arm confidence (ARM_CONFIDENCE[arm]); callers may boost. */
|
||||
confidence: number;
|
||||
/**
|
||||
* normalizeAlias form of the CANDIDATE that resolved this pointer (v0.43
|
||||
* #2095) — lets the volunteer layer join pointers back to window-salience
|
||||
* metadata without guessing from the display label (which falls back to the
|
||||
* page title when the candidate surface differs, e.g. alias "Swami" →
|
||||
* title "Swami X"). Absent only when suffix classification couldn't
|
||||
* recover the source candidate.
|
||||
*/
|
||||
matchedNorm?: string;
|
||||
}
|
||||
|
||||
export interface PointerBlock {
|
||||
@@ -55,10 +85,30 @@ export interface ResolvePointersOpts {
|
||||
* message's own mention would suppress every pointer (eng-review/Codex fix).
|
||||
*/
|
||||
priorContextText?: string;
|
||||
/**
|
||||
* v0.43 (#2095, codex D7) — suppression mode.
|
||||
* 'slug-and-title' (default, window=1 legacy): suppress when the slug OR
|
||||
* the page title appears whole-word in prior context.
|
||||
* 'slug-only' (REQUIRED under multi-turn windowing): suppress on slug
|
||||
* presence only. Slugs only enter context when a pointer block or page
|
||||
* body was actually injected; a bare mention of "Alice Example" in a
|
||||
* prior turn never contains `people/alice-example`. The title rule
|
||||
* would suppress every entity merely MENTIONED in a prior window turn
|
||||
* — breaking window extraction by construction.
|
||||
*/
|
||||
suppression?: 'slug-and-title' | 'slug-only';
|
||||
/**
|
||||
* v0.43 (#2095) — federated scope: resolve across these sources instead of
|
||||
* the single positional sourceId. Precedence mirrors sourceScopeOpts
|
||||
* (federated array > scalar). Alias arm loops per source; the title/slug
|
||||
* arm uses source_id = ANY(...) in one query.
|
||||
*/
|
||||
sourceIds?: string[];
|
||||
}
|
||||
|
||||
interface PageRow {
|
||||
slug: string;
|
||||
source_id: string;
|
||||
title: string;
|
||||
type: string | null;
|
||||
frontmatter: Record<string, unknown> | null;
|
||||
@@ -88,40 +138,63 @@ export async function resolveEntitiesToPointers(
|
||||
const titlesLc: string[] = [];
|
||||
const exactSlugs: string[] = [];
|
||||
const slugSuffixes: string[] = [];
|
||||
// Reverse maps for arm-2 provenance (which candidate produced a row) —
|
||||
// populated in this same pass so the derivations happen exactly once.
|
||||
const titleToNorm = new Map<string, string>();
|
||||
const slugToNorm = new Map<string, string>();
|
||||
for (const c of candidates) {
|
||||
const norm = normalizeAlias(c.query);
|
||||
if (!norm) continue;
|
||||
if (!displayByNorm.has(norm)) displayByNorm.set(norm, c.display);
|
||||
aliasNorms.push(norm);
|
||||
titlesLc.push(c.query.toLowerCase());
|
||||
const tl = c.query.toLowerCase();
|
||||
titlesLc.push(tl);
|
||||
if (!titleToNorm.has(tl)) titleToNorm.set(tl, norm);
|
||||
const s = slugify(c.query);
|
||||
if (s) {
|
||||
exactSlugs.push(s);
|
||||
slugSuffixes.push(`%/${s}`);
|
||||
if (!slugToNorm.has(s)) slugToNorm.set(s, norm);
|
||||
}
|
||||
}
|
||||
if (!aliasNorms.length) return null;
|
||||
|
||||
// Ordered set of resolved slugs (alias hits first → higher confidence).
|
||||
const resolvedSlugs: string[] = [];
|
||||
// Federated scope (v0.43 #2095): explicit sourceIds win over the scalar.
|
||||
const sourceIds = opts.sourceIds?.length ? opts.sourceIds : [sourceId];
|
||||
|
||||
// Ordered set of resolved (source, slug) pairs with arm provenance —
|
||||
// alias hits pushed first → higher confidence.
|
||||
const resolved: Array<{ slug: string; source_id: string; arm: ResolveArm; matchedNorm?: string }> = [];
|
||||
const seen = new Set<string>();
|
||||
const pushSlug = (slug: string) => {
|
||||
if (slug && !seen.has(slug)) {
|
||||
seen.add(slug);
|
||||
resolvedSlugs.push(slug);
|
||||
// Neither source ids nor slugs contain spaces, so a space separator is safe.
|
||||
const keyOf = (src: string, slug: string) => `${src} ${slug}`;
|
||||
const push = (slug: string, src: string, arm: ResolveArm, matchedNorm?: string) => {
|
||||
if (!slug) return;
|
||||
const k = keyOf(src, slug);
|
||||
if (!seen.has(k)) {
|
||||
seen.add(k);
|
||||
resolved.push({ slug, source_id: src, arm, matchedNorm });
|
||||
}
|
||||
};
|
||||
|
||||
// Arm 1 — alias-first. Unambiguous single-slug hits only. Guarded: pre-v110
|
||||
// brains throw "relation page_aliases does not exist" — swallow and continue.
|
||||
try {
|
||||
const aliasMap = await engine.resolveAliases(aliasNorms, { sourceId });
|
||||
// Arm 1 — alias-first. Unambiguous single-slug hits only, per source (no
|
||||
// engine-interface change for federation). Guarded: pre-v110 brains throw
|
||||
// "relation page_aliases does not exist" — swallow and continue.
|
||||
// Per-source lookups are independent — run them concurrently so a
|
||||
// federated caller (M granted sources) pays one RTT, not M sequential
|
||||
// ones (~71ms each cross-region; the reflex runs under a 1.5s budget).
|
||||
// Results are folded back in sourceIds order so pointer ordering stays
|
||||
// deterministic. Per-source failures degrade independently (pre-v110
|
||||
// brains have no page_aliases table).
|
||||
const aliasResults = await Promise.allSettled(
|
||||
sourceIds.map((src) => engine.resolveAliases(aliasNorms, { sourceId: src })),
|
||||
);
|
||||
for (let i = 0; i < sourceIds.length; i++) {
|
||||
const r = aliasResults[i];
|
||||
if (r.status !== 'fulfilled') continue;
|
||||
for (const norm of aliasNorms) {
|
||||
const hits = aliasMap.get(norm);
|
||||
if (hits && hits.length === 1) pushSlug(hits[0].slug);
|
||||
const hits = r.value.get(norm);
|
||||
if (hits && hits.length === 1) push(hits[0].slug, sourceIds[i], 'alias', norm);
|
||||
}
|
||||
} catch {
|
||||
/* no page_aliases table (pre-v110) — degrade to the title/slug arm */
|
||||
}
|
||||
|
||||
// Arm 2 — exact title OR slug-suffix. This is the recall fix: a bare "Alice
|
||||
@@ -130,53 +203,72 @@ export async function resolveEntitiesToPointers(
|
||||
let rows: PageRow[] = [];
|
||||
try {
|
||||
rows = await engine.executeRaw<PageRow>(
|
||||
`SELECT slug, title, type, frontmatter, compiled_truth
|
||||
`SELECT slug, source_id, title, type, frontmatter, compiled_truth
|
||||
FROM pages
|
||||
WHERE deleted_at IS NULL
|
||||
AND source_id = $1
|
||||
AND source_id = ANY($1::text[])
|
||||
AND ( lower(title) = ANY($2::text[])
|
||||
OR slug = ANY($3::text[])
|
||||
OR slug LIKE ANY($4::text[]) )`,
|
||||
[sourceId, titlesLc, exactSlugs, slugSuffixes],
|
||||
[sourceIds, titlesLc, exactSlugs, slugSuffixes],
|
||||
);
|
||||
} catch {
|
||||
rows = [];
|
||||
}
|
||||
// Hydrate alias-resolved slugs too (their bodies for the synopsis) if not in rows.
|
||||
const rowBySlug = new Map<string, PageRow>();
|
||||
for (const r of rows) rowBySlug.set(r.slug, r);
|
||||
const aliasOnly = resolvedSlugs.filter((s) => !rowBySlug.has(s));
|
||||
// Hydrate alias-resolved pages too (their bodies for the synopsis) if not in rows.
|
||||
const rowByKey = new Map<string, PageRow>();
|
||||
for (const r of rows) rowByKey.set(keyOf(r.source_id, r.slug), r);
|
||||
const aliasOnly = resolved.filter((p) => !rowByKey.has(keyOf(p.source_id, p.slug)));
|
||||
if (aliasOnly.length) {
|
||||
try {
|
||||
const extra = await engine.executeRaw<PageRow>(
|
||||
`SELECT slug, title, type, frontmatter, compiled_truth
|
||||
`SELECT slug, source_id, title, type, frontmatter, compiled_truth
|
||||
FROM pages
|
||||
WHERE deleted_at IS NULL AND source_id = $1 AND slug = ANY($2::text[])`,
|
||||
[sourceId, aliasOnly],
|
||||
WHERE deleted_at IS NULL AND source_id = ANY($1::text[]) AND slug = ANY($2::text[])`,
|
||||
[sourceIds, aliasOnly.map((p) => p.slug)],
|
||||
);
|
||||
for (const r of extra) rowBySlug.set(r.slug, r);
|
||||
for (const r of extra) rowByKey.set(keyOf(r.source_id, r.slug), r);
|
||||
} catch {
|
||||
/* ignore — alias slug may be stale */
|
||||
}
|
||||
}
|
||||
// Title/slug matches that weren't alias hits, appended after alias hits.
|
||||
for (const r of rows) pushSlug(r.slug);
|
||||
// Arm provenance per row is classified in JS (codex D8) — the combined OR
|
||||
// can't report which predicate matched: an exact lower(title) hit is the
|
||||
// 'title' arm; anything else got in via slug / slug-suffix.
|
||||
const titleSet = new Set(titlesLc);
|
||||
for (const r of rows) {
|
||||
const titleLc = (r.title ?? '').toLowerCase();
|
||||
if (titleSet.has(titleLc)) {
|
||||
push(r.slug, r.source_id, 'title', titleToNorm.get(titleLc));
|
||||
} else {
|
||||
// Slug arm: exact slugified-candidate match, else suffix scan.
|
||||
const tail = r.slug.includes('/') ? r.slug.slice(r.slug.lastIndexOf('/') + 1) : r.slug;
|
||||
push(r.slug, r.source_id, 'slug-suffix', slugToNorm.get(r.slug) ?? slugToNorm.get(tail));
|
||||
}
|
||||
}
|
||||
|
||||
// Build pointers in confidence order, applying suppression + cap.
|
||||
const suppression = opts.suppression ?? 'slug-and-title';
|
||||
const pointers: ReflexPointer[] = [];
|
||||
for (const slug of resolvedSlugs) {
|
||||
const row = rowBySlug.get(slug);
|
||||
for (const { slug, source_id, arm, matchedNorm } of resolved) {
|
||||
const row = rowByKey.get(keyOf(source_id, slug));
|
||||
if (!row) continue;
|
||||
// Suppression: already present in PRIOR context (slug or title). The current
|
||||
// turn is deliberately excluded from priorContextText.
|
||||
// Suppression: already present in PRIOR context. The current turn is
|
||||
// deliberately excluded from priorContextText. Under windowing
|
||||
// ('slug-only', codex D7) only the slug counts — a slug appears in prior
|
||||
// context only when a pointer/page was actually surfaced there, while a
|
||||
// title appears on any bare mention.
|
||||
if (priorLc) {
|
||||
const titleLc = (row.title ?? '').toLowerCase();
|
||||
if (priorLc.includes(slug.toLowerCase())) continue;
|
||||
if (titleLc && wholeWordIncludes(priorLc, titleLc)) continue;
|
||||
if (suppression === 'slug-and-title') {
|
||||
const titleLc = (row.title ?? '').toLowerCase();
|
||||
if (titleLc && wholeWordIncludes(priorLc, titleLc)) continue;
|
||||
}
|
||||
}
|
||||
const display = displayForRow(row, displayByNorm);
|
||||
const synopsis = safeSynopsis(row);
|
||||
pointers.push({ display, slug, synopsis });
|
||||
pointers.push({ display, slug, source_id, synopsis, arm, confidence: ARM_CONFIDENCE[arm], matchedNorm });
|
||||
if (pointers.length >= maxPointers) break;
|
||||
}
|
||||
|
||||
@@ -246,3 +338,30 @@ export function renderPointerBlock(pointers: ReflexPointer[]): string {
|
||||
}
|
||||
return lines.join('\n');
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.43 (#2095, codex D11 + red-team) — ambient-channel feedback logging,
|
||||
* ACCEPT-SIDE ONLY. Called by the delivery points (the serve IPC server after
|
||||
* a successful write; the direct-Postgres reflex rung after its per-turn
|
||||
* timeout admitted the block) — never inside the resolver itself, because a
|
||||
* pointer block that timed out client-side was NEVER injected into a prompt,
|
||||
* and logging it would inflate "volunteered" counts and drag the measured
|
||||
* precision toward zero (corrupting the exact stats users tune
|
||||
* min_confidence with).
|
||||
*/
|
||||
export function logDeliveredReflexPointers(engine: BrainEngine, pointers: ReflexPointer[]): void {
|
||||
if (!pointers.length) return;
|
||||
void import('./volunteer-events.ts')
|
||||
.then(({ logVolunteerEventsFireAndForget, volunteerEventRowsFrom }) => {
|
||||
logVolunteerEventsFireAndForget(
|
||||
engine,
|
||||
volunteerEventRowsFrom(
|
||||
pointers.map((p) => ({ ...p, rationale: `${p.arm} match "${p.display}"` })),
|
||||
{ channel: 'reflex' },
|
||||
),
|
||||
);
|
||||
})
|
||||
.catch(() => {
|
||||
/* telemetry only */
|
||||
});
|
||||
}
|
||||
|
||||
@@ -0,0 +1,186 @@
|
||||
/**
|
||||
* v0.43 (#2095) — context_volunteer_events: the feedback-loop log behind
|
||||
* push-based context. One row per page the brain VOLUNTEERED, written
|
||||
* fire-and-forget by the volunteer_context op, the retrieval-reflex pointer
|
||||
* path (channel 'reflex'), and `gbrain watch` (channel 'watch').
|
||||
*
|
||||
* "Used" is DERIVED, never written: a volunteered page counts as used when
|
||||
* pages.last_retrieved_at > volunteered_at (the existing bumpLastRetrievedAt
|
||||
* write-back on get_page/search/query is the open/cite signal). The join is
|
||||
* approximate by design — last-retrieved is 5-min throttled (false negatives)
|
||||
* and unrelated reads match too (false positives); stats output carries the
|
||||
* caveat.
|
||||
*
|
||||
* Retention: rows older than VOLUNTEER_EVENTS_TTL_DAYS are pruned by the
|
||||
* dream cycle's purge phase so conversation-adjacent telemetry never grows
|
||||
* unbounded. rationale is a deterministic template that may embed the matched
|
||||
* entity's surface form (which by construction resolved to an existing
|
||||
* alias/title/slug) — never free conversation text.
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from './../engine.ts';
|
||||
import { registerBackgroundWorkDrainer } from '../background-work.ts';
|
||||
|
||||
export const VOLUNTEER_EVENTS_TTL_DAYS = 90;
|
||||
|
||||
export type VolunteerChannel = 'op' | 'reflex' | 'watch';
|
||||
|
||||
/**
|
||||
* Map volunteered pages to event rows for one channel — the ONE place the
|
||||
* VolunteerEventRow shape is assembled (op / reflex / watch all call this,
|
||||
* so adding a column is a one-site change).
|
||||
*/
|
||||
export function volunteerEventRowsFrom(
|
||||
pages: Array<{ source_id: string; slug: string; confidence: number; arm: string; rationale: string }>,
|
||||
opts: { channel: VolunteerChannel; session_id?: string | null; turn?: number | null },
|
||||
): VolunteerEventRow[] {
|
||||
return pages.map((p) => ({
|
||||
source_id: p.source_id,
|
||||
slug: p.slug,
|
||||
confidence: p.confidence,
|
||||
match_arm: p.arm,
|
||||
rationale: p.rationale,
|
||||
channel: opts.channel,
|
||||
session_id: opts.session_id ?? null,
|
||||
turn: opts.turn ?? null,
|
||||
}));
|
||||
}
|
||||
|
||||
export interface VolunteerEventRow {
|
||||
source_id: string;
|
||||
slug: string;
|
||||
confidence: number;
|
||||
match_arm: string;
|
||||
rationale: string;
|
||||
channel: VolunteerChannel;
|
||||
session_id?: string | null;
|
||||
turn?: number | null;
|
||||
}
|
||||
|
||||
/**
|
||||
* ONE multi-row parameterized INSERT for a batch of volunteered pages (max 5
|
||||
* per call by the volunteer cap) — never per-row awaited INSERTs (up to 5
|
||||
* RTTs ≈ 355ms on a cross-region deployment; eng-review D4). Throws on
|
||||
* failure; callers run it through the volunteer-events background-work sink
|
||||
* with try/catch so logging can never fail the op.
|
||||
*/
|
||||
export async function insertVolunteerEvents(
|
||||
engine: BrainEngine,
|
||||
rows: VolunteerEventRow[],
|
||||
): Promise<void> {
|
||||
if (!rows.length) return;
|
||||
const params: unknown[] = [];
|
||||
const tuples = rows.map((r) => {
|
||||
const base = params.length;
|
||||
params.push(
|
||||
r.source_id,
|
||||
r.slug,
|
||||
r.confidence,
|
||||
r.match_arm,
|
||||
r.rationale,
|
||||
r.channel,
|
||||
r.session_id ?? null,
|
||||
r.turn ?? null,
|
||||
);
|
||||
const ph = Array.from({ length: 8 }, (_, i) => `$${base + i + 1}`);
|
||||
return `(${ph.join(', ')})`;
|
||||
});
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO context_volunteer_events
|
||||
(source_id, slug, confidence, match_arm, rationale, channel, session_id, turn)
|
||||
VALUES ${tuples.join(', ')}`,
|
||||
params,
|
||||
);
|
||||
}
|
||||
|
||||
// ── Fire-and-forget sink (eng-review D4) ─────────────────────────────────
|
||||
// Mirrors src/core/last-retrieved.ts: track every dangling INSERT promise in
|
||||
// a module Set, register a drainer so finishCliTeardown settles them
|
||||
// against a live engine before teardown on EVERY CLI exit path (the commit-1
|
||||
// drain hoist). Logging failure never fails the caller.
|
||||
|
||||
const pendingVolunteerEventWrites = new Set<Promise<unknown>>();
|
||||
|
||||
/**
|
||||
* Log volunteered pages without blocking the hot path. The batched INSERT
|
||||
* runs as a tracked dangling promise; errors are swallowed (pre-v117 brains,
|
||||
* transient DB failures — the volunteer result is unaffected).
|
||||
*/
|
||||
export function logVolunteerEventsFireAndForget(
|
||||
engine: BrainEngine,
|
||||
rows: VolunteerEventRow[],
|
||||
): void {
|
||||
if (!rows.length) return;
|
||||
const p = insertVolunteerEvents(engine, rows).catch(() => {
|
||||
/* best-effort telemetry — never surfaces */
|
||||
});
|
||||
pendingVolunteerEventWrites.add(p);
|
||||
void p.finally(() => pendingVolunteerEventWrites.delete(p));
|
||||
}
|
||||
|
||||
/** Drain pending event writes (bounded). Same snapshot semantics as last-retrieved. */
|
||||
export async function awaitPendingVolunteerEventWrites(
|
||||
timeoutMs = 5_000,
|
||||
): Promise<{ unfinished: number }> {
|
||||
if (pendingVolunteerEventWrites.size === 0) return { unfinished: 0 };
|
||||
const snapshot = Array.from(pendingVolunteerEventWrites);
|
||||
let timer: ReturnType<typeof setTimeout> | undefined;
|
||||
const timeout = new Promise<'timeout'>((resolve) => {
|
||||
timer = setTimeout(() => resolve('timeout'), timeoutMs);
|
||||
});
|
||||
const drain = Promise.allSettled(snapshot).then(() => 'drained' as const);
|
||||
const outcome = await Promise.race([drain, timeout]);
|
||||
if (timer) clearTimeout(timer);
|
||||
if (outcome === 'timeout') {
|
||||
const unfinished = pendingVolunteerEventWrites.size;
|
||||
// Drop the snapshot so a long-lived process (`gbrain watch`) doesn't
|
||||
// accumulate references to forever-pending work (last-retrieved C1).
|
||||
for (const p of snapshot) pendingVolunteerEventWrites.delete(p);
|
||||
return { unfinished };
|
||||
}
|
||||
return { unfinished: 0 };
|
||||
}
|
||||
|
||||
// Registered in the enqueue-owning module (background-work contract): module
|
||||
// not imported ⇒ nothing enqueued ⇒ nothing to drain. Order 4 — after facts /
|
||||
// last-retrieved / search-cache / eval-capture; bare INSERTs, no abort.
|
||||
registerBackgroundWorkDrainer({
|
||||
name: 'volunteer-events',
|
||||
order: 4,
|
||||
drain: (ms) => awaitPendingVolunteerEventWrites(ms),
|
||||
});
|
||||
|
||||
/** Test seam — clears the pending set so each test starts clean. */
|
||||
export function _resetPendingVolunteerEventWritesForTests(): void {
|
||||
pendingVolunteerEventWrites.clear();
|
||||
}
|
||||
|
||||
/** Test seam — peek the current pending count. */
|
||||
export function _peekPendingVolunteerEventWritesForTests(): number {
|
||||
return pendingVolunteerEventWrites.size;
|
||||
}
|
||||
|
||||
/**
|
||||
* 90-day GC, called from the dream cycle's purge phase (mirrors
|
||||
* purgeStaleCheckpoints). Best-effort: returns 0 on any failure (pre-v117
|
||||
* brains have no table yet).
|
||||
*/
|
||||
export async function purgeStaleVolunteerEvents(
|
||||
engine: BrainEngine,
|
||||
ttlDays = VOLUNTEER_EVENTS_TTL_DAYS,
|
||||
): Promise<number> {
|
||||
try {
|
||||
const rows = await engine.executeRaw<{ count: string | number }>(
|
||||
`WITH deleted AS (
|
||||
DELETE FROM context_volunteer_events
|
||||
WHERE volunteered_at < now() - ($1 || ' days')::interval
|
||||
RETURNING 1
|
||||
)
|
||||
SELECT count(*)::text AS count FROM deleted`,
|
||||
[String(ttlDays)],
|
||||
);
|
||||
return Number(rows[0]?.count ?? 0);
|
||||
} catch {
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,268 @@
|
||||
/**
|
||||
* v0.43 (#2095) — push-based context: the brain VOLUNTEERS relevant pages
|
||||
* from a rolling conversation window instead of waiting to be asked.
|
||||
*
|
||||
* window text ─ parseWindow() ─→ turns[] ─ extractCandidatesFromWindow()
|
||||
* │ (recency + frequency +
|
||||
* │ user-role weights)
|
||||
* ▼
|
||||
* resolveEntitiesToPointers(sourceIds[]) alias 0.9 / title 0.8 /
|
||||
* │ slug-suffix 0.6 (+0.05 boost
|
||||
* ▼ for ≥2-turn or newest-turn
|
||||
* gate min_confidence (default 0.7) → mentions)
|
||||
* suppression (slug-only — windowing) →
|
||||
* cap (3 default / 5 max)
|
||||
*
|
||||
* Zero-LLM, deterministic, precision-biased: push noise is worse than pull
|
||||
* silence (#2095). At the default gate, slug-suffix matches (0.6+0.05 < 0.7)
|
||||
* never volunteer — they need an explicit lower min_confidence.
|
||||
*
|
||||
* Consumed by three channels: the volunteer_context op, the retrieval-reflex
|
||||
* window path, and `gbrain watch`. Event logging lives in
|
||||
* volunteer-events.ts; usage stats here derive "used" from
|
||||
* pages.last_retrieved_at — APPROXIMATE by design (the 5-min last-retrieved
|
||||
* throttle causes false negatives; unrelated reads cause false positives).
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from '../engine.ts';
|
||||
import { normalizeAlias } from '../search/alias-normalize.ts';
|
||||
import {
|
||||
extractCandidatesFromWindow,
|
||||
type WindowTurn,
|
||||
type WindowEntityCandidate,
|
||||
} from './entity-salience.ts';
|
||||
import {
|
||||
resolveEntitiesToPointers,
|
||||
ARM_CONFIDENCE,
|
||||
type ResolveArm,
|
||||
} from './retrieval-reflex.ts';
|
||||
|
||||
export const VOLUNTEER_DEFAULT_MAX_PAGES = 3;
|
||||
export const VOLUNTEER_MAX_PAGES_CAP = 5;
|
||||
export const VOLUNTEER_DEFAULT_MIN_CONFIDENCE = 0.7;
|
||||
/** Deterministic boost for ≥2-turn or newest-turn mentions. */
|
||||
export const VOLUNTEER_SALIENCE_BOOST = 0.05;
|
||||
|
||||
export interface VolunteeredPage {
|
||||
slug: string;
|
||||
source_id: string;
|
||||
display: string;
|
||||
confidence: number;
|
||||
arm: ResolveArm;
|
||||
/** Deterministic template string — never raw conversation text. */
|
||||
rationale: string;
|
||||
synopsis: string;
|
||||
}
|
||||
|
||||
export interface VolunteerOpts {
|
||||
/** Resolved source scope (federated array > scalar — sourceScopeOpts shape). */
|
||||
sourceIds: string[];
|
||||
/** Prior context (already-surfaced pointers/pages) for slug-only suppression. */
|
||||
priorContext?: string;
|
||||
/**
|
||||
* Slugs to skip BEFORE the confidence gate and the maxPages cap (O(1)
|
||||
* membership). `gbrain watch` passes its session-dedupe set here — a
|
||||
* post-call filter would let a recurring already-pushed entity consume cap
|
||||
* slots every turn and starve new pages behind it (red-team finding).
|
||||
*/
|
||||
excludeSlugs?: ReadonlySet<string>;
|
||||
maxPages?: number;
|
||||
minConfidence?: number;
|
||||
}
|
||||
|
||||
/** Shared wire protocol for window turns — watch.ts imports this so the two
|
||||
* channels can never desynchronize on the prefix grammar. */
|
||||
export const TURN_PREFIX_RE = /^(user|assistant)\s*:\s?(.*)$/i;
|
||||
|
||||
/**
|
||||
* Lenient window parser: `user:` / `assistant:` line prefixes start a new
|
||||
* turn (oldest → newest); unprefixed lines continue the current turn; input
|
||||
* with no prefixes at all is ONE user turn (so `echo "..." | volunteer-context`
|
||||
* just works). CRLF-tolerant. Empty/whitespace input → [].
|
||||
*/
|
||||
export function parseWindow(text: string): WindowTurn[] {
|
||||
if (!text || !text.trim()) return [];
|
||||
const turns: WindowTurn[] = [];
|
||||
let current: WindowTurn | null = null;
|
||||
for (const rawLine of text.split(/\r?\n/)) {
|
||||
const m = TURN_PREFIX_RE.exec(rawLine);
|
||||
if (m) {
|
||||
if (current) turns.push(current);
|
||||
current = { role: m[1].toLowerCase() as WindowTurn['role'], text: m[2] ?? '' };
|
||||
} else if (current) {
|
||||
current.text += (current.text ? '\n' : '') + rawLine;
|
||||
} else if (rawLine.trim()) {
|
||||
// No prefix seen yet — accumulate into an implicit user turn.
|
||||
current = { role: 'user', text: rawLine };
|
||||
}
|
||||
}
|
||||
if (current) turns.push(current);
|
||||
// Trim trailing whitespace-only turn bodies.
|
||||
return turns
|
||||
.map((t) => ({ role: t.role, text: t.text.trim() }))
|
||||
.filter((t) => t.text.length > 0);
|
||||
}
|
||||
|
||||
function clampMaxPages(n: number | undefined): number {
|
||||
if (typeof n !== 'number' || !Number.isFinite(n) || n < 1) return VOLUNTEER_DEFAULT_MAX_PAGES;
|
||||
return Math.min(Math.floor(n), VOLUNTEER_MAX_PAGES_CAP);
|
||||
}
|
||||
|
||||
function rationaleFor(arm: ResolveArm, display: string, c: WindowEntityCandidate | undefined, windowSize: number): string {
|
||||
const armText =
|
||||
arm === 'alias' ? `alias match "${display}"`
|
||||
: arm === 'title' ? `exact title match "${display}"`
|
||||
: `slug match "${display}"`;
|
||||
if (!c) return armText;
|
||||
const parts = [armText];
|
||||
if (c.occurrences >= 2) parts.push(`mentioned in ${c.occurrences} of last ${windowSize} turns`);
|
||||
else if (c.inNewestTurn) parts.push('mentioned in the newest turn');
|
||||
if (!c.userMention) parts.push('assistant-introduced');
|
||||
return parts.join('; ');
|
||||
}
|
||||
|
||||
/**
|
||||
* Volunteer confidence-gated pages for a conversation window. Pure read —
|
||||
* event logging is the CALLER's job (through the volunteer-events sink).
|
||||
* Non-relational, zero-LLM; returns [] when nothing clears the gate.
|
||||
*/
|
||||
export async function volunteerContext(
|
||||
engine: BrainEngine,
|
||||
turns: WindowTurn[],
|
||||
opts: VolunteerOpts,
|
||||
): Promise<VolunteeredPage[]> {
|
||||
if (!turns.length || !opts.sourceIds?.length) return [];
|
||||
const candidates = extractCandidatesFromWindow(turns);
|
||||
if (!candidates.length) return [];
|
||||
const byNorm = new Map<string, WindowEntityCandidate>();
|
||||
for (const c of candidates) {
|
||||
const norm = normalizeAlias(c.query);
|
||||
if (norm && !byNorm.has(norm)) byNorm.set(norm, c);
|
||||
}
|
||||
|
||||
const maxPages = clampMaxPages(opts.maxPages);
|
||||
const minConfidence =
|
||||
typeof opts.minConfidence === 'number' && opts.minConfidence >= 0 && opts.minConfidence <= 1
|
||||
? opts.minConfidence
|
||||
: VOLUNTEER_DEFAULT_MIN_CONFIDENCE;
|
||||
|
||||
// Resolve up to the hard cap so the confidence gate sees the full pool —
|
||||
// a gated-out alias hit must not shadow a passing title hit behind it.
|
||||
const block = await resolveEntitiesToPointers(engine, opts.sourceIds[0], candidates, {
|
||||
sourceIds: opts.sourceIds,
|
||||
priorContextText: opts.priorContext,
|
||||
suppression: 'slug-only',
|
||||
maxPointers: VOLUNTEER_MAX_PAGES_CAP * 2,
|
||||
});
|
||||
if (!block) return [];
|
||||
|
||||
const out: VolunteeredPage[] = [];
|
||||
for (const p of block.pointers) {
|
||||
if (opts.excludeSlugs?.has(p.slug)) continue; // before gate + cap — see VolunteerOpts
|
||||
// matchedNorm is the resolver's provenance join-key (the candidate that
|
||||
// resolved the pointer); display-based lookup is the fallback for the
|
||||
// rare suffix rows where provenance couldn't be recovered.
|
||||
const cand = (p.matchedNorm ? byNorm.get(p.matchedNorm) : undefined) ?? byNorm.get(normalizeAlias(p.display));
|
||||
const boost = cand && (cand.occurrences >= 2 || cand.inNewestTurn) ? VOLUNTEER_SALIENCE_BOOST : 0;
|
||||
const confidence = Math.min(0.99, ARM_CONFIDENCE[p.arm] + boost);
|
||||
if (confidence < minConfidence) continue;
|
||||
out.push({
|
||||
slug: p.slug,
|
||||
source_id: p.source_id,
|
||||
display: p.display,
|
||||
confidence,
|
||||
arm: p.arm,
|
||||
rationale: rationaleFor(p.arm, p.display, cand, turns.length),
|
||||
synopsis: p.synopsis,
|
||||
});
|
||||
if (out.length >= maxPages) break;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* Canonical human rendering of one volunteered page — shared by
|
||||
* `gbrain volunteer-context` (cli.ts formatResult) and `gbrain watch` so the
|
||||
* two surfaces can't drift.
|
||||
*/
|
||||
export function formatVolunteeredPage(p: VolunteeredPage): string {
|
||||
return (
|
||||
`${p.display} → ${p.slug} (${p.confidence.toFixed(2)}, ${p.arm}) — ${p.rationale}` +
|
||||
(p.synopsis ? `\n ${p.synopsis}` : '')
|
||||
);
|
||||
}
|
||||
|
||||
// ── Usage stats (the feedback loop) ──────────────────────────────────────
|
||||
|
||||
export interface VolunteerArmStats {
|
||||
match_arm: string;
|
||||
channel: string;
|
||||
volunteered: number;
|
||||
used: number;
|
||||
/** used / volunteered, 0 when nothing volunteered. */
|
||||
precision: number;
|
||||
}
|
||||
|
||||
export interface VolunteerUsageStats {
|
||||
days: number;
|
||||
/** The join is approximate — see the note (false +/- documented in #2095 D9). */
|
||||
approximate: true;
|
||||
note: string;
|
||||
total_volunteered: number;
|
||||
total_used: number;
|
||||
by_arm: VolunteerArmStats[];
|
||||
}
|
||||
|
||||
export const VOLUNTEER_STATS_NOTE =
|
||||
'approximate: "used" = pages.last_retrieved_at > volunteered_at. The 5-min ' +
|
||||
'last-retrieved throttle causes false negatives; unrelated reads of the same ' +
|
||||
'page cause false positives.';
|
||||
|
||||
/**
|
||||
* Per-arm/channel precision over the last N days, source-scoped. Read-only;
|
||||
* returns zeroed stats on pre-v117 brains (no table).
|
||||
*/
|
||||
export async function volunteerUsageStats(
|
||||
engine: BrainEngine,
|
||||
sourceIds: string[],
|
||||
days = 30,
|
||||
): Promise<VolunteerUsageStats> {
|
||||
const safeDays = Number.isFinite(days) && days > 0 ? Math.floor(days) : 30;
|
||||
let rows: Array<{ match_arm: string; channel: string; volunteered: string | number; used: string | number }> = [];
|
||||
try {
|
||||
rows = await engine.executeRaw(
|
||||
`SELECT e.match_arm, e.channel,
|
||||
count(*)::text AS volunteered,
|
||||
count(*) FILTER (WHERE p.last_retrieved_at > e.volunteered_at)::text AS used
|
||||
FROM context_volunteer_events e
|
||||
LEFT JOIN pages p
|
||||
ON p.source_id = e.source_id AND p.slug = e.slug AND p.deleted_at IS NULL
|
||||
WHERE e.source_id = ANY($1::text[])
|
||||
AND e.volunteered_at > now() - ($2 || ' days')::interval
|
||||
GROUP BY e.match_arm, e.channel
|
||||
ORDER BY e.match_arm, e.channel`,
|
||||
[sourceIds, String(safeDays)],
|
||||
);
|
||||
} catch {
|
||||
rows = []; // pre-v117 brain — table doesn't exist yet
|
||||
}
|
||||
const by_arm: VolunteerArmStats[] = rows.map((r) => {
|
||||
const volunteered = Number(r.volunteered);
|
||||
const used = Number(r.used);
|
||||
return {
|
||||
match_arm: r.match_arm,
|
||||
channel: r.channel,
|
||||
volunteered,
|
||||
used,
|
||||
precision: volunteered > 0 ? Number((used / volunteered).toFixed(3)) : 0,
|
||||
};
|
||||
});
|
||||
return {
|
||||
days: safeDays,
|
||||
approximate: true,
|
||||
note: VOLUNTEER_STATS_NOTE,
|
||||
total_volunteered: by_arm.reduce((s, a) => s + a.volunteered, 0),
|
||||
total_used: by_arm.reduce((s, a) => s + a.used, 0),
|
||||
by_arm,
|
||||
};
|
||||
}
|
||||
+13
-1
@@ -1285,6 +1285,16 @@ async function runPhasePurge(engine: BrainEngine, dryRun: boolean): Promise<Phas
|
||||
} catch {
|
||||
// Non-fatal.
|
||||
}
|
||||
// v0.43 (#2095) — 90-day GC of the volunteered-context feedback log.
|
||||
// Conversation-adjacent telemetry must never grow unbounded. Best-effort:
|
||||
// purgeStaleVolunteerEvents returns 0 on pre-v117 brains (no table).
|
||||
let purgedVolunteerEvents = 0;
|
||||
try {
|
||||
const { purgeStaleVolunteerEvents } = await import('./context/volunteer-events.ts');
|
||||
purgedVolunteerEvents = await purgeStaleVolunteerEvents(engine);
|
||||
} catch {
|
||||
// Non-fatal.
|
||||
}
|
||||
return {
|
||||
phase: 'purge',
|
||||
status: 'ok',
|
||||
@@ -1293,7 +1303,8 @@ async function runPhasePurge(engine: BrainEngine, dryRun: boolean): Promise<Phas
|
||||
`purged ${purgedSources.length} source(s), ${purgedPages.count} page(s), ` +
|
||||
`${purgedClones.count} orphan clone temp dir(s), ${purgedCheckpoints} stale op_checkpoint(s), ` +
|
||||
`${purgedBrainstormCheckpoints} stale brainstorm checkpoint(s), ` +
|
||||
`and ${purgedBatchRetryAuditFiles} stale batch-retry audit file(s)`,
|
||||
`${purgedBatchRetryAuditFiles} stale batch-retry audit file(s), ` +
|
||||
`and ${purgedVolunteerEvents} stale volunteer event(s)`,
|
||||
details: {
|
||||
purged_sources_count: purgedSources.length,
|
||||
purged_pages_count: purgedPages.count,
|
||||
@@ -1304,6 +1315,7 @@ async function runPhasePurge(engine: BrainEngine, dryRun: boolean): Promise<Phas
|
||||
purged_checkpoints_count: purgedCheckpoints,
|
||||
purged_brainstorm_checkpoints_count: purgedBrainstormCheckpoints,
|
||||
purged_batch_retry_audit_files_count: purgedBatchRetryAuditFiles,
|
||||
purged_volunteer_events_count: purgedVolunteerEvents,
|
||||
},
|
||||
};
|
||||
} catch (e) {
|
||||
|
||||
@@ -5232,6 +5232,44 @@ export const MIGRATIONS: Migration[] = [
|
||||
ON code_edges_chunk (from_symbol_qualified);
|
||||
`,
|
||||
},
|
||||
{
|
||||
// Renumbered 116 -> 117 at merge: master's v0.42.41.0 triage wave
|
||||
// claimed v116 (code_edges_source_backfill_and_callee_index) first.
|
||||
version: 117,
|
||||
name: 'context_volunteer_events_table',
|
||||
// #2095 push-based context: feedback-loop log of every page the brain
|
||||
// VOLUNTEERED (via the volunteer_context op, the retrieval-reflex pointer
|
||||
// path, or `gbrain watch`). "Used" is derived later by joining
|
||||
// pages.last_retrieved_at > volunteered_at — no second write path.
|
||||
// session_id/turn are caller-supplied attribution (nullable). rationale is
|
||||
// a deterministic template string, NEVER raw conversation text. Rows are
|
||||
// pruned past 90 days by the dream cycle's purge phase
|
||||
// (purgeStaleVolunteerEvents). No ::jsonb anywhere. RLS: covered by the
|
||||
// v35 auto_rls_on_create_table event trigger on Postgres (same as
|
||||
// v110/v115 tables); pinned by the volunteer-context Postgres e2e.
|
||||
// Created empty; plain CREATE INDEX is instant — no CONCURRENTLY needed.
|
||||
// Keep in sync with src/schema.sql, src/core/pglite-schema.ts,
|
||||
// src/core/schema-embedded.ts.
|
||||
idempotent: true,
|
||||
sql: `
|
||||
CREATE TABLE IF NOT EXISTS context_volunteer_events (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
source_id TEXT NOT NULL,
|
||||
slug TEXT NOT NULL,
|
||||
confidence DOUBLE PRECISION NOT NULL,
|
||||
match_arm TEXT NOT NULL,
|
||||
rationale TEXT NOT NULL DEFAULT '',
|
||||
channel TEXT NOT NULL DEFAULT 'op',
|
||||
session_id TEXT,
|
||||
turn INTEGER,
|
||||
volunteered_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS context_volunteer_events_src_time_idx
|
||||
ON context_volunteer_events (source_id, volunteered_at DESC);
|
||||
CREATE INDEX IF NOT EXISTS context_volunteer_events_src_slug_idx
|
||||
ON context_volunteer_events (source_id, slug);
|
||||
`,
|
||||
},
|
||||
];
|
||||
|
||||
export const LATEST_VERSION = MIGRATIONS.length > 0
|
||||
|
||||
@@ -3145,6 +3145,99 @@ const get_recent_salience: Operation = {
|
||||
cliHints: { name: 'salience' },
|
||||
};
|
||||
|
||||
// --- v0.43 (#2095): push-based context — the brain volunteers pages ---
|
||||
|
||||
const volunteer_context: Operation = {
|
||||
name: 'volunteer_context',
|
||||
description:
|
||||
'Push-based context: volunteer brain pages relevant to a rolling conversation window ' +
|
||||
'WITHOUT being asked. Zero-LLM, confidence-gated (alias 0.9 / exact-title 0.8 / ' +
|
||||
'slug-suffix 0.6, +0.05 for multi-turn or newest-turn mentions; default gate 0.7), ' +
|
||||
'capped at 3 pages (max 5). Returns pointers with one-line rationales + synopses — ' +
|
||||
'open the page (get_page) before relying on details. Pass stats: true for the ' +
|
||||
'approximate volunteered-vs-used precision summary (the feedback loop).',
|
||||
scope: 'read',
|
||||
params: {
|
||||
window: {
|
||||
type: 'string',
|
||||
description:
|
||||
"Recent conversation turns, oldest → newest, as 'user:' / 'assistant:' prefixed " +
|
||||
'lines (unprefixed text = one user turn). Required unless stats: true. ' +
|
||||
'CLI: piped stdin fills this.',
|
||||
},
|
||||
prior_context: {
|
||||
type: 'string',
|
||||
description:
|
||||
'Already-surfaced context (pointer blocks / opened page bodies). Pages whose slug ' +
|
||||
'appears here are not re-volunteered.',
|
||||
},
|
||||
max_pages: { type: 'number', description: 'Max pages to volunteer (default 3, hard cap 5).' },
|
||||
min_confidence: {
|
||||
type: 'number',
|
||||
description:
|
||||
'Confidence gate 0..1 (default 0.7 — slug-suffix matches need an explicit lower gate).',
|
||||
},
|
||||
session_id: { type: 'string', description: 'Optional caller session id, logged for attribution.' },
|
||||
turn: { type: 'number', description: 'Optional caller turn number, logged for attribution.' },
|
||||
stats: {
|
||||
type: 'boolean',
|
||||
description:
|
||||
'Return the volunteered-vs-used precision summary instead of volunteering. ' +
|
||||
'APPROXIMATE: "used" = pages.last_retrieved_at > volunteered_at.',
|
||||
},
|
||||
days: { type: 'number', description: 'Stats window in days (default 30; stats mode only).' },
|
||||
},
|
||||
handler: async (ctx, p) => {
|
||||
const { parseWindow, volunteerContext, volunteerUsageStats } = await import('./context/volunteer.ts');
|
||||
const scope = sourceScopeOpts(ctx);
|
||||
const sourceIds = scope.sourceIds ?? (scope.sourceId ? [scope.sourceId] : ['default']);
|
||||
|
||||
if (p.stats === true) {
|
||||
return volunteerUsageStats(ctx.engine, sourceIds, typeof p.days === 'number' ? p.days : undefined);
|
||||
}
|
||||
|
||||
if (typeof p.window !== 'string' || !p.window.trim()) {
|
||||
throw new OperationError(
|
||||
'invalid_params',
|
||||
'window is required unless stats: true',
|
||||
'Pass the recent turns as a string (CLI: pipe them on stdin), or use --stats.',
|
||||
);
|
||||
}
|
||||
const turns = parseWindow(p.window);
|
||||
const pages = await volunteerContext(ctx.engine, turns, {
|
||||
sourceIds,
|
||||
priorContext: typeof p.prior_context === 'string' ? p.prior_context : undefined,
|
||||
maxPages: typeof p.max_pages === 'number' ? p.max_pages : undefined,
|
||||
minConfidence: typeof p.min_confidence === 'number' ? p.min_confidence : undefined,
|
||||
});
|
||||
|
||||
// Feedback-loop logging: fire-and-forget batched INSERT through the
|
||||
// volunteer-events sink (drained at exit). Never fails the op.
|
||||
if (pages.length) {
|
||||
try {
|
||||
const { logVolunteerEventsFireAndForget, volunteerEventRowsFrom } = await import('./context/volunteer-events.ts');
|
||||
// Trust-boundary clamps (remote MCP callers): cap session_id length so
|
||||
// a read-scoped token can't bank unbounded TEXT per request, and only
|
||||
// log integer turns — a non-integer would throw inside the single
|
||||
// multi-row INSERT and silently drop the whole batch.
|
||||
const sessionId = typeof p.session_id === 'string' ? p.session_id.slice(0, 256) : null;
|
||||
const turn =
|
||||
typeof p.turn === 'number' && Number.isInteger(p.turn) && Math.abs(p.turn) <= 2_147_483_647
|
||||
? p.turn
|
||||
: null;
|
||||
logVolunteerEventsFireAndForget(
|
||||
ctx.engine,
|
||||
volunteerEventRowsFrom(pages, { channel: 'op', session_id: sessionId, turn }),
|
||||
);
|
||||
} catch {
|
||||
/* telemetry only */
|
||||
}
|
||||
}
|
||||
return { pages, count: pages.length, window_turns: turns.length };
|
||||
},
|
||||
cliHints: { name: 'volunteer-context', stdin: 'window' },
|
||||
};
|
||||
|
||||
const find_anomalies: Operation = {
|
||||
name: 'find_anomalies',
|
||||
description: FIND_ANOMALIES_DESCRIPTION,
|
||||
@@ -4856,6 +4949,8 @@ export const operations: Operation[] = [
|
||||
whoami, sources_add, sources_list, sources_remove, sources_status,
|
||||
// v0.29: Salience + anomalies + recent transcripts
|
||||
get_recent_salience, find_anomalies, get_recent_transcripts,
|
||||
// v0.43 (#2095): push-based context
|
||||
volunteer_context,
|
||||
// v0.31: hot memory (facts table)
|
||||
extract_facts, recall, forget_fact,
|
||||
// v0.32.6: contradiction probe MCP surface (M3)
|
||||
|
||||
@@ -264,6 +264,12 @@ export class PGLiteEngine implements BrainEngine {
|
||||
}
|
||||
}
|
||||
|
||||
// NOTE (#2084): PGLite's Emscripten runtime writes the WASM backend's
|
||||
// proc_exit status into `process.exitCode` (initdb here at create-time,
|
||||
// the postmaster at close-time), and the writes land asynchronously —
|
||||
// a snapshot/restore around these awaits does NOT contain them. That is
|
||||
// why the CLI's exit paths read gbrain's own verdict
|
||||
// (cli-force-exit.ts currentExitCode), never ambient process.exitCode.
|
||||
try {
|
||||
this._db = await preservingProcessExitCode(() =>
|
||||
PGlite.create({
|
||||
|
||||
@@ -945,6 +945,27 @@ CREATE TABLE IF NOT EXISTS op_checkpoint_paths (
|
||||
FOREIGN KEY (op, fingerprint) REFERENCES op_checkpoints (op, fingerprint) ON DELETE CASCADE
|
||||
);
|
||||
|
||||
-- #2095 push-based context: feedback-loop log of volunteered pages.
|
||||
-- Mirrors migration v117 + src/schema.sql. "Used" derives from
|
||||
-- pages.last_retrieved_at > volunteered_at; 90-day prune in the dream
|
||||
-- cycle's purge phase.
|
||||
CREATE TABLE IF NOT EXISTS context_volunteer_events (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
source_id TEXT NOT NULL,
|
||||
slug TEXT NOT NULL,
|
||||
confidence DOUBLE PRECISION NOT NULL,
|
||||
match_arm TEXT NOT NULL,
|
||||
rationale TEXT NOT NULL DEFAULT '',
|
||||
channel TEXT NOT NULL DEFAULT 'op',
|
||||
session_id TEXT,
|
||||
turn INTEGER,
|
||||
volunteered_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS context_volunteer_events_src_time_idx
|
||||
ON context_volunteer_events (source_id, volunteered_at DESC);
|
||||
CREATE INDEX IF NOT EXISTS context_volunteer_events_src_slug_idx
|
||||
ON context_volunteer_events (source_id, slug);
|
||||
|
||||
-- ============================================================
|
||||
-- migration_impact_log (v0.41.18.0 — gbrain onboard wave)
|
||||
-- ============================================================
|
||||
|
||||
@@ -711,6 +711,28 @@ CREATE TABLE IF NOT EXISTS op_checkpoint_paths (
|
||||
FOREIGN KEY (op, fingerprint) REFERENCES op_checkpoints (op, fingerprint) ON DELETE CASCADE
|
||||
);
|
||||
|
||||
-- #2095 push-based context: feedback-loop log of volunteered pages
|
||||
-- (volunteer_context op / retrieval-reflex pointers / gbrain watch).
|
||||
-- "Used" derives from pages.last_retrieved_at > volunteered_at; rationale is
|
||||
-- a deterministic template string, never raw conversation text. 90-day prune
|
||||
-- in the dream cycle's purge phase. Mirrors migration v117.
|
||||
CREATE TABLE IF NOT EXISTS context_volunteer_events (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
source_id TEXT NOT NULL,
|
||||
slug TEXT NOT NULL,
|
||||
confidence DOUBLE PRECISION NOT NULL,
|
||||
match_arm TEXT NOT NULL,
|
||||
rationale TEXT NOT NULL DEFAULT '',
|
||||
channel TEXT NOT NULL DEFAULT 'op',
|
||||
session_id TEXT,
|
||||
turn INTEGER,
|
||||
volunteered_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS context_volunteer_events_src_time_idx
|
||||
ON context_volunteer_events (source_id, volunteered_at DESC);
|
||||
CREATE INDEX IF NOT EXISTS context_volunteer_events_src_slug_idx
|
||||
ON context_volunteer_events (source_id, slug);
|
||||
|
||||
-- migration_impact_log moved BELOW minion_jobs (was here, lines 645-676)
|
||||
-- because its \`job_id BIGINT REFERENCES minion_jobs(id)\` FK requires
|
||||
-- minion_jobs to exist FIRST during SCHEMA_SQL replay. v0.41.25.0 fix.
|
||||
|
||||
+19
-8
@@ -13,7 +13,7 @@ import {
|
||||
startResolveIpcServer,
|
||||
cleanupStaleSocket,
|
||||
} from '../core/context/resolve-ipc.ts';
|
||||
import { resolveEntitiesToPointers } from '../core/context/retrieval-reflex.ts';
|
||||
import { resolveEntitiesToPointers, logDeliveredReflexPointers } from '../core/context/retrieval-reflex.ts';
|
||||
|
||||
export async function startMcpServer(engine: BrainEngine) {
|
||||
const server = new Server(
|
||||
@@ -68,13 +68,24 @@ export async function startMcpServer(engine: BrainEngine) {
|
||||
if (cfg?.engine === 'pglite' && cfg.database_path) {
|
||||
resolveSocket = resolveSocketPath(cfg.database_path);
|
||||
const defaultSource = process.env.GBRAIN_SOURCE || 'default';
|
||||
resolveServer = await startResolveIpcServer(resolveSocket, (req) =>
|
||||
resolveEntitiesToPointers(
|
||||
engine,
|
||||
req.sourceId || defaultSource,
|
||||
req.candidates ?? [],
|
||||
{ priorContextText: req.priorContextText, maxPointers: req.maxPointers },
|
||||
),
|
||||
resolveServer = await startResolveIpcServer(
|
||||
resolveSocket,
|
||||
(req) =>
|
||||
resolveEntitiesToPointers(
|
||||
engine,
|
||||
req.sourceId || defaultSource,
|
||||
req.candidates ?? [],
|
||||
{
|
||||
priorContextText: req.priorContextText,
|
||||
maxPointers: req.maxPointers,
|
||||
suppression: req.suppression,
|
||||
},
|
||||
),
|
||||
// The IPC resolve path IS the ambient reflex channel. Logging happens
|
||||
// at DELIVERY (post-write), not inside the resolver — a block the
|
||||
// client's 250ms budget abandoned was never injected, and counting it
|
||||
// would corrupt the volunteered-vs-used precision stats (red-team).
|
||||
(block) => logDeliveredReflexPointers(engine, block.pointers),
|
||||
);
|
||||
}
|
||||
} catch {
|
||||
|
||||
@@ -707,6 +707,28 @@ CREATE TABLE IF NOT EXISTS op_checkpoint_paths (
|
||||
FOREIGN KEY (op, fingerprint) REFERENCES op_checkpoints (op, fingerprint) ON DELETE CASCADE
|
||||
);
|
||||
|
||||
-- #2095 push-based context: feedback-loop log of volunteered pages
|
||||
-- (volunteer_context op / retrieval-reflex pointers / gbrain watch).
|
||||
-- "Used" derives from pages.last_retrieved_at > volunteered_at; rationale is
|
||||
-- a deterministic template string, never raw conversation text. 90-day prune
|
||||
-- in the dream cycle's purge phase. Mirrors migration v117.
|
||||
CREATE TABLE IF NOT EXISTS context_volunteer_events (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
source_id TEXT NOT NULL,
|
||||
slug TEXT NOT NULL,
|
||||
confidence DOUBLE PRECISION NOT NULL,
|
||||
match_arm TEXT NOT NULL,
|
||||
rationale TEXT NOT NULL DEFAULT '',
|
||||
channel TEXT NOT NULL DEFAULT 'op',
|
||||
session_id TEXT,
|
||||
turn INTEGER,
|
||||
volunteered_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS context_volunteer_events_src_time_idx
|
||||
ON context_volunteer_events (source_id, volunteered_at DESC);
|
||||
CREATE INDEX IF NOT EXISTS context_volunteer_events_src_slug_idx
|
||||
ON context_volunteer_events (source_id, slug);
|
||||
|
||||
-- migration_impact_log moved BELOW minion_jobs (was here, lines 645-676)
|
||||
-- because its `job_id BIGINT REFERENCES minion_jobs(id)` FK requires
|
||||
-- minion_jobs to exist FIRST during SCHEMA_SQL replay. v0.41.25.0 fix.
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
/**
|
||||
* #2084 class pin — every exit-code write in src/ routes through
|
||||
* setCliExitVerdict.
|
||||
*
|
||||
* A RAW `process.exitCode = N` is silently ZEROED by the deliberate
|
||||
* flush-exit: currentExitCode() reads only gbrain's owned verdict (the
|
||||
* PGLite-Emscripten-pollution defense), so a command that bypasses the
|
||||
* setter reports success on failure. Caught live twice in one day: doctor's
|
||||
* FAIL path exited 0 after a merge introduced a raw write. Runtime variants
|
||||
* are pinned in test/cli-finish-teardown.test.ts; this is the structural
|
||||
* guard that catches the NEXT raw write at review time.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { execSync } from 'child_process';
|
||||
|
||||
describe('exit-verdict ownership — no raw process.exitCode assignments', () => {
|
||||
test('every exit-code write in src/ routes through setCliExitVerdict', () => {
|
||||
// Two legitimate writers are exempt:
|
||||
// - cli-force-exit.ts: setCliExitVerdict's own mirror-write.
|
||||
// - pglite-engine.ts preservingProcessExitCode: #2141's containment
|
||||
// RESTORE around PGlite.create() — it keeps the GLOBAL tidy for
|
||||
// external readers and is explicitly not a verdict write (the owned
|
||||
// channel never reads process.exitCode).
|
||||
// Whitespace/operator-tolerant: catches `process.exitCode=1`,
|
||||
// `process.exitCode ??= 1`, `process.exitCode ||= 1`, and the bracket
|
||||
// form — every shape is equally zeroed by the owned-verdict read.
|
||||
const hits = execSync(
|
||||
String.raw`grep -rnE "process(\.|\[')exitCode('\])?[[:space:]]*([?|&]{2})?=[^=]" src --include='*.ts' | grep -v "core/cli-force-exit.ts" | grep -v "core/pglite-engine.ts" || true`,
|
||||
{ encoding: 'utf-8', cwd: new URL('..', import.meta.url).pathname },
|
||||
).trim();
|
||||
expect(hits).toBe('');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,67 @@
|
||||
/**
|
||||
* v0.43 (#2095) — formatResult's volunteer_context human rendering
|
||||
* (ship coverage G3): pointer lines with confidence/arm/rationale,
|
||||
* the empty-result message, and the approximate stats summary.
|
||||
*/
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { formatResult } from '../src/cli.ts';
|
||||
|
||||
describe('formatResult — volunteer_context', () => {
|
||||
test('renders pointer lines with confidence, arm, rationale, and synopsis', () => {
|
||||
const out = formatResult('volunteer_context', {
|
||||
pages: [
|
||||
{
|
||||
slug: 'people/alice-example',
|
||||
source_id: 'default',
|
||||
display: 'Alice Example',
|
||||
confidence: 0.85,
|
||||
arm: 'title',
|
||||
rationale: 'exact title match "Alice Example"; mentioned in the newest turn',
|
||||
synopsis: 'Alice is an early founder.',
|
||||
},
|
||||
],
|
||||
count: 1,
|
||||
window_turns: 3,
|
||||
});
|
||||
expect(out).toContain('Alice Example → people/alice-example (0.85, title)');
|
||||
expect(out).toContain('exact title match');
|
||||
expect(out).toContain('Alice is an early founder.');
|
||||
});
|
||||
|
||||
test('empty result explains the confidence gate', () => {
|
||||
const out = formatResult('volunteer_context', { pages: [], count: 0, window_turns: 1 });
|
||||
expect(out).toContain('Nothing volunteered');
|
||||
expect(out).toContain('confidence gate');
|
||||
});
|
||||
|
||||
test('stats mode renders totals, per-arm precision, and the approximate note', () => {
|
||||
const out = formatResult('volunteer_context', {
|
||||
days: 30,
|
||||
approximate: true,
|
||||
note: 'approximate: "used" = pages.last_retrieved_at > volunteered_at.',
|
||||
total_volunteered: 4,
|
||||
total_used: 3,
|
||||
by_arm: [
|
||||
{ match_arm: 'alias', channel: 'reflex', volunteered: 2, used: 2, precision: 1 },
|
||||
{ match_arm: 'title', channel: 'op', volunteered: 2, used: 1, precision: 0.5 },
|
||||
],
|
||||
});
|
||||
expect(out).toContain('last 30 day(s)');
|
||||
expect(out).toContain('approximate');
|
||||
expect(out).toContain('total: 4 volunteered, 3 used');
|
||||
expect(out).toContain('alias/reflex: 2/2 used (precision 1)');
|
||||
expect(out).toContain('title/op: 1/2 used (precision 0.5)');
|
||||
});
|
||||
|
||||
test('stats mode with zero events prints the empty-window line', () => {
|
||||
const out = formatResult('volunteer_context', {
|
||||
days: 7,
|
||||
approximate: true,
|
||||
note: 'approximate',
|
||||
total_volunteered: 0,
|
||||
total_used: 0,
|
||||
by_arm: [],
|
||||
});
|
||||
expect(out).toContain('no volunteer events in the window');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,47 @@
|
||||
/**
|
||||
* v0.43 (#2084) — real-CLI pipe completeness pin (the incident #1959 class).
|
||||
*
|
||||
* The synthetic flush-mechanism coverage lives in
|
||||
* test/flush-then-exit-harness.test.ts (4MB late-reader byte-complete pin).
|
||||
* This file keeps the IMPLEMENTATION-AGNOSTIC check: the actual CLI, run the
|
||||
* way agents run it (piped stdout), produces complete, parseable, byte-stable
|
||||
* output and exits deliberately — well under the teardown backstop.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { spawnSync } from 'child_process';
|
||||
import { join, resolve } from 'path';
|
||||
|
||||
const REPO = resolve(import.meta.dir, '..');
|
||||
const CLI = join(REPO, 'src', 'cli.ts');
|
||||
|
||||
describe('cli pipe completeness — deliberate exit never truncates piped stdout (#2084)', () => {
|
||||
test('real CLI: --tools-json over a pipe is complete, parseable, byte-stable, and prompt', () => {
|
||||
const run = () => {
|
||||
const t0 = Date.now();
|
||||
const res = spawnSync('bun', [CLI, '--tools-json'], {
|
||||
stdio: ['ignore', 'pipe', 'pipe'],
|
||||
encoding: 'utf-8',
|
||||
timeout: 60_000,
|
||||
env: { ...process.env, GBRAIN_SKIP_STARTUP_HOOKS: '1' },
|
||||
maxBuffer: 64 * 1024 * 1024,
|
||||
});
|
||||
return { stdout: res.stdout ?? '', stderr: res.stderr ?? '', status: res.status, ms: Date.now() - t0 };
|
||||
};
|
||||
const first = run();
|
||||
expect(first.status).toBe(0);
|
||||
expect(Buffer.byteLength(first.stdout, 'utf-8')).toBeGreaterThan(16 * 1024);
|
||||
// Truncated JSON does not parse — the strongest single-run completeness check.
|
||||
const parsed = JSON.parse(first.stdout);
|
||||
expect(Array.isArray(parsed)).toBe(true);
|
||||
// Deliberate exit, not the teardown backstop. A wall-clock bound is flaky
|
||||
// on cold CI (bun parse alone runs 10-20s there) — the backstop's banner
|
||||
// is the truthful signal, same assertion the pgbouncer e2e uses.
|
||||
expect(first.stderr).not.toContain('force-exiting');
|
||||
expect(first.stderr).not.toContain('did not return within');
|
||||
|
||||
const second = run();
|
||||
expect(second.status).toBe(0);
|
||||
expect(second.stdout).toBe(first.stdout);
|
||||
}, 180_000);
|
||||
});
|
||||
@@ -82,4 +82,20 @@ describe('shouldForceExitAfterMain — daemon survival gate', () => {
|
||||
expect(shouldForceExitAfterMain(['serves'])).toBe(true);
|
||||
expect(shouldForceExitAfterMain(['serve-cluster'])).toBe(true);
|
||||
});
|
||||
|
||||
test('awaited long-runners exit deliberately when their handler resolves', () => {
|
||||
// `jobs work`, `jobs watch --follow`, `autopilot`, and `gbrain watch`
|
||||
// (#2095) all BLOCK inside their awaited handler until done — when
|
||||
// main() resolves for them, the work is over and the deliberate exit is
|
||||
// correct (v0.43 #2084 contract). Only commands that RETURN from main()
|
||||
// while the event loop carries the daemon (`serve`) belong in
|
||||
// DAEMON_COMMANDS — `watch` blocks in its stdin iteration, so piped EOF
|
||||
// must flow through the flush-exit instead of hanging on lingering
|
||||
// sockets.
|
||||
expect(shouldForceExitAfterMain(['jobs', 'work'])).toBe(true);
|
||||
expect(shouldForceExitAfterMain(['jobs', 'watch', '--follow'])).toBe(true);
|
||||
expect(shouldForceExitAfterMain(['autopilot'])).toBe(true);
|
||||
expect(shouldForceExitAfterMain(['watch'])).toBe(true);
|
||||
expect(shouldForceExitAfterMain(['watch', '--json'])).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -26,7 +26,10 @@ describe('resolve IPC', () => {
|
||||
test('round-trip: client gets the pointer block the server returns', async () => {
|
||||
const dir = tmpDir();
|
||||
const sock = resolveSocketPath(dir);
|
||||
const block: PointerBlock = { pointers: [{ display: 'Alice', slug: 'people/alice', synopsis: 'x' }], text: 'BLOCK' };
|
||||
const block: PointerBlock = {
|
||||
pointers: [{ display: 'Alice', slug: 'people/alice', source_id: 'default', synopsis: 'x', arm: 'alias', confidence: 0.9 }],
|
||||
text: 'BLOCK',
|
||||
};
|
||||
const server = await startResolveIpcServer(sock, async (req) => {
|
||||
expect(req.candidates[0].query).toBe('Alice');
|
||||
return block;
|
||||
|
||||
@@ -8,11 +8,24 @@
|
||||
*/
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { configureGateway } from '../src/core/ai/gateway.ts';
|
||||
import { checkFederationHealth } from '../src/commands/doctor.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
// Pin the legacy OpenAI/1536 embedding shape BEFORE initSchema builds the
|
||||
// content_chunks vector column. beforeAll runs before the legacy-embedding
|
||||
// preload's restoring beforeEach fires, so the gateway here is whatever the
|
||||
// PRIOR file in this shard left it — if that file configured a 1280-d model
|
||||
// and didn't reset, initSchema would build a vector(1280) column and the
|
||||
// 1536-d coverage fixture below would fail CheckExpectedDim. Establish the
|
||||
// dimension this file's fixtures assume rather than inheriting it.
|
||||
configureGateway({
|
||||
embedding_model: 'openai:text-embedding-3-large',
|
||||
embedding_dimensions: 1536,
|
||||
env: { ...process.env },
|
||||
});
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
|
||||
@@ -49,6 +49,9 @@ const ALL_TABLES = [
|
||||
'page_versions',
|
||||
'ingest_log',
|
||||
'files',
|
||||
// v0.43 (#2095): volunteered-context feedback log — no FK to pages (slug
|
||||
// join), but stale rows poison stats/count assertions across runs.
|
||||
'context_volunteer_events',
|
||||
'pages', // last because of foreign keys
|
||||
'config',
|
||||
'minion_attachments',
|
||||
|
||||
@@ -0,0 +1,148 @@
|
||||
/**
|
||||
* v0.43 (#2084 / eng-review TD1) — PgBouncer transaction-mode teardown E2E.
|
||||
*
|
||||
* Three consecutive waves (#1972 → #2015 → #2084) fixed pooler-teardown bugs
|
||||
* that were verified only against one production deployment, because CI had
|
||||
* no transaction-mode pooler. This file pins the bug CLASS, not exact
|
||||
* timings: a CLI op against a txn-mode pooled URL must
|
||||
*
|
||||
* 1. exit zero with intact stdout, and
|
||||
* 2. NOT ride the 10s hard-deadline backstop (the
|
||||
* "[cli] engine.disconnect() did not return within 10000ms" banner is
|
||||
* the smoking gun — pre-#2084 it printed on 100% of query-shaped ops).
|
||||
*
|
||||
* Topology: docker-compose.ci.yml runs `pgbouncer` (transaction mode) in
|
||||
* front of postgres-1. The test uses a DEDICATED database
|
||||
* (`gbrain_pgbouncer`) created via the direct URL, so it never races the
|
||||
* TRUNCATE-based fixtures any shard runs against `gbrain_test`.
|
||||
*
|
||||
* Gated by GBRAIN_PGBOUNCER_URL + GBRAIN_PGBOUNCER_DIRECT_URL — skips
|
||||
* gracefully outside the docker CI gate. Run manually:
|
||||
*
|
||||
* GBRAIN_PGBOUNCER_URL=postgresql://postgres:postgres@localhost:6543/gbrain_pgbouncer \
|
||||
* GBRAIN_PGBOUNCER_DIRECT_URL=postgresql://postgres:postgres@localhost:5434/gbrain_test \
|
||||
* bun test test/e2e/pgbouncer-teardown.test.ts
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from 'fs';
|
||||
import { join, resolve } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import postgres from 'postgres';
|
||||
import { PostgresEngine } from '../../src/core/postgres-engine.ts';
|
||||
|
||||
const POOLED_URL = process.env.GBRAIN_PGBOUNCER_URL;
|
||||
const DIRECT_ADMIN_URL = process.env.GBRAIN_PGBOUNCER_DIRECT_URL;
|
||||
const SKIP = !POOLED_URL || !DIRECT_ADMIN_URL;
|
||||
const describePooled = SKIP ? describe.skip : describe;
|
||||
|
||||
const REPO = resolve(import.meta.dir, '..', '..');
|
||||
const TEST_DB = 'gbrain_pgbouncer';
|
||||
const SLUG = 'test/pgbouncer-teardown-fixture';
|
||||
const MARKER = 'pgbouncer-teardown-marker-content-7c4f';
|
||||
|
||||
/** Direct URL pointing at the dedicated test database (same server). */
|
||||
function directTestDbUrl(): string {
|
||||
const u = new URL(DIRECT_ADMIN_URL!);
|
||||
u.pathname = `/${TEST_DB}`;
|
||||
return u.toString();
|
||||
}
|
||||
|
||||
async function runCli(
|
||||
args: string[],
|
||||
env: Record<string, string>,
|
||||
timeoutMs: number,
|
||||
): Promise<{ exitCode: number; stdout: string; stderr: string; wallMs: number }> {
|
||||
const t0 = Date.now();
|
||||
const proc = Bun.spawn(['bun', 'run', join(REPO, 'src', 'cli.ts'), ...args], {
|
||||
cwd: REPO,
|
||||
env: { ...process.env, ...env, GBRAIN_SKIP_STARTUP_HOOKS: '1' },
|
||||
stdout: 'pipe',
|
||||
stderr: 'pipe',
|
||||
});
|
||||
const killer = setTimeout(() => {
|
||||
try { proc.kill('SIGKILL'); } catch { /* already dead */ }
|
||||
}, timeoutMs);
|
||||
try {
|
||||
const [stdout, stderr, exitCode] = await Promise.all([
|
||||
new Response(proc.stdout).text(),
|
||||
new Response(proc.stderr).text(),
|
||||
proc.exited,
|
||||
]);
|
||||
return { exitCode, stdout, stderr, wallMs: Date.now() - t0 };
|
||||
} finally {
|
||||
clearTimeout(killer);
|
||||
}
|
||||
}
|
||||
|
||||
describePooled('pgbouncer txn-mode teardown (#2084 / TD1)', () => {
|
||||
let home: string;
|
||||
|
||||
beforeAll(async () => {
|
||||
// Dedicated database on the same server, created via the DIRECT url
|
||||
// (CREATE DATABASE cannot run through a transaction-mode pooler).
|
||||
const admin = postgres(DIRECT_ADMIN_URL!, { max: 1 });
|
||||
try {
|
||||
await admin.unsafe(`DROP DATABASE IF EXISTS ${TEST_DB} WITH (FORCE)`);
|
||||
await admin.unsafe(`CREATE DATABASE ${TEST_DB}`);
|
||||
} finally {
|
||||
await admin.end({ timeout: 5 });
|
||||
}
|
||||
|
||||
// Schema + fixture via the direct connection (DDL stays off the pooler,
|
||||
// matching the production split-pool discipline).
|
||||
const eng = new PostgresEngine();
|
||||
await eng.connect({ engine: 'postgres', database_url: directTestDbUrl() });
|
||||
await eng.initSchema();
|
||||
await eng.putPage(SLUG, {
|
||||
type: 'note',
|
||||
title: 'PgBouncer teardown fixture',
|
||||
compiled_truth: MARKER,
|
||||
timeline: '',
|
||||
});
|
||||
await eng.disconnect();
|
||||
|
||||
// Brain config pointing the CLI at the POOLED url.
|
||||
home = mkdtempSync(join(tmpdir(), 'gbrain-pgbouncer-'));
|
||||
mkdirSync(join(home, '.gbrain'), { recursive: true });
|
||||
const pooled = new URL(POOLED_URL!);
|
||||
pooled.pathname = `/${TEST_DB}`;
|
||||
writeFileSync(
|
||||
join(home, '.gbrain', 'config.json'),
|
||||
JSON.stringify({ engine: 'postgres', database_url: pooled.toString() }) + '\n',
|
||||
);
|
||||
}, 240_000);
|
||||
|
||||
afterAll(() => {
|
||||
try { rmSync(home, { recursive: true, force: true }); } catch { /* best effort */ }
|
||||
});
|
||||
|
||||
test('op against the pooled URL exits clean — output intact, no force-exit banner', async () => {
|
||||
const env = { HOME: home, GBRAIN_HOME: home };
|
||||
const res = await runCli(['get', SLUG], env, 90_000);
|
||||
|
||||
if (res.exitCode !== 0 || /force-exiting/.test(res.stderr)) {
|
||||
console.error('--- stdout ---\n' + res.stdout);
|
||||
console.error('--- stderr ---\n' + res.stderr);
|
||||
}
|
||||
expect(res.exitCode).toBe(0);
|
||||
// Output is complete — the #1959 truncation class.
|
||||
expect(res.stdout).toContain(MARKER);
|
||||
// The smoking gun: pre-#2084 the hard-deadline banner printed every time
|
||||
// a query-shaped op ran against a txn-mode pooler.
|
||||
expect(res.stderr).not.toMatch(/force-exiting/);
|
||||
expect(res.stderr).not.toMatch(/did not return within/);
|
||||
// Generous CLASS bound (cold bun parse on CI is 10-20s): the op itself is
|
||||
// milliseconds; anything that ALSO waited out a 10s teardown backstop
|
||||
// lands well past this.
|
||||
expect(res.wallMs).toBeLessThan(60_000);
|
||||
}, 120_000);
|
||||
|
||||
test('second run (warm schema probe) also exits clean through the pooler', async () => {
|
||||
const env = { HOME: home, GBRAIN_HOME: home };
|
||||
const res = await runCli(['get', SLUG], env, 90_000);
|
||||
expect(res.exitCode).toBe(0);
|
||||
expect(res.stdout).toContain(MARKER);
|
||||
expect(res.stderr).not.toMatch(/force-exiting/);
|
||||
}, 120_000);
|
||||
});
|
||||
@@ -0,0 +1,97 @@
|
||||
/**
|
||||
* v0.43 (#2095) — volunteer_context on REAL Postgres (engine parity beyond
|
||||
* PGLite) + the RLS pin for the v117 table.
|
||||
*
|
||||
* The unit suite covers the volunteer core hermetically on PGLite; this file
|
||||
* proves the same op handler against pgvector Postgres: resolution arms,
|
||||
* the fire-and-forget event sink landing rows, the stats join, and that
|
||||
* context_volunteer_events (migration v117) has ROW LEVEL SECURITY enabled (the v35
|
||||
* auto_rls_on_create_table event trigger covers migration-created tables —
|
||||
* this is the assertion that keeps that mechanism honest for new tables).
|
||||
*
|
||||
* Gated by DATABASE_URL — skips gracefully without real Postgres.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { hasDatabase, setupDB, teardownDB, getEngine } from './helpers.ts';
|
||||
import { operationsByName } from '../../src/core/operations.ts';
|
||||
import {
|
||||
awaitPendingVolunteerEventWrites,
|
||||
_resetPendingVolunteerEventWritesForTests,
|
||||
} from '../../src/core/context/volunteer-events.ts';
|
||||
|
||||
const SKIP = !hasDatabase();
|
||||
const describePg = SKIP ? describe.skip : describe;
|
||||
|
||||
function mkCtx(engine: unknown, overrides: Record<string, unknown> = {}) {
|
||||
return {
|
||||
engine,
|
||||
config: {} as never,
|
||||
logger: { info: () => {}, warn: () => {}, error: () => {} } as never,
|
||||
dryRun: false,
|
||||
remote: false,
|
||||
sourceId: 'default',
|
||||
...overrides,
|
||||
} as never;
|
||||
}
|
||||
|
||||
describePg('volunteer_context on real Postgres (#2095)', () => {
|
||||
beforeAll(async () => {
|
||||
await setupDB();
|
||||
const engine = getEngine();
|
||||
await engine.putPage('people/alice-example', {
|
||||
type: 'person',
|
||||
title: 'Alice Example',
|
||||
compiled_truth: 'Alice is an early founder.',
|
||||
timeline: '',
|
||||
});
|
||||
}, 240_000);
|
||||
|
||||
afterAll(async () => {
|
||||
await teardownDB();
|
||||
});
|
||||
|
||||
test('op volunteers, logs through the sink, and stats join works', async () => {
|
||||
_resetPendingVolunteerEventWritesForTests();
|
||||
const engine = getEngine();
|
||||
const op = operationsByName.volunteer_context;
|
||||
|
||||
const result = (await op.handler(mkCtx(engine), {
|
||||
window: 'user: who knows the seed market?\nassistant: Alice Example does.\nuser: ask her',
|
||||
session_id: 'e2e-pg',
|
||||
})) as any;
|
||||
expect(result.count).toBe(1);
|
||||
expect(result.pages[0].slug).toBe('people/alice-example');
|
||||
expect(result.pages[0].arm).toBe('title');
|
||||
|
||||
const { unfinished } = await awaitPendingVolunteerEventWrites(10_000);
|
||||
expect(unfinished).toBe(0);
|
||||
|
||||
const rows = await engine.executeRaw<{ slug: string; session_id: string }>(
|
||||
`SELECT slug, session_id FROM context_volunteer_events WHERE session_id = 'e2e-pg'`,
|
||||
[],
|
||||
);
|
||||
expect(rows.length).toBe(1);
|
||||
|
||||
// Mark the page as opened after volunteering → stats counts it used.
|
||||
await engine.executeRaw(
|
||||
`UPDATE pages SET last_retrieved_at = now() + interval '1 minute' WHERE slug = 'people/alice-example'`,
|
||||
[],
|
||||
);
|
||||
const stats = (await op.handler(mkCtx(engine), { stats: true })) as any;
|
||||
expect(stats.approximate).toBe(true);
|
||||
expect(stats.total_volunteered).toBeGreaterThanOrEqual(1);
|
||||
expect(stats.total_used).toBeGreaterThanOrEqual(1);
|
||||
}, 120_000);
|
||||
|
||||
test('RLS is enabled on context_volunteer_events (auto-RLS covers v117)', async () => {
|
||||
const engine = getEngine();
|
||||
const rows = await engine.executeRaw<{ relrowsecurity: boolean }>(
|
||||
`SELECT relrowsecurity FROM pg_class
|
||||
WHERE oid = 'public.context_volunteer_events'::regclass`,
|
||||
[],
|
||||
);
|
||||
expect(rows.length).toBe(1);
|
||||
expect(rows[0].relrowsecurity).toBe(true);
|
||||
}, 60_000);
|
||||
});
|
||||
@@ -250,3 +250,20 @@ describe('v0.41.8.0 #1340 — PGLite WASM init classifier', () => {
|
||||
expect(src).toMatch(/buildPgliteInitErrorMessage\(verdict, original\)/);
|
||||
});
|
||||
});
|
||||
|
||||
describe('v0.42.43.0 #2095 — volunteer-events sink + cycle purge wiring (structural pins)', () => {
|
||||
test('volunteer-events registers a background-work drainer (order 4)', () => {
|
||||
// Deleting this registration would silently drop volunteer events on
|
||||
// every CLI exit with no behavioral test failing — same pin class as the
|
||||
// other four sinks above.
|
||||
const src = readFileSync('src/core/context/volunteer-events.ts', 'utf8');
|
||||
expect(src).toMatch(/registerBackgroundWorkDrainer\(\{[\s\S]*?name:\s*'volunteer-events'/);
|
||||
expect(src).toMatch(/order:\s*4/);
|
||||
});
|
||||
|
||||
test("the dream cycle's purge phase invokes purgeStaleVolunteerEvents and reports the count", () => {
|
||||
const src = readFileSync('src/core/cycle.ts', 'utf8');
|
||||
expect(src).toMatch(/purgeStaleVolunteerEvents\(engine\)/);
|
||||
expect(src).toMatch(/purged_volunteer_events_count/);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -2179,3 +2179,67 @@ describe('v112 — pages_links_extracted_at', () => {
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
// v0.43 (#2095): context_volunteer_events — push-based-context feedback log.
|
||||
describe('v117 — context_volunteer_events_table', () => {
|
||||
let engine: PGLiteEngine;
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
}, 60_000);
|
||||
afterAll(async () => { if (engine) await engine.disconnect(); }, 60_000);
|
||||
|
||||
test('v117 entry exists, named + idempotent', () => {
|
||||
const m = MIGRATIONS.find(x => x.version === 117);
|
||||
expect(m).toBeDefined();
|
||||
expect(m!.name).toBe('context_volunteer_events_table');
|
||||
expect(m!.idempotent).toBe(true);
|
||||
});
|
||||
|
||||
test('LATEST_VERSION is at or above 117', () => {
|
||||
expect(LATEST_VERSION).toBeGreaterThanOrEqual(117);
|
||||
});
|
||||
|
||||
test('table exists after initSchema with the documented columns', async () => {
|
||||
const rows = await engine.executeRaw<{ column_name: string; is_nullable: string }>(
|
||||
`SELECT column_name, is_nullable FROM information_schema.columns
|
||||
WHERE table_name = 'context_volunteer_events' ORDER BY ordinal_position`, [],
|
||||
);
|
||||
const byName = new Map(rows.map(r => [r.column_name, r.is_nullable]));
|
||||
for (const required of ['source_id', 'slug', 'confidence', 'match_arm', 'rationale', 'channel', 'volunteered_at']) {
|
||||
expect(byName.get(required)).toBe('NO');
|
||||
}
|
||||
// Caller-supplied attribution columns are nullable by contract (codex D9).
|
||||
expect(byName.get('session_id')).toBe('YES');
|
||||
expect(byName.get('turn')).toBe('YES');
|
||||
});
|
||||
|
||||
test('both source-scoped indexes exist after initSchema', async () => {
|
||||
const rows = await engine.executeRaw<{ indexname: string }>(
|
||||
`SELECT indexname FROM pg_indexes WHERE tablename = 'context_volunteer_events'`, [],
|
||||
);
|
||||
const names = rows.map(r => r.indexname);
|
||||
expect(names).toContain('context_volunteer_events_src_time_idx');
|
||||
expect(names).toContain('context_volunteer_events_src_slug_idx');
|
||||
});
|
||||
|
||||
test('insert + 90-day purge round-trip (purgeStaleVolunteerEvents)', async () => {
|
||||
const { insertVolunteerEvents, purgeStaleVolunteerEvents } = await import('../src/core/context/volunteer-events.ts');
|
||||
await insertVolunteerEvents(engine, [
|
||||
{ source_id: 'default', slug: 'people/alice-example', confidence: 0.9, match_arm: 'alias', rationale: 'alias match', channel: 'op' },
|
||||
{ source_id: 'default', slug: 'projects/acme-example', confidence: 0.8, match_arm: 'title', rationale: 'exact title', channel: 'reflex', session_id: 's1', turn: 3 },
|
||||
]);
|
||||
// Age one row past the TTL, leave the other fresh.
|
||||
await engine.executeRaw(
|
||||
`UPDATE context_volunteer_events SET volunteered_at = now() - interval '120 days'
|
||||
WHERE slug = 'projects/acme-example'`, [],
|
||||
);
|
||||
const purged = await purgeStaleVolunteerEvents(engine, 90);
|
||||
expect(purged).toBe(1);
|
||||
const left = await engine.executeRaw<{ slug: string }>(
|
||||
`SELECT slug FROM context_volunteer_events`, [],
|
||||
);
|
||||
expect(left.map(r => r.slug)).toEqual(['people/alice-example']);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -182,3 +182,258 @@ describe('context-engine assemble() — Retrieval Reflex integration', () => {
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('v0.43 (#2095) — rolling window extraction through assemble()', () => {
|
||||
const REFLEX_ON = { GBRAIN_RETRIEVAL_REFLEX: 'true' };
|
||||
|
||||
test('entity named ONLY in a previous assistant turn yields a pointer now', async () => {
|
||||
await withEnv(REFLEX_ON, async () => {
|
||||
await seed('people/alice-example', 'Alice Example', 'Alice is a founder.');
|
||||
const ce = createGBrainContextEngine({
|
||||
workspaceDir: '/tmp/rr-test-ws-w1',
|
||||
resolveEntities: (candidates, opts) =>
|
||||
resolveEntitiesToPointers(engine, 'default', candidates, opts),
|
||||
});
|
||||
// Current turn is a pronoun follow-up; the antecedent was NAMED two
|
||||
// turns back by the ASSISTANT. Pre-window this never fired.
|
||||
const res = await ce.assemble({
|
||||
sessionId: 'w1',
|
||||
messages: [
|
||||
{ role: 'user', content: 'who should I talk to about the seed round?' },
|
||||
{ role: 'assistant', content: 'Alice Example led a similar round last year.' },
|
||||
{ role: 'user', content: 'what did she invest in?' },
|
||||
],
|
||||
});
|
||||
expect(res.systemPromptAddition).toContain('people/alice-example');
|
||||
});
|
||||
});
|
||||
|
||||
test('window=1 reproduces the legacy current-turn-only behavior', async () => {
|
||||
await withEnv({ ...REFLEX_ON, GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS: '1' }, async () => {
|
||||
await seed('people/alice-example', 'Alice Example', 'Alice is a founder.');
|
||||
const ce = createGBrainContextEngine({
|
||||
workspaceDir: '/tmp/rr-test-ws-w2',
|
||||
resolveEntities: (candidates, opts) =>
|
||||
resolveEntitiesToPointers(engine, 'default', candidates, opts),
|
||||
});
|
||||
const res = await ce.assemble({
|
||||
sessionId: 'w2',
|
||||
messages: [
|
||||
{ role: 'assistant', content: 'Alice Example led a similar round last year.' },
|
||||
{ role: 'user', content: 'what did she invest in?' },
|
||||
],
|
||||
});
|
||||
// Current turn has no extractable entity; window=1 must NOT widen.
|
||||
expect(res.systemPromptAddition).not.toContain('people/alice-example');
|
||||
});
|
||||
});
|
||||
|
||||
test('windowed suppression is slug-only: a prior-turn MENTION does not suppress (codex D7)', async () => {
|
||||
await withEnv(REFLEX_ON, async () => {
|
||||
await seed('people/alice-example', 'Alice Example', 'A founder.');
|
||||
const ce = createGBrainContextEngine({
|
||||
workspaceDir: '/tmp/rr-test-ws-w3',
|
||||
resolveEntities: (candidates, opts) =>
|
||||
resolveEntitiesToPointers(engine, 'default', candidates, opts),
|
||||
});
|
||||
// "Alice Example" appears in a PRIOR turn (a bare mention — prior
|
||||
// context contains the TITLE). Under the legacy title rule the pointer
|
||||
// would be suppressed; slug-only windowing must still fire.
|
||||
const res = await ce.assemble({
|
||||
sessionId: 'w3',
|
||||
messages: [
|
||||
{ role: 'user', content: 'I met Alice Example yesterday' },
|
||||
{ role: 'assistant', content: 'How did the meeting with Alice Example go?' },
|
||||
{ role: 'user', content: 'she wants to invest — thoughts?' },
|
||||
],
|
||||
});
|
||||
expect(res.systemPromptAddition).toContain('people/alice-example');
|
||||
});
|
||||
});
|
||||
|
||||
test('windowed suppression still drops an already-surfaced page (slug in prior context)', async () => {
|
||||
await withEnv(REFLEX_ON, async () => {
|
||||
await seed('people/alice-example', 'Alice Example', 'A founder.');
|
||||
const ce = createGBrainContextEngine({
|
||||
workspaceDir: '/tmp/rr-test-ws-w4',
|
||||
resolveEntities: (candidates, opts) =>
|
||||
resolveEntitiesToPointers(engine, 'default', candidates, opts),
|
||||
});
|
||||
const res = await ce.assemble({
|
||||
sessionId: 'w4',
|
||||
messages: [
|
||||
{ role: 'assistant', content: 'Pointer: **Alice Example** → `people/alice-example` (use get_page)' },
|
||||
{ role: 'user', content: 'tell me more about Alice Example' },
|
||||
],
|
||||
});
|
||||
expect(res.systemPromptAddition).not.toContain('Brain pages mentioned this turn');
|
||||
});
|
||||
});
|
||||
|
||||
test('fail-open: a throwing resolver under windowing never breaks the turn', async () => {
|
||||
await withEnv(REFLEX_ON, async () => {
|
||||
await seed('people/alice-example', 'Alice Example', 'A founder.');
|
||||
const ce = createGBrainContextEngine({
|
||||
workspaceDir: '/tmp/rr-test-ws-w5',
|
||||
resolveEntities: async () => { throw new Error('resolver exploded'); },
|
||||
});
|
||||
const res = await ce.assemble({
|
||||
sessionId: 'w5',
|
||||
messages: [
|
||||
{ role: 'assistant', content: 'Alice Example is relevant here.' },
|
||||
{ role: 'user', content: 'ok tell me about her' },
|
||||
],
|
||||
});
|
||||
expect(res.systemPromptAddition).toContain('Live Context');
|
||||
expect(res.systemPromptAddition).not.toContain('Brain pages mentioned this turn');
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('ambient-channel event logging (codex D11 — accept-side logDeliveredReflexPointers)', () => {
|
||||
test('logDeliveredReflexPointers logs channel=reflex events through the drained sink', async () => {
|
||||
const { logDeliveredReflexPointers } = await import('../src/core/context/retrieval-reflex.ts');
|
||||
const { awaitPendingVolunteerEventWrites, _resetPendingVolunteerEventWritesForTests } =
|
||||
await import('../src/core/context/volunteer-events.ts');
|
||||
_resetPendingVolunteerEventWritesForTests();
|
||||
await engine.executeRaw('DELETE FROM context_volunteer_events').catch(() => {});
|
||||
await seed('people/alice-example', 'Alice Example', 'A founder.');
|
||||
|
||||
const block = await resolveEntitiesToPointers(
|
||||
engine,
|
||||
'default',
|
||||
extractCandidates('what do you think about Alice Example?'),
|
||||
{},
|
||||
);
|
||||
expect(block).not.toBeNull();
|
||||
logDeliveredReflexPointers(engine, block!.pointers);
|
||||
const { unfinished } = await awaitPendingVolunteerEventWrites(5_000);
|
||||
expect(unfinished).toBe(0);
|
||||
const rows = await engine.executeRaw<{ channel: string; slug: string; match_arm: string }>(
|
||||
'SELECT channel, slug, match_arm FROM context_volunteer_events',
|
||||
[],
|
||||
);
|
||||
expect(rows.length).toBe(1);
|
||||
expect(rows[0].channel).toBe('reflex');
|
||||
expect(rows[0].slug).toBe('people/alice-example');
|
||||
expect(rows[0].match_arm).toBe('title');
|
||||
});
|
||||
|
||||
test('the bare resolver logs nothing — delivery is the only logging seam', async () => {
|
||||
await engine.executeRaw('DELETE FROM context_volunteer_events').catch(() => {});
|
||||
await seed('people/alice-example', 'Alice Example', 'A founder.');
|
||||
const block = await resolveEntitiesToPointers(
|
||||
engine,
|
||||
'default',
|
||||
extractCandidates('about Alice Example'),
|
||||
{},
|
||||
);
|
||||
expect(block).not.toBeNull();
|
||||
const { awaitPendingVolunteerEventWrites } = await import('../src/core/context/volunteer-events.ts');
|
||||
await awaitPendingVolunteerEventWrites(5_000);
|
||||
const rows = await engine.executeRaw<{ channel: string }>('SELECT channel FROM context_volunteer_events', []);
|
||||
expect(rows.length).toBe(0);
|
||||
});
|
||||
|
||||
test('logDeliveredReflexPointers with an empty pointer list is a no-op', async () => {
|
||||
const { logDeliveredReflexPointers } = await import('../src/core/context/retrieval-reflex.ts');
|
||||
const { awaitPendingVolunteerEventWrites } = await import('../src/core/context/volunteer-events.ts');
|
||||
await engine.executeRaw('DELETE FROM context_volunteer_events').catch(() => {});
|
||||
logDeliveredReflexPointers(engine, []);
|
||||
await awaitPendingVolunteerEventWrites(5_000);
|
||||
const rows = await engine.executeRaw<{ channel: string }>('SELECT channel FROM context_volunteer_events', []);
|
||||
expect(rows.length).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('serve IPC wiring — suppression passthrough + reflex-channel logging (review hardening)', () => {
|
||||
test('the IPC round-trip honors slug-only suppression and logs channel=reflex', async () => {
|
||||
const { startResolveIpcServer, resolveViaIpc, resolveSocketPath, IPC_UNAVAILABLE } =
|
||||
await import('../src/core/context/resolve-ipc.ts');
|
||||
const { awaitPendingVolunteerEventWrites, _resetPendingVolunteerEventWritesForTests } =
|
||||
await import('../src/core/context/volunteer-events.ts');
|
||||
const { mkdtempSync, rmSync } = await import('fs');
|
||||
const { join } = await import('path');
|
||||
const { tmpdir } = await import('os');
|
||||
|
||||
_resetPendingVolunteerEventWritesForTests();
|
||||
await engine.executeRaw('DELETE FROM context_volunteer_events').catch(() => {});
|
||||
await seed('people/alice-example', 'Alice Example', 'A founder.');
|
||||
|
||||
const dir = mkdtempSync(join(tmpdir(), 'rr-ipc-'));
|
||||
const sock = resolveSocketPath(dir);
|
||||
// The SAME wiring shape src/mcp/server.ts uses for serve: forwards
|
||||
// suppression from the request; logging happens at DELIVERY via the
|
||||
// onDelivered hook (post-write), never inside the resolver.
|
||||
const { logDeliveredReflexPointers } = await import('../src/core/context/retrieval-reflex.ts');
|
||||
const server = await startResolveIpcServer(
|
||||
sock,
|
||||
(req) =>
|
||||
resolveEntitiesToPointers(engine, req.sourceId || 'default', req.candidates ?? [], {
|
||||
priorContextText: req.priorContextText,
|
||||
maxPointers: req.maxPointers,
|
||||
suppression: req.suppression,
|
||||
}),
|
||||
(block) => logDeliveredReflexPointers(engine, block.pointers),
|
||||
);
|
||||
expect(server).not.toBeNull();
|
||||
try {
|
||||
// slug-only suppression: a TITLE mention in prior context must NOT
|
||||
// suppress (the windowing contract), and the resolve must log.
|
||||
const block = await resolveViaIpc(sock, {
|
||||
candidates: extractCandidates('tell me about Alice Example'),
|
||||
priorContextText: 'earlier turn merely mentioned Alice Example',
|
||||
suppression: 'slug-only',
|
||||
});
|
||||
expect(block).not.toBe(IPC_UNAVAILABLE);
|
||||
expect(block).not.toBeNull();
|
||||
expect((block as { pointers: Array<{ slug: string }> }).pointers[0].slug).toBe('people/alice-example');
|
||||
|
||||
const { unfinished } = await awaitPendingVolunteerEventWrites(5_000);
|
||||
expect(unfinished).toBe(0);
|
||||
const rows = await engine.executeRaw<{ channel: string }>(
|
||||
'SELECT channel FROM context_volunteer_events', [],
|
||||
);
|
||||
expect(rows.length).toBe(1);
|
||||
expect(rows[0].channel).toBe('reflex');
|
||||
} finally {
|
||||
server!.close();
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('windowTurnCount — knob edge semantics', () => {
|
||||
test('0, negative, NaN, and absent all fall back to the default of 4 (1 = legacy off)', async () => {
|
||||
const { windowTurnCount, DEFAULT_WINDOW_TURNS } = await import('../src/core/context/reflex.ts');
|
||||
expect(DEFAULT_WINDOW_TURNS).toBe(4);
|
||||
expect(windowTurnCount(null)).toBe(4);
|
||||
expect(windowTurnCount({ retrieval_reflex_window_turns: 0 } as never)).toBe(4);
|
||||
expect(windowTurnCount({ retrieval_reflex_window_turns: -3 } as never)).toBe(4);
|
||||
expect(windowTurnCount({ retrieval_reflex_window_turns: Number.NaN } as never)).toBe(4);
|
||||
// The documented "off" switch is 1 (legacy single-turn), not 0.
|
||||
expect(windowTurnCount({ retrieval_reflex_window_turns: 1 } as never)).toBe(1);
|
||||
expect(windowTurnCount({ retrieval_reflex_window_turns: 6.9 } as never)).toBe(6);
|
||||
});
|
||||
|
||||
test('the env escape hatch is honored even when config is null (no config file / DB)', async () => {
|
||||
const { windowTurnCount } = await import('../src/core/context/reflex.ts');
|
||||
// loadConfig() returns null in a config-less environment (clean CI shard,
|
||||
// no brain) and drops its env→config mapping — windowTurnCount must still
|
||||
// read the env var directly, or the documented escape hatch is dead and
|
||||
// the window silently defaults to 4. withEnv() (not raw process.env
|
||||
// mutation) keeps the linter + isolation guard happy.
|
||||
await withEnv({ GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS: '1' }, async () => {
|
||||
expect(windowTurnCount(null)).toBe(1);
|
||||
});
|
||||
await withEnv({ GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS: '7' }, async () => {
|
||||
expect(windowTurnCount(null)).toBe(7);
|
||||
// Env wins over a config value too (env is the higher-precedence plane).
|
||||
expect(windowTurnCount({ retrieval_reflex_window_turns: 3 } as never)).toBe(7);
|
||||
});
|
||||
await withEnv({ GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS: 'not-a-number' }, async () => {
|
||||
// Garbage env falls through to config / default, not a crash.
|
||||
expect(windowTurnCount(null)).toBe(4);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
@@ -13,7 +13,7 @@
|
||||
* envelope paths don't depend on DB state.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { describe, test, expect, beforeEach } from 'bun:test';
|
||||
import {
|
||||
EMBEDDING_COST_PER_1K_TOKENS,
|
||||
estimateEmbeddingCostUsd,
|
||||
@@ -21,9 +21,20 @@ import {
|
||||
shouldBlockSync,
|
||||
} from '../src/core/embedding.ts';
|
||||
import { lookupEmbeddingPrice } from '../src/core/embedding-pricing.ts';
|
||||
import { resetGateway } from '../src/core/ai/gateway.ts';
|
||||
import { estimateTokens } from '../src/core/chunkers/code.ts';
|
||||
|
||||
describe('Layer 8 D1 — embedding cost model', () => {
|
||||
// The estimateEmbeddingCostUsd tests assert the gateway-UNCONFIGURED OpenAI
|
||||
// fallback ($0.13/Mtok). That fallback only fires when no gateway is
|
||||
// configured — so guarantee that precondition rather than depending on
|
||||
// ambient state. Without this, a sibling test file in the same shard that
|
||||
// configured (and didn't reset) a cheaper model leaks its rate in here and
|
||||
// these assertions flip (the legacy-embedding preload only restores when the
|
||||
// gateway slot is EMPTY, so a non-empty foreign config survives).
|
||||
beforeEach(() => resetGateway());
|
||||
|
||||
|
||||
test('EMBEDDING_COST_PER_1K_TOKENS back-compat constant is the OpenAI 3-large rate', () => {
|
||||
// Retained only for back-compat imports. Live cost math now resolves the
|
||||
// CONFIGURED model's rate via embedding-pricing.ts (see model-aware test
|
||||
|
||||
@@ -0,0 +1,483 @@
|
||||
/**
|
||||
* v0.43 (#2095) — push-based context core: window parsing, multi-turn
|
||||
* extraction, confidence-gated volunteering, slug-only suppression, privacy,
|
||||
* and the usage-stats join. Hermetic in-memory PGLite (no file lock),
|
||||
* modeled on test/retrieval-reflex.test.ts.
|
||||
*/
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { normalizeAlias } from '../src/core/search/alias-normalize.ts';
|
||||
import {
|
||||
extractCandidatesFromWindow,
|
||||
MAX_CANDIDATES,
|
||||
type WindowTurn,
|
||||
} from '../src/core/context/entity-salience.ts';
|
||||
import { resolveEntitiesToPointers } from '../src/core/context/retrieval-reflex.ts';
|
||||
import {
|
||||
parseWindow,
|
||||
volunteerContext,
|
||||
volunteerUsageStats,
|
||||
VOLUNTEER_DEFAULT_MIN_CONFIDENCE,
|
||||
} from '../src/core/context/volunteer.ts';
|
||||
import { insertVolunteerEvents } from '../src/core/context/volunteer-events.ts';
|
||||
import { TAKES_FENCE_BEGIN, TAKES_FENCE_END } from '../src/core/takes-fence.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
async function seed(slug: string, title: string, body: string, source = 'default') {
|
||||
if (source !== 'default') {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, local_path) VALUES ($1, $1, $2)
|
||||
ON CONFLICT (id) DO NOTHING`,
|
||||
[source, `/tmp/${source}`],
|
||||
);
|
||||
}
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO pages (slug, source_id, type, title, compiled_truth, timeline)
|
||||
VALUES ($1, $2, 'person', $3, $4, '')`,
|
||||
[slug, source, title, body],
|
||||
);
|
||||
}
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
}, 60_000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await engine.executeRaw('DELETE FROM page_aliases').catch(() => {});
|
||||
await engine.executeRaw('DELETE FROM context_volunteer_events').catch(() => {});
|
||||
await engine.executeRaw('DELETE FROM pages');
|
||||
});
|
||||
|
||||
describe('parseWindow', () => {
|
||||
test('role prefixes split turns oldest → newest', () => {
|
||||
const turns = parseWindow('user: ask Alice about the deal\nassistant: noted\nuser: what did she say?');
|
||||
expect(turns).toEqual([
|
||||
{ role: 'user', text: 'ask Alice about the deal' },
|
||||
{ role: 'assistant', text: 'noted' },
|
||||
{ role: 'user', text: 'what did she say?' },
|
||||
]);
|
||||
});
|
||||
|
||||
test('CRLF input parses identically', () => {
|
||||
const turns = parseWindow('user: hello\r\nassistant: hi\r\n');
|
||||
expect(turns).toEqual([
|
||||
{ role: 'user', text: 'hello' },
|
||||
{ role: 'assistant', text: 'hi' },
|
||||
]);
|
||||
});
|
||||
|
||||
test('unprefixed text is ONE user turn (echo | volunteer-context just works)', () => {
|
||||
const turns = parseWindow('met with Alice Example today\nshe asked about acme');
|
||||
expect(turns).toEqual([{ role: 'user', text: 'met with Alice Example today\nshe asked about acme' }]);
|
||||
});
|
||||
|
||||
test('continuation lines append to the open turn', () => {
|
||||
const turns = parseWindow('user: first line\nsecond line\nassistant: ok');
|
||||
expect(turns[0]).toEqual({ role: 'user', text: 'first line\nsecond line' });
|
||||
expect(turns[1]).toEqual({ role: 'assistant', text: 'ok' });
|
||||
});
|
||||
|
||||
test('empty / whitespace input → []', () => {
|
||||
expect(parseWindow('')).toEqual([]);
|
||||
expect(parseWindow(' \n \n')).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('extractCandidatesFromWindow', () => {
|
||||
test('merges across turns with occurrence + newest-turn metadata', () => {
|
||||
const turns: WindowTurn[] = [
|
||||
{ role: 'user', text: 'ask Alice Example about the deal' },
|
||||
{ role: 'assistant', text: 'Alice Example said she will follow up' },
|
||||
{ role: 'user', text: 'and ping Bob Sample too' },
|
||||
];
|
||||
const cands = extractCandidatesFromWindow(turns);
|
||||
const alice = cands.find((c) => normalizeAlias(c.query) === normalizeAlias('Alice Example'));
|
||||
const bob = cands.find((c) => normalizeAlias(c.query) === normalizeAlias('Bob Sample'));
|
||||
expect(alice).toBeDefined();
|
||||
expect(alice!.occurrences).toBe(2);
|
||||
expect(alice!.inNewestTurn).toBe(false);
|
||||
expect(alice!.userMention).toBe(true);
|
||||
expect(bob).toBeDefined();
|
||||
expect(bob!.inNewestTurn).toBe(true);
|
||||
});
|
||||
|
||||
test('assistant-only mention is flagged (userMention=false)', () => {
|
||||
const cands = extractCandidatesFromWindow([
|
||||
{ role: 'assistant', text: 'You should talk to Charlie Demo about this' },
|
||||
{ role: 'user', text: 'good idea' },
|
||||
]);
|
||||
const charlie = cands.find((c) => normalizeAlias(c.query) === normalizeAlias('Charlie Demo'));
|
||||
expect(charlie).toBeDefined();
|
||||
expect(charlie!.userMention).toBe(false);
|
||||
});
|
||||
|
||||
test('cap holds across a noisy window', () => {
|
||||
const noisy = Array.from({ length: 30 }, (_, i) => `Entity Number${i} did something.`).join(' ');
|
||||
const cands = extractCandidatesFromWindow([{ role: 'user', text: noisy }]);
|
||||
expect(cands.length).toBeLessThanOrEqual(MAX_CANDIDATES);
|
||||
});
|
||||
});
|
||||
|
||||
describe('volunteerContext', () => {
|
||||
test('assistant-introduced entity two turns back resolves on a pronoun follow-up', async () => {
|
||||
await seed('people/alice-example', 'Alice Example', 'Alice is an early founder.');
|
||||
const turns = parseWindow(
|
||||
'user: who should I talk to about the seed round?\n' +
|
||||
'assistant: Alice Example led a similar round last year.\n' +
|
||||
'user: what did she invest in?',
|
||||
);
|
||||
const pages = await volunteerContext(engine, turns, { sourceIds: ['default'] });
|
||||
expect(pages.length).toBe(1);
|
||||
expect(pages[0].slug).toBe('people/alice-example');
|
||||
expect(pages[0].arm).toBe('title');
|
||||
expect(pages[0].rationale).toContain('assistant-introduced');
|
||||
});
|
||||
|
||||
test('excludeSlugs skips BEFORE the cap: an excluded entity never starves a fresh page', async () => {
|
||||
await seed('people/alice-example', 'Alice Example', 'Founder.');
|
||||
await seed('people/bob-sample', 'Bob Sample', 'Engineer.');
|
||||
const turns = parseWindow('user: intro Alice Example to Bob Sample');
|
||||
const pages = await volunteerContext(engine, turns, {
|
||||
sourceIds: ['default'],
|
||||
maxPages: 1,
|
||||
excludeSlugs: new Set(['people/alice-example']),
|
||||
});
|
||||
// A post-call filter would return [] here (Alice burns the single cap
|
||||
// slot, then gets filtered). The pre-cap exclusion hands Bob the slot.
|
||||
expect(pages.length).toBe(1);
|
||||
expect(pages[0].slug).toBe('people/bob-sample');
|
||||
});
|
||||
|
||||
test('confidence gate drops slug-suffix matches at the default threshold', async () => {
|
||||
// Page resolvable ONLY via slug-suffix: title differs from the mention.
|
||||
await seed('projects/widget-co', 'The Widget Company Project', 'A project page.');
|
||||
const turns = parseWindow('user: any updates on Widget-Co this week?');
|
||||
const gated = await volunteerContext(engine, turns, { sourceIds: ['default'] });
|
||||
expect(gated).toEqual([]);
|
||||
// Lowering min_confidence lets it through with honest provenance.
|
||||
const loose = await volunteerContext(engine, turns, { sourceIds: ['default'], minConfidence: 0.5 });
|
||||
expect(loose.length).toBe(1);
|
||||
expect(loose[0].arm).toBe('slug-suffix');
|
||||
expect(loose[0].confidence).toBeLessThan(VOLUNTEER_DEFAULT_MIN_CONFIDENCE);
|
||||
});
|
||||
|
||||
test('alias arm volunteers with boost when mentioned in the newest turn', async () => {
|
||||
await seed('people/swami-x', 'Swami X', 'A close friend.');
|
||||
await engine.setPageAliases('people/swami-x', 'default', [normalizeAlias('Swami')]);
|
||||
const pages = await volunteerContext(
|
||||
engine,
|
||||
parseWindow('user: Spoke with Swami today'),
|
||||
{ sourceIds: ['default'] },
|
||||
);
|
||||
expect(pages.length).toBe(1);
|
||||
expect(pages[0].arm).toBe('alias');
|
||||
expect(pages[0].confidence).toBeCloseTo(0.95, 5); // 0.9 + newest-turn boost
|
||||
expect(pages[0].rationale).toContain('alias match');
|
||||
});
|
||||
|
||||
test('suppression is slug-only under windowing: a prior-turn MENTION does not suppress', async () => {
|
||||
await seed('people/alice-example', 'Alice Example', 'A founder.');
|
||||
// Prior context contains the TITLE (a bare mention from an earlier turn)
|
||||
// but NOT the slug — the page was never actually surfaced.
|
||||
const pages = await volunteerContext(
|
||||
engine,
|
||||
parseWindow('user: ping Alice Example again'),
|
||||
{ sourceIds: ['default'], priorContext: 'earlier the user said: tell Alice Example the news' },
|
||||
);
|
||||
expect(pages.length).toBe(1);
|
||||
// A slug in prior context (the page WAS surfaced) does suppress.
|
||||
const suppressed = await volunteerContext(
|
||||
engine,
|
||||
parseWindow('user: ping Alice Example again'),
|
||||
{ sourceIds: ['default'], priorContext: 'pointer: people/alice-example was injected last turn' },
|
||||
);
|
||||
expect(suppressed).toEqual([]);
|
||||
});
|
||||
|
||||
test('privacy: takes-fence content never leaks into the synopsis', async () => {
|
||||
const body = `${TAKES_FENCE_BEGIN}\nSECRET_HUNCH_DO_NOT_LEAK\n${TAKES_FENCE_END}\nAlice is a founder.`;
|
||||
await seed('people/alice-example', 'Alice Example', body);
|
||||
const pages = await volunteerContext(engine, parseWindow('user: about Alice Example'), { sourceIds: ['default'] });
|
||||
expect(pages.length).toBe(1);
|
||||
expect(JSON.stringify(pages)).not.toContain('SECRET_HUNCH_DO_NOT_LEAK');
|
||||
});
|
||||
|
||||
test('multi-source scope: resolves from both sources, never outside the scope', async () => {
|
||||
await seed('people/alice-example', 'Alice Example', 'Founder.', 'default');
|
||||
await seed('people/bob-sample', 'Bob Sample', 'Engineer.', 'team-brain');
|
||||
await seed('people/eve-other', 'Eve Other', 'Out of scope.', 'private-brain');
|
||||
const turns = parseWindow('user: intro Alice Example to Bob Sample and Eve Other');
|
||||
const pages = await volunteerContext(engine, turns, { sourceIds: ['default', 'team-brain'] });
|
||||
const keys = pages.map((p) => `${p.source_id}:${p.slug}`).sort();
|
||||
expect(keys).toEqual(['default:people/alice-example', 'team-brain:people/bob-sample']);
|
||||
});
|
||||
|
||||
test('zero-candidate fast path: no entities → [] without touching resolution', async () => {
|
||||
const pages = await volunteerContext(engine, parseWindow('user: ok thanks, sounds good'), {
|
||||
sourceIds: ['default'],
|
||||
});
|
||||
expect(pages).toEqual([]);
|
||||
});
|
||||
|
||||
test('max_pages caps at 5 even when asked for more', async () => {
|
||||
for (let i = 0; i < 8; i++) {
|
||||
await seed(`people/person-${i}`, `Person Alpha${i}`, 'A person.');
|
||||
}
|
||||
const text = Array.from({ length: 8 }, (_, i) => `Person Alpha${i}`).join(' and ');
|
||||
const pages = await volunteerContext(engine, parseWindow(`user: intro ${text}`), {
|
||||
sourceIds: ['default'],
|
||||
maxPages: 50,
|
||||
});
|
||||
expect(pages.length).toBeLessThanOrEqual(5);
|
||||
});
|
||||
});
|
||||
|
||||
describe('volunteerUsageStats', () => {
|
||||
test('join math: used = last_retrieved_at > volunteered_at, labeled approximate', async () => {
|
||||
await seed('people/alice-example', 'Alice Example', 'Founder.');
|
||||
await seed('people/bob-sample', 'Bob Sample', 'Engineer.');
|
||||
await insertVolunteerEvents(engine, [
|
||||
{ source_id: 'default', slug: 'people/alice-example', confidence: 0.9, match_arm: 'alias', rationale: 'r', channel: 'op' },
|
||||
{ source_id: 'default', slug: 'people/bob-sample', confidence: 0.8, match_arm: 'title', rationale: 'r', channel: 'op' },
|
||||
]);
|
||||
// Alice was opened AFTER being volunteered; Bob never was.
|
||||
await engine.executeRaw(
|
||||
`UPDATE pages SET last_retrieved_at = now() + interval '1 minute' WHERE slug = 'people/alice-example'`, [],
|
||||
);
|
||||
const stats = await volunteerUsageStats(engine, ['default'], 30);
|
||||
expect(stats.approximate).toBe(true);
|
||||
expect(stats.note).toContain('approximate');
|
||||
expect(stats.total_volunteered).toBe(2);
|
||||
expect(stats.total_used).toBe(1);
|
||||
const alias = stats.by_arm.find((a) => a.match_arm === 'alias')!;
|
||||
expect(alias.used).toBe(1);
|
||||
expect(alias.precision).toBe(1);
|
||||
const title = stats.by_arm.find((a) => a.match_arm === 'title')!;
|
||||
expect(title.used).toBe(0);
|
||||
});
|
||||
|
||||
test('zero events → zeroed stats, not an error', async () => {
|
||||
const stats = await volunteerUsageStats(engine, ['default'], 7);
|
||||
expect(stats.total_volunteered).toBe(0);
|
||||
expect(stats.by_arm).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('resolveEntitiesToPointers — new provenance surface (back-compat)', () => {
|
||||
test('pointers carry arm + confidence + source_id', async () => {
|
||||
await seed('people/alice-example', 'Alice Example', 'Founder.');
|
||||
const block = await resolveEntitiesToPointers(
|
||||
engine,
|
||||
'default',
|
||||
[{ display: 'Alice Example', query: 'Alice Example' }],
|
||||
{},
|
||||
);
|
||||
expect(block).not.toBeNull();
|
||||
expect(block!.pointers[0].arm).toBe('title');
|
||||
expect(block!.pointers[0].confidence).toBe(0.8);
|
||||
expect(block!.pointers[0].source_id).toBe('default');
|
||||
});
|
||||
|
||||
test('legacy suppression (slug-and-title) still drops a title mention in prior context', async () => {
|
||||
await seed('people/alice-example', 'Alice Example', 'Founder.');
|
||||
const block = await resolveEntitiesToPointers(
|
||||
engine,
|
||||
'default',
|
||||
[{ display: 'Alice Example', query: 'Alice Example' }],
|
||||
{ priorContextText: 'we discussed Alice Example earlier' },
|
||||
);
|
||||
expect(block).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('volunteer_context op (contract surface)', () => {
|
||||
const { operationsByName } = require('../src/core/operations.ts') as typeof import('../src/core/operations.ts');
|
||||
const {
|
||||
awaitPendingVolunteerEventWrites,
|
||||
_resetPendingVolunteerEventWritesForTests,
|
||||
} = require('../src/core/context/volunteer-events.ts') as typeof import('../src/core/context/volunteer-events.ts');
|
||||
|
||||
function mkCtx(overrides: Record<string, unknown> = {}) {
|
||||
return {
|
||||
engine,
|
||||
config: {} as never,
|
||||
logger: { info: () => {}, warn: () => {}, error: () => {} } as never,
|
||||
dryRun: false,
|
||||
remote: false,
|
||||
sourceId: 'default',
|
||||
...overrides,
|
||||
} as never;
|
||||
}
|
||||
|
||||
test('registered, read-scope, not localOnly, stdin cliHint on window', () => {
|
||||
const op = operationsByName.volunteer_context;
|
||||
expect(op).toBeDefined();
|
||||
expect(op.scope).toBe('read');
|
||||
expect(op.localOnly).toBeFalsy();
|
||||
expect(op.cliHints?.name).toBe('volunteer-context');
|
||||
expect(op.cliHints?.stdin).toBe('window');
|
||||
});
|
||||
|
||||
test('window required unless stats: true', async () => {
|
||||
const op = operationsByName.volunteer_context;
|
||||
await expect(op.handler(mkCtx(), {})).rejects.toThrow(/window is required/);
|
||||
const stats = (await op.handler(mkCtx(), { stats: true })) as any;
|
||||
expect(stats.approximate).toBe(true);
|
||||
});
|
||||
|
||||
test('volunteers pages and logs events through the drained sink', async () => {
|
||||
_resetPendingVolunteerEventWritesForTests();
|
||||
await seed('people/alice-example', 'Alice Example', 'Founder.');
|
||||
const op = operationsByName.volunteer_context;
|
||||
const result = (await op.handler(mkCtx(), {
|
||||
window: 'user: ping Alice Example about the deal',
|
||||
session_id: 's-42',
|
||||
turn: 7,
|
||||
})) as any;
|
||||
expect(result.count).toBe(1);
|
||||
expect(result.pages[0].slug).toBe('people/alice-example');
|
||||
expect(result.window_turns).toBe(1);
|
||||
// Event row lands via the fire-and-forget sink once drained.
|
||||
const { unfinished } = await awaitPendingVolunteerEventWrites(5_000);
|
||||
expect(unfinished).toBe(0);
|
||||
const rows = await engine.executeRaw<{ slug: string; channel: string; session_id: string; turn: number }>(
|
||||
`SELECT slug, channel, session_id, turn FROM context_volunteer_events`, [],
|
||||
);
|
||||
expect(rows.length).toBe(1);
|
||||
expect(rows[0].slug).toBe('people/alice-example');
|
||||
expect(rows[0].channel).toBe('op');
|
||||
expect(rows[0].session_id).toBe('s-42');
|
||||
expect(Number(rows[0].turn)).toBe(7);
|
||||
});
|
||||
|
||||
test('event-log failure never fails the op (failing engine injected for the INSERT)', async () => {
|
||||
_resetPendingVolunteerEventWritesForTests();
|
||||
await seed('people/alice-example', 'Alice Example', 'Founder.');
|
||||
const op = operationsByName.volunteer_context;
|
||||
// Wrap the engine: reads pass through, INSERTs into the events table throw.
|
||||
const failingEngine = new Proxy(engine, {
|
||||
get(target, prop, receiver) {
|
||||
if (prop === 'executeRaw') {
|
||||
return (sql: string, params: unknown[]) => {
|
||||
if (/INSERT INTO context_volunteer_events/.test(sql)) {
|
||||
return Promise.reject(new Error('telemetry db down'));
|
||||
}
|
||||
return target.executeRaw(sql, params);
|
||||
};
|
||||
}
|
||||
return Reflect.get(target, prop, receiver);
|
||||
},
|
||||
});
|
||||
const result = (await op.handler(mkCtx({ engine: failingEngine }), {
|
||||
window: 'user: ping Alice Example',
|
||||
})) as any;
|
||||
expect(result.count).toBe(1); // the volunteer result is unaffected
|
||||
const { unfinished } = await awaitPendingVolunteerEventWrites(5_000);
|
||||
expect(unfinished).toBe(0); // failed write settled (swallowed), not stuck
|
||||
});
|
||||
|
||||
test('federated grant scopes the volunteer (allowedSources)', async () => {
|
||||
await seed('people/alice-example', 'Alice Example', 'Founder.', 'default');
|
||||
await seed('people/bob-sample', 'Bob Sample', 'Engineer.', 'grant-brain');
|
||||
await seed('people/eve-other', 'Eve Other', 'Out of grant.', 'secret-brain');
|
||||
const op = operationsByName.volunteer_context;
|
||||
const result = (await op.handler(
|
||||
mkCtx({ remote: true, auth: { allowedSources: ['grant-brain'] } }),
|
||||
{ window: 'user: intro Alice Example to Bob Sample and Eve Other' },
|
||||
)) as any;
|
||||
expect(result.pages.map((p: any) => p.slug)).toEqual(['people/bob-sample']);
|
||||
});
|
||||
|
||||
test('stats mode is source-scoped and returns the approximate note', async () => {
|
||||
await seed('people/alice-example', 'Alice Example', 'Founder.');
|
||||
await insertVolunteerEvents(engine, [
|
||||
{ source_id: 'default', slug: 'people/alice-example', confidence: 0.9, match_arm: 'alias', rationale: 'r', channel: 'watch' },
|
||||
]);
|
||||
const op = operationsByName.volunteer_context;
|
||||
const stats = (await op.handler(mkCtx(), { stats: true, days: 7 })) as any;
|
||||
expect(stats.days).toBe(7);
|
||||
expect(stats.total_volunteered).toBe(1);
|
||||
expect(stats.by_arm[0].channel).toBe('watch');
|
||||
expect(stats.note).toContain('approximate');
|
||||
});
|
||||
});
|
||||
|
||||
describe('knob clamps — untrusted MCP caller inputs (review hardening)', () => {
|
||||
test('minConfidence outside [0,1] (or NaN) falls back to the 0.7 default gate', async () => {
|
||||
await seed('projects/widget-co', 'The Widget Company Project', 'A project.');
|
||||
const turns = parseWindow('user: updates on Widget-Co?');
|
||||
for (const bad of [5, -1, Number.NaN]) {
|
||||
const pages = await volunteerContext(engine, turns, { sourceIds: ['default'], minConfidence: bad });
|
||||
expect(pages).toEqual([]); // slug-suffix (0.6+boost) stays gated at the default 0.7
|
||||
}
|
||||
});
|
||||
|
||||
test('maxPages 0 / negative / NaN fall back to the 3-page default', async () => {
|
||||
for (let i = 0; i < 5; i++) await seed(`people/person-${i}`, `Person Alpha${i}`, 'A person.');
|
||||
const text = Array.from({ length: 5 }, (_, i) => `Person Alpha${i}`).join(' and ');
|
||||
for (const bad of [0, -3, Number.NaN]) {
|
||||
const pages = await volunteerContext(engine, parseWindow(`user: intro ${text}`), {
|
||||
sourceIds: ['default'],
|
||||
maxPages: bad,
|
||||
});
|
||||
expect(pages.length).toBeLessThanOrEqual(3);
|
||||
expect(pages.length).toBeGreaterThan(0);
|
||||
}
|
||||
});
|
||||
|
||||
test('stats days <= 0 / NaN falls back to 30', async () => {
|
||||
const stats = await volunteerUsageStats(engine, ['default'], -5);
|
||||
expect(stats.days).toBe(30);
|
||||
const stats2 = await volunteerUsageStats(engine, ['default'], Number.NaN);
|
||||
expect(stats2.days).toBe(30);
|
||||
});
|
||||
});
|
||||
|
||||
describe('window-cap ordering — the newest user mention survives the cap', () => {
|
||||
test('stale assistant-only chatter is dropped before a newest-turn user entity', async () => {
|
||||
const { extractCandidatesFromWindow: extract, MAX_CANDIDATES: CAP } = await import('../src/core/context/entity-salience.ts');
|
||||
// 14 stale assistant-introduced entities in turn 1, then the user names
|
||||
// ONE entity in the newest turn. The cap (12) must keep the user's.
|
||||
const stale = Array.from({ length: 14 }, (_, i) => `Stale Chatter${i}`).join(', ');
|
||||
const cands = extract([
|
||||
{ role: 'assistant', text: `consider ${stale}.` },
|
||||
{ role: 'user', text: 'actually ask Alice Example first' },
|
||||
]);
|
||||
expect(cands.length).toBeLessThanOrEqual(CAP);
|
||||
const alice = cands.find((c) => normalizeAlias(c.query) === normalizeAlias('Alice Example'));
|
||||
expect(alice).toBeDefined();
|
||||
// Recency + user-role weighting puts the newest user mention FIRST.
|
||||
expect(normalizeAlias(cands[0].query)).toBe(normalizeAlias('Alice Example'));
|
||||
});
|
||||
});
|
||||
|
||||
describe('volunteer-events sink — timeout branch (long-lived process safety)', () => {
|
||||
test('a hung write reports unfinished and drops the snapshot (no ghost references)', async () => {
|
||||
const {
|
||||
logVolunteerEventsFireAndForget,
|
||||
awaitPendingVolunteerEventWrites,
|
||||
_resetPendingVolunteerEventWritesForTests,
|
||||
_peekPendingVolunteerEventWritesForTests,
|
||||
} = await import('../src/core/context/volunteer-events.ts');
|
||||
_resetPendingVolunteerEventWritesForTests();
|
||||
const hangingEngine = {
|
||||
executeRaw: () => new Promise(() => { /* never settles */ }),
|
||||
} as never;
|
||||
logVolunteerEventsFireAndForget(hangingEngine, [
|
||||
{ source_id: 'default', slug: 'people/x', confidence: 0.9, match_arm: 'alias', rationale: 'r', channel: 'watch' },
|
||||
]);
|
||||
const { unfinished } = await awaitPendingVolunteerEventWrites(20);
|
||||
expect(unfinished).toBe(1);
|
||||
// Snapshot dropped so a long-lived `gbrain watch` never accumulates
|
||||
// references to forever-pending work (the last-retrieved C1 class).
|
||||
expect(_peekPendingVolunteerEventWritesForTests()).toBe(0);
|
||||
_resetPendingVolunteerEventWritesForTests();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,231 @@
|
||||
/**
|
||||
* v0.43 (#2095) — `gbrain watch` push transport: streaming loop, rolling
|
||||
* window, session dedupe, --json shape, event logging on channel 'watch',
|
||||
* and clean EOF return. Hermetic PGLite + injected line/write deps (no
|
||||
* subprocess, no real stdin).
|
||||
*/
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { runWatch, WATCH_HELP } from '../src/commands/watch.ts';
|
||||
import { awaitPendingVolunteerEventWrites, _resetPendingVolunteerEventWritesForTests } from '../src/core/context/volunteer-events.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
async function seed(slug: string, title: string, body: string) {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO pages (slug, source_id, type, title, compiled_truth, timeline)
|
||||
VALUES ($1, 'default', 'person', $2, $3, '')`,
|
||||
[slug, title, body],
|
||||
);
|
||||
}
|
||||
|
||||
async function* feed(lines: string[]): AsyncGenerator<string> {
|
||||
for (const l of lines) yield l;
|
||||
}
|
||||
|
||||
async function watchRun(lines: string[], extraArgs: string[] = []): Promise<string[]> {
|
||||
const out: string[] = [];
|
||||
await runWatch(engine, ['--source', 'default', ...extraArgs], {
|
||||
lines: feed(lines),
|
||||
write: (s) => out.push(s),
|
||||
isTTY: false,
|
||||
});
|
||||
return out;
|
||||
}
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
}, 60_000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
_resetPendingVolunteerEventWritesForTests();
|
||||
await engine.executeRaw('DELETE FROM context_volunteer_events').catch(() => {});
|
||||
await engine.executeRaw('DELETE FROM pages');
|
||||
});
|
||||
|
||||
describe('gbrain watch (#2095)', () => {
|
||||
test('--help prints WATCH_HELP and touches nothing', async () => {
|
||||
const out = await watchRun([], ['--help']);
|
||||
expect(out.join('')).toBe(WATCH_HELP);
|
||||
});
|
||||
|
||||
test('volunteers per turn and returns cleanly at EOF', async () => {
|
||||
await seed('people/alice-example', 'Alice Example', 'Alice is a founder.');
|
||||
const out = await watchRun(['ping Alice Example about the deal']);
|
||||
const text = out.join('');
|
||||
expect(text).toContain('people/alice-example');
|
||||
expect(text).toContain('exact title match');
|
||||
// runWatch RESOLVED — the EOF → clean-return contract (the entrypoint
|
||||
// flush-exit + finally drain handle the rest in the real CLI).
|
||||
});
|
||||
|
||||
test('rolling window: assistant-introduced entity fires on the pronoun follow-up turn', async () => {
|
||||
await seed('people/alice-example', 'Alice Example', 'Alice is a founder.');
|
||||
const out = await watchRun([
|
||||
'user: who should I ask about the round?',
|
||||
'assistant: Alice Example led one last year.',
|
||||
'user: what did she invest in?',
|
||||
]);
|
||||
expect(out.join('')).toContain('people/alice-example');
|
||||
});
|
||||
|
||||
test('session dedupe: a slug is volunteered at most once per session', async () => {
|
||||
await seed('people/alice-example', 'Alice Example', 'Alice is a founder.');
|
||||
const out = await watchRun([
|
||||
'user: ping Alice Example',
|
||||
'user: ok',
|
||||
'user: Alice Example again please',
|
||||
]);
|
||||
const hits = out.join('').split('people/alice-example').length - 1;
|
||||
expect(hits).toBe(1);
|
||||
});
|
||||
|
||||
test('--json emits one JSONL row per volunteered page with turn attribution', async () => {
|
||||
await seed('people/alice-example', 'Alice Example', 'Alice is a founder.');
|
||||
const out = await watchRun(['user: hello there', 'user: ping Alice Example'], ['--json']);
|
||||
expect(out.length).toBe(1);
|
||||
const row = JSON.parse(out[0]);
|
||||
expect(row.slug).toBe('people/alice-example');
|
||||
expect(row.turn).toBe(2);
|
||||
expect(row.arm).toBe('title');
|
||||
expect(typeof row.confidence).toBe('number');
|
||||
});
|
||||
|
||||
test('events land on channel watch with session_id + turn (drained sink)', async () => {
|
||||
await seed('people/alice-example', 'Alice Example', 'Alice is a founder.');
|
||||
await watchRun(['user: ping Alice Example']);
|
||||
const { unfinished } = await awaitPendingVolunteerEventWrites(5_000);
|
||||
expect(unfinished).toBe(0);
|
||||
const rows = await engine.executeRaw<{ channel: string; session_id: string; turn: number }>(
|
||||
`SELECT channel, session_id, turn FROM context_volunteer_events`, [],
|
||||
);
|
||||
expect(rows.length).toBe(1);
|
||||
expect(rows[0].channel).toBe('watch');
|
||||
expect(rows[0].session_id).toMatch(/^watch-/);
|
||||
expect(Number(rows[0].turn)).toBe(1);
|
||||
});
|
||||
|
||||
test('min-confidence flag gates exactly like the op', async () => {
|
||||
await seed('projects/widget-co', 'The Widget Company Project', 'A project.');
|
||||
const gated = await watchRun(['user: updates on Widget-Co?']);
|
||||
expect(gated.join('')).toBe('');
|
||||
const loose = await watchRun(['user: updates on Widget-Co?'], ['--min-confidence', '0.5']);
|
||||
expect(loose.join('')).toContain('projects/widget-co');
|
||||
});
|
||||
|
||||
test('blank lines and CRLF are tolerated; no turn, no volunteer', async () => {
|
||||
await seed('people/alice-example', 'Alice Example', 'Alice is a founder.');
|
||||
const out = await watchRun(['', ' ', 'user: ping Alice Example\r']);
|
||||
expect(out.join('')).toContain('people/alice-example');
|
||||
});
|
||||
});
|
||||
|
||||
describe('gbrain watch — window + cap flags (ship coverage G4)', () => {
|
||||
test('--window-turns 1: fires on the mention turn itself; the pronoun follow-up adds nothing', async () => {
|
||||
await seed('people/alice-example', 'Alice Example', 'Alice is a founder.');
|
||||
const out = await watchRun(
|
||||
[
|
||||
'assistant: Alice Example led one last year.',
|
||||
'user: what did she invest in?',
|
||||
],
|
||||
['--window-turns', '1', '--json'],
|
||||
);
|
||||
// Watch volunteers per turn: the entity fires at turn 1 (its mention
|
||||
// turn). With window=1 the follow-up turn extracts nothing, so exactly
|
||||
// one row exists and it carries turn 1 attribution.
|
||||
expect(out.length).toBe(1);
|
||||
expect(JSON.parse(out[0]).turn).toBe(1);
|
||||
});
|
||||
|
||||
test('--max-pages 1 caps a multi-entity turn to one volunteered page', async () => {
|
||||
await seed('people/alice-example', 'Alice Example', 'Founder.');
|
||||
await seed('people/bob-sample', 'Bob Sample', 'Engineer.');
|
||||
const out = await watchRun(
|
||||
['user: intro Alice Example to Bob Sample'],
|
||||
['--max-pages', '1', '--json'],
|
||||
);
|
||||
expect(out.length).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe('gbrain watch — red-team hardening', () => {
|
||||
test('starvation guard: an already-pushed slug never burns the --max-pages cap slot', async () => {
|
||||
await seed('people/alice-example', 'Alice Example', 'Founder.');
|
||||
await seed('people/bob-sample', 'Bob Sample', 'Engineer.');
|
||||
const out = await watchRun(
|
||||
[
|
||||
'user: ping Alice Example about the round',
|
||||
'user: also intro Alice Example to Bob Sample',
|
||||
],
|
||||
['--max-pages', '1', '--json'],
|
||||
);
|
||||
// Turn 1: Alice takes the single cap slot. Turn 2: Alice is excluded
|
||||
// BEFORE the cap (not filtered after), so Bob gets the slot instead of
|
||||
// the turn volunteering nothing forever.
|
||||
expect(out.length).toBe(2);
|
||||
expect(JSON.parse(out[0]).slug).toBe('people/alice-example');
|
||||
expect(JSON.parse(out[1]).slug).toBe('people/bob-sample');
|
||||
});
|
||||
|
||||
test('--window-turns clamps: 0 and negatives behave as window 1', async () => {
|
||||
await seed('people/alice-example', 'Alice Example', 'Founder.');
|
||||
const out = await watchRun(
|
||||
[
|
||||
'assistant: Alice Example led one last year.',
|
||||
'user: what did she invest in?',
|
||||
],
|
||||
['--window-turns', '0', '--json'],
|
||||
);
|
||||
// Clamped to 1: fires on the mention turn only — identical to the
|
||||
// --window-turns 1 contract above, not a crash or an unbounded window.
|
||||
expect(out.length).toBe(1);
|
||||
expect(JSON.parse(out[0]).turn).toBe(1);
|
||||
});
|
||||
|
||||
test('--window-turns absurdly large still runs (hard cap, no unbounded re-scan)', async () => {
|
||||
await seed('people/alice-example', 'Alice Example', 'Founder.');
|
||||
const filler = Array.from({ length: 70 }, (_, i) => `user: filler line ${i}`);
|
||||
const out = await watchRun(
|
||||
[...filler, 'user: ping Alice Example'],
|
||||
['--window-turns', '999999', '--json'],
|
||||
);
|
||||
expect(out.length).toBe(1);
|
||||
expect(JSON.parse(out[0]).slug).toBe('people/alice-example');
|
||||
});
|
||||
});
|
||||
|
||||
describe('gbrain watch — per-turn fail-open (review hardening)', () => {
|
||||
test('a transient DB error on one turn never kills the stream', async () => {
|
||||
await seed('people/alice-example', 'Alice Example', 'Alice is a founder.');
|
||||
// Fail the FIRST resolver query against pages (turn 1's resolution);
|
||||
// everything else — incl. resolveSourceId's pre-loop check — passes.
|
||||
let pagesQueries = 0;
|
||||
const flaky = new Proxy(engine, {
|
||||
get(target, prop, receiver) {
|
||||
if (prop === 'executeRaw') {
|
||||
return (sql: string, params: unknown[]) => {
|
||||
if (/FROM pages/.test(sql) && ++pagesQueries === 1) {
|
||||
return Promise.reject(new Error('transient db hiccup'));
|
||||
}
|
||||
return target.executeRaw(sql, params);
|
||||
};
|
||||
}
|
||||
return Reflect.get(target, prop, receiver);
|
||||
},
|
||||
});
|
||||
const out: string[] = [];
|
||||
await runWatch(flaky as never, ['--source', 'default'], {
|
||||
lines: feed(['user: ping Alice Example', 'user: ping Alice Example please']),
|
||||
write: (s) => out.push(s),
|
||||
isTTY: false,
|
||||
});
|
||||
// Turn 1 failed open; turn 2 volunteered. The stream survived.
|
||||
expect(out.join('')).toContain('people/alice-example');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,75 @@
|
||||
/**
|
||||
* v0.43 (#2095) — `gbrain watch` SIGINT lifecycle. SERIAL: spawns a real CLI
|
||||
* subprocess with a tmpdir brain (the parallel unit shards flake on
|
||||
* concurrent subprocess spawns — same isolation rationale as
|
||||
* apply-migrations-pglite-spawn.serial.test.ts).
|
||||
*/
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
|
||||
describe('gbrain watch — SIGINT lifecycle (real subprocess)', () => {
|
||||
test('SIGINT mid-stream closes the stream and exits cleanly (drain path, exit 0)', async () => {
|
||||
const { mkdtempSync, mkdirSync, writeFileSync, rmSync } = await import('fs');
|
||||
const { join, resolve } = await import('path');
|
||||
const { tmpdir } = await import('os');
|
||||
const REPO = resolve(import.meta.dir, '..');
|
||||
const home = mkdtempSync(join(tmpdir(), 'gbrain-watch-sigint-'));
|
||||
try {
|
||||
mkdirSync(join(home, '.gbrain'), { recursive: true });
|
||||
writeFileSync(
|
||||
join(home, '.gbrain', 'config.json'),
|
||||
JSON.stringify({
|
||||
engine: 'pglite',
|
||||
database_path: join(home, '.gbrain', 'brain.pglite'),
|
||||
embedding_dimensions: 1536,
|
||||
}) + '\n',
|
||||
);
|
||||
// Piped stdin that NEVER reaches EOF — only SIGINT can end the stream.
|
||||
const proc = Bun.spawn(['bun', 'run', join(REPO, 'src', 'cli.ts'), 'watch'], {
|
||||
cwd: REPO,
|
||||
env: { ...process.env, HOME: home, GBRAIN_HOME: home, GBRAIN_SKIP_STARTUP_HOOKS: '1' },
|
||||
stdin: 'pipe',
|
||||
stdout: 'pipe',
|
||||
stderr: 'pipe',
|
||||
});
|
||||
proc.stdin.write('user: nothing relevant here\n');
|
||||
await proc.stdin.flush();
|
||||
// Readiness probe: watch prints "[watch] session <id> ready" on stderr
|
||||
// once engine + source resolution are done and the stdin loop is live.
|
||||
// A fixed sleep raced cold PGLite init (other tests budget 60s for it)
|
||||
// — SIGINT before the handler registers means default-disposition kill.
|
||||
const stderrChunks: string[] = [];
|
||||
const reader = (proc.stderr as ReadableStream<Uint8Array>).getReader();
|
||||
const decoder = new TextDecoder();
|
||||
const deadline = Date.now() + 90_000;
|
||||
let ready = false;
|
||||
while (Date.now() < deadline) {
|
||||
const { value, done } = await reader.read();
|
||||
if (done) break;
|
||||
stderrChunks.push(decoder.decode(value, { stream: true }));
|
||||
if (stderrChunks.join('').includes('ready')) { ready = true; break; }
|
||||
}
|
||||
expect(ready).toBe(true);
|
||||
// Brief settle so the first turn's volunteerContext round-trip is in
|
||||
// flight or done, then interrupt mid-stream.
|
||||
await new Promise((r) => setTimeout(r, 500));
|
||||
proc.kill('SIGINT');
|
||||
const killer = setTimeout(() => {
|
||||
try { proc.kill('SIGKILL'); } catch { /* already dead */ }
|
||||
}, 30_000);
|
||||
const exitCode = await proc.exited;
|
||||
clearTimeout(killer);
|
||||
// Drain the rest of stderr for the banner assertion.
|
||||
while (true) {
|
||||
const { value, done } = await reader.read();
|
||||
if (done) break;
|
||||
stderrChunks.push(decoder.decode(value, { stream: true }));
|
||||
}
|
||||
const stderr = stderrChunks.join('');
|
||||
// Clean drain-then-exit: no force-exit banner, no SIGKILL (137), exit 0.
|
||||
expect(stderr).not.toContain('force-exiting');
|
||||
expect(exitCode).toBe(0);
|
||||
} finally {
|
||||
rmSync(home, { recursive: true, force: true });
|
||||
}
|
||||
}, 120_000);
|
||||
});
|
||||
Reference in New Issue
Block a user