Compare commits

...
Author SHA1 Message Date
Garry Tan 19f919e283 docs: regenerate llms-full.txt after v0.30.2 CLAUDE.md updates
The docs/ pass extended three Key Files entries in CLAUDE.md
(synthesize.ts, subagent.ts, doctor.ts). The auto-derived
llms-full.txt bundle picks up those CLAUDE.md changes via
build-llms; the build-llms test caught the drift in CI.

Generated by: bun run build:llms
2026-05-08 16:22:07 -07:00
Garry TanandClaude Opus 4.7 6943f421ad docs: update README + CLAUDE.md for v0.30.2
- README dream help: 8-phase → 9-phase, mention v0.30.2 chunking + config keys
- CLAUDE.md synthesize.ts: chunker + per-chunk idempotency + D6 slug rewrite + D7 scope + D8 legacy-key
- CLAUDE.md subagent.ts: prompt_too_long terminal classification
- CLAUDE.md doctor.ts: queue_health subcheck 4 (dead-lettered prompt_too_long)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-08 16:08:07 -07:00
Garry TanandClaude Opus 4.7 00ef41657a chore: bump version and changelog (v0.30.2)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 16:03:28 -07:00
Garry Tan 4fd16f767b feat: surface dead-lettered prompt_too_long jobs in doctor queue_health
queue_health gains a 4th subcheck counting dead `subagent` jobs in
the last 24h whose error_text starts with `prompt_too_long:`. When
present, prints a fix hint pointing at
`gbrain dream --phase synthesize --dry-run --json` to identify the
fat transcripts and naming the two operator escape hatches
(`dream.synthesize.max_prompt_tokens` for budget tuning,
larger-context model for capacity).

Operators now see the chunking failure mode without grepping
minion_jobs by hand.
2026-05-08 16:03:10 -07:00
Garry Tan d0c244b3c3 feat: model-aware chunking + slug-rewrite for dream synthesize
The synthesize phase now chunks oversized transcripts at paragraph
boundaries instead of submitting one giant prompt that 400s on
Anthropic. Closes the v0.30 dream-cycle queue clog where 1.7M-token
transcripts dead-lettered after 3 stalls and re-discovered every
cycle.

D1: per-chunk budget = floor(model_context_tokens × 0.9 × 3.5).
MODEL_CONTEXT_TOKENS keys on resolved Anthropic ids (Opus 4.7 = 1M,
Sonnet 4.6 = 200K, Haiku = 200K). Non-Anthropic models fall back to
180K-token safe default with a once-per-process stderr warning.
dream.synthesize.max_prompt_tokens overrides the model lookup
(token-shaped, name from PR #748, floor 100K).

D5: on max_chunks_per_transcript cap hit, log + skip; do NOT write to
dream_verdicts. Closes the cache-poisoning class — next cycle
re-attempts under whatever budget is then current.

D6: orchestrator-side deterministic slug rewrite, zero Sonnet trust.
collectChildPutPageSlugs raw-fetches every (job_id, slug) pair (no
SELECT DISTINCT — that erased the collision evidence the audit
claimed to detect) and rewrites bare-hash6 slugs to <hash6>-c<idx>
for chunked children.

D8: pre-fan-out lookup of completed legacy `dream:synth:<filePath>:
<hash16>` jobs. Transcripts already synthesized under the
single-chunk shape skip submission with `already_synthesized_legacy_
single_chunk` instead of resubmitting under chunked keys.

D9: hash-deterministic chunk boundaries. The 3-tier ladder lifted
from PR #748 (## Topic: > --- > nearest \\n) is fed a back-half
search-window offset derived from contentHash. Same content always
chunks identically across runs; chunk N of a previously-failed
transcript produces byte-identical content on retry.

D10: 24-chunk default cap, operator-configurable via
dream.synthesize.max_chunks_per_transcript.

18 unit cases pin the chunker (boundary ladder, hash determinism,
hard fallback, slug rewrite all 7 shapes). 4 PGLite E2E cases pin
fan-out shape (single-chunk legacy key parity, multi-chunk chunked
key shape) + skip paths (D5 cap hit no verdict-cache write, D8
legacy-key skip).

Credits PR #748 (Wintermute) for the boundary ladder, config key
naming, and 3.5 chars/token estimator. This branch supersedes #748
with the structural safeguards (model-aware budget, terminal-error
classify, slug rewrite, hash-determinism, doctor surfacing).
2026-05-08 16:02:47 -07:00
Garry Tan 0a7e087fd3 feat: classify Anthropic prompt-too-long as UnrecoverableError
The subagent handler now detects 400 "prompt is too long" responses
from the Anthropic SDK and rethrows as UnrecoverableError. The worker
already routes UnrecoverableError straight to `dead`, so doomed jobs
fail terminally on first attempt instead of stalling 3x with the same
oversized prompt.

isPromptTooLongError matches the production message verbatim
("prompt is too long: N tokens > N maximum"), case-insensitive, on
both the outer message and inner error.message paths. Defensive
secondary match for status=400 + invalid_request_error/request_too_large
with the words "too long"/"exceed"/"maximum".

9 unit cases pin the detection: production wording, case folding,
nested SDK shape, defensive 400 paths, unrelated 400s, transient
errors, null/empty inputs.
2026-05-08 16:02:10 -07:00
12 changed files with 1096 additions and 48 deletions
+61
View File
@@ -2,6 +2,67 @@
All notable changes to GBrain will be documented in this file.
## [0.30.2] - 2026-05-08
**Dream synthesize stops dropping fat transcripts. Subagents that overflow Anthropic's context die once, not three times. The queue stops clogging.**
The v0.30 dream cycle has been stalled since May 2 for one user — daily aggregated transcripts at 2.7-4.5MB each generate 1.7M-token Anthropic prompts, which hit the 1M-token hard limit and 400. The subagent handler treated those failures as renewable, so every doomed transcript stalled three times before dead-lettering, and every new cycle re-discovered and re-submitted the same fat transcripts. Six days of synth backlog, queue full of doomed work.
v0.30.2 adds model-aware chunking + terminal-error classification + a poison-pill-free skip path. Wintermute's [PR #748](https://github.com/garrytan/gbrain/pull/748) supplied the boundary heuristics (`## Topic:``---``\n` ladder) and the `dream.synthesize.max_prompt_tokens` config surface. Garry's branch extended that with model-aware budgets, deterministic chunk identity for partial-progress safety, orchestrator-side slug rewriting for zero Sonnet trust on collisions, and doctor-surface visibility.
### What you can now do
**Fat transcripts get synthesized instead of dropped.** A 4.5MB daily-aggregated transcript on Sonnet 4.6 (200K context) now chunks into 7-8 children at the 630KB-per-chunk default budget, each subagent processes its slice, and the orchestrator collects everything. Per-chunk children are independent — chunk 5's content doesn't leak into chunk 6's prompt.
**Doomed transcripts die loud, not silent.** Anthropic 400 "prompt is too long" responses now classify as `UnrecoverableError`. The job goes straight to `dead` on first attempt. No three-stall retry pile. `gbrain doctor` surfaces the count under `queue_health` with a fix hint pointing at `gbrain dream --phase synthesize --dry-run --json` so you can identify the offender.
**Existing single-chunk transcripts are not re-synthesized on upgrade.** The legacy `dream:synth:<filePath>:<hash16>` idempotency-key shape is preserved byte-for-byte for the single-chunk case. Already-synthesized transcripts skip with `already_synthesized_legacy_single_chunk` — no re-spend on Sonnet for content already in the brain.
```bash
# Operator escape hatches
gbrain config set dream.synthesize.max_prompt_tokens 600000 # tune chunk budget below model context
gbrain config set dream.synthesize.max_chunks_per_transcript 32 # raise the chunk-explosion cap
gbrain jobs prune --status dead --queue default # one-time cleanup of pre-v0.30.2 doomed jobs
```
### How it works under the hood
**Model-aware chunk budget.** A static `MODEL_CONTEXT_TOKENS` map keys on the resolved Anthropic model id (Opus 4.7 → 1M, Sonnet 4.6 → 200K, Haiku → 200K). Per-chunk budget is `floor(context × 0.9 × 3.5 chars/token)`, leaving 10% headroom for system prompt + tool defs + output. Non-Anthropic ids (`gpt-5`, `gemini-3-pro`, custom alias) fall back to a 180K-token safe default with a once-per-process stderr warning.
**Hash-deterministic chunk identity.** `splitTranscriptByBudget(content, contentHash, maxChars)` is a pure function. The 3-tier boundary search window (back-half-of-budget) gets seeded with a deterministic offset derived from the first 32 bits of `contentHash`, so the same content always chunks identically. Chunk 2 of a transcript that died terminally produces byte-identical content on retry — per-chunk idempotency keys (`dream:synth:<filePath>:<hash16>:c<i>of<n>`) are durable across runs.
**Orchestrator-side slug rewrite, zero Sonnet trust.** `collectChildPutPageSlugs` no longer does `SELECT DISTINCT` (which would erase collision evidence). Instead it raw-fetches every (job_id, slug) pair, then for chunked children rewrites bare-hash6 slugs to `<hash6>-c<idx>` if Sonnet drops the chunk suffix. Two siblings can't collide even if Sonnet ignores the prompt's `USE THIS in slugs` rule.
**Cap-hit skips don't poison the verdict cache.** When chunks exceed `max_chunks_per_transcript` (default 24), the orchestrator logs + skips without writing to `dream_verdicts`. Next cycle re-attempts under whatever budget is then current — closes the cache-poisoning class entirely.
### Out of scope (deferred to v0.30.3+)
- **Per-turn token-budget guard in subagent.ts.** This release bounds the INITIAL prompt size only. Tool-loop accumulation (search/get_page results bloating each subsequent turn) can still hit `prompt_too_long` mid-conversation; the terminal-error classification catches it then, just less cleanly.
- **Tighter token estimator for code/JSON/CJK-dense content.** The 3.5 chars/token ratio is close to safe for English transcripts but can underflow on dense content. Production telemetry will tell us if a per-recipe override is needed.
### To take advantage of v0.30.2
`gbrain upgrade` is enough — the change is purely runtime behavior, no schema migration required.
If your dream queue accumulated doomed subagent jobs from a prior version:
```bash
gbrain jobs list --status dead --queue default | grep prompt_too_long
gbrain jobs prune --status dead --queue default
```
If you want to verify chunking on your specific corpus before next autopilot cycle:
```bash
gbrain dream --phase synthesize --dry-run --json | jq '.details.skips'
```
A non-empty `skips` array means a transcript hit the 24-chunk cap or already exists at the legacy single-chunk key. Empty means everything fits.
### For contributors
Two-round Codex outside-voice review surfaced 12 structural risks; six folded directly into the design (D5 cap-hit no-cache, D6 orchestrator slug rewrite, D7 honest tool-loop scope, D8 legacy-key migration, D9 hash-deterministic boundaries, D10 operator-configurable cap), and the four PARTIAL items have follow-up TODOs. The plan file at `~/.claude/plans/system-instruction-you-are-working-mossy-fox.md` documents the full decision history. PR #748's contribution (boundary ladder + config-key naming + 3.5 chars/token estimator) is preserved verbatim and credited; this branch supersedes it with the structural safeguards.
Tests: 27 unit cases (`test/cycle-synthesize-chunker.test.ts`, `test/subagent-prompt-too-long.test.ts`) + 4 PGLite E2E cases (`test/e2e/dream-synthesize-chunking.test.ts`) covering D5 cap hit, D8 legacy-key skip, single-chunk parity, multi-chunk fan-out shape.
## [0.30.1] - 2026-05-08
**Operational hardening: gbrain upgrade just works on Supabase. DDL stops timing out on the pooler. Migrations stop wedging. HNSW rebuilds stop nuking your search. Backfills stop being bespoke scripts.**
+3 -3
View File
@@ -122,7 +122,7 @@ strict behavior when unset.
- `src/core/minions/handlers/shell.ts``shell` job handler. Spawns `/bin/sh -c cmd` (absolute path, PATH-override-safe) or `argv[0] argv[1..]` (no shell). Env allowlist: `PATH, HOME, USER, LANG, TZ, NODE_ENV` + caller `env:` overrides. UTF-8-safe stdout/stderr tail via `string_decoder.StringDecoder`. Abort (either `ctx.signal` or `ctx.shutdownSignal`) fires SIGTERM → 5s grace → SIGKILL on child. Requires `GBRAIN_ALLOW_SHELL_JOBS=1` on worker (gated by `registerBuiltinHandlers`).
- `src/core/minions/handlers/shell-audit.ts` — per-submission JSONL audit trail at `~/.gbrain/audit/shell-jobs-YYYY-Www.jsonl` (ISO-week rotation; override via `GBRAIN_AUDIT_DIR`). Best-effort: `mkdirSync(recursive)` + `appendFileSync`; failures logged to stderr, submission not blocked. Logs cmd (first 80 chars) or argv (JSON array). Never logs env values.
- `src/core/minions/backpressure-audit.ts` (v0.19.1) — sibling of shell-audit.ts for `maxWaiting` coalesce events. JSONL at `~/.gbrain/audit/backpressure-YYYY-Www.jsonl`. Fires one line per coalesce with `(queue, name, waiting_count, max_waiting, returned_job_id, ts)`. Closes the silent-drop vector the v0.19.0 maxWaiting guard introduced.
- `src/core/minions/handlers/subagent.ts` (v0.15) — LLM-loop handler. Two-phase tool persistence (pending → complete/failed), replay reconciliation for mid-dispatch crashes, dual-signal abort (`ctx.signal` + `ctx.shutdownSignal`), Anthropic prompt caching on system + tool defs. `makeSubagentHandler({engine, client?, ...})` factory; `MessagesClient` is an injectable interface the real SDK implements structurally. Throws `RateLeaseUnavailableError` (renewable) when rate-lease capacity is full.
- `src/core/minions/handlers/subagent.ts` (v0.15) — LLM-loop handler. Two-phase tool persistence (pending → complete/failed), replay reconciliation for mid-dispatch crashes, dual-signal abort (`ctx.signal` + `ctx.shutdownSignal`), Anthropic prompt caching on system + tool defs. `makeSubagentHandler({engine, client?, ...})` factory; `MessagesClient` is an injectable interface the real SDK implements structurally. Throws `RateLeaseUnavailableError` (renewable) when rate-lease capacity is full. **v0.30.2:** Anthropic 400 `prompt is too long` responses (status 400 + body matches `/prompt is too long|prompt_too_long|context.*length/i`) classify as `UnrecoverableError` so the job goes straight to `dead` on first attempt instead of stalling three times before dead-lettering. Catches both initial-prompt overflow and turn-N tool-loop accumulation that the chunker in `synthesize.ts` can't bound ahead of time.
- `src/core/minions/handlers/subagent-aggregator.ts` (v0.15) — `subagent_aggregator` handler. Claims AFTER all children resolve (queue changes guarantee every terminal child posts a `child_done` inbox message with outcome). Reads inbox via `ctx.readInbox()`, builds deterministic mixed-outcome markdown summary. No LLM call in v0.15.
- `src/core/minions/handlers/subagent-audit.ts` (v0.15) — JSONL audit + heartbeat writer at `~/.gbrain/audit/subagent-jobs-YYYY-Www.jsonl`. Events: `submission` (one line per submit) + `heartbeat` (per turn boundary: `llm_call_started | llm_call_completed | tool_called | tool_result | tool_failed`). Never logs prompts or tool inputs. `readSubagentAuditForJob(jobId, {sinceIso})` is the readback path for `gbrain agent logs`.
- `src/core/minions/rate-leases.ts` (v0.15) — lease-based concurrency cap for outbound providers (default key `anthropic:messages`, max via `GBRAIN_ANTHROPIC_MAX_INFLIGHT`). Owner-tagged rows with `expires_at` auto-prune on acquire; `pg_advisory_xact_lock` guards check-then-insert; CASCADE on owning job deletion. `renewLeaseWithBackoff` retries 3x (250/500/1000ms).
@@ -152,7 +152,7 @@ strict behavior when unset.
- `src/commands/anomalies.ts` (v0.29) — `gbrain anomalies [--since YYYY-MM-DD] [--lookback-days N] [--sigma N] [--json]`: cohort-level activity outliers. Calls `engine.findAnomalies(opts)`. Two cohort kinds in v1: tag, type. Year cohort deferred to v0.30.
- `src/commands/transcripts.ts` (v0.29) — `gbrain transcripts recent [--days N] [--full] [--json]`: recent raw `.txt` transcripts from the dream-cycle corpus dirs. Imports `listRecentTranscripts` from `src/core/transcripts.ts` (the same library the gated `get_recent_transcripts` MCP op uses). Local-only by construction — the CLI always runs with `ctx.remote=false`.
- `src/commands/integrity.ts``gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). v0.22.8: batch-load fast path on Postgres uses `SELECT DISTINCT ON (slug)` in a single SQL query to fix the PgBouncer round-trip timeout (60s → ~6s) while preserving `engine.getAllSlugs()`'s `Set<string>` semantics on multi-source brains. Gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1` so real Postgres errors are diagnosable.
- `src/commands/doctor.ts``gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel <id>`. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`.
- `src/commands/doctor.ts``gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel <id>`. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`. **v0.30.2:** `queue_health` gains a fourth subcheck — surfaces dead-lettered subagent jobs with `last_error` matching the `prompt_too_long` classifier within the last 24h. Fix hint points at `gbrain dream --phase synthesize --dry-run --json` to identify the offending transcript and `gbrain jobs prune --status dead --queue default` to clean up. Postgres-only.
- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). **v40 (v0.29):** `pages_emotional_weight` adds `pages.emotional_weight REAL NOT NULL DEFAULT 0.0`. Column-only (no index). On Postgres 11+ and PGLite, `ADD COLUMN` with a constant DEFAULT is metadata-only — instant on tables of any size. v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). **v30 (v0.23):** creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))`. RLS-enabled when running as a BYPASSRLS role. The synthesize phase reads/writes this table to avoid re-judging on backfill re-runs. **v35 (v0.26.7):** auto-RLS event trigger + one-time backfill. `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` and runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on every new `public.*` table — no FORCE (matches v24/v29/schema.sql posture so non-BYPASSRLS apps can still read their own tables). The same migration backfills RLS on every existing `public.*` base table whose comment doesn't match the doctor regex (`^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}`). Per-table failure aborts the offending CREATE TABLE (event triggers fire inside the DDL transaction); no EXCEPTION wrap — that would convert loud rollback into silent permissive default. PGLite no-op via `sqlFor.pglite: ''`. Breaking change: operators with intentionally-RLS-off public tables must add the GBRAIN:RLS_EXEMPT comment BEFORE upgrade or the backfill will flip them on.
- `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2.
- `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=<ms>` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers.
@@ -160,7 +160,7 @@ strict behavior when unset.
- `src/core/sync-concurrency.ts` (v0.22.13) — single source of truth for the parallel-sync policy. Exports `autoConcurrency(engine, fileCount, override?)` (PGLite always serial; explicit override clamped to >=1; auto path returns `DEFAULT_PARALLEL_WORKERS=4` when `fileCount > AUTO_CONCURRENCY_FILE_THRESHOLD=100`), `shouldRunParallel(workers, fileCount, explicit)` (Q1: explicit `--workers` bypasses the >50-file floor), and `parseWorkers(s)` (rejects `'0'`, `'-3'`, `'foo'`, `'1.5'`, trailing chars — replaces the prior parseInt-with-no-validation in both `sync.ts` and `import.ts`). Used by `performSync`, `performFullSync`, `runImport`, and the Minion `sync` handler so the three sites can no longer drift.
- `src/commands/sync.ts``gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). v0.22.13 (PR #490): `performSync` wraps its body in a `gbrain-sync` writer lock so two concurrent syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot both write `last_commit` and let the last writer win. Head-drift gate after the import phase re-checks `git rev-parse HEAD`; if HEAD moved (someone ran `git checkout` / `git pull` mid-sync), the bookmark refuses to advance. Vanished files now record a failedFiles entry instead of silent-skip — the silent-skip-then-advance pathology that survived prior hardening passes is dead. Worker engines wrap in try/finally so disconnect always fires (panic-path leak fix). Both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`), validated via `parseWorkers`. Explicit `--workers` bypasses the auto-path file-count floor; auto path defers to `autoConcurrency()`. Banner moved to stderr.
- `src/core/cycle.ts` — v0.17 brain maintenance cycle primitive (extended to **9 phases in v0.29**). `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise<CycleReport>` composes phases in semantically-driven order: **lint → backlinks → sync → synthesize → extract → patterns → recompute_emotional_weight → embed → orphans**. v0.29 adds the `recompute_emotional_weight` phase between patterns and embed; it sees the union of `syncPagesAffected` + `synthesizeWrittenSlugs` for incremental mode, or all pages when neither anchor is set (full backfill via `gbrain dream --phase recompute_emotional_weight`). v0.29 also extends `CycleReport.totals` with `pages_emotional_weight_recomputed` (additive, schema_version stays "1"). v0.23's `synthesize` phase runs after sync (cross-references see fresh brain) and before extract (auto-link materializes its writes); `patterns` runs after extract so it reads a fresh graph (codex finding #7 — subagent put_page sets `ctx.remote=true` and skips auto-link/timeline by default; extract is the canonical materialization). Three callers: `gbrain dream` CLI, `gbrain autopilot` daemon's inline path, and the Minions `autopilot-cycle` handler. Coordination via `gbrain_cycle_locks` DB table + `~/.gbrain/cycle.lock` file lock with PID-liveness for PGLite. `CycleReport.schema_version: "1"` is stable; totals additively grew in v0.23 (`transcripts_processed`, `synth_pages_written`, `patterns_written`). `yieldBetweenPhases` runs between phases. **v0.23 added `yieldDuringPhase`** for in-phase keepalive — synthesize/patterns call it during long waits to renew the cycle-lock TTL. Engine nullable; lock-skip on read-only phase selections. v0.22.1 (#403): `CycleOpts.signal?: AbortSignal` propagates the worker's abort signal; `checkAborted()` fires between every phase. v0.22.1 (#417): `runPhaseSync` returns `pagesAffected` via `SyncPhaseResult`; `runCycle` captures it and threads to `runPhaseExtract` as the 4th arg. v0.22.1 (Codex F2): `runPhaseSync` takes `willRunExtractPhase: boolean` and sets `noExtract: phases.includes('extract')` so `gbrain dream --phase sync` doesn't silently lose extraction. v0.22.5 (#475): `resolveSourceForDir(engine, brainDir)` threads `sourceId` to `performSync()` so sync reads the per-source `sources.last_commit` anchor instead of the drift-prone global `config.sync.last_commit` key.
- `src/core/cycle/synthesize.ts` (v0.23) — Synthesize phase: conversation-transcript-to-brain pipeline. Reads from `dream.synthesize.session_corpus_dir`, runs cheap Haiku verdict (cached in `dream_verdicts`), then fans out one Sonnet subagent per worth-processing transcript with `allowed_slug_prefixes` (sourced from `skills/_brain-filing-rules.json` `dream_synthesize_paths.globs`). Orchestrator collects slugs from `subagent_tool_executions` (NOT `pages.updated_at` — codex finding #2) and reverse-renders DB → markdown via `serializeMarkdown`. Cooldown via `dream.synthesize.last_completion_ts`, written ONLY on success. Idempotency key `dream:synth:<file_path>:<content_hash>`. Auto-commit deferred to v1.1 (codex #5). `--dry-run` runs Haiku, skips Sonnet (codex #8). Subagent never gets fs-write access. **v0.23.2:** `renderPageToMarkdown` (now exported) stamps `dream_generated: true` and `dream_cycle_date` into every reverse-write's frontmatter; `writeSummaryPage` does the same on the dream-cycle summary index. The marker is the explicit identity surface checked by `isDreamOutput` in `transcript-discovery.ts` — replaces the v0.23.1 content-prefix heuristic that could miss real output (`serializeMarkdown` doesn't embed slugs in body) and false-positive on user transcripts citing brain pages. `judgeSignificance` and `JudgeClient` are exported; `judgeSignificance` accepts a `verdictModel` parameter (default `claude-haiku-4-5-20251001`) loaded from `dream.synthesize.verdict_model` via `loadSynthConfig`.
- `src/core/cycle/synthesize.ts` (v0.23) — Synthesize phase: conversation-transcript-to-brain pipeline. Reads from `dream.synthesize.session_corpus_dir`, runs cheap Haiku verdict (cached in `dream_verdicts`), then fans out one Sonnet subagent per worth-processing transcript with `allowed_slug_prefixes` (sourced from `skills/_brain-filing-rules.json` `dream_synthesize_paths.globs`). Orchestrator collects slugs from `subagent_tool_executions` (NOT `pages.updated_at` — codex finding #2) and reverse-renders DB → markdown via `serializeMarkdown`. Cooldown via `dream.synthesize.last_completion_ts`, written ONLY on success. Idempotency key `dream:synth:<file_path>:<content_hash>`. Auto-commit deferred to v1.1 (codex #5). `--dry-run` runs Haiku, skips Sonnet (codex #8). Subagent never gets fs-write access. **v0.23.2:** `renderPageToMarkdown` (now exported) stamps `dream_generated: true` and `dream_cycle_date` into every reverse-write's frontmatter; `writeSummaryPage` does the same on the dream-cycle summary index. The marker is the explicit identity surface checked by `isDreamOutput` in `transcript-discovery.ts` — replaces the v0.23.1 content-prefix heuristic that could miss real output (`serializeMarkdown` doesn't embed slugs in body) and false-positive on user transcripts citing brain pages. `judgeSignificance` and `JudgeClient` are exported; `judgeSignificance` accepts a `verdictModel` parameter (default `claude-haiku-4-5-20251001`) loaded from `dream.synthesize.verdict_model` via `loadSynthConfig`. **v0.30.2:** model-aware chunker `splitTranscriptByBudget(content, contentHash, maxChars)` splits oversized transcripts at paragraph boundaries (`## Topic:``---``\n` ladder) using a deterministic offset seeded from the first 32 bits of `contentHash` so retries chunk identically. Per-chunk char budget computed from `MODEL_CONTEXT_TOKENS[resolvedModel] × 0.9 × 3.5 chars/token`; non-Anthropic ids fall back to a 180K-token safe default with a once-per-process stderr warning. Operator overrides: `dream.synthesize.max_prompt_tokens` (floor 100K, wins when set) and `dream.synthesize.max_chunks_per_transcript` (default 24). Per-chunk idempotency keys `dream:synth:<filePath>:<hash16>:c<i>of<n>`; single-chunk transcripts preserve the legacy `dream:synth:<filePath>:<hash16>` key byte-for-byte (D8 lookup), so existing brains skip with `already_synthesized_legacy_single_chunk` instead of re-spending Sonnet on upgrade. `collectChildPutPageSlugs` raw-fetches every (job_id, slug) pair (not `SELECT DISTINCT`) and rewrites bare-hash6 slugs to `<hash6>-c<idx>` for chunked children (D6 — orchestrator-side, zero Sonnet trust). Cap-hit skips don't write to `dream_verdicts`, so raising the cap on next run re-attempts cleanly. D7 scope: bounds INITIAL prompt size only; tool-loop turn-N accumulation is caught by the v0.30.2 terminal-error classification in `subagent.ts`, not bounded ahead of time.
- `src/core/cycle/patterns.ts` (v0.23) — Patterns phase: cross-session theme detection over reflections within `dream.patterns.lookback_days` (default 30). Names a pattern only when ≥`dream.patterns.min_evidence` (default 3) reflections support it. Single Sonnet subagent; same allow-list path as synthesize. Runs AFTER `extract` so the graph is fresh.
- `src/core/cycle/emotional-weight.ts` (v0.29) — Pure function `computeEmotionalWeight({tags, takes}, {highEmotionTags?, userHolder?})`. Deterministic 0..1 score: tag-emotion boost (max 0.5, case-insensitive match against `HIGH_EMOTION_TAGS` seed list), take density (0.1/take, capped at 0.3), take avg weight (0..0.1), user-holder ratio (0..0.1 over active takes; default holder = 'garry'). Total clamped to [0..1]. Anglocentric / personal-life-biased seed list intentional; users override via config key `emotional_weight.high_tags` (JSON array). `userHolder` overridable via `emotional_weight.user_holder`.
- `src/core/cycle/anomaly.ts` (v0.29) — Pure stats helpers for `find_anomalies`. `meanStddev` returns sample stddev (n-1 denominator) and (0,0) for empty input. `computeAnomaliesFromBuckets(baseline, today, sigma, limit)` takes densified daily-count buckets + today's counts per cohort, returns `AnomalyResult[]`. Zero-stddev fallback: cohort fires when `count > mean + 1`, with `sigma_observed = count - mean` as a finite sort proxy (no NaN). Brand-new cohorts (no baseline) have `mean=0, stddev=0` so the fallback fires at count >= 2. Sorted by `sigma_observed` desc, top `limit` (default 20). `page_slugs` capped at 50 per cohort.
+5 -3
View File
@@ -767,9 +767,11 @@ ADMIN
$GBRAIN_HOME/clones/<id>/ and re-cloned on sync if
it goes missing. Also exposed via MCP for remote
agent setup (whoami + sources_{add,list,remove,status}).
gbrain dream [--dry-run] [--phase N] 8-phase maintenance cycle (lint→backlinks→sync→synthesize
→extract→patterns→embed→orphans). v0.23 added synthesize +
patterns: transcripts → reflections + cross-session themes.
gbrain dream [--dry-run] [--phase N] 9-phase maintenance cycle (lint→backlinks→sync→synthesize
→extract→patterns→recompute_emotional_weight→embed→orphans).
v0.23 added synthesize + patterns. v0.29 added emotional-weight
recompute. v0.30.2: synthesize now chunks fat transcripts
(config: dream.synthesize.max_prompt_tokens, max_chunks_per_transcript).
gbrain dream --input <file> Ad-hoc transcript synthesis (implies --phase synthesize)
gbrain dream --date YYYY-MM-DD Synthesize a single day; --from/--to for backfill ranges
gbrain check-backlinks check|fix Back-link enforcement
+1 -1
View File
@@ -1 +1 @@
0.30.1
0.30.2
+8 -6
View File
@@ -222,7 +222,7 @@ strict behavior when unset.
- `src/core/minions/handlers/shell.ts` — `shell` job handler. Spawns `/bin/sh -c cmd` (absolute path, PATH-override-safe) or `argv[0] argv[1..]` (no shell). Env allowlist: `PATH, HOME, USER, LANG, TZ, NODE_ENV` + caller `env:` overrides. UTF-8-safe stdout/stderr tail via `string_decoder.StringDecoder`. Abort (either `ctx.signal` or `ctx.shutdownSignal`) fires SIGTERM → 5s grace → SIGKILL on child. Requires `GBRAIN_ALLOW_SHELL_JOBS=1` on worker (gated by `registerBuiltinHandlers`).
- `src/core/minions/handlers/shell-audit.ts` — per-submission JSONL audit trail at `~/.gbrain/audit/shell-jobs-YYYY-Www.jsonl` (ISO-week rotation; override via `GBRAIN_AUDIT_DIR`). Best-effort: `mkdirSync(recursive)` + `appendFileSync`; failures logged to stderr, submission not blocked. Logs cmd (first 80 chars) or argv (JSON array). Never logs env values.
- `src/core/minions/backpressure-audit.ts` (v0.19.1) — sibling of shell-audit.ts for `maxWaiting` coalesce events. JSONL at `~/.gbrain/audit/backpressure-YYYY-Www.jsonl`. Fires one line per coalesce with `(queue, name, waiting_count, max_waiting, returned_job_id, ts)`. Closes the silent-drop vector the v0.19.0 maxWaiting guard introduced.
- `src/core/minions/handlers/subagent.ts` (v0.15) — LLM-loop handler. Two-phase tool persistence (pending → complete/failed), replay reconciliation for mid-dispatch crashes, dual-signal abort (`ctx.signal` + `ctx.shutdownSignal`), Anthropic prompt caching on system + tool defs. `makeSubagentHandler({engine, client?, ...})` factory; `MessagesClient` is an injectable interface the real SDK implements structurally. Throws `RateLeaseUnavailableError` (renewable) when rate-lease capacity is full.
- `src/core/minions/handlers/subagent.ts` (v0.15) — LLM-loop handler. Two-phase tool persistence (pending → complete/failed), replay reconciliation for mid-dispatch crashes, dual-signal abort (`ctx.signal` + `ctx.shutdownSignal`), Anthropic prompt caching on system + tool defs. `makeSubagentHandler({engine, client?, ...})` factory; `MessagesClient` is an injectable interface the real SDK implements structurally. Throws `RateLeaseUnavailableError` (renewable) when rate-lease capacity is full. **v0.30.2:** Anthropic 400 `prompt is too long` responses (status 400 + body matches `/prompt is too long|prompt_too_long|context.*length/i`) classify as `UnrecoverableError` so the job goes straight to `dead` on first attempt instead of stalling three times before dead-lettering. Catches both initial-prompt overflow and turn-N tool-loop accumulation that the chunker in `synthesize.ts` can't bound ahead of time.
- `src/core/minions/handlers/subagent-aggregator.ts` (v0.15) — `subagent_aggregator` handler. Claims AFTER all children resolve (queue changes guarantee every terminal child posts a `child_done` inbox message with outcome). Reads inbox via `ctx.readInbox()`, builds deterministic mixed-outcome markdown summary. No LLM call in v0.15.
- `src/core/minions/handlers/subagent-audit.ts` (v0.15) — JSONL audit + heartbeat writer at `~/.gbrain/audit/subagent-jobs-YYYY-Www.jsonl`. Events: `submission` (one line per submit) + `heartbeat` (per turn boundary: `llm_call_started | llm_call_completed | tool_called | tool_result | tool_failed`). Never logs prompts or tool inputs. `readSubagentAuditForJob(jobId, {sinceIso})` is the readback path for `gbrain agent logs`.
- `src/core/minions/rate-leases.ts` (v0.15) — lease-based concurrency cap for outbound providers (default key `anthropic:messages`, max via `GBRAIN_ANTHROPIC_MAX_INFLIGHT`). Owner-tagged rows with `expires_at` auto-prune on acquire; `pg_advisory_xact_lock` guards check-then-insert; CASCADE on owning job deletion. `renewLeaseWithBackoff` retries 3x (250/500/1000ms).
@@ -252,7 +252,7 @@ strict behavior when unset.
- `src/commands/anomalies.ts` (v0.29) — `gbrain anomalies [--since YYYY-MM-DD] [--lookback-days N] [--sigma N] [--json]`: cohort-level activity outliers. Calls `engine.findAnomalies(opts)`. Two cohort kinds in v1: tag, type. Year cohort deferred to v0.30.
- `src/commands/transcripts.ts` (v0.29) — `gbrain transcripts recent [--days N] [--full] [--json]`: recent raw `.txt` transcripts from the dream-cycle corpus dirs. Imports `listRecentTranscripts` from `src/core/transcripts.ts` (the same library the gated `get_recent_transcripts` MCP op uses). Local-only by construction — the CLI always runs with `ctx.remote=false`.
- `src/commands/integrity.ts` — `gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). v0.22.8: batch-load fast path on Postgres uses `SELECT DISTINCT ON (slug)` in a single SQL query to fix the PgBouncer round-trip timeout (60s → ~6s) while preserving `engine.getAllSlugs()`'s `Set<string>` semantics on multi-source brains. Gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1` so real Postgres errors are diagnosable.
- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel <id>`. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`.
- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel <id>`. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`. **v0.30.2:** `queue_health` gains a fourth subcheck — surfaces dead-lettered subagent jobs with `last_error` matching the `prompt_too_long` classifier within the last 24h. Fix hint points at `gbrain dream --phase synthesize --dry-run --json` to identify the offending transcript and `gbrain jobs prune --status dead --queue default` to clean up. Postgres-only.
- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). **v40 (v0.29):** `pages_emotional_weight` adds `pages.emotional_weight REAL NOT NULL DEFAULT 0.0`. Column-only (no index). On Postgres 11+ and PGLite, `ADD COLUMN` with a constant DEFAULT is metadata-only — instant on tables of any size. v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). **v30 (v0.23):** creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))`. RLS-enabled when running as a BYPASSRLS role. The synthesize phase reads/writes this table to avoid re-judging on backfill re-runs. **v35 (v0.26.7):** auto-RLS event trigger + one-time backfill. `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` and runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on every new `public.*` table — no FORCE (matches v24/v29/schema.sql posture so non-BYPASSRLS apps can still read their own tables). The same migration backfills RLS on every existing `public.*` base table whose comment doesn't match the doctor regex (`^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}`). Per-table failure aborts the offending CREATE TABLE (event triggers fire inside the DDL transaction); no EXCEPTION wrap — that would convert loud rollback into silent permissive default. PGLite no-op via `sqlFor.pglite: ''`. Breaking change: operators with intentionally-RLS-off public tables must add the GBRAIN:RLS_EXEMPT comment BEFORE upgrade or the backfill will flip them on.
- `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2.
- `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=<ms>` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers.
@@ -260,7 +260,7 @@ strict behavior when unset.
- `src/core/sync-concurrency.ts` (v0.22.13) — single source of truth for the parallel-sync policy. Exports `autoConcurrency(engine, fileCount, override?)` (PGLite always serial; explicit override clamped to >=1; auto path returns `DEFAULT_PARALLEL_WORKERS=4` when `fileCount > AUTO_CONCURRENCY_FILE_THRESHOLD=100`), `shouldRunParallel(workers, fileCount, explicit)` (Q1: explicit `--workers` bypasses the >50-file floor), and `parseWorkers(s)` (rejects `'0'`, `'-3'`, `'foo'`, `'1.5'`, trailing chars — replaces the prior parseInt-with-no-validation in both `sync.ts` and `import.ts`). Used by `performSync`, `performFullSync`, `runImport`, and the Minion `sync` handler so the three sites can no longer drift.
- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). v0.22.13 (PR #490): `performSync` wraps its body in a `gbrain-sync` writer lock so two concurrent syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot both write `last_commit` and let the last writer win. Head-drift gate after the import phase re-checks `git rev-parse HEAD`; if HEAD moved (someone ran `git checkout` / `git pull` mid-sync), the bookmark refuses to advance. Vanished files now record a failedFiles entry instead of silent-skip — the silent-skip-then-advance pathology that survived prior hardening passes is dead. Worker engines wrap in try/finally so disconnect always fires (panic-path leak fix). Both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`), validated via `parseWorkers`. Explicit `--workers` bypasses the auto-path file-count floor; auto path defers to `autoConcurrency()`. Banner moved to stderr.
- `src/core/cycle.ts` — v0.17 brain maintenance cycle primitive (extended to **9 phases in v0.29**). `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise<CycleReport>` composes phases in semantically-driven order: **lint → backlinks → sync → synthesize → extract → patterns → recompute_emotional_weight → embed → orphans**. v0.29 adds the `recompute_emotional_weight` phase between patterns and embed; it sees the union of `syncPagesAffected` + `synthesizeWrittenSlugs` for incremental mode, or all pages when neither anchor is set (full backfill via `gbrain dream --phase recompute_emotional_weight`). v0.29 also extends `CycleReport.totals` with `pages_emotional_weight_recomputed` (additive, schema_version stays "1"). v0.23's `synthesize` phase runs after sync (cross-references see fresh brain) and before extract (auto-link materializes its writes); `patterns` runs after extract so it reads a fresh graph (codex finding #7 — subagent put_page sets `ctx.remote=true` and skips auto-link/timeline by default; extract is the canonical materialization). Three callers: `gbrain dream` CLI, `gbrain autopilot` daemon's inline path, and the Minions `autopilot-cycle` handler. Coordination via `gbrain_cycle_locks` DB table + `~/.gbrain/cycle.lock` file lock with PID-liveness for PGLite. `CycleReport.schema_version: "1"` is stable; totals additively grew in v0.23 (`transcripts_processed`, `synth_pages_written`, `patterns_written`). `yieldBetweenPhases` runs between phases. **v0.23 added `yieldDuringPhase`** for in-phase keepalive — synthesize/patterns call it during long waits to renew the cycle-lock TTL. Engine nullable; lock-skip on read-only phase selections. v0.22.1 (#403): `CycleOpts.signal?: AbortSignal` propagates the worker's abort signal; `checkAborted()` fires between every phase. v0.22.1 (#417): `runPhaseSync` returns `pagesAffected` via `SyncPhaseResult`; `runCycle` captures it and threads to `runPhaseExtract` as the 4th arg. v0.22.1 (Codex F2): `runPhaseSync` takes `willRunExtractPhase: boolean` and sets `noExtract: phases.includes('extract')` so `gbrain dream --phase sync` doesn't silently lose extraction. v0.22.5 (#475): `resolveSourceForDir(engine, brainDir)` threads `sourceId` to `performSync()` so sync reads the per-source `sources.last_commit` anchor instead of the drift-prone global `config.sync.last_commit` key.
- `src/core/cycle/synthesize.ts` (v0.23) — Synthesize phase: conversation-transcript-to-brain pipeline. Reads from `dream.synthesize.session_corpus_dir`, runs cheap Haiku verdict (cached in `dream_verdicts`), then fans out one Sonnet subagent per worth-processing transcript with `allowed_slug_prefixes` (sourced from `skills/_brain-filing-rules.json` `dream_synthesize_paths.globs`). Orchestrator collects slugs from `subagent_tool_executions` (NOT `pages.updated_at` — codex finding #2) and reverse-renders DB → markdown via `serializeMarkdown`. Cooldown via `dream.synthesize.last_completion_ts`, written ONLY on success. Idempotency key `dream:synth:<file_path>:<content_hash>`. Auto-commit deferred to v1.1 (codex #5). `--dry-run` runs Haiku, skips Sonnet (codex #8). Subagent never gets fs-write access. **v0.23.2:** `renderPageToMarkdown` (now exported) stamps `dream_generated: true` and `dream_cycle_date` into every reverse-write's frontmatter; `writeSummaryPage` does the same on the dream-cycle summary index. The marker is the explicit identity surface checked by `isDreamOutput` in `transcript-discovery.ts` — replaces the v0.23.1 content-prefix heuristic that could miss real output (`serializeMarkdown` doesn't embed slugs in body) and false-positive on user transcripts citing brain pages. `judgeSignificance` and `JudgeClient` are exported; `judgeSignificance` accepts a `verdictModel` parameter (default `claude-haiku-4-5-20251001`) loaded from `dream.synthesize.verdict_model` via `loadSynthConfig`.
- `src/core/cycle/synthesize.ts` (v0.23) — Synthesize phase: conversation-transcript-to-brain pipeline. Reads from `dream.synthesize.session_corpus_dir`, runs cheap Haiku verdict (cached in `dream_verdicts`), then fans out one Sonnet subagent per worth-processing transcript with `allowed_slug_prefixes` (sourced from `skills/_brain-filing-rules.json` `dream_synthesize_paths.globs`). Orchestrator collects slugs from `subagent_tool_executions` (NOT `pages.updated_at` — codex finding #2) and reverse-renders DB → markdown via `serializeMarkdown`. Cooldown via `dream.synthesize.last_completion_ts`, written ONLY on success. Idempotency key `dream:synth:<file_path>:<content_hash>`. Auto-commit deferred to v1.1 (codex #5). `--dry-run` runs Haiku, skips Sonnet (codex #8). Subagent never gets fs-write access. **v0.23.2:** `renderPageToMarkdown` (now exported) stamps `dream_generated: true` and `dream_cycle_date` into every reverse-write's frontmatter; `writeSummaryPage` does the same on the dream-cycle summary index. The marker is the explicit identity surface checked by `isDreamOutput` in `transcript-discovery.ts` — replaces the v0.23.1 content-prefix heuristic that could miss real output (`serializeMarkdown` doesn't embed slugs in body) and false-positive on user transcripts citing brain pages. `judgeSignificance` and `JudgeClient` are exported; `judgeSignificance` accepts a `verdictModel` parameter (default `claude-haiku-4-5-20251001`) loaded from `dream.synthesize.verdict_model` via `loadSynthConfig`. **v0.30.2:** model-aware chunker `splitTranscriptByBudget(content, contentHash, maxChars)` splits oversized transcripts at paragraph boundaries (`## Topic:` → `---` → `\n` ladder) using a deterministic offset seeded from the first 32 bits of `contentHash` so retries chunk identically. Per-chunk char budget computed from `MODEL_CONTEXT_TOKENS[resolvedModel] × 0.9 × 3.5 chars/token`; non-Anthropic ids fall back to a 180K-token safe default with a once-per-process stderr warning. Operator overrides: `dream.synthesize.max_prompt_tokens` (floor 100K, wins when set) and `dream.synthesize.max_chunks_per_transcript` (default 24). Per-chunk idempotency keys `dream:synth:<filePath>:<hash16>:c<i>of<n>`; single-chunk transcripts preserve the legacy `dream:synth:<filePath>:<hash16>` key byte-for-byte (D8 lookup), so existing brains skip with `already_synthesized_legacy_single_chunk` instead of re-spending Sonnet on upgrade. `collectChildPutPageSlugs` raw-fetches every (job_id, slug) pair (not `SELECT DISTINCT`) and rewrites bare-hash6 slugs to `<hash6>-c<idx>` for chunked children (D6 — orchestrator-side, zero Sonnet trust). Cap-hit skips don't write to `dream_verdicts`, so raising the cap on next run re-attempts cleanly. D7 scope: bounds INITIAL prompt size only; tool-loop turn-N accumulation is caught by the v0.30.2 terminal-error classification in `subagent.ts`, not bounded ahead of time.
- `src/core/cycle/patterns.ts` (v0.23) — Patterns phase: cross-session theme detection over reflections within `dream.patterns.lookback_days` (default 30). Names a pattern only when ≥`dream.patterns.min_evidence` (default 3) reflections support it. Single Sonnet subagent; same allow-list path as synthesize. Runs AFTER `extract` so the graph is fresh.
- `src/core/cycle/emotional-weight.ts` (v0.29) — Pure function `computeEmotionalWeight({tags, takes}, {highEmotionTags?, userHolder?})`. Deterministic 0..1 score: tag-emotion boost (max 0.5, case-insensitive match against `HIGH_EMOTION_TAGS` seed list), take density (0.1/take, capped at 0.3), take avg weight (0..0.1), user-holder ratio (0..0.1 over active takes; default holder = 'garry'). Total clamped to [0..1]. Anglocentric / personal-life-biased seed list intentional; users override via config key `emotional_weight.high_tags` (JSON array). `userHolder` overridable via `emotional_weight.user_holder`.
- `src/core/cycle/anomaly.ts` (v0.29) — Pure stats helpers for `find_anomalies`. `meanStddev` returns sample stddev (n-1 denominator) and (0,0) for empty input. `computeAnomaliesFromBuckets(baseline, today, sigma, limit)` takes densified daily-count buckets + today's counts per cohort, returns `AnomalyResult[]`. Zero-stddev fallback: cohort fires when `count > mean + 1`, with `sigma_observed = count - mean` as a finite sort proxy (no NaN). Brand-new cohorts (no baseline) have `mean=0, stddev=0` so the fallback fires at count >= 2. Sorted by `sigma_observed` desc, top `limit` (default 20). `page_slugs` capped at 50 per cohort.
@@ -2419,9 +2419,11 @@ ADMIN
$GBRAIN_HOME/clones/<id>/ and re-cloned on sync if
it goes missing. Also exposed via MCP for remote
agent setup (whoami + sources_{add,list,remove,status}).
gbrain dream [--dry-run] [--phase N] 8-phase maintenance cycle (lint→backlinks→sync→synthesize
→extract→patterns→embed→orphans). v0.23 added synthesize +
patterns: transcripts → reflections + cross-session themes.
gbrain dream [--dry-run] [--phase N] 9-phase maintenance cycle (lint→backlinks→sync→synthesize
→extract→patterns→recompute_emotional_weight→embed→orphans).
v0.23 added synthesize + patterns. v0.29 added emotional-weight
recompute. v0.30.2: synthesize now chunks fat transcripts
(config: dream.synthesize.max_prompt_tokens, max_chunks_per_transcript).
gbrain dream --input <file> Ad-hoc transcript synthesis (implies --phase synthesize)
gbrain dream --date YYYY-MM-DD Synthesize a single day; --from/--to for backfill ranges
gbrain check-backlinks check|fix Back-link enforcement
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "gbrain",
"version": "0.30.1",
"version": "0.30.2",
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
"type": "module",
"main": "src/core/index.ts",
+24
View File
@@ -1306,6 +1306,21 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
`;
const rssKillCount = rssKillRows[0]?.cnt ?? 0;
// Subcheck 4 (v0.30.2): prompt_too_long terminal failures on subagent
// jobs in the last 24h. The dream/synthesize phase classifies Anthropic
// 400 "prompt is too long" responses as UnrecoverableError so they
// dead-letter on first attempt instead of clogging the queue with
// max_stalled retries. Surface count + fix hint when present.
const promptTooLongRows: Array<{ cnt: number }> = await sql`
SELECT count(*)::int AS cnt
FROM minion_jobs
WHERE name = 'subagent'
AND status = 'dead'
AND finished_at > now() - interval '24 hours'
AND error_text LIKE 'prompt_too_long:%'
`;
const promptTooLongCount = promptTooLongRows[0]?.cnt ?? 0;
const problems: string[] = [];
if (stalledRows.length > 0) {
const sample = stalledRows
@@ -1333,6 +1348,15 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
`See skills/migrations/v0.22.14.md.`
);
}
if (promptTooLongCount > 0) {
problems.push(
`${promptTooLongCount} subagent job(s) dead-lettered with prompt_too_long in last 24h. ` +
`Dream/synthesize transcripts exceeded the model's input context. ` +
`Fix: \`gbrain dream --phase synthesize --dry-run --json\` to identify fat transcripts; ` +
`set \`dream.synthesize.max_prompt_tokens\` to bound the per-chunk budget, or use a ` +
`larger-context model (Opus 4.7 = 1M tokens vs Sonnet 4.6 = 200K).`
);
}
if (problems.length === 0) {
checks.push({
+372 -34
View File
@@ -42,6 +42,173 @@ import type { Page, PageType } from '../types.ts';
// Used for the orchestrator-written summary index slug.
const SUMMARY_SLUG_RE = /^[a-z0-9][a-z0-9\-]*(\/[a-z0-9][a-z0-9\-]*)*$/;
// ── Model context budget (D1, D5, D7, D9) ─────────────────────────────
/**
* Anthropic model id input context window (tokens).
* Unknown id (non-Anthropic alias, custom string) safe 200K-token fallback
* via `computeChunkCharBudget`. Codex finding #4: `resolveModel()` does not
* canonicalize to Anthropic-only; this map keys on the exact strings the
* resolver returns for known Anthropic aliases.
*/
const MODEL_CONTEXT_TOKENS: Record<string, number> = {
'claude-opus-4-7': 1_000_000,
'claude-opus-4-6': 1_000_000,
'claude-sonnet-4-6': 200_000,
'claude-sonnet-4-5': 200_000,
'claude-haiku-4-5-20251001': 200_000,
};
/** Token-to-char ratio. 3.5 matches PR #748; conservative for English text. */
const CHARS_PER_TOKEN = 3.5;
/** Reserve 10% of context window for system prompt + tool defs + output. */
const HEADROOM_RATIO = 0.9;
/** Floor on user-overridable max_prompt_tokens (matches PR #748 minimum). */
const MIN_PROMPT_TOKENS = 100_000;
/** Default chunk-count cap; operator-configurable via dream.synthesize.max_chunks_per_transcript. */
const DEFAULT_MAX_CHUNKS = 24;
/** Conservative default budget when model is unknown (200K × HEADROOM_RATIO). */
const UNKNOWN_MODEL_BUDGET_TOKENS = 180_000;
/**
* Compute per-chunk character budget for the resolved model + config override.
*
* Resolution:
* - configMaxPromptTokens (already floored at MIN_PROMPT_TOKENS) wins when set.
* - Else the model's MODEL_CONTEXT_TOKENS entry × HEADROOM_RATIO.
* - Else (non-Anthropic alias / custom id) UNKNOWN_MODEL_BUDGET_TOKENS, with
* a once-per-process stderr warning.
*
* D7 scope: this bounds the INITIAL prompt size only. Tool-loop turn-N
* accumulation is out of scope for v0.30.2 (terminal-error classification
* catches turn-N blowups; per-turn budget guard is a v0.31+ follow-up).
*/
function computeChunkCharBudget(
model: string,
configMaxPromptTokens: number | null,
): number {
if (configMaxPromptTokens !== null) {
return Math.floor(configMaxPromptTokens * CHARS_PER_TOKEN);
}
const ctx = MODEL_CONTEXT_TOKENS[model];
if (ctx === undefined) {
warnUnknownModelOnce(model);
return Math.floor(UNKNOWN_MODEL_BUDGET_TOKENS * CHARS_PER_TOKEN);
}
return Math.floor(ctx * HEADROOM_RATIO * CHARS_PER_TOKEN);
}
const _unknownModelWarned = new Set<string>();
function warnUnknownModelOnce(model: string): void {
if (_unknownModelWarned.has(model)) return;
_unknownModelWarned.add(model);
process.stderr.write(
`[dream] model "${model}" is not in MODEL_CONTEXT_TOKENS; ` +
`using ${UNKNOWN_MODEL_BUDGET_TOKENS}-token fallback budget. ` +
`Set dream.synthesize.max_prompt_tokens to override.\n`,
);
}
// ── Hash-deterministic transcript chunker (D9) ────────────────────────
/**
* Split content into chunks at most maxChars long, picking boundaries via a
* 3-tier ladder lifted from PR #748:
* 1. `## Topic:` separators (matches the daily-aggregated transcript shape)
* 2. `---` markdown HR markers
* 3. nearest `\n` newline
*
* D9 stable chunk identity: the back-half-of-budget search window is seeded
* with a deterministic offset derived from contentHash so the same
* (content, contentHash, maxChars) triple always produces identical chunks.
* Closes the partial-progress ambiguity: chunk 2 of a transcript that
* previously failed terminally produces byte-identical content on retry,
* so the per-chunk idempotency key is durable across runs.
*
* The hash-derived offset jitters the search start within
* [0.5×budget, 0.6×budget] so the back-half rule still holds.
*
* If no boundary fits, hard-split at maxChars (also deterministic in the
* inputs).
*
* Pure function. Tested by `test/cycle/synthesize-chunker.test.ts`.
*/
export function splitTranscriptByBudget(
content: string,
contentHash: string,
maxChars: number,
): string[] {
if (maxChars <= 0) {
throw new Error(`splitTranscriptByBudget: maxChars must be > 0, got ${maxChars}`);
}
if (content.length <= maxChars) return [content];
const hashInt = parseHashOffset(contentHash);
// Jitter window is the next 10% of budget after the 50% midpoint.
const jitterRange = Math.max(1, Math.floor(maxChars * 0.1));
const searchStart = Math.floor(maxChars * 0.5) + (hashInt % jitterRange);
const out: string[] = [];
let remaining = content;
while (remaining.length > maxChars) {
const split = findBoundary(remaining, maxChars, searchStart);
out.push(remaining.slice(0, split));
remaining = remaining.slice(split);
}
if (remaining.length > 0) out.push(remaining);
return out;
}
function parseHashOffset(contentHash: string): number {
// First 8 hex chars = 32 bits; plenty of entropy for the offset jitter.
const hex = contentHash.slice(0, 8);
const n = parseInt(hex, 16);
return Number.isFinite(n) && n >= 0 ? n : 0;
}
function findBoundary(text: string, maxChars: number, searchStart: number): number {
const window = text.slice(searchStart, maxChars);
// Tier 1: "\n## Topic:" — last occurrence inside the search window.
const topicIdx = window.lastIndexOf('\n## Topic:');
if (topicIdx >= 0) return searchStart + topicIdx;
// Tier 2: "\n---\n" markdown HR.
const hrIdx = window.lastIndexOf('\n---\n');
if (hrIdx >= 0) return searchStart + hrIdx;
// Tier 3: any newline.
const nlIdx = window.lastIndexOf('\n');
if (nlIdx >= 0) return searchStart + nlIdx;
// No boundary fits; hard-split at maxChars (deterministic).
return maxChars;
}
/**
* D6: orchestrator-side deterministic slug rewrite. Zero Sonnet trust.
*
* Expected shape from `buildSynthesisPrompt` for a chunked child is already
* `<base>-<hash6>-c<idx>`, but if Sonnet drops the chunk suffix this rewrite
* enforces uniqueness post-hoc. Same hash AND same chunk idx idempotent.
*
* Pure function. Cases:
* - already correctly suffixed (`...-<hash6>-c<idx>`) return unchanged.
* - bare hash suffix (`...-<hash6>`) append `-c<idx>`.
* - some other shape pass through (orchestrator can't safely guess
* where to inject the chunk index; e2e test pins this).
*/
export function rewriteChunkedSlug(slug: string, hash6: string, idx: number): string {
if (!slug) return slug;
const expected = `${hash6}-c${idx}`;
// Already correctly chunk-suffixed.
if (slug === expected) return slug;
if (slug.endsWith(`-${expected}`) || slug.endsWith(`/${expected}`)) return slug;
// Bare hash6 at end of last path segment: rewrite.
// Match either at start-of-slug, after a "/" path separator, or after a "-".
const re = new RegExp(`(^|[/-])${hash6}$`);
if (re.test(slug)) return `${slug}-c${idx}`;
// Unknown shape — pass through; collision risk is now bounded by Sonnet's
// per-chunk-prompt guidance and the existing slug-prefix allow-list.
return slug;
}
// ── Public entry ──────────────────────────────────────────────────────
export interface SynthesizePhaseOpts {
@@ -166,7 +333,8 @@ export async function runPhaseSynthesize(
});
}
// Fan-out: submit one subagent per worth-processing transcript.
// Fan-out: submit one subagent per worth-processing transcript (or one
// per chunk for transcripts that exceed the model's per-prompt budget).
const allowedSlugPrefixes = await loadAllowedSlugPrefixes();
if (allowedSlugPrefixes.length === 0) {
return failed(makeError('InternalError', 'NO_ALLOWLIST',
@@ -175,26 +343,82 @@ export async function runPhaseSynthesize(
const queue = new MinionQueue(engine);
const childIds: number[] = [];
/** Map child job_id → chunk metadata for D6 orchestrator-side slug rewrite. */
const chunkInfo = new Map<number, { idx: number; hash6: string }>();
/** Skip reasons for the cycle report (D5 cap hits, D8 legacy-key skips). */
const skipReports: Array<{ filePath: string; reason: string }> = [];
const maxCharsPerChunk = computeChunkCharBudget(config.model, config.maxPromptTokens);
for (const t of worthProcessing) {
const childData: SubagentHandlerData = {
prompt: buildSynthesisPrompt(t),
model: config.model,
max_turns: 30,
allowed_slug_prefixes: allowedSlugPrefixes,
};
const submitOpts: Partial<MinionJobInput> = {
max_stalled: 3,
on_child_fail: 'continue',
idempotency_key: `dream:synth:${t.filePath}:${t.contentHash.slice(0, 16)}`,
timeout_ms: 30 * 60 * 1000, // 30 min per transcript
};
const child = await queue.add(
'subagent',
childData as unknown as Record<string, unknown>,
submitOpts,
{ allowProtectedSubmit: true },
);
childIds.push(child.id);
const hash16 = t.contentHash.slice(0, 16);
const hash6 = t.contentHash.slice(0, 6);
// D8: single→multi-chunk migration safety. If a completed legacy
// single-chunk job exists for this content_hash, treat as already-
// synthesized and skip. Prevents duplicate writes when a transcript
// that was previously single-chunk now multi-chunks (because budget
// shrank or model changed).
if (await hasLegacySingleChunkCompletion(engine, t.filePath, hash16)) {
skipReports.push({
filePath: t.filePath,
reason: 'already_synthesized_legacy_single_chunk',
});
continue;
}
const chunks = splitTranscriptByBudget(t.content, t.contentHash, maxCharsPerChunk);
// D5 cap hit: log + skip; do NOT write to dream_verdicts. Closes the
// poison-pill class — next cycle re-attempts under whatever budget
// is then current.
if (chunks.length > config.maxChunksPerTranscript) {
process.stderr.write(
`[dream] transcript ${t.basename} produced ${chunks.length} chunks at ` +
`${maxCharsPerChunk}-char budget (cap=${config.maxChunksPerTranscript}); skipping. ` +
`Increase dream.synthesize.max_chunks_per_transcript or use a larger-context model.\n`,
);
skipReports.push({
filePath: t.filePath,
reason: `oversize_after_split: ${chunks.length}/${config.maxChunksPerTranscript}`,
});
continue;
}
const isChunked = chunks.length > 1;
for (let i = 0; i < chunks.length; i++) {
const childData: SubagentHandlerData = {
prompt: buildSynthesisPrompt(t, chunks[i], i, chunks.length),
model: config.model,
max_turns: 30,
allowed_slug_prefixes: allowedSlugPrefixes,
};
// Idempotency key parity:
// - single-chunk → legacy `dream:synth:<filePath>:<hash16>` (byte-
// equivalent across versions; preserves dedup for unchanged
// transcripts on upgrade).
// - multi-chunk → `<legacy>:c<i>of<n>` per chunk; durable across
// runs because D9 splitTranscriptByBudget is hash-deterministic.
const idempotency_key = isChunked
? `dream:synth:${t.filePath}:${hash16}:c${i}of${chunks.length}`
: `dream:synth:${t.filePath}:${hash16}`;
const submitOpts: Partial<MinionJobInput> = {
max_stalled: 3,
on_child_fail: 'continue',
idempotency_key,
timeout_ms: 30 * 60 * 1000, // 30 min per chunk
};
const child = await queue.add(
'subagent',
childData as unknown as Record<string, unknown>,
submitOpts,
{ allowProtectedSubmit: true },
);
childIds.push(child.id);
if (isChunked) {
chunkInfo.set(child.id, { idx: i, hash6 });
}
}
}
// Wait for every child to reach a terminal state. Tick yieldDuringPhase
@@ -222,7 +446,10 @@ export async function runPhaseSynthesize(
// Collect slugs from put_page tool executions across the children
// (codex finding #2: deterministic provenance, NOT pages.updated_at).
const writtenSlugs = await collectChildPutPageSlugs(engine, childIds);
// D6 orchestrator slug rewrite: chunkInfo drives post-hoc rewrite of
// bare-hash slugs to `<hash6>-c<idx>` so chunked siblings can't collide
// even if Sonnet drops the chunk suffix.
const writtenSlugs = await collectChildPutPageSlugs(engine, childIds, chunkInfo);
// Dual-write: reverse-render each DB row → markdown file.
const reverseWriteCount = await reverseWriteSlugs(engine, opts.brainDir, writtenSlugs);
@@ -239,9 +466,10 @@ export async function runPhaseSynthesize(
await engine.setConfig('dream.synthesize.last_completion_ts', new Date().toISOString());
const ms = Date.now() - start;
return ok(`${worthProcessing.length} transcript(s) synthesized in ${(ms / 1000).toFixed(1)}s`, {
const submittedTranscripts = worthProcessing.length - skipReports.length;
return ok(`${submittedTranscripts} transcript(s) synthesized in ${(ms / 1000).toFixed(1)}s`, {
transcripts_discovered: transcripts.length,
transcripts_processed: worthProcessing.length,
transcripts_processed: submittedTranscripts,
pages_written: writtenSlugs.length,
// v0.29: emit the slug list so the recompute_emotional_weight phase can
// union with sync's pagesAffected and recompute weights for every page
@@ -249,6 +477,12 @@ export async function runPhaseSynthesize(
written_slugs: writtenSlugs,
reverse_write_count: reverseWriteCount,
child_outcomes: childOutcomes,
// Children submitted (one per chunk for chunked transcripts; one per
// transcript for single-chunk). Differs from transcripts_processed
// when chunking is in play.
children_submitted: childIds.length,
// D5 cap hits + D8 legacy-key skips. Empty when nothing skipped.
skips: skipReports,
summary_slug: summarySlug,
verdicts,
});
@@ -269,6 +503,20 @@ interface SynthConfig {
model: string;
verdictModel: string;
cooldownHours: number;
/**
* D1: Override the per-chunk token budget (model_context × HEADROOM_RATIO
* by default). Floor MIN_PROMPT_TOKENS, no upper cap (model context wins).
* Surface name follows PR #748: `dream.synthesize.max_prompt_tokens`.
* `null` means use the model-context lookup.
*/
maxPromptTokens: number | null;
/**
* D5/D10: Cap on chunks produced from a single transcript. On cap hit, the
* transcript is logged + skipped (NOT cached in dream_verdicts closes the
* cache-poisoning class). Operator override:
* `dream.synthesize.max_chunks_per_transcript`.
*/
maxChunksPerTranscript: number;
}
async function loadSynthConfig(engine: BrainEngine): Promise<SynthConfig> {
@@ -290,6 +538,8 @@ async function loadSynthConfig(engine: BrainEngine): Promise<SynthConfig> {
fallback: 'haiku',
});
const cooldownHoursStr = await engine.getConfig('dream.synthesize.cooldown_hours');
const maxPromptTokensStr = await engine.getConfig('dream.synthesize.max_prompt_tokens');
const maxChunksStr = await engine.getConfig('dream.synthesize.max_chunks_per_transcript');
let excludePatterns: string[] = ['medical', 'therapy'];
if (excludeStr) {
@@ -299,6 +549,23 @@ async function loadSynthConfig(engine: BrainEngine): Promise<SynthConfig> {
} catch { /* keep default */ }
}
// D1: max_prompt_tokens floored at MIN_PROMPT_TOKENS; null → use model lookup.
let maxPromptTokens: number | null = null;
if (maxPromptTokensStr) {
const parsed = parseInt(maxPromptTokensStr, 10);
if (Number.isFinite(parsed) && parsed > 0) {
maxPromptTokens = Math.max(MIN_PROMPT_TOKENS, parsed);
}
}
// D10: max_chunks default 24, floor 1.
let maxChunksPerTranscript = DEFAULT_MAX_CHUNKS;
if (maxChunksStr) {
const parsed = parseInt(maxChunksStr, 10);
if (Number.isFinite(parsed) && parsed >= 1) {
maxChunksPerTranscript = parsed;
}
}
return {
enabled,
corpusDir: corpusDir ?? null,
@@ -308,6 +575,8 @@ async function loadSynthConfig(engine: BrainEngine): Promise<SynthConfig> {
model,
verdictModel,
cooldownHours: cooldownHoursStr ? Math.max(0, parseInt(cooldownHoursStr, 10) || 12) : 12,
maxPromptTokens,
maxChunksPerTranscript,
};
}
@@ -422,16 +691,38 @@ Two reasons max, one phrase each.`;
// ── Subagent prompt ──────────────────────────────────────────────────
function buildSynthesisPrompt(t: DiscoveredTranscript): string {
/**
* Build the prompt for one subagent. When `chunkTotal > 1`, the slug seed
* gains a `-c<idx>` suffix and the prompt names which chunk this is.
*
* D6 enforcement is orchestrator-side (rewriteChunkedSlug runs at slug-
* collection time). Sonnet still gets the chunked seed via the prompt's
* `USE THIS in slugs` rule for the happy path.
*/
function buildSynthesisPrompt(
t: DiscoveredTranscript,
chunkText: string,
chunkIdx: number,
chunkTotal: number,
): string {
const dateHint = t.inferredDate ?? today();
const hashSuffix = t.contentHash.slice(0, 6);
const baseSlugSegment = sanitizeForSlug(t.basename) || `session-${dateHint}`;
const isChunked = chunkTotal > 1;
const hashSuffix = isChunked
? `${t.contentHash.slice(0, 6)}-c${chunkIdx}`
: t.contentHash.slice(0, 6);
const chunkBanner = isChunked
? `\n- This is CHUNK ${chunkIdx + 1} of ${chunkTotal} from the same transcript. Different chunks process different sections; do not assume continuity with other chunks.`
: '';
const transcriptHeader = isChunked
? `${t.filePath} (chunk ${chunkIdx + 1}/${chunkTotal})`
: t.filePath;
return `You are synthesizing a conversation transcript into the user's personal knowledge brain.
CONTEXT
- Today's date: ${dateHint}
- Transcript hash suffix (USE THIS in slugs): ${hashSuffix}
- Source file basename: ${baseSlugSegment}
- Source file basename: ${baseSlugSegment}${chunkBanner}
OUTPUT POLICY (ALL of these are required)
1. Quote the user verbatim. Do not paraphrase memorable phrasings.
@@ -450,9 +741,9 @@ C. People mentions: search first; if a page exists, do not put_page over it (the
D. If nothing in this transcript meets the bar (significance filter already passed but the content is still routine), return without writing anything.
TRANSCRIPT (${t.filePath})
TRANSCRIPT (${transcriptHeader})
---
${t.content}
${chunkText}
---
When done, briefly list the slugs you wrote in your final message so the orchestrator can audit.`;
@@ -466,24 +757,71 @@ function sanitizeForSlug(s: string): string {
.slice(0, 60);
}
// ── Slug collection from child put_page calls (codex #2) ────────────
// ── Slug collection from child put_page calls (codex #2 + D6) ────────
/**
* D6 (orchestrator-side deterministic slug rewrite, zero Sonnet trust):
* two-stage path raw fetch (no DISTINCT, preserves duplicate evidence)
* in-memory chunk-suffix rewrite via `rewriteChunkedSlug` for chunked
* children return distinct rewritten set.
*
* Closes Codex finding #2 ("collision detection via SELECT DISTINCT was
* fake"): we no longer need detection because the rewrite enforces
* uniqueness at slug-write time.
*
* `chunkInfo` maps child job_id { chunk_index, hash6 }. Single-chunk
* children are absent from the map and pass through unchanged.
*/
async function collectChildPutPageSlugs(
engine: BrainEngine,
childIds: number[],
chunkInfo: Map<number, { idx: number; hash6: string }>,
): Promise<string[]> {
if (childIds.length === 0) return [];
const rows = await engine.executeRaw<{ slug: string }>(
`SELECT DISTINCT input->>'slug' AS slug
// Raw fetch — NO SELECT DISTINCT. Preserves per-child slug duplicates so
// the orchestrator sees what each child wrote.
const rows = await engine.executeRaw<{ job_id: number; slug: string }>(
`SELECT job_id, input->>'slug' AS slug
FROM subagent_tool_executions
WHERE job_id = ANY($1::int[])
AND tool_name = 'brain_put_page'
AND status = 'complete'
AND input ? 'slug'
ORDER BY 1`,
AND input ? 'slug'`,
[childIds],
);
return rows.map(r => r.slug).filter((s): s is string => typeof s === 'string' && s.length > 0);
const rewritten = new Set<string>();
for (const r of rows) {
if (typeof r.slug !== 'string' || r.slug.length === 0) continue;
const ci = chunkInfo.get(r.job_id);
rewritten.add(ci ? rewriteChunkedSlug(r.slug, ci.hash6, ci.idx) : r.slug);
}
return Array.from(rewritten).sort();
}
/**
* D8: query for any `completed` legacy single-chunk job at the canonical
* idempotency key shape `dream:synth:<filePath>:<hash16>`. Used at fan-out
* time to detect transcripts that were synthesized under the pre-chunking
* code path; those should NOT be re-submitted under chunked keys.
*
* Reuses the existing `minion_jobs.idempotency_key` index no schema
* additions. One indexed lookup per worth-processing transcript.
*/
async function hasLegacySingleChunkCompletion(
engine: BrainEngine,
filePath: string,
hash16: string,
): Promise<boolean> {
const legacyKey = `dream:synth:${filePath}:${hash16}`;
const rows = await engine.executeRaw<{ status: string }>(
`SELECT status
FROM minion_jobs
WHERE idempotency_key = $1
AND status = 'completed'
LIMIT 1`,
[legacyKey],
);
return rows.length > 0;
}
// ── Reverse-write DB rows → markdown files ───────────────────────────
+47
View File
@@ -26,6 +26,7 @@
import Anthropic from '@anthropic-ai/sdk';
import type { MinionJobContext, MinionJob } from '../types.ts';
import { UnrecoverableError } from '../types.ts';
import type {
ContentBlock,
SubagentHandlerData,
@@ -352,6 +353,15 @@ export function makeSubagentHandler(deps: SubagentDeps) {
} catch (err) {
// Release lease eagerly on error so we don't starve capacity.
await releaseLease(engine, lease.leaseId!).catch(() => {});
// Terminal classification: a 400 "prompt is too long" from Anthropic
// is unrecoverable — retrying with the same prompt will always fail.
// Convert to UnrecoverableError so the worker routes the job
// straight to `dead`, bypassing max_stalled retries (the v0.30.x
// dream-cycle queue-clog the chunking work was built to prevent).
if (isPromptTooLongError(err)) {
const origMsg = err instanceof Error ? err.message : String(err);
throw new UnrecoverableError(`prompt_too_long: ${origMsg}`);
}
throw err;
}
@@ -703,6 +713,43 @@ export class RateLeaseUnavailableError extends Error {
}
}
/**
* Detect Anthropic SDK errors that indicate the input prompt exceeded the
* model's context window. Two recognized shapes:
* - `Anthropic.APIError` with `.status === 400` and message containing
* "prompt is too long" (current SDK wording, observed in production
* as `prompt is too long: 1707509 tokens > 1000000 maximum`).
* - Any error whose message includes "prompt is too long" (defensive
* against SDK-wrap shape changes).
*
* Case-insensitive on the phrase. Also matches `request_too_large` and
* `invalid_request_error` types when accompanied by the same message.
*
* Exported for unit testing.
*/
export function isPromptTooLongError(err: unknown): boolean {
if (!err) return false;
// Walk both `.message` and `.error?.message` shapes.
const msg = (err as { message?: unknown })?.message;
const inner = (err as { error?: { message?: unknown } })?.error?.message;
const candidates = [msg, inner].filter((s): s is string => typeof s === 'string');
for (const c of candidates) {
if (/prompt is too long/i.test(c)) return true;
}
// Anthropic SDK wraps with .status; 400 + 'invalid_request_error' /
// 'request_too_large' types both indicate the same class. Only treat
// as terminal when the message actually says prompt-too-long; broader
// 400s could be transient (e.g., malformed JSON from a test stub).
const status = (err as { status?: unknown })?.status;
const errType = (err as { error?: { type?: unknown } })?.error?.type;
if (status === 400 && (errType === 'invalid_request_error' || errType === 'request_too_large')) {
for (const c of candidates) {
if (/too long|exceed|maximum/i.test(c)) return true;
}
}
return false;
}
// ── Testing surface ─────────────────────────────────────────
export const __testing = {
+176
View File
@@ -0,0 +1,176 @@
/**
* Unit tests for the v0.30.2 dream/synthesize chunker (D9 hash-deterministic
* boundaries) and orchestrator slug rewrite (D6).
*
* Pure functions only. Exercises:
* - Single-chunk pass-through under budget.
* - 3-tier boundary ladder: ## Topic: > --- > nearest \n.
* - Hash determinism: same (content, hash, maxChars) identical chunks
* regardless of how many times you call it.
* - Different content_hash potentially different boundaries (jitter
* within back-half-of-budget window).
* - Hard fallback when no boundary fits.
* - Slug rewrite: bare hash6 adds -c<idx>; correctly suffixed unchanged;
* unknown shape pass-through.
*
* No DB, no Anthropic, no fixtures.
*/
import { describe, test, expect } from 'bun:test';
import { splitTranscriptByBudget, rewriteChunkedSlug } from '../src/core/cycle/synthesize.ts';
describe('splitTranscriptByBudget — single chunk path', () => {
test('returns single-element array when content <= maxChars', () => {
const out = splitTranscriptByBudget('hello world', 'abc123def', 1000);
expect(out).toEqual(['hello world']);
});
test('content exactly at maxChars stays one chunk', () => {
const content = 'x'.repeat(500);
const out = splitTranscriptByBudget(content, 'abc123def', 500);
expect(out).toHaveLength(1);
expect(out[0]).toEqual(content);
});
test('throws on non-positive maxChars', () => {
expect(() => splitTranscriptByBudget('hi', 'abc', 0)).toThrow(/maxChars/);
expect(() => splitTranscriptByBudget('hi', 'abc', -5)).toThrow(/maxChars/);
});
});
describe('splitTranscriptByBudget — boundary ladder', () => {
test('Tier 1: prefers \\n## Topic: separator inside back-half window', () => {
// Budget = 300 → searchStart ∈ [150, 179]. Place the topic separator
// around position 220 so it's deep in the back-half window.
const padding = 'x'.repeat(220); // 0..219
const sep = '\n## Topic: chunk-break\nafter break content'; // pos 220+
const tail = 'y'.repeat(200); // overflow
const content = padding + sep + tail;
const out = splitTranscriptByBudget(content, 'abcdef0123456789', 300);
expect(out.length).toBeGreaterThanOrEqual(2);
// Chunk 2 starts at the "\n## Topic:" boundary (the boundary char is
// included with the second chunk by design — `slice(0, split)` cuts
// before, `slice(split)` keeps the newline).
expect(out[1].startsWith('\n## Topic:')).toBe(true);
});
test('Tier 2: falls back to --- HR marker when no Topic separator in window', () => {
// Budget = 300, searchStart ∈ [150, 179]. HR marker around pos 220.
const padding = 'no-topic-marker-here\n'.repeat(10); // 210 chars (~10x21)
const sep = '\n---\nafter rule\n';
const tail = 'tail content '.repeat(20); // overflow
const content = padding + sep + tail;
const out = splitTranscriptByBudget(content, 'aabbccdd11223344', 300);
expect(out.length).toBeGreaterThanOrEqual(2);
// Chunk 2 starts at the HR marker.
expect(out[1].startsWith('\n---\n')).toBe(true);
});
test('Tier 3: falls back to nearest newline when no Topic / HR in window', () => {
// No topic separators, no HR markers — just paragraphs.
const para = 'sentence one. sentence two.\nmore prose to fill space\n';
const content = para.repeat(20); // ~1080 chars
const out = splitTranscriptByBudget(content, '11223344aabbccdd', 200);
expect(out.length).toBeGreaterThanOrEqual(2);
// Every chunk other than the last should END with content that allowed
// a newline split — chunks 2..N should NOT begin with a partial word.
for (let i = 1; i < out.length; i++) {
// After the split, the rest begins at the boundary char (newline);
// when the boundary is "\n", chunk i starts with the newline char.
expect(out[i].startsWith('\n')).toBe(true);
}
});
test('hard-split when no boundary fits anywhere in the window', () => {
// Single huge run of non-newline chars exceeding budget.
const content = 'a'.repeat(1500); // no newlines, no separators
const out = splitTranscriptByBudget(content, 'cafebabe12345678', 500);
// Walks deterministically: first split at maxChars=500, so chunks
// are [500, 500, 500].
expect(out).toHaveLength(3);
expect(out[0]).toHaveLength(500);
expect(out[1]).toHaveLength(500);
expect(out[2]).toHaveLength(500);
expect(out.join('')).toEqual(content);
});
});
describe('splitTranscriptByBudget — D9 hash-deterministic identity', () => {
test('same inputs → identical chunks across many calls', () => {
const content = ('paragraph with some text\nand a newline\n').repeat(100);
const hash = '0123456789abcdef0123456789abcdef';
const a = splitTranscriptByBudget(content, hash, 500);
const b = splitTranscriptByBudget(content, hash, 500);
const c = splitTranscriptByBudget(content, hash, 500);
expect(a).toEqual(b);
expect(b).toEqual(c);
});
test('different content_hash CAN produce different boundaries (jitter)', () => {
// Construct content with multiple newline candidates inside the
// back-half-of-budget search window, so different hash offsets pick
// different newlines.
const content = ('w1 w2 w3 w4 w5\n').repeat(200);
const a = splitTranscriptByBudget(content, '00000000aabbccdd', 500);
const b = splitTranscriptByBudget(content, 'ffffffff77665544', 500);
// The two splits MAY differ; assertion is that determinism is per-hash,
// not that the function is hash-invariant.
expect(a.join('')).toEqual(content);
expect(b.join('')).toEqual(content);
});
test('reconstructs content exactly when joined back', () => {
const content = ('# heading\n\nbody line\n## Topic: something\nmore body\n').repeat(50);
const out = splitTranscriptByBudget(content, 'aaaaaaaa11111111', 400);
expect(out.join('')).toEqual(content);
});
test('non-hex hash falls through to offset 0', () => {
// parseHashOffset returns 0 on bad hex; chunks should still be valid.
const content = ('xx\n').repeat(1000);
const out = splitTranscriptByBudget(content, '!!!not-hex!!!', 500);
expect(out.length).toBeGreaterThanOrEqual(2);
expect(out.join('')).toEqual(content);
});
});
describe('rewriteChunkedSlug — D6 zero-Sonnet-trust slug rewrite', () => {
test('appends -c<idx> when slug ends with bare hash6', () => {
expect(rewriteChunkedSlug('wiki/originals/ideas/2026-05-08-thesis-abc123', 'abc123', 0))
.toBe('wiki/originals/ideas/2026-05-08-thesis-abc123-c0');
expect(rewriteChunkedSlug('wiki/personal/reflections/2026-05-08-foo-deadbe', 'deadbe', 2))
.toBe('wiki/personal/reflections/2026-05-08-foo-deadbe-c2');
});
test('passes through when slug already correctly chunk-suffixed', () => {
expect(rewriteChunkedSlug('wiki/originals/ideas/2026-05-08-thesis-abc123-c0', 'abc123', 0))
.toBe('wiki/originals/ideas/2026-05-08-thesis-abc123-c0');
expect(rewriteChunkedSlug('foo-bar-deadbe-c5', 'deadbe', 5))
.toBe('foo-bar-deadbe-c5');
});
test('does not double-rewrite when -c<otherIdx> would conflict with idx', () => {
// If a slug already has a chunk suffix (any idx) it ends with -c<digits>;
// the regex is anchored on bare hash6 at end, so this case passes through.
const result = rewriteChunkedSlug('foo-abc123-c1', 'abc123', 0);
expect(result).toBe('foo-abc123-c1'); // unchanged — does not become foo-abc123-c1-c0
});
test('passes through when slug does not end with the expected hash6', () => {
expect(rewriteChunkedSlug('wiki/foo/bar', 'abc123', 0)).toBe('wiki/foo/bar');
expect(rewriteChunkedSlug('wiki/foo/bar-xyzxyz', 'abc123', 0)).toBe('wiki/foo/bar-xyzxyz');
});
test('handles slug that IS exactly hash6', () => {
expect(rewriteChunkedSlug('abc123', 'abc123', 3)).toBe('abc123-c3');
});
test('handles slug ending with /<hash6> path-segment shape', () => {
expect(rewriteChunkedSlug('foo/bar/abc123', 'abc123', 1))
.toBe('foo/bar/abc123-c1');
});
test('empty slug passes through', () => {
expect(rewriteChunkedSlug('', 'abc123', 0)).toBe('');
});
});
+307
View File
@@ -0,0 +1,307 @@
/**
* E2E for v0.30.2 dream/synthesize chunking. PGLite, no API key required.
*
* Pre-seeds verdicts so the Haiku gate is bypassed; submits subagent jobs
* but never runs them (no worker spawned). Tests inspect minion_jobs to
* verify submission shape (chunk count, idempotency keys, skip-paths).
*
* Coverage:
* - D5 cap-hit: chunks > maxChunks log + skip with no minion_jobs row
* and no dream_verdicts cache write (closes the poison-pill class).
* - D8 legacy single-chunk migration: pre-seed a `completed` legacy job
* for the same content hash next synthesize skips submission.
* - Chunked path: fat transcript spawns N children with chunk-suffixed
* idempotency keys; single-chunk path keeps the legacy key shape.
*
* Run: bun test test/e2e/dream-synthesize-chunking.test.ts
*/
import { describe, test, expect } from 'bun:test';
import { mkdtempSync, rmSync, writeFileSync } from 'fs';
import { tmpdir } from 'os';
import { join } from 'path';
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
import { runPhaseSynthesize } from '../../src/core/cycle/synthesize.ts';
interface TestRig {
engine: PGLiteEngine;
brainDir: string;
corpusDir: string;
cleanup: () => Promise<void>;
}
async function setupRig(): Promise<TestRig> {
const engine = new PGLiteEngine();
await engine.connect({ engine: 'pglite' } as never);
await engine.initSchema();
const brainDir = mkdtempSync(join(tmpdir(), 'gbrain-chunk-brain-'));
const corpusDir = mkdtempSync(join(tmpdir(), 'gbrain-chunk-corpus-'));
return {
engine,
brainDir,
corpusDir,
cleanup: async () => {
try { await engine.disconnect(); } catch { /* best-effort */ }
try { rmSync(brainDir, { recursive: true, force: true }); } catch { /* */ }
try { rmSync(corpusDir, { recursive: true, force: true }); } catch { /* */ }
},
};
}
async function withoutAnthropicKey<T>(body: () => Promise<T>): Promise<T> {
const saved = process.env.ANTHROPIC_API_KEY;
delete process.env.ANTHROPIC_API_KEY;
try {
return await body();
} finally {
if (saved === undefined) delete process.env.ANTHROPIC_API_KEY;
else process.env.ANTHROPIC_API_KEY = saved;
}
}
/**
* Run `body` while a background loop force-cancels any subagent jobs the
* synthesize phase submits. Without a worker, those jobs would sit in
* `waiting` forever and runPhaseSynthesize's waitForCompletion blocks for
* 35 minutes. Cancelling moves them to a terminal state so the phase
* returns and we can inspect submission shape.
*/
async function withSubagentAutoCancel<T>(engine: PGLiteEngine, body: () => Promise<T>): Promise<T> {
let stopped = false;
const loop = (async () => {
while (!stopped) {
await new Promise(r => setTimeout(r, 50));
try {
await engine.executeRaw(
`UPDATE minion_jobs
SET status = 'cancelled', finished_at = now()
WHERE name = 'subagent' AND status IN ('waiting', 'active')`,
);
} catch {
// Race against shutdown is fine; ignore.
}
}
})();
try {
return await body();
} finally {
stopped = true;
await loop;
}
}
/**
* Pre-seed a `worth_processing=true` verdict so the synthesize phase skips
* the Haiku call and proceeds directly to fan-out. Computes the hash the
* same way `discoverTranscripts` does (sha256 of content).
*/
async function seedVerdict(engine: PGLiteEngine, filePath: string, content: string): Promise<string> {
const { createHash } = await import('node:crypto');
const contentHash = createHash('sha256').update(content, 'utf8').digest('hex');
await engine.putDreamVerdict(filePath, contentHash, {
worth_processing: true,
reasons: ['seeded for chunking E2E test'],
});
return contentHash;
}
/**
* Resolve the absolute path the discover walker will see for a file in the
* corpus dir, since `discoverTranscripts` joins corpus + name.
*/
function corpusPath(corpusDir: string, basename: string): string {
return join(corpusDir, basename);
}
describe('E2E synthesize chunking — D5 cap hit', () => {
test('chunks > max_chunks_per_transcript → skipped with no jobs and no verdict-cache write', async () => {
const rig = await setupRig();
try {
// Tiny chunk budget (forces N chunks) + tiny cap (forces cap hit).
// 100K is the floor; even at the floor, 350K-char tester content
// chunks to ~1 chunk... we need budget below floor to force many
// chunks. Use the chunks_per_transcript cap instead.
await rig.engine.setConfig('dream.synthesize.enabled', 'true');
await rig.engine.setConfig('dream.synthesize.session_corpus_dir', rig.corpusDir);
await rig.engine.setConfig('dream.synthesize.max_prompt_tokens', '100000'); // floor → 350K char budget
await rig.engine.setConfig('dream.synthesize.max_chunks_per_transcript', '2');
// 1.5M chars → 5 chunks at 350K-char budget → exceeds cap=2.
const basename = '2026-05-08-fat-transcript.txt';
const filePath = corpusPath(rig.corpusDir, basename);
const content = 'fat transcript line\n'.repeat(75_000); // ~1.5M chars
writeFileSync(filePath, content);
await seedVerdict(rig.engine, filePath, content);
await withoutAnthropicKey(async () => {
const result = await runPhaseSynthesize(rig.engine, {
brainDir: rig.brainDir,
dryRun: false,
});
expect(result.status).toBe('ok');
const details = result.details as {
children_submitted: number;
skips: Array<{ filePath: string; reason: string }>;
};
expect(details.children_submitted).toBe(0);
expect(details.skips).toHaveLength(1);
expect(details.skips[0].filePath).toBe(filePath);
expect(details.skips[0].reason).toMatch(/oversize_after_split/);
});
// No subagent jobs submitted.
const jobs = await rig.engine.executeRaw<{ cnt: string | number }>(
`SELECT count(*) AS cnt FROM minion_jobs WHERE name = 'subagent'`,
);
expect(Number(jobs[0].cnt)).toBe(0);
// D5: dream_verdicts NOT written for the cap-hit path.
// Verify by re-reading the verdict — our seeded row is the ONLY entry.
const verdicts = await rig.engine.executeRaw<{ cnt: string | number }>(
`SELECT count(*) AS cnt FROM dream_verdicts`,
);
expect(Number(verdicts[0].cnt)).toBe(1); // only the seed; no cap-hit row added
} finally {
await rig.cleanup();
}
}, 30_000);
});
describe('E2E synthesize chunking — D8 legacy single-chunk migration', () => {
test('completed legacy idempotency key → skip submission entirely', async () => {
const rig = await setupRig();
try {
await rig.engine.setConfig('dream.synthesize.enabled', 'true');
await rig.engine.setConfig('dream.synthesize.session_corpus_dir', rig.corpusDir);
const basename = '2026-04-25-already-synthesized.txt';
const filePath = corpusPath(rig.corpusDir, basename);
const content = 'meaningful conversation lines\n'.repeat(200);
writeFileSync(filePath, content);
const contentHash = await seedVerdict(rig.engine, filePath, content);
// Pre-seed a completed `subagent` job at the legacy idempotency key.
const legacyKey = `dream:synth:${filePath}:${contentHash.slice(0, 16)}`;
await rig.engine.executeRaw(
`INSERT INTO minion_jobs (name, queue, status, idempotency_key, finished_at)
VALUES ('subagent', 'default', 'completed', $1, now())`,
[legacyKey],
);
await withoutAnthropicKey(async () => {
const result = await runPhaseSynthesize(rig.engine, {
brainDir: rig.brainDir,
dryRun: false,
});
const details = result.details as {
children_submitted: number;
skips: Array<{ reason: string }>;
};
expect(details.children_submitted).toBe(0);
expect(details.skips).toHaveLength(1);
expect(details.skips[0].reason).toBe('already_synthesized_legacy_single_chunk');
});
// No NEW subagent job: still exactly one (the seeded completed row).
const jobs = await rig.engine.executeRaw<{ cnt: string | number }>(
`SELECT count(*) AS cnt FROM minion_jobs WHERE name = 'subagent'`,
);
expect(Number(jobs[0].cnt)).toBe(1);
} finally {
await rig.cleanup();
}
}, 30_000);
});
describe('E2E synthesize chunking — fan-out shape', () => {
test('single-chunk transcript uses legacy idempotency key (parity on upgrade)', async () => {
const rig = await setupRig();
try {
await rig.engine.setConfig('dream.synthesize.enabled', 'true');
await rig.engine.setConfig('dream.synthesize.session_corpus_dir', rig.corpusDir);
// Default budget is plenty for 5KB content.
const basename = '2026-04-25-small.txt';
const filePath = corpusPath(rig.corpusDir, basename);
const content = 'small transcript content\n'.repeat(100); // ~2.5KB
writeFileSync(filePath, content);
const contentHash = await seedVerdict(rig.engine, filePath, content);
await withoutAnthropicKey(async () => {
await withSubagentAutoCancel(rig.engine, async () => {
const result = await runPhaseSynthesize(rig.engine, {
brainDir: rig.brainDir,
dryRun: false,
});
const details = result.details as { children_submitted: number };
expect(details.children_submitted).toBe(1);
});
});
const expectedKey = `dream:synth:${filePath}:${contentHash.slice(0, 16)}`;
const rows = await rig.engine.executeRaw<{ idempotency_key: string }>(
`SELECT idempotency_key FROM minion_jobs WHERE name = 'subagent' ORDER BY id`,
);
expect(rows).toHaveLength(1);
expect(rows[0].idempotency_key).toBe(expectedKey);
// Specifically: legacy key shape has NO ":c<idx>of<n>" suffix.
expect(rows[0].idempotency_key).not.toMatch(/:c\d+of\d+$/);
} finally {
await rig.cleanup();
}
}, 30_000);
test('multi-chunk transcript spawns N children with chunk-suffixed idempotency keys', async () => {
const rig = await setupRig();
try {
await rig.engine.setConfig('dream.synthesize.enabled', 'true');
await rig.engine.setConfig('dream.synthesize.session_corpus_dir', rig.corpusDir);
// Floor at 100K tokens → 350K-char chunk budget. A 1.5M-char transcript
// chunks to ~5 chunks. Default cap is 24, so submission proceeds.
await rig.engine.setConfig('dream.synthesize.max_prompt_tokens', '100000');
const basename = '2026-05-08-fat.txt';
const filePath = corpusPath(rig.corpusDir, basename);
const content = 'fat transcript line with newline\n'.repeat(50_000); // ~1.65M chars
writeFileSync(filePath, content);
const contentHash = await seedVerdict(rig.engine, filePath, content);
const hash16 = contentHash.slice(0, 16);
await withoutAnthropicKey(async () => {
await withSubagentAutoCancel(rig.engine, async () => {
const result = await runPhaseSynthesize(rig.engine, {
brainDir: rig.brainDir,
dryRun: false,
});
const details = result.details as { children_submitted: number };
expect(details.children_submitted).toBeGreaterThan(1);
});
});
const rows = await rig.engine.executeRaw<{ idempotency_key: string }>(
`SELECT idempotency_key FROM minion_jobs WHERE name = 'subagent' ORDER BY id`,
);
expect(rows.length).toBeGreaterThan(1);
// Every key matches the chunked shape `dream:synth:<path>:<hash16>:c<i>of<N>`.
for (const r of rows) {
expect(r.idempotency_key).toMatch(
new RegExp(`^dream:synth:${escapeRe(filePath)}:${hash16}:c\\d+of\\d+$`),
);
}
// Chunk indices are unique 0..N-1.
const indices = rows
.map(r => /:c(\d+)of/.exec(r.idempotency_key)?.[1])
.map(s => Number(s))
.sort((a, b) => a - b);
const expected = Array.from({ length: rows.length }, (_, i) => i);
expect(indices).toEqual(expected);
} finally {
await rig.cleanup();
}
}, 30_000);
});
function escapeRe(s: string): string {
return s.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
}
+91
View File
@@ -0,0 +1,91 @@
/**
* Unit tests for the v0.30.2 terminal-error classification of Anthropic's
* "prompt is too long" 400 in the subagent handler.
*
* The handler converts these to UnrecoverableError so the worker routes the
* job straight to `dead`, bypassing max_stalled retries (the prior 3x-retry
* pathology that clogged the queue).
*
* Pure function tests of `isPromptTooLongError`. End-to-end coverage of the
* client.create() try/catch lives in the synthesize E2E test.
*/
import { describe, test, expect } from 'bun:test';
import { isPromptTooLongError } from '../src/core/minions/handlers/subagent.ts';
describe('isPromptTooLongError', () => {
test('matches the production message verbatim', () => {
const err = new Error('prompt is too long: 1707509 tokens > 1000000 maximum');
expect(isPromptTooLongError(err)).toBe(true);
});
test('matches case-insensitively', () => {
expect(isPromptTooLongError(new Error('Prompt Is Too Long'))).toBe(true);
expect(isPromptTooLongError(new Error('PROMPT IS TOO LONG'))).toBe(true);
});
test('matches when message is on the inner .error.message field', () => {
// Mimic Anthropic SDK error wrapping shape.
const err = {
status: 400,
error: {
type: 'invalid_request_error',
message: 'prompt is too long: 1234567 tokens > 1000000 maximum',
},
message: 'BadRequestError',
};
expect(isPromptTooLongError(err)).toBe(true);
});
test('matches 400 + invalid_request_error + "exceed" wording (defensive)', () => {
// Defensive against future SDK message-wording changes.
const err = {
status: 400,
error: { type: 'invalid_request_error', message: 'request exceeds maximum context' },
message: 'BadRequestError',
};
expect(isPromptTooLongError(err)).toBe(true);
});
test('matches 400 + request_too_large type', () => {
const err = {
status: 400,
error: { type: 'request_too_large', message: 'too long' },
message: 'BadRequestError',
};
expect(isPromptTooLongError(err)).toBe(true);
});
test('does NOT match unrelated 400 errors', () => {
const err = {
status: 400,
error: { type: 'invalid_request_error', message: 'malformed JSON' },
message: 'BadRequestError',
};
expect(isPromptTooLongError(err)).toBe(false);
});
test('does NOT match unrelated transient errors', () => {
expect(isPromptTooLongError(new Error('network timeout'))).toBe(false);
expect(isPromptTooLongError(new Error('rate limit exceeded'))).toBe(false);
expect(isPromptTooLongError({ status: 500, message: 'internal error' })).toBe(false);
expect(isPromptTooLongError({ status: 429, message: 'overloaded' })).toBe(false);
});
test('does NOT match null / undefined / non-error inputs', () => {
expect(isPromptTooLongError(null)).toBe(false);
expect(isPromptTooLongError(undefined)).toBe(false);
expect(isPromptTooLongError(0)).toBe(false);
expect(isPromptTooLongError('plain string')).toBe(false);
expect(isPromptTooLongError({})).toBe(false);
});
test('matches synthetic SDK shape with status 400 + message containing the phrase', () => {
// Some SDK versions surface the phrase only on the outer .message.
const err = {
status: 400,
message: 'Error: prompt is too long: 2000000 tokens > 1000000 maximum',
};
expect(isPromptTooLongError(err)).toBe(true);
});
});