From 410c6978a4dcc0abea336ef1b80b4792206955ab Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Fri, 8 May 2026 17:07:51 -0700 Subject: [PATCH] v0.30.2 feat: dream synthesize stops dropping fat transcripts (#754) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat: classify Anthropic prompt-too-long as UnrecoverableError The subagent handler now detects 400 "prompt is too long" responses from the Anthropic SDK and rethrows as UnrecoverableError. The worker already routes UnrecoverableError straight to `dead`, so doomed jobs fail terminally on first attempt instead of stalling 3x with the same oversized prompt. isPromptTooLongError matches the production message verbatim ("prompt is too long: N tokens > N maximum"), case-insensitive, on both the outer message and inner error.message paths. Defensive secondary match for status=400 + invalid_request_error/request_too_large with the words "too long"/"exceed"/"maximum". 9 unit cases pin the detection: production wording, case folding, nested SDK shape, defensive 400 paths, unrelated 400s, transient errors, null/empty inputs. * feat: model-aware chunking + slug-rewrite for dream synthesize The synthesize phase now chunks oversized transcripts at paragraph boundaries instead of submitting one giant prompt that 400s on Anthropic. Closes the v0.30 dream-cycle queue clog where 1.7M-token transcripts dead-lettered after 3 stalls and re-discovered every cycle. D1: per-chunk budget = floor(model_context_tokens × 0.9 × 3.5). MODEL_CONTEXT_TOKENS keys on resolved Anthropic ids (Opus 4.7 = 1M, Sonnet 4.6 = 200K, Haiku = 200K). Non-Anthropic models fall back to 180K-token safe default with a once-per-process stderr warning. dream.synthesize.max_prompt_tokens overrides the model lookup (token-shaped, name from PR #748, floor 100K). D5: on max_chunks_per_transcript cap hit, log + skip; do NOT write to dream_verdicts. Closes the cache-poisoning class — next cycle re-attempts under whatever budget is then current. D6: orchestrator-side deterministic slug rewrite, zero Sonnet trust. collectChildPutPageSlugs raw-fetches every (job_id, slug) pair (no SELECT DISTINCT — that erased the collision evidence the audit claimed to detect) and rewrites bare-hash6 slugs to -c for chunked children. D8: pre-fan-out lookup of completed legacy `dream:synth:: ` jobs. Transcripts already synthesized under the single-chunk shape skip submission with `already_synthesized_legacy_ single_chunk` instead of resubmitting under chunked keys. D9: hash-deterministic chunk boundaries. The 3-tier ladder lifted from PR #748 (## Topic: > --- > nearest \\n) is fed a back-half search-window offset derived from contentHash. Same content always chunks identically across runs; chunk N of a previously-failed transcript produces byte-identical content on retry. D10: 24-chunk default cap, operator-configurable via dream.synthesize.max_chunks_per_transcript. 18 unit cases pin the chunker (boundary ladder, hash determinism, hard fallback, slug rewrite all 7 shapes). 4 PGLite E2E cases pin fan-out shape (single-chunk legacy key parity, multi-chunk chunked key shape) + skip paths (D5 cap hit no verdict-cache write, D8 legacy-key skip). Credits PR #748 (Wintermute) for the boundary ladder, config key naming, and 3.5 chars/token estimator. This branch supersedes #748 with the structural safeguards (model-aware budget, terminal-error classify, slug rewrite, hash-determinism, doctor surfacing). * feat: surface dead-lettered prompt_too_long jobs in doctor queue_health queue_health gains a 4th subcheck counting dead `subagent` jobs in the last 24h whose error_text starts with `prompt_too_long:`. When present, prints a fix hint pointing at `gbrain dream --phase synthesize --dry-run --json` to identify the fat transcripts and naming the two operator escape hatches (`dream.synthesize.max_prompt_tokens` for budget tuning, larger-context model for capacity). Operators now see the chunking failure mode without grepping minion_jobs by hand. * chore: bump version and changelog (v0.30.2) Co-Authored-By: Claude Opus 4.7 (1M context) * docs: update README + CLAUDE.md for v0.30.2 - README dream help: 8-phase → 9-phase, mention v0.30.2 chunking + config keys - CLAUDE.md synthesize.ts: chunker + per-chunk idempotency + D6 slug rewrite + D7 scope + D8 legacy-key - CLAUDE.md subagent.ts: prompt_too_long terminal classification - CLAUDE.md doctor.ts: queue_health subcheck 4 (dead-lettered prompt_too_long) Co-Authored-By: Claude Opus 4.7 * docs: regenerate llms-full.txt after v0.30.2 CLAUDE.md updates The docs/ pass extended three Key Files entries in CLAUDE.md (synthesize.ts, subagent.ts, doctor.ts). The auto-derived llms-full.txt bundle picks up those CLAUDE.md changes via build-llms; the build-llms test caught the drift in CI. Generated by: bun run build:llms --------- Co-authored-by: Claude Opus 4.7 (1M context) --- CHANGELOG.md | 61 ++++ CLAUDE.md | 6 +- README.md | 8 +- VERSION | 2 +- llms-full.txt | 14 +- package.json | 2 +- src/commands/doctor.ts | 24 ++ src/core/cycle/synthesize.ts | 406 +++++++++++++++++++-- src/core/minions/handlers/subagent.ts | 47 +++ test/cycle-synthesize-chunker.test.ts | 176 +++++++++ test/e2e/dream-synthesize-chunking.test.ts | 307 ++++++++++++++++ test/subagent-prompt-too-long.test.ts | 91 +++++ 12 files changed, 1096 insertions(+), 48 deletions(-) create mode 100644 test/cycle-synthesize-chunker.test.ts create mode 100644 test/e2e/dream-synthesize-chunking.test.ts create mode 100644 test/subagent-prompt-too-long.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 2a2e3c46b..fb73d3c9e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,67 @@ All notable changes to GBrain will be documented in this file. +## [0.30.2] - 2026-05-08 + +**Dream synthesize stops dropping fat transcripts. Subagents that overflow Anthropic's context die once, not three times. The queue stops clogging.** + +The v0.30 dream cycle has been stalled since May 2 for one user — daily aggregated transcripts at 2.7-4.5MB each generate 1.7M-token Anthropic prompts, which hit the 1M-token hard limit and 400. The subagent handler treated those failures as renewable, so every doomed transcript stalled three times before dead-lettering, and every new cycle re-discovered and re-submitted the same fat transcripts. Six days of synth backlog, queue full of doomed work. + +v0.30.2 adds model-aware chunking + terminal-error classification + a poison-pill-free skip path. Wintermute's [PR #748](https://github.com/garrytan/gbrain/pull/748) supplied the boundary heuristics (`## Topic:` → `---` → `\n` ladder) and the `dream.synthesize.max_prompt_tokens` config surface. Garry's branch extended that with model-aware budgets, deterministic chunk identity for partial-progress safety, orchestrator-side slug rewriting for zero Sonnet trust on collisions, and doctor-surface visibility. + +### What you can now do + +**Fat transcripts get synthesized instead of dropped.** A 4.5MB daily-aggregated transcript on Sonnet 4.6 (200K context) now chunks into 7-8 children at the 630KB-per-chunk default budget, each subagent processes its slice, and the orchestrator collects everything. Per-chunk children are independent — chunk 5's content doesn't leak into chunk 6's prompt. + +**Doomed transcripts die loud, not silent.** Anthropic 400 "prompt is too long" responses now classify as `UnrecoverableError`. The job goes straight to `dead` on first attempt. No three-stall retry pile. `gbrain doctor` surfaces the count under `queue_health` with a fix hint pointing at `gbrain dream --phase synthesize --dry-run --json` so you can identify the offender. + +**Existing single-chunk transcripts are not re-synthesized on upgrade.** The legacy `dream:synth::` idempotency-key shape is preserved byte-for-byte for the single-chunk case. Already-synthesized transcripts skip with `already_synthesized_legacy_single_chunk` — no re-spend on Sonnet for content already in the brain. + +```bash +# Operator escape hatches +gbrain config set dream.synthesize.max_prompt_tokens 600000 # tune chunk budget below model context +gbrain config set dream.synthesize.max_chunks_per_transcript 32 # raise the chunk-explosion cap +gbrain jobs prune --status dead --queue default # one-time cleanup of pre-v0.30.2 doomed jobs +``` + +### How it works under the hood + +**Model-aware chunk budget.** A static `MODEL_CONTEXT_TOKENS` map keys on the resolved Anthropic model id (Opus 4.7 → 1M, Sonnet 4.6 → 200K, Haiku → 200K). Per-chunk budget is `floor(context × 0.9 × 3.5 chars/token)`, leaving 10% headroom for system prompt + tool defs + output. Non-Anthropic ids (`gpt-5`, `gemini-3-pro`, custom alias) fall back to a 180K-token safe default with a once-per-process stderr warning. + +**Hash-deterministic chunk identity.** `splitTranscriptByBudget(content, contentHash, maxChars)` is a pure function. The 3-tier boundary search window (back-half-of-budget) gets seeded with a deterministic offset derived from the first 32 bits of `contentHash`, so the same content always chunks identically. Chunk 2 of a transcript that died terminally produces byte-identical content on retry — per-chunk idempotency keys (`dream:synth:::cof`) are durable across runs. + +**Orchestrator-side slug rewrite, zero Sonnet trust.** `collectChildPutPageSlugs` no longer does `SELECT DISTINCT` (which would erase collision evidence). Instead it raw-fetches every (job_id, slug) pair, then for chunked children rewrites bare-hash6 slugs to `-c` if Sonnet drops the chunk suffix. Two siblings can't collide even if Sonnet ignores the prompt's `USE THIS in slugs` rule. + +**Cap-hit skips don't poison the verdict cache.** When chunks exceed `max_chunks_per_transcript` (default 24), the orchestrator logs + skips without writing to `dream_verdicts`. Next cycle re-attempts under whatever budget is then current — closes the cache-poisoning class entirely. + +### Out of scope (deferred to v0.30.3+) + +- **Per-turn token-budget guard in subagent.ts.** This release bounds the INITIAL prompt size only. Tool-loop accumulation (search/get_page results bloating each subsequent turn) can still hit `prompt_too_long` mid-conversation; the terminal-error classification catches it then, just less cleanly. +- **Tighter token estimator for code/JSON/CJK-dense content.** The 3.5 chars/token ratio is close to safe for English transcripts but can underflow on dense content. Production telemetry will tell us if a per-recipe override is needed. + +### To take advantage of v0.30.2 + +`gbrain upgrade` is enough — the change is purely runtime behavior, no schema migration required. + +If your dream queue accumulated doomed subagent jobs from a prior version: +```bash +gbrain jobs list --status dead --queue default | grep prompt_too_long +gbrain jobs prune --status dead --queue default +``` + +If you want to verify chunking on your specific corpus before next autopilot cycle: +```bash +gbrain dream --phase synthesize --dry-run --json | jq '.details.skips' +``` + +A non-empty `skips` array means a transcript hit the 24-chunk cap or already exists at the legacy single-chunk key. Empty means everything fits. + +### For contributors + +Two-round Codex outside-voice review surfaced 12 structural risks; six folded directly into the design (D5 cap-hit no-cache, D6 orchestrator slug rewrite, D7 honest tool-loop scope, D8 legacy-key migration, D9 hash-deterministic boundaries, D10 operator-configurable cap), and the four PARTIAL items have follow-up TODOs. The plan file at `~/.claude/plans/system-instruction-you-are-working-mossy-fox.md` documents the full decision history. PR #748's contribution (boundary ladder + config-key naming + 3.5 chars/token estimator) is preserved verbatim and credited; this branch supersedes it with the structural safeguards. + +Tests: 27 unit cases (`test/cycle-synthesize-chunker.test.ts`, `test/subagent-prompt-too-long.test.ts`) + 4 PGLite E2E cases (`test/e2e/dream-synthesize-chunking.test.ts`) covering D5 cap hit, D8 legacy-key skip, single-chunk parity, multi-chunk fan-out shape. + ## [0.30.1] - 2026-05-08 **Operational hardening: gbrain upgrade just works on Supabase. DDL stops timing out on the pooler. Migrations stop wedging. HNSW rebuilds stop nuking your search. Backfills stop being bespoke scripts.** diff --git a/CLAUDE.md b/CLAUDE.md index 0f5a2a384..50cfd829b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -122,7 +122,7 @@ strict behavior when unset. - `src/core/minions/handlers/shell.ts` — `shell` job handler. Spawns `/bin/sh -c cmd` (absolute path, PATH-override-safe) or `argv[0] argv[1..]` (no shell). Env allowlist: `PATH, HOME, USER, LANG, TZ, NODE_ENV` + caller `env:` overrides. UTF-8-safe stdout/stderr tail via `string_decoder.StringDecoder`. Abort (either `ctx.signal` or `ctx.shutdownSignal`) fires SIGTERM → 5s grace → SIGKILL on child. Requires `GBRAIN_ALLOW_SHELL_JOBS=1` on worker (gated by `registerBuiltinHandlers`). - `src/core/minions/handlers/shell-audit.ts` — per-submission JSONL audit trail at `~/.gbrain/audit/shell-jobs-YYYY-Www.jsonl` (ISO-week rotation; override via `GBRAIN_AUDIT_DIR`). Best-effort: `mkdirSync(recursive)` + `appendFileSync`; failures logged to stderr, submission not blocked. Logs cmd (first 80 chars) or argv (JSON array). Never logs env values. - `src/core/minions/backpressure-audit.ts` (v0.19.1) — sibling of shell-audit.ts for `maxWaiting` coalesce events. JSONL at `~/.gbrain/audit/backpressure-YYYY-Www.jsonl`. Fires one line per coalesce with `(queue, name, waiting_count, max_waiting, returned_job_id, ts)`. Closes the silent-drop vector the v0.19.0 maxWaiting guard introduced. -- `src/core/minions/handlers/subagent.ts` (v0.15) — LLM-loop handler. Two-phase tool persistence (pending → complete/failed), replay reconciliation for mid-dispatch crashes, dual-signal abort (`ctx.signal` + `ctx.shutdownSignal`), Anthropic prompt caching on system + tool defs. `makeSubagentHandler({engine, client?, ...})` factory; `MessagesClient` is an injectable interface the real SDK implements structurally. Throws `RateLeaseUnavailableError` (renewable) when rate-lease capacity is full. +- `src/core/minions/handlers/subagent.ts` (v0.15) — LLM-loop handler. Two-phase tool persistence (pending → complete/failed), replay reconciliation for mid-dispatch crashes, dual-signal abort (`ctx.signal` + `ctx.shutdownSignal`), Anthropic prompt caching on system + tool defs. `makeSubagentHandler({engine, client?, ...})` factory; `MessagesClient` is an injectable interface the real SDK implements structurally. Throws `RateLeaseUnavailableError` (renewable) when rate-lease capacity is full. **v0.30.2:** Anthropic 400 `prompt is too long` responses (status 400 + body matches `/prompt is too long|prompt_too_long|context.*length/i`) classify as `UnrecoverableError` so the job goes straight to `dead` on first attempt instead of stalling three times before dead-lettering. Catches both initial-prompt overflow and turn-N tool-loop accumulation that the chunker in `synthesize.ts` can't bound ahead of time. - `src/core/minions/handlers/subagent-aggregator.ts` (v0.15) — `subagent_aggregator` handler. Claims AFTER all children resolve (queue changes guarantee every terminal child posts a `child_done` inbox message with outcome). Reads inbox via `ctx.readInbox()`, builds deterministic mixed-outcome markdown summary. No LLM call in v0.15. - `src/core/minions/handlers/subagent-audit.ts` (v0.15) — JSONL audit + heartbeat writer at `~/.gbrain/audit/subagent-jobs-YYYY-Www.jsonl`. Events: `submission` (one line per submit) + `heartbeat` (per turn boundary: `llm_call_started | llm_call_completed | tool_called | tool_result | tool_failed`). Never logs prompts or tool inputs. `readSubagentAuditForJob(jobId, {sinceIso})` is the readback path for `gbrain agent logs`. - `src/core/minions/rate-leases.ts` (v0.15) — lease-based concurrency cap for outbound providers (default key `anthropic:messages`, max via `GBRAIN_ANTHROPIC_MAX_INFLIGHT`). Owner-tagged rows with `expires_at` auto-prune on acquire; `pg_advisory_xact_lock` guards check-then-insert; CASCADE on owning job deletion. `renewLeaseWithBackoff` retries 3x (250/500/1000ms). @@ -152,7 +152,7 @@ strict behavior when unset. - `src/commands/anomalies.ts` (v0.29) — `gbrain anomalies [--since YYYY-MM-DD] [--lookback-days N] [--sigma N] [--json]`: cohort-level activity outliers. Calls `engine.findAnomalies(opts)`. Two cohort kinds in v1: tag, type. Year cohort deferred to v0.30. - `src/commands/transcripts.ts` (v0.29) — `gbrain transcripts recent [--days N] [--full] [--json]`: recent raw `.txt` transcripts from the dream-cycle corpus dirs. Imports `listRecentTranscripts` from `src/core/transcripts.ts` (the same library the gated `get_recent_transcripts` MCP op uses). Local-only by construction — the CLI always runs with `ctx.remote=false`. - `src/commands/integrity.ts` — `gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). v0.22.8: batch-load fast path on Postgres uses `SELECT DISTINCT ON (slug)` in a single SQL query to fix the PgBouncer round-trip timeout (60s → ~6s) while preserving `engine.getAllSlugs()`'s `Set` semantics on multi-source brains. Gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1` so real Postgres errors are diagnosable. -- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel `. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`. +- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel `. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`. **v0.30.2:** `queue_health` gains a fourth subcheck — surfaces dead-lettered subagent jobs with `last_error` matching the `prompt_too_long` classifier within the last 24h. Fix hint points at `gbrain dream --phase synthesize --dry-run --json` to identify the offending transcript and `gbrain jobs prune --status dead --queue default` to clean up. Postgres-only. - `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). **v40 (v0.29):** `pages_emotional_weight` adds `pages.emotional_weight REAL NOT NULL DEFAULT 0.0`. Column-only (no index). On Postgres 11+ and PGLite, `ADD COLUMN` with a constant DEFAULT is metadata-only — instant on tables of any size. v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). **v30 (v0.23):** creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))`. RLS-enabled when running as a BYPASSRLS role. The synthesize phase reads/writes this table to avoid re-judging on backfill re-runs. **v35 (v0.26.7):** auto-RLS event trigger + one-time backfill. `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` and runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on every new `public.*` table — no FORCE (matches v24/v29/schema.sql posture so non-BYPASSRLS apps can still read their own tables). The same migration backfills RLS on every existing `public.*` base table whose comment doesn't match the doctor regex (`^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}`). Per-table failure aborts the offending CREATE TABLE (event triggers fire inside the DDL transaction); no EXCEPTION wrap — that would convert loud rollback into silent permissive default. PGLite no-op via `sqlFor.pglite: ''`. Breaking change: operators with intentionally-RLS-off public tables must add the GBRAIN:RLS_EXEMPT comment BEFORE upgrade or the backfill will flip them on. - `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2. - `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers. @@ -160,7 +160,7 @@ strict behavior when unset. - `src/core/sync-concurrency.ts` (v0.22.13) — single source of truth for the parallel-sync policy. Exports `autoConcurrency(engine, fileCount, override?)` (PGLite always serial; explicit override clamped to >=1; auto path returns `DEFAULT_PARALLEL_WORKERS=4` when `fileCount > AUTO_CONCURRENCY_FILE_THRESHOLD=100`), `shouldRunParallel(workers, fileCount, explicit)` (Q1: explicit `--workers` bypasses the >50-file floor), and `parseWorkers(s)` (rejects `'0'`, `'-3'`, `'foo'`, `'1.5'`, trailing chars — replaces the prior parseInt-with-no-validation in both `sync.ts` and `import.ts`). Used by `performSync`, `performFullSync`, `runImport`, and the Minion `sync` handler so the three sites can no longer drift. - `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). v0.22.13 (PR #490): `performSync` wraps its body in a `gbrain-sync` writer lock so two concurrent syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot both write `last_commit` and let the last writer win. Head-drift gate after the import phase re-checks `git rev-parse HEAD`; if HEAD moved (someone ran `git checkout` / `git pull` mid-sync), the bookmark refuses to advance. Vanished files now record a failedFiles entry instead of silent-skip — the silent-skip-then-advance pathology that survived prior hardening passes is dead. Worker engines wrap in try/finally so disconnect always fires (panic-path leak fix). Both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`), validated via `parseWorkers`. Explicit `--workers` bypasses the auto-path file-count floor; auto path defers to `autoConcurrency()`. Banner moved to stderr. - `src/core/cycle.ts` — v0.17 brain maintenance cycle primitive (extended to **9 phases in v0.29**). `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise` composes phases in semantically-driven order: **lint → backlinks → sync → synthesize → extract → patterns → recompute_emotional_weight → embed → orphans**. v0.29 adds the `recompute_emotional_weight` phase between patterns and embed; it sees the union of `syncPagesAffected` + `synthesizeWrittenSlugs` for incremental mode, or all pages when neither anchor is set (full backfill via `gbrain dream --phase recompute_emotional_weight`). v0.29 also extends `CycleReport.totals` with `pages_emotional_weight_recomputed` (additive, schema_version stays "1"). v0.23's `synthesize` phase runs after sync (cross-references see fresh brain) and before extract (auto-link materializes its writes); `patterns` runs after extract so it reads a fresh graph (codex finding #7 — subagent put_page sets `ctx.remote=true` and skips auto-link/timeline by default; extract is the canonical materialization). Three callers: `gbrain dream` CLI, `gbrain autopilot` daemon's inline path, and the Minions `autopilot-cycle` handler. Coordination via `gbrain_cycle_locks` DB table + `~/.gbrain/cycle.lock` file lock with PID-liveness for PGLite. `CycleReport.schema_version: "1"` is stable; totals additively grew in v0.23 (`transcripts_processed`, `synth_pages_written`, `patterns_written`). `yieldBetweenPhases` runs between phases. **v0.23 added `yieldDuringPhase`** for in-phase keepalive — synthesize/patterns call it during long waits to renew the cycle-lock TTL. Engine nullable; lock-skip on read-only phase selections. v0.22.1 (#403): `CycleOpts.signal?: AbortSignal` propagates the worker's abort signal; `checkAborted()` fires between every phase. v0.22.1 (#417): `runPhaseSync` returns `pagesAffected` via `SyncPhaseResult`; `runCycle` captures it and threads to `runPhaseExtract` as the 4th arg. v0.22.1 (Codex F2): `runPhaseSync` takes `willRunExtractPhase: boolean` and sets `noExtract: phases.includes('extract')` so `gbrain dream --phase sync` doesn't silently lose extraction. v0.22.5 (#475): `resolveSourceForDir(engine, brainDir)` threads `sourceId` to `performSync()` so sync reads the per-source `sources.last_commit` anchor instead of the drift-prone global `config.sync.last_commit` key. -- `src/core/cycle/synthesize.ts` (v0.23) — Synthesize phase: conversation-transcript-to-brain pipeline. Reads from `dream.synthesize.session_corpus_dir`, runs cheap Haiku verdict (cached in `dream_verdicts`), then fans out one Sonnet subagent per worth-processing transcript with `allowed_slug_prefixes` (sourced from `skills/_brain-filing-rules.json` `dream_synthesize_paths.globs`). Orchestrator collects slugs from `subagent_tool_executions` (NOT `pages.updated_at` — codex finding #2) and reverse-renders DB → markdown via `serializeMarkdown`. Cooldown via `dream.synthesize.last_completion_ts`, written ONLY on success. Idempotency key `dream:synth::`. Auto-commit deferred to v1.1 (codex #5). `--dry-run` runs Haiku, skips Sonnet (codex #8). Subagent never gets fs-write access. **v0.23.2:** `renderPageToMarkdown` (now exported) stamps `dream_generated: true` and `dream_cycle_date` into every reverse-write's frontmatter; `writeSummaryPage` does the same on the dream-cycle summary index. The marker is the explicit identity surface checked by `isDreamOutput` in `transcript-discovery.ts` — replaces the v0.23.1 content-prefix heuristic that could miss real output (`serializeMarkdown` doesn't embed slugs in body) and false-positive on user transcripts citing brain pages. `judgeSignificance` and `JudgeClient` are exported; `judgeSignificance` accepts a `verdictModel` parameter (default `claude-haiku-4-5-20251001`) loaded from `dream.synthesize.verdict_model` via `loadSynthConfig`. +- `src/core/cycle/synthesize.ts` (v0.23) — Synthesize phase: conversation-transcript-to-brain pipeline. Reads from `dream.synthesize.session_corpus_dir`, runs cheap Haiku verdict (cached in `dream_verdicts`), then fans out one Sonnet subagent per worth-processing transcript with `allowed_slug_prefixes` (sourced from `skills/_brain-filing-rules.json` `dream_synthesize_paths.globs`). Orchestrator collects slugs from `subagent_tool_executions` (NOT `pages.updated_at` — codex finding #2) and reverse-renders DB → markdown via `serializeMarkdown`. Cooldown via `dream.synthesize.last_completion_ts`, written ONLY on success. Idempotency key `dream:synth::`. Auto-commit deferred to v1.1 (codex #5). `--dry-run` runs Haiku, skips Sonnet (codex #8). Subagent never gets fs-write access. **v0.23.2:** `renderPageToMarkdown` (now exported) stamps `dream_generated: true` and `dream_cycle_date` into every reverse-write's frontmatter; `writeSummaryPage` does the same on the dream-cycle summary index. The marker is the explicit identity surface checked by `isDreamOutput` in `transcript-discovery.ts` — replaces the v0.23.1 content-prefix heuristic that could miss real output (`serializeMarkdown` doesn't embed slugs in body) and false-positive on user transcripts citing brain pages. `judgeSignificance` and `JudgeClient` are exported; `judgeSignificance` accepts a `verdictModel` parameter (default `claude-haiku-4-5-20251001`) loaded from `dream.synthesize.verdict_model` via `loadSynthConfig`. **v0.30.2:** model-aware chunker `splitTranscriptByBudget(content, contentHash, maxChars)` splits oversized transcripts at paragraph boundaries (`## Topic:` → `---` → `\n` ladder) using a deterministic offset seeded from the first 32 bits of `contentHash` so retries chunk identically. Per-chunk char budget computed from `MODEL_CONTEXT_TOKENS[resolvedModel] × 0.9 × 3.5 chars/token`; non-Anthropic ids fall back to a 180K-token safe default with a once-per-process stderr warning. Operator overrides: `dream.synthesize.max_prompt_tokens` (floor 100K, wins when set) and `dream.synthesize.max_chunks_per_transcript` (default 24). Per-chunk idempotency keys `dream:synth:::cof`; single-chunk transcripts preserve the legacy `dream:synth::` key byte-for-byte (D8 lookup), so existing brains skip with `already_synthesized_legacy_single_chunk` instead of re-spending Sonnet on upgrade. `collectChildPutPageSlugs` raw-fetches every (job_id, slug) pair (not `SELECT DISTINCT`) and rewrites bare-hash6 slugs to `-c` for chunked children (D6 — orchestrator-side, zero Sonnet trust). Cap-hit skips don't write to `dream_verdicts`, so raising the cap on next run re-attempts cleanly. D7 scope: bounds INITIAL prompt size only; tool-loop turn-N accumulation is caught by the v0.30.2 terminal-error classification in `subagent.ts`, not bounded ahead of time. - `src/core/cycle/patterns.ts` (v0.23) — Patterns phase: cross-session theme detection over reflections within `dream.patterns.lookback_days` (default 30). Names a pattern only when ≥`dream.patterns.min_evidence` (default 3) reflections support it. Single Sonnet subagent; same allow-list path as synthesize. Runs AFTER `extract` so the graph is fresh. - `src/core/cycle/emotional-weight.ts` (v0.29) — Pure function `computeEmotionalWeight({tags, takes}, {highEmotionTags?, userHolder?})`. Deterministic 0..1 score: tag-emotion boost (max 0.5, case-insensitive match against `HIGH_EMOTION_TAGS` seed list), take density (0.1/take, capped at 0.3), take avg weight (0..0.1), user-holder ratio (0..0.1 over active takes; default holder = 'garry'). Total clamped to [0..1]. Anglocentric / personal-life-biased seed list intentional; users override via config key `emotional_weight.high_tags` (JSON array). `userHolder` overridable via `emotional_weight.user_holder`. - `src/core/cycle/anomaly.ts` (v0.29) — Pure stats helpers for `find_anomalies`. `meanStddev` returns sample stddev (n-1 denominator) and (0,0) for empty input. `computeAnomaliesFromBuckets(baseline, today, sigma, limit)` takes densified daily-count buckets + today's counts per cohort, returns `AnomalyResult[]`. Zero-stddev fallback: cohort fires when `count > mean + 1`, with `sigma_observed = count - mean` as a finite sort proxy (no NaN). Brand-new cohorts (no baseline) have `mean=0, stddev=0` so the fallback fires at count >= 2. Sorted by `sigma_observed` desc, top `limit` (default 20). `page_slugs` capped at 50 per cohort. diff --git a/README.md b/README.md index f104786ad..4fb717608 100644 --- a/README.md +++ b/README.md @@ -767,9 +767,11 @@ ADMIN $GBRAIN_HOME/clones// and re-cloned on sync if it goes missing. Also exposed via MCP for remote agent setup (whoami + sources_{add,list,remove,status}). - gbrain dream [--dry-run] [--phase N] 8-phase maintenance cycle (lint→backlinks→sync→synthesize - →extract→patterns→embed→orphans). v0.23 added synthesize + - patterns: transcripts → reflections + cross-session themes. + gbrain dream [--dry-run] [--phase N] 9-phase maintenance cycle (lint→backlinks→sync→synthesize + →extract→patterns→recompute_emotional_weight→embed→orphans). + v0.23 added synthesize + patterns. v0.29 added emotional-weight + recompute. v0.30.2: synthesize now chunks fat transcripts + (config: dream.synthesize.max_prompt_tokens, max_chunks_per_transcript). gbrain dream --input Ad-hoc transcript synthesis (implies --phase synthesize) gbrain dream --date YYYY-MM-DD Synthesize a single day; --from/--to for backfill ranges gbrain check-backlinks check|fix Back-link enforcement diff --git a/VERSION b/VERSION index 1a44cad74..0f7217737 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.30.1 +0.30.2 diff --git a/llms-full.txt b/llms-full.txt index df46fb6c4..2b6f86455 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -222,7 +222,7 @@ strict behavior when unset. - `src/core/minions/handlers/shell.ts` — `shell` job handler. Spawns `/bin/sh -c cmd` (absolute path, PATH-override-safe) or `argv[0] argv[1..]` (no shell). Env allowlist: `PATH, HOME, USER, LANG, TZ, NODE_ENV` + caller `env:` overrides. UTF-8-safe stdout/stderr tail via `string_decoder.StringDecoder`. Abort (either `ctx.signal` or `ctx.shutdownSignal`) fires SIGTERM → 5s grace → SIGKILL on child. Requires `GBRAIN_ALLOW_SHELL_JOBS=1` on worker (gated by `registerBuiltinHandlers`). - `src/core/minions/handlers/shell-audit.ts` — per-submission JSONL audit trail at `~/.gbrain/audit/shell-jobs-YYYY-Www.jsonl` (ISO-week rotation; override via `GBRAIN_AUDIT_DIR`). Best-effort: `mkdirSync(recursive)` + `appendFileSync`; failures logged to stderr, submission not blocked. Logs cmd (first 80 chars) or argv (JSON array). Never logs env values. - `src/core/minions/backpressure-audit.ts` (v0.19.1) — sibling of shell-audit.ts for `maxWaiting` coalesce events. JSONL at `~/.gbrain/audit/backpressure-YYYY-Www.jsonl`. Fires one line per coalesce with `(queue, name, waiting_count, max_waiting, returned_job_id, ts)`. Closes the silent-drop vector the v0.19.0 maxWaiting guard introduced. -- `src/core/minions/handlers/subagent.ts` (v0.15) — LLM-loop handler. Two-phase tool persistence (pending → complete/failed), replay reconciliation for mid-dispatch crashes, dual-signal abort (`ctx.signal` + `ctx.shutdownSignal`), Anthropic prompt caching on system + tool defs. `makeSubagentHandler({engine, client?, ...})` factory; `MessagesClient` is an injectable interface the real SDK implements structurally. Throws `RateLeaseUnavailableError` (renewable) when rate-lease capacity is full. +- `src/core/minions/handlers/subagent.ts` (v0.15) — LLM-loop handler. Two-phase tool persistence (pending → complete/failed), replay reconciliation for mid-dispatch crashes, dual-signal abort (`ctx.signal` + `ctx.shutdownSignal`), Anthropic prompt caching on system + tool defs. `makeSubagentHandler({engine, client?, ...})` factory; `MessagesClient` is an injectable interface the real SDK implements structurally. Throws `RateLeaseUnavailableError` (renewable) when rate-lease capacity is full. **v0.30.2:** Anthropic 400 `prompt is too long` responses (status 400 + body matches `/prompt is too long|prompt_too_long|context.*length/i`) classify as `UnrecoverableError` so the job goes straight to `dead` on first attempt instead of stalling three times before dead-lettering. Catches both initial-prompt overflow and turn-N tool-loop accumulation that the chunker in `synthesize.ts` can't bound ahead of time. - `src/core/minions/handlers/subagent-aggregator.ts` (v0.15) — `subagent_aggregator` handler. Claims AFTER all children resolve (queue changes guarantee every terminal child posts a `child_done` inbox message with outcome). Reads inbox via `ctx.readInbox()`, builds deterministic mixed-outcome markdown summary. No LLM call in v0.15. - `src/core/minions/handlers/subagent-audit.ts` (v0.15) — JSONL audit + heartbeat writer at `~/.gbrain/audit/subagent-jobs-YYYY-Www.jsonl`. Events: `submission` (one line per submit) + `heartbeat` (per turn boundary: `llm_call_started | llm_call_completed | tool_called | tool_result | tool_failed`). Never logs prompts or tool inputs. `readSubagentAuditForJob(jobId, {sinceIso})` is the readback path for `gbrain agent logs`. - `src/core/minions/rate-leases.ts` (v0.15) — lease-based concurrency cap for outbound providers (default key `anthropic:messages`, max via `GBRAIN_ANTHROPIC_MAX_INFLIGHT`). Owner-tagged rows with `expires_at` auto-prune on acquire; `pg_advisory_xact_lock` guards check-then-insert; CASCADE on owning job deletion. `renewLeaseWithBackoff` retries 3x (250/500/1000ms). @@ -252,7 +252,7 @@ strict behavior when unset. - `src/commands/anomalies.ts` (v0.29) — `gbrain anomalies [--since YYYY-MM-DD] [--lookback-days N] [--sigma N] [--json]`: cohort-level activity outliers. Calls `engine.findAnomalies(opts)`. Two cohort kinds in v1: tag, type. Year cohort deferred to v0.30. - `src/commands/transcripts.ts` (v0.29) — `gbrain transcripts recent [--days N] [--full] [--json]`: recent raw `.txt` transcripts from the dream-cycle corpus dirs. Imports `listRecentTranscripts` from `src/core/transcripts.ts` (the same library the gated `get_recent_transcripts` MCP op uses). Local-only by construction — the CLI always runs with `ctx.remote=false`. - `src/commands/integrity.ts` — `gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). v0.22.8: batch-load fast path on Postgres uses `SELECT DISTINCT ON (slug)` in a single SQL query to fix the PgBouncer round-trip timeout (60s → ~6s) while preserving `engine.getAllSlugs()`'s `Set` semantics on multi-source brains. Gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1` so real Postgres errors are diagnosable. -- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel `. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`. +- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel `. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`. **v0.30.2:** `queue_health` gains a fourth subcheck — surfaces dead-lettered subagent jobs with `last_error` matching the `prompt_too_long` classifier within the last 24h. Fix hint points at `gbrain dream --phase synthesize --dry-run --json` to identify the offending transcript and `gbrain jobs prune --status dead --queue default` to clean up. Postgres-only. - `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). **v40 (v0.29):** `pages_emotional_weight` adds `pages.emotional_weight REAL NOT NULL DEFAULT 0.0`. Column-only (no index). On Postgres 11+ and PGLite, `ADD COLUMN` with a constant DEFAULT is metadata-only — instant on tables of any size. v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). **v30 (v0.23):** creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))`. RLS-enabled when running as a BYPASSRLS role. The synthesize phase reads/writes this table to avoid re-judging on backfill re-runs. **v35 (v0.26.7):** auto-RLS event trigger + one-time backfill. `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` and runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on every new `public.*` table — no FORCE (matches v24/v29/schema.sql posture so non-BYPASSRLS apps can still read their own tables). The same migration backfills RLS on every existing `public.*` base table whose comment doesn't match the doctor regex (`^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}`). Per-table failure aborts the offending CREATE TABLE (event triggers fire inside the DDL transaction); no EXCEPTION wrap — that would convert loud rollback into silent permissive default. PGLite no-op via `sqlFor.pglite: ''`. Breaking change: operators with intentionally-RLS-off public tables must add the GBRAIN:RLS_EXEMPT comment BEFORE upgrade or the backfill will flip them on. - `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2. - `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers. @@ -260,7 +260,7 @@ strict behavior when unset. - `src/core/sync-concurrency.ts` (v0.22.13) — single source of truth for the parallel-sync policy. Exports `autoConcurrency(engine, fileCount, override?)` (PGLite always serial; explicit override clamped to >=1; auto path returns `DEFAULT_PARALLEL_WORKERS=4` when `fileCount > AUTO_CONCURRENCY_FILE_THRESHOLD=100`), `shouldRunParallel(workers, fileCount, explicit)` (Q1: explicit `--workers` bypasses the >50-file floor), and `parseWorkers(s)` (rejects `'0'`, `'-3'`, `'foo'`, `'1.5'`, trailing chars — replaces the prior parseInt-with-no-validation in both `sync.ts` and `import.ts`). Used by `performSync`, `performFullSync`, `runImport`, and the Minion `sync` handler so the three sites can no longer drift. - `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). v0.22.13 (PR #490): `performSync` wraps its body in a `gbrain-sync` writer lock so two concurrent syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot both write `last_commit` and let the last writer win. Head-drift gate after the import phase re-checks `git rev-parse HEAD`; if HEAD moved (someone ran `git checkout` / `git pull` mid-sync), the bookmark refuses to advance. Vanished files now record a failedFiles entry instead of silent-skip — the silent-skip-then-advance pathology that survived prior hardening passes is dead. Worker engines wrap in try/finally so disconnect always fires (panic-path leak fix). Both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`), validated via `parseWorkers`. Explicit `--workers` bypasses the auto-path file-count floor; auto path defers to `autoConcurrency()`. Banner moved to stderr. - `src/core/cycle.ts` — v0.17 brain maintenance cycle primitive (extended to **9 phases in v0.29**). `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise` composes phases in semantically-driven order: **lint → backlinks → sync → synthesize → extract → patterns → recompute_emotional_weight → embed → orphans**. v0.29 adds the `recompute_emotional_weight` phase between patterns and embed; it sees the union of `syncPagesAffected` + `synthesizeWrittenSlugs` for incremental mode, or all pages when neither anchor is set (full backfill via `gbrain dream --phase recompute_emotional_weight`). v0.29 also extends `CycleReport.totals` with `pages_emotional_weight_recomputed` (additive, schema_version stays "1"). v0.23's `synthesize` phase runs after sync (cross-references see fresh brain) and before extract (auto-link materializes its writes); `patterns` runs after extract so it reads a fresh graph (codex finding #7 — subagent put_page sets `ctx.remote=true` and skips auto-link/timeline by default; extract is the canonical materialization). Three callers: `gbrain dream` CLI, `gbrain autopilot` daemon's inline path, and the Minions `autopilot-cycle` handler. Coordination via `gbrain_cycle_locks` DB table + `~/.gbrain/cycle.lock` file lock with PID-liveness for PGLite. `CycleReport.schema_version: "1"` is stable; totals additively grew in v0.23 (`transcripts_processed`, `synth_pages_written`, `patterns_written`). `yieldBetweenPhases` runs between phases. **v0.23 added `yieldDuringPhase`** for in-phase keepalive — synthesize/patterns call it during long waits to renew the cycle-lock TTL. Engine nullable; lock-skip on read-only phase selections. v0.22.1 (#403): `CycleOpts.signal?: AbortSignal` propagates the worker's abort signal; `checkAborted()` fires between every phase. v0.22.1 (#417): `runPhaseSync` returns `pagesAffected` via `SyncPhaseResult`; `runCycle` captures it and threads to `runPhaseExtract` as the 4th arg. v0.22.1 (Codex F2): `runPhaseSync` takes `willRunExtractPhase: boolean` and sets `noExtract: phases.includes('extract')` so `gbrain dream --phase sync` doesn't silently lose extraction. v0.22.5 (#475): `resolveSourceForDir(engine, brainDir)` threads `sourceId` to `performSync()` so sync reads the per-source `sources.last_commit` anchor instead of the drift-prone global `config.sync.last_commit` key. -- `src/core/cycle/synthesize.ts` (v0.23) — Synthesize phase: conversation-transcript-to-brain pipeline. Reads from `dream.synthesize.session_corpus_dir`, runs cheap Haiku verdict (cached in `dream_verdicts`), then fans out one Sonnet subagent per worth-processing transcript with `allowed_slug_prefixes` (sourced from `skills/_brain-filing-rules.json` `dream_synthesize_paths.globs`). Orchestrator collects slugs from `subagent_tool_executions` (NOT `pages.updated_at` — codex finding #2) and reverse-renders DB → markdown via `serializeMarkdown`. Cooldown via `dream.synthesize.last_completion_ts`, written ONLY on success. Idempotency key `dream:synth::`. Auto-commit deferred to v1.1 (codex #5). `--dry-run` runs Haiku, skips Sonnet (codex #8). Subagent never gets fs-write access. **v0.23.2:** `renderPageToMarkdown` (now exported) stamps `dream_generated: true` and `dream_cycle_date` into every reverse-write's frontmatter; `writeSummaryPage` does the same on the dream-cycle summary index. The marker is the explicit identity surface checked by `isDreamOutput` in `transcript-discovery.ts` — replaces the v0.23.1 content-prefix heuristic that could miss real output (`serializeMarkdown` doesn't embed slugs in body) and false-positive on user transcripts citing brain pages. `judgeSignificance` and `JudgeClient` are exported; `judgeSignificance` accepts a `verdictModel` parameter (default `claude-haiku-4-5-20251001`) loaded from `dream.synthesize.verdict_model` via `loadSynthConfig`. +- `src/core/cycle/synthesize.ts` (v0.23) — Synthesize phase: conversation-transcript-to-brain pipeline. Reads from `dream.synthesize.session_corpus_dir`, runs cheap Haiku verdict (cached in `dream_verdicts`), then fans out one Sonnet subagent per worth-processing transcript with `allowed_slug_prefixes` (sourced from `skills/_brain-filing-rules.json` `dream_synthesize_paths.globs`). Orchestrator collects slugs from `subagent_tool_executions` (NOT `pages.updated_at` — codex finding #2) and reverse-renders DB → markdown via `serializeMarkdown`. Cooldown via `dream.synthesize.last_completion_ts`, written ONLY on success. Idempotency key `dream:synth::`. Auto-commit deferred to v1.1 (codex #5). `--dry-run` runs Haiku, skips Sonnet (codex #8). Subagent never gets fs-write access. **v0.23.2:** `renderPageToMarkdown` (now exported) stamps `dream_generated: true` and `dream_cycle_date` into every reverse-write's frontmatter; `writeSummaryPage` does the same on the dream-cycle summary index. The marker is the explicit identity surface checked by `isDreamOutput` in `transcript-discovery.ts` — replaces the v0.23.1 content-prefix heuristic that could miss real output (`serializeMarkdown` doesn't embed slugs in body) and false-positive on user transcripts citing brain pages. `judgeSignificance` and `JudgeClient` are exported; `judgeSignificance` accepts a `verdictModel` parameter (default `claude-haiku-4-5-20251001`) loaded from `dream.synthesize.verdict_model` via `loadSynthConfig`. **v0.30.2:** model-aware chunker `splitTranscriptByBudget(content, contentHash, maxChars)` splits oversized transcripts at paragraph boundaries (`## Topic:` → `---` → `\n` ladder) using a deterministic offset seeded from the first 32 bits of `contentHash` so retries chunk identically. Per-chunk char budget computed from `MODEL_CONTEXT_TOKENS[resolvedModel] × 0.9 × 3.5 chars/token`; non-Anthropic ids fall back to a 180K-token safe default with a once-per-process stderr warning. Operator overrides: `dream.synthesize.max_prompt_tokens` (floor 100K, wins when set) and `dream.synthesize.max_chunks_per_transcript` (default 24). Per-chunk idempotency keys `dream:synth:::cof`; single-chunk transcripts preserve the legacy `dream:synth::` key byte-for-byte (D8 lookup), so existing brains skip with `already_synthesized_legacy_single_chunk` instead of re-spending Sonnet on upgrade. `collectChildPutPageSlugs` raw-fetches every (job_id, slug) pair (not `SELECT DISTINCT`) and rewrites bare-hash6 slugs to `-c` for chunked children (D6 — orchestrator-side, zero Sonnet trust). Cap-hit skips don't write to `dream_verdicts`, so raising the cap on next run re-attempts cleanly. D7 scope: bounds INITIAL prompt size only; tool-loop turn-N accumulation is caught by the v0.30.2 terminal-error classification in `subagent.ts`, not bounded ahead of time. - `src/core/cycle/patterns.ts` (v0.23) — Patterns phase: cross-session theme detection over reflections within `dream.patterns.lookback_days` (default 30). Names a pattern only when ≥`dream.patterns.min_evidence` (default 3) reflections support it. Single Sonnet subagent; same allow-list path as synthesize. Runs AFTER `extract` so the graph is fresh. - `src/core/cycle/emotional-weight.ts` (v0.29) — Pure function `computeEmotionalWeight({tags, takes}, {highEmotionTags?, userHolder?})`. Deterministic 0..1 score: tag-emotion boost (max 0.5, case-insensitive match against `HIGH_EMOTION_TAGS` seed list), take density (0.1/take, capped at 0.3), take avg weight (0..0.1), user-holder ratio (0..0.1 over active takes; default holder = 'garry'). Total clamped to [0..1]. Anglocentric / personal-life-biased seed list intentional; users override via config key `emotional_weight.high_tags` (JSON array). `userHolder` overridable via `emotional_weight.user_holder`. - `src/core/cycle/anomaly.ts` (v0.29) — Pure stats helpers for `find_anomalies`. `meanStddev` returns sample stddev (n-1 denominator) and (0,0) for empty input. `computeAnomaliesFromBuckets(baseline, today, sigma, limit)` takes densified daily-count buckets + today's counts per cohort, returns `AnomalyResult[]`. Zero-stddev fallback: cohort fires when `count > mean + 1`, with `sigma_observed = count - mean` as a finite sort proxy (no NaN). Brand-new cohorts (no baseline) have `mean=0, stddev=0` so the fallback fires at count >= 2. Sorted by `sigma_observed` desc, top `limit` (default 20). `page_slugs` capped at 50 per cohort. @@ -2419,9 +2419,11 @@ ADMIN $GBRAIN_HOME/clones// and re-cloned on sync if it goes missing. Also exposed via MCP for remote agent setup (whoami + sources_{add,list,remove,status}). - gbrain dream [--dry-run] [--phase N] 8-phase maintenance cycle (lint→backlinks→sync→synthesize - →extract→patterns→embed→orphans). v0.23 added synthesize + - patterns: transcripts → reflections + cross-session themes. + gbrain dream [--dry-run] [--phase N] 9-phase maintenance cycle (lint→backlinks→sync→synthesize + →extract→patterns→recompute_emotional_weight→embed→orphans). + v0.23 added synthesize + patterns. v0.29 added emotional-weight + recompute. v0.30.2: synthesize now chunks fat transcripts + (config: dream.synthesize.max_prompt_tokens, max_chunks_per_transcript). gbrain dream --input Ad-hoc transcript synthesis (implies --phase synthesize) gbrain dream --date YYYY-MM-DD Synthesize a single day; --from/--to for backfill ranges gbrain check-backlinks check|fix Back-link enforcement diff --git a/package.json b/package.json index 6470606c5..e6b27e370 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "gbrain", - "version": "0.30.1", + "version": "0.30.2", "description": "Postgres-native personal knowledge brain with hybrid RAG search", "type": "module", "main": "src/core/index.ts", diff --git a/src/commands/doctor.ts b/src/commands/doctor.ts index 2b5e1cd50..6d2b654be 100644 --- a/src/commands/doctor.ts +++ b/src/commands/doctor.ts @@ -1306,6 +1306,21 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo `; const rssKillCount = rssKillRows[0]?.cnt ?? 0; + // Subcheck 4 (v0.30.2): prompt_too_long terminal failures on subagent + // jobs in the last 24h. The dream/synthesize phase classifies Anthropic + // 400 "prompt is too long" responses as UnrecoverableError so they + // dead-letter on first attempt instead of clogging the queue with + // max_stalled retries. Surface count + fix hint when present. + const promptTooLongRows: Array<{ cnt: number }> = await sql` + SELECT count(*)::int AS cnt + FROM minion_jobs + WHERE name = 'subagent' + AND status = 'dead' + AND finished_at > now() - interval '24 hours' + AND error_text LIKE 'prompt_too_long:%' + `; + const promptTooLongCount = promptTooLongRows[0]?.cnt ?? 0; + const problems: string[] = []; if (stalledRows.length > 0) { const sample = stalledRows @@ -1333,6 +1348,15 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo `See skills/migrations/v0.22.14.md.` ); } + if (promptTooLongCount > 0) { + problems.push( + `${promptTooLongCount} subagent job(s) dead-lettered with prompt_too_long in last 24h. ` + + `Dream/synthesize transcripts exceeded the model's input context. ` + + `Fix: \`gbrain dream --phase synthesize --dry-run --json\` to identify fat transcripts; ` + + `set \`dream.synthesize.max_prompt_tokens\` to bound the per-chunk budget, or use a ` + + `larger-context model (Opus 4.7 = 1M tokens vs Sonnet 4.6 = 200K).` + ); + } if (problems.length === 0) { checks.push({ diff --git a/src/core/cycle/synthesize.ts b/src/core/cycle/synthesize.ts index d3a2025c4..76198439f 100644 --- a/src/core/cycle/synthesize.ts +++ b/src/core/cycle/synthesize.ts @@ -42,6 +42,173 @@ import type { Page, PageType } from '../types.ts'; // Used for the orchestrator-written summary index slug. const SUMMARY_SLUG_RE = /^[a-z0-9][a-z0-9\-]*(\/[a-z0-9][a-z0-9\-]*)*$/; +// ── Model context budget (D1, D5, D7, D9) ───────────────────────────── + +/** + * Anthropic model id → input context window (tokens). + * Unknown id (non-Anthropic alias, custom string) → safe 200K-token fallback + * via `computeChunkCharBudget`. Codex finding #4: `resolveModel()` does not + * canonicalize to Anthropic-only; this map keys on the exact strings the + * resolver returns for known Anthropic aliases. + */ +const MODEL_CONTEXT_TOKENS: Record = { + 'claude-opus-4-7': 1_000_000, + 'claude-opus-4-6': 1_000_000, + 'claude-sonnet-4-6': 200_000, + 'claude-sonnet-4-5': 200_000, + 'claude-haiku-4-5-20251001': 200_000, +}; + +/** Token-to-char ratio. 3.5 matches PR #748; conservative for English text. */ +const CHARS_PER_TOKEN = 3.5; +/** Reserve 10% of context window for system prompt + tool defs + output. */ +const HEADROOM_RATIO = 0.9; +/** Floor on user-overridable max_prompt_tokens (matches PR #748 minimum). */ +const MIN_PROMPT_TOKENS = 100_000; +/** Default chunk-count cap; operator-configurable via dream.synthesize.max_chunks_per_transcript. */ +const DEFAULT_MAX_CHUNKS = 24; +/** Conservative default budget when model is unknown (200K × HEADROOM_RATIO). */ +const UNKNOWN_MODEL_BUDGET_TOKENS = 180_000; + +/** + * Compute per-chunk character budget for the resolved model + config override. + * + * Resolution: + * - configMaxPromptTokens (already floored at MIN_PROMPT_TOKENS) wins when set. + * - Else the model's MODEL_CONTEXT_TOKENS entry × HEADROOM_RATIO. + * - Else (non-Anthropic alias / custom id) UNKNOWN_MODEL_BUDGET_TOKENS, with + * a once-per-process stderr warning. + * + * D7 scope: this bounds the INITIAL prompt size only. Tool-loop turn-N + * accumulation is out of scope for v0.30.2 (terminal-error classification + * catches turn-N blowups; per-turn budget guard is a v0.31+ follow-up). + */ +function computeChunkCharBudget( + model: string, + configMaxPromptTokens: number | null, +): number { + if (configMaxPromptTokens !== null) { + return Math.floor(configMaxPromptTokens * CHARS_PER_TOKEN); + } + const ctx = MODEL_CONTEXT_TOKENS[model]; + if (ctx === undefined) { + warnUnknownModelOnce(model); + return Math.floor(UNKNOWN_MODEL_BUDGET_TOKENS * CHARS_PER_TOKEN); + } + return Math.floor(ctx * HEADROOM_RATIO * CHARS_PER_TOKEN); +} + +const _unknownModelWarned = new Set(); +function warnUnknownModelOnce(model: string): void { + if (_unknownModelWarned.has(model)) return; + _unknownModelWarned.add(model); + process.stderr.write( + `[dream] model "${model}" is not in MODEL_CONTEXT_TOKENS; ` + + `using ${UNKNOWN_MODEL_BUDGET_TOKENS}-token fallback budget. ` + + `Set dream.synthesize.max_prompt_tokens to override.\n`, + ); +} + +// ── Hash-deterministic transcript chunker (D9) ──────────────────────── + +/** + * Split content into chunks at most maxChars long, picking boundaries via a + * 3-tier ladder lifted from PR #748: + * 1. `## Topic:` separators (matches the daily-aggregated transcript shape) + * 2. `---` markdown HR markers + * 3. nearest `\n` newline + * + * D9 stable chunk identity: the back-half-of-budget search window is seeded + * with a deterministic offset derived from contentHash so the same + * (content, contentHash, maxChars) triple always produces identical chunks. + * Closes the partial-progress ambiguity: chunk 2 of a transcript that + * previously failed terminally produces byte-identical content on retry, + * so the per-chunk idempotency key is durable across runs. + * + * The hash-derived offset jitters the search start within + * [0.5×budget, 0.6×budget] so the back-half rule still holds. + * + * If no boundary fits, hard-split at maxChars (also deterministic in the + * inputs). + * + * Pure function. Tested by `test/cycle/synthesize-chunker.test.ts`. + */ +export function splitTranscriptByBudget( + content: string, + contentHash: string, + maxChars: number, +): string[] { + if (maxChars <= 0) { + throw new Error(`splitTranscriptByBudget: maxChars must be > 0, got ${maxChars}`); + } + if (content.length <= maxChars) return [content]; + + const hashInt = parseHashOffset(contentHash); + // Jitter window is the next 10% of budget after the 50% midpoint. + const jitterRange = Math.max(1, Math.floor(maxChars * 0.1)); + const searchStart = Math.floor(maxChars * 0.5) + (hashInt % jitterRange); + + const out: string[] = []; + let remaining = content; + while (remaining.length > maxChars) { + const split = findBoundary(remaining, maxChars, searchStart); + out.push(remaining.slice(0, split)); + remaining = remaining.slice(split); + } + if (remaining.length > 0) out.push(remaining); + return out; +} + +function parseHashOffset(contentHash: string): number { + // First 8 hex chars = 32 bits; plenty of entropy for the offset jitter. + const hex = contentHash.slice(0, 8); + const n = parseInt(hex, 16); + return Number.isFinite(n) && n >= 0 ? n : 0; +} + +function findBoundary(text: string, maxChars: number, searchStart: number): number { + const window = text.slice(searchStart, maxChars); + // Tier 1: "\n## Topic:" — last occurrence inside the search window. + const topicIdx = window.lastIndexOf('\n## Topic:'); + if (topicIdx >= 0) return searchStart + topicIdx; + // Tier 2: "\n---\n" markdown HR. + const hrIdx = window.lastIndexOf('\n---\n'); + if (hrIdx >= 0) return searchStart + hrIdx; + // Tier 3: any newline. + const nlIdx = window.lastIndexOf('\n'); + if (nlIdx >= 0) return searchStart + nlIdx; + // No boundary fits; hard-split at maxChars (deterministic). + return maxChars; +} + +/** + * D6: orchestrator-side deterministic slug rewrite. Zero Sonnet trust. + * + * Expected shape from `buildSynthesisPrompt` for a chunked child is already + * `--c`, but if Sonnet drops the chunk suffix this rewrite + * enforces uniqueness post-hoc. Same hash AND same chunk idx → idempotent. + * + * Pure function. Cases: + * - already correctly suffixed (`...--c`) → return unchanged. + * - bare hash suffix (`...-`) → append `-c`. + * - some other shape → pass through (orchestrator can't safely guess + * where to inject the chunk index; e2e test pins this). + */ +export function rewriteChunkedSlug(slug: string, hash6: string, idx: number): string { + if (!slug) return slug; + const expected = `${hash6}-c${idx}`; + // Already correctly chunk-suffixed. + if (slug === expected) return slug; + if (slug.endsWith(`-${expected}`) || slug.endsWith(`/${expected}`)) return slug; + // Bare hash6 at end of last path segment: rewrite. + // Match either at start-of-slug, after a "/" path separator, or after a "-". + const re = new RegExp(`(^|[/-])${hash6}$`); + if (re.test(slug)) return `${slug}-c${idx}`; + // Unknown shape — pass through; collision risk is now bounded by Sonnet's + // per-chunk-prompt guidance and the existing slug-prefix allow-list. + return slug; +} + // ── Public entry ────────────────────────────────────────────────────── export interface SynthesizePhaseOpts { @@ -166,7 +333,8 @@ export async function runPhaseSynthesize( }); } - // Fan-out: submit one subagent per worth-processing transcript. + // Fan-out: submit one subagent per worth-processing transcript (or one + // per chunk for transcripts that exceed the model's per-prompt budget). const allowedSlugPrefixes = await loadAllowedSlugPrefixes(); if (allowedSlugPrefixes.length === 0) { return failed(makeError('InternalError', 'NO_ALLOWLIST', @@ -175,26 +343,82 @@ export async function runPhaseSynthesize( const queue = new MinionQueue(engine); const childIds: number[] = []; + /** Map child job_id → chunk metadata for D6 orchestrator-side slug rewrite. */ + const chunkInfo = new Map(); + /** Skip reasons for the cycle report (D5 cap hits, D8 legacy-key skips). */ + const skipReports: Array<{ filePath: string; reason: string }> = []; + + const maxCharsPerChunk = computeChunkCharBudget(config.model, config.maxPromptTokens); + for (const t of worthProcessing) { - const childData: SubagentHandlerData = { - prompt: buildSynthesisPrompt(t), - model: config.model, - max_turns: 30, - allowed_slug_prefixes: allowedSlugPrefixes, - }; - const submitOpts: Partial = { - max_stalled: 3, - on_child_fail: 'continue', - idempotency_key: `dream:synth:${t.filePath}:${t.contentHash.slice(0, 16)}`, - timeout_ms: 30 * 60 * 1000, // 30 min per transcript - }; - const child = await queue.add( - 'subagent', - childData as unknown as Record, - submitOpts, - { allowProtectedSubmit: true }, - ); - childIds.push(child.id); + const hash16 = t.contentHash.slice(0, 16); + const hash6 = t.contentHash.slice(0, 6); + + // D8: single→multi-chunk migration safety. If a completed legacy + // single-chunk job exists for this content_hash, treat as already- + // synthesized and skip. Prevents duplicate writes when a transcript + // that was previously single-chunk now multi-chunks (because budget + // shrank or model changed). + if (await hasLegacySingleChunkCompletion(engine, t.filePath, hash16)) { + skipReports.push({ + filePath: t.filePath, + reason: 'already_synthesized_legacy_single_chunk', + }); + continue; + } + + const chunks = splitTranscriptByBudget(t.content, t.contentHash, maxCharsPerChunk); + + // D5 cap hit: log + skip; do NOT write to dream_verdicts. Closes the + // poison-pill class — next cycle re-attempts under whatever budget + // is then current. + if (chunks.length > config.maxChunksPerTranscript) { + process.stderr.write( + `[dream] transcript ${t.basename} produced ${chunks.length} chunks at ` + + `${maxCharsPerChunk}-char budget (cap=${config.maxChunksPerTranscript}); skipping. ` + + `Increase dream.synthesize.max_chunks_per_transcript or use a larger-context model.\n`, + ); + skipReports.push({ + filePath: t.filePath, + reason: `oversize_after_split: ${chunks.length}/${config.maxChunksPerTranscript}`, + }); + continue; + } + + const isChunked = chunks.length > 1; + for (let i = 0; i < chunks.length; i++) { + const childData: SubagentHandlerData = { + prompt: buildSynthesisPrompt(t, chunks[i], i, chunks.length), + model: config.model, + max_turns: 30, + allowed_slug_prefixes: allowedSlugPrefixes, + }; + // Idempotency key parity: + // - single-chunk → legacy `dream:synth::` (byte- + // equivalent across versions; preserves dedup for unchanged + // transcripts on upgrade). + // - multi-chunk → `:cof` per chunk; durable across + // runs because D9 splitTranscriptByBudget is hash-deterministic. + const idempotency_key = isChunked + ? `dream:synth:${t.filePath}:${hash16}:c${i}of${chunks.length}` + : `dream:synth:${t.filePath}:${hash16}`; + const submitOpts: Partial = { + max_stalled: 3, + on_child_fail: 'continue', + idempotency_key, + timeout_ms: 30 * 60 * 1000, // 30 min per chunk + }; + const child = await queue.add( + 'subagent', + childData as unknown as Record, + submitOpts, + { allowProtectedSubmit: true }, + ); + childIds.push(child.id); + if (isChunked) { + chunkInfo.set(child.id, { idx: i, hash6 }); + } + } } // Wait for every child to reach a terminal state. Tick yieldDuringPhase @@ -222,7 +446,10 @@ export async function runPhaseSynthesize( // Collect slugs from put_page tool executions across the children // (codex finding #2: deterministic provenance, NOT pages.updated_at). - const writtenSlugs = await collectChildPutPageSlugs(engine, childIds); + // D6 orchestrator slug rewrite: chunkInfo drives post-hoc rewrite of + // bare-hash slugs to `-c` so chunked siblings can't collide + // even if Sonnet drops the chunk suffix. + const writtenSlugs = await collectChildPutPageSlugs(engine, childIds, chunkInfo); // Dual-write: reverse-render each DB row → markdown file. const reverseWriteCount = await reverseWriteSlugs(engine, opts.brainDir, writtenSlugs); @@ -239,9 +466,10 @@ export async function runPhaseSynthesize( await engine.setConfig('dream.synthesize.last_completion_ts', new Date().toISOString()); const ms = Date.now() - start; - return ok(`${worthProcessing.length} transcript(s) synthesized in ${(ms / 1000).toFixed(1)}s`, { + const submittedTranscripts = worthProcessing.length - skipReports.length; + return ok(`${submittedTranscripts} transcript(s) synthesized in ${(ms / 1000).toFixed(1)}s`, { transcripts_discovered: transcripts.length, - transcripts_processed: worthProcessing.length, + transcripts_processed: submittedTranscripts, pages_written: writtenSlugs.length, // v0.29: emit the slug list so the recompute_emotional_weight phase can // union with sync's pagesAffected and recompute weights for every page @@ -249,6 +477,12 @@ export async function runPhaseSynthesize( written_slugs: writtenSlugs, reverse_write_count: reverseWriteCount, child_outcomes: childOutcomes, + // Children submitted (one per chunk for chunked transcripts; one per + // transcript for single-chunk). Differs from transcripts_processed + // when chunking is in play. + children_submitted: childIds.length, + // D5 cap hits + D8 legacy-key skips. Empty when nothing skipped. + skips: skipReports, summary_slug: summarySlug, verdicts, }); @@ -269,6 +503,20 @@ interface SynthConfig { model: string; verdictModel: string; cooldownHours: number; + /** + * D1: Override the per-chunk token budget (model_context × HEADROOM_RATIO + * by default). Floor MIN_PROMPT_TOKENS, no upper cap (model context wins). + * Surface name follows PR #748: `dream.synthesize.max_prompt_tokens`. + * `null` means use the model-context lookup. + */ + maxPromptTokens: number | null; + /** + * D5/D10: Cap on chunks produced from a single transcript. On cap hit, the + * transcript is logged + skipped (NOT cached in dream_verdicts — closes the + * cache-poisoning class). Operator override: + * `dream.synthesize.max_chunks_per_transcript`. + */ + maxChunksPerTranscript: number; } async function loadSynthConfig(engine: BrainEngine): Promise { @@ -290,6 +538,8 @@ async function loadSynthConfig(engine: BrainEngine): Promise { fallback: 'haiku', }); const cooldownHoursStr = await engine.getConfig('dream.synthesize.cooldown_hours'); + const maxPromptTokensStr = await engine.getConfig('dream.synthesize.max_prompt_tokens'); + const maxChunksStr = await engine.getConfig('dream.synthesize.max_chunks_per_transcript'); let excludePatterns: string[] = ['medical', 'therapy']; if (excludeStr) { @@ -299,6 +549,23 @@ async function loadSynthConfig(engine: BrainEngine): Promise { } catch { /* keep default */ } } + // D1: max_prompt_tokens floored at MIN_PROMPT_TOKENS; null → use model lookup. + let maxPromptTokens: number | null = null; + if (maxPromptTokensStr) { + const parsed = parseInt(maxPromptTokensStr, 10); + if (Number.isFinite(parsed) && parsed > 0) { + maxPromptTokens = Math.max(MIN_PROMPT_TOKENS, parsed); + } + } + // D10: max_chunks default 24, floor 1. + let maxChunksPerTranscript = DEFAULT_MAX_CHUNKS; + if (maxChunksStr) { + const parsed = parseInt(maxChunksStr, 10); + if (Number.isFinite(parsed) && parsed >= 1) { + maxChunksPerTranscript = parsed; + } + } + return { enabled, corpusDir: corpusDir ?? null, @@ -308,6 +575,8 @@ async function loadSynthConfig(engine: BrainEngine): Promise { model, verdictModel, cooldownHours: cooldownHoursStr ? Math.max(0, parseInt(cooldownHoursStr, 10) || 12) : 12, + maxPromptTokens, + maxChunksPerTranscript, }; } @@ -422,16 +691,38 @@ Two reasons max, one phrase each.`; // ── Subagent prompt ────────────────────────────────────────────────── -function buildSynthesisPrompt(t: DiscoveredTranscript): string { +/** + * Build the prompt for one subagent. When `chunkTotal > 1`, the slug seed + * gains a `-c` suffix and the prompt names which chunk this is. + * + * D6 enforcement is orchestrator-side (rewriteChunkedSlug runs at slug- + * collection time). Sonnet still gets the chunked seed via the prompt's + * `USE THIS in slugs` rule for the happy path. + */ +function buildSynthesisPrompt( + t: DiscoveredTranscript, + chunkText: string, + chunkIdx: number, + chunkTotal: number, +): string { const dateHint = t.inferredDate ?? today(); - const hashSuffix = t.contentHash.slice(0, 6); const baseSlugSegment = sanitizeForSlug(t.basename) || `session-${dateHint}`; + const isChunked = chunkTotal > 1; + const hashSuffix = isChunked + ? `${t.contentHash.slice(0, 6)}-c${chunkIdx}` + : t.contentHash.slice(0, 6); + const chunkBanner = isChunked + ? `\n- This is CHUNK ${chunkIdx + 1} of ${chunkTotal} from the same transcript. Different chunks process different sections; do not assume continuity with other chunks.` + : ''; + const transcriptHeader = isChunked + ? `${t.filePath} (chunk ${chunkIdx + 1}/${chunkTotal})` + : t.filePath; return `You are synthesizing a conversation transcript into the user's personal knowledge brain. CONTEXT - Today's date: ${dateHint} - Transcript hash suffix (USE THIS in slugs): ${hashSuffix} -- Source file basename: ${baseSlugSegment} +- Source file basename: ${baseSlugSegment}${chunkBanner} OUTPUT POLICY (ALL of these are required) 1. Quote the user verbatim. Do not paraphrase memorable phrasings. @@ -450,9 +741,9 @@ C. People mentions: search first; if a page exists, do not put_page over it (the D. If nothing in this transcript meets the bar (significance filter already passed but the content is still routine), return without writing anything. -TRANSCRIPT (${t.filePath}) +TRANSCRIPT (${transcriptHeader}) --- -${t.content} +${chunkText} --- When done, briefly list the slugs you wrote in your final message so the orchestrator can audit.`; @@ -466,24 +757,71 @@ function sanitizeForSlug(s: string): string { .slice(0, 60); } -// ── Slug collection from child put_page calls (codex #2) ──────────── +// ── Slug collection from child put_page calls (codex #2 + D6) ──────── +/** + * D6 (orchestrator-side deterministic slug rewrite, zero Sonnet trust): + * two-stage path — raw fetch (no DISTINCT, preserves duplicate evidence) → + * in-memory chunk-suffix rewrite via `rewriteChunkedSlug` for chunked + * children → return distinct rewritten set. + * + * Closes Codex finding #2 ("collision detection via SELECT DISTINCT was + * fake"): we no longer need detection because the rewrite enforces + * uniqueness at slug-write time. + * + * `chunkInfo` maps child job_id → { chunk_index, hash6 }. Single-chunk + * children are absent from the map and pass through unchanged. + */ async function collectChildPutPageSlugs( engine: BrainEngine, childIds: number[], + chunkInfo: Map, ): Promise { if (childIds.length === 0) return []; - const rows = await engine.executeRaw<{ slug: string }>( - `SELECT DISTINCT input->>'slug' AS slug + // Raw fetch — NO SELECT DISTINCT. Preserves per-child slug duplicates so + // the orchestrator sees what each child wrote. + const rows = await engine.executeRaw<{ job_id: number; slug: string }>( + `SELECT job_id, input->>'slug' AS slug FROM subagent_tool_executions WHERE job_id = ANY($1::int[]) AND tool_name = 'brain_put_page' AND status = 'complete' - AND input ? 'slug' - ORDER BY 1`, + AND input ? 'slug'`, [childIds], ); - return rows.map(r => r.slug).filter((s): s is string => typeof s === 'string' && s.length > 0); + const rewritten = new Set(); + for (const r of rows) { + if (typeof r.slug !== 'string' || r.slug.length === 0) continue; + const ci = chunkInfo.get(r.job_id); + rewritten.add(ci ? rewriteChunkedSlug(r.slug, ci.hash6, ci.idx) : r.slug); + } + return Array.from(rewritten).sort(); +} + +/** + * D8: query for any `completed` legacy single-chunk job at the canonical + * idempotency key shape `dream:synth::`. Used at fan-out + * time to detect transcripts that were synthesized under the pre-chunking + * code path; those should NOT be re-submitted under chunked keys. + * + * Reuses the existing `minion_jobs.idempotency_key` index — no schema + * additions. One indexed lookup per worth-processing transcript. + */ +async function hasLegacySingleChunkCompletion( + engine: BrainEngine, + filePath: string, + hash16: string, +): Promise { + const legacyKey = `dream:synth:${filePath}:${hash16}`; + const rows = await engine.executeRaw<{ status: string }>( + `SELECT status + FROM minion_jobs + WHERE idempotency_key = $1 + AND status = 'completed' + LIMIT 1`, + [legacyKey], + ); + return rows.length > 0; } // ── Reverse-write DB rows → markdown files ─────────────────────────── diff --git a/src/core/minions/handlers/subagent.ts b/src/core/minions/handlers/subagent.ts index a6c69e6a5..4418e31d7 100644 --- a/src/core/minions/handlers/subagent.ts +++ b/src/core/minions/handlers/subagent.ts @@ -26,6 +26,7 @@ import Anthropic from '@anthropic-ai/sdk'; import type { MinionJobContext, MinionJob } from '../types.ts'; +import { UnrecoverableError } from '../types.ts'; import type { ContentBlock, SubagentHandlerData, @@ -352,6 +353,15 @@ export function makeSubagentHandler(deps: SubagentDeps) { } catch (err) { // Release lease eagerly on error so we don't starve capacity. await releaseLease(engine, lease.leaseId!).catch(() => {}); + // Terminal classification: a 400 "prompt is too long" from Anthropic + // is unrecoverable — retrying with the same prompt will always fail. + // Convert to UnrecoverableError so the worker routes the job + // straight to `dead`, bypassing max_stalled retries (the v0.30.x + // dream-cycle queue-clog the chunking work was built to prevent). + if (isPromptTooLongError(err)) { + const origMsg = err instanceof Error ? err.message : String(err); + throw new UnrecoverableError(`prompt_too_long: ${origMsg}`); + } throw err; } @@ -703,6 +713,43 @@ export class RateLeaseUnavailableError extends Error { } } +/** + * Detect Anthropic SDK errors that indicate the input prompt exceeded the + * model's context window. Two recognized shapes: + * - `Anthropic.APIError` with `.status === 400` and message containing + * "prompt is too long" (current SDK wording, observed in production + * as `prompt is too long: 1707509 tokens > 1000000 maximum`). + * - Any error whose message includes "prompt is too long" (defensive + * against SDK-wrap shape changes). + * + * Case-insensitive on the phrase. Also matches `request_too_large` and + * `invalid_request_error` types when accompanied by the same message. + * + * Exported for unit testing. + */ +export function isPromptTooLongError(err: unknown): boolean { + if (!err) return false; + // Walk both `.message` and `.error?.message` shapes. + const msg = (err as { message?: unknown })?.message; + const inner = (err as { error?: { message?: unknown } })?.error?.message; + const candidates = [msg, inner].filter((s): s is string => typeof s === 'string'); + for (const c of candidates) { + if (/prompt is too long/i.test(c)) return true; + } + // Anthropic SDK wraps with .status; 400 + 'invalid_request_error' / + // 'request_too_large' types both indicate the same class. Only treat + // as terminal when the message actually says prompt-too-long; broader + // 400s could be transient (e.g., malformed JSON from a test stub). + const status = (err as { status?: unknown })?.status; + const errType = (err as { error?: { type?: unknown } })?.error?.type; + if (status === 400 && (errType === 'invalid_request_error' || errType === 'request_too_large')) { + for (const c of candidates) { + if (/too long|exceed|maximum/i.test(c)) return true; + } + } + return false; +} + // ── Testing surface ───────────────────────────────────────── export const __testing = { diff --git a/test/cycle-synthesize-chunker.test.ts b/test/cycle-synthesize-chunker.test.ts new file mode 100644 index 000000000..a12d538c9 --- /dev/null +++ b/test/cycle-synthesize-chunker.test.ts @@ -0,0 +1,176 @@ +/** + * Unit tests for the v0.30.2 dream/synthesize chunker (D9 hash-deterministic + * boundaries) and orchestrator slug rewrite (D6). + * + * Pure functions only. Exercises: + * - Single-chunk pass-through under budget. + * - 3-tier boundary ladder: ## Topic: > --- > nearest \n. + * - Hash determinism: same (content, hash, maxChars) → identical chunks + * regardless of how many times you call it. + * - Different content_hash → potentially different boundaries (jitter + * within back-half-of-budget window). + * - Hard fallback when no boundary fits. + * - Slug rewrite: bare hash6 → adds -c; correctly suffixed → unchanged; + * unknown shape → pass-through. + * + * No DB, no Anthropic, no fixtures. + */ + +import { describe, test, expect } from 'bun:test'; +import { splitTranscriptByBudget, rewriteChunkedSlug } from '../src/core/cycle/synthesize.ts'; + +describe('splitTranscriptByBudget — single chunk path', () => { + test('returns single-element array when content <= maxChars', () => { + const out = splitTranscriptByBudget('hello world', 'abc123def', 1000); + expect(out).toEqual(['hello world']); + }); + + test('content exactly at maxChars stays one chunk', () => { + const content = 'x'.repeat(500); + const out = splitTranscriptByBudget(content, 'abc123def', 500); + expect(out).toHaveLength(1); + expect(out[0]).toEqual(content); + }); + + test('throws on non-positive maxChars', () => { + expect(() => splitTranscriptByBudget('hi', 'abc', 0)).toThrow(/maxChars/); + expect(() => splitTranscriptByBudget('hi', 'abc', -5)).toThrow(/maxChars/); + }); +}); + +describe('splitTranscriptByBudget — boundary ladder', () => { + test('Tier 1: prefers \\n## Topic: separator inside back-half window', () => { + // Budget = 300 → searchStart ∈ [150, 179]. Place the topic separator + // around position 220 so it's deep in the back-half window. + const padding = 'x'.repeat(220); // 0..219 + const sep = '\n## Topic: chunk-break\nafter break content'; // pos 220+ + const tail = 'y'.repeat(200); // overflow + const content = padding + sep + tail; + const out = splitTranscriptByBudget(content, 'abcdef0123456789', 300); + expect(out.length).toBeGreaterThanOrEqual(2); + // Chunk 2 starts at the "\n## Topic:" boundary (the boundary char is + // included with the second chunk by design — `slice(0, split)` cuts + // before, `slice(split)` keeps the newline). + expect(out[1].startsWith('\n## Topic:')).toBe(true); + }); + + test('Tier 2: falls back to --- HR marker when no Topic separator in window', () => { + // Budget = 300, searchStart ∈ [150, 179]. HR marker around pos 220. + const padding = 'no-topic-marker-here\n'.repeat(10); // 210 chars (~10x21) + const sep = '\n---\nafter rule\n'; + const tail = 'tail content '.repeat(20); // overflow + const content = padding + sep + tail; + const out = splitTranscriptByBudget(content, 'aabbccdd11223344', 300); + expect(out.length).toBeGreaterThanOrEqual(2); + // Chunk 2 starts at the HR marker. + expect(out[1].startsWith('\n---\n')).toBe(true); + }); + + test('Tier 3: falls back to nearest newline when no Topic / HR in window', () => { + // No topic separators, no HR markers — just paragraphs. + const para = 'sentence one. sentence two.\nmore prose to fill space\n'; + const content = para.repeat(20); // ~1080 chars + const out = splitTranscriptByBudget(content, '11223344aabbccdd', 200); + expect(out.length).toBeGreaterThanOrEqual(2); + // Every chunk other than the last should END with content that allowed + // a newline split — chunks 2..N should NOT begin with a partial word. + for (let i = 1; i < out.length; i++) { + // After the split, the rest begins at the boundary char (newline); + // when the boundary is "\n", chunk i starts with the newline char. + expect(out[i].startsWith('\n')).toBe(true); + } + }); + + test('hard-split when no boundary fits anywhere in the window', () => { + // Single huge run of non-newline chars exceeding budget. + const content = 'a'.repeat(1500); // no newlines, no separators + const out = splitTranscriptByBudget(content, 'cafebabe12345678', 500); + // Walks deterministically: first split at maxChars=500, so chunks + // are [500, 500, 500]. + expect(out).toHaveLength(3); + expect(out[0]).toHaveLength(500); + expect(out[1]).toHaveLength(500); + expect(out[2]).toHaveLength(500); + expect(out.join('')).toEqual(content); + }); +}); + +describe('splitTranscriptByBudget — D9 hash-deterministic identity', () => { + test('same inputs → identical chunks across many calls', () => { + const content = ('paragraph with some text\nand a newline\n').repeat(100); + const hash = '0123456789abcdef0123456789abcdef'; + const a = splitTranscriptByBudget(content, hash, 500); + const b = splitTranscriptByBudget(content, hash, 500); + const c = splitTranscriptByBudget(content, hash, 500); + expect(a).toEqual(b); + expect(b).toEqual(c); + }); + + test('different content_hash CAN produce different boundaries (jitter)', () => { + // Construct content with multiple newline candidates inside the + // back-half-of-budget search window, so different hash offsets pick + // different newlines. + const content = ('w1 w2 w3 w4 w5\n').repeat(200); + const a = splitTranscriptByBudget(content, '00000000aabbccdd', 500); + const b = splitTranscriptByBudget(content, 'ffffffff77665544', 500); + // The two splits MAY differ; assertion is that determinism is per-hash, + // not that the function is hash-invariant. + expect(a.join('')).toEqual(content); + expect(b.join('')).toEqual(content); + }); + + test('reconstructs content exactly when joined back', () => { + const content = ('# heading\n\nbody line\n## Topic: something\nmore body\n').repeat(50); + const out = splitTranscriptByBudget(content, 'aaaaaaaa11111111', 400); + expect(out.join('')).toEqual(content); + }); + + test('non-hex hash falls through to offset 0', () => { + // parseHashOffset returns 0 on bad hex; chunks should still be valid. + const content = ('xx\n').repeat(1000); + const out = splitTranscriptByBudget(content, '!!!not-hex!!!', 500); + expect(out.length).toBeGreaterThanOrEqual(2); + expect(out.join('')).toEqual(content); + }); +}); + +describe('rewriteChunkedSlug — D6 zero-Sonnet-trust slug rewrite', () => { + test('appends -c when slug ends with bare hash6', () => { + expect(rewriteChunkedSlug('wiki/originals/ideas/2026-05-08-thesis-abc123', 'abc123', 0)) + .toBe('wiki/originals/ideas/2026-05-08-thesis-abc123-c0'); + expect(rewriteChunkedSlug('wiki/personal/reflections/2026-05-08-foo-deadbe', 'deadbe', 2)) + .toBe('wiki/personal/reflections/2026-05-08-foo-deadbe-c2'); + }); + + test('passes through when slug already correctly chunk-suffixed', () => { + expect(rewriteChunkedSlug('wiki/originals/ideas/2026-05-08-thesis-abc123-c0', 'abc123', 0)) + .toBe('wiki/originals/ideas/2026-05-08-thesis-abc123-c0'); + expect(rewriteChunkedSlug('foo-bar-deadbe-c5', 'deadbe', 5)) + .toBe('foo-bar-deadbe-c5'); + }); + + test('does not double-rewrite when -c would conflict with idx', () => { + // If a slug already has a chunk suffix (any idx) it ends with -c; + // the regex is anchored on bare hash6 at end, so this case passes through. + const result = rewriteChunkedSlug('foo-abc123-c1', 'abc123', 0); + expect(result).toBe('foo-abc123-c1'); // unchanged — does not become foo-abc123-c1-c0 + }); + + test('passes through when slug does not end with the expected hash6', () => { + expect(rewriteChunkedSlug('wiki/foo/bar', 'abc123', 0)).toBe('wiki/foo/bar'); + expect(rewriteChunkedSlug('wiki/foo/bar-xyzxyz', 'abc123', 0)).toBe('wiki/foo/bar-xyzxyz'); + }); + + test('handles slug that IS exactly hash6', () => { + expect(rewriteChunkedSlug('abc123', 'abc123', 3)).toBe('abc123-c3'); + }); + + test('handles slug ending with / path-segment shape', () => { + expect(rewriteChunkedSlug('foo/bar/abc123', 'abc123', 1)) + .toBe('foo/bar/abc123-c1'); + }); + + test('empty slug passes through', () => { + expect(rewriteChunkedSlug('', 'abc123', 0)).toBe(''); + }); +}); diff --git a/test/e2e/dream-synthesize-chunking.test.ts b/test/e2e/dream-synthesize-chunking.test.ts new file mode 100644 index 000000000..fedb06aea --- /dev/null +++ b/test/e2e/dream-synthesize-chunking.test.ts @@ -0,0 +1,307 @@ +/** + * E2E for v0.30.2 dream/synthesize chunking. PGLite, no API key required. + * + * Pre-seeds verdicts so the Haiku gate is bypassed; submits subagent jobs + * but never runs them (no worker spawned). Tests inspect minion_jobs to + * verify submission shape (chunk count, idempotency keys, skip-paths). + * + * Coverage: + * - D5 cap-hit: chunks > maxChunks → log + skip with no minion_jobs row + * and no dream_verdicts cache write (closes the poison-pill class). + * - D8 legacy single-chunk migration: pre-seed a `completed` legacy job + * for the same content hash → next synthesize skips submission. + * - Chunked path: fat transcript spawns N children with chunk-suffixed + * idempotency keys; single-chunk path keeps the legacy key shape. + * + * Run: bun test test/e2e/dream-synthesize-chunking.test.ts + */ + +import { describe, test, expect } from 'bun:test'; +import { mkdtempSync, rmSync, writeFileSync } from 'fs'; +import { tmpdir } from 'os'; +import { join } from 'path'; +import { PGLiteEngine } from '../../src/core/pglite-engine.ts'; +import { runPhaseSynthesize } from '../../src/core/cycle/synthesize.ts'; + +interface TestRig { + engine: PGLiteEngine; + brainDir: string; + corpusDir: string; + cleanup: () => Promise; +} + +async function setupRig(): Promise { + const engine = new PGLiteEngine(); + await engine.connect({ engine: 'pglite' } as never); + await engine.initSchema(); + const brainDir = mkdtempSync(join(tmpdir(), 'gbrain-chunk-brain-')); + const corpusDir = mkdtempSync(join(tmpdir(), 'gbrain-chunk-corpus-')); + return { + engine, + brainDir, + corpusDir, + cleanup: async () => { + try { await engine.disconnect(); } catch { /* best-effort */ } + try { rmSync(brainDir, { recursive: true, force: true }); } catch { /* */ } + try { rmSync(corpusDir, { recursive: true, force: true }); } catch { /* */ } + }, + }; +} + +async function withoutAnthropicKey(body: () => Promise): Promise { + const saved = process.env.ANTHROPIC_API_KEY; + delete process.env.ANTHROPIC_API_KEY; + try { + return await body(); + } finally { + if (saved === undefined) delete process.env.ANTHROPIC_API_KEY; + else process.env.ANTHROPIC_API_KEY = saved; + } +} + +/** + * Run `body` while a background loop force-cancels any subagent jobs the + * synthesize phase submits. Without a worker, those jobs would sit in + * `waiting` forever and runPhaseSynthesize's waitForCompletion blocks for + * 35 minutes. Cancelling moves them to a terminal state so the phase + * returns and we can inspect submission shape. + */ +async function withSubagentAutoCancel(engine: PGLiteEngine, body: () => Promise): Promise { + let stopped = false; + const loop = (async () => { + while (!stopped) { + await new Promise(r => setTimeout(r, 50)); + try { + await engine.executeRaw( + `UPDATE minion_jobs + SET status = 'cancelled', finished_at = now() + WHERE name = 'subagent' AND status IN ('waiting', 'active')`, + ); + } catch { + // Race against shutdown is fine; ignore. + } + } + })(); + try { + return await body(); + } finally { + stopped = true; + await loop; + } +} + +/** + * Pre-seed a `worth_processing=true` verdict so the synthesize phase skips + * the Haiku call and proceeds directly to fan-out. Computes the hash the + * same way `discoverTranscripts` does (sha256 of content). + */ +async function seedVerdict(engine: PGLiteEngine, filePath: string, content: string): Promise { + const { createHash } = await import('node:crypto'); + const contentHash = createHash('sha256').update(content, 'utf8').digest('hex'); + await engine.putDreamVerdict(filePath, contentHash, { + worth_processing: true, + reasons: ['seeded for chunking E2E test'], + }); + return contentHash; +} + +/** + * Resolve the absolute path the discover walker will see for a file in the + * corpus dir, since `discoverTranscripts` joins corpus + name. + */ +function corpusPath(corpusDir: string, basename: string): string { + return join(corpusDir, basename); +} + +describe('E2E synthesize chunking — D5 cap hit', () => { + test('chunks > max_chunks_per_transcript → skipped with no jobs and no verdict-cache write', async () => { + const rig = await setupRig(); + try { + // Tiny chunk budget (forces N chunks) + tiny cap (forces cap hit). + // 100K is the floor; even at the floor, 350K-char tester content + // chunks to ~1 chunk... we need budget below floor to force many + // chunks. Use the chunks_per_transcript cap instead. + await rig.engine.setConfig('dream.synthesize.enabled', 'true'); + await rig.engine.setConfig('dream.synthesize.session_corpus_dir', rig.corpusDir); + await rig.engine.setConfig('dream.synthesize.max_prompt_tokens', '100000'); // floor → 350K char budget + await rig.engine.setConfig('dream.synthesize.max_chunks_per_transcript', '2'); + + // 1.5M chars → 5 chunks at 350K-char budget → exceeds cap=2. + const basename = '2026-05-08-fat-transcript.txt'; + const filePath = corpusPath(rig.corpusDir, basename); + const content = 'fat transcript line\n'.repeat(75_000); // ~1.5M chars + writeFileSync(filePath, content); + await seedVerdict(rig.engine, filePath, content); + + await withoutAnthropicKey(async () => { + const result = await runPhaseSynthesize(rig.engine, { + brainDir: rig.brainDir, + dryRun: false, + }); + + expect(result.status).toBe('ok'); + const details = result.details as { + children_submitted: number; + skips: Array<{ filePath: string; reason: string }>; + }; + expect(details.children_submitted).toBe(0); + expect(details.skips).toHaveLength(1); + expect(details.skips[0].filePath).toBe(filePath); + expect(details.skips[0].reason).toMatch(/oversize_after_split/); + }); + + // No subagent jobs submitted. + const jobs = await rig.engine.executeRaw<{ cnt: string | number }>( + `SELECT count(*) AS cnt FROM minion_jobs WHERE name = 'subagent'`, + ); + expect(Number(jobs[0].cnt)).toBe(0); + + // D5: dream_verdicts NOT written for the cap-hit path. + // Verify by re-reading the verdict — our seeded row is the ONLY entry. + const verdicts = await rig.engine.executeRaw<{ cnt: string | number }>( + `SELECT count(*) AS cnt FROM dream_verdicts`, + ); + expect(Number(verdicts[0].cnt)).toBe(1); // only the seed; no cap-hit row added + } finally { + await rig.cleanup(); + } + }, 30_000); +}); + +describe('E2E synthesize chunking — D8 legacy single-chunk migration', () => { + test('completed legacy idempotency key → skip submission entirely', async () => { + const rig = await setupRig(); + try { + await rig.engine.setConfig('dream.synthesize.enabled', 'true'); + await rig.engine.setConfig('dream.synthesize.session_corpus_dir', rig.corpusDir); + + const basename = '2026-04-25-already-synthesized.txt'; + const filePath = corpusPath(rig.corpusDir, basename); + const content = 'meaningful conversation lines\n'.repeat(200); + writeFileSync(filePath, content); + const contentHash = await seedVerdict(rig.engine, filePath, content); + + // Pre-seed a completed `subagent` job at the legacy idempotency key. + const legacyKey = `dream:synth:${filePath}:${contentHash.slice(0, 16)}`; + await rig.engine.executeRaw( + `INSERT INTO minion_jobs (name, queue, status, idempotency_key, finished_at) + VALUES ('subagent', 'default', 'completed', $1, now())`, + [legacyKey], + ); + + await withoutAnthropicKey(async () => { + const result = await runPhaseSynthesize(rig.engine, { + brainDir: rig.brainDir, + dryRun: false, + }); + const details = result.details as { + children_submitted: number; + skips: Array<{ reason: string }>; + }; + expect(details.children_submitted).toBe(0); + expect(details.skips).toHaveLength(1); + expect(details.skips[0].reason).toBe('already_synthesized_legacy_single_chunk'); + }); + + // No NEW subagent job: still exactly one (the seeded completed row). + const jobs = await rig.engine.executeRaw<{ cnt: string | number }>( + `SELECT count(*) AS cnt FROM minion_jobs WHERE name = 'subagent'`, + ); + expect(Number(jobs[0].cnt)).toBe(1); + } finally { + await rig.cleanup(); + } + }, 30_000); +}); + +describe('E2E synthesize chunking — fan-out shape', () => { + test('single-chunk transcript uses legacy idempotency key (parity on upgrade)', async () => { + const rig = await setupRig(); + try { + await rig.engine.setConfig('dream.synthesize.enabled', 'true'); + await rig.engine.setConfig('dream.synthesize.session_corpus_dir', rig.corpusDir); + // Default budget is plenty for 5KB content. + + const basename = '2026-04-25-small.txt'; + const filePath = corpusPath(rig.corpusDir, basename); + const content = 'small transcript content\n'.repeat(100); // ~2.5KB + writeFileSync(filePath, content); + const contentHash = await seedVerdict(rig.engine, filePath, content); + + await withoutAnthropicKey(async () => { + await withSubagentAutoCancel(rig.engine, async () => { + const result = await runPhaseSynthesize(rig.engine, { + brainDir: rig.brainDir, + dryRun: false, + }); + const details = result.details as { children_submitted: number }; + expect(details.children_submitted).toBe(1); + }); + }); + + const expectedKey = `dream:synth:${filePath}:${contentHash.slice(0, 16)}`; + const rows = await rig.engine.executeRaw<{ idempotency_key: string }>( + `SELECT idempotency_key FROM minion_jobs WHERE name = 'subagent' ORDER BY id`, + ); + expect(rows).toHaveLength(1); + expect(rows[0].idempotency_key).toBe(expectedKey); + // Specifically: legacy key shape has NO ":cof" suffix. + expect(rows[0].idempotency_key).not.toMatch(/:c\d+of\d+$/); + } finally { + await rig.cleanup(); + } + }, 30_000); + + test('multi-chunk transcript spawns N children with chunk-suffixed idempotency keys', async () => { + const rig = await setupRig(); + try { + await rig.engine.setConfig('dream.synthesize.enabled', 'true'); + await rig.engine.setConfig('dream.synthesize.session_corpus_dir', rig.corpusDir); + // Floor at 100K tokens → 350K-char chunk budget. A 1.5M-char transcript + // chunks to ~5 chunks. Default cap is 24, so submission proceeds. + await rig.engine.setConfig('dream.synthesize.max_prompt_tokens', '100000'); + + const basename = '2026-05-08-fat.txt'; + const filePath = corpusPath(rig.corpusDir, basename); + const content = 'fat transcript line with newline\n'.repeat(50_000); // ~1.65M chars + writeFileSync(filePath, content); + const contentHash = await seedVerdict(rig.engine, filePath, content); + const hash16 = contentHash.slice(0, 16); + + await withoutAnthropicKey(async () => { + await withSubagentAutoCancel(rig.engine, async () => { + const result = await runPhaseSynthesize(rig.engine, { + brainDir: rig.brainDir, + dryRun: false, + }); + const details = result.details as { children_submitted: number }; + expect(details.children_submitted).toBeGreaterThan(1); + }); + }); + + const rows = await rig.engine.executeRaw<{ idempotency_key: string }>( + `SELECT idempotency_key FROM minion_jobs WHERE name = 'subagent' ORDER BY id`, + ); + expect(rows.length).toBeGreaterThan(1); + // Every key matches the chunked shape `dream:synth:::cof`. + for (const r of rows) { + expect(r.idempotency_key).toMatch( + new RegExp(`^dream:synth:${escapeRe(filePath)}:${hash16}:c\\d+of\\d+$`), + ); + } + // Chunk indices are unique 0..N-1. + const indices = rows + .map(r => /:c(\d+)of/.exec(r.idempotency_key)?.[1]) + .map(s => Number(s)) + .sort((a, b) => a - b); + const expected = Array.from({ length: rows.length }, (_, i) => i); + expect(indices).toEqual(expected); + } finally { + await rig.cleanup(); + } + }, 30_000); +}); + +function escapeRe(s: string): string { + return s.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); +} diff --git a/test/subagent-prompt-too-long.test.ts b/test/subagent-prompt-too-long.test.ts new file mode 100644 index 000000000..28bfa54b5 --- /dev/null +++ b/test/subagent-prompt-too-long.test.ts @@ -0,0 +1,91 @@ +/** + * Unit tests for the v0.30.2 terminal-error classification of Anthropic's + * "prompt is too long" 400 in the subagent handler. + * + * The handler converts these to UnrecoverableError so the worker routes the + * job straight to `dead`, bypassing max_stalled retries (the prior 3x-retry + * pathology that clogged the queue). + * + * Pure function tests of `isPromptTooLongError`. End-to-end coverage of the + * client.create() try/catch lives in the synthesize E2E test. + */ + +import { describe, test, expect } from 'bun:test'; +import { isPromptTooLongError } from '../src/core/minions/handlers/subagent.ts'; + +describe('isPromptTooLongError', () => { + test('matches the production message verbatim', () => { + const err = new Error('prompt is too long: 1707509 tokens > 1000000 maximum'); + expect(isPromptTooLongError(err)).toBe(true); + }); + + test('matches case-insensitively', () => { + expect(isPromptTooLongError(new Error('Prompt Is Too Long'))).toBe(true); + expect(isPromptTooLongError(new Error('PROMPT IS TOO LONG'))).toBe(true); + }); + + test('matches when message is on the inner .error.message field', () => { + // Mimic Anthropic SDK error wrapping shape. + const err = { + status: 400, + error: { + type: 'invalid_request_error', + message: 'prompt is too long: 1234567 tokens > 1000000 maximum', + }, + message: 'BadRequestError', + }; + expect(isPromptTooLongError(err)).toBe(true); + }); + + test('matches 400 + invalid_request_error + "exceed" wording (defensive)', () => { + // Defensive against future SDK message-wording changes. + const err = { + status: 400, + error: { type: 'invalid_request_error', message: 'request exceeds maximum context' }, + message: 'BadRequestError', + }; + expect(isPromptTooLongError(err)).toBe(true); + }); + + test('matches 400 + request_too_large type', () => { + const err = { + status: 400, + error: { type: 'request_too_large', message: 'too long' }, + message: 'BadRequestError', + }; + expect(isPromptTooLongError(err)).toBe(true); + }); + + test('does NOT match unrelated 400 errors', () => { + const err = { + status: 400, + error: { type: 'invalid_request_error', message: 'malformed JSON' }, + message: 'BadRequestError', + }; + expect(isPromptTooLongError(err)).toBe(false); + }); + + test('does NOT match unrelated transient errors', () => { + expect(isPromptTooLongError(new Error('network timeout'))).toBe(false); + expect(isPromptTooLongError(new Error('rate limit exceeded'))).toBe(false); + expect(isPromptTooLongError({ status: 500, message: 'internal error' })).toBe(false); + expect(isPromptTooLongError({ status: 429, message: 'overloaded' })).toBe(false); + }); + + test('does NOT match null / undefined / non-error inputs', () => { + expect(isPromptTooLongError(null)).toBe(false); + expect(isPromptTooLongError(undefined)).toBe(false); + expect(isPromptTooLongError(0)).toBe(false); + expect(isPromptTooLongError('plain string')).toBe(false); + expect(isPromptTooLongError({})).toBe(false); + }); + + test('matches synthetic SDK shape with status 400 + message containing the phrase', () => { + // Some SDK versions surface the phrase only on the outer .message. + const err = { + status: 400, + message: 'Error: prompt is too long: 2000000 tokens > 1000000 maximum', + }; + expect(isPromptTooLongError(err)).toBe(true); + }); +});