mirror of
https://github.com/garrytan/gbrain.git
synced 2026-08-14 08:53:22 +00:00
Compare commits
6
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
19f919e283 | ||
|
|
6943f421ad | ||
|
|
00ef41657a | ||
|
|
4fd16f767b | ||
|
|
d0c244b3c3 | ||
|
|
0a7e087fd3 |
@@ -2,6 +2,67 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.30.2] - 2026-05-08
|
||||
|
||||
**Dream synthesize stops dropping fat transcripts. Subagents that overflow Anthropic's context die once, not three times. The queue stops clogging.**
|
||||
|
||||
The v0.30 dream cycle has been stalled since May 2 for one user — daily aggregated transcripts at 2.7-4.5MB each generate 1.7M-token Anthropic prompts, which hit the 1M-token hard limit and 400. The subagent handler treated those failures as renewable, so every doomed transcript stalled three times before dead-lettering, and every new cycle re-discovered and re-submitted the same fat transcripts. Six days of synth backlog, queue full of doomed work.
|
||||
|
||||
v0.30.2 adds model-aware chunking + terminal-error classification + a poison-pill-free skip path. Wintermute's [PR #748](https://github.com/garrytan/gbrain/pull/748) supplied the boundary heuristics (`## Topic:` → `---` → `\n` ladder) and the `dream.synthesize.max_prompt_tokens` config surface. Garry's branch extended that with model-aware budgets, deterministic chunk identity for partial-progress safety, orchestrator-side slug rewriting for zero Sonnet trust on collisions, and doctor-surface visibility.
|
||||
|
||||
### What you can now do
|
||||
|
||||
**Fat transcripts get synthesized instead of dropped.** A 4.5MB daily-aggregated transcript on Sonnet 4.6 (200K context) now chunks into 7-8 children at the 630KB-per-chunk default budget, each subagent processes its slice, and the orchestrator collects everything. Per-chunk children are independent — chunk 5's content doesn't leak into chunk 6's prompt.
|
||||
|
||||
**Doomed transcripts die loud, not silent.** Anthropic 400 "prompt is too long" responses now classify as `UnrecoverableError`. The job goes straight to `dead` on first attempt. No three-stall retry pile. `gbrain doctor` surfaces the count under `queue_health` with a fix hint pointing at `gbrain dream --phase synthesize --dry-run --json` so you can identify the offender.
|
||||
|
||||
**Existing single-chunk transcripts are not re-synthesized on upgrade.** The legacy `dream:synth:<filePath>:<hash16>` idempotency-key shape is preserved byte-for-byte for the single-chunk case. Already-synthesized transcripts skip with `already_synthesized_legacy_single_chunk` — no re-spend on Sonnet for content already in the brain.
|
||||
|
||||
```bash
|
||||
# Operator escape hatches
|
||||
gbrain config set dream.synthesize.max_prompt_tokens 600000 # tune chunk budget below model context
|
||||
gbrain config set dream.synthesize.max_chunks_per_transcript 32 # raise the chunk-explosion cap
|
||||
gbrain jobs prune --status dead --queue default # one-time cleanup of pre-v0.30.2 doomed jobs
|
||||
```
|
||||
|
||||
### How it works under the hood
|
||||
|
||||
**Model-aware chunk budget.** A static `MODEL_CONTEXT_TOKENS` map keys on the resolved Anthropic model id (Opus 4.7 → 1M, Sonnet 4.6 → 200K, Haiku → 200K). Per-chunk budget is `floor(context × 0.9 × 3.5 chars/token)`, leaving 10% headroom for system prompt + tool defs + output. Non-Anthropic ids (`gpt-5`, `gemini-3-pro`, custom alias) fall back to a 180K-token safe default with a once-per-process stderr warning.
|
||||
|
||||
**Hash-deterministic chunk identity.** `splitTranscriptByBudget(content, contentHash, maxChars)` is a pure function. The 3-tier boundary search window (back-half-of-budget) gets seeded with a deterministic offset derived from the first 32 bits of `contentHash`, so the same content always chunks identically. Chunk 2 of a transcript that died terminally produces byte-identical content on retry — per-chunk idempotency keys (`dream:synth:<filePath>:<hash16>:c<i>of<n>`) are durable across runs.
|
||||
|
||||
**Orchestrator-side slug rewrite, zero Sonnet trust.** `collectChildPutPageSlugs` no longer does `SELECT DISTINCT` (which would erase collision evidence). Instead it raw-fetches every (job_id, slug) pair, then for chunked children rewrites bare-hash6 slugs to `<hash6>-c<idx>` if Sonnet drops the chunk suffix. Two siblings can't collide even if Sonnet ignores the prompt's `USE THIS in slugs` rule.
|
||||
|
||||
**Cap-hit skips don't poison the verdict cache.** When chunks exceed `max_chunks_per_transcript` (default 24), the orchestrator logs + skips without writing to `dream_verdicts`. Next cycle re-attempts under whatever budget is then current — closes the cache-poisoning class entirely.
|
||||
|
||||
### Out of scope (deferred to v0.30.3+)
|
||||
|
||||
- **Per-turn token-budget guard in subagent.ts.** This release bounds the INITIAL prompt size only. Tool-loop accumulation (search/get_page results bloating each subsequent turn) can still hit `prompt_too_long` mid-conversation; the terminal-error classification catches it then, just less cleanly.
|
||||
- **Tighter token estimator for code/JSON/CJK-dense content.** The 3.5 chars/token ratio is close to safe for English transcripts but can underflow on dense content. Production telemetry will tell us if a per-recipe override is needed.
|
||||
|
||||
### To take advantage of v0.30.2
|
||||
|
||||
`gbrain upgrade` is enough — the change is purely runtime behavior, no schema migration required.
|
||||
|
||||
If your dream queue accumulated doomed subagent jobs from a prior version:
|
||||
```bash
|
||||
gbrain jobs list --status dead --queue default | grep prompt_too_long
|
||||
gbrain jobs prune --status dead --queue default
|
||||
```
|
||||
|
||||
If you want to verify chunking on your specific corpus before next autopilot cycle:
|
||||
```bash
|
||||
gbrain dream --phase synthesize --dry-run --json | jq '.details.skips'
|
||||
```
|
||||
|
||||
A non-empty `skips` array means a transcript hit the 24-chunk cap or already exists at the legacy single-chunk key. Empty means everything fits.
|
||||
|
||||
### For contributors
|
||||
|
||||
Two-round Codex outside-voice review surfaced 12 structural risks; six folded directly into the design (D5 cap-hit no-cache, D6 orchestrator slug rewrite, D7 honest tool-loop scope, D8 legacy-key migration, D9 hash-deterministic boundaries, D10 operator-configurable cap), and the four PARTIAL items have follow-up TODOs. The plan file at `~/.claude/plans/system-instruction-you-are-working-mossy-fox.md` documents the full decision history. PR #748's contribution (boundary ladder + config-key naming + 3.5 chars/token estimator) is preserved verbatim and credited; this branch supersedes it with the structural safeguards.
|
||||
|
||||
Tests: 27 unit cases (`test/cycle-synthesize-chunker.test.ts`, `test/subagent-prompt-too-long.test.ts`) + 4 PGLite E2E cases (`test/e2e/dream-synthesize-chunking.test.ts`) covering D5 cap hit, D8 legacy-key skip, single-chunk parity, multi-chunk fan-out shape.
|
||||
|
||||
## [0.30.1] - 2026-05-08
|
||||
|
||||
**Operational hardening: gbrain upgrade just works on Supabase. DDL stops timing out on the pooler. Migrations stop wedging. HNSW rebuilds stop nuking your search. Backfills stop being bespoke scripts.**
|
||||
|
||||
@@ -122,7 +122,7 @@ strict behavior when unset.
|
||||
- `src/core/minions/handlers/shell.ts` — `shell` job handler. Spawns `/bin/sh -c cmd` (absolute path, PATH-override-safe) or `argv[0] argv[1..]` (no shell). Env allowlist: `PATH, HOME, USER, LANG, TZ, NODE_ENV` + caller `env:` overrides. UTF-8-safe stdout/stderr tail via `string_decoder.StringDecoder`. Abort (either `ctx.signal` or `ctx.shutdownSignal`) fires SIGTERM → 5s grace → SIGKILL on child. Requires `GBRAIN_ALLOW_SHELL_JOBS=1` on worker (gated by `registerBuiltinHandlers`).
|
||||
- `src/core/minions/handlers/shell-audit.ts` — per-submission JSONL audit trail at `~/.gbrain/audit/shell-jobs-YYYY-Www.jsonl` (ISO-week rotation; override via `GBRAIN_AUDIT_DIR`). Best-effort: `mkdirSync(recursive)` + `appendFileSync`; failures logged to stderr, submission not blocked. Logs cmd (first 80 chars) or argv (JSON array). Never logs env values.
|
||||
- `src/core/minions/backpressure-audit.ts` (v0.19.1) — sibling of shell-audit.ts for `maxWaiting` coalesce events. JSONL at `~/.gbrain/audit/backpressure-YYYY-Www.jsonl`. Fires one line per coalesce with `(queue, name, waiting_count, max_waiting, returned_job_id, ts)`. Closes the silent-drop vector the v0.19.0 maxWaiting guard introduced.
|
||||
- `src/core/minions/handlers/subagent.ts` (v0.15) — LLM-loop handler. Two-phase tool persistence (pending → complete/failed), replay reconciliation for mid-dispatch crashes, dual-signal abort (`ctx.signal` + `ctx.shutdownSignal`), Anthropic prompt caching on system + tool defs. `makeSubagentHandler({engine, client?, ...})` factory; `MessagesClient` is an injectable interface the real SDK implements structurally. Throws `RateLeaseUnavailableError` (renewable) when rate-lease capacity is full.
|
||||
- `src/core/minions/handlers/subagent.ts` (v0.15) — LLM-loop handler. Two-phase tool persistence (pending → complete/failed), replay reconciliation for mid-dispatch crashes, dual-signal abort (`ctx.signal` + `ctx.shutdownSignal`), Anthropic prompt caching on system + tool defs. `makeSubagentHandler({engine, client?, ...})` factory; `MessagesClient` is an injectable interface the real SDK implements structurally. Throws `RateLeaseUnavailableError` (renewable) when rate-lease capacity is full. **v0.30.2:** Anthropic 400 `prompt is too long` responses (status 400 + body matches `/prompt is too long|prompt_too_long|context.*length/i`) classify as `UnrecoverableError` so the job goes straight to `dead` on first attempt instead of stalling three times before dead-lettering. Catches both initial-prompt overflow and turn-N tool-loop accumulation that the chunker in `synthesize.ts` can't bound ahead of time.
|
||||
- `src/core/minions/handlers/subagent-aggregator.ts` (v0.15) — `subagent_aggregator` handler. Claims AFTER all children resolve (queue changes guarantee every terminal child posts a `child_done` inbox message with outcome). Reads inbox via `ctx.readInbox()`, builds deterministic mixed-outcome markdown summary. No LLM call in v0.15.
|
||||
- `src/core/minions/handlers/subagent-audit.ts` (v0.15) — JSONL audit + heartbeat writer at `~/.gbrain/audit/subagent-jobs-YYYY-Www.jsonl`. Events: `submission` (one line per submit) + `heartbeat` (per turn boundary: `llm_call_started | llm_call_completed | tool_called | tool_result | tool_failed`). Never logs prompts or tool inputs. `readSubagentAuditForJob(jobId, {sinceIso})` is the readback path for `gbrain agent logs`.
|
||||
- `src/core/minions/rate-leases.ts` (v0.15) — lease-based concurrency cap for outbound providers (default key `anthropic:messages`, max via `GBRAIN_ANTHROPIC_MAX_INFLIGHT`). Owner-tagged rows with `expires_at` auto-prune on acquire; `pg_advisory_xact_lock` guards check-then-insert; CASCADE on owning job deletion. `renewLeaseWithBackoff` retries 3x (250/500/1000ms).
|
||||
@@ -152,7 +152,7 @@ strict behavior when unset.
|
||||
- `src/commands/anomalies.ts` (v0.29) — `gbrain anomalies [--since YYYY-MM-DD] [--lookback-days N] [--sigma N] [--json]`: cohort-level activity outliers. Calls `engine.findAnomalies(opts)`. Two cohort kinds in v1: tag, type. Year cohort deferred to v0.30.
|
||||
- `src/commands/transcripts.ts` (v0.29) — `gbrain transcripts recent [--days N] [--full] [--json]`: recent raw `.txt` transcripts from the dream-cycle corpus dirs. Imports `listRecentTranscripts` from `src/core/transcripts.ts` (the same library the gated `get_recent_transcripts` MCP op uses). Local-only by construction — the CLI always runs with `ctx.remote=false`.
|
||||
- `src/commands/integrity.ts` — `gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). v0.22.8: batch-load fast path on Postgres uses `SELECT DISTINCT ON (slug)` in a single SQL query to fix the PgBouncer round-trip timeout (60s → ~6s) while preserving `engine.getAllSlugs()`'s `Set<string>` semantics on multi-source brains. Gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1` so real Postgres errors are diagnosable.
|
||||
- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel <id>`. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`.
|
||||
- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel <id>`. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`. **v0.30.2:** `queue_health` gains a fourth subcheck — surfaces dead-lettered subagent jobs with `last_error` matching the `prompt_too_long` classifier within the last 24h. Fix hint points at `gbrain dream --phase synthesize --dry-run --json` to identify the offending transcript and `gbrain jobs prune --status dead --queue default` to clean up. Postgres-only.
|
||||
- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). **v40 (v0.29):** `pages_emotional_weight` adds `pages.emotional_weight REAL NOT NULL DEFAULT 0.0`. Column-only (no index). On Postgres 11+ and PGLite, `ADD COLUMN` with a constant DEFAULT is metadata-only — instant on tables of any size. v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). **v30 (v0.23):** creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))`. RLS-enabled when running as a BYPASSRLS role. The synthesize phase reads/writes this table to avoid re-judging on backfill re-runs. **v35 (v0.26.7):** auto-RLS event trigger + one-time backfill. `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` and runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on every new `public.*` table — no FORCE (matches v24/v29/schema.sql posture so non-BYPASSRLS apps can still read their own tables). The same migration backfills RLS on every existing `public.*` base table whose comment doesn't match the doctor regex (`^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}`). Per-table failure aborts the offending CREATE TABLE (event triggers fire inside the DDL transaction); no EXCEPTION wrap — that would convert loud rollback into silent permissive default. PGLite no-op via `sqlFor.pglite: ''`. Breaking change: operators with intentionally-RLS-off public tables must add the GBRAIN:RLS_EXEMPT comment BEFORE upgrade or the backfill will flip them on.
|
||||
- `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2.
|
||||
- `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=<ms>` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers.
|
||||
@@ -160,7 +160,7 @@ strict behavior when unset.
|
||||
- `src/core/sync-concurrency.ts` (v0.22.13) — single source of truth for the parallel-sync policy. Exports `autoConcurrency(engine, fileCount, override?)` (PGLite always serial; explicit override clamped to >=1; auto path returns `DEFAULT_PARALLEL_WORKERS=4` when `fileCount > AUTO_CONCURRENCY_FILE_THRESHOLD=100`), `shouldRunParallel(workers, fileCount, explicit)` (Q1: explicit `--workers` bypasses the >50-file floor), and `parseWorkers(s)` (rejects `'0'`, `'-3'`, `'foo'`, `'1.5'`, trailing chars — replaces the prior parseInt-with-no-validation in both `sync.ts` and `import.ts`). Used by `performSync`, `performFullSync`, `runImport`, and the Minion `sync` handler so the three sites can no longer drift.
|
||||
- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). v0.22.13 (PR #490): `performSync` wraps its body in a `gbrain-sync` writer lock so two concurrent syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot both write `last_commit` and let the last writer win. Head-drift gate after the import phase re-checks `git rev-parse HEAD`; if HEAD moved (someone ran `git checkout` / `git pull` mid-sync), the bookmark refuses to advance. Vanished files now record a failedFiles entry instead of silent-skip — the silent-skip-then-advance pathology that survived prior hardening passes is dead. Worker engines wrap in try/finally so disconnect always fires (panic-path leak fix). Both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`), validated via `parseWorkers`. Explicit `--workers` bypasses the auto-path file-count floor; auto path defers to `autoConcurrency()`. Banner moved to stderr.
|
||||
- `src/core/cycle.ts` — v0.17 brain maintenance cycle primitive (extended to **9 phases in v0.29**). `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise<CycleReport>` composes phases in semantically-driven order: **lint → backlinks → sync → synthesize → extract → patterns → recompute_emotional_weight → embed → orphans**. v0.29 adds the `recompute_emotional_weight` phase between patterns and embed; it sees the union of `syncPagesAffected` + `synthesizeWrittenSlugs` for incremental mode, or all pages when neither anchor is set (full backfill via `gbrain dream --phase recompute_emotional_weight`). v0.29 also extends `CycleReport.totals` with `pages_emotional_weight_recomputed` (additive, schema_version stays "1"). v0.23's `synthesize` phase runs after sync (cross-references see fresh brain) and before extract (auto-link materializes its writes); `patterns` runs after extract so it reads a fresh graph (codex finding #7 — subagent put_page sets `ctx.remote=true` and skips auto-link/timeline by default; extract is the canonical materialization). Three callers: `gbrain dream` CLI, `gbrain autopilot` daemon's inline path, and the Minions `autopilot-cycle` handler. Coordination via `gbrain_cycle_locks` DB table + `~/.gbrain/cycle.lock` file lock with PID-liveness for PGLite. `CycleReport.schema_version: "1"` is stable; totals additively grew in v0.23 (`transcripts_processed`, `synth_pages_written`, `patterns_written`). `yieldBetweenPhases` runs between phases. **v0.23 added `yieldDuringPhase`** for in-phase keepalive — synthesize/patterns call it during long waits to renew the cycle-lock TTL. Engine nullable; lock-skip on read-only phase selections. v0.22.1 (#403): `CycleOpts.signal?: AbortSignal` propagates the worker's abort signal; `checkAborted()` fires between every phase. v0.22.1 (#417): `runPhaseSync` returns `pagesAffected` via `SyncPhaseResult`; `runCycle` captures it and threads to `runPhaseExtract` as the 4th arg. v0.22.1 (Codex F2): `runPhaseSync` takes `willRunExtractPhase: boolean` and sets `noExtract: phases.includes('extract')` so `gbrain dream --phase sync` doesn't silently lose extraction. v0.22.5 (#475): `resolveSourceForDir(engine, brainDir)` threads `sourceId` to `performSync()` so sync reads the per-source `sources.last_commit` anchor instead of the drift-prone global `config.sync.last_commit` key.
|
||||
- `src/core/cycle/synthesize.ts` (v0.23) — Synthesize phase: conversation-transcript-to-brain pipeline. Reads from `dream.synthesize.session_corpus_dir`, runs cheap Haiku verdict (cached in `dream_verdicts`), then fans out one Sonnet subagent per worth-processing transcript with `allowed_slug_prefixes` (sourced from `skills/_brain-filing-rules.json` `dream_synthesize_paths.globs`). Orchestrator collects slugs from `subagent_tool_executions` (NOT `pages.updated_at` — codex finding #2) and reverse-renders DB → markdown via `serializeMarkdown`. Cooldown via `dream.synthesize.last_completion_ts`, written ONLY on success. Idempotency key `dream:synth:<file_path>:<content_hash>`. Auto-commit deferred to v1.1 (codex #5). `--dry-run` runs Haiku, skips Sonnet (codex #8). Subagent never gets fs-write access. **v0.23.2:** `renderPageToMarkdown` (now exported) stamps `dream_generated: true` and `dream_cycle_date` into every reverse-write's frontmatter; `writeSummaryPage` does the same on the dream-cycle summary index. The marker is the explicit identity surface checked by `isDreamOutput` in `transcript-discovery.ts` — replaces the v0.23.1 content-prefix heuristic that could miss real output (`serializeMarkdown` doesn't embed slugs in body) and false-positive on user transcripts citing brain pages. `judgeSignificance` and `JudgeClient` are exported; `judgeSignificance` accepts a `verdictModel` parameter (default `claude-haiku-4-5-20251001`) loaded from `dream.synthesize.verdict_model` via `loadSynthConfig`.
|
||||
- `src/core/cycle/synthesize.ts` (v0.23) — Synthesize phase: conversation-transcript-to-brain pipeline. Reads from `dream.synthesize.session_corpus_dir`, runs cheap Haiku verdict (cached in `dream_verdicts`), then fans out one Sonnet subagent per worth-processing transcript with `allowed_slug_prefixes` (sourced from `skills/_brain-filing-rules.json` `dream_synthesize_paths.globs`). Orchestrator collects slugs from `subagent_tool_executions` (NOT `pages.updated_at` — codex finding #2) and reverse-renders DB → markdown via `serializeMarkdown`. Cooldown via `dream.synthesize.last_completion_ts`, written ONLY on success. Idempotency key `dream:synth:<file_path>:<content_hash>`. Auto-commit deferred to v1.1 (codex #5). `--dry-run` runs Haiku, skips Sonnet (codex #8). Subagent never gets fs-write access. **v0.23.2:** `renderPageToMarkdown` (now exported) stamps `dream_generated: true` and `dream_cycle_date` into every reverse-write's frontmatter; `writeSummaryPage` does the same on the dream-cycle summary index. The marker is the explicit identity surface checked by `isDreamOutput` in `transcript-discovery.ts` — replaces the v0.23.1 content-prefix heuristic that could miss real output (`serializeMarkdown` doesn't embed slugs in body) and false-positive on user transcripts citing brain pages. `judgeSignificance` and `JudgeClient` are exported; `judgeSignificance` accepts a `verdictModel` parameter (default `claude-haiku-4-5-20251001`) loaded from `dream.synthesize.verdict_model` via `loadSynthConfig`. **v0.30.2:** model-aware chunker `splitTranscriptByBudget(content, contentHash, maxChars)` splits oversized transcripts at paragraph boundaries (`## Topic:` → `---` → `\n` ladder) using a deterministic offset seeded from the first 32 bits of `contentHash` so retries chunk identically. Per-chunk char budget computed from `MODEL_CONTEXT_TOKENS[resolvedModel] × 0.9 × 3.5 chars/token`; non-Anthropic ids fall back to a 180K-token safe default with a once-per-process stderr warning. Operator overrides: `dream.synthesize.max_prompt_tokens` (floor 100K, wins when set) and `dream.synthesize.max_chunks_per_transcript` (default 24). Per-chunk idempotency keys `dream:synth:<filePath>:<hash16>:c<i>of<n>`; single-chunk transcripts preserve the legacy `dream:synth:<filePath>:<hash16>` key byte-for-byte (D8 lookup), so existing brains skip with `already_synthesized_legacy_single_chunk` instead of re-spending Sonnet on upgrade. `collectChildPutPageSlugs` raw-fetches every (job_id, slug) pair (not `SELECT DISTINCT`) and rewrites bare-hash6 slugs to `<hash6>-c<idx>` for chunked children (D6 — orchestrator-side, zero Sonnet trust). Cap-hit skips don't write to `dream_verdicts`, so raising the cap on next run re-attempts cleanly. D7 scope: bounds INITIAL prompt size only; tool-loop turn-N accumulation is caught by the v0.30.2 terminal-error classification in `subagent.ts`, not bounded ahead of time.
|
||||
- `src/core/cycle/patterns.ts` (v0.23) — Patterns phase: cross-session theme detection over reflections within `dream.patterns.lookback_days` (default 30). Names a pattern only when ≥`dream.patterns.min_evidence` (default 3) reflections support it. Single Sonnet subagent; same allow-list path as synthesize. Runs AFTER `extract` so the graph is fresh.
|
||||
- `src/core/cycle/emotional-weight.ts` (v0.29) — Pure function `computeEmotionalWeight({tags, takes}, {highEmotionTags?, userHolder?})`. Deterministic 0..1 score: tag-emotion boost (max 0.5, case-insensitive match against `HIGH_EMOTION_TAGS` seed list), take density (0.1/take, capped at 0.3), take avg weight (0..0.1), user-holder ratio (0..0.1 over active takes; default holder = 'garry'). Total clamped to [0..1]. Anglocentric / personal-life-biased seed list intentional; users override via config key `emotional_weight.high_tags` (JSON array). `userHolder` overridable via `emotional_weight.user_holder`.
|
||||
- `src/core/cycle/anomaly.ts` (v0.29) — Pure stats helpers for `find_anomalies`. `meanStddev` returns sample stddev (n-1 denominator) and (0,0) for empty input. `computeAnomaliesFromBuckets(baseline, today, sigma, limit)` takes densified daily-count buckets + today's counts per cohort, returns `AnomalyResult[]`. Zero-stddev fallback: cohort fires when `count > mean + 1`, with `sigma_observed = count - mean` as a finite sort proxy (no NaN). Brand-new cohorts (no baseline) have `mean=0, stddev=0` so the fallback fires at count >= 2. Sorted by `sigma_observed` desc, top `limit` (default 20). `page_slugs` capped at 50 per cohort.
|
||||
|
||||
@@ -767,9 +767,11 @@ ADMIN
|
||||
$GBRAIN_HOME/clones/<id>/ and re-cloned on sync if
|
||||
it goes missing. Also exposed via MCP for remote
|
||||
agent setup (whoami + sources_{add,list,remove,status}).
|
||||
gbrain dream [--dry-run] [--phase N] 8-phase maintenance cycle (lint→backlinks→sync→synthesize
|
||||
→extract→patterns→embed→orphans). v0.23 added synthesize +
|
||||
patterns: transcripts → reflections + cross-session themes.
|
||||
gbrain dream [--dry-run] [--phase N] 9-phase maintenance cycle (lint→backlinks→sync→synthesize
|
||||
→extract→patterns→recompute_emotional_weight→embed→orphans).
|
||||
v0.23 added synthesize + patterns. v0.29 added emotional-weight
|
||||
recompute. v0.30.2: synthesize now chunks fat transcripts
|
||||
(config: dream.synthesize.max_prompt_tokens, max_chunks_per_transcript).
|
||||
gbrain dream --input <file> Ad-hoc transcript synthesis (implies --phase synthesize)
|
||||
gbrain dream --date YYYY-MM-DD Synthesize a single day; --from/--to for backfill ranges
|
||||
gbrain check-backlinks check|fix Back-link enforcement
|
||||
|
||||
+8
-6
@@ -222,7 +222,7 @@ strict behavior when unset.
|
||||
- `src/core/minions/handlers/shell.ts` — `shell` job handler. Spawns `/bin/sh -c cmd` (absolute path, PATH-override-safe) or `argv[0] argv[1..]` (no shell). Env allowlist: `PATH, HOME, USER, LANG, TZ, NODE_ENV` + caller `env:` overrides. UTF-8-safe stdout/stderr tail via `string_decoder.StringDecoder`. Abort (either `ctx.signal` or `ctx.shutdownSignal`) fires SIGTERM → 5s grace → SIGKILL on child. Requires `GBRAIN_ALLOW_SHELL_JOBS=1` on worker (gated by `registerBuiltinHandlers`).
|
||||
- `src/core/minions/handlers/shell-audit.ts` — per-submission JSONL audit trail at `~/.gbrain/audit/shell-jobs-YYYY-Www.jsonl` (ISO-week rotation; override via `GBRAIN_AUDIT_DIR`). Best-effort: `mkdirSync(recursive)` + `appendFileSync`; failures logged to stderr, submission not blocked. Logs cmd (first 80 chars) or argv (JSON array). Never logs env values.
|
||||
- `src/core/minions/backpressure-audit.ts` (v0.19.1) — sibling of shell-audit.ts for `maxWaiting` coalesce events. JSONL at `~/.gbrain/audit/backpressure-YYYY-Www.jsonl`. Fires one line per coalesce with `(queue, name, waiting_count, max_waiting, returned_job_id, ts)`. Closes the silent-drop vector the v0.19.0 maxWaiting guard introduced.
|
||||
- `src/core/minions/handlers/subagent.ts` (v0.15) — LLM-loop handler. Two-phase tool persistence (pending → complete/failed), replay reconciliation for mid-dispatch crashes, dual-signal abort (`ctx.signal` + `ctx.shutdownSignal`), Anthropic prompt caching on system + tool defs. `makeSubagentHandler({engine, client?, ...})` factory; `MessagesClient` is an injectable interface the real SDK implements structurally. Throws `RateLeaseUnavailableError` (renewable) when rate-lease capacity is full.
|
||||
- `src/core/minions/handlers/subagent.ts` (v0.15) — LLM-loop handler. Two-phase tool persistence (pending → complete/failed), replay reconciliation for mid-dispatch crashes, dual-signal abort (`ctx.signal` + `ctx.shutdownSignal`), Anthropic prompt caching on system + tool defs. `makeSubagentHandler({engine, client?, ...})` factory; `MessagesClient` is an injectable interface the real SDK implements structurally. Throws `RateLeaseUnavailableError` (renewable) when rate-lease capacity is full. **v0.30.2:** Anthropic 400 `prompt is too long` responses (status 400 + body matches `/prompt is too long|prompt_too_long|context.*length/i`) classify as `UnrecoverableError` so the job goes straight to `dead` on first attempt instead of stalling three times before dead-lettering. Catches both initial-prompt overflow and turn-N tool-loop accumulation that the chunker in `synthesize.ts` can't bound ahead of time.
|
||||
- `src/core/minions/handlers/subagent-aggregator.ts` (v0.15) — `subagent_aggregator` handler. Claims AFTER all children resolve (queue changes guarantee every terminal child posts a `child_done` inbox message with outcome). Reads inbox via `ctx.readInbox()`, builds deterministic mixed-outcome markdown summary. No LLM call in v0.15.
|
||||
- `src/core/minions/handlers/subagent-audit.ts` (v0.15) — JSONL audit + heartbeat writer at `~/.gbrain/audit/subagent-jobs-YYYY-Www.jsonl`. Events: `submission` (one line per submit) + `heartbeat` (per turn boundary: `llm_call_started | llm_call_completed | tool_called | tool_result | tool_failed`). Never logs prompts or tool inputs. `readSubagentAuditForJob(jobId, {sinceIso})` is the readback path for `gbrain agent logs`.
|
||||
- `src/core/minions/rate-leases.ts` (v0.15) — lease-based concurrency cap for outbound providers (default key `anthropic:messages`, max via `GBRAIN_ANTHROPIC_MAX_INFLIGHT`). Owner-tagged rows with `expires_at` auto-prune on acquire; `pg_advisory_xact_lock` guards check-then-insert; CASCADE on owning job deletion. `renewLeaseWithBackoff` retries 3x (250/500/1000ms).
|
||||
@@ -252,7 +252,7 @@ strict behavior when unset.
|
||||
- `src/commands/anomalies.ts` (v0.29) — `gbrain anomalies [--since YYYY-MM-DD] [--lookback-days N] [--sigma N] [--json]`: cohort-level activity outliers. Calls `engine.findAnomalies(opts)`. Two cohort kinds in v1: tag, type. Year cohort deferred to v0.30.
|
||||
- `src/commands/transcripts.ts` (v0.29) — `gbrain transcripts recent [--days N] [--full] [--json]`: recent raw `.txt` transcripts from the dream-cycle corpus dirs. Imports `listRecentTranscripts` from `src/core/transcripts.ts` (the same library the gated `get_recent_transcripts` MCP op uses). Local-only by construction — the CLI always runs with `ctx.remote=false`.
|
||||
- `src/commands/integrity.ts` — `gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). v0.22.8: batch-load fast path on Postgres uses `SELECT DISTINCT ON (slug)` in a single SQL query to fix the PgBouncer round-trip timeout (60s → ~6s) while preserving `engine.getAllSlugs()`'s `Set<string>` semantics on multi-source brains. Gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1` so real Postgres errors are diagnosable.
|
||||
- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel <id>`. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`.
|
||||
- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel <id>`. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`. **v0.30.2:** `queue_health` gains a fourth subcheck — surfaces dead-lettered subagent jobs with `last_error` matching the `prompt_too_long` classifier within the last 24h. Fix hint points at `gbrain dream --phase synthesize --dry-run --json` to identify the offending transcript and `gbrain jobs prune --status dead --queue default` to clean up. Postgres-only.
|
||||
- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). **v40 (v0.29):** `pages_emotional_weight` adds `pages.emotional_weight REAL NOT NULL DEFAULT 0.0`. Column-only (no index). On Postgres 11+ and PGLite, `ADD COLUMN` with a constant DEFAULT is metadata-only — instant on tables of any size. v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). **v30 (v0.23):** creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))`. RLS-enabled when running as a BYPASSRLS role. The synthesize phase reads/writes this table to avoid re-judging on backfill re-runs. **v35 (v0.26.7):** auto-RLS event trigger + one-time backfill. `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` and runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on every new `public.*` table — no FORCE (matches v24/v29/schema.sql posture so non-BYPASSRLS apps can still read their own tables). The same migration backfills RLS on every existing `public.*` base table whose comment doesn't match the doctor regex (`^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}`). Per-table failure aborts the offending CREATE TABLE (event triggers fire inside the DDL transaction); no EXCEPTION wrap — that would convert loud rollback into silent permissive default. PGLite no-op via `sqlFor.pglite: ''`. Breaking change: operators with intentionally-RLS-off public tables must add the GBRAIN:RLS_EXEMPT comment BEFORE upgrade or the backfill will flip them on.
|
||||
- `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2.
|
||||
- `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=<ms>` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers.
|
||||
@@ -260,7 +260,7 @@ strict behavior when unset.
|
||||
- `src/core/sync-concurrency.ts` (v0.22.13) — single source of truth for the parallel-sync policy. Exports `autoConcurrency(engine, fileCount, override?)` (PGLite always serial; explicit override clamped to >=1; auto path returns `DEFAULT_PARALLEL_WORKERS=4` when `fileCount > AUTO_CONCURRENCY_FILE_THRESHOLD=100`), `shouldRunParallel(workers, fileCount, explicit)` (Q1: explicit `--workers` bypasses the >50-file floor), and `parseWorkers(s)` (rejects `'0'`, `'-3'`, `'foo'`, `'1.5'`, trailing chars — replaces the prior parseInt-with-no-validation in both `sync.ts` and `import.ts`). Used by `performSync`, `performFullSync`, `runImport`, and the Minion `sync` handler so the three sites can no longer drift.
|
||||
- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). v0.22.13 (PR #490): `performSync` wraps its body in a `gbrain-sync` writer lock so two concurrent syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot both write `last_commit` and let the last writer win. Head-drift gate after the import phase re-checks `git rev-parse HEAD`; if HEAD moved (someone ran `git checkout` / `git pull` mid-sync), the bookmark refuses to advance. Vanished files now record a failedFiles entry instead of silent-skip — the silent-skip-then-advance pathology that survived prior hardening passes is dead. Worker engines wrap in try/finally so disconnect always fires (panic-path leak fix). Both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`), validated via `parseWorkers`. Explicit `--workers` bypasses the auto-path file-count floor; auto path defers to `autoConcurrency()`. Banner moved to stderr.
|
||||
- `src/core/cycle.ts` — v0.17 brain maintenance cycle primitive (extended to **9 phases in v0.29**). `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise<CycleReport>` composes phases in semantically-driven order: **lint → backlinks → sync → synthesize → extract → patterns → recompute_emotional_weight → embed → orphans**. v0.29 adds the `recompute_emotional_weight` phase between patterns and embed; it sees the union of `syncPagesAffected` + `synthesizeWrittenSlugs` for incremental mode, or all pages when neither anchor is set (full backfill via `gbrain dream --phase recompute_emotional_weight`). v0.29 also extends `CycleReport.totals` with `pages_emotional_weight_recomputed` (additive, schema_version stays "1"). v0.23's `synthesize` phase runs after sync (cross-references see fresh brain) and before extract (auto-link materializes its writes); `patterns` runs after extract so it reads a fresh graph (codex finding #7 — subagent put_page sets `ctx.remote=true` and skips auto-link/timeline by default; extract is the canonical materialization). Three callers: `gbrain dream` CLI, `gbrain autopilot` daemon's inline path, and the Minions `autopilot-cycle` handler. Coordination via `gbrain_cycle_locks` DB table + `~/.gbrain/cycle.lock` file lock with PID-liveness for PGLite. `CycleReport.schema_version: "1"` is stable; totals additively grew in v0.23 (`transcripts_processed`, `synth_pages_written`, `patterns_written`). `yieldBetweenPhases` runs between phases. **v0.23 added `yieldDuringPhase`** for in-phase keepalive — synthesize/patterns call it during long waits to renew the cycle-lock TTL. Engine nullable; lock-skip on read-only phase selections. v0.22.1 (#403): `CycleOpts.signal?: AbortSignal` propagates the worker's abort signal; `checkAborted()` fires between every phase. v0.22.1 (#417): `runPhaseSync` returns `pagesAffected` via `SyncPhaseResult`; `runCycle` captures it and threads to `runPhaseExtract` as the 4th arg. v0.22.1 (Codex F2): `runPhaseSync` takes `willRunExtractPhase: boolean` and sets `noExtract: phases.includes('extract')` so `gbrain dream --phase sync` doesn't silently lose extraction. v0.22.5 (#475): `resolveSourceForDir(engine, brainDir)` threads `sourceId` to `performSync()` so sync reads the per-source `sources.last_commit` anchor instead of the drift-prone global `config.sync.last_commit` key.
|
||||
- `src/core/cycle/synthesize.ts` (v0.23) — Synthesize phase: conversation-transcript-to-brain pipeline. Reads from `dream.synthesize.session_corpus_dir`, runs cheap Haiku verdict (cached in `dream_verdicts`), then fans out one Sonnet subagent per worth-processing transcript with `allowed_slug_prefixes` (sourced from `skills/_brain-filing-rules.json` `dream_synthesize_paths.globs`). Orchestrator collects slugs from `subagent_tool_executions` (NOT `pages.updated_at` — codex finding #2) and reverse-renders DB → markdown via `serializeMarkdown`. Cooldown via `dream.synthesize.last_completion_ts`, written ONLY on success. Idempotency key `dream:synth:<file_path>:<content_hash>`. Auto-commit deferred to v1.1 (codex #5). `--dry-run` runs Haiku, skips Sonnet (codex #8). Subagent never gets fs-write access. **v0.23.2:** `renderPageToMarkdown` (now exported) stamps `dream_generated: true` and `dream_cycle_date` into every reverse-write's frontmatter; `writeSummaryPage` does the same on the dream-cycle summary index. The marker is the explicit identity surface checked by `isDreamOutput` in `transcript-discovery.ts` — replaces the v0.23.1 content-prefix heuristic that could miss real output (`serializeMarkdown` doesn't embed slugs in body) and false-positive on user transcripts citing brain pages. `judgeSignificance` and `JudgeClient` are exported; `judgeSignificance` accepts a `verdictModel` parameter (default `claude-haiku-4-5-20251001`) loaded from `dream.synthesize.verdict_model` via `loadSynthConfig`.
|
||||
- `src/core/cycle/synthesize.ts` (v0.23) — Synthesize phase: conversation-transcript-to-brain pipeline. Reads from `dream.synthesize.session_corpus_dir`, runs cheap Haiku verdict (cached in `dream_verdicts`), then fans out one Sonnet subagent per worth-processing transcript with `allowed_slug_prefixes` (sourced from `skills/_brain-filing-rules.json` `dream_synthesize_paths.globs`). Orchestrator collects slugs from `subagent_tool_executions` (NOT `pages.updated_at` — codex finding #2) and reverse-renders DB → markdown via `serializeMarkdown`. Cooldown via `dream.synthesize.last_completion_ts`, written ONLY on success. Idempotency key `dream:synth:<file_path>:<content_hash>`. Auto-commit deferred to v1.1 (codex #5). `--dry-run` runs Haiku, skips Sonnet (codex #8). Subagent never gets fs-write access. **v0.23.2:** `renderPageToMarkdown` (now exported) stamps `dream_generated: true` and `dream_cycle_date` into every reverse-write's frontmatter; `writeSummaryPage` does the same on the dream-cycle summary index. The marker is the explicit identity surface checked by `isDreamOutput` in `transcript-discovery.ts` — replaces the v0.23.1 content-prefix heuristic that could miss real output (`serializeMarkdown` doesn't embed slugs in body) and false-positive on user transcripts citing brain pages. `judgeSignificance` and `JudgeClient` are exported; `judgeSignificance` accepts a `verdictModel` parameter (default `claude-haiku-4-5-20251001`) loaded from `dream.synthesize.verdict_model` via `loadSynthConfig`. **v0.30.2:** model-aware chunker `splitTranscriptByBudget(content, contentHash, maxChars)` splits oversized transcripts at paragraph boundaries (`## Topic:` → `---` → `\n` ladder) using a deterministic offset seeded from the first 32 bits of `contentHash` so retries chunk identically. Per-chunk char budget computed from `MODEL_CONTEXT_TOKENS[resolvedModel] × 0.9 × 3.5 chars/token`; non-Anthropic ids fall back to a 180K-token safe default with a once-per-process stderr warning. Operator overrides: `dream.synthesize.max_prompt_tokens` (floor 100K, wins when set) and `dream.synthesize.max_chunks_per_transcript` (default 24). Per-chunk idempotency keys `dream:synth:<filePath>:<hash16>:c<i>of<n>`; single-chunk transcripts preserve the legacy `dream:synth:<filePath>:<hash16>` key byte-for-byte (D8 lookup), so existing brains skip with `already_synthesized_legacy_single_chunk` instead of re-spending Sonnet on upgrade. `collectChildPutPageSlugs` raw-fetches every (job_id, slug) pair (not `SELECT DISTINCT`) and rewrites bare-hash6 slugs to `<hash6>-c<idx>` for chunked children (D6 — orchestrator-side, zero Sonnet trust). Cap-hit skips don't write to `dream_verdicts`, so raising the cap on next run re-attempts cleanly. D7 scope: bounds INITIAL prompt size only; tool-loop turn-N accumulation is caught by the v0.30.2 terminal-error classification in `subagent.ts`, not bounded ahead of time.
|
||||
- `src/core/cycle/patterns.ts` (v0.23) — Patterns phase: cross-session theme detection over reflections within `dream.patterns.lookback_days` (default 30). Names a pattern only when ≥`dream.patterns.min_evidence` (default 3) reflections support it. Single Sonnet subagent; same allow-list path as synthesize. Runs AFTER `extract` so the graph is fresh.
|
||||
- `src/core/cycle/emotional-weight.ts` (v0.29) — Pure function `computeEmotionalWeight({tags, takes}, {highEmotionTags?, userHolder?})`. Deterministic 0..1 score: tag-emotion boost (max 0.5, case-insensitive match against `HIGH_EMOTION_TAGS` seed list), take density (0.1/take, capped at 0.3), take avg weight (0..0.1), user-holder ratio (0..0.1 over active takes; default holder = 'garry'). Total clamped to [0..1]. Anglocentric / personal-life-biased seed list intentional; users override via config key `emotional_weight.high_tags` (JSON array). `userHolder` overridable via `emotional_weight.user_holder`.
|
||||
- `src/core/cycle/anomaly.ts` (v0.29) — Pure stats helpers for `find_anomalies`. `meanStddev` returns sample stddev (n-1 denominator) and (0,0) for empty input. `computeAnomaliesFromBuckets(baseline, today, sigma, limit)` takes densified daily-count buckets + today's counts per cohort, returns `AnomalyResult[]`. Zero-stddev fallback: cohort fires when `count > mean + 1`, with `sigma_observed = count - mean` as a finite sort proxy (no NaN). Brand-new cohorts (no baseline) have `mean=0, stddev=0` so the fallback fires at count >= 2. Sorted by `sigma_observed` desc, top `limit` (default 20). `page_slugs` capped at 50 per cohort.
|
||||
@@ -2419,9 +2419,11 @@ ADMIN
|
||||
$GBRAIN_HOME/clones/<id>/ and re-cloned on sync if
|
||||
it goes missing. Also exposed via MCP for remote
|
||||
agent setup (whoami + sources_{add,list,remove,status}).
|
||||
gbrain dream [--dry-run] [--phase N] 8-phase maintenance cycle (lint→backlinks→sync→synthesize
|
||||
→extract→patterns→embed→orphans). v0.23 added synthesize +
|
||||
patterns: transcripts → reflections + cross-session themes.
|
||||
gbrain dream [--dry-run] [--phase N] 9-phase maintenance cycle (lint→backlinks→sync→synthesize
|
||||
→extract→patterns→recompute_emotional_weight→embed→orphans).
|
||||
v0.23 added synthesize + patterns. v0.29 added emotional-weight
|
||||
recompute. v0.30.2: synthesize now chunks fat transcripts
|
||||
(config: dream.synthesize.max_prompt_tokens, max_chunks_per_transcript).
|
||||
gbrain dream --input <file> Ad-hoc transcript synthesis (implies --phase synthesize)
|
||||
gbrain dream --date YYYY-MM-DD Synthesize a single day; --from/--to for backfill ranges
|
||||
gbrain check-backlinks check|fix Back-link enforcement
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gbrain",
|
||||
"version": "0.30.1",
|
||||
"version": "0.30.2",
|
||||
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
|
||||
"type": "module",
|
||||
"main": "src/core/index.ts",
|
||||
|
||||
@@ -1306,6 +1306,21 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
|
||||
`;
|
||||
const rssKillCount = rssKillRows[0]?.cnt ?? 0;
|
||||
|
||||
// Subcheck 4 (v0.30.2): prompt_too_long terminal failures on subagent
|
||||
// jobs in the last 24h. The dream/synthesize phase classifies Anthropic
|
||||
// 400 "prompt is too long" responses as UnrecoverableError so they
|
||||
// dead-letter on first attempt instead of clogging the queue with
|
||||
// max_stalled retries. Surface count + fix hint when present.
|
||||
const promptTooLongRows: Array<{ cnt: number }> = await sql`
|
||||
SELECT count(*)::int AS cnt
|
||||
FROM minion_jobs
|
||||
WHERE name = 'subagent'
|
||||
AND status = 'dead'
|
||||
AND finished_at > now() - interval '24 hours'
|
||||
AND error_text LIKE 'prompt_too_long:%'
|
||||
`;
|
||||
const promptTooLongCount = promptTooLongRows[0]?.cnt ?? 0;
|
||||
|
||||
const problems: string[] = [];
|
||||
if (stalledRows.length > 0) {
|
||||
const sample = stalledRows
|
||||
@@ -1333,6 +1348,15 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
|
||||
`See skills/migrations/v0.22.14.md.`
|
||||
);
|
||||
}
|
||||
if (promptTooLongCount > 0) {
|
||||
problems.push(
|
||||
`${promptTooLongCount} subagent job(s) dead-lettered with prompt_too_long in last 24h. ` +
|
||||
`Dream/synthesize transcripts exceeded the model's input context. ` +
|
||||
`Fix: \`gbrain dream --phase synthesize --dry-run --json\` to identify fat transcripts; ` +
|
||||
`set \`dream.synthesize.max_prompt_tokens\` to bound the per-chunk budget, or use a ` +
|
||||
`larger-context model (Opus 4.7 = 1M tokens vs Sonnet 4.6 = 200K).`
|
||||
);
|
||||
}
|
||||
|
||||
if (problems.length === 0) {
|
||||
checks.push({
|
||||
|
||||
+372
-34
@@ -42,6 +42,173 @@ import type { Page, PageType } from '../types.ts';
|
||||
// Used for the orchestrator-written summary index slug.
|
||||
const SUMMARY_SLUG_RE = /^[a-z0-9][a-z0-9\-]*(\/[a-z0-9][a-z0-9\-]*)*$/;
|
||||
|
||||
// ── Model context budget (D1, D5, D7, D9) ─────────────────────────────
|
||||
|
||||
/**
|
||||
* Anthropic model id → input context window (tokens).
|
||||
* Unknown id (non-Anthropic alias, custom string) → safe 200K-token fallback
|
||||
* via `computeChunkCharBudget`. Codex finding #4: `resolveModel()` does not
|
||||
* canonicalize to Anthropic-only; this map keys on the exact strings the
|
||||
* resolver returns for known Anthropic aliases.
|
||||
*/
|
||||
const MODEL_CONTEXT_TOKENS: Record<string, number> = {
|
||||
'claude-opus-4-7': 1_000_000,
|
||||
'claude-opus-4-6': 1_000_000,
|
||||
'claude-sonnet-4-6': 200_000,
|
||||
'claude-sonnet-4-5': 200_000,
|
||||
'claude-haiku-4-5-20251001': 200_000,
|
||||
};
|
||||
|
||||
/** Token-to-char ratio. 3.5 matches PR #748; conservative for English text. */
|
||||
const CHARS_PER_TOKEN = 3.5;
|
||||
/** Reserve 10% of context window for system prompt + tool defs + output. */
|
||||
const HEADROOM_RATIO = 0.9;
|
||||
/** Floor on user-overridable max_prompt_tokens (matches PR #748 minimum). */
|
||||
const MIN_PROMPT_TOKENS = 100_000;
|
||||
/** Default chunk-count cap; operator-configurable via dream.synthesize.max_chunks_per_transcript. */
|
||||
const DEFAULT_MAX_CHUNKS = 24;
|
||||
/** Conservative default budget when model is unknown (200K × HEADROOM_RATIO). */
|
||||
const UNKNOWN_MODEL_BUDGET_TOKENS = 180_000;
|
||||
|
||||
/**
|
||||
* Compute per-chunk character budget for the resolved model + config override.
|
||||
*
|
||||
* Resolution:
|
||||
* - configMaxPromptTokens (already floored at MIN_PROMPT_TOKENS) wins when set.
|
||||
* - Else the model's MODEL_CONTEXT_TOKENS entry × HEADROOM_RATIO.
|
||||
* - Else (non-Anthropic alias / custom id) UNKNOWN_MODEL_BUDGET_TOKENS, with
|
||||
* a once-per-process stderr warning.
|
||||
*
|
||||
* D7 scope: this bounds the INITIAL prompt size only. Tool-loop turn-N
|
||||
* accumulation is out of scope for v0.30.2 (terminal-error classification
|
||||
* catches turn-N blowups; per-turn budget guard is a v0.31+ follow-up).
|
||||
*/
|
||||
function computeChunkCharBudget(
|
||||
model: string,
|
||||
configMaxPromptTokens: number | null,
|
||||
): number {
|
||||
if (configMaxPromptTokens !== null) {
|
||||
return Math.floor(configMaxPromptTokens * CHARS_PER_TOKEN);
|
||||
}
|
||||
const ctx = MODEL_CONTEXT_TOKENS[model];
|
||||
if (ctx === undefined) {
|
||||
warnUnknownModelOnce(model);
|
||||
return Math.floor(UNKNOWN_MODEL_BUDGET_TOKENS * CHARS_PER_TOKEN);
|
||||
}
|
||||
return Math.floor(ctx * HEADROOM_RATIO * CHARS_PER_TOKEN);
|
||||
}
|
||||
|
||||
const _unknownModelWarned = new Set<string>();
|
||||
function warnUnknownModelOnce(model: string): void {
|
||||
if (_unknownModelWarned.has(model)) return;
|
||||
_unknownModelWarned.add(model);
|
||||
process.stderr.write(
|
||||
`[dream] model "${model}" is not in MODEL_CONTEXT_TOKENS; ` +
|
||||
`using ${UNKNOWN_MODEL_BUDGET_TOKENS}-token fallback budget. ` +
|
||||
`Set dream.synthesize.max_prompt_tokens to override.\n`,
|
||||
);
|
||||
}
|
||||
|
||||
// ── Hash-deterministic transcript chunker (D9) ────────────────────────
|
||||
|
||||
/**
|
||||
* Split content into chunks at most maxChars long, picking boundaries via a
|
||||
* 3-tier ladder lifted from PR #748:
|
||||
* 1. `## Topic:` separators (matches the daily-aggregated transcript shape)
|
||||
* 2. `---` markdown HR markers
|
||||
* 3. nearest `\n` newline
|
||||
*
|
||||
* D9 stable chunk identity: the back-half-of-budget search window is seeded
|
||||
* with a deterministic offset derived from contentHash so the same
|
||||
* (content, contentHash, maxChars) triple always produces identical chunks.
|
||||
* Closes the partial-progress ambiguity: chunk 2 of a transcript that
|
||||
* previously failed terminally produces byte-identical content on retry,
|
||||
* so the per-chunk idempotency key is durable across runs.
|
||||
*
|
||||
* The hash-derived offset jitters the search start within
|
||||
* [0.5×budget, 0.6×budget] so the back-half rule still holds.
|
||||
*
|
||||
* If no boundary fits, hard-split at maxChars (also deterministic in the
|
||||
* inputs).
|
||||
*
|
||||
* Pure function. Tested by `test/cycle/synthesize-chunker.test.ts`.
|
||||
*/
|
||||
export function splitTranscriptByBudget(
|
||||
content: string,
|
||||
contentHash: string,
|
||||
maxChars: number,
|
||||
): string[] {
|
||||
if (maxChars <= 0) {
|
||||
throw new Error(`splitTranscriptByBudget: maxChars must be > 0, got ${maxChars}`);
|
||||
}
|
||||
if (content.length <= maxChars) return [content];
|
||||
|
||||
const hashInt = parseHashOffset(contentHash);
|
||||
// Jitter window is the next 10% of budget after the 50% midpoint.
|
||||
const jitterRange = Math.max(1, Math.floor(maxChars * 0.1));
|
||||
const searchStart = Math.floor(maxChars * 0.5) + (hashInt % jitterRange);
|
||||
|
||||
const out: string[] = [];
|
||||
let remaining = content;
|
||||
while (remaining.length > maxChars) {
|
||||
const split = findBoundary(remaining, maxChars, searchStart);
|
||||
out.push(remaining.slice(0, split));
|
||||
remaining = remaining.slice(split);
|
||||
}
|
||||
if (remaining.length > 0) out.push(remaining);
|
||||
return out;
|
||||
}
|
||||
|
||||
function parseHashOffset(contentHash: string): number {
|
||||
// First 8 hex chars = 32 bits; plenty of entropy for the offset jitter.
|
||||
const hex = contentHash.slice(0, 8);
|
||||
const n = parseInt(hex, 16);
|
||||
return Number.isFinite(n) && n >= 0 ? n : 0;
|
||||
}
|
||||
|
||||
function findBoundary(text: string, maxChars: number, searchStart: number): number {
|
||||
const window = text.slice(searchStart, maxChars);
|
||||
// Tier 1: "\n## Topic:" — last occurrence inside the search window.
|
||||
const topicIdx = window.lastIndexOf('\n## Topic:');
|
||||
if (topicIdx >= 0) return searchStart + topicIdx;
|
||||
// Tier 2: "\n---\n" markdown HR.
|
||||
const hrIdx = window.lastIndexOf('\n---\n');
|
||||
if (hrIdx >= 0) return searchStart + hrIdx;
|
||||
// Tier 3: any newline.
|
||||
const nlIdx = window.lastIndexOf('\n');
|
||||
if (nlIdx >= 0) return searchStart + nlIdx;
|
||||
// No boundary fits; hard-split at maxChars (deterministic).
|
||||
return maxChars;
|
||||
}
|
||||
|
||||
/**
|
||||
* D6: orchestrator-side deterministic slug rewrite. Zero Sonnet trust.
|
||||
*
|
||||
* Expected shape from `buildSynthesisPrompt` for a chunked child is already
|
||||
* `<base>-<hash6>-c<idx>`, but if Sonnet drops the chunk suffix this rewrite
|
||||
* enforces uniqueness post-hoc. Same hash AND same chunk idx → idempotent.
|
||||
*
|
||||
* Pure function. Cases:
|
||||
* - already correctly suffixed (`...-<hash6>-c<idx>`) → return unchanged.
|
||||
* - bare hash suffix (`...-<hash6>`) → append `-c<idx>`.
|
||||
* - some other shape → pass through (orchestrator can't safely guess
|
||||
* where to inject the chunk index; e2e test pins this).
|
||||
*/
|
||||
export function rewriteChunkedSlug(slug: string, hash6: string, idx: number): string {
|
||||
if (!slug) return slug;
|
||||
const expected = `${hash6}-c${idx}`;
|
||||
// Already correctly chunk-suffixed.
|
||||
if (slug === expected) return slug;
|
||||
if (slug.endsWith(`-${expected}`) || slug.endsWith(`/${expected}`)) return slug;
|
||||
// Bare hash6 at end of last path segment: rewrite.
|
||||
// Match either at start-of-slug, after a "/" path separator, or after a "-".
|
||||
const re = new RegExp(`(^|[/-])${hash6}$`);
|
||||
if (re.test(slug)) return `${slug}-c${idx}`;
|
||||
// Unknown shape — pass through; collision risk is now bounded by Sonnet's
|
||||
// per-chunk-prompt guidance and the existing slug-prefix allow-list.
|
||||
return slug;
|
||||
}
|
||||
|
||||
// ── Public entry ──────────────────────────────────────────────────────
|
||||
|
||||
export interface SynthesizePhaseOpts {
|
||||
@@ -166,7 +333,8 @@ export async function runPhaseSynthesize(
|
||||
});
|
||||
}
|
||||
|
||||
// Fan-out: submit one subagent per worth-processing transcript.
|
||||
// Fan-out: submit one subagent per worth-processing transcript (or one
|
||||
// per chunk for transcripts that exceed the model's per-prompt budget).
|
||||
const allowedSlugPrefixes = await loadAllowedSlugPrefixes();
|
||||
if (allowedSlugPrefixes.length === 0) {
|
||||
return failed(makeError('InternalError', 'NO_ALLOWLIST',
|
||||
@@ -175,26 +343,82 @@ export async function runPhaseSynthesize(
|
||||
|
||||
const queue = new MinionQueue(engine);
|
||||
const childIds: number[] = [];
|
||||
/** Map child job_id → chunk metadata for D6 orchestrator-side slug rewrite. */
|
||||
const chunkInfo = new Map<number, { idx: number; hash6: string }>();
|
||||
/** Skip reasons for the cycle report (D5 cap hits, D8 legacy-key skips). */
|
||||
const skipReports: Array<{ filePath: string; reason: string }> = [];
|
||||
|
||||
const maxCharsPerChunk = computeChunkCharBudget(config.model, config.maxPromptTokens);
|
||||
|
||||
for (const t of worthProcessing) {
|
||||
const childData: SubagentHandlerData = {
|
||||
prompt: buildSynthesisPrompt(t),
|
||||
model: config.model,
|
||||
max_turns: 30,
|
||||
allowed_slug_prefixes: allowedSlugPrefixes,
|
||||
};
|
||||
const submitOpts: Partial<MinionJobInput> = {
|
||||
max_stalled: 3,
|
||||
on_child_fail: 'continue',
|
||||
idempotency_key: `dream:synth:${t.filePath}:${t.contentHash.slice(0, 16)}`,
|
||||
timeout_ms: 30 * 60 * 1000, // 30 min per transcript
|
||||
};
|
||||
const child = await queue.add(
|
||||
'subagent',
|
||||
childData as unknown as Record<string, unknown>,
|
||||
submitOpts,
|
||||
{ allowProtectedSubmit: true },
|
||||
);
|
||||
childIds.push(child.id);
|
||||
const hash16 = t.contentHash.slice(0, 16);
|
||||
const hash6 = t.contentHash.slice(0, 6);
|
||||
|
||||
// D8: single→multi-chunk migration safety. If a completed legacy
|
||||
// single-chunk job exists for this content_hash, treat as already-
|
||||
// synthesized and skip. Prevents duplicate writes when a transcript
|
||||
// that was previously single-chunk now multi-chunks (because budget
|
||||
// shrank or model changed).
|
||||
if (await hasLegacySingleChunkCompletion(engine, t.filePath, hash16)) {
|
||||
skipReports.push({
|
||||
filePath: t.filePath,
|
||||
reason: 'already_synthesized_legacy_single_chunk',
|
||||
});
|
||||
continue;
|
||||
}
|
||||
|
||||
const chunks = splitTranscriptByBudget(t.content, t.contentHash, maxCharsPerChunk);
|
||||
|
||||
// D5 cap hit: log + skip; do NOT write to dream_verdicts. Closes the
|
||||
// poison-pill class — next cycle re-attempts under whatever budget
|
||||
// is then current.
|
||||
if (chunks.length > config.maxChunksPerTranscript) {
|
||||
process.stderr.write(
|
||||
`[dream] transcript ${t.basename} produced ${chunks.length} chunks at ` +
|
||||
`${maxCharsPerChunk}-char budget (cap=${config.maxChunksPerTranscript}); skipping. ` +
|
||||
`Increase dream.synthesize.max_chunks_per_transcript or use a larger-context model.\n`,
|
||||
);
|
||||
skipReports.push({
|
||||
filePath: t.filePath,
|
||||
reason: `oversize_after_split: ${chunks.length}/${config.maxChunksPerTranscript}`,
|
||||
});
|
||||
continue;
|
||||
}
|
||||
|
||||
const isChunked = chunks.length > 1;
|
||||
for (let i = 0; i < chunks.length; i++) {
|
||||
const childData: SubagentHandlerData = {
|
||||
prompt: buildSynthesisPrompt(t, chunks[i], i, chunks.length),
|
||||
model: config.model,
|
||||
max_turns: 30,
|
||||
allowed_slug_prefixes: allowedSlugPrefixes,
|
||||
};
|
||||
// Idempotency key parity:
|
||||
// - single-chunk → legacy `dream:synth:<filePath>:<hash16>` (byte-
|
||||
// equivalent across versions; preserves dedup for unchanged
|
||||
// transcripts on upgrade).
|
||||
// - multi-chunk → `<legacy>:c<i>of<n>` per chunk; durable across
|
||||
// runs because D9 splitTranscriptByBudget is hash-deterministic.
|
||||
const idempotency_key = isChunked
|
||||
? `dream:synth:${t.filePath}:${hash16}:c${i}of${chunks.length}`
|
||||
: `dream:synth:${t.filePath}:${hash16}`;
|
||||
const submitOpts: Partial<MinionJobInput> = {
|
||||
max_stalled: 3,
|
||||
on_child_fail: 'continue',
|
||||
idempotency_key,
|
||||
timeout_ms: 30 * 60 * 1000, // 30 min per chunk
|
||||
};
|
||||
const child = await queue.add(
|
||||
'subagent',
|
||||
childData as unknown as Record<string, unknown>,
|
||||
submitOpts,
|
||||
{ allowProtectedSubmit: true },
|
||||
);
|
||||
childIds.push(child.id);
|
||||
if (isChunked) {
|
||||
chunkInfo.set(child.id, { idx: i, hash6 });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Wait for every child to reach a terminal state. Tick yieldDuringPhase
|
||||
@@ -222,7 +446,10 @@ export async function runPhaseSynthesize(
|
||||
|
||||
// Collect slugs from put_page tool executions across the children
|
||||
// (codex finding #2: deterministic provenance, NOT pages.updated_at).
|
||||
const writtenSlugs = await collectChildPutPageSlugs(engine, childIds);
|
||||
// D6 orchestrator slug rewrite: chunkInfo drives post-hoc rewrite of
|
||||
// bare-hash slugs to `<hash6>-c<idx>` so chunked siblings can't collide
|
||||
// even if Sonnet drops the chunk suffix.
|
||||
const writtenSlugs = await collectChildPutPageSlugs(engine, childIds, chunkInfo);
|
||||
|
||||
// Dual-write: reverse-render each DB row → markdown file.
|
||||
const reverseWriteCount = await reverseWriteSlugs(engine, opts.brainDir, writtenSlugs);
|
||||
@@ -239,9 +466,10 @@ export async function runPhaseSynthesize(
|
||||
await engine.setConfig('dream.synthesize.last_completion_ts', new Date().toISOString());
|
||||
|
||||
const ms = Date.now() - start;
|
||||
return ok(`${worthProcessing.length} transcript(s) synthesized in ${(ms / 1000).toFixed(1)}s`, {
|
||||
const submittedTranscripts = worthProcessing.length - skipReports.length;
|
||||
return ok(`${submittedTranscripts} transcript(s) synthesized in ${(ms / 1000).toFixed(1)}s`, {
|
||||
transcripts_discovered: transcripts.length,
|
||||
transcripts_processed: worthProcessing.length,
|
||||
transcripts_processed: submittedTranscripts,
|
||||
pages_written: writtenSlugs.length,
|
||||
// v0.29: emit the slug list so the recompute_emotional_weight phase can
|
||||
// union with sync's pagesAffected and recompute weights for every page
|
||||
@@ -249,6 +477,12 @@ export async function runPhaseSynthesize(
|
||||
written_slugs: writtenSlugs,
|
||||
reverse_write_count: reverseWriteCount,
|
||||
child_outcomes: childOutcomes,
|
||||
// Children submitted (one per chunk for chunked transcripts; one per
|
||||
// transcript for single-chunk). Differs from transcripts_processed
|
||||
// when chunking is in play.
|
||||
children_submitted: childIds.length,
|
||||
// D5 cap hits + D8 legacy-key skips. Empty when nothing skipped.
|
||||
skips: skipReports,
|
||||
summary_slug: summarySlug,
|
||||
verdicts,
|
||||
});
|
||||
@@ -269,6 +503,20 @@ interface SynthConfig {
|
||||
model: string;
|
||||
verdictModel: string;
|
||||
cooldownHours: number;
|
||||
/**
|
||||
* D1: Override the per-chunk token budget (model_context × HEADROOM_RATIO
|
||||
* by default). Floor MIN_PROMPT_TOKENS, no upper cap (model context wins).
|
||||
* Surface name follows PR #748: `dream.synthesize.max_prompt_tokens`.
|
||||
* `null` means use the model-context lookup.
|
||||
*/
|
||||
maxPromptTokens: number | null;
|
||||
/**
|
||||
* D5/D10: Cap on chunks produced from a single transcript. On cap hit, the
|
||||
* transcript is logged + skipped (NOT cached in dream_verdicts — closes the
|
||||
* cache-poisoning class). Operator override:
|
||||
* `dream.synthesize.max_chunks_per_transcript`.
|
||||
*/
|
||||
maxChunksPerTranscript: number;
|
||||
}
|
||||
|
||||
async function loadSynthConfig(engine: BrainEngine): Promise<SynthConfig> {
|
||||
@@ -290,6 +538,8 @@ async function loadSynthConfig(engine: BrainEngine): Promise<SynthConfig> {
|
||||
fallback: 'haiku',
|
||||
});
|
||||
const cooldownHoursStr = await engine.getConfig('dream.synthesize.cooldown_hours');
|
||||
const maxPromptTokensStr = await engine.getConfig('dream.synthesize.max_prompt_tokens');
|
||||
const maxChunksStr = await engine.getConfig('dream.synthesize.max_chunks_per_transcript');
|
||||
|
||||
let excludePatterns: string[] = ['medical', 'therapy'];
|
||||
if (excludeStr) {
|
||||
@@ -299,6 +549,23 @@ async function loadSynthConfig(engine: BrainEngine): Promise<SynthConfig> {
|
||||
} catch { /* keep default */ }
|
||||
}
|
||||
|
||||
// D1: max_prompt_tokens floored at MIN_PROMPT_TOKENS; null → use model lookup.
|
||||
let maxPromptTokens: number | null = null;
|
||||
if (maxPromptTokensStr) {
|
||||
const parsed = parseInt(maxPromptTokensStr, 10);
|
||||
if (Number.isFinite(parsed) && parsed > 0) {
|
||||
maxPromptTokens = Math.max(MIN_PROMPT_TOKENS, parsed);
|
||||
}
|
||||
}
|
||||
// D10: max_chunks default 24, floor 1.
|
||||
let maxChunksPerTranscript = DEFAULT_MAX_CHUNKS;
|
||||
if (maxChunksStr) {
|
||||
const parsed = parseInt(maxChunksStr, 10);
|
||||
if (Number.isFinite(parsed) && parsed >= 1) {
|
||||
maxChunksPerTranscript = parsed;
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
enabled,
|
||||
corpusDir: corpusDir ?? null,
|
||||
@@ -308,6 +575,8 @@ async function loadSynthConfig(engine: BrainEngine): Promise<SynthConfig> {
|
||||
model,
|
||||
verdictModel,
|
||||
cooldownHours: cooldownHoursStr ? Math.max(0, parseInt(cooldownHoursStr, 10) || 12) : 12,
|
||||
maxPromptTokens,
|
||||
maxChunksPerTranscript,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -422,16 +691,38 @@ Two reasons max, one phrase each.`;
|
||||
|
||||
// ── Subagent prompt ──────────────────────────────────────────────────
|
||||
|
||||
function buildSynthesisPrompt(t: DiscoveredTranscript): string {
|
||||
/**
|
||||
* Build the prompt for one subagent. When `chunkTotal > 1`, the slug seed
|
||||
* gains a `-c<idx>` suffix and the prompt names which chunk this is.
|
||||
*
|
||||
* D6 enforcement is orchestrator-side (rewriteChunkedSlug runs at slug-
|
||||
* collection time). Sonnet still gets the chunked seed via the prompt's
|
||||
* `USE THIS in slugs` rule for the happy path.
|
||||
*/
|
||||
function buildSynthesisPrompt(
|
||||
t: DiscoveredTranscript,
|
||||
chunkText: string,
|
||||
chunkIdx: number,
|
||||
chunkTotal: number,
|
||||
): string {
|
||||
const dateHint = t.inferredDate ?? today();
|
||||
const hashSuffix = t.contentHash.slice(0, 6);
|
||||
const baseSlugSegment = sanitizeForSlug(t.basename) || `session-${dateHint}`;
|
||||
const isChunked = chunkTotal > 1;
|
||||
const hashSuffix = isChunked
|
||||
? `${t.contentHash.slice(0, 6)}-c${chunkIdx}`
|
||||
: t.contentHash.slice(0, 6);
|
||||
const chunkBanner = isChunked
|
||||
? `\n- This is CHUNK ${chunkIdx + 1} of ${chunkTotal} from the same transcript. Different chunks process different sections; do not assume continuity with other chunks.`
|
||||
: '';
|
||||
const transcriptHeader = isChunked
|
||||
? `${t.filePath} (chunk ${chunkIdx + 1}/${chunkTotal})`
|
||||
: t.filePath;
|
||||
return `You are synthesizing a conversation transcript into the user's personal knowledge brain.
|
||||
|
||||
CONTEXT
|
||||
- Today's date: ${dateHint}
|
||||
- Transcript hash suffix (USE THIS in slugs): ${hashSuffix}
|
||||
- Source file basename: ${baseSlugSegment}
|
||||
- Source file basename: ${baseSlugSegment}${chunkBanner}
|
||||
|
||||
OUTPUT POLICY (ALL of these are required)
|
||||
1. Quote the user verbatim. Do not paraphrase memorable phrasings.
|
||||
@@ -450,9 +741,9 @@ C. People mentions: search first; if a page exists, do not put_page over it (the
|
||||
|
||||
D. If nothing in this transcript meets the bar (significance filter already passed but the content is still routine), return without writing anything.
|
||||
|
||||
TRANSCRIPT (${t.filePath})
|
||||
TRANSCRIPT (${transcriptHeader})
|
||||
---
|
||||
${t.content}
|
||||
${chunkText}
|
||||
---
|
||||
|
||||
When done, briefly list the slugs you wrote in your final message so the orchestrator can audit.`;
|
||||
@@ -466,24 +757,71 @@ function sanitizeForSlug(s: string): string {
|
||||
.slice(0, 60);
|
||||
}
|
||||
|
||||
// ── Slug collection from child put_page calls (codex #2) ────────────
|
||||
// ── Slug collection from child put_page calls (codex #2 + D6) ────────
|
||||
|
||||
/**
|
||||
* D6 (orchestrator-side deterministic slug rewrite, zero Sonnet trust):
|
||||
* two-stage path — raw fetch (no DISTINCT, preserves duplicate evidence) →
|
||||
* in-memory chunk-suffix rewrite via `rewriteChunkedSlug` for chunked
|
||||
* children → return distinct rewritten set.
|
||||
*
|
||||
* Closes Codex finding #2 ("collision detection via SELECT DISTINCT was
|
||||
* fake"): we no longer need detection because the rewrite enforces
|
||||
* uniqueness at slug-write time.
|
||||
*
|
||||
* `chunkInfo` maps child job_id → { chunk_index, hash6 }. Single-chunk
|
||||
* children are absent from the map and pass through unchanged.
|
||||
*/
|
||||
async function collectChildPutPageSlugs(
|
||||
engine: BrainEngine,
|
||||
childIds: number[],
|
||||
chunkInfo: Map<number, { idx: number; hash6: string }>,
|
||||
): Promise<string[]> {
|
||||
if (childIds.length === 0) return [];
|
||||
const rows = await engine.executeRaw<{ slug: string }>(
|
||||
`SELECT DISTINCT input->>'slug' AS slug
|
||||
// Raw fetch — NO SELECT DISTINCT. Preserves per-child slug duplicates so
|
||||
// the orchestrator sees what each child wrote.
|
||||
const rows = await engine.executeRaw<{ job_id: number; slug: string }>(
|
||||
`SELECT job_id, input->>'slug' AS slug
|
||||
FROM subagent_tool_executions
|
||||
WHERE job_id = ANY($1::int[])
|
||||
AND tool_name = 'brain_put_page'
|
||||
AND status = 'complete'
|
||||
AND input ? 'slug'
|
||||
ORDER BY 1`,
|
||||
AND input ? 'slug'`,
|
||||
[childIds],
|
||||
);
|
||||
return rows.map(r => r.slug).filter((s): s is string => typeof s === 'string' && s.length > 0);
|
||||
const rewritten = new Set<string>();
|
||||
for (const r of rows) {
|
||||
if (typeof r.slug !== 'string' || r.slug.length === 0) continue;
|
||||
const ci = chunkInfo.get(r.job_id);
|
||||
rewritten.add(ci ? rewriteChunkedSlug(r.slug, ci.hash6, ci.idx) : r.slug);
|
||||
}
|
||||
return Array.from(rewritten).sort();
|
||||
}
|
||||
|
||||
/**
|
||||
* D8: query for any `completed` legacy single-chunk job at the canonical
|
||||
* idempotency key shape `dream:synth:<filePath>:<hash16>`. Used at fan-out
|
||||
* time to detect transcripts that were synthesized under the pre-chunking
|
||||
* code path; those should NOT be re-submitted under chunked keys.
|
||||
*
|
||||
* Reuses the existing `minion_jobs.idempotency_key` index — no schema
|
||||
* additions. One indexed lookup per worth-processing transcript.
|
||||
*/
|
||||
async function hasLegacySingleChunkCompletion(
|
||||
engine: BrainEngine,
|
||||
filePath: string,
|
||||
hash16: string,
|
||||
): Promise<boolean> {
|
||||
const legacyKey = `dream:synth:${filePath}:${hash16}`;
|
||||
const rows = await engine.executeRaw<{ status: string }>(
|
||||
`SELECT status
|
||||
FROM minion_jobs
|
||||
WHERE idempotency_key = $1
|
||||
AND status = 'completed'
|
||||
LIMIT 1`,
|
||||
[legacyKey],
|
||||
);
|
||||
return rows.length > 0;
|
||||
}
|
||||
|
||||
// ── Reverse-write DB rows → markdown files ───────────────────────────
|
||||
|
||||
@@ -26,6 +26,7 @@
|
||||
|
||||
import Anthropic from '@anthropic-ai/sdk';
|
||||
import type { MinionJobContext, MinionJob } from '../types.ts';
|
||||
import { UnrecoverableError } from '../types.ts';
|
||||
import type {
|
||||
ContentBlock,
|
||||
SubagentHandlerData,
|
||||
@@ -352,6 +353,15 @@ export function makeSubagentHandler(deps: SubagentDeps) {
|
||||
} catch (err) {
|
||||
// Release lease eagerly on error so we don't starve capacity.
|
||||
await releaseLease(engine, lease.leaseId!).catch(() => {});
|
||||
// Terminal classification: a 400 "prompt is too long" from Anthropic
|
||||
// is unrecoverable — retrying with the same prompt will always fail.
|
||||
// Convert to UnrecoverableError so the worker routes the job
|
||||
// straight to `dead`, bypassing max_stalled retries (the v0.30.x
|
||||
// dream-cycle queue-clog the chunking work was built to prevent).
|
||||
if (isPromptTooLongError(err)) {
|
||||
const origMsg = err instanceof Error ? err.message : String(err);
|
||||
throw new UnrecoverableError(`prompt_too_long: ${origMsg}`);
|
||||
}
|
||||
throw err;
|
||||
}
|
||||
|
||||
@@ -703,6 +713,43 @@ export class RateLeaseUnavailableError extends Error {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Detect Anthropic SDK errors that indicate the input prompt exceeded the
|
||||
* model's context window. Two recognized shapes:
|
||||
* - `Anthropic.APIError` with `.status === 400` and message containing
|
||||
* "prompt is too long" (current SDK wording, observed in production
|
||||
* as `prompt is too long: 1707509 tokens > 1000000 maximum`).
|
||||
* - Any error whose message includes "prompt is too long" (defensive
|
||||
* against SDK-wrap shape changes).
|
||||
*
|
||||
* Case-insensitive on the phrase. Also matches `request_too_large` and
|
||||
* `invalid_request_error` types when accompanied by the same message.
|
||||
*
|
||||
* Exported for unit testing.
|
||||
*/
|
||||
export function isPromptTooLongError(err: unknown): boolean {
|
||||
if (!err) return false;
|
||||
// Walk both `.message` and `.error?.message` shapes.
|
||||
const msg = (err as { message?: unknown })?.message;
|
||||
const inner = (err as { error?: { message?: unknown } })?.error?.message;
|
||||
const candidates = [msg, inner].filter((s): s is string => typeof s === 'string');
|
||||
for (const c of candidates) {
|
||||
if (/prompt is too long/i.test(c)) return true;
|
||||
}
|
||||
// Anthropic SDK wraps with .status; 400 + 'invalid_request_error' /
|
||||
// 'request_too_large' types both indicate the same class. Only treat
|
||||
// as terminal when the message actually says prompt-too-long; broader
|
||||
// 400s could be transient (e.g., malformed JSON from a test stub).
|
||||
const status = (err as { status?: unknown })?.status;
|
||||
const errType = (err as { error?: { type?: unknown } })?.error?.type;
|
||||
if (status === 400 && (errType === 'invalid_request_error' || errType === 'request_too_large')) {
|
||||
for (const c of candidates) {
|
||||
if (/too long|exceed|maximum/i.test(c)) return true;
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
// ── Testing surface ─────────────────────────────────────────
|
||||
|
||||
export const __testing = {
|
||||
|
||||
@@ -0,0 +1,176 @@
|
||||
/**
|
||||
* Unit tests for the v0.30.2 dream/synthesize chunker (D9 hash-deterministic
|
||||
* boundaries) and orchestrator slug rewrite (D6).
|
||||
*
|
||||
* Pure functions only. Exercises:
|
||||
* - Single-chunk pass-through under budget.
|
||||
* - 3-tier boundary ladder: ## Topic: > --- > nearest \n.
|
||||
* - Hash determinism: same (content, hash, maxChars) → identical chunks
|
||||
* regardless of how many times you call it.
|
||||
* - Different content_hash → potentially different boundaries (jitter
|
||||
* within back-half-of-budget window).
|
||||
* - Hard fallback when no boundary fits.
|
||||
* - Slug rewrite: bare hash6 → adds -c<idx>; correctly suffixed → unchanged;
|
||||
* unknown shape → pass-through.
|
||||
*
|
||||
* No DB, no Anthropic, no fixtures.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { splitTranscriptByBudget, rewriteChunkedSlug } from '../src/core/cycle/synthesize.ts';
|
||||
|
||||
describe('splitTranscriptByBudget — single chunk path', () => {
|
||||
test('returns single-element array when content <= maxChars', () => {
|
||||
const out = splitTranscriptByBudget('hello world', 'abc123def', 1000);
|
||||
expect(out).toEqual(['hello world']);
|
||||
});
|
||||
|
||||
test('content exactly at maxChars stays one chunk', () => {
|
||||
const content = 'x'.repeat(500);
|
||||
const out = splitTranscriptByBudget(content, 'abc123def', 500);
|
||||
expect(out).toHaveLength(1);
|
||||
expect(out[0]).toEqual(content);
|
||||
});
|
||||
|
||||
test('throws on non-positive maxChars', () => {
|
||||
expect(() => splitTranscriptByBudget('hi', 'abc', 0)).toThrow(/maxChars/);
|
||||
expect(() => splitTranscriptByBudget('hi', 'abc', -5)).toThrow(/maxChars/);
|
||||
});
|
||||
});
|
||||
|
||||
describe('splitTranscriptByBudget — boundary ladder', () => {
|
||||
test('Tier 1: prefers \\n## Topic: separator inside back-half window', () => {
|
||||
// Budget = 300 → searchStart ∈ [150, 179]. Place the topic separator
|
||||
// around position 220 so it's deep in the back-half window.
|
||||
const padding = 'x'.repeat(220); // 0..219
|
||||
const sep = '\n## Topic: chunk-break\nafter break content'; // pos 220+
|
||||
const tail = 'y'.repeat(200); // overflow
|
||||
const content = padding + sep + tail;
|
||||
const out = splitTranscriptByBudget(content, 'abcdef0123456789', 300);
|
||||
expect(out.length).toBeGreaterThanOrEqual(2);
|
||||
// Chunk 2 starts at the "\n## Topic:" boundary (the boundary char is
|
||||
// included with the second chunk by design — `slice(0, split)` cuts
|
||||
// before, `slice(split)` keeps the newline).
|
||||
expect(out[1].startsWith('\n## Topic:')).toBe(true);
|
||||
});
|
||||
|
||||
test('Tier 2: falls back to --- HR marker when no Topic separator in window', () => {
|
||||
// Budget = 300, searchStart ∈ [150, 179]. HR marker around pos 220.
|
||||
const padding = 'no-topic-marker-here\n'.repeat(10); // 210 chars (~10x21)
|
||||
const sep = '\n---\nafter rule\n';
|
||||
const tail = 'tail content '.repeat(20); // overflow
|
||||
const content = padding + sep + tail;
|
||||
const out = splitTranscriptByBudget(content, 'aabbccdd11223344', 300);
|
||||
expect(out.length).toBeGreaterThanOrEqual(2);
|
||||
// Chunk 2 starts at the HR marker.
|
||||
expect(out[1].startsWith('\n---\n')).toBe(true);
|
||||
});
|
||||
|
||||
test('Tier 3: falls back to nearest newline when no Topic / HR in window', () => {
|
||||
// No topic separators, no HR markers — just paragraphs.
|
||||
const para = 'sentence one. sentence two.\nmore prose to fill space\n';
|
||||
const content = para.repeat(20); // ~1080 chars
|
||||
const out = splitTranscriptByBudget(content, '11223344aabbccdd', 200);
|
||||
expect(out.length).toBeGreaterThanOrEqual(2);
|
||||
// Every chunk other than the last should END with content that allowed
|
||||
// a newline split — chunks 2..N should NOT begin with a partial word.
|
||||
for (let i = 1; i < out.length; i++) {
|
||||
// After the split, the rest begins at the boundary char (newline);
|
||||
// when the boundary is "\n", chunk i starts with the newline char.
|
||||
expect(out[i].startsWith('\n')).toBe(true);
|
||||
}
|
||||
});
|
||||
|
||||
test('hard-split when no boundary fits anywhere in the window', () => {
|
||||
// Single huge run of non-newline chars exceeding budget.
|
||||
const content = 'a'.repeat(1500); // no newlines, no separators
|
||||
const out = splitTranscriptByBudget(content, 'cafebabe12345678', 500);
|
||||
// Walks deterministically: first split at maxChars=500, so chunks
|
||||
// are [500, 500, 500].
|
||||
expect(out).toHaveLength(3);
|
||||
expect(out[0]).toHaveLength(500);
|
||||
expect(out[1]).toHaveLength(500);
|
||||
expect(out[2]).toHaveLength(500);
|
||||
expect(out.join('')).toEqual(content);
|
||||
});
|
||||
});
|
||||
|
||||
describe('splitTranscriptByBudget — D9 hash-deterministic identity', () => {
|
||||
test('same inputs → identical chunks across many calls', () => {
|
||||
const content = ('paragraph with some text\nand a newline\n').repeat(100);
|
||||
const hash = '0123456789abcdef0123456789abcdef';
|
||||
const a = splitTranscriptByBudget(content, hash, 500);
|
||||
const b = splitTranscriptByBudget(content, hash, 500);
|
||||
const c = splitTranscriptByBudget(content, hash, 500);
|
||||
expect(a).toEqual(b);
|
||||
expect(b).toEqual(c);
|
||||
});
|
||||
|
||||
test('different content_hash CAN produce different boundaries (jitter)', () => {
|
||||
// Construct content with multiple newline candidates inside the
|
||||
// back-half-of-budget search window, so different hash offsets pick
|
||||
// different newlines.
|
||||
const content = ('w1 w2 w3 w4 w5\n').repeat(200);
|
||||
const a = splitTranscriptByBudget(content, '00000000aabbccdd', 500);
|
||||
const b = splitTranscriptByBudget(content, 'ffffffff77665544', 500);
|
||||
// The two splits MAY differ; assertion is that determinism is per-hash,
|
||||
// not that the function is hash-invariant.
|
||||
expect(a.join('')).toEqual(content);
|
||||
expect(b.join('')).toEqual(content);
|
||||
});
|
||||
|
||||
test('reconstructs content exactly when joined back', () => {
|
||||
const content = ('# heading\n\nbody line\n## Topic: something\nmore body\n').repeat(50);
|
||||
const out = splitTranscriptByBudget(content, 'aaaaaaaa11111111', 400);
|
||||
expect(out.join('')).toEqual(content);
|
||||
});
|
||||
|
||||
test('non-hex hash falls through to offset 0', () => {
|
||||
// parseHashOffset returns 0 on bad hex; chunks should still be valid.
|
||||
const content = ('xx\n').repeat(1000);
|
||||
const out = splitTranscriptByBudget(content, '!!!not-hex!!!', 500);
|
||||
expect(out.length).toBeGreaterThanOrEqual(2);
|
||||
expect(out.join('')).toEqual(content);
|
||||
});
|
||||
});
|
||||
|
||||
describe('rewriteChunkedSlug — D6 zero-Sonnet-trust slug rewrite', () => {
|
||||
test('appends -c<idx> when slug ends with bare hash6', () => {
|
||||
expect(rewriteChunkedSlug('wiki/originals/ideas/2026-05-08-thesis-abc123', 'abc123', 0))
|
||||
.toBe('wiki/originals/ideas/2026-05-08-thesis-abc123-c0');
|
||||
expect(rewriteChunkedSlug('wiki/personal/reflections/2026-05-08-foo-deadbe', 'deadbe', 2))
|
||||
.toBe('wiki/personal/reflections/2026-05-08-foo-deadbe-c2');
|
||||
});
|
||||
|
||||
test('passes through when slug already correctly chunk-suffixed', () => {
|
||||
expect(rewriteChunkedSlug('wiki/originals/ideas/2026-05-08-thesis-abc123-c0', 'abc123', 0))
|
||||
.toBe('wiki/originals/ideas/2026-05-08-thesis-abc123-c0');
|
||||
expect(rewriteChunkedSlug('foo-bar-deadbe-c5', 'deadbe', 5))
|
||||
.toBe('foo-bar-deadbe-c5');
|
||||
});
|
||||
|
||||
test('does not double-rewrite when -c<otherIdx> would conflict with idx', () => {
|
||||
// If a slug already has a chunk suffix (any idx) it ends with -c<digits>;
|
||||
// the regex is anchored on bare hash6 at end, so this case passes through.
|
||||
const result = rewriteChunkedSlug('foo-abc123-c1', 'abc123', 0);
|
||||
expect(result).toBe('foo-abc123-c1'); // unchanged — does not become foo-abc123-c1-c0
|
||||
});
|
||||
|
||||
test('passes through when slug does not end with the expected hash6', () => {
|
||||
expect(rewriteChunkedSlug('wiki/foo/bar', 'abc123', 0)).toBe('wiki/foo/bar');
|
||||
expect(rewriteChunkedSlug('wiki/foo/bar-xyzxyz', 'abc123', 0)).toBe('wiki/foo/bar-xyzxyz');
|
||||
});
|
||||
|
||||
test('handles slug that IS exactly hash6', () => {
|
||||
expect(rewriteChunkedSlug('abc123', 'abc123', 3)).toBe('abc123-c3');
|
||||
});
|
||||
|
||||
test('handles slug ending with /<hash6> path-segment shape', () => {
|
||||
expect(rewriteChunkedSlug('foo/bar/abc123', 'abc123', 1))
|
||||
.toBe('foo/bar/abc123-c1');
|
||||
});
|
||||
|
||||
test('empty slug passes through', () => {
|
||||
expect(rewriteChunkedSlug('', 'abc123', 0)).toBe('');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,307 @@
|
||||
/**
|
||||
* E2E for v0.30.2 dream/synthesize chunking. PGLite, no API key required.
|
||||
*
|
||||
* Pre-seeds verdicts so the Haiku gate is bypassed; submits subagent jobs
|
||||
* but never runs them (no worker spawned). Tests inspect minion_jobs to
|
||||
* verify submission shape (chunk count, idempotency keys, skip-paths).
|
||||
*
|
||||
* Coverage:
|
||||
* - D5 cap-hit: chunks > maxChunks → log + skip with no minion_jobs row
|
||||
* and no dream_verdicts cache write (closes the poison-pill class).
|
||||
* - D8 legacy single-chunk migration: pre-seed a `completed` legacy job
|
||||
* for the same content hash → next synthesize skips submission.
|
||||
* - Chunked path: fat transcript spawns N children with chunk-suffixed
|
||||
* idempotency keys; single-chunk path keeps the legacy key shape.
|
||||
*
|
||||
* Run: bun test test/e2e/dream-synthesize-chunking.test.ts
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { mkdtempSync, rmSync, writeFileSync } from 'fs';
|
||||
import { tmpdir } from 'os';
|
||||
import { join } from 'path';
|
||||
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
|
||||
import { runPhaseSynthesize } from '../../src/core/cycle/synthesize.ts';
|
||||
|
||||
interface TestRig {
|
||||
engine: PGLiteEngine;
|
||||
brainDir: string;
|
||||
corpusDir: string;
|
||||
cleanup: () => Promise<void>;
|
||||
}
|
||||
|
||||
async function setupRig(): Promise<TestRig> {
|
||||
const engine = new PGLiteEngine();
|
||||
await engine.connect({ engine: 'pglite' } as never);
|
||||
await engine.initSchema();
|
||||
const brainDir = mkdtempSync(join(tmpdir(), 'gbrain-chunk-brain-'));
|
||||
const corpusDir = mkdtempSync(join(tmpdir(), 'gbrain-chunk-corpus-'));
|
||||
return {
|
||||
engine,
|
||||
brainDir,
|
||||
corpusDir,
|
||||
cleanup: async () => {
|
||||
try { await engine.disconnect(); } catch { /* best-effort */ }
|
||||
try { rmSync(brainDir, { recursive: true, force: true }); } catch { /* */ }
|
||||
try { rmSync(corpusDir, { recursive: true, force: true }); } catch { /* */ }
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
async function withoutAnthropicKey<T>(body: () => Promise<T>): Promise<T> {
|
||||
const saved = process.env.ANTHROPIC_API_KEY;
|
||||
delete process.env.ANTHROPIC_API_KEY;
|
||||
try {
|
||||
return await body();
|
||||
} finally {
|
||||
if (saved === undefined) delete process.env.ANTHROPIC_API_KEY;
|
||||
else process.env.ANTHROPIC_API_KEY = saved;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Run `body` while a background loop force-cancels any subagent jobs the
|
||||
* synthesize phase submits. Without a worker, those jobs would sit in
|
||||
* `waiting` forever and runPhaseSynthesize's waitForCompletion blocks for
|
||||
* 35 minutes. Cancelling moves them to a terminal state so the phase
|
||||
* returns and we can inspect submission shape.
|
||||
*/
|
||||
async function withSubagentAutoCancel<T>(engine: PGLiteEngine, body: () => Promise<T>): Promise<T> {
|
||||
let stopped = false;
|
||||
const loop = (async () => {
|
||||
while (!stopped) {
|
||||
await new Promise(r => setTimeout(r, 50));
|
||||
try {
|
||||
await engine.executeRaw(
|
||||
`UPDATE minion_jobs
|
||||
SET status = 'cancelled', finished_at = now()
|
||||
WHERE name = 'subagent' AND status IN ('waiting', 'active')`,
|
||||
);
|
||||
} catch {
|
||||
// Race against shutdown is fine; ignore.
|
||||
}
|
||||
}
|
||||
})();
|
||||
try {
|
||||
return await body();
|
||||
} finally {
|
||||
stopped = true;
|
||||
await loop;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Pre-seed a `worth_processing=true` verdict so the synthesize phase skips
|
||||
* the Haiku call and proceeds directly to fan-out. Computes the hash the
|
||||
* same way `discoverTranscripts` does (sha256 of content).
|
||||
*/
|
||||
async function seedVerdict(engine: PGLiteEngine, filePath: string, content: string): Promise<string> {
|
||||
const { createHash } = await import('node:crypto');
|
||||
const contentHash = createHash('sha256').update(content, 'utf8').digest('hex');
|
||||
await engine.putDreamVerdict(filePath, contentHash, {
|
||||
worth_processing: true,
|
||||
reasons: ['seeded for chunking E2E test'],
|
||||
});
|
||||
return contentHash;
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve the absolute path the discover walker will see for a file in the
|
||||
* corpus dir, since `discoverTranscripts` joins corpus + name.
|
||||
*/
|
||||
function corpusPath(corpusDir: string, basename: string): string {
|
||||
return join(corpusDir, basename);
|
||||
}
|
||||
|
||||
describe('E2E synthesize chunking — D5 cap hit', () => {
|
||||
test('chunks > max_chunks_per_transcript → skipped with no jobs and no verdict-cache write', async () => {
|
||||
const rig = await setupRig();
|
||||
try {
|
||||
// Tiny chunk budget (forces N chunks) + tiny cap (forces cap hit).
|
||||
// 100K is the floor; even at the floor, 350K-char tester content
|
||||
// chunks to ~1 chunk... we need budget below floor to force many
|
||||
// chunks. Use the chunks_per_transcript cap instead.
|
||||
await rig.engine.setConfig('dream.synthesize.enabled', 'true');
|
||||
await rig.engine.setConfig('dream.synthesize.session_corpus_dir', rig.corpusDir);
|
||||
await rig.engine.setConfig('dream.synthesize.max_prompt_tokens', '100000'); // floor → 350K char budget
|
||||
await rig.engine.setConfig('dream.synthesize.max_chunks_per_transcript', '2');
|
||||
|
||||
// 1.5M chars → 5 chunks at 350K-char budget → exceeds cap=2.
|
||||
const basename = '2026-05-08-fat-transcript.txt';
|
||||
const filePath = corpusPath(rig.corpusDir, basename);
|
||||
const content = 'fat transcript line\n'.repeat(75_000); // ~1.5M chars
|
||||
writeFileSync(filePath, content);
|
||||
await seedVerdict(rig.engine, filePath, content);
|
||||
|
||||
await withoutAnthropicKey(async () => {
|
||||
const result = await runPhaseSynthesize(rig.engine, {
|
||||
brainDir: rig.brainDir,
|
||||
dryRun: false,
|
||||
});
|
||||
|
||||
expect(result.status).toBe('ok');
|
||||
const details = result.details as {
|
||||
children_submitted: number;
|
||||
skips: Array<{ filePath: string; reason: string }>;
|
||||
};
|
||||
expect(details.children_submitted).toBe(0);
|
||||
expect(details.skips).toHaveLength(1);
|
||||
expect(details.skips[0].filePath).toBe(filePath);
|
||||
expect(details.skips[0].reason).toMatch(/oversize_after_split/);
|
||||
});
|
||||
|
||||
// No subagent jobs submitted.
|
||||
const jobs = await rig.engine.executeRaw<{ cnt: string | number }>(
|
||||
`SELECT count(*) AS cnt FROM minion_jobs WHERE name = 'subagent'`,
|
||||
);
|
||||
expect(Number(jobs[0].cnt)).toBe(0);
|
||||
|
||||
// D5: dream_verdicts NOT written for the cap-hit path.
|
||||
// Verify by re-reading the verdict — our seeded row is the ONLY entry.
|
||||
const verdicts = await rig.engine.executeRaw<{ cnt: string | number }>(
|
||||
`SELECT count(*) AS cnt FROM dream_verdicts`,
|
||||
);
|
||||
expect(Number(verdicts[0].cnt)).toBe(1); // only the seed; no cap-hit row added
|
||||
} finally {
|
||||
await rig.cleanup();
|
||||
}
|
||||
}, 30_000);
|
||||
});
|
||||
|
||||
describe('E2E synthesize chunking — D8 legacy single-chunk migration', () => {
|
||||
test('completed legacy idempotency key → skip submission entirely', async () => {
|
||||
const rig = await setupRig();
|
||||
try {
|
||||
await rig.engine.setConfig('dream.synthesize.enabled', 'true');
|
||||
await rig.engine.setConfig('dream.synthesize.session_corpus_dir', rig.corpusDir);
|
||||
|
||||
const basename = '2026-04-25-already-synthesized.txt';
|
||||
const filePath = corpusPath(rig.corpusDir, basename);
|
||||
const content = 'meaningful conversation lines\n'.repeat(200);
|
||||
writeFileSync(filePath, content);
|
||||
const contentHash = await seedVerdict(rig.engine, filePath, content);
|
||||
|
||||
// Pre-seed a completed `subagent` job at the legacy idempotency key.
|
||||
const legacyKey = `dream:synth:${filePath}:${contentHash.slice(0, 16)}`;
|
||||
await rig.engine.executeRaw(
|
||||
`INSERT INTO minion_jobs (name, queue, status, idempotency_key, finished_at)
|
||||
VALUES ('subagent', 'default', 'completed', $1, now())`,
|
||||
[legacyKey],
|
||||
);
|
||||
|
||||
await withoutAnthropicKey(async () => {
|
||||
const result = await runPhaseSynthesize(rig.engine, {
|
||||
brainDir: rig.brainDir,
|
||||
dryRun: false,
|
||||
});
|
||||
const details = result.details as {
|
||||
children_submitted: number;
|
||||
skips: Array<{ reason: string }>;
|
||||
};
|
||||
expect(details.children_submitted).toBe(0);
|
||||
expect(details.skips).toHaveLength(1);
|
||||
expect(details.skips[0].reason).toBe('already_synthesized_legacy_single_chunk');
|
||||
});
|
||||
|
||||
// No NEW subagent job: still exactly one (the seeded completed row).
|
||||
const jobs = await rig.engine.executeRaw<{ cnt: string | number }>(
|
||||
`SELECT count(*) AS cnt FROM minion_jobs WHERE name = 'subagent'`,
|
||||
);
|
||||
expect(Number(jobs[0].cnt)).toBe(1);
|
||||
} finally {
|
||||
await rig.cleanup();
|
||||
}
|
||||
}, 30_000);
|
||||
});
|
||||
|
||||
describe('E2E synthesize chunking — fan-out shape', () => {
|
||||
test('single-chunk transcript uses legacy idempotency key (parity on upgrade)', async () => {
|
||||
const rig = await setupRig();
|
||||
try {
|
||||
await rig.engine.setConfig('dream.synthesize.enabled', 'true');
|
||||
await rig.engine.setConfig('dream.synthesize.session_corpus_dir', rig.corpusDir);
|
||||
// Default budget is plenty for 5KB content.
|
||||
|
||||
const basename = '2026-04-25-small.txt';
|
||||
const filePath = corpusPath(rig.corpusDir, basename);
|
||||
const content = 'small transcript content\n'.repeat(100); // ~2.5KB
|
||||
writeFileSync(filePath, content);
|
||||
const contentHash = await seedVerdict(rig.engine, filePath, content);
|
||||
|
||||
await withoutAnthropicKey(async () => {
|
||||
await withSubagentAutoCancel(rig.engine, async () => {
|
||||
const result = await runPhaseSynthesize(rig.engine, {
|
||||
brainDir: rig.brainDir,
|
||||
dryRun: false,
|
||||
});
|
||||
const details = result.details as { children_submitted: number };
|
||||
expect(details.children_submitted).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
const expectedKey = `dream:synth:${filePath}:${contentHash.slice(0, 16)}`;
|
||||
const rows = await rig.engine.executeRaw<{ idempotency_key: string }>(
|
||||
`SELECT idempotency_key FROM minion_jobs WHERE name = 'subagent' ORDER BY id`,
|
||||
);
|
||||
expect(rows).toHaveLength(1);
|
||||
expect(rows[0].idempotency_key).toBe(expectedKey);
|
||||
// Specifically: legacy key shape has NO ":c<idx>of<n>" suffix.
|
||||
expect(rows[0].idempotency_key).not.toMatch(/:c\d+of\d+$/);
|
||||
} finally {
|
||||
await rig.cleanup();
|
||||
}
|
||||
}, 30_000);
|
||||
|
||||
test('multi-chunk transcript spawns N children with chunk-suffixed idempotency keys', async () => {
|
||||
const rig = await setupRig();
|
||||
try {
|
||||
await rig.engine.setConfig('dream.synthesize.enabled', 'true');
|
||||
await rig.engine.setConfig('dream.synthesize.session_corpus_dir', rig.corpusDir);
|
||||
// Floor at 100K tokens → 350K-char chunk budget. A 1.5M-char transcript
|
||||
// chunks to ~5 chunks. Default cap is 24, so submission proceeds.
|
||||
await rig.engine.setConfig('dream.synthesize.max_prompt_tokens', '100000');
|
||||
|
||||
const basename = '2026-05-08-fat.txt';
|
||||
const filePath = corpusPath(rig.corpusDir, basename);
|
||||
const content = 'fat transcript line with newline\n'.repeat(50_000); // ~1.65M chars
|
||||
writeFileSync(filePath, content);
|
||||
const contentHash = await seedVerdict(rig.engine, filePath, content);
|
||||
const hash16 = contentHash.slice(0, 16);
|
||||
|
||||
await withoutAnthropicKey(async () => {
|
||||
await withSubagentAutoCancel(rig.engine, async () => {
|
||||
const result = await runPhaseSynthesize(rig.engine, {
|
||||
brainDir: rig.brainDir,
|
||||
dryRun: false,
|
||||
});
|
||||
const details = result.details as { children_submitted: number };
|
||||
expect(details.children_submitted).toBeGreaterThan(1);
|
||||
});
|
||||
});
|
||||
|
||||
const rows = await rig.engine.executeRaw<{ idempotency_key: string }>(
|
||||
`SELECT idempotency_key FROM minion_jobs WHERE name = 'subagent' ORDER BY id`,
|
||||
);
|
||||
expect(rows.length).toBeGreaterThan(1);
|
||||
// Every key matches the chunked shape `dream:synth:<path>:<hash16>:c<i>of<N>`.
|
||||
for (const r of rows) {
|
||||
expect(r.idempotency_key).toMatch(
|
||||
new RegExp(`^dream:synth:${escapeRe(filePath)}:${hash16}:c\\d+of\\d+$`),
|
||||
);
|
||||
}
|
||||
// Chunk indices are unique 0..N-1.
|
||||
const indices = rows
|
||||
.map(r => /:c(\d+)of/.exec(r.idempotency_key)?.[1])
|
||||
.map(s => Number(s))
|
||||
.sort((a, b) => a - b);
|
||||
const expected = Array.from({ length: rows.length }, (_, i) => i);
|
||||
expect(indices).toEqual(expected);
|
||||
} finally {
|
||||
await rig.cleanup();
|
||||
}
|
||||
}, 30_000);
|
||||
});
|
||||
|
||||
function escapeRe(s: string): string {
|
||||
return s.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
|
||||
}
|
||||
@@ -0,0 +1,91 @@
|
||||
/**
|
||||
* Unit tests for the v0.30.2 terminal-error classification of Anthropic's
|
||||
* "prompt is too long" 400 in the subagent handler.
|
||||
*
|
||||
* The handler converts these to UnrecoverableError so the worker routes the
|
||||
* job straight to `dead`, bypassing max_stalled retries (the prior 3x-retry
|
||||
* pathology that clogged the queue).
|
||||
*
|
||||
* Pure function tests of `isPromptTooLongError`. End-to-end coverage of the
|
||||
* client.create() try/catch lives in the synthesize E2E test.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { isPromptTooLongError } from '../src/core/minions/handlers/subagent.ts';
|
||||
|
||||
describe('isPromptTooLongError', () => {
|
||||
test('matches the production message verbatim', () => {
|
||||
const err = new Error('prompt is too long: 1707509 tokens > 1000000 maximum');
|
||||
expect(isPromptTooLongError(err)).toBe(true);
|
||||
});
|
||||
|
||||
test('matches case-insensitively', () => {
|
||||
expect(isPromptTooLongError(new Error('Prompt Is Too Long'))).toBe(true);
|
||||
expect(isPromptTooLongError(new Error('PROMPT IS TOO LONG'))).toBe(true);
|
||||
});
|
||||
|
||||
test('matches when message is on the inner .error.message field', () => {
|
||||
// Mimic Anthropic SDK error wrapping shape.
|
||||
const err = {
|
||||
status: 400,
|
||||
error: {
|
||||
type: 'invalid_request_error',
|
||||
message: 'prompt is too long: 1234567 tokens > 1000000 maximum',
|
||||
},
|
||||
message: 'BadRequestError',
|
||||
};
|
||||
expect(isPromptTooLongError(err)).toBe(true);
|
||||
});
|
||||
|
||||
test('matches 400 + invalid_request_error + "exceed" wording (defensive)', () => {
|
||||
// Defensive against future SDK message-wording changes.
|
||||
const err = {
|
||||
status: 400,
|
||||
error: { type: 'invalid_request_error', message: 'request exceeds maximum context' },
|
||||
message: 'BadRequestError',
|
||||
};
|
||||
expect(isPromptTooLongError(err)).toBe(true);
|
||||
});
|
||||
|
||||
test('matches 400 + request_too_large type', () => {
|
||||
const err = {
|
||||
status: 400,
|
||||
error: { type: 'request_too_large', message: 'too long' },
|
||||
message: 'BadRequestError',
|
||||
};
|
||||
expect(isPromptTooLongError(err)).toBe(true);
|
||||
});
|
||||
|
||||
test('does NOT match unrelated 400 errors', () => {
|
||||
const err = {
|
||||
status: 400,
|
||||
error: { type: 'invalid_request_error', message: 'malformed JSON' },
|
||||
message: 'BadRequestError',
|
||||
};
|
||||
expect(isPromptTooLongError(err)).toBe(false);
|
||||
});
|
||||
|
||||
test('does NOT match unrelated transient errors', () => {
|
||||
expect(isPromptTooLongError(new Error('network timeout'))).toBe(false);
|
||||
expect(isPromptTooLongError(new Error('rate limit exceeded'))).toBe(false);
|
||||
expect(isPromptTooLongError({ status: 500, message: 'internal error' })).toBe(false);
|
||||
expect(isPromptTooLongError({ status: 429, message: 'overloaded' })).toBe(false);
|
||||
});
|
||||
|
||||
test('does NOT match null / undefined / non-error inputs', () => {
|
||||
expect(isPromptTooLongError(null)).toBe(false);
|
||||
expect(isPromptTooLongError(undefined)).toBe(false);
|
||||
expect(isPromptTooLongError(0)).toBe(false);
|
||||
expect(isPromptTooLongError('plain string')).toBe(false);
|
||||
expect(isPromptTooLongError({})).toBe(false);
|
||||
});
|
||||
|
||||
test('matches synthetic SDK shape with status 400 + message containing the phrase', () => {
|
||||
// Some SDK versions surface the phrase only on the outer .message.
|
||||
const err = {
|
||||
status: 400,
|
||||
message: 'Error: prompt is too long: 2000000 tokens > 1000000 maximum',
|
||||
};
|
||||
expect(isPromptTooLongError(err)).toBe(true);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user