gbrain

admin/gbrain

Fork 0

mirror of https://github.com/garrytan/gbrain.git synced 2026-08-16 18:02:30 +00:00

Author	SHA1	Message	Date
Garry TanandClaude Fable 5	5ef85ac9e3	v0.46.11.0 fix: five-issue operational wave — backlinks corruption, queue admission, junk paths, source scoping, type visibility (#4219 ) * fix(backlinks): frontmatter-safe fixer — canonical body offset, validate, atomic write, page lock The check-backlinks fixer could glue a generated timeline bullet above the frontmatter fence (naive split on '## Timeline' matched inside YAML and as a substring of longer headings; byte arithmetic then landed the insert at byte 0), breaking YAML parsing for the whole page. The write was also a bare writeFileSync: non-atomic, unvalidated, unlocked. fixBacklinkGaps is now a safety pipeline per target file: - pre-validate with parseMarkdown({validate:true}); fence/YAML-broken pages (YAML_PARSE / MISSING_CLOSE / NULL_BYTES) are skipped and reported, byte-identical. MISSING_OPEN is deliberately NOT a blocker: legacy pages without frontmatter have no fence to corrupt and stay fixable. - insertion anchors only on a real '## Timeline' heading line (CRLF-tolerant) at/after the canonical frontmatter body offset (new markdown.ts frontmatterBodyOffset, mirroring collectValidationErrors' fence semantics); first heading wins deterministically; near-misses ('### Timeline', '## Timeline (2026)') never match. - post-validate the candidate, then write via new src/core/atomic-write.ts (unique tmp sibling + fsync + mode preservation + on-disk re-validation before the atomic rename). Migrating the older per-module atomic-write copies is a filed follow-up. - each file is wrapped in withPageLock + try/catch: one bad page or write error isolates to a skip entry instead of killing the batch. fixBacklinkGaps returns {fixed, skipped[]}; BacklinksResult carries the skip report (additive), the CLI prints it, and the fix loop runs under a new backlinks.fix progress phase. Dead sourceDir var removed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(sync): reject malformed filenames at import — classifier reason, delete-lane carve-outs, doctor discovery Files literally named like markdown links polluted search because slugifySegment STRIPS brackets instead of rejecting them, minting plausible slugs for junk files. New SyncableReason 'malformed-path' (brackets + ASCII control chars; parens deliberately legal) rejects such filenames at every ingestion route: classifySync (incremental sync), isCollectibleForWalker (bulk import, both FS-walk and git fast-path), and a defense-in-depth check in importFromFile for direct callers. The subtle halves (both caught by outside-voice review): - Deletions and reconciles must still SEE malformed paths. manifest.deleted is filtered by isSyncable, and the full-sync reconcile requires isSyncable(source_path) — naively classifying junk as unsyncable would have orphaned previously-ingested rows forever. Both lanes now carry an explicit malformed-path carve-out, so deleting a junk file removes its DB row, editing one sweeps its stale row, and a full sync reconciles poison away even while the junk file still sits in the repo. Classifier ordering (strategy before malformed-path) keeps the reconcile strategy-safe; the #1433 metafile protection is untouched. - Malformed skips are INFORMATIONAL, never failures. importFromFile marks them skip_reason='malformed_path'; sync's failure classification keeps them out of failedFiles and the failure ledger (they can never gate bookmark advancement) and checkpoints them as done. Surfaced instead in the dry-run listing, a summary count, SyncResult.malformedSkipped, and a new doctor check (malformed_path_pages) that reports existing poisoned rows with the reconcile fix hint. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(schema): warn on alias/undeclared explicit frontmatter types at ingest + lint An explicit frontmatter type that is an ALIAS of a canonical pack type (or undeclared entirely) is stored literally and never re-normalized, so agents can silently file the same concept under different types and directories. gbrain can't stop agent-side filing decisions; it can make every non-canonical explicit type loud. - src/core/schema-pack/type-usage.ts: classifyStoredType(type, pack) → canonical \| alias_of (with the canonical type + its path_prefixes[0] filing directory) \| undeclared; sanitizeTypeForDisplay (control-char strip + length cap for terminal hygiene — type strings come from frontmatter); renderTypeWarningSummary (once-per-type lines). - importFromContent classifies at the typeExplicit site and returns an advisory ImportResult.type_warning — the type is still stored as-is, zero filing behavior change. - gbrain sync + gbrain import aggregate warnings once per distinct type per run (O(types), not O(files)) to stderr, and sync carries the counts on SyncResult.type_warnings so worker-driven syncs surface them in job results where daemon stderr is invisible. - Two data-plane schema lint rules (stored_type_is_alias, stored_type_undeclared) audit the EXISTING corpus via gbrain schema lint. - Config off-switch schema.type_warnings (default on) registered in the known-keys list; lint rules stay active regardless. Acknowledged limit: this warns at ingest and audit — it cannot prevent routing that happened in the agent's filing layer before import. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(scoping): close the unscoped-check/scoped-write bug class — writer, import, integrity + CI guard engine.getPage without opts matches a slug in ANY source while putPage defaults to 'default' — an existence check can succeed against source B and the paired write then targets a different row (duplicates, clobbers, spurious slug disambiguation; this class broke dream cycles for weeks). Fixes: - SlugRegistry + BrainWriter/WriteTxImpl thread a sourceId (default 'default') through every probe, read, write, addTimelineEntry, and the validator pass (which previously hardcoded 'default' regardless of the pages being validated). - integrity auto uses one writer PER SOURCE, and its resume progress is keyed by (source_id, slug) instead of slug alone — a resume no longer skips same-slug pages in other sources (legacy slug-only progress entries match the default source). - import-file reads flip from any-source-when-unset to the canonical { sourceId: sourceId ?? 'default' } (mirrors the writes' schema default). Caller audit: every importFromContent/importFromFile call site either passes sourceId explicitly or is default-source-intentional (brain-global synthesis artifacts, eval sandboxes) — none relied on the any-source fallback. - quarantine clear resolves the target source deterministically: new --source-id flag; without it, an ambiguous multi-source slug errors with the candidate list instead of acting on an arbitrary row. - link-extraction's exact-probe steps reuse the resolver's source scope (same posture as its basename index). - Both engines' unscoped getPage gains a deterministic tiebreak: ORDER BY (source_id = 'default') DESC, source_id ASC — default-first, then stable alpha (plain LIMIT 1 returned an arbitrary row). Guard: scripts/check-getpage-scoped-write.mjs (comment/string-aware span scanner) fails the build when a file contains BOTH an unscoped or ternary-undefined getPage AND a write-path call; opt-out marker gbrain-allow-unscoped-getpage for documented read-only first-match sites. Wired into verify CHECKS + guards-manifest + bad/good fixtures + unit test; the grandfathered allowlist is empty. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(minions): queue admission control — param-coalescing, waiting-TTL, quota + divergence alerting The submit-side half of the queue-divergence incident (drain-side pool starvation landed in v0.46.1.0): subagent intake ran ~25x drain with no backpressure, no expiry, and stats that couldn't even express the divergence. Admission-side only by design — claim fairness / lane scheduling is a filed follow-up, not this wave. New src/core/minions/admission.ts (per-name defaults tables, the handler-timeouts.ts pattern; config under the new 'minions.' prefix; GBRAIN_MINIONS_ADMISSION=0 kill-switch; fail-open to defaults with a once-per-process warning): - PARAM-COALESCING (default on for subagent): an identical parentless submit — same (name, queue, payload hash) — returns the newest matching WAITING row with coalesced:true instead of enqueuing a duplicate. __owner_client_id is INCLUDED in the hash so owner lanes never cross; parented submits never coalesce (aggregator bookkeeping); matches are age-bounded to ttl/2 so a fresh submit can't coalesce onto a nearly-expired row and silently die an hour later; waiting-only (a running job never suppresses a re-run). Audited via the existing backpressure JSONL. - WAITING-TTL (default 48h for subagent): the worker's 4th maintenance sweep cancels jobs still waiting past their TTL — through the canonical cancel path (new cancelJobs(ids, {reason}) batch variant: one transaction, descendant cancellation, child_done inbox messages, aggregator-parent resolution — a raw UPDATE would wedge parents in waiting-children), capped at 500/tick, oldest-first, with a reason error_text every alerting surface keys on (the single-id cancel path never wrote one). Warn-before-act: tick 1 counts + warns + sets minions.ttl_notice_shown, sweeping starts tick 2; gbrain upgrade prints the same one-shot notice on the interactive channel. - NAME-GLOBAL QUOTA (config-only, NO shipped default per operator decision): waiting count for a name across ALL queues (per-run private fanout queues can't dodge it) at/over minions.quota_max_waiting.<name> rejects with a typed QueueQuotaExceededError; approximate under concurrency, documented as a backstop. Dream submitters (synthesize/patterns) record the rejection as a phase skip — never a phase crash. Alerting: getStats by_type gains drained_completed/failed/dead/cancelled (true outflow keyed on finished_at, split so TTL-cancel storms can't masquerade as throughput), waiting_now, and oldest_waiting_minutes. jobs stats gains Drained/Waiting columns, a --json document, a per-type DIVERGENT-QUEUE scream (intake > GBRAIN_QUEUE_DIVERGENCE_RATIO x completed with waiting > GBRAIN_QUEUE_DIVERGENCE_MIN_WAITING) carrying the quota opt-in hint, and a 24h waiting-TTL cancellation report. doctor checkQueueHealth screams the same divergence + TTL activity. The advisor's stalled-jobs fix command now points at the real subcommand (jobs stats; 'jobs status' never existed). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: five-issue fix-wave reference updates — KEY_FILES, TODOS, invariants, flag registry - CLAUDE.md source-isolation invariant gains the unscoped-check/scoped-write corollary + the new guard's opt-out marker (llms bundles regenerated in the same commit per the CLAUDE.md-edit rule). - KEY_FILES.md: current-state entries for src/core/atomic-write.ts, src/core/minions/admission.ts, src/core/schema-pack/type-usage.ts, and scripts/check-getpage-scoped-write.mjs. - TODOS.md: six wave follow-ups (atomic-write migration + page-lock unification, splitFrontmatter relocation, admission/stats indexes, getPage type-boundary redesign, per-name claim fairness, per-queue divergence scoping) and the existing --max-pending entry now notes param-coalescing as the shipped payload-distinct dedupe primitive. - docs/progress-events.md: new backlinks.fix phase (additive). - Flag registry regenerated for quarantine --source-id; scrubbed a prose-bleed '--refresh' token that page-lock.ts's comment would have injected into check-backlinks' allowlist via the new import edge (the known generator pitfall — flags in comments must be spelled dash-less). - test/minions-admission.test.ts: env mutation moved onto withEnv (test isolation guard). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(minions): empty payloads never param-coalesce — no dedupe signal Full-suite fallout (budget-tracker.test.ts): scaffolding jobs submitted with literal {} data hashed identically and coalesced into one row, collapsing the tests' owner/child fixtures. Two no-param submits are far more likely distinct placeholder jobs than a runaway producer (which always carries a prompt), so add() now skips coalescing when the payload has no keys beyond __param_hash. Pinned in test/minions-admission.test.ts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: adversarial-review hardening — 15 confirmed findings across the wave A 35-agent adversarial verification pass (5 dimension finders, 2 independent skeptics per finding) confirmed 15 defects the suites missed; all fixed: Queue (the serious ones): - P0: waiting-TTL keyed on created_at cancelled jobs the SAME maintenance tick had just requeued/retried/promoted (those paths bump updated_at, not created_at) with a factually-false 'waited > Nh' reason. The sweep, the notice counts, and the coalesce age-bound all key on updated_at now — a state transition refreshes the TTL window; pure-waiting backlog semantics unchanged. - P1: the sweep's SELECT→cancel race could cancel a JUST-CLAIMED active job (claimer and sweep both target the oldest waiting rows). cancelJobs gains rootStatuses to re-check status atomically inside the cancel CTE; the TTL sweep passes ['waiting']. - P1: param-coalescing silently discarded a caller's idempotency_key (returned a row the key was never registered against → a later same-key submit ran the work twice). Producer-owned idempotency now disables param-coalescing outright — it is the stronger contract. - P2: a quota rejection mid-fanout left aggregators torn (children_ids never written). The fanout now cancels the whole tree and surfaces the quota message — all-or-nothing. - P2: the quota error message no longer includes the live global waiting count (it travels to remote MCP clients via submit_agent; counts stay on the error object and in local jobs stats). Scoping: - P1: reindex-code iterated ALL sources' code pages but re-imported with the CLI-level sourceId (undefined without a source flag) — under the new default-scoped reads that duplicated every non-default-source code page into 'default' and re-embedded it. fetchCodePages now SELECTs source_id and each page re-imports into its OWN source. - P2: importImageFile kept the variable-bound unscoped-read ternary the CI guard's inline heuristic can't see; flipped to the canonical sourceId-or-default read. Malformed paths: - P2: the FS-walk route only checked basenames, so a bracket-named DIRECTORY of clean files passed the walker (and its rows were unsweepable on non-git brains); descent now applies the malformed-segment check. - P2: doctor's malformed_path_pages SQL prefilter was brackets-only; a [[:cntrl:]] regex arm covers control-character-only paths. - P2: junk filenames are control-character carriers by definition — every place sync/import echoes one now routes through sanitizePathForDisplay (control chars → U+FFFD, length-capped) so a crafted filename can't inject terminal escapes. Filesystem: - P2: atomicWriteFileSync documented mode preservation but open(2)'s mode arg is umask-masked; an explicit chmod on the tmp file now actually preserves the target's mode across the rename (group-writable brain repos no longer lose their group-write bit on a backlinks fix). Test honesty: - P1: the divergence test's seed screamed under BOTH the correct (completed-based) and naive (cancellations-count-as-drain) metrics; reseeded so only the correct metric fires. - P2: the backlinks frontmatter-safety fixtures used quoted mid-line '## Timeline' strings that the ^-anchored regex ignores even without the bodyStart guard; replaced with line-start YAML-comment forms that genuinely distinguish guard-on from guard-off. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: ship pre-landing review fixes — guard regex, sanitization, TTL grace window Pre-landing review army (5 specialists + red team) findings, all applied: - check-getpage-scoped-write guard: catch the EXPANDED ternary (x ? { sourceId: x } : undefined) and {}/null false branches the shorthand-only regex missed; the sharpened probe immediately found one latent expanded-ternary site in operations.ts (safe pattern — write keys on the returned row's source_id — now carrying the documented opt-out). - Terminal hygiene: job names from the MCP-exposed submit surface are control/ANSI-stripped before echoing into jobs-stats + doctor screams. - Waiting-TTL warn-before-act: one ~30s tick between warn and sweep was a courtesy log line, not a warning. runWaitingTtlTick (admission.ts, testable per the lock-renewal-tick pattern) now stamps an ISO timestamp and holds a 1h grace window (GBRAIN_MINIONS_TTL_NOTICE_GRACE_MS seam) before the first sweep; the upgrade banner stamps the same clock; legacy 'true' flags sweep immediately. Worker + upgrade share countTtlExpiredWaiting. - TTL_REASON_PREFIX constant shared by the sweep reason + both LIKE consumers (jobs stats, doctor); patterns parameterized. - isQueueQuotaExceededError code-based guard replaces error-name string matching in agent fanout, cycle patterns, synthesize; submit_agent maps quota rejections to a structured rate_limited OperationError. - Empty-payload coalesce guard derives from PARAM_HASH_EXCLUDED_KEYS. - Red team: stored_type_is_alias lint hint pointed at a non-existent 'schema unify' subcommand — now points at the real unify-types job; reconcile prints a distinct count for malformed-path rows (their files usually still exist on disk — rename to re-import) instead of lumping them into 'source file was removed'. - integrity: one-time note when legacy (pre-source_id) resume-ledger entries force a partial re-scan. Tests: atomic-write direct unit (verify-abort, umask/mode preservation); cancelJobs reason/rootStatuses direct; idempotency-key-disables-coalesce; runWaitingTtlTick notice→grace→sweep + legacy flag + kill-switch; walker bracket-directory descent; sanitizePathForDisplay unit rows; backlinks multi-gap batch; quarantine clear ambiguity exit-2 + source scoping; doctor divergence/TTL/malformed_path_pages; engine-parity unscoped-getPage ORDER BY (DATABASE_URL-gated). TODOS: requeue surface, dream-path quota integration tests, coalesce concurrency e2e, canonical-json consolidation, reconcile quarantine option, cross-source clobber audit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: adversarial cross-model review fixes — bracket scoping, quota lock, writer scope Step-11 adversarial passes (Claude subagent + Codex exec + Codex structured review). Cross-model high-confidence finding, fixed: blanket bracket rejection over-captured — Next.js/Nuxt dynamic-route paths (app/[id]/…) in code-strategy lanes stopped importing AND their indexed rows became reconcile-deletable; bare-bracket markdown files imported by pre-gate releases would be hard-deleted by a routine post-upgrade full sync while their file sat on disk. Two-tier fix in core/sync.ts: ADMISSION rejects control chars anywhere + brackets on markdown paths only (hasMalformedPathSegment); the row-DELETING lanes (reconcile, modified-lane cleanup) act only on the poison signature isPoisonedPath (`](` or control chars). Walker now checks the RELATIVE path per file (basename checks can't see bracket directories) and descends bracket dirs for code lanes. Also fixed: - Name-global quota: pg_advisory_xact_lock('minion_quota:'\|\|name) serializes check+insert — concurrent distinct-payload submits could each observe capacity and overshoot without bound (quota is now exact, not a backstop; lock cost applies only to quota'd names). - cancelJobs stamps the reason on ROOT ids only — descendants carried a factually false waiting_ttl_expired text and inflated the LIKE-prefix stats the alerting surfaces count. - synthesize: a mid-transcript quota rejection now rolls back that transcript's fresh (non-coalesced) chunk submissions — previously the skip report said 'skipped' while earlier chunks drained and wrote partial pages. - BrainWriter: putRawData + addLink (both directions) now carry the writer's source scope — unscoped they defaulted to 'default'-source rows (both Codex passes flagged this P1). - page-lock: acquisition is O_EXCL ('wx') — the existsSync→write sequence let two stale-reclaimers both 'acquire' and lose one side's writes. - atomic-write: writeSync loops until all bytes land (legal short writes could atomically install truncated content) + best-effort parent-dir fsync after rename. - Copy-pasteable command hints gate untrusted names/types through strict token checks (safeConfigSegment/safeCliToken) — display sanitization keeps shell metacharacters, so injected names could ride into pasteable commands (jobs stats, doctor, schema lint hints). - admission isOffValue: case-insensitive + trimmed + 'no' — 'FALSE'/'Off' silently left the off-switch ON. - Walker/import/sync surface malformed-filename exclusions on BOTH collection routes (git + FS-walk), in directory imports, full-sync dry runs, and the totalChanges===0 up_to_date early return (a commit whose only changes are malformed files advances the anchor past them forever — now reported + counted in SyncResult). - submit_agent response carries coalesced:true so remote clients can detect param-coalescing instead of silently receiving an existing job. Rejected (by design, documented): default-on subagent param-coalescing semantics (user-approved; per-call + config opt-outs, response flag makes it detectable). Tests: markdown-scoped bracket rows + code-strategy framework paths + isPoisonedPath units (shape suite); bare-bracket row SURVIVES reconcile (serial suite); walker dir-descent pin retained via relative-path check. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: current-state reference updates for the review-fix batch KEY_FILES: SyncableReason union gains 'malformed-path' with the two-tier admission/destruction semantics; admission.ts entry reflects the grace-window warn-gate, exact quota lock, root-only cancel reasons, and the shared TTL/quota helpers; atomic-write entry reflects the full-write loop + parent-dir fsync. llms bundles regenerated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: raise module-size ceilings for wave growth + refresh structural manifest Nine ratcheted modules grew with the five-issue wave's feature code (admission control in queue/worker/jobs, malformed-path lanes in sync/import-file, divergence findings in doctor, ORDER BY in both engines, quota rollback in synthesize) — conscious TSV raises per the ratchet's documented growth path. structural-suites.tsv regenerated for the new test files. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * v0.46.11.0 fix: five-issue operational wave — backlinks corruption, queue admission, junk paths, source scoping, type visibility Version re-bumped from the user-pinned 0.46.10.0: master claimed that version mid-ship (queue collision; natural next off master). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: update project documentation for v0.46.11.0 Post-ship /document-release pass for the five-issue operational wave: - queue-operations-runbook: new DIVERGENT QUEUE section (jobs stats Drained/Waiting columns, divergence thresholds, admission remediations, GBRAIN_MINIONS_ADMISSION kill-switch) + doctor divergent-queue and waiting-TTL subcheck entries - minions-deployment: "jobs sit in waiting forever" nuanced for the waiting-TTL (48h default for subagent) - schema-packs: schema lint data-plane rules documented behind --with-db (per-source scoping exists at the rule layer, not yet CLI-threaded); ingest-time type warnings + schema.type_warnings off-switch - minion-orchestrator skill: coalesced-submit semantics (treat as success, monitor the matched id, never resubmit), waiting-TTL, rate_limited quota backoff + anti-pattern bullet - KEY_FILES: lint-rules entry 12->14 rules / 2->4 DB-aware, guards-manifest count 48->52 - schema-author skill + tutorial: DB-aware rule list and 14-rule surface - TODOS: filed P3 for threading --source-id into schema lint --with-db (+ jobs stats --json usage line, agent-run coalesce hint) - regenerated skills.lock.json, plugin/ tree, llms bundles Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-08-16 10:22:52 -07:00

Author

SHA1

Message

Date

Garry TanandClaude Fable 5

5ef85ac9e3

v0.46.11.0 fix: five-issue operational wave — backlinks corruption, queue admission, junk paths, source scoping, type visibility (#4219 )

* fix(backlinks): frontmatter-safe fixer — canonical body offset, validate, atomic write, page lock

The check-backlinks fixer could glue a generated timeline bullet above the
frontmatter fence (naive split on '## Timeline' matched inside YAML and as a
substring of longer headings; byte arithmetic then landed the insert at byte 0),
breaking YAML parsing for the whole page. The write was also a bare
writeFileSync: non-atomic, unvalidated, unlocked.

fixBacklinkGaps is now a safety pipeline per target file:
- pre-validate with parseMarkdown({validate:true}); fence/YAML-broken pages
  (YAML_PARSE / MISSING_CLOSE / NULL_BYTES) are skipped and reported,
  byte-identical. MISSING_OPEN is deliberately NOT a blocker: legacy pages
  without frontmatter have no fence to corrupt and stay fixable.
- insertion anchors only on a real '## Timeline' heading line (CRLF-tolerant)
  at/after the canonical frontmatter body offset (new markdown.ts
  frontmatterBodyOffset, mirroring collectValidationErrors' fence semantics);
  first heading wins deterministically; near-misses ('### Timeline',
  '## Timeline (2026)') never match.
- post-validate the candidate, then write via new src/core/atomic-write.ts
  (unique tmp sibling + fsync + mode preservation + on-disk re-validation
  before the atomic rename). Migrating the older per-module atomic-write
  copies is a filed follow-up.
- each file is wrapped in withPageLock + try/catch: one bad page or write
  error isolates to a skip entry instead of killing the batch.

fixBacklinkGaps returns {fixed, skipped[]}; BacklinksResult carries the skip
report (additive), the CLI prints it, and the fix loop runs under a new
backlinks.fix progress phase. Dead sourceDir var removed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(sync): reject malformed filenames at import — classifier reason, delete-lane carve-outs, doctor discovery

Files literally named like markdown links polluted search because
slugifySegment STRIPS brackets instead of rejecting them, minting
plausible slugs for junk files. New SyncableReason 'malformed-path'
(brackets + ASCII control chars; parens deliberately legal) rejects such
filenames at every ingestion route: classifySync (incremental sync),
isCollectibleForWalker (bulk import, both FS-walk and git fast-path),
and a defense-in-depth check in importFromFile for direct callers.

The subtle halves (both caught by outside-voice review):

- Deletions and reconciles must still SEE malformed paths. manifest.deleted
  is filtered by isSyncable, and the full-sync reconcile requires
  isSyncable(source_path) — naively classifying junk as unsyncable would
  have orphaned previously-ingested rows forever. Both lanes now carry an
  explicit malformed-path carve-out, so deleting a junk file removes its
  DB row, editing one sweeps its stale row, and a full sync reconciles
  poison away even while the junk file still sits in the repo. Classifier
  ordering (strategy before malformed-path) keeps the reconcile
  strategy-safe; the #1433 metafile protection is untouched.

- Malformed skips are INFORMATIONAL, never failures. importFromFile marks
  them skip_reason='malformed_path'; sync's failure classification keeps
  them out of failedFiles and the failure ledger (they can never gate
  bookmark advancement) and checkpoints them as done. Surfaced instead in
  the dry-run listing, a summary count, SyncResult.malformedSkipped, and a
  new doctor check (malformed_path_pages) that reports existing poisoned
  rows with the reconcile fix hint.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(schema): warn on alias/undeclared explicit frontmatter types at ingest + lint

An explicit frontmatter type that is an ALIAS of a canonical pack type (or
undeclared entirely) is stored literally and never re-normalized, so agents
can silently file the same concept under different types and directories.
gbrain can't stop agent-side filing decisions; it can make every
non-canonical explicit type loud.

- src/core/schema-pack/type-usage.ts: classifyStoredType(type, pack) →
  canonical | alias_of (with the canonical type + its path_prefixes[0]
  filing directory) | undeclared; sanitizeTypeForDisplay (control-char
  strip + length cap for terminal hygiene — type strings come from
  frontmatter); renderTypeWarningSummary (once-per-type lines).
- importFromContent classifies at the typeExplicit site and returns an
  advisory ImportResult.type_warning — the type is still stored as-is,
  zero filing behavior change.
- gbrain sync + gbrain import aggregate warnings once per distinct type per
  run (O(types), not O(files)) to stderr, and sync carries the counts on
  SyncResult.type_warnings so worker-driven syncs surface them in job
  results where daemon stderr is invisible.
- Two data-plane schema lint rules (stored_type_is_alias,
  stored_type_undeclared) audit the EXISTING corpus via gbrain schema lint.
- Config off-switch schema.type_warnings (default on) registered in the
  known-keys list; lint rules stay active regardless.

Acknowledged limit: this warns at ingest and audit — it cannot prevent
routing that happened in the agent's filing layer before import.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(scoping): close the unscoped-check/scoped-write bug class — writer, import, integrity + CI guard

engine.getPage without opts matches a slug in ANY source while putPage
defaults to 'default' — an existence check can succeed against source B
and the paired write then targets a different row (duplicates, clobbers,
spurious slug disambiguation; this class broke dream cycles for weeks).

Fixes:
- SlugRegistry + BrainWriter/WriteTxImpl thread a sourceId (default
  'default') through every probe, read, write, addTimelineEntry, and the
  validator pass (which previously hardcoded 'default' regardless of the
  pages being validated).
- integrity auto uses one writer PER SOURCE, and its resume progress is
  keyed by (source_id, slug) instead of slug alone — a resume no longer
  skips same-slug pages in other sources (legacy slug-only progress
  entries match the default source).
- import-file reads flip from any-source-when-unset to the canonical
  { sourceId: sourceId ?? 'default' } (mirrors the writes' schema
  default). Caller audit: every importFromContent/importFromFile call
  site either passes sourceId explicitly or is default-source-intentional
  (brain-global synthesis artifacts, eval sandboxes) — none relied on the
  any-source fallback.
- quarantine clear resolves the target source deterministically: new
  --source-id flag; without it, an ambiguous multi-source slug errors
  with the candidate list instead of acting on an arbitrary row.
- link-extraction's exact-probe steps reuse the resolver's source scope
  (same posture as its basename index).
- Both engines' unscoped getPage gains a deterministic tiebreak:
  ORDER BY (source_id = 'default') DESC, source_id ASC — default-first,
  then stable alpha (plain LIMIT 1 returned an arbitrary row).

Guard: scripts/check-getpage-scoped-write.mjs (comment/string-aware span
scanner) fails the build when a file contains BOTH an unscoped or
ternary-undefined getPage AND a write-path call; opt-out marker
gbrain-allow-unscoped-getpage for documented read-only first-match sites.
Wired into verify CHECKS + guards-manifest + bad/good fixtures + unit
test; the grandfathered allowlist is empty.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(minions): queue admission control — param-coalescing, waiting-TTL, quota + divergence alerting

The submit-side half of the queue-divergence incident (drain-side pool
starvation landed in v0.46.1.0): subagent intake ran ~25x drain with no
backpressure, no expiry, and stats that couldn't even express the
divergence. Admission-side only by design — claim fairness / lane
scheduling is a filed follow-up, not this wave.

New src/core/minions/admission.ts (per-name defaults tables, the
handler-timeouts.ts pattern; config under the new 'minions.' prefix;
GBRAIN_MINIONS_ADMISSION=0 kill-switch; fail-open to defaults with a
once-per-process warning):

- PARAM-COALESCING (default on for subagent): an identical parentless
  submit — same (name, queue, payload hash) — returns the newest matching
  WAITING row with coalesced:true instead of enqueuing a duplicate.
  __owner_client_id is INCLUDED in the hash so owner lanes never cross;
  parented submits never coalesce (aggregator bookkeeping); matches are
  age-bounded to ttl/2 so a fresh submit can't coalesce onto a
  nearly-expired row and silently die an hour later; waiting-only (a
  running job never suppresses a re-run). Audited via the existing
  backpressure JSONL.
- WAITING-TTL (default 48h for subagent): the worker's 4th maintenance
  sweep cancels jobs still waiting past their TTL — through the canonical
  cancel path (new cancelJobs(ids, {reason}) batch variant: one
  transaction, descendant cancellation, child_done inbox messages,
  aggregator-parent resolution — a raw UPDATE would wedge parents in
  waiting-children), capped at 500/tick, oldest-first, with a reason
  error_text every alerting surface keys on (the single-id cancel path
  never wrote one). Warn-before-act: tick 1 counts + warns + sets
  minions.ttl_notice_shown, sweeping starts tick 2; gbrain upgrade prints
  the same one-shot notice on the interactive channel.
- NAME-GLOBAL QUOTA (config-only, NO shipped default per operator
  decision): waiting count for a name across ALL queues (per-run private
  fanout queues can't dodge it) at/over minions.quota_max_waiting.<name>
  rejects with a typed QueueQuotaExceededError; approximate under
  concurrency, documented as a backstop. Dream submitters
  (synthesize/patterns) record the rejection as a phase skip — never a
  phase crash.

Alerting: getStats by_type gains drained_completed/failed/dead/cancelled
(true outflow keyed on finished_at, split so TTL-cancel storms can't
masquerade as throughput), waiting_now, and oldest_waiting_minutes.
jobs stats gains Drained/Waiting columns, a --json document, a per-type
DIVERGENT-QUEUE scream (intake > GBRAIN_QUEUE_DIVERGENCE_RATIO x
completed with waiting > GBRAIN_QUEUE_DIVERGENCE_MIN_WAITING) carrying
the quota opt-in hint, and a 24h waiting-TTL cancellation report. doctor
checkQueueHealth screams the same divergence + TTL activity. The
advisor's stalled-jobs fix command now points at the real subcommand
(jobs stats; 'jobs status' never existed).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: five-issue fix-wave reference updates — KEY_FILES, TODOS, invariants, flag registry

- CLAUDE.md source-isolation invariant gains the unscoped-check/scoped-write
  corollary + the new guard's opt-out marker (llms bundles regenerated in the
  same commit per the CLAUDE.md-edit rule).
- KEY_FILES.md: current-state entries for src/core/atomic-write.ts,
  src/core/minions/admission.ts, src/core/schema-pack/type-usage.ts, and
  scripts/check-getpage-scoped-write.mjs.
- TODOS.md: six wave follow-ups (atomic-write migration + page-lock
  unification, splitFrontmatter relocation, admission/stats indexes, getPage
  type-boundary redesign, per-name claim fairness, per-queue divergence
  scoping) and the existing --max-pending entry now notes param-coalescing as
  the shipped payload-distinct dedupe primitive.
- docs/progress-events.md: new backlinks.fix phase (additive).
- Flag registry regenerated for quarantine --source-id; scrubbed a
  prose-bleed '--refresh' token that page-lock.ts's comment would have
  injected into check-backlinks' allowlist via the new import edge (the
  known generator pitfall — flags in comments must be spelled dash-less).
- test/minions-admission.test.ts: env mutation moved onto withEnv (test
  isolation guard).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(minions): empty payloads never param-coalesce — no dedupe signal

Full-suite fallout (budget-tracker.test.ts): scaffolding jobs submitted with
literal {} data hashed identically and coalesced into one row, collapsing the
tests' owner/child fixtures. Two no-param submits are far more likely distinct
placeholder jobs than a runaway producer (which always carries a prompt), so
add() now skips coalescing when the payload has no keys beyond __param_hash.
Pinned in test/minions-admission.test.ts.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: adversarial-review hardening — 15 confirmed findings across the wave

A 35-agent adversarial verification pass (5 dimension finders, 2 independent
skeptics per finding) confirmed 15 defects the suites missed; all fixed:

Queue (the serious ones):
- P0: waiting-TTL keyed on created_at cancelled jobs the SAME maintenance
  tick had just requeued/retried/promoted (those paths bump updated_at, not
  created_at) with a factually-false 'waited > Nh' reason. The sweep, the
  notice counts, and the coalesce age-bound all key on updated_at now — a
  state transition refreshes the TTL window; pure-waiting backlog semantics
  unchanged.
- P1: the sweep's SELECT→cancel race could cancel a JUST-CLAIMED active job
  (claimer and sweep both target the oldest waiting rows). cancelJobs gains
  rootStatuses to re-check status atomically inside the cancel CTE; the TTL
  sweep passes ['waiting'].
- P1: param-coalescing silently discarded a caller's idempotency_key
  (returned a row the key was never registered against → a later same-key
  submit ran the work twice). Producer-owned idempotency now disables
  param-coalescing outright — it is the stronger contract.
- P2: a quota rejection mid-fanout left  aggregators torn
  (children_ids never written). The fanout now cancels the whole tree and
  surfaces the quota message — all-or-nothing.
- P2: the quota error message no longer includes the live global waiting
  count (it travels to remote MCP clients via submit_agent; counts stay on
  the error object and in local jobs stats).

Scoping:
- P1: reindex-code iterated ALL sources' code pages but re-imported with the
  CLI-level sourceId (undefined without a source flag) — under the new
  default-scoped reads that duplicated every non-default-source code page
  into 'default' and re-embedded it. fetchCodePages now SELECTs source_id
  and each page re-imports into its OWN source.
- P2: importImageFile kept the variable-bound unscoped-read ternary the CI
  guard's inline heuristic can't see; flipped to the canonical
  sourceId-or-default read.

Malformed paths:
- P2: the FS-walk route only checked basenames, so a bracket-named DIRECTORY
  of clean files passed the walker (and its rows were unsweepable on non-git
  brains); descent now applies the malformed-segment check.
- P2: doctor's malformed_path_pages SQL prefilter was brackets-only; a
  [[:cntrl:]] regex arm covers control-character-only paths.
- P2: junk filenames are control-character carriers by definition — every
  place sync/import echoes one now routes through sanitizePathForDisplay
  (control chars → U+FFFD, length-capped) so a crafted filename can't inject
  terminal escapes.

Filesystem:
- P2: atomicWriteFileSync documented mode preservation but open(2)'s mode
  arg is umask-masked; an explicit chmod on the tmp file now actually
  preserves the target's mode across the rename (group-writable brain repos
  no longer lose their group-write bit on a backlinks fix).

Test honesty:
- P1: the divergence test's seed screamed under BOTH the correct
  (completed-based) and naive (cancellations-count-as-drain) metrics;
  reseeded so only the correct metric fires.
- P2: the backlinks frontmatter-safety fixtures used quoted mid-line
  '## Timeline' strings that the ^-anchored regex ignores even without the
  bodyStart guard; replaced with line-start YAML-comment forms that
  genuinely distinguish guard-on from guard-off.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: ship pre-landing review fixes — guard regex, sanitization, TTL grace window

Pre-landing review army (5 specialists + red team) findings, all applied:

- check-getpage-scoped-write guard: catch the EXPANDED ternary
  (x ? { sourceId: x } : undefined) and {}/null false branches the
  shorthand-only regex missed; the sharpened probe immediately found one
  latent expanded-ternary site in operations.ts (safe pattern — write keys
  on the returned row's source_id — now carrying the documented opt-out).
- Terminal hygiene: job names from the MCP-exposed submit surface are
  control/ANSI-stripped before echoing into jobs-stats + doctor screams.
- Waiting-TTL warn-before-act: one ~30s tick between warn and sweep was a
  courtesy log line, not a warning. runWaitingTtlTick (admission.ts, testable
  per the lock-renewal-tick pattern) now stamps an ISO timestamp and holds a
  1h grace window (GBRAIN_MINIONS_TTL_NOTICE_GRACE_MS seam) before the first
  sweep; the upgrade banner stamps the same clock; legacy 'true' flags sweep
  immediately. Worker + upgrade share countTtlExpiredWaiting.
- TTL_REASON_PREFIX constant shared by the sweep reason + both LIKE
  consumers (jobs stats, doctor); patterns parameterized.
- isQueueQuotaExceededError code-based guard replaces error-name string
  matching in agent fanout, cycle patterns, synthesize; submit_agent maps
  quota rejections to a structured rate_limited OperationError.
- Empty-payload coalesce guard derives from PARAM_HASH_EXCLUDED_KEYS.
- Red team: stored_type_is_alias lint hint pointed at a non-existent
  'schema unify' subcommand — now points at the real unify-types job;
  reconcile prints a distinct count for malformed-path rows (their files
  usually still exist on disk — rename to re-import) instead of lumping
  them into 'source file was removed'.
- integrity: one-time note when legacy (pre-source_id) resume-ledger
  entries force a partial re-scan.

Tests: atomic-write direct unit (verify-abort, umask/mode preservation);
cancelJobs reason/rootStatuses direct; idempotency-key-disables-coalesce;
runWaitingTtlTick notice→grace→sweep + legacy flag + kill-switch; walker
bracket-directory descent; sanitizePathForDisplay unit rows; backlinks
multi-gap batch; quarantine clear ambiguity exit-2 + source scoping;
doctor divergence/TTL/malformed_path_pages; engine-parity unscoped-getPage
ORDER BY (DATABASE_URL-gated). TODOS: requeue surface, dream-path quota
integration tests, coalesce concurrency e2e, canonical-json consolidation,
reconcile quarantine option, cross-source clobber audit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: adversarial cross-model review fixes — bracket scoping, quota lock, writer scope

Step-11 adversarial passes (Claude subagent + Codex exec + Codex structured
review). Cross-model high-confidence finding, fixed: blanket bracket
rejection over-captured — Next.js/Nuxt dynamic-route paths (app/[id]/…)
in code-strategy lanes stopped importing AND their indexed rows became
reconcile-deletable; bare-bracket markdown files imported by pre-gate
releases would be hard-deleted by a routine post-upgrade full sync while
their file sat on disk. Two-tier fix in core/sync.ts: ADMISSION rejects
control chars anywhere + brackets on markdown paths only
(hasMalformedPathSegment); the row-DELETING lanes (reconcile, modified-lane
cleanup) act only on the poison signature isPoisonedPath (`](` or control
chars). Walker now checks the RELATIVE path per file (basename checks
can't see bracket directories) and descends bracket dirs for code lanes.

Also fixed:
- Name-global quota: pg_advisory_xact_lock('minion_quota:'||name)
  serializes check+insert — concurrent distinct-payload submits could each
  observe capacity and overshoot without bound (quota is now exact, not a
  backstop; lock cost applies only to quota'd names).
- cancelJobs stamps the reason on ROOT ids only — descendants carried a
  factually false waiting_ttl_expired text and inflated the LIKE-prefix
  stats the alerting surfaces count.
- synthesize: a mid-transcript quota rejection now rolls back that
  transcript's fresh (non-coalesced) chunk submissions — previously the
  skip report said 'skipped' while earlier chunks drained and wrote
  partial pages.
- BrainWriter: putRawData + addLink (both directions) now carry the
  writer's source scope — unscoped they defaulted to 'default'-source rows
  (both Codex passes flagged this P1).
- page-lock: acquisition is O_EXCL ('wx') — the existsSync→write sequence
  let two stale-reclaimers both 'acquire' and lose one side's writes.
- atomic-write: writeSync loops until all bytes land (legal short writes
  could atomically install truncated content) + best-effort parent-dir
  fsync after rename.
- Copy-pasteable command hints gate untrusted names/types through strict
  token checks (safeConfigSegment/safeCliToken) — display sanitization
  keeps shell metacharacters, so injected names could ride into pasteable
  commands (jobs stats, doctor, schema lint hints).
- admission isOffValue: case-insensitive + trimmed + 'no' — 'FALSE'/'Off'
  silently left the off-switch ON.
- Walker/import/sync surface malformed-filename exclusions on BOTH
  collection routes (git + FS-walk), in directory imports, full-sync
  dry runs, and the totalChanges===0 up_to_date early return (a commit
  whose only changes are malformed files advances the anchor past them
  forever — now reported + counted in SyncResult).
- submit_agent response carries coalesced:true so remote clients can
  detect param-coalescing instead of silently receiving an existing job.

Rejected (by design, documented): default-on subagent param-coalescing
semantics (user-approved; per-call + config opt-outs, response flag makes
it detectable).

Tests: markdown-scoped bracket rows + code-strategy framework paths +
isPoisonedPath units (shape suite); bare-bracket row SURVIVES reconcile
(serial suite); walker dir-descent pin retained via relative-path check.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: current-state reference updates for the review-fix batch

KEY_FILES: SyncableReason union gains 'malformed-path' with the two-tier
admission/destruction semantics; admission.ts entry reflects the grace-window
warn-gate, exact quota lock, root-only cancel reasons, and the shared
TTL/quota helpers; atomic-write entry reflects the full-write loop +
parent-dir fsync. llms bundles regenerated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: raise module-size ceilings for wave growth + refresh structural manifest

Nine ratcheted modules grew with the five-issue wave's feature code
(admission control in queue/worker/jobs, malformed-path lanes in
sync/import-file, divergence findings in doctor, ORDER BY in both engines,
quota rollback in synthesize) — conscious TSV raises per the ratchet's
documented growth path. structural-suites.tsv regenerated for the new
test files.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* v0.46.11.0 fix: five-issue operational wave — backlinks corruption, queue admission, junk paths, source scoping, type visibility

Version re-bumped from the user-pinned 0.46.10.0: master claimed that
version mid-ship (queue collision; natural next off master).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: update project documentation for v0.46.11.0

Post-ship /document-release pass for the five-issue operational wave:

- queue-operations-runbook: new DIVERGENT QUEUE section (jobs stats
  Drained/Waiting columns, divergence thresholds, admission remediations,
  GBRAIN_MINIONS_ADMISSION kill-switch) + doctor divergent-queue and
  waiting-TTL subcheck entries
- minions-deployment: "jobs sit in waiting forever" nuanced for the
  waiting-TTL (48h default for subagent)
- schema-packs: schema lint data-plane rules documented behind --with-db
  (per-source scoping exists at the rule layer, not yet CLI-threaded);
  ingest-time type warnings + schema.type_warnings off-switch
- minion-orchestrator skill: coalesced-submit semantics (treat as success,
  monitor the matched id, never resubmit), waiting-TTL, rate_limited quota
  backoff + anti-pattern bullet
- KEY_FILES: lint-rules entry 12->14 rules / 2->4 DB-aware,
  guards-manifest count 48->52
- schema-author skill + tutorial: DB-aware rule list and 14-rule surface
- TODOS: filed P3 for threading --source-id into schema lint --with-db
  (+ jobs stats --json usage line, agent-run coalesce hint)
- regenerated skills.lock.json, plugin/ tree, llms bundles

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

2026-08-16 10:22:52 -07:00

Compare commits

Diff Content Not Available