gbrain

mirror of https://github.com/garrytan/gbrain.git synced 2026-08-16 18:02:30 +00:00

Author	SHA1	Message	Date
Garry TanandClaude Fable 5	5fbc73a23f	docs: update project documentation for v0.46.11.0 Post-ship /document-release pass for the five-issue operational wave: - queue-operations-runbook: new DIVERGENT QUEUE section (jobs stats Drained/Waiting columns, divergence thresholds, admission remediations, GBRAIN_MINIONS_ADMISSION kill-switch) + doctor divergent-queue and waiting-TTL subcheck entries - minions-deployment: "jobs sit in waiting forever" nuanced for the waiting-TTL (48h default for subagent) - schema-packs: schema lint data-plane rules documented behind --with-db (per-source scoping exists at the rule layer, not yet CLI-threaded); ingest-time type warnings + schema.type_warnings off-switch - minion-orchestrator skill: coalesced-submit semantics (treat as success, monitor the matched id, never resubmit), waiting-TTL, rate_limited quota backoff + anti-pattern bullet - KEY_FILES: lint-rules entry 12->14 rules / 2->4 DB-aware, guards-manifest count 48->52 - schema-author skill + tutorial: DB-aware rule list and 14-rule surface - TODOS: filed P3 for threading --source-id into schema lint --with-db (+ jobs stats --json usage line, agent-run coalesce hint) - regenerated skills.lock.json, plugin/ tree, llms bundles Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-16 10:11:51 -07:00
Garry TanandClaude Fable 5	877d1ca479	v0.46.11.0 fix: five-issue operational wave — backlinks corruption, queue admission, junk paths, source scoping, type visibility Version re-bumped from the user-pinned 0.46.10.0: master claimed that version mid-ship (queue collision; natural next off master). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-16 09:47:42 -07:00
Garry TanandClaude Fable 5	0b3bafeb06	chore: raise module-size ceilings for wave growth + refresh structural manifest Nine ratcheted modules grew with the five-issue wave's feature code (admission control in queue/worker/jobs, malformed-path lanes in sync/import-file, divergence findings in doctor, ORDER BY in both engines, quota rollback in synthesize) — conscious TSV raises per the ratchet's documented growth path. structural-suites.tsv regenerated for the new test files. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-16 09:33:05 -07:00
Garry Tan	506cfbf787	Merge remote-tracking branch 'origin/master' into garrytan/fix-five-issue-wave # Conflicts: # TODOS.md # src/commands/doctor.ts # src/core/operations.ts	2026-08-16 09:28:13 -07:00
Garry TanandClaude Fable 5	a944a7a6d2	docs: current-state reference updates for the review-fix batch KEY_FILES: SyncableReason union gains 'malformed-path' with the two-tier admission/destruction semantics; admission.ts entry reflects the grace-window warn-gate, exact quota lock, root-only cancel reasons, and the shared TTL/quota helpers; atomic-write entry reflects the full-write loop + parent-dir fsync. llms bundles regenerated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-16 09:16:21 -07:00
Garry TanandClaude Fable 5	70202a24f9	fix: adversarial cross-model review fixes — bracket scoping, quota lock, writer scope Step-11 adversarial passes (Claude subagent + Codex exec + Codex structured review). Cross-model high-confidence finding, fixed: blanket bracket rejection over-captured — Next.js/Nuxt dynamic-route paths (app/[id]/…) in code-strategy lanes stopped importing AND their indexed rows became reconcile-deletable; bare-bracket markdown files imported by pre-gate releases would be hard-deleted by a routine post-upgrade full sync while their file sat on disk. Two-tier fix in core/sync.ts: ADMISSION rejects control chars anywhere + brackets on markdown paths only (hasMalformedPathSegment); the row-DELETING lanes (reconcile, modified-lane cleanup) act only on the poison signature isPoisonedPath (`](` or control chars). Walker now checks the RELATIVE path per file (basename checks can't see bracket directories) and descends bracket dirs for code lanes. Also fixed: - Name-global quota: pg_advisory_xact_lock('minion_quota:'\|\|name) serializes check+insert — concurrent distinct-payload submits could each observe capacity and overshoot without bound (quota is now exact, not a backstop; lock cost applies only to quota'd names). - cancelJobs stamps the reason on ROOT ids only — descendants carried a factually false waiting_ttl_expired text and inflated the LIKE-prefix stats the alerting surfaces count. - synthesize: a mid-transcript quota rejection now rolls back that transcript's fresh (non-coalesced) chunk submissions — previously the skip report said 'skipped' while earlier chunks drained and wrote partial pages. - BrainWriter: putRawData + addLink (both directions) now carry the writer's source scope — unscoped they defaulted to 'default'-source rows (both Codex passes flagged this P1). - page-lock: acquisition is O_EXCL ('wx') — the existsSync→write sequence let two stale-reclaimers both 'acquire' and lose one side's writes. - atomic-write: writeSync loops until all bytes land (legal short writes could atomically install truncated content) + best-effort parent-dir fsync after rename. - Copy-pasteable command hints gate untrusted names/types through strict token checks (safeConfigSegment/safeCliToken) — display sanitization keeps shell metacharacters, so injected names could ride into pasteable commands (jobs stats, doctor, schema lint hints). - admission isOffValue: case-insensitive + trimmed + 'no' — 'FALSE'/'Off' silently left the off-switch ON. - Walker/import/sync surface malformed-filename exclusions on BOTH collection routes (git + FS-walk), in directory imports, full-sync dry runs, and the totalChanges===0 up_to_date early return (a commit whose only changes are malformed files advances the anchor past them forever — now reported + counted in SyncResult). - submit_agent response carries coalesced:true so remote clients can detect param-coalescing instead of silently receiving an existing job. Rejected (by design, documented): default-on subagent param-coalescing semantics (user-approved; per-call + config opt-outs, response flag makes it detectable). Tests: markdown-scoped bracket rows + code-strategy framework paths + isPoisonedPath units (shape suite); bare-bracket row SURVIVES reconcile (serial suite); walker dir-descent pin retained via relative-path check. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-16 09:11:39 -07:00
Garry TanandClaude Fable 5	bf51bf500f	fix: ship pre-landing review fixes — guard regex, sanitization, TTL grace window Pre-landing review army (5 specialists + red team) findings, all applied: - check-getpage-scoped-write guard: catch the EXPANDED ternary (x ? { sourceId: x } : undefined) and {}/null false branches the shorthand-only regex missed; the sharpened probe immediately found one latent expanded-ternary site in operations.ts (safe pattern — write keys on the returned row's source_id — now carrying the documented opt-out). - Terminal hygiene: job names from the MCP-exposed submit surface are control/ANSI-stripped before echoing into jobs-stats + doctor screams. - Waiting-TTL warn-before-act: one ~30s tick between warn and sweep was a courtesy log line, not a warning. runWaitingTtlTick (admission.ts, testable per the lock-renewal-tick pattern) now stamps an ISO timestamp and holds a 1h grace window (GBRAIN_MINIONS_TTL_NOTICE_GRACE_MS seam) before the first sweep; the upgrade banner stamps the same clock; legacy 'true' flags sweep immediately. Worker + upgrade share countTtlExpiredWaiting. - TTL_REASON_PREFIX constant shared by the sweep reason + both LIKE consumers (jobs stats, doctor); patterns parameterized. - isQueueQuotaExceededError code-based guard replaces error-name string matching in agent fanout, cycle patterns, synthesize; submit_agent maps quota rejections to a structured rate_limited OperationError. - Empty-payload coalesce guard derives from PARAM_HASH_EXCLUDED_KEYS. - Red team: stored_type_is_alias lint hint pointed at a non-existent 'schema unify' subcommand — now points at the real unify-types job; reconcile prints a distinct count for malformed-path rows (their files usually still exist on disk — rename to re-import) instead of lumping them into 'source file was removed'. - integrity: one-time note when legacy (pre-source_id) resume-ledger entries force a partial re-scan. Tests: atomic-write direct unit (verify-abort, umask/mode preservation); cancelJobs reason/rootStatuses direct; idempotency-key-disables-coalesce; runWaitingTtlTick notice→grace→sweep + legacy flag + kill-switch; walker bracket-directory descent; sanitizePathForDisplay unit rows; backlinks multi-gap batch; quarantine clear ambiguity exit-2 + source scoping; doctor divergence/TTL/malformed_path_pages; engine-parity unscoped-getPage ORDER BY (DATABASE_URL-gated). TODOS: requeue surface, dream-path quota integration tests, coalesce concurrency e2e, canonical-json consolidation, reconcile quarantine option, cross-source clobber audit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-16 08:43:52 -07:00
Garry Tan	6624f1a53b	Merge remote-tracking branch 'origin/master' into garrytan/fix-five-issue-wave # Conflicts: # TODOS.md # src/core/cli-flag-registry.generated.ts	2026-08-16 07:33:59 -07:00
Garry TanandClaude Fable 5	e3fd862caf	fix: adversarial-review hardening — 15 confirmed findings across the wave A 35-agent adversarial verification pass (5 dimension finders, 2 independent skeptics per finding) confirmed 15 defects the suites missed; all fixed: Queue (the serious ones): - P0: waiting-TTL keyed on created_at cancelled jobs the SAME maintenance tick had just requeued/retried/promoted (those paths bump updated_at, not created_at) with a factually-false 'waited > Nh' reason. The sweep, the notice counts, and the coalesce age-bound all key on updated_at now — a state transition refreshes the TTL window; pure-waiting backlog semantics unchanged. - P1: the sweep's SELECT→cancel race could cancel a JUST-CLAIMED active job (claimer and sweep both target the oldest waiting rows). cancelJobs gains rootStatuses to re-check status atomically inside the cancel CTE; the TTL sweep passes ['waiting']. - P1: param-coalescing silently discarded a caller's idempotency_key (returned a row the key was never registered against → a later same-key submit ran the work twice). Producer-owned idempotency now disables param-coalescing outright — it is the stronger contract. - P2: a quota rejection mid-fanout left aggregators torn (children_ids never written). The fanout now cancels the whole tree and surfaces the quota message — all-or-nothing. - P2: the quota error message no longer includes the live global waiting count (it travels to remote MCP clients via submit_agent; counts stay on the error object and in local jobs stats). Scoping: - P1: reindex-code iterated ALL sources' code pages but re-imported with the CLI-level sourceId (undefined without a source flag) — under the new default-scoped reads that duplicated every non-default-source code page into 'default' and re-embedded it. fetchCodePages now SELECTs source_id and each page re-imports into its OWN source. - P2: importImageFile kept the variable-bound unscoped-read ternary the CI guard's inline heuristic can't see; flipped to the canonical sourceId-or-default read. Malformed paths: - P2: the FS-walk route only checked basenames, so a bracket-named DIRECTORY of clean files passed the walker (and its rows were unsweepable on non-git brains); descent now applies the malformed-segment check. - P2: doctor's malformed_path_pages SQL prefilter was brackets-only; a [[:cntrl:]] regex arm covers control-character-only paths. - P2: junk filenames are control-character carriers by definition — every place sync/import echoes one now routes through sanitizePathForDisplay (control chars → U+FFFD, length-capped) so a crafted filename can't inject terminal escapes. Filesystem: - P2: atomicWriteFileSync documented mode preservation but open(2)'s mode arg is umask-masked; an explicit chmod on the tmp file now actually preserves the target's mode across the rename (group-writable brain repos no longer lose their group-write bit on a backlinks fix). Test honesty: - P1: the divergence test's seed screamed under BOTH the correct (completed-based) and naive (cancellations-count-as-drain) metrics; reseeded so only the correct metric fires. - P2: the backlinks frontmatter-safety fixtures used quoted mid-line '## Timeline' strings that the ^-anchored regex ignores even without the bodyStart guard; replaced with line-start YAML-comment forms that genuinely distinguish guard-on from guard-off. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-16 00:39:15 -07:00
Garry TanandClaude Fable 5	70ed8060da	fix(minions): empty payloads never param-coalesce — no dedupe signal Full-suite fallout (budget-tracker.test.ts): scaffolding jobs submitted with literal {} data hashed identically and coalesced into one row, collapsing the tests' owner/child fixtures. Two no-param submits are far more likely distinct placeholder jobs than a runaway producer (which always carries a prompt), so add() now skips coalescing when the payload has no keys beyond __param_hash. Pinned in test/minions-admission.test.ts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-16 00:06:07 -07:00
Garry TanandClaude Fable 5	3d203b7258	docs: five-issue fix-wave reference updates — KEY_FILES, TODOS, invariants, flag registry - CLAUDE.md source-isolation invariant gains the unscoped-check/scoped-write corollary + the new guard's opt-out marker (llms bundles regenerated in the same commit per the CLAUDE.md-edit rule). - KEY_FILES.md: current-state entries for src/core/atomic-write.ts, src/core/minions/admission.ts, src/core/schema-pack/type-usage.ts, and scripts/check-getpage-scoped-write.mjs. - TODOS.md: six wave follow-ups (atomic-write migration + page-lock unification, splitFrontmatter relocation, admission/stats indexes, getPage type-boundary redesign, per-name claim fairness, per-queue divergence scoping) and the existing --max-pending entry now notes param-coalescing as the shipped payload-distinct dedupe primitive. - docs/progress-events.md: new backlinks.fix phase (additive). - Flag registry regenerated for quarantine --source-id; scrubbed a prose-bleed '--refresh' token that page-lock.ts's comment would have injected into check-backlinks' allowlist via the new import edge (the known generator pitfall — flags in comments must be spelled dash-less). - test/minions-admission.test.ts: env mutation moved onto withEnv (test isolation guard). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-15 23:55:33 -07:00
Garry TanandClaude Fable 5	ee8d48e1b3	feat(minions): queue admission control — param-coalescing, waiting-TTL, quota + divergence alerting The submit-side half of the queue-divergence incident (drain-side pool starvation landed in v0.46.1.0): subagent intake ran ~25x drain with no backpressure, no expiry, and stats that couldn't even express the divergence. Admission-side only by design — claim fairness / lane scheduling is a filed follow-up, not this wave. New src/core/minions/admission.ts (per-name defaults tables, the handler-timeouts.ts pattern; config under the new 'minions.' prefix; GBRAIN_MINIONS_ADMISSION=0 kill-switch; fail-open to defaults with a once-per-process warning): - PARAM-COALESCING (default on for subagent): an identical parentless submit — same (name, queue, payload hash) — returns the newest matching WAITING row with coalesced:true instead of enqueuing a duplicate. __owner_client_id is INCLUDED in the hash so owner lanes never cross; parented submits never coalesce (aggregator bookkeeping); matches are age-bounded to ttl/2 so a fresh submit can't coalesce onto a nearly-expired row and silently die an hour later; waiting-only (a running job never suppresses a re-run). Audited via the existing backpressure JSONL. - WAITING-TTL (default 48h for subagent): the worker's 4th maintenance sweep cancels jobs still waiting past their TTL — through the canonical cancel path (new cancelJobs(ids, {reason}) batch variant: one transaction, descendant cancellation, child_done inbox messages, aggregator-parent resolution — a raw UPDATE would wedge parents in waiting-children), capped at 500/tick, oldest-first, with a reason error_text every alerting surface keys on (the single-id cancel path never wrote one). Warn-before-act: tick 1 counts + warns + sets minions.ttl_notice_shown, sweeping starts tick 2; gbrain upgrade prints the same one-shot notice on the interactive channel. - NAME-GLOBAL QUOTA (config-only, NO shipped default per operator decision): waiting count for a name across ALL queues (per-run private fanout queues can't dodge it) at/over minions.quota_max_waiting.<name> rejects with a typed QueueQuotaExceededError; approximate under concurrency, documented as a backstop. Dream submitters (synthesize/patterns) record the rejection as a phase skip — never a phase crash. Alerting: getStats by_type gains drained_completed/failed/dead/cancelled (true outflow keyed on finished_at, split so TTL-cancel storms can't masquerade as throughput), waiting_now, and oldest_waiting_minutes. jobs stats gains Drained/Waiting columns, a --json document, a per-type DIVERGENT-QUEUE scream (intake > GBRAIN_QUEUE_DIVERGENCE_RATIO x completed with waiting > GBRAIN_QUEUE_DIVERGENCE_MIN_WAITING) carrying the quota opt-in hint, and a 24h waiting-TTL cancellation report. doctor checkQueueHealth screams the same divergence + TTL activity. The advisor's stalled-jobs fix command now points at the real subcommand (jobs stats; 'jobs status' never existed). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-15 23:42:01 -07:00
Garry TanandClaude Fable 5	bf0b28cd64	fix(scoping): close the unscoped-check/scoped-write bug class — writer, import, integrity + CI guard engine.getPage without opts matches a slug in ANY source while putPage defaults to 'default' — an existence check can succeed against source B and the paired write then targets a different row (duplicates, clobbers, spurious slug disambiguation; this class broke dream cycles for weeks). Fixes: - SlugRegistry + BrainWriter/WriteTxImpl thread a sourceId (default 'default') through every probe, read, write, addTimelineEntry, and the validator pass (which previously hardcoded 'default' regardless of the pages being validated). - integrity auto uses one writer PER SOURCE, and its resume progress is keyed by (source_id, slug) instead of slug alone — a resume no longer skips same-slug pages in other sources (legacy slug-only progress entries match the default source). - import-file reads flip from any-source-when-unset to the canonical { sourceId: sourceId ?? 'default' } (mirrors the writes' schema default). Caller audit: every importFromContent/importFromFile call site either passes sourceId explicitly or is default-source-intentional (brain-global synthesis artifacts, eval sandboxes) — none relied on the any-source fallback. - quarantine clear resolves the target source deterministically: new --source-id flag; without it, an ambiguous multi-source slug errors with the candidate list instead of acting on an arbitrary row. - link-extraction's exact-probe steps reuse the resolver's source scope (same posture as its basename index). - Both engines' unscoped getPage gains a deterministic tiebreak: ORDER BY (source_id = 'default') DESC, source_id ASC — default-first, then stable alpha (plain LIMIT 1 returned an arbitrary row). Guard: scripts/check-getpage-scoped-write.mjs (comment/string-aware span scanner) fails the build when a file contains BOTH an unscoped or ternary-undefined getPage AND a write-path call; opt-out marker gbrain-allow-unscoped-getpage for documented read-only first-match sites. Wired into verify CHECKS + guards-manifest + bad/good fixtures + unit test; the grandfathered allowlist is empty. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-15 23:27:41 -07:00
Garry TanandClaude Fable 5	86f0da5eb9	feat(schema): warn on alias/undeclared explicit frontmatter types at ingest + lint An explicit frontmatter type that is an ALIAS of a canonical pack type (or undeclared entirely) is stored literally and never re-normalized, so agents can silently file the same concept under different types and directories. gbrain can't stop agent-side filing decisions; it can make every non-canonical explicit type loud. - src/core/schema-pack/type-usage.ts: classifyStoredType(type, pack) → canonical \| alias_of (with the canonical type + its path_prefixes[0] filing directory) \| undeclared; sanitizeTypeForDisplay (control-char strip + length cap for terminal hygiene — type strings come from frontmatter); renderTypeWarningSummary (once-per-type lines). - importFromContent classifies at the typeExplicit site and returns an advisory ImportResult.type_warning — the type is still stored as-is, zero filing behavior change. - gbrain sync + gbrain import aggregate warnings once per distinct type per run (O(types), not O(files)) to stderr, and sync carries the counts on SyncResult.type_warnings so worker-driven syncs surface them in job results where daemon stderr is invisible. - Two data-plane schema lint rules (stored_type_is_alias, stored_type_undeclared) audit the EXISTING corpus via gbrain schema lint. - Config off-switch schema.type_warnings (default on) registered in the known-keys list; lint rules stay active regardless. Acknowledged limit: this warns at ingest and audit — it cannot prevent routing that happened in the agent's filing layer before import. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-15 23:18:16 -07:00
Garry TanandClaude Fable 5	b8d07a0850	fix(sync): reject malformed filenames at import — classifier reason, delete-lane carve-outs, doctor discovery Files literally named like markdown links polluted search because slugifySegment STRIPS brackets instead of rejecting them, minting plausible slugs for junk files. New SyncableReason 'malformed-path' (brackets + ASCII control chars; parens deliberately legal) rejects such filenames at every ingestion route: classifySync (incremental sync), isCollectibleForWalker (bulk import, both FS-walk and git fast-path), and a defense-in-depth check in importFromFile for direct callers. The subtle halves (both caught by outside-voice review): - Deletions and reconciles must still SEE malformed paths. manifest.deleted is filtered by isSyncable, and the full-sync reconcile requires isSyncable(source_path) — naively classifying junk as unsyncable would have orphaned previously-ingested rows forever. Both lanes now carry an explicit malformed-path carve-out, so deleting a junk file removes its DB row, editing one sweeps its stale row, and a full sync reconciles poison away even while the junk file still sits in the repo. Classifier ordering (strategy before malformed-path) keeps the reconcile strategy-safe; the #1433 metafile protection is untouched. - Malformed skips are INFORMATIONAL, never failures. importFromFile marks them skip_reason='malformed_path'; sync's failure classification keeps them out of failedFiles and the failure ledger (they can never gate bookmark advancement) and checkpoints them as done. Surfaced instead in the dry-run listing, a summary count, SyncResult.malformedSkipped, and a new doctor check (malformed_path_pages) that reports existing poisoned rows with the reconcile fix hint. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-15 23:10:22 -07:00
Garry TanandClaude Fable 5	f3931df6a4	fix(backlinks): frontmatter-safe fixer — canonical body offset, validate, atomic write, page lock The check-backlinks fixer could glue a generated timeline bullet above the frontmatter fence (naive split on '## Timeline' matched inside YAML and as a substring of longer headings; byte arithmetic then landed the insert at byte 0), breaking YAML parsing for the whole page. The write was also a bare writeFileSync: non-atomic, unvalidated, unlocked. fixBacklinkGaps is now a safety pipeline per target file: - pre-validate with parseMarkdown({validate:true}); fence/YAML-broken pages (YAML_PARSE / MISSING_CLOSE / NULL_BYTES) are skipped and reported, byte-identical. MISSING_OPEN is deliberately NOT a blocker: legacy pages without frontmatter have no fence to corrupt and stay fixable. - insertion anchors only on a real '## Timeline' heading line (CRLF-tolerant) at/after the canonical frontmatter body offset (new markdown.ts frontmatterBodyOffset, mirroring collectValidationErrors' fence semantics); first heading wins deterministically; near-misses ('### Timeline', '## Timeline (2026)') never match. - post-validate the candidate, then write via new src/core/atomic-write.ts (unique tmp sibling + fsync + mode preservation + on-disk re-validation before the atomic rename). Migrating the older per-module atomic-write copies is a filed follow-up. - each file is wrapped in withPageLock + try/catch: one bad page or write error isolates to a skip entry instead of killing the batch. fixBacklinkGaps returns {fixed, skipped[]}; BacklinksResult carries the skip report (additive), the CLI prints it, and the fix loop runs under a new backlinks.fix progress phase. Dead sourceDir var removed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-15 22:59:09 -07:00

Compare commits

Diff Content Not Available