Compare commits

...
Author SHA1 Message Date
Garry TanandClaude Fable 5 e5dc0ac399 test(ci): R5 isolation guard — configureGateway/__setEmbedTransportForTests require resetGateway teardown (#3066)
check-test-isolation.sh gains rule R5 (comment-stripped grep so prose
mentions of resetGateway() don't satisfy it, exact call syntax on the
trigger so test-name prose doesn't fire it). Fixes the 11 current
violators with resetGateway() in afterAll; 5 fixture cases pin the rule.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 11:10:33 -07:00
Garry TanandClaude Fable 5 c11ec1166f fix(facts): surface chat-gateway unavailability instead of silent success-shaped no-op (#3062)
- runFactsBackstop records skipped: 'chat_unavailable' (both modes) and
  declines to enqueue facts-absorb jobs guaranteed to no-op.
- New gateway unavailableReason()/warnUnavailableOnce() name the configured
  model, the missing auth_env keys, and the recipe setup_url; once-per-
  process stderr warn fires at the extractFactsFromTurn guard and at
  expand()'s silent single-query degrade (tokenmax's headline knob).
- doctor facts_health distinguishes '0 active facts, chat gateway
  unreachable' (warn) from a genuinely empty brain (ok).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 11:10:33 -07:00
393805ab2c fix(extract): link-aware Source — Summary delimiter in timeline bullets (#3059)
Takeover of #3060: the Format-1 timeline parser split 'Source — Summary' at
the first dash after the pipe, which lands inside markdown links (hyphenated
link targets, em-dash link labels), shattering one entry into two fragments
that re-insert on every sync. Delimiter scan is now bracket-depth-aware and
whitespace-anchored; delimiterless bullets are kept whole under the
'markdown' source sentinel instead of dropped. Test fixtures renamed to the
repo's generic placeholders.

Co-authored-by: wright-io <wright-io@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 11:10:20 -07:00
spiky02plateau e79b8d5780 fix(migrations): let force-retry escape completed ledger entries (#2616)
statusForVersion short-circuited on any 'complete' entry before checking
the trailing 'retry' marker, so --force-retry appended an inert row and a
version marked complete with zero work done could never be re-run without
hand-editing completed.jsonl. Check retry-latest first: an explicit
--force-retry now yields 'pending' even past an earlier 'complete', while
a stray 'partial' after 'complete' still cannot regress the version.
2026-07-23 09:17:01 -07:00
spiky02plateau fc1f88cdcb fix(minions): default timeout for contextual reindex (#2611) 2026-07-23 09:16:55 -07:00
70ffe4a2a2 fix: honor explicit list_pages limit for local callers, warn on remote clamp, thread offset (#2591)
gbrain list --limit 100000 silently returned 100 rows (default 50) with
no warning, and --offset was accepted but dropped at the op layer even
though PageFilters has supported it all along.

- Local CLI callers (ctx.remote === false, the same trust boundary that
  already bypasses scope enforcement) get an explicit limit above 100
  honored — full enumeration is a legitimate local operation.
- Remote MCP/OAuth callers keep the 100-row DoS cap, now loud: one
  logger.warn (stderr, stdout stays script-clean) with both numbers,
  parity with the three search-path clamp warnings.
- offset is declared as a param (so the CLI coerces it to number) and
  threaded to engine.listPages for real pagination.


Claude-Session: https://claude.ai/code/session_01Vswwe1y5fQbJWfbaSK3enT

Co-authored-by: deacon-botdoctor <291411030+deacon-botdoctor@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 09:16:50 -07:00
FloridaStyleandClaude Opus 4.8 5a295bc293 fix(storage): Supabase signed URLs — prepend /storage/v1 (#2565)
SupabaseStorage.getSignedUrl built the download URL as `${projectUrl}${signedURL}`,
but Supabase's sign API returns `signedURL` relative to the Storage API root
(/object/sign/<bucket>/<path>?token=...), so the generated link dropped /storage/v1
and returned 404. Now prepends `${projectUrl}/storage/v1`, tolerating an
already-absolute URL or a value that already carries the prefix. `gbrain files
signed-url` links resolve again.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 09:16:45 -07:00
qaz8545355andqaz8545355 2724c3b6c9 fix: handle <think> reasoning tags in parseExtractorOutput (#2559)
Reasoning models (MiniMax-M3, DeepSeek-R1, etc.) return <think>...</think>
tags in the content field before the actual JSON output. This caused
parseExtractorOutput to fail in two ways:

1. The fence regex /^\`\`\`(json)?...$/ requires the fence at text start;
   <think> preceding it prevents matching, so the raw text (with trailing
   fences) hits JSON.parse and throws.

2. When think tags contain [ or { characters, indexOf finds them inside
   the reasoning block instead of the actual JSON array.

Changes:
- Strip <think>...</think> tags before any parsing (covers all reasoning models)
- Add JSON.parse fallback: truncate at last ] or } to handle trailing
  noise (leftover markdown fences after stripping)

Tests: 28/28 pass (3 new cases for think tags + trailing noise).

Co-authored-by: qaz8545355 <junjun@openclaw.local>
2026-07-23 09:16:39 -07:00
ivandebotandivandebot 1233051a20 fix(propose_takes): memoize empty extractions so zero-claim pages don't re-spend every cycle (#2514)
The idempotency row is only written inside `for (const p of proposals)`, so a
page that extracts ZERO gradeable claims never records an idempotency tuple
and is re-sent to the LLM on every cycle forever. The docstring's "unchanged
page never re-spends tokens" contract only holds for pages that produce >=1
claim; a page that legitimately has no gradeable claims (or any machine-
generated page) is a perpetual cache miss and re-spends tokens indefinitely.

Fix: when `proposals.length === 0`, write one tombstone row keyed by the same
(source_id, page_slug, content_hash, prompt_version) tuple, with
status='rejected' so it never surfaces in a pending-review query (the pending
index filters status='pending'). Content changes (new content_hash) or a
PROPOSE_TAKES_PROMPT_VERSION bump still miss the tombstone and re-extract. The
extractor-throw path `continue`s before the tombstone, so failed pages are
retried rather than cached.

Guard against a subtle regression: `parseExtractorOutput` returns [] for BOTH
a genuine empty extraction AND malformed/prose/truncated model output, so
naively tombstoning every [] would permanently suppress a page that has claims
but hit a transient parse failure. `defaultExtractor` now throws when the
output is empty-but-not-a-clean-`[]` (new `isWellFormedEmptyExtraction`
predicate), routing transient failures into the existing retry path; only a
cleanly-parsed empty array is memoized.

Adds a `tombstones_written` counter for observability.

Tests: tombstone written on genuine empty extraction; two-cycle idempotency
(no repeat LLM call on an unchanged zero-claim page); extractor error writes
no tombstone; isWellFormedEmptyExtraction discriminates clean-[] from
malformed/prose/non-empty output. propose-takes suite: 36 pass / 0 fail.

Co-authored-by: ivandebot <187176982+ivandebot@users.noreply.github.com>
2026-07-23 09:16:34 -07:00
53c9086945 fix(cycle): extract_facts guard requires live backing page, not just non-NULL entity_slug (#2497)
The empty-fence guard counted every `row_num IS NULL AND entity_slug IS NOT NULL`
row as a pending v0_32_2 backfill, but the inline facts writer keeps producing
rows of exactly that shape post-migration: when a resolved slug has no fenceable
page (slugify-floor / stub-guard-blocked unprefixed slugs like `wingman`,
`people-jane-doe`), backstop.ts falls through to a DB-only insert with row_num
NULL. Those rows are structurally unfenceable — no page to fence onto, and the
ledger-complete migration won't re-run — so they jammed the phase forever
(~16/day observed) and the warning advised a no-op `apply-migrations --yes`.

Discriminator: a row is a genuine backfill candidate only if its entity_slug
resolves to a LIVE page in the same source (EXISTS in `pages` with deleted_at
NULL) — mirroring the migration's Phase B, which only fences slugs that map to
a writable page. Genuine pre-v0.32.2 rows (their entity page exists) still gate;
inline-writer unfenceable rows no longer do. Warning text updated to name the
"entity page present, not yet fenced" condition.

Regression tests pin both sides: unfenceable rows (no page / soft-deleted page)
do NOT gate and the phase converges; a legacy row WITH a backing page still
gates. Fails pre-fix, passes post-fix.

(#2484)

Co-authored-by: Javier Aldape <javieraldape@Javiers-Laptop.local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-23 09:16:28 -07:00
033fd24fe8 fix(import): fall back to body H1 for title when frontmatter lacks title: instead of slug-derived junk (#2446) (#2495)
Co-authored-by: Javier Aldape <javieraldape@Javiers-Laptop.local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-23 09:16:22 -07:00
Garry Tan 8915fba476 Revert "fix(search): honor recency decay config on the hybrid path (#2386)"
This reverts commit 0367c800a4.
2026-07-23 09:16:17 -07:00
Garry Tan 372f013158 Revert "feat(links): resolve [[wikilink]] frontmatter values via global_basename (#2406)"
This reverts commit 503f61e6e4.
2026-07-23 09:16:17 -07:00
Garry Tan 439bbaac3a Revert "fix: Bun+Windows write-through EEXIST, non-Anthropic --max-cost pricing, dream-page exclusion in enrich (#2407)"
This reverts commit 2941e17798.
2026-07-23 09:16:17 -07:00
Garry Tan a1bb7683d0 Revert "fix(import): canonicalize slug in importFromContent so mixed-case put_page with tags doesn't roll back (#2436)"
This reverts commit 1b099aeaca.
2026-07-23 09:16:17 -07:00
Garry Tan 94535fc0e0 Revert "fix(chunkers/code): tolerate tiktoken special tokens in estimateTokens (#2453)"
This reverts commit b7f70970c1.
2026-07-23 09:16:17 -07:00
Garry Tan a6aafddd23 Revert "fix(init): point soul-audit hint at the conversational skill, not a nonexistent CLI verb (#2486)"
This reverts commit f8dbfca2f5.
2026-07-23 09:16:17 -07:00
Garry Tan c92af9a7d6 Revert "fix(schema-pack): narrow stats catch-all so masked errors surface, not fake 0 pages (#2493)"
This reverts commit beedacde56.
2026-07-23 09:16:17 -07:00
beedacde56 fix(schema-pack): narrow stats catch-all so masked errors surface, not fake 0 pages (#2493)
`gbrain schema stats` reported "Total pages: 0" on populated brains because
fetchCountRows wrapped its count query in a bare `catch { return []; }` that
converted EVERY error into zero rows — false 0 pages, false "100% coverage"
(0/0 → vacuous 1.0), and a starved `schema suggest`. A sibling bare catch in
detectDeadPrefixes had the same defect.

Root cause is the masked error, NOT a PGLite query incompatibility: reproduced
the exact COUNT query (COALESCE/NULLIF/GROUP BY/ORDER BY ... NULLS LAST) against
the pinned PGLite 0.4.3 (PG17.5) through the real engine + full schema, plus
PG18 and NULL/empty edge-case data — it returns correct counts every time and
never throws. The issue's "the query is failing on PGLite" premise doesn't
reproduce; the actual failure on the reporter's brain was hidden by the catch
(they could not capture it, consistent with an engine/init-level throw). The
honest fix is to stop hiding it.

Both catches now swallow ONLY the genuine missing-table case via the existing
isUndefinedTableError helper (SQLSTATE 42P01 + PGLite "relation ... does not
exist") and rethrow everything else, so the next occurrence shows the real
error instead of a fake zero. Pre-init/empty-brain behavior is preserved.

Regression: 4 new cases in test/schema-pack-stats.test.ts pin (1) real non-zero
count on a populated PGLite brain, (2) fetchCountRows rethrows a non-missing-
table error, (3) fetchCountRows still degrades to empty on 42P01, (4)
detectDeadPrefixes rethrows via the sibling catch. Each error-surfacing test
verified to fail when its catch is re-broadened.

Co-authored-by: Javier Aldape <javieraldape@Javiers-Laptop.local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-23 06:13:07 -07:00
Sean Gearin f8dbfca2f5 fix(init): point soul-audit hint at the conversational skill, not a nonexistent CLI verb (#2486) 2026-07-23 05:12:14 -07:00
Jim TangandClaude Opus 4.8 b7f70970c1 fix(chunkers/code): tolerate tiktoken special tokens in estimateTokens (#2453)
Code legitimately contains tiktoken special-token strings (e.g. CLIP/GPT tokenizers embed the literal <|endoftext|>). The default encode() uses disallowed_special='all' and THROWS on those, crashing reindex-code on valid source files. Re-encode treating them as ordinary text (allowed=[], disallowed=[]); heuristic fallback if even that fails. A token COUNT needs no special-token semantics.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-23 05:12:08 -07:00
Fahd Akhtar 1b099aeaca fix(import): canonicalize slug in importFromContent so mixed-case put_page with tags doesn't roll back (#2436)
putPage lowercases the slug via validateSlug, but the tag/link/timeline
reconcilers (tx.addTag, addLink, addTimelineEntry) query the slug as passed.
A remote put_page with a capitalized slug (e.g. 'Projects/Team-Wiki/Quarterly-Roadmap')
stored the page under 'projects/team-wiki/quarterly-roadmap', then threw
'addTag failed: page "…" not found' on the existence check, rolling back the
entire write — so the page never persisted under either casing. Any agent
driving the HTTP MCP server (where slugs arrive verbatim) lost every page whose
slug carried a capital letter plus a frontmatter tag.

Normalize the slug once at the top of importFromContent (the shared chokepoint
for MCP put_page and CLI capture) so putPage and every reconciler agree on the
canonical lowercased slug. No-op for disk imports (already slugifyPath output),
idempotent with putPage's own validateSlug call. Engine-agnostic, so PGLite and
Postgres move together.

Adds test/put-page-mixed-case-slug-tags.test.ts pinning the regression on PGLite.
2026-07-23 05:03:59 -07:00
nguyenchivietandClaude Opus 4.8 2941e17798 fix: Bun+Windows write-through EEXIST, non-Anthropic --max-cost pricing, dream-page exclusion in enrich (#2407)
* fix(write-through): guard mkdir against EEXIST on Bun+Windows

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(budget): resolve non-Anthropic model pricing via canonical table under --max-cost

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(enrich): exclude dream-generated pages from thin candidates

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 05:03:54 -07:00
spiky02plateauandClaude Opus 4.8 503f61e6e4 feat(links): resolve [[wikilink]] frontmatter values via global_basename (#2406)
When link_resolution.global_basename is enabled, extend basename-index
resolution to frontmatter link fields (FRONTMATTER_LINK_MAP), mirroring the
body bare-wikilink path added in #972.

Problem: a bare-title wikilink in a frontmatter list -- e.g.
  sources:
    - "[[2025-12-25_mentor-extraction]]"
never resolves. SlugResolver.resolve() has no '/' to hit the slug-direct
getPage, and the field's dirHint (sources -> ['source','media']) may name
folders absent from the brain, so the dir-scoped exact + fuzzy steps also
miss. The frontmatter path never consulted resolveBasenameMatches -- that was
wired only for body bare-wikilinks. On a PARA/Obsidian vault this silently
drops the bulk of sources:/related: provenance edges.

Fix: extractFrontmatterLinks takes a globalBasename flag (threaded from
extractPageLinks). On a resolve() miss, unwrap [[ ]] and fall back to
resolver.resolveBasenameMatches -- UNIQUE-MATCH-ONLY, so ambiguous basenames
(archive dupes, generic hubs like _index) stay unresolved rather than create
a wrong edge. Purely additive; resolved frontmatter edges are unchanged.

Scope: covers the db-source extract and live put_page paths (real
makeResolver). The --source fs extract uses an inline resolver without a
basename index, so it gracefully no-ops there (typeof guard).

Tested: 3 new cases (resolves-when-on, ambiguous-stays-unresolved,
gated-off-by-flag); full link-extraction suite green (130 pass).

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-23 05:03:48 -07:00
Richard Baker 0367c800a4 fix(search): honor recency decay config on the hybrid path (#2386)
The hybrid recency stage in runPostFusionStages imported
DEFAULT_RECENCY_DECAY directly, so operator overrides via the
GBRAIN_RECENCY_DECAY env var and the gbrain.yml `recency:` section were
honored only on the get_recent_salience SQL path and silently ignored on
the hot hybridSearch path. Non-default vault layouts therefore stayed on
the baked-in defaults / DEFAULT_FALLBACK (90d / 0.5) regardless of
tuning.

Call resolveRecencyDecayMap() (already used by the SQL path) so the
configured decay map reaches the boost stage. Behavior is unchanged when
no override is set — resolveRecencyDecayMap() returns DEFAULT_RECENCY_DECAY.

Adds test/hybrid-recency-config.test.ts asserting the env override
reaches the applied recency factor (fails against the prior wiring).
2026-07-23 05:03:43 -07:00
Garry Tan 23df0227bd Revert "Reject unknown init flags before migrations (#2201)"
This reverts commit d67be8b570.
2026-07-23 05:03:38 -07:00
Garry Tan 3225bdf768 Revert "fix(queue): dead/cancelled jobs no longer block idempotency re-submission (#2253)"
This reverts commit c0cb6c533b.
2026-07-23 05:03:38 -07:00
Garry Tan 6ec5261700 Revert "feat(ai): claude-cli recipe for native gateway-based subagent dispatch (#2277)"
This reverts commit 5ac81b0d0a.
2026-07-23 05:03:38 -07:00
Garry Tan b0d136ee6d Revert "fix(dims): handle prefixed model IDs on openai-compatible path (#2325)"
This reverts commit 7c06af281d.
2026-07-23 05:03:38 -07:00
Garry Tan 47d7e95b74 Revert "fix(frontmatter): derive validate slug from brain root, not absolute path (#2340)"
This reverts commit 1a9ab6a95f.
2026-07-23 05:03:38 -07:00
Garry Tan c0d4def5bc Revert "fix dream orphan source scope (#2368)"
This reverts commit 6e4c2435e3.
2026-07-23 05:03:38 -07:00
Garry Tan c0a4b80f0d Revert "fix: meter extract atoms haiku calls (#2371)"
This reverts commit 0bd752b3f7.
2026-07-23 05:03:38 -07:00
TheRealMrSystem 0bd752b3f7 fix: meter extract atoms haiku calls (#2371) 2026-07-23 02:09:10 -07:00
Haoqian 6e4c2435e3 fix dream orphan source scope (#2368) 2026-07-23 02:09:05 -07:00
alessioalioncoandClaude Opus 4.8 1a9ab6a95f fix(frontmatter): derive validate slug from brain root, not absolute path (#2340)
Single-file `frontmatter validate` derived the expected slug from the
absolute path: relative(resolve(target), file) is empty when target IS the
file, so it fell back to `|| file` (the full path), yielding "root/<abs>"
slugs and a false SLUG_MISMATCH. The pre-commit hook from install-hook
validates staged files one-by-one, so this rejected every commit in a
markdown brain (only bypassable with --no-verify).

Walk up to the brain root (nearest .git) and use relative(brainRoot, file)
|| basename(file), matching runAudit/runGenerate and sync/extract. Files
above the root fall back to basename instead of a ../-prefixed slug.

Reopens #565. Present since v0.32.0; reproduced on v0.42.51.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-23 02:08:58 -07:00
Noetherly 7c06af281d fix(dims): handle prefixed model IDs on openai-compatible path (#2325)
OpenRouter (and potentially other proxy providers) expose OpenAI's
text-embedding-3 models with a provider prefix in the model ID, e.g.
`openai/text-embedding-3-large` rather than bare `text-embedding-3-large`.

`dimsProviderOptions()` checks `modelId.startsWith('text-embedding-3')`
which fails for the prefixed form, so the `dimensions` parameter is never
sent. The upstream provider returns its native dimensionality (3072 for
-large) instead of the configured value (e.g. 1536), causing an immediate
"dim mismatch" error on first embed.

The default OpenRouter embedding (`text-embedding-3-small` at 1536d)
masked this because its native size happens to match the default config.
The bug surfaces when using `-large`, or `-small` with a non-1536 dim
(512, 768, 1024 — all listed in the recipe's `dims_options`).

Fix: strip the provider prefix before the `startsWith` check. The full
prefixed ID is preserved in the error message for user clarity.
2026-07-23 02:08:51 -07:00
5ac81b0d0a feat(ai): claude-cli recipe for native gateway-based subagent dispatch (#2277)
* feat(subagent): claude-cli MessagesClient adapter (baseline, no tool use)

Closes #334 (partially — text-only baseline; tool use lands in the next
commit on this branch).

Adds a MessagesClient adapter that shells out to `claude --print
--output-format json --model <model>` instead of the Anthropic SDK. When
`GBRAIN_USE_CLAUDE_CLI=1` is set, the subagent worker registers the adapter
in place of the SDK client; the default path (Anthropic SDK with
ANTHROPIC_API_KEY) is unchanged when the env var is unset or set to
anything else.

The benefit is that Claude Max subscribers can run Minions subagents
against their existing OAuth subscription, no ANTHROPIC_API_KEY needed.

New: src/core/minions/handlers/claude-cli-adapter.ts
- Implements the MessagesClient interface exported from subagent.ts.
- Strips provider prefixes (`anthropic:`, `litellm:`) from the model id
  because `claude --print` only accepts CLI-native aliases (`sonnet`,
  `opus`, `haiku`, or the bare `claude-*-N-M` form).
- Flattens the Anthropic messages array into a single text prompt for
  claude-cli stdin. Tool blocks (tool_use / tool_result) are stringified
  as placeholders so multi-turn conversations stay coherent in this
  baseline; native tool_use round-tripping is the follow-up commit.
- Spawns claude with stdio piped, captures stdout, parses the
  `{type:"result", subtype:"success", result, usage, ...}` JSON envelope,
  and returns it as a properly shaped Anthropic.Message with
  `stop_reason: 'end_turn'`.
- Token totals propagate from the claude usage block so the subagent
  handler's `ctx.updateTokens()` reports usable numbers.
- AbortSignal is wired through to SIGTERM the child so the subagent loop's
  cancellation path stays correct.

Modified: src/commands/jobs.ts (worker registration)
- Conditionally constructs a MessagesClient via the new adapter when
  GBRAIN_USE_CLAUDE_CLI=1.
- Passes it into makeSubagentHandler({ engine, client: subagentClient }).
- Logs `[minion worker] subagent routing via claude-cli (GBRAIN_USE_CLAUDE_CLI=1)`
  on startup so the env var status is operator-visible.

Limitations of this commit (addressed in the follow-up):
- Tool use is not yet supported. Tools in params.tools are ignored; the
  adapter returns a single text block with stop_reason='end_turn'.
- Token counts come from claude-cli's reporting and may not match the
  Anthropic API's accounting precisely (especially for cache tiers).

Original design from #334; this commit preserves that author's attribution.
The follow-up commits on this branch carry the tool-use implementation.

* feat(subagent): tool use + context isolation + convention rename on top of jarvisdoes baseline

Builds on the previous commit (jarvisdoes's #334 baseline) by adding three
things the upstream issue called out as gaps or that surfaced during review:

1. Tool use support via system-prompt-instructed JSON emission.
2. Context isolation flags so claude-cli does not load operator-level
   CLAUDE.md, skills, and local project context into every subagent call.
3. Env var rename from GBRAIN_USE_CLAUDE_CLI=1 to
   GBRAIN_SUBAGENT_PROVIDER=claude-cli to match the existing
   GBRAIN_<noun>_<role>=<value> convention used by GBRAIN_CHAT_MODEL,
   GBRAIN_EMBEDDING_MODEL, GBRAIN_EXPANSION_MODEL.

## Tool use

The MessagesClient interface returns Anthropic.Message objects whose
content array may include tool_use blocks. The subagent handler filters
those blocks and dispatches each tool, so any backend that produces
correctly shaped tool_use blocks gets the same loop behavior as the
Anthropic SDK.

The adapter injects a system-prompt addendum describing the tool registry
plus an emission protocol:

  <use_tools>
  [{"id": "...", "name": "...", "input": {...}}, ...]
  </use_tools>

After the response comes back, extractToolCalls() scans for the block,
parses the JSON (tolerant of optional ```json fencing), and converts each
entry into a tool_use content block. Multiple parallel tool calls in one
turn are supported via the array shape; this is the exact case that
breaks today on the codex-proxy / litellm GPT-5.x bridge where parallel
tool-call response IDs get dropped.

Defensive fallbacks:
 - Malformed JSON inside the block: drop to text-only, stop_reason='end_turn'.
 - Unterminated <use_tools> (no close tag): drop to text-only.
 - Model omits id field: adapter synthesizes a toolu_claude_cli_<rand> id.
 - Empty response: still hand the subagent loop a well-formed content
   array so the .filter chain does not crash.

## Context isolation

claude-cli auto-discovers CLAUDE.md from cwd upward and injects the
operator's skills + plugins + auto-memory into the default system prompt.
On a real install that is ~42-65k tokens of contamination per subagent
call, with both cost and behavioral consequences (the subagent picks up
the operator's coding conventions, opinions, and preferences).

The maximum suppression that still preserves OAuth / Claude Max
subscription auth is:
 - Spawn from a dedicated clean cwd (tmpdir-based) so LOCAL CLAUDE.md
   auto-discovery has nothing to find. -13k tokens on a real gbrain
   install where CLAUDE.md is substantial.
 - --disable-slash-commands so skill resolution does not pull in
   /skill-name handlers.
 - --system-prompt <gbrain prompt> so the default system prompt is
   replaced rather than appended to.

The --bare flag would also strip user-level ~/.claude/CLAUDE.md but it
forces ANTHROPIC_API_KEY auth, defeating the whole point of this adapter.
The remaining ~42k cached tokens from user-level instructions are
accepted as a cost-trivial trade-off because the Max subscription absorbs
the per-call cost. Behavioral contamination is mitigated by gbrain's
strong per-call system prompt overriding any operator-level drift.

## Env var rename

Surveyed all ~140 GBRAIN_* env vars in src/. The codebase uses three
patterns: GBRAIN_NO_<feature> (negative toggles), GBRAIN_<noun>_<role>
=<value> (routing keys), GBRAIN_ALLOW_<feature> (permissive toggles).
GBRAIN_USE_* does not appear anywhere except jarvisdoes's original
commit; it would introduce a fourth pattern.

GBRAIN_SUBAGENT_PROVIDER=claude-cli aligns with the routing-keys family
and is value-extensible — adding codex-cli / meridian-proxy / etc. later
means a new value, not a new env var. The scope ('SUBAGENT_*') is also
unambiguous about which calls the toggle covers; GBRAIN_USE_CLAUDE_CLI
was silent on whether it applied to all gbrain LLM calls or only the
subagent path.

Unknown values are rejected with a fail-fast error message naming the
two valid values rather than silently falling through to the default.

## Tests

New file: test/claude-cli-adapter.test.ts — 12 tests, 33 assertions:
 - Text-only round trip (single text block, usage propagation, end_turn).
 - Provider prefix stripping ('anthropic:claude-sonnet-4-6' -> 'claude-sonnet-4-6').
 - Single tool_use parsing.
 - Multiple parallel tool calls in one block (the case that triggered
   the codex-proxy regression).
 - Fenced JSON inside <use_tools> block.
 - Model-omitted id gets synthesized to toolu_claude_cli_<rand>.
 - Malformed JSON falls back to text.
 - Unterminated block falls back to text.
 - AbortSignal SIGTERMs the child.
 - Error envelope rejected with informative message.
 - Non-JSON output rejected with raw-output excerpt in the error.
 - argv + cwd assertion: --disable-slash-commands + --system-prompt are
   present and cwd is the dedicated tmpdir.

Tests use a POSIX shell stub at GBRAIN_CLAUDE_CLI_BIN that emits a
scripted --output-format json envelope, so the suite runs without
claude-cli installed and without API credits.

* feat(ai): claude-cli recipe with native gateway integration (supersedes #334 baseline)

Replaces the MessagesClient adapter + GBRAIN_USE_CLAUDE_CLI=1 env-var
gate from the previous commit on this branch with a proper gateway recipe.
The recipe path gives per-call routing as a native capability: a model
string like `claude-cli:claude-sonnet-4-6` lands here while a sibling
`litellm:gpt-5.4` continues through the litellm-proxy / codex-proxy path
in the same worker. No global env-var switch, no agent.use_gateway_loop
bypass, no MessagesClient injection at jobs.ts worker startup.

The previous commit on this branch (jarvisdoes baseline) is preserved
in the history for #334 authorship attribution. Its functional changes
are backed out here because the recipe pattern is gbrain's established
integration seam; introducing a parallel MessagesClient + env-var path
would have created two routing mechanisms competing for the same job.

New: src/core/ai/recipes/claude-cli.ts
- Recipe declaration: id 'claude-cli', tier 'native', implementation
  'claude-cli', chat-only (no embedding or expansion touchpoints).
- Models: claude-opus-4-7, claude-sonnet-4-6, claude-haiku-4-5-20251001.
- supports_tools and supports_subagent_loop both true.
- supports_prompt_cache false because the CLI handles caching internally
  and does not surface cache_control via the standard control plane.
- auth_env.required is the empty array because the CLI owns auth (OAuth
  session managed by `claude login`).
- Friendly aliases mirror the `anthropic` recipe: `sonnet`, `haiku`,
  `opus` and the same legacy-id rewrites for back-compat with stale
  config strings.

New: src/core/ai/providers/claude-cli-language-model.ts
- ClaudeCliLanguageModel class implementing the ai-sdk LanguageModelV2
  interface.
- doGenerate: renders the ai-sdk prompt array into a system text + user
  text, injects the use_tools protocol instructions when tools are
  present, spawns `claude --print --output-format json --model <X>
  --disable-slash-commands --system-prompt <gbrain prompt>` from a
  dedicated tmpdir (contamination suppression: no local CLAUDE.md
  auto-discovery), parses the JSON envelope, extracts <use_tools>
  blocks, and returns ai-sdk-shaped LanguageModelV2Content (text +
  tool-call parts with stringified-JSON input matching the V2 contract).
- Tolerates fenced JSON inside use_tools blocks, malformed JSON
  (falls back to text), missing close tag (falls back to text),
  model-omitted ids (synthesizes toolu_claude_cli_<rand>).
- Parallel tool calls in one block round-trip cleanly: this is the
  case that drops IDs on the litellm + codex-proxy bridge today.
- AbortSignal SIGTERMs the child for proper cancellation.
- doStream throws not-supported (gateway.toolLoop is non-streaming).

Modified: src/core/ai/gateway.ts
- Adds case 'claude-cli' to instantiateChat (returns ClaudeCliLanguageModel).
- Adds case 'claude-cli' to instantiateExpansion (same wrapper, reserved
  for a future expansion touchpoint declaration).
- Adds case 'claude-cli' to instantiateEmbedding (throws, no embedding
  model, mirrors the native-anthropic path).
- Lazy require() at the call site keeps the gateway module load cheap
  for users who never use the claude-cli path.

Modified: src/core/ai/recipes/index.ts
- Registers `claudeCli` in the ALL[] array next to `anthropic`.

Modified: src/core/ai/types.ts
- Adds 'claude-cli' to the Implementation union so the gateway switch
  is exhaustive at compile time.

Reverted: src/commands/jobs.ts
- Drops the GBRAIN_USE_CLAUDE_CLI=1 env-var gate the prior commit
  added. Routing now happens at the gateway based on the model string.

Deleted: src/core/minions/handlers/claude-cli-adapter.ts
- The MessagesClient adapter is superseded by the recipe + LanguageModelV2
  path. Two routing mechanisms competing for the same job would have
  forced users to reason about which one wins; the recipe is the single
  source of truth.

New file: test/claude-cli-recipe.test.ts (16 tests, 46 assertions):
- Recipe registration: getRecipe returns chat-only Recipe; aliases map
  short names (sonnet/haiku/opus) to canonical model ids.
- Text round trip: single text content block, usage propagation, stop
  finish reason.
- Provider prefix stripping.
- Single tool-call parsing.
- Multiple parallel tool calls in one block.
- Fenced JSON inside the block.
- Model-omitted id synthesizes toolu_claude_cli_<rand>.
- Malformed JSON falls back to text + stop reason.
- Unterminated block falls back to text + stop reason.
- Tools offered but model declines: returns text-only with stop reason
  so the gateway-loop treats it as a final answer rather than wedging
  for tool calls that never come.
- AbortSignal SIGTERMs the child.
- is_error envelope rejected.
- Non-JSON output rejected.
- doStream throws.
- argv + cwd assertion: --print, --disable-slash-commands,
  --system-prompt are present and cwd is the dedicated tmpdir.

Tests use a POSIX shell stub at GBRAIN_CLAUDE_CLI_BIN so the suite runs
without claude-cli installed and without API credits.

End-to-end smoke verified against a real `claude --print --model haiku`
invocation: model emitted `<use_tools>` block with toolu_add_001 +
{"a":12,"b":30}, adapter parsed back into a `tool-call` content block,
finishReason 'tool-calls'.

* feat(ai/claude-cli): harden subagent isolation, env scrub, verbose + stdin robustness

Four defensive fixes to the claude-cli provider so a subagent call behaves
identically regardless of the host's ambient Claude Code config:

- Agent isolation: pass `--tools ''` and `--strict-mcp-config` so the subprocess
  runs as a raw LLM with no built-in tools and no inherited user MCP servers.
  Without `--strict-mcp-config`, each call boots the user's MCP servers (including
  gbrain's own), causing recursion plus PGLite single-writer lock contention.
- Env scrub: drop ANTHROPIC_API_KEY / ANTHROPIC_AUTH_TOKEN / ANTHROPIC_BASE_URL
  from the child env so the CLI authenticates via its own OAuth subscription
  session. An inherited API key silently flips billing to per-token API usage,
  the exact setup this recipe exists to replace.
- Verbose-mode compat: with `"verbose": true` in ~/.claude/settings.json,
  `--print --output-format json` emits an event array instead of a bare result
  object. Tolerate both shapes and select the result event.
- stdin robustness: handle the child stdin 'error' event and wrap write/end so a
  missing binary (ENOENT) or early child death (EPIPE) rejects cleanly instead of
  crashing the worker with an unhandled error.

Adds unit coverage for the env scrub, the isolation argv, and the verbose event
array. Verified against claude CLI 2.1.x.

* test(ai/claude-cli): cover verbose-array no-result + missing-binary reject paths

Two error branches in the hardened claude-cli provider had no coverage: the
verbose event-array path when no result event is present, and a missing binary
surfacing as a clean spawn-failed rejection. The missing-binary case is the
deterministic form of the stdin/EPIPE robustness; a synchronous stdin-write
throw is not reliably triggerable in a unit test, so the real ENOENT path the
handlers defend is exercised instead. Both reuse the existing shell-stub harness.

---------

Co-authored-by: jarvisdoes <258486803+jarvisdoes@users.noreply.github.com>
Co-authored-by: Marco Maldonado <34176133+loweaxerium@users.noreply.github.com>
2026-07-23 02:08:46 -07:00
Rafael ReisandRafael Reis c0cb6c533b fix(queue): dead/cancelled jobs no longer block idempotency re-submission (#2253)
queue.add() with an idempotency_key returns any existing row regardless
of status. This means dead jobs (exhausted retries from a transient
provider outage) permanently block re-submission of the same work —
even after the underlying issue is fixed.

Fix: when the existing row is dead or cancelled, NULL its
idempotency_key (preserving the row for audit) and fall through to the
INSERT path so a fresh job can be created.

Affects dream synthesize children that died during provider migrations
(429 rate-limit on old Anthropic proxy, tool-results-missing on old
OpenRouter). 45 dead children were blocking re-synthesis of transcripts
in production.

Includes 4 new tests covering dead, cancelled, completed, and active
status interactions with idempotency dedup.

Co-authored-by: Rafael Reis <rafael.reis@contabilizei.com.br>
2026-07-23 02:08:40 -07:00
caioribeiroclw-pixel d67be8b570 Reject unknown init flags before migrations (#2201) 2026-07-23 02:08:35 -07:00
Garry Tan a356f64e4f Revert "fix(autopilot): export ~/.bun/bin onto PATH in cron wrapper (#2013)"
This reverts commit b928f40bcd.
2026-07-23 01:07:57 -07:00
Garry Tan a1dadebd60 Revert "fix(extract): recognize reference wikilinks (#2071)"
This reverts commit 49cf5202cb.
2026-07-23 01:07:57 -07:00
Garry Tan c43ed81c72 Revert "fix(cycle): extract_atoms stamps concepts so synthesize_concepts has material (#2123) (#2124)"
This reverts commit f065eb1509.
2026-07-23 01:07:57 -07:00
Garry Tan 1d0df706fe Revert "fix(cycle): tombstone zero-yield pages so extract_atoms stops rediscovering them (#2144) (#2145)"
This reverts commit a8a94f5742.
2026-07-23 01:07:57 -07:00
Garry Tan 8078c46ab7 Revert "fix(doctor): stop claiming "Brain is at target" when the target is unreachable (#2151)"
This reverts commit 74358329e1.
2026-07-23 01:07:57 -07:00
Garry Tan e20a6a5328 Revert "feat(recipes): add reranker touchpoint to OpenRouter (#2164)"
This reverts commit 1a449bf501.
2026-07-23 01:07:57 -07:00
Ryan XieandHippityy 1a449bf501 feat(recipes): add reranker touchpoint to OpenRouter (#2164)
OpenRouter's POST /api/v1/rerank is wire-compatible with gateway.rerank()
({query, documents, model} → {results: [{index, relevance_score}]}). This
adds a recipe-only reranker touchpoint declaring four models:

  - cohere/rerank-v3.5          (default; $0.001/search)
  - cohere/rerank-4-fast        ($0.002/search, 32K context)
  - cohere/rerank-4-pro         ($0.0025/search, SOTA quality)
  - nvidia/llama-nemotron-rerank-vl-1b-v2:free  (multimodal)

Unlike embedding/chat, the reranker path strictly enforces the models
allowlist — the openai-compat extended-model bypass does not apply. New
rerank models must be added to this recipe before they can be called.

The cost_per_1m_tokens_usd value is a pseudo-rate for the budget tracker's
chars/4 heuristic — Cohere bills per-search, not per-token. At ~4K chars
the estimated cost is in the right ballpark.

Recipe-only change; no gateway or search-layer modifications. gateway
auto-concatenates path → .../api/v1/rerank.

Adds hermetic unit test (test/openrouter-reranker-recipe.test.ts) covering
shape, models, default_model, path, max_payload_bytes, default_timeout_ms,
and cost field. No DB, no env mutation — survives the parallel 8-shard
fan-out.

Verified: bun run verify (30/30 green); 285 targeted recipe+rerank+budget
tests pass.

Co-authored-by: Hippityy <Hippityy@users.noreply.github.com>
2026-07-22 18:51:09 -07:00
Brett 74358329e1 fix(doctor): stop claiming "Brain is at target" when the target is unreachable (#2151)
`gbrain doctor --remediation-plan` printed two consecutive lines that
contradicted each other when the brain was below target AND the target
was unreachable with autonomous remediation:

    Brain score: 45/100 → target 90
    Target unreachable: max with autonomous remediation is 70/100.
    No remediations needed. Brain is at target.

The second sentence hid the real next step (configure the prereqs that
would lift `max_reachable_score`) and made the brain look healthy when it
was not.

Fix: gate the "Brain is at target" line on `brain_score_current >=
targetScore`. When the plan is empty AND the brain is below target, the
"Target unreachable" line above is already the user-facing explanation;
the `Blocked checks` block below surfaces the manual gap.

Extracted `renderRemediationPlanLines(plan, targetScore): string[]` as a
pure helper alongside `runRemediationPlan` so the regression coverage
asserts on the rendered output directly rather than mocking
`console.log`. `runRemediationPlan` now joins the lines verbatim through
console.log; behavior is byte-identical for every case other than the
fixed contradiction.

Five regression tests cover: unreachable-and-below-target (the bug
case), reachable-and-at-target, exact-target, below-target-with-plan,
unreachable-with-partial-plan. 38 tests across the adjacent doctor test
files stay green; `bun run typecheck` clean.
2026-07-22 18:51:04 -07:00
a8a94f5742 fix(cycle): tombstone zero-yield pages so extract_atoms stops rediscovering them (#2144) (#2145)
Idempotency was keyed on atom rows alone — a page the LLM judges
un-atomizable leaves no row, so it re-entered the discovery window every
run. Two production consequences: --drain false-stopped with
no_progress once the window head was mostly zero-yield pages (remaining
frozen while batches report +0), and every nightly re-spent extraction
budget on the same pages.

Fix:
- After a SUCCESSFUL chat call that parses to zero atoms, stamp the
  source page with frontmatter.atoms_scan_hash = contentHash16. LLM
  failures take the catch path and stay retryable.
- discoverExtractablePages + countExtractAtomsBacklog (both variants)
  exclude pages whose stamp matches the CURRENT content hash prefix —
  content edits re-eligibilize, mirroring atom-row staleness semantics.
- Drain no_progress now recounts the backlog on a zero-atom batch and
  only stops when it genuinely didn't shrink — tombstoning IS progress.

Tests: +2 pure-loop drain cases (shrinking backlog continues / flat
backlog stops) and +3 PGLite integration cases (stamp + exclusion /
content-change re-eligibility / failed chat does not stamp).
29 pass / 0 fail across the two files; tsc clean.

Co-authored-by: 陈源泉 <chenyuanquan@chenyuanquandeMac-mini.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 18:50:59 -07:00
f065eb1509 fix(cycle): extract_atoms stamps concepts so synthesize_concepts has material (#2123) (#2124)
synthesize-concepts.ts's design comment says extract_atoms stamps a
`concepts:` frontmatter field on each atom and :92 consumes ONLY that
field — but the extractor never wrote it, so the atoms → concepts
pipeline was dead end-to-end: every cycle reported "synthesize_concepts:
skipped — no atoms with concept refs" no matter how many atoms
accumulated (696 page-derived atoms / 0 with concepts on our production
brain before an external backfill).

Fix, all on the extractor side (no synthesize change needed):
- EXTRACT_PROMPT asks for `concepts` (1-3 kebab-case TOPIC labels) with
  an explicit reuse-over-coinage instruction — labels must cluster,
  since synthesize_concepts only materializes groups of >=2.
- parseAtomsResponse validates labels (kebab regex, max 3, drop
  invalid; empty -> undefined).
- The putPage frontmatter write stamps `concepts` alongside lesson /
  source_quote.

Tests: 4 parse cases + an end-to-end regression that goes extractor ->
real frontmatter -> synthesize_concepts' OWN DB query path -> concept
page. The existing tests fed synthesize via the `_atoms` seam, which is
exactly how this gap survived.

Validated in production ahead of this PR by stamping the same shape
externally: the next synthesize_concepts run wrote 33 concept pages
(T2=7/T3=26) from 60 stamped atoms, zero failures.

Co-authored-by: 陈源泉 <chenyuanquan@chenyuanquandeMac-mini.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 18:50:54 -07:00
mzkarami 49cf5202cb fix(extract): recognize reference wikilinks (#2071) 2026-07-22 18:50:49 -07:00
klampatech b928f40bcd fix(autopilot): export ~/.bun/bin onto PATH in cron wrapper (#2013)
The wrapper script that 'gbrain autopilot --install' writes to
~/.gbrain/autopilot-run.sh sources ~/.bashrc to inherit PATH for the
exec'd gbrain binary (which has a '#!/usr/bin/env bun' shebang). The
standard Debian/Ubuntu ~/.bashrc ships a non-interactive guard that
returns early when bash is launched non-interactively (cron, launchd,
systemd) — so PATH exports operators add to ~/.bashrc never reach the
wrapper subprocess.

The result: the wrapper dies silently with 'env: bun: No such file or
directory', leaves a stale lockfile, and every subsequent cron tick
hits the lockfile and bails. The nightly dream cycle hangs waiting on
a worker that never comes back, and the wrapper's own 10-min
stale-lock window is the only thing that can recover it.

This bites every operator whose bashrc is the standard distro default
(which is the default), and there is no warning at install time.

Fix: prepend ~/.bun/bin to PATH directly in the wrapper, so it is
self-contained regardless of which init file the OS loaded. Add a
regression test alongside the existing zshenv/zshrc source-order test
(v0.36.1.x #966) so this class of bug stays caught.
2026-07-22 18:49:05 -07:00
Elliot DrelandClaude Opus 4.8 bb5a66942d fix(doctor): drop dead llm_fallback_enabled recommendation from conversation_format_coverage (#1903)
The conversation_format_coverage check recommended `gbrain config set
conversation_parser.llm_fallback_enabled true`, but that config key is dead
(never read) — see #1890. The recommendation is a no-op and misleads users into
thinking a fallback will kick in. Drop it; keep the actionable `scan` hint.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-22 18:47:03 -07:00
6cf4f3122d fix(jobs): backlinks worker defaults to check, not fix (#1853)
Backlinks Minion jobs submitted with an empty payload (the
sync→embed→backlinks chains enqueued after every ingestion) defaulted to
action='fix', rewriting tracked brain pages with generated "Referenced in"
timeline bullets on every routine run — 129 vault files polluted in one
day on our production brain before we traced it.

This contradicts the documented intent in src/core/cycle.ts
(runPhaseBacklinks): "Maintenance cycles must not rewrite tracked brain
pages with generated 'Referenced in' timeline bullets. [...] the legacy
filesystem fixer remains available explicitly via `gbrain check-backlinks
fix`." — the jobs-worker handler simply inverted that default.

Fix: default to 'check'; 'fix' requires explicit opt-in via
'{"action":"fix"}' (the documented submit shape) or
`gbrain check-backlinks fix`. Both explicit paths are unchanged.

Adds a structural regression test (fix-wave-structural.test.ts precedent)
pinning the default, since the handler dynamically imports
runBacklinksCore and walks a real repo dir — a behavioral test would
require mocking that hides the regression behind a test seam.

Co-authored-by: Valentin Ferriere <valentin@v-labs.fr>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 18:46:58 -07:00
Khaja NazimuddinandClaude Opus 4.8 6cf8d8d66c fix(extract): clear pre-version-bump pages in extract --stale (#1791)
`extractStaleFromDB` stamped `links_extracted_at` with each page's read
`updated_at` (the D4 race-fix). But the stale predicate also flags
`links_extracted_at < LINK_EXTRACTOR_VERSION_TS`. Any page last edited
BEFORE the version timestamp got stamped below the threshold, so the
version arm re-flagged it stale on every run — an infinite re-extract
loop that never cleared the lag.

Since the v112 watermark column ships with no backfill, every
pre-existing page starts stale, and most pre-date the version bump. In
practice this left ~97% of pages permanently stale: `extract --stale`
reported "done" each run but `links_extraction_lag` never dropped.

Fix: stamp `GREATEST(read updated_at, versionTs)`. Old pages lift to the
threshold so the version arm clears; a real future edit still advances
`updated_at` past the stamp, so the CDX-1 edited-after-stamp race
protection is preserved.

Adds a regression test: a page with `updated_at` before
LINK_EXTRACTOR_VERSION_TS must clear after extract AND stay clear on a
second run (the existing tests only used now()-dated pages, so the
old-page case was uncovered).

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-22 18:46:54 -07:00
sonlndvandSon Le 355fbc6947 fix(doctor): two false-positive/timeout fixes — drift walk skips node_modules; bare-tweet skips inline-code + cited lines (#1772)
* fix(drift): skip node_modules/dist/build in multi-source drift walk

The drift walker recursed into node_modules (50k+ files in RN/Astro repos),
exhausting the time budget before completing, so multi_source_drift always
reported 'walk hit limit/timeout' on real projects. Skip heavy non-content
dirs + add a deadline check on directory descent.

* fix(integrity): skip inline-code spans + [Source:] citations in bare-tweet detection

Recipe/doc pages that show the CORRECT citation format inline (e.g.
`Tweeted about {topic} [Source: X, @handle, date]`) were false-flagged.
The fenced-code skip didn't cover inline backticks; add inline-code
stripping + an explicit-citation exemption.

---------

Co-authored-by: Son Le <tuanson1200@gmail.com>
2026-07-22 18:46:49 -07:00
Aurora Capital 2c96787867 test(e2e): harden suite — kill flakes, no-op assertions, cross-test coupling (#1704)
* test(e2e): drop flaky wall-clock bounds in minions-resilience

The runaway-dead-letter and cascade-kill tests asserted tight real-clock
upper bounds (<2000ms, <3000ms) on top of already-complete terminal-state
checks. Those bounds carry no correctness signal — the dead/cancelled status
and abortedChildren==10 assertions fully prove behavior — and flake on loaded
CI runners where the stall/timeout sweep cadence varies. Removed both bounds,
kept the diagnostic values, de-promised the test titles.

* test(e2e): kill order-dependence + no-op assertions in mechanical

- traverse_graph: self-contained (re-adds its own idempotent link) and asserts
  the linked company is reachable, instead of depending on a prior it() and
  only checking array shape.
- file_list-without-slug: seeds its own >100 rows instead of relying on the
  previous test's 150 surviving in the DB; asserts the cap is exercised.
- precision@5: add a loose floor (every known-item query surfaces >=1 truth doc
  in top-5) so a 0% retrieval regression no longer passes silently.
- get_health: assert value bounds (page_count==16, embed_coverage 0..1) not just
  typeof; get_chunks: assert non-empty text, numeric non-decreasing chunk_index,
  and that the page name appears, instead of toBeTruthy on chunks[0].

* test(e2e): strengthen graph/search quality assertions + close coverage gaps

graph-quality: truncate+reseed 'config' in truncateAll (kills config leak where a
setConfig test throwing before its finally bleeds into later tests); replace
toBeGreaterThan(0) link/timeline floors with fixture-derived minimums; assert exact
attendee slugs are 'attended' instead of a vacuous .every; pin autoLinks.created to
the provable 2 (Alice+Acme); add direction out/both + depth:2 multi-hop and a
cycle-safety (A->B->A terminates) test.
search-quality: fix the vacuous detail=low vector test; assert pedro returns >=2
chunks; assert detail=high includes the timeline chunk; add empty-query and
zero-vector no-throw edge tests.

* test(e2e): self-contain multi-source sync test + assert ledger cascade

Break the sequential dependency where 'performSync no sourceId' relied on a prior
test writing sync.repo_path — it now sets its own config. Add the missing
file_migration_ledger COUNT(*)==0 cascade assertion. Tighten the source_id default
check from toContain('default') to exact "'default'::text".

* test(e2e): make migration-flow HOME/PATH swap throw-safe

The suite repoints process.env.HOME/PATH to a temp dir and only restored them in
afterAll, so a mid-test throw left HOME dead for the rest of the bun process and
silently broke sibling suites. Wrap each test body in try/finally restore + a
defensive restore at the top of beforeEach.

* test(e2e): loud-skip jsonb-roundtrip + doctor-progress

Both skipped silently with no DATABASE_URL, giving zero signal the regression guard
never ran. Add the console.log skip line matching the sibling e2e files.

* test(e2e): robust check-update contract + find_orphans tool coverage

upgrade: the 'no-releases' test hard-asserted update_available===false, which flips
to failing the moment the repo has a real release. Assert the JSON contract shape
(boolean update_available, current_version===VERSION, typed optional fields) instead.
mcp: add find_orphans to the asserted generated tool names.
2026-07-22 18:46:44 -07:00
The Lord ArgusandThe Lord Argus 8a5296f3cb fix: merge provider base URL config from DB (#1676)
Co-authored-by: The Lord Argus <215461619+TheLordArgus@users.noreply.github.com>
2026-07-22 18:46:39 -07:00
Lubos BuracinskyandClaude Opus 4.8 8837bfe5f2 fix(chunker): cap oversized code chunks so they stay embeddable (#1675)
splitLargeNode can only break up a node that exposes a `body` with >= 2
named children. A node without one -- a giant object/array literal, a single
huge assignment, a massive template literal -- is emitted whole. On real
source that yields a chunk far larger than the embedder's context window; the
embedder then rejects it ("input exceeds context length") and it is never
embedded. Example: a 372 KB service file produced 113 chunks, one a single
281 KB (~70k-token) node -> permanently unembedded.

Add a final safety-net pass (capOversizedChunks) that recursively re-splits
any chunk over a token budget (default 2000, configurable via maxChunkTokens),
with a hard character split as a last resort for no-whitespace content
(minified one-liners). Normal files are untouched.

Verified: that 372 KB file now yields 188 chunks, max ~1.5k tokens, zero
oversized; a small file is unchanged.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 18:46:34 -07:00
e1526bfebe fix(think): render the Gaps section once instead of twice (#1662)
gbrain think printed "## Gaps" twice: the synthesis prompt asked the model for a Gaps section inside the answer body AND a separate structured gaps array, then both render paths printed both — the CLI human output (src/commands/think.ts) and the --save page (persistSynthesis in src/core/think/index.ts).

Make the structured gaps array the single source. The prompt now routes gaps into the array, not an answer-body section. New exported stripGapsSection(answer) defensively removes any "## Gaps" section a model still emits (any heading level, case-insensitive, bounded by the next same/higher heading); both render sites call it, so the dedup is structural rather than dependent on the model obeying the prompt.

Adds test/think-gaps.test.ts (hermetic): strip helper across heading levels / case / no-section / mid-document / false-match, the one-render-only repro, and the prompt contract.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-07-22 18:46:29 -07:00
44cae62324 fix(pglite): guard putPage against zero-row RETURNING (#1649)
PGLite can return zero rows from INSERT ... ON CONFLICT DO UPDATE ...
RETURNING in no-op/trigger edge cases. The previous code called
rowToPage(rows[0]) unconditionally, so rows[0] was undefined and
rowToPage threw "undefined is not an object (evaluating 'row.deleted_at')",
which aborted the import and silently skipped the file during sync.

getPage() already has the empty-rows guard; putPage() was missing the
parallel one. The row was in fact written by the upsert, so re-read it
via getPage() instead of crashing. On a real monorepo index this
recovered ~19% of files (985/5148) that were failing to embed.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-07-22 18:26:48 -07:00
56ccc14bcc fix(code-def): surface method/constructor/field/struct definitions (#1628)
DEF_TYPES listed only canonical symbol-type names (function, class, interface, ...). But normalizeSymbolType in the code chunker canonicalizes only some tree-sitter node types and lets the rest fall through type.replace(/_/g, ' '). So method_declaration is stored as 'method declaration', struct_specifier as 'struct specifier', protocol_declaration as 'protocol declaration'. None were in DEF_TYPES, so code-def returned 0 hits for every method, constructor, field, C struct, and Swift protocol. The plain 'struct' entry never matched either. Add the fallthrough definition forms. Read-path only; no reindex needed (0 -> N on existing indexes).

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-07-22 17:56:33 -07:00
e7ffbc057c fix(lint): code-fence-wrap detector and fixer regex now agree (#1597)
The code-fence-wrap detector in lintContent used the /m multiline flag, so
^/$ matched start/end of any line. The rule fired on any page that simply
contained a ```markdown code block, not only pages wrapped end-to-end.

The matching fixer in fixContent has no /m flag, so it can only strip
whole-file wrappers. Result: detected issues were marked fixable: true,
yet fixContent could never strip them. `gbrain dream` reported
"0 fix(es) applied, N remaining" perpetually for the rule.

Drops the /m flag from the detector so detector and fixer stay in sync.
Whole-file wrapper detection is preserved; inner code blocks no longer
trigger the rule.

Real-world impact: a brain with 5 docs pages containing markdown examples
(skill READMEs, decision registry, journal templates) reports 5 phantom
"fixable: true" issues every dream cycle, never converging. After this
fix, the dream-cycle lint phase reports only real-and-unfixable issues
(missing frontmatter, missing title/type) which is the intended behavior.

Two regression tests added in test/lint.test.ts:
- Page contains a single inner ```markdown block
- Page contains multiple inner ```markdown blocks

Both assert no code-fence-wrap issue is reported. The existing
"detects wrapping code fences" test (true-positive case) continues to
pass; total tests in the file are 18 -> 20.

Co-authored-by: Thomas Chung <thomaschung@macbookair.lan>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-07-22 17:19:29 -07:00
0xTimandTime Attakc 0a757bf780 fix(entities): thread sourceId through findByTitleFuzzy + skip soft-deleted (#1508)
`findByTitleFuzzy` on both `postgres-engine.ts` and `pglite-engine.ts`
has no `source_id` filter and no `deleted_at IS NULL` filter. `tryFuzzyMatch`
in `src/core/entities/resolve.ts` got both of those filters via #1436
(v0.41.13.0) for exactly the reasons that apply to its sibling here:
fuzzy resolution can suggest cross-source slug candidates that the
caller then silently drops at the FK filter (or worse, picks a
soft-deleted page).

This is the missing twin of #1436. In multi-source brains, the live-mode
auto-link resolver invoked from `put_page` (`operations.ts:937`) calls
`engine.findByTitleFuzzy` with no scope. When two sources contain pages
with similar titles (`people/alice-example` on `source-a`,
`people/alice-other` on `source-b`), the fuzzy lookup can return the
wrong-source slug, which then fails the downstream `allSlugs` /
`addLink` FK filter — the link silently doesn't get created, and from
the caller's view the resolver "failed" even though the page existed
under the right source.

Reproducible with a 2-source PGLite setup + identical-title pages on
both sides; the fuzzy call returns a slug whose `source_id` doesn't
match the put_page caller's source.

- 2-source brains: auto-links between same-title-different-source pages
  now resolve under the caller's source instead of the wrong neighbor.
- Soft-deleted pages can no longer be returned as fuzzy candidates
  (mirroring the resolve.ts fix from #1436).
- 1-source brains: no behavior change. `sourceId` is optional; when
  omitted the SQL takes the pre-existing unscoped path.

- `engine.ts`: add optional 4th `sourceId` param to the
  `findByTitleFuzzy` interface + JSDoc explaining the scope semantics.
- `postgres-engine.ts` / `pglite-engine.ts`: implement the param via a
  conditional SQL branch that adds `AND source_id = $N AND
  deleted_at IS NULL` when `sourceId` is set; existing query path
  unchanged when omitted.
- `link-extraction.ts`: add optional `sourceId` to `makeResolver` opts,
  forward to `findByTitleFuzzy` in step 3 of the resolve chain.
- `operations.ts`: pass `opts?.sourceId` to `makeResolver` from the
  live-mode put_page resolver (the place that already knows the
  caller's source).

- New unit tests in `test/link-extraction.test.ts` (2 cases):
  - `opts.sourceId` is forwarded to `findByTitleFuzzy` when set.
  - `opts.sourceId` omitted → `findByTitleFuzzy` receives `undefined`
    (back-compat).
- `bun run typecheck` clean.
- `bun test test/link-extraction.test.ts test/entity-resolve.test.ts
  test/operations.test.ts test/extract.test.ts` — 145/145 pass.

Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-07-22 16:50:52 -07:00
Benjamin D. SmithandTime Attakc 2b020ba2bd fix(models): dispatch subcommand reads args[0] not args[1] (#1428)
* fix(models): dispatch subcommand reads args[0] not args[1]

`gbrain models doctor` silently fell through to the read view
instead of running the reachability probe.

`runModels` checks `args[1] === 'doctor'`, but the caller —
`handleCliOnly(command, subArgs)` in `src/cli.ts:113` — passes
`subArgs` (the leading command token already stripped). So inside
`runModels`, args[0] is the subcommand. args[1] is undefined.

The doctor probe path has been unreachable from the CLI since the
handleCliOnly refactor. `gbrain models help` happened to work by
falling through to the `--help` flag detection.

Two-char fix: `args[1]` → `args[0]` on both branches of the
ternary.

Verified by manual probe — `gbrain models doctor` now prints
"Model reachability probe:" with per-model results (real production
brain, 4 touchpoints probed):

```
Model reachability probe:
  embedding_config  ollama:bge-m3                              ok (0ms)
  reranker_config   (none)                                     ok (0ms)
  chat              lmstudio:mistralai/magistral-small-2509    unknown (5012ms)
      [chat(lmstudio:mistralai/magistral-small-2509)] probe timed out after 5s
  expansion         lmstudio:google/gemma-4-e2b                ok (535ms)
Summary: 3/4 reachable.
```

RECOVERY REBUILD 2026-05-26 of original 20ed0eee.

* fix: honor --help before doctor dispatch to avoid running probes on `models doctor --help`

Codex review of #1428 flagged that the args[1]→args[0] rewrite
regressed `gbrain models doctor --help` into running network
probes instead of printing usage. The original args[1]-shaped
ternary happened to dodge this by always falling through to the
args.includes('--help') branch when args[1] === 'doctor' was
false; the new args[0] code checks doctor first, so --help no
longer wins.

Reorder ternary: `hasHelp` is computed FIRST from
(--help / -h / args[0] === 'help'), then the sub is hasHelp ?
'help' : args[0] === 'doctor' ? 'doctor' : 'read'.

Addresses codex review P2 on PR #1428.

---------

Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-07-22 16:17:06 -07:00
d43fb631bc fix(serve-http): add resource_metadata to WWW-Authenticate per MCP spec + RFC 9728 (#1410)
The HTTP MCP server's 401 responses missed the `resource_metadata`
parameter in the WWW-Authenticate header. MCP authorization spec
(2025-06-18 draft §5.1) and RFC 9728 require:

  WWW-Authenticate: Bearer resource_metadata="<url>"

MCP-aware OAuth clients (claude.ai, Cursor, etc.) use that URL to find
the authorization-server discovery doc without the user manually
configuring the issuer. Pre-fix the header shipped only `Bearer
error="invalid_token", error_description="..."` and MCP clients silently
failed to begin the OAuth flow — symptom on claude.ai's UI was "Couldn't
reach the MCP server" even when discovery + /token + /register all
responded 200 individually.

The `requireBearerAuth` middleware in @modelcontextprotocol/sdk's
BearerAuthMiddlewareOptions already supports a `resourceMetadataUrl`
parameter. Two call sites (`/mcp` and `/ingest`) now pass it.

Verified against a real claude.ai connector attempt: pre-fix the
connector showed "Couldn't reach the MCP server" with no OAuth redirect.
Post-fix the connector successfully begins the authorization flow.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-07-22 15:52:59 -07:00
mmekkaouiandTime Attakc 292b8b1637 fix(ai): cap llama-server embedding batches at its 32-input request limit (#1281)
llama.cpp's llama-server rejects /v1/embeddings requests with more inputs
than its launch --batch-size (default 32): "batch size 100 > maximum allowed
batch size 32". gbrain sends batches of 100, so any page with >32 chunks fails
to embed, and embed --stale then trips the Postgres statement_timeout retrying
the doomed batches. The existing token-based protection (max_batch_tokens)
can't bound item count — N tiny chunks fit under any token budget.

Add an optional max_batch_items count cap to EmbeddingTouchpoint, enforced as a
hard re-split after the token split in embed(), and set it to 32 on the
llama-server recipe (replacing no_batch_cap: true, which wrongly assumed
llama.cpp has no per-request item cap). A declared item cap also suppresses the
missing-max_batch_tokens startup warning.

Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-07-22 15:22:09 -07:00
e78ad9ff9e fix(salience): exclude briefings/* from their own Brain Pulse (TIM-37) (#1202)
The cron daily briefing writes 90_Briefings/<date>.md, which gets
re-ingested on the next sync and then dominates tomorrow's
getRecentSalience output as pure self-reference (observed: top
result score 0.9956, everyone else clustered at 0.587).

Filter `p.slug LIKE 'briefings/%'` out of getRecentSalience in both
the PG and PGLite engines. Suppressed by default; callers can still
opt in by passing `slugPrefix: 'briefings/'` (or `--kind briefings/`
from the CLI). search and list_pages are unaffected.

Co-authored-by: CTO <cto@timelycare.local>
Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-07-22 14:47:11 -07:00
2840734d70 fix(doctor): normalize CRLF in extractTriggers so Windows skill triggers parse (#1149)
On Windows, `core.autocrlf=true` is the default and SKILL.md files are
checked out with CRLF line endings. `extractTriggers` used regexes
anchored to `\n` (`/^---\n.../` and `/^triggers:\s*\n.../`), which
never matched `\r\n`, so the parser returned `[]` for every skill.

Result: `gbrain doctor --fast --json` on Windows reported every skill
not in `OVERLAP_WHITELIST` (39 of 42) as a false `mece_gap` warning —
even though `skill_conformance` in the same run reported "42/42 skills
pass". CI runs Ubuntu-only so the divergence never surfaced.

Fix: normalize CRLF → LF at the top of `extractTriggers`. Single-line
change preserves existing LF behavior. Function is now exported so the
test can target it directly.

Tests: added `describe("extractTriggers")` block covering LF input,
CRLF input (regression case), missing frontmatter, missing triggers
field, and quote-stripping. All 30 tests in `check-resolvable.test.ts`
pass.

Verified locally on Windows: `gbrain doctor --fast --json` now reports
`resolver_health: ok, 42 skills, all reachable` (health_score 90 → 95).

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-07-22 14:13:54 -07:00
d69f211629 fix(ci): delta-assert reporter leak test + raise shard timeout to 22min (#3231)
The signal-handler test asserted an absolute liveReporters===0 on a
module-global set, so any other test file in the shard holding a live
reporter flaked it — it red-flagged ~12 unrelated PR runs and one master
push in two days, purely as a function of shard composition. The delta
form pins the same claim (50 reporter lifecycles leak nothing).

The 15-minute shard timeout cancelled 13 fully-passing runs under
parallel PR load (PGLite WASM cold-starts stretch shards); the
test-status gate then reported the cancellations as failures.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 13:41:32 -07:00
02ba4b4fc2 dims: thread Matryoshka dimensions for Qwen3-Embedding on Ollama (#1072)
Qwen3-Embedding family on Ollama supports Matryoshka truncation via the
'dimensions' field on /v1/embeddings. Without this passthrough, gbrain
ignores user-selected reduced dims and the provider returns its native
size, causing dim-mismatch errors against brains configured for narrower
widths (e.g. existing 1536-dim brains).

Matches by bare name 'qwen3-embedding' or any tag variant
'qwen3-embedding:0.6b' / ':4b' / ':8b'.

Native dims: 0.6B=1024, 4B=2560, 8B=4096. All MRL-truncatable.

5 new tests; full AI suite 137/137 green.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-07-22 12:38:30 -07:00
d9eb027bdd fix(openclaw): declare gbrain plugin manifest entry (takeover of #2551) (#3185)
Add the OpenClaw-required top-level id to openclaw.plugin.json, export a
direct register(api) entrypoint from src/openclaw-context-engine.ts, add a
manifest regression test, and document that skillpack harvest must preserve
OpenClaw-native manifest fields (id, configSchema, contracts).

llms bundles regenerated (bun run build:llms) — no content drift.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Filip <FilipHarald@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 01:32:11 -07:00
62e009d192 fix(skillopt): emit proposed.md in no-mutate mode (#2635) (#3182)
Takeover of #2719 (fork head; rebased onto origin/master).

- writeProposed now writes both best.md (current-best pointer) and
  proposed.md (stable human-review artifact); returns the proposal path.
- Orchestrator reports the real proposed.md path for accepted --no-mutate runs.
- Tutorial updated; llms bundles regenerated (no content drift — tutorial
  is not inlined in the bundle).

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Ziyang Guo <121015044+RerankerGuo@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 00:23:10 -07:00
314fefa560 fix(readme): correct broken OpenClaw and Hermes project links (#1961) (#3179)
Point the OpenClaw and Hermes anchors in the "Have your agent install
it" section at their real upstream repos; the previous openclawagents
org URLs 404. Regenerated llms-full.txt to match.

Takeover of #1961 (fork branch) rebased onto current master.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: jessems <jessems@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-21 20:22:27 -07:00
7f841fae7f feat(maintain): safe maintenance automation + shared orphan-exclusion policy (#3015) (#3023)
Ports #3015 by @jdewoski-cmd onto current master:

- src/core/orphan-policy.ts centralizes the orphan-reporting exclusion
  convention so `gbrain orphans`, doctor's orphan_ratio, and both engines'
  getHealth orphan_pages can no longer drift.
- getHealth stale_pages now uses the link-extractor stale watermark
  (countStalePagesForExtraction) so health agrees with what `gbrain extract
  --stale` will actually process.
- New `gbrain maintain` command: dry-run by default, `--safe` applies only
  the conservative runbook actions (DB-backed stale extraction + source-scoped
  dream cycles for doctor cycle_freshness findings), `--json` for structured
  before/action/after reports. Frontmatter mutations, schema-pack upgrades,
  and semantic hub links stay review-only by design.

Changed from the original PR: the shared defaults carried slugs specific to
the contributor's own brain ('josa-secrets/', '*-ga4-property-id.md',
'*-josa-test', literal 'welcome'/'untitled' fixtures). Global defaults now
carry only GBrain-wide conventions; brain-specific exclusions move to a new
per-brain config plane the policy reads through loadOrphanPolicyOverrides:

    gbrain config set orphans.exclude_prefixes "my-private-folder/,archive/"
    gbrain config set orphans.exclude_slugs "some-one-off-page"

Both engines' getHealth and the orphans command thread the overrides;
tests cover the neutral defaults, the override plane, and health parity.
Also registered `maintain` in CLI_ONLY_SELF_HELP so `gbrain maintain --help`
reaches the command's own usage block.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: jdewoski-cmd <jdewoski@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-21 19:52:18 -07:00
1fabbb9849 fix(links): resolve path-qualified wikilinks outside DIR_PATTERN in the DB/put_page path (#2866)
The generic wikilink pass (issue #972) forwarded the raw literal to
resolveBasenameMatches, whose index is keyed by final path segments —
so [[notes/struktura]] (any dir outside DIR_PATTERN) silently produced
zero edges from `extract links --source db` and put_page auto-link,
while the FS extractor resolves the identical content (resolveSlugAll
strips the dirname before its basename lookup).

Query by the literal's final segment, then keep only matches whose slug
ends with the written path — [[notes/struktura]] can resolve to
vault/notes/struktura but never attach to wiki/struktura. Bare literals
are untouched. Flag-gated by link_resolution.global_basename as before.

Co-authored-by: YMYD <53603073+OJ-OnJourney@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-07-21 18:45:23 -07:00
64920f83c9 fix(embed): preserve code-chunk metadata across re-embed (#769) (#1232)
Closes #769. Every re-embed pass clobbered code-chunk metadata
(language, symbol_name, symbol_type, start_line, end_line,
parent_symbol_path, doc_comment, symbol_name_qualified) to NULL,
disabling code-def queries across thousands of indexed chunks.

Two complementary fixes:

embed.ts — three re-upsert call sites (embedPage, embedAll
non-stale, embedAllStale autopilot path) build ChunkInputs from
loaded chunks; they were stripping the 8 metadata fields. New
preserveCodeMetadata helper threads those fields through
consistently. Integrated cleanly with v0.34.4.0's cursor-paginated
--stale hardening — the wrap sits inside the worker function
between embedBatchWithBackoff and engine.upsertChunks.

postgres-engine.ts + pglite-engine.ts — upsertChunks ON CONFLICT
clause OVERWROTE metadata columns from EXCLUDED. Asymmetric vs the
embedding/embedded_at columns which already used a chunk_text-gated
CASE pattern (re-chunk → trust EXCLUDED, re-embed → COALESCE
preserve). Applied the same pattern to all 8 metadata columns.

Three regression tests in test/embed.serial.test.ts cover --stale
(autopilot), --all, and --slugs paths. Each loads a chunk with
full metadata, runs runEmbed, and asserts engine.upsertChunks
receives the metadata round-tripped. Coexists with master's D5
embedBatchWithBackoff test block.

Backfill required after deploy: \`gbrain sync --strategy code
--force --source <id>\` per code source to re-populate metadata via
the chunker. Without backfill, existing NULL columns stay NULL —
re-embed alone never produces metadata, only the chunker does.

Originally landed as part of PR #768 (the wave that bundled #767 +
fix; this PR carries the #769 fix alone with no scope overlap.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-07-21 18:13:45 -07:00
Benjamin D. SmithandTime Attakc e861b92da7 feat(synopsis): tail-truncate documentText for small-model chat handlers (#1427)
* feat(synopsis): tail-truncate documentText for small-model chat handlers

Small local chat models (Gemma 4 E2B, Qwen3 4B) get dramatically
slower on long contexts even at 131K declared windows. A 73K-char
page synopsis on Gemma 4 E2B takes 60-120s, exceeding the worker's
default 30s `lockDuration` and tripping `lock-lost` errors.

Add `SYNOPSIS_DOC_MAX_CHARS` env-overridable cap (default 32768
chars, ~8K tokens) applied in `buildUserPrompt`. Truncate the TAIL
so the head (title, frontmatter, intro) preserves the document-level
anchor the synopsis needs.

Anthropic Haiku is unaffected at this cap; bump via
`GBRAIN_SYNOPSIS_DOC_MAX_CHARS` for frontier models that want
richer document anchoring.

Belt-and-suspenders companion to commit 0aaff691 (--lock-duration
flag on the worker). Combined: bumping lock TTL gives the handler
more time, AND truncating doc cap makes the handler complete faster.
Either alone helps; both together get the synopsis backfill running
reliably on small local LLMs.

Verified: real 1383-chunk personal brain backfill at
GBRAIN_SYNOPSIS_MODEL=lmstudio:google/gemma-4-e2b +
GBRAIN_SYNOPSIS_DOC_MAX_CHARS=16384 +
`gbrain jobs work --concurrency 4 --lock-duration 300000`
transitions from "lock-lost on every transcript page" to "no
deaths, no stalls, steady throughput."

RECOVERY REBUILD 2026-05-26 of original ac213aa6.

* fix: fold SYNOPSIS_DOC_MAX_CHARS into corpus_generation hash

Codex review of #1427 flagged that changing GBRAIN_SYNOPSIS_DOC_MAX_CHARS
shifts the synopsis prompt + downstream embeddings for long documents
but was NOT folded into the computeCorpusGeneration hash. Pages
re-embedded with a different cap would retain the same
corpus_generation, defeating the v0.40.3.0 D27 P1-5 cache invalidation
contract.

Three changes:

1. Export SYNOPSIS_DOC_MAX_CHARS from src/core/page-summary.ts
2. computeCorpusGeneration accepts optional synopsisDocMaxChars param.
   When set, folded into hash via '|doc_cap=<N>'. Omitted for
   non-synopsis modes (title / none don't consult the cap) so existing
   pre-PR caches stay valid for those.
3. Service-layer call sites (2 in contextual-retrieval-service.ts)
   pass SYNOPSIS_DOC_MAX_CHARS when attemptMode/resolution.mode is
   per_chunk_synopsis, undefined otherwise.
4. import-file.ts inline path passes undefined (per_chunk_synopsis
   refused upstream there).

One-time effect: per_chunk_synopsis pages re-embedded post-PR get a
NEW corpus_generation including the cap. v0.40.3.0 query_cache.page_generations
contract auto-invalidates cached query results on first re-embed.
Future cap changes track correctly.

Addresses codex review P2 on PR #1427.

---------

Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-07-21 16:50:43 -07:00
103 changed files with 3303 additions and 398 deletions
+5 -1
View File
@@ -206,7 +206,11 @@ jobs:
needs: cache-check
if: needs.cache-check.outputs.hit != 'true'
runs-on: ubuntu-latest
timeout-minutes: 15
# 22, not 15: under parallel PR load the PGLite WASM cold-starts stretch a
# shard past 15 min while every test is still passing — the timeout then
# cancels the job and the test-status gate reads it as a failure. 13 runs
# died this way on 2026-07-21/22 alone.
timeout-minutes: 22
strategy:
fail-fast: false
matrix:
+2 -2
View File
@@ -71,8 +71,8 @@ GBrain is designed to be installed and operated by an AI agent. The fastest path
If you don't already have an AI agent platform running, start with one of these. Both are designed to read GBrain's install protocol and execute it:
- **[OpenClaw](https://github.com/openclawagents/openclaw)** — deploy [AlphaClaw on Render](https://render.com/deploy?repo=https://github.com/chrysb/alphaclaw) (one click, 8GB+ RAM)
- **[Hermes](https://github.com/openclawagents/hermes)** — deploy on [Railway](https://github.com/praveen-ks-2001/hermes-agent-template) (one click)
- **[OpenClaw](https://github.com/openclaw/openclaw)** — deploy [AlphaClaw on Render](https://render.com/deploy?repo=https://github.com/chrysb/alphaclaw) (one click, 8GB+ RAM)
- **[Hermes](https://github.com/NousResearch/hermes-agent)** — deploy on [Railway](https://github.com/praveen-ks-2001/hermes-agent-template) (one click)
Then paste this into your agent:
+3 -1
View File
@@ -131,7 +131,9 @@ into gbrain so other clients can scaffold it. Default behavior:
`~/.gbrain/harvest-private-patterns.txt` plus built-in defaults
(canonical private fork name, common email regex, Slack channel pattern). Any
match → rollback (delete the harvested files) and exit non-zero.
- `openclaw.plugin.json` updated with the new slug, sorted.
- `openclaw.plugin.json` updated with the new slug, sorted. Harvest must preserve
the top-level OpenClaw-native plugin fields (`id`, `configSchema`, `contracts`)
because OpenClaw validates those before it can install the package.
- `--no-lint` bypasses the linter (after a manual editorial scrub).
Use the `skillpack-harvest` skill (its companion editorial workflow)
@@ -233,13 +233,14 @@ keep it or `git checkout` to throw it away. Nothing is committed for you.
**For a skill that ships with gbrain** (anything under the gbrain repo's own
`skills/`): SkillOpt refuses to overwrite it by default and writes the winner to
`skills/<name>/skillopt/best.md` instead, so an optimization pass can never
silently mutate a skill other people depend on. Two ways to handle that:
`skills/<name>/skillopt/proposed.md` instead (while keeping `best.md` as the
optimizer's current-best pointer), so an optimization pass can never silently
mutate a skill other people depend on. Two ways to handle that:
```bash
# See the proposed improvement without touching SKILL.md (works for ANY skill):
gbrain skillopt meeting-prep --split 1:1:1 --no-mutate
# → writes skills/meeting-prep/skillopt/best.md (the proposed rewrite), prints its path. Copy what you want.
# → writes skills/meeting-prep/skillopt/proposed.md, updates best.md, and prints the proposal path.
# Actually rewrite a bundled skill (explicit opt-in + an independent held-out set):
gbrain skillopt brain-ops --split 1:1:1 --allow-mutate-bundled \
+2 -2
View File
@@ -1565,8 +1565,8 @@ GBrain is designed to be installed and operated by an AI agent. The fastest path
If you don't already have an AI agent platform running, start with one of these. Both are designed to read GBrain's install protocol and execute it:
- **[OpenClaw](https://github.com/openclawagents/openclaw)** — deploy [AlphaClaw on Render](https://render.com/deploy?repo=https://github.com/chrysb/alphaclaw) (one click, 8GB+ RAM)
- **[Hermes](https://github.com/openclawagents/hermes)** — deploy on [Railway](https://github.com/praveen-ks-2001/hermes-agent-template) (one click)
- **[OpenClaw](https://github.com/openclaw/openclaw)** — deploy [AlphaClaw on Render](https://render.com/deploy?repo=https://github.com/chrysb/alphaclaw) (one click, 8GB+ RAM)
- **[Hermes](https://github.com/NousResearch/hermes-agent)** — deploy on [Railway](https://github.com/praveen-ks-2001/hermes-agent-template) (one click)
Then paste this into your agent:
+1
View File
@@ -1,4 +1,5 @@
{
"id": "gbrain-context-engine",
"name": "gbrain",
"version": "0.32.3.0",
"description": "Personal knowledge brain with Postgres + pgvector hybrid search",
+22
View File
@@ -18,6 +18,12 @@
# R4: any file that creates `new PGLiteEngine(` must call `.disconnect(`
# inside an `afterAll(` block. Without disconnect, engines leak across
# file boundaries within a shard process.
# R5: any file that calls `configureGateway(` or
# `__setEmbedTransportForTests(` must also call `resetGateway()` and
# have an `afterAll(`/`afterEach(` hook. The gateway is module-global;
# a configured remote provider + fake key left past the file boundary
# makes the next embed-triggering file in the shard fire a live HTTP
# call (issue #3066; master shard 6 broke twice this way).
#
# Scope:
# - Recursively scans `test/**/*.test.ts`.
@@ -132,6 +138,22 @@ while IFS= read -r f; do
emit_violation "$f" "R4" "creates PGLiteEngine but missing afterAll(() => engine.disconnect()); engine leaks across files in the shard process" ""
fi
fi
# R5: gateway configuration requires resetGateway + an afterAll/afterEach
# hook. Same loose two-grep shape as R4 — `resetGateway(` present plus at
# least one afterAll(/afterEach( — but on comment-stripped lines: a prose
# mention like "a test that calls resetGateway()" must not satisfy the
# rule (that exact false pass hid the cycle-consolidate leaker).
# No [[:space:]]* before the paren on the trigger side: prose in a test
# name ("works WITHOUT configureGateway (reads registry...)") must not
# trigger the rule; real call sites are always `configureGateway(`.
r5_code=$(grep -vE '^[[:space:]]*(//|\*)' "$f" 2>/dev/null || true)
if printf '%s\n' "$r5_code" | grep -qE 'configureGateway\(|__setEmbedTransportForTests\('; then
if ! printf '%s\n' "$r5_code" | grep -qE 'resetGateway\(' \
|| ! printf '%s\n' "$r5_code" | grep -qE 'afterAll[[:space:]]*\(|afterEach[[:space:]]*\('; then
emit_violation "$f" "R5" "configures the AI gateway but never calls resetGateway() in afterAll/afterEach; gateway state (provider, fake keys, transports) leaks across files in the shard process" ""
fi
fi
done <<EOF
$FILE_LIST
EOF
+2 -1
View File
@@ -266,4 +266,5 @@ editorial pass.
(e.g. `src/commands/<slug>.ts` if the host SKILL.md declares it
in frontmatter)
- gbrain's `openclaw.plugin.json` — adds the slug to `skills:`
array, sorted alphabetically
array, sorted alphabetically, without removing OpenClaw-native plugin fields
like `id`, `configSchema`, or `contracts`
+3 -1
View File
@@ -57,6 +57,8 @@ This mode guarantees:
- `skills/manifest.json` lists every skill directory
- `skills/RESOLVER.md` references every skill in the manifest
- `openclaw.plugin.json` `skills[]` round-trips with both
- `openclaw.plugin.json` keeps OpenClaw install-required native plugin fields
(`id`, object `configSchema`, and `contracts.contextEngines` when applicable)
- No MECE violations (duplicate triggers across skills)
### Phases
@@ -72,7 +74,7 @@ This mode guarantees:
### Automation
```bash
bun test test/skills-conformance.test.ts test/resolver.test.ts
bun test test/skills-conformance.test.ts test/resolver.test.ts test/openclaw-plugin-manifest.test.ts
```
The CI-gated check is the package.json `test` script.
+8 -1
View File
@@ -54,7 +54,7 @@ export function bigintToStringReplacer(_key: string, value: unknown): unknown {
}
// CLI-only commands that bypass the operation layer
export const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'reconcile-links', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'calibration', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'advisor', 'watch', 'reindex-search-vector']);
export const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'maintain', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'reconcile-links', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'calibration', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'advisor', 'watch', 'reindex-search-vector']);
// CLI-only commands whose handlers print their own --help text. These are
// excluded from the generic short-circuit so detailed per-command and
// per-subcommand usage stays reachable.
@@ -78,6 +78,8 @@ const CLI_ONLY_SELF_HELP = new Set([
'capture',
// v0.42 self-upgrade ships its own usage (flags + the agent-skill story).
'self-upgrade',
// maintain (#3015) prints its own usage block (modes + not-auto-applied list).
'maintain',
// v0.43 (#2095): watch ships WATCH_HELP (flags + the stdin-turn protocol).
'watch',
// v0.37 fix wave (Lane D.4 + CDX2-12): sync's --no-embed flag was
@@ -1757,6 +1759,11 @@ async function handleCliOnly(command: string, args: string[]) {
await runOrphans(engine, args);
break;
}
case 'maintain': {
const { runMaintain } = await import('./commands/maintain.ts');
await runMaintain(engine, args);
break;
}
// v0.32.7 CJK wave — post-upgrade markdown re-chunk sweep.
// v0.36 Phase 3 wave — `gbrain reindex --multimodal` re-embeds content_chunks
// into the unified Voyage multimodal-3 column.
+7 -6
View File
@@ -133,14 +133,15 @@ function indexCompleted(entries: CompletedMigrationEntry[]): CompletedIndex {
* Returns the resolved status for a migration based on its entries.
*
* Semantics (Bug 3 keep "complete wins" safety):
* - If any entry is `complete`, the version is complete. Terminal state.
* - Otherwise, if the latest entry is `retry`, the version is pending
* (user requested a fresh attempt).
* - If the latest entry is `retry`, the version is pending. This is the
* explicit escape hatch written by `--force-retry`, and it overrides an
* earlier `complete` entry without hand-editing the ledger.
* - Otherwise, if any entry is `complete`, the version is complete.
* - Otherwise, if any entry is `partial`, the version is partial.
* - Otherwise, pending.
*
* `complete` never regresses. A later accidental `partial` append cannot
* undo a completed migration.
* `complete` never regresses accidentally. A later `partial` append cannot
* undo a completed migration; only a trailing, explicit `retry` marker can.
*/
function statusForVersion(
version: string,
@@ -148,9 +149,9 @@ function statusForVersion(
): 'complete' | 'partial' | 'pending' | 'wedged' {
const entries = idx.byVersion.get(version) ?? [];
if (entries.length === 0) return 'pending';
if (entries.some(e => e.status === 'complete')) return 'complete';
const latest = entries[entries.length - 1];
if (latest.status === 'retry') return 'pending';
if (entries.some(e => e.status === 'complete')) return 'complete';
// Bug 3 attempt cap — count consecutive partials from the end (stopping
// at any 'retry' or 'complete'). If we hit MAX_CONSECUTIVE_PARTIALS,
// the migration is wedged and needs explicit --force-retry to try again.
+10
View File
@@ -37,9 +37,19 @@ export async function findCodeDef(
// trigger) are first-class definitions in the SQL sense. The chunker's
// normalizeSymbolType maps create_table → 'table' etc, so adding the SQL
// kinds here is what makes `gbrain code-def users` work against SQL.
// Method-level + member definitions. normalizeSymbolType only canonicalizes
// some node types; the rest fall through `type.replace(/_/g, ' ')`, so
// tree-sitter's method_declaration → 'method declaration', struct_specifier →
// 'struct specifier', protocol_declaration → 'protocol declaration', etc.
// Without these, code-def is blind to every method, constructor, field, C
// struct, and Swift protocol — which is most of an OO codebase. The plain
// 'struct' entry above never matched for the same reason (C emits the
// 'struct specifier' fallback form).
const DEF_TYPES = [
'function', 'class', 'interface', 'type', 'enum', 'struct', 'trait', 'module', 'contract',
'table', 'view', 'index', 'procedure', 'schema', 'database', 'trigger',
'method declaration', 'method definition', 'constructor declaration',
'field declaration', 'field definition', 'struct specifier', 'protocol declaration',
];
const params: unknown[] = [symbol, limit];
let whereLang = '';
+9 -4
View File
@@ -4963,8 +4963,7 @@ export async function buildChecks(
message:
`${unmatched}/${sample.length} conversation pages (${unmatchedPct.toFixed(1)}%) match NO built-in pattern. ` +
`Breakdown: ${breakdown}. ` +
`Investigate: gbrain conversation-parser scan <slug> | ` +
`Enable LLM fallback (opt-in): gbrain config set conversation_parser.llm_fallback_enabled true`,
`Investigate: gbrain conversation-parser scan <slug>`,
});
} else {
checks.push({
@@ -7091,11 +7090,16 @@ export async function buildChecks(
);
if (factsExists[0]?.exists) {
const health = await engine.getFactsHealth('default');
const status: 'ok' | 'warn' = health.total_active >= 0 ? 'ok' : 'warn';
const top = health.top_entities
.slice(0, 3)
.map(t => `${t.entity_slug}:${t.count}`)
.join(', ') || '—';
// #3062: "0 active facts" used to read [OK] even when the chat
// gateway had never been reachable — indistinguishable from a brain
// with genuinely nothing to extract. Distinguish the two.
const { unavailableReason } = await import('../core/ai/gateway.ts');
const chatDown = health.total_active === 0 ? unavailableReason('chat') : null;
const status: 'ok' | 'warn' = chatDown ? 'warn' : health.total_active >= 0 ? 'ok' : 'warn';
checks.push({
name: 'facts_health',
status,
@@ -7103,7 +7107,8 @@ export async function buildChecks(
`facts_health(default): ${health.total_active} active, ` +
`${health.total_today} today, ${health.total_week} this week, ` +
`${health.total_consolidated} consolidated, ` +
`top entities ${top}`,
`top entities ${top}` +
(chatDown ? ` — extraction has no chat gateway: ${chatDown}` : ''),
});
} else {
checks.push({
+34 -4
View File
@@ -581,7 +581,7 @@ async function embedPage(
for (let j = 0; j < toEmbed.length; j++) {
embeddingMap.set(toEmbed[j].chunk_index, embeddings[j]);
}
const updated: ChunkInput[] = chunks.map(c => ({
const updated: ChunkInput[] = chunks.map(c => preserveCodeMetadata(c, {
chunk_index: c.chunk_index,
chunk_text: c.chunk_text,
chunk_source: c.chunk_source,
@@ -605,6 +605,31 @@ async function embedPage(
slog(`${slug}: embedded ${toEmbed.length} chunks`);
}
/**
* Carry code-chunk metadata (language, symbol_name, symbol_type, line range,
* parent scope, doc comment, qualified name) from a loaded Chunk back into a
* ChunkInput destined for upsertChunks.
*
* Issue #769: every re-embed used to strip these fields, and upsertChunks
* overwrites (does not COALESCE) the metadata columns from EXCLUDED, so
* each pass clobbered code-def's primary index to NULL. Pulling the
* preservation into one helper keeps the three re-embed call sites
* (embedPage, embedAll non-stale, embedAllStale) in lock-step.
*/
function preserveCodeMetadata(loaded: any, base: ChunkInput): ChunkInput {
return {
...base,
language: loaded.language ?? undefined,
symbol_name: loaded.symbol_name ?? undefined,
symbol_type: loaded.symbol_type ?? undefined,
start_line: loaded.start_line ?? undefined,
end_line: loaded.end_line ?? undefined,
parent_symbol_path: loaded.parent_symbol_path ?? undefined,
doc_comment: loaded.doc_comment ?? undefined,
symbol_name_qualified: loaded.symbol_name_qualified ?? undefined,
};
}
async function embedAll(
engine: BrainEngine,
staleOnly: boolean,
@@ -717,8 +742,10 @@ async function embedAll(
for (let j = 0; j < toEmbed.length; j++) {
embeddingMap.set(toEmbed[j].chunk_index, embeddings[j]);
}
// Preserve ALL chunks, only update embeddings for stale ones
const updated: ChunkInput[] = chunks.map(c => ({
// Preserve ALL chunks, only update embeddings for stale ones.
// preserveCodeMetadata threads code-chunk metadata (#769) so re-embed
// doesn't clobber language/symbol_name/symbol_type to NULL.
const updated: ChunkInput[] = chunks.map(c => preserveCodeMetadata(c, {
chunk_index: c.chunk_index,
chunk_text: c.chunk_text,
chunk_source: c.chunk_source,
@@ -1012,7 +1039,10 @@ async function embedAllStale(
for (let j = 0; j < stale.length; j++) {
staleIdxToEmbedding.set(stale[j].chunk_index, embeddings[j]);
}
const merged: ChunkInput[] = existing.map(c => ({
// preserveCodeMetadata threads code-chunk metadata (#769) so the
// autopilot --stale path doesn't clobber language/symbol_name/etc
// to NULL on every cycle.
const merged: ChunkInput[] = existing.map(c => preserveCodeMetadata(c, {
chunk_index: c.chunk_index,
chunk_text: c.chunk_text,
chunk_source: c.chunk_source,
+53 -4
View File
@@ -469,15 +469,53 @@ export async function extractLinksFromFile(
// --- Timeline extraction ---
/**
* Index of the first dash (, , -) that can serve as the Source Summary
* delimiter: it must have whitespace on both sides and sit outside every
* markdown-link span. Hyphens inside link targets
* (`../people/alice-example.md`) and dashes inside link labels
* (`[Deals — Q1 Review](...)`) are content, not delimiters splitting on
* them shatters one entry into two fragments whose halves re-insert on
* every sync (the (page_id, date, summary, source) uniqueness sees each
* fragment shape as a new row). Returns -1 when the line has no delimiter.
*/
function findDelimiterOutsideLinks(text: string): number {
let depth = 0;
for (let i = 0; i < text.length; i++) {
const c = text[i];
if (c === '[' || c === '(') depth++;
else if (c === ']' || c === ')') { if (depth > 0) depth--; }
else if (
depth === 0 &&
(c === '—' || c === '' || c === '-') &&
i > 0 && /\s/.test(text[i - 1]) &&
i + 1 < text.length && /\s/.test(text[i + 1])
) {
return i;
}
}
return -1;
}
/** Extract timeline entries from markdown content */
export function extractTimelineFromContent(content: string, slug: string): ExtractedTimelineEntry[] {
const entries: ExtractedTimelineEntry[] = [];
// Format 1: Bullet — - **YYYY-MM-DD** | Source — Summary
const bulletPattern = /^-\s+\*\*(\d{4}-\d{2}-\d{2})\*\*\s*\|\s*(.+?)\s*[—–-]\s*(.+)$/gm;
// The delimiter search is link-aware (see findDelimiterOutsideLinks); a
// bullet with no delimiter (e.g. an auto-generated backlink line
// `- **date** | Referenced in [X](y.md)`) is kept whole as the summary
// rather than dropped or fragmented.
const bulletPattern = /^-\s+\*\*(\d{4}-\d{2}-\d{2})\*\*\s*\|\s*(.+)$/gm;
let match;
while ((match = bulletPattern.exec(content)) !== null) {
entries.push({ slug, date: match[1], source: match[2].trim(), summary: match[3].trim() });
const rest = match[2].trim();
const at = findDelimiterOutsideLinks(rest);
if (at >= 0) {
entries.push({ slug, date: match[1], source: rest.slice(0, at).trim(), summary: rest.slice(at + 1).trim() });
} else {
entries.push({ slug, date: match[1], source: 'markdown', summary: rest });
}
}
// Format 2: Header — ### YYYY-MM-DD — Title
@@ -1651,7 +1689,7 @@ async function extractTimelineFromDB(
* make re-extraction idempotent). EVERY processed page is stamped, including
* zero-link pages they WERE processed.
*/
async function extractStaleFromDB(
export async function extractStaleFromDB(
engine: BrainEngine,
opts: {
dryRun: boolean;
@@ -1743,7 +1781,18 @@ async function extractStaleFromDB(
// `page.updated_at.toISOString()` — the JS Date is ms-truncated, so the
// µs-precision DB updated_at stayed strictly greater and the page never
// cleared on Postgres. Stamping the exact value makes them equal.
processedRefs.push({ slug: page.slug, source_id: page.source_id, extractedAt: page.updated_at_iso });
//
// BUT the stamp must also clear the version-staleness clause
// (`links_extracted_at < versionTs`). A page whose updated_at predates
// versionTs would otherwise be stamped below the threshold and read as
// stale forever — a permanent re-extract loop that never clears the lag.
// GREATEST(updated_at, versionTs) preserves the race semantics (a real
// future edit advances updated_at > versionTs >= stamp → re-extracts)
// while lifting old pages to the threshold so they clear.
const stampIso = page.updated_at.getTime() >= Date.parse(versionTs)
? page.updated_at_iso
: versionTs;
processedRefs.push({ slug: page.slug, source_id: page.source_id, extractedAt: stampIso });
}
// Flush NON-swallowing (CDX-4): a throw here propagates out of the sweep so
+10 -1
View File
@@ -98,8 +98,17 @@ export function findBareTweetHits(compiledTruth: string, slug: string): BareTwee
}
// If the line already contains a tweet URL, it's cited — skip
if (URL_NEARBY_RE.test(line)) continue;
// If the line carries an explicit source citation (e.g.
// "[Source: X, @handle, 2026-05-28]"), it's already attributed — skip.
// Catches instructional/example lines in recipe docs that demonstrate
// the CORRECT citation format. (v0.42.x)
if (/\[\s*source:/i.test(line)) continue;
// Strip inline-code spans (`...`) before matching: phrases shown as
// inline-code templates in docs are examples, not bare claims. The
// fenced-code skip above only covers ``` blocks, not inline backticks.
const lineForMatch = line.replace(/`[^`]*`/g, '');
for (const re of BARE_TWEET_PHRASES) {
const m = line.match(re);
const m = lineForMatch.match(re);
if (m) {
hits.push({ slug, line: i + 1, rawLine: line.trim(), phrase: m[0] });
break; // one finding per line is enough
+7 -1
View File
@@ -1664,7 +1664,13 @@ export async function registerBuiltinHandlers(
worker.register('backlinks', async (job) => {
const { runBacklinksCore } = await import('./backlinks.ts');
const action: 'check' | 'fix' = job.data.action === 'check' ? 'check' : 'fix';
// Default to 'check', not 'fix': backlinks jobs submitted with an empty
// payload (e.g. the sync→embed→backlinks chains enqueued after ingestion)
// must never rewrite tracked brain pages with generated "Referenced in"
// timeline bullets. Mirrors the documented intent in src/core/cycle.ts
// (runPhaseBacklinks). The filesystem fixer stays available explicitly
// via '{"action":"fix"}' or `gbrain check-backlinks fix`.
const action: 'check' | 'fix' = job.data.action === 'fix' ? 'fix' : 'check';
const dir = typeof job.data.dir === 'string'
? job.data.dir
: (await engine.getConfig('sync.repo_path')) ?? '.';
+6 -1
View File
@@ -127,7 +127,12 @@ export function lintContent(content: string, filePath: string, opts: LintContent
}
// Rule: Wrapping code fences (```markdown ... ```)
if (content.match(/^```(?:markdown|md)\s*\n/m) && content.match(/\n```\s*$/m)) {
// Detector intentionally has NO /m flag so ^/$ match start/end of the whole
// file, not inner lines. Keeps detector in sync with fixContent() below,
// which also has no /m flag. Without this, lint reports "fixable" false
// positives on any page that simply contains a ```markdown code block, but
// fixContent can never strip them (its regex only matches whole-file wrappers).
if (content.match(/^```(?:markdown|md)\s*\n/) && content.match(/\n```\s*$/)) {
issues.push({
file: filePath, line: 1, rule: 'code-fence-wrap',
message: 'Page wrapped in ```markdown code fences (LLM artifact)',
+224
View File
@@ -0,0 +1,224 @@
/**
* gbrain maintain conservative self-healing maintenance.
*
* This command automates the safe parts of the operator runbook:
* - stale link/timeline extraction
* - stale per-source dream cycles when doctor reports cycle_freshness
*
* It deliberately does NOT mutate source files, apply schema-pack upgrades, or
* invent semantic hub links. Those need review or a separate command with an
* auditable proposal surface.
*/
import { existsSync } from 'fs';
import type { BrainEngine } from '../core/engine.ts';
import type { BrainHealth } from '../core/types.ts';
import { buildChecks, computeDoctorReport, type DoctorReport, type Check } from './doctor.ts';
import { extractStaleFromDB } from './extract.ts';
import { runCycle, type CycleReport } from '../core/cycle.ts';
type ActionStatus = 'ok' | 'would_apply' | 'applied' | 'blocked' | 'skipped';
export interface MaintenanceAction {
name: string;
status: ActionStatus;
message: string;
details?: Record<string, unknown>;
}
export interface MaintainOptions {
json: boolean;
safe: boolean;
dryRun: boolean;
help: boolean;
}
export interface MaintainReport {
mode: 'dry-run' | 'safe';
before: {
health: BrainHealth;
doctor: DoctorReport;
};
actions: MaintenanceAction[];
after: {
health: BrainHealth;
doctor: DoctorReport;
};
}
export function parseMaintainArgs(args: string[]): MaintainOptions {
const safe = args.includes('--safe');
return {
json: args.includes('--json'),
safe,
dryRun: args.includes('--dry-run') || !safe,
help: args.includes('--help') || args.includes('-h'),
};
}
export function extractCycleFreshnessSourceIds(checks: Check[]): string[] {
const ids = new Set<string>();
for (const check of checks) {
if (check.name !== 'cycle_freshness' || check.status === 'ok') continue;
const re = /Source '([^']+)' last cycled/g;
for (const match of check.message.matchAll(re)) {
const id = match[1]?.trim();
if (id) ids.add(id);
}
}
return [...ids].sort();
}
async function buildDoctorReport(engine: BrainEngine): Promise<DoctorReport> {
const checks = await buildChecks(engine, ['--json', '--scope=brain']);
return computeDoctorReport(checks);
}
async function runStaleExtraction(
engine: BrainEngine,
beforeHealth: BrainHealth,
dryRun: boolean,
): Promise<MaintenanceAction> {
if (beforeHealth.stale_pages <= 0) {
return { name: 'extract_stale', status: 'ok', message: 'No stale pages.' };
}
if (dryRun) {
return {
name: 'extract_stale',
status: 'would_apply',
message: `Would run DB-backed stale extraction for ${beforeHealth.stale_pages} page(s).`,
details: { stale_pages: beforeHealth.stale_pages },
};
}
const result = await extractStaleFromDB(engine, {
dryRun: false,
jsonMode: false,
includeFrontmatter: false,
catchUp: false,
});
return {
name: 'extract_stale',
status: 'applied',
message: `Processed ${result.pagesProcessed} stale page(s); ${result.staleRemaining} remain.`,
details: {
links_created: result.linksCreated,
timeline_created: result.timelineCreated,
pages_processed: result.pagesProcessed,
stale_remaining: result.staleRemaining,
},
};
}
async function runCycleFreshnessMaintenance(
engine: BrainEngine,
beforeDoctor: DoctorReport,
dryRun: boolean,
): Promise<MaintenanceAction[]> {
const sourceIds = extractCycleFreshnessSourceIds(beforeDoctor.checks);
if (sourceIds.length === 0) {
return [{ name: 'cycle_freshness', status: 'ok', message: 'All sources cycled recently.' }];
}
if (dryRun) {
return sourceIds.map((sourceId) => ({
name: 'cycle_freshness',
status: 'would_apply',
message: `Would run source-scoped dream cycle for ${sourceId}.`,
details: { source_id: sourceId },
}));
}
const sources = await engine.listAllSources();
const actions: MaintenanceAction[] = [];
for (const sourceId of sourceIds) {
const source = sources.find((s) => s.id === sourceId);
const localPath = source?.local_path ?? null;
const brainDir = localPath && existsSync(localPath) ? localPath : null;
const report: CycleReport = await runCycle(engine, {
brainDir,
dryRun: false,
pull: false,
sourceId,
});
actions.push({
name: 'cycle_freshness',
status: report.status === 'failed' ? 'blocked' : 'applied',
message: `Ran source-scoped dream cycle for ${sourceId}: ${report.status}.`,
details: {
source_id: sourceId,
brain_dir: brainDir,
cycle_status: report.status,
phases: report.phases.map((p) => ({ phase: p.phase, status: p.status })),
},
});
}
return actions;
}
export async function runMaintain(engine: BrainEngine, args: string[]): Promise<MaintainReport | void> {
const opts = parseMaintainArgs(args);
if (opts.help) {
console.log(`Usage: gbrain maintain [--safe] [--dry-run] [--json]
Conservative self-healing maintenance.
Modes:
--dry-run Preview safe actions without writes. Default when --safe is absent.
--safe Apply safe actions: stale extraction and source cycle freshness.
--json Emit a structured before/action/after report.
Not auto-applied:
source-file frontmatter fixes, schema-pack upgrades, atom-pack changes,
semantic hub-link guesses, and destructive cleanup.
`);
return;
}
const beforeHealth = await engine.getHealth();
const beforeDoctor = await buildDoctorReport(engine);
const actions: MaintenanceAction[] = [];
actions.push(await runStaleExtraction(engine, beforeHealth, opts.dryRun));
actions.push(...await runCycleFreshnessMaintenance(engine, beforeDoctor, opts.dryRun));
const afterHealth = await engine.getHealth();
const afterDoctor = await buildDoctorReport(engine);
const report: MaintainReport = {
mode: opts.dryRun ? 'dry-run' : 'safe',
before: { health: beforeHealth, doctor: beforeDoctor },
actions,
after: { health: afterHealth, doctor: afterDoctor },
};
if (opts.json) {
console.log(JSON.stringify(report, null, 2));
} else {
printMaintainReport(report);
}
return report;
}
function printMaintainReport(report: MaintainReport): void {
console.log(`GBrain maintain (${report.mode})`);
console.log(
`Before: brain_score=${Math.round(report.before.health.brain_score)}/100 ` +
`stale=${report.before.health.stale_pages} islands=${report.before.health.orphan_pages} ` +
`doctor=${report.before.doctor.status}`,
);
for (const action of report.actions) {
console.log(` ${action.status}: ${action.name}${action.message}`);
}
console.log(
`After: brain_score=${Math.round(report.after.health.brain_score)}/100 ` +
`stale=${report.after.health.stale_pages} islands=${report.after.health.orphan_pages} ` +
`doctor=${report.after.doctor.status}`,
);
if (report.mode === 'dry-run') {
console.log('Run `gbrain maintain --safe` to apply safe actions.');
}
}
+14 -1
View File
@@ -536,7 +536,20 @@ function shouldSkipProvider(modelStr: string, skip: string[]): boolean {
export async function runModels(engine: BrainEngine, args: string[]): Promise<void> {
const json = args.includes('--json');
const sub = args[1] === 'doctor' ? 'doctor' : args[1] === 'help' || args.includes('--help') || args.includes('-h') ? 'help' : 'read';
// args is `subArgs` from cli.ts `handleCliOnly` — the leading 'models'
// token has already been stripped. The subcommand is at args[0], NOT
// args[1]. Pre-fix this check was `args[1]`, so `gbrain models doctor`
// silently fell through to the read view. The doctor probe path was
// unreachable from the CLI.
//
// --help honored FIRST so `gbrain models doctor --help` shows usage
// instead of running network probes (which would spend tokens or
// exit nonzero when the user only asked for help). Pre-fix the
// args[1] ternary happened to dodge this by always falling through
// to the args.includes('--help') branch; the args[0] rewrite needs
// explicit ordering to preserve that behavior.
const hasHelp = args.includes('--help') || args.includes('-h') || args[0] === 'help';
const sub = hasHelp ? 'help' : args[0] === 'doctor' ? 'doctor' : 'read';
if (sub === 'help') {
process.stdout.write(
+10 -55
View File
@@ -15,6 +15,11 @@
import type { BrainEngine } from '../core/engine.ts';
import { createProgress, startHeartbeat } from '../core/progress.ts';
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
import {
shouldExcludeFromOrphanReporting,
loadOrphanPolicyOverrides,
type OrphanPolicyOverrides,
} from '../core/orphan-policy.ts';
// --- Types ---
@@ -32,65 +37,14 @@ export interface OrphanResult {
excluded: number;
}
// --- Filter constants ---
/** Slug suffixes that are always auto-generated root files */
const AUTO_SUFFIX_PATTERNS = ['/_index', '/log'];
/** Page slugs that are pseudo-pages by convention */
const PSEUDO_SLUGS = new Set(['_atlas', '_index', '_stats', '_orphans', '_scratch', 'claude']);
/** Slug segment that marks raw sources */
const RAW_SEGMENT = '/raw/';
/** Slug prefixes where no inbound links is expected */
const DENY_PREFIXES = [
'output/',
'dashboards/',
'scripts/',
'templates/',
'openclaw/config/',
];
/** First slug segments where no inbound links is expected */
const FIRST_SEGMENT_EXCLUSIONS = new Set([
'scratch',
'thoughts',
'catalog',
'entities',
'raw',
'atoms',
'skills',
]);
// --- Filter logic ---
/**
* Returns true if a slug should be excluded from orphan reporting by default.
* These are pages where having no inbound links is expected / not a content problem.
*/
export function shouldExclude(slug: string): boolean {
// Pseudo-pages (exact match)
if (PSEUDO_SLUGS.has(slug)) return true;
// Auto-generated suffix patterns
for (const suffix of AUTO_SUFFIX_PATTERNS) {
if (slug.endsWith(suffix)) return true;
}
// Raw source slugs
if (slug.includes(RAW_SEGMENT)) return true;
// Deny-prefix slugs
for (const prefix of DENY_PREFIXES) {
if (slug.startsWith(prefix)) return true;
}
// First-segment exclusions
const firstSegment = slug.split('/')[0];
if (FIRST_SEGMENT_EXCLUSIONS.has(firstSegment)) return true;
return false;
export function shouldExclude(slug: string, overrides?: OrphanPolicyOverrides): boolean {
return shouldExcludeFromOrphanReporting(slug, overrides);
}
/**
@@ -156,6 +110,7 @@ export async function findOrphans(
let allOrphans: { slug: string; title: string; domain: string | null }[];
let total: number;
let excludedAll: number;
const overrides = includePseudo ? undefined : await loadOrphanPolicyOverrides(engine);
try {
allOrphans = await engine.findOrphanPages(
sourceIds ? { sourceIds } : sourceId ? { sourceId } : undefined,
@@ -184,7 +139,7 @@ export async function findOrphans(
total = liveRows.length;
excludedAll = includePseudo
? 0
: liveRows.reduce((n, r) => n + (shouldExclude(r.slug) ? 1 : 0), 0);
: liveRows.reduce((n, r) => n + (shouldExclude(r.slug, overrides) ? 1 : 0), 0);
} finally {
stopHb();
progress.finish();
@@ -192,7 +147,7 @@ export async function findOrphans(
const filtered = includePseudo
? allOrphans
: allOrphans.filter(row => !shouldExclude(row.slug));
: allOrphans.filter(row => !shouldExclude(row.slug, overrides));
const orphans: OrphanPage[] = filtered.map(row => ({
slug: row.slug,
+17 -2
View File
@@ -843,6 +843,21 @@ export async function runServeHttp(engine: BrainEngine, options: ServeHttpOption
// reverse proxies / tunnels; default to localhost for dev.
const issuerUrl = new URL(publicUrl || `http://localhost:${port}`);
// MCP authorization spec (2025-06-18 draft §5.1) and RFC 9728 require the
// protected resource server to return its discovery metadata URL in the
// WWW-Authenticate header on 401 responses:
//
// WWW-Authenticate: Bearer resource_metadata="<URL>"
//
// Clients (claude.ai, Cursor, every other MCP-aware OAuth client) use that
// URL to find the authorization-server discovery doc + token endpoint
// without the user having to paste those URLs manually. Pre-fix the header
// shipped `Bearer error="invalid_token", ...` with no resource_metadata
// parameter, so MCP clients couldn't begin the OAuth flow from a fresh
// 401 — they would silently fail to connect with a generic "couldn't
// reach the MCP server" error.
const resourceMetadataUrl = `${issuerUrl.toString().replace(/\/$/, '')}/.well-known/oauth-protected-resource`;
// F9: cookie `secure` flag honors both the request's TLS state (req.secure
// is set when express trust-proxy lands an X-Forwarded-Proto: https) AND
// the operator's declared issuer protocol (so a Cloudflare-tunnel deploy
@@ -1601,7 +1616,7 @@ export async function runServeHttp(engine: BrainEngine, options: ServeHttpOption
res.status(405).json({ jsonrpc: '2.0', error: { code: -32000, message: 'Method not allowed' }, id: null });
});
app.post('/mcp', requireBearerAuth({ verifier: oauthProvider }), async (req: Request, res: Response) => {
app.post('/mcp', requireBearerAuth({ verifier: oauthProvider, resourceMetadataUrl }), async (req: Request, res: Response) => {
const startTime = Date.now();
const authInfo = (req as any).auth as AuthInfo;
@@ -1944,7 +1959,7 @@ export async function runServeHttp(engine: BrainEngine, options: ServeHttpOption
app.post(
'/ingest',
ingestRateLimiter,
requireBearerAuth({ verifier: oauthProvider, requiredScopes: ['write'] }),
requireBearerAuth({ verifier: oauthProvider, requiredScopes: ['write'], resourceMetadataUrl }),
express.raw({ type: '*/*', limit: ingestMaxBytes }),
async (req: Request, res: Response) => {
const startTime = Date.now();
+2 -2
View File
@@ -6,7 +6,7 @@
* degrades to gather-only output with a warning if missing.
*/
import type { BrainEngine } from '../core/engine.ts';
import { runThink, persistSynthesis } from '../core/think/index.ts';
import { runThink, persistSynthesis, stripGapsSection } from '../core/think/index.ts';
import { loadConfig, isThinClient } from '../core/config.ts';
import { callRemoteTool, unpackToolResult } from '../core/mcp-client.ts';
@@ -157,7 +157,7 @@ prints what would have been the input (exit 0).
// Human-readable output
console.log(`# ${question}\n`);
console.log(result.answer);
console.log(stripGapsSection(result.answer));
console.log('');
if (result.gaps.length > 0) {
console.log('## Gaps');
+9
View File
@@ -263,6 +263,15 @@ export function dimsProviderOptions(
if (modelId === 'text-embedding-v3' || modelId === 'embedding-3') {
return { openaiCompatible: { dimensions: dims } };
}
// Qwen3-Embedding family on Ollama (and any other openai-compatible
// provider serving it) supports Matryoshka truncation via `dimensions`.
// Native sizes: 0.6B=1024, 4B=2560, 8B=4096. Without `dimensions`,
// Ollama returns the native size and brains configured for narrower
// widths hard-fail with a dim-mismatch error. Pattern match the bare
// model name + any `:tag` (e.g. `qwen3-embedding:4b`, `qwen3-embedding:0.6b`).
if (modelId === 'qwen3-embedding' || modelId.startsWith('qwen3-embedding:')) {
return { openaiCompatible: { dimensions: dims } };
}
// MiniMax embo-01 takes a `type: 'db' | 'query'` field for asymmetric
// retrieval. Today still hardcoded to 'db' for back-compat — opting
// into the new inputType seam is a follow-up (see plan's deferred
+75 -2
View File
@@ -599,6 +599,8 @@ function warnRecipesMissingBatchTokens(): void {
// LiteLLM proxy, llama-server) — they ship without a static cap because
// the cap depends on a user-launched server. Warning is noise for them.
if (embedding.no_batch_cap === true) continue;
// A declared item-count cap is a real batch cap — no warning needed.
if (embedding.max_batch_items !== undefined) continue;
if (_warnedRecipes.has(recipe.id)) continue;
_warnedRecipes.add(recipe.id);
// eslint-disable-next-line no-console
@@ -619,6 +621,7 @@ export function resetGateway(): void {
_embedTransportInstalled = false;
_chatTransport = null;
_warnedRecipes.clear();
_warnedUnavailable.clear();
_extendedModels.clear();
}
@@ -885,6 +888,47 @@ export function isAvailable(touchpoint: TouchpointKind, modelOverride?: string):
}
}
/**
* Human-readable reason a chat-capable touchpoint is unavailable, or null
* when it IS available. Names the configured model, the missing
* `auth_env.required` keys, and the recipe's `setup_url` so silent-degrade
* guards (#3062: facts extraction, query expansion) can warn with an
* actionable message instead of no-oping invisibly.
*/
export function unavailableReason(touchpoint: 'chat' | 'expansion'): string | null {
if (isAvailable(touchpoint)) return null;
if (!_config) return `${touchpoint} gateway not configured (no AI gateway config loaded)`;
try {
const modelStr = touchpoint === 'expansion' ? getExpansionModel() : getChatModel();
const { recipe } = resolveRecipe(modelStr);
if (!recipe.touchpoints[touchpoint]) {
return `${touchpoint} model ${modelStr}: provider recipe "${recipe.id}" does not support the ${touchpoint} touchpoint`;
}
const missing = (recipe.auth_env?.required ?? []).filter(k => !_config!.env[k]);
if (missing.length > 0) {
const setup = recipe.auth_env?.setup_url ? ` (get a key: ${recipe.auth_env.setup_url})` : '';
return `${touchpoint} model ${modelStr} needs ${missing.join(', ')}${setup}`;
}
return `${touchpoint} model ${modelStr} is unavailable`;
} catch (e) {
return `${touchpoint} gateway unavailable: ${e instanceof Error ? e.message : String(e)}`;
}
}
/**
* Once-per-process memo for silent-degrade warnings (#3062). Cleared by
* resetGateway() so a reconfigure gets a fresh warning if still broken.
*/
const _warnedUnavailable = new Set<string>();
export function warnUnavailableOnce(touchpoint: 'chat' | 'expansion', context: string): void {
if (_warnedUnavailable.has(touchpoint)) return;
const reason = unavailableReason(touchpoint);
if (!reason) return;
_warnedUnavailable.add(touchpoint);
// eslint-disable-next-line no-console
console.warn(`[ai.gateway] WARN: ${context}${reason}`);
}
// ---- Embedding ----
/**
@@ -1517,10 +1561,17 @@ export async function embed(texts: string[], opts?: EmbedOpts): Promise<Float32A
// Pre-split is gated on max_batch_tokens. Recipes without it (e.g. OpenAI)
// ride the fast path: one embedMany call, no recursion safety net.
const batches = maxBatchTokens
const tokenBatches = maxBatchTokens
? splitByTokenBudget(truncated, Math.floor(maxBatchTokens * effectiveSafetyFactor(recipe)), charsPerToken)
: [truncated];
// Hard COUNT cap (e.g. llama-server's "maximum allowed batch size 32").
// Token budget can't bound item count, so re-split any oversized batch.
const maxBatchItems = embedding?.max_batch_items;
const batches = maxBatchItems
? tokenBatches.flatMap(b => capBatchItems(b, maxBatchItems))
: tokenBatches;
const allEmbeddings: Float32Array[] = [];
let _embedThrew = false;
try {
@@ -1596,6 +1647,23 @@ export function splitByTokenBudget(
return batches;
}
/**
* Split a batch into sub-batches of at most `maxItems` inputs. Enforces a
* hard COUNT cap that the token-budget split can't (many tiny inputs fit
* under any token budget). Used for endpoints like llama.cpp's llama-server
* that reject requests exceeding their launch batch size.
*
* @internal exported for tests; not part of the public gateway API.
*/
export function capBatchItems(texts: string[], maxItems: number): string[][] {
if (maxItems <= 0 || texts.length <= maxItems) return [texts];
const batches: string[][] = [];
for (let i = 0; i < texts.length; i += maxItems) {
batches.push(texts.slice(i, i + maxItems));
}
return batches;
}
/**
* Returns true if the error looks like a provider batch-token-limit error.
*
@@ -2280,7 +2348,12 @@ const ExpansionSchema = z.object({
*/
export async function expand(query: string): Promise<string[]> {
if (!query || !query.trim()) return [query];
if (!isAvailable('expansion')) return [query];
if (!isAvailable('expansion')) {
// #3062: tokenmax's headline knob silently degrading to single-query
// was invisible. Warn once per process; still degrade gracefully.
warnUnavailableOnce('expansion', 'query expansion is configured but inert; searches run single-query');
return [query];
}
// Guardrail seam: classify the query before the expansion model call.
await classifyGatewayGuardrail({
+6 -3
View File
@@ -35,9 +35,12 @@ export const llamaServer: Recipe = {
trust_custom_dims: true, // #2271: user knows the launched model's native dim
cost_per_1m_tokens_usd: 0,
price_last_verified: '2026-05-10',
// llama-server's batch capacity is set by `--ctx-size` at launch
// time; no static cap to declare. v0.32 (#779).
no_batch_cap: true,
// llama-server enforces a hard request-COUNT cap equal to its launch
// batch size (`--batch-size`, default 32): it rejects requests with
// more inputs with `batch size N > maximum allowed batch size 32`.
// The token-budget split can't bound item count, so cap it here. A
// server launched with a larger `-b` can raise this. v0.32 (#779).
max_batch_items: 32,
},
},
/**
+10
View File
@@ -54,6 +54,16 @@ export interface EmbeddingTouchpoint {
* `max_batch_tokens` is also set.
*/
safety_factor?: number;
/**
* Maximum number of inputs per embedding request. Some endpoints enforce a
* hard COUNT cap independent of token budget notably llama.cpp's
* `llama-server`, which rejects requests with more inputs than its launch
* batch size (e.g. `batch size 100 > maximum allowed batch size 32`). The
* token-budget pre-split cannot bound item count (many tiny chunks fit under
* any token budget), so this is enforced as a separate hard re-split after
* the token split. When unset, no count cap is applied.
*/
max_batch_items?: number;
/**
* v0.27.1: when true, at least one model in this recipe accepts image
* inputs via a multimodal embedding endpoint (e.g. Voyage's
+13 -3
View File
@@ -217,9 +217,19 @@ export function parseResolverEntries(resolverContent: string): ResolverEntry[] {
// `skillsDir/*/SKILL.md` when manifest.json is missing — the scenario
// needed for AGENTS.md-only OpenClaw deployments. See D-CX-12 / F-ENG-1.
/** Simple YAML frontmatter parser — extracts triggers array if present. */
function extractTriggers(skillContent: string): string[] {
const fmMatch = skillContent.match(/^---\n([\s\S]*?)\n---/);
/**
* Simple YAML frontmatter parser extracts triggers array if present.
*
* Normalizes CRLF LF before parsing so Windows checkouts (where
* `core.autocrlf=true` is the default) parse correctly. Without this,
* the `^---\n` and `^triggers:\s*\n` regexes never match because the
* file content is `---\r\n` / `triggers:\r\n`, and every skill on
* Windows is reported as `mece_gap` regardless of its actual content.
* CI runs on Ubuntu-only so the bug only surfaces in user environments.
*/
export function extractTriggers(skillContent: string): string[] {
const content = skillContent.replace(/\r\n/g, '\n');
const fmMatch = content.match(/^---\n([\s\S]*?)\n---/);
if (!fmMatch) return [];
const fm = fmMatch[1];
const triggersMatch = fm.match(/^triggers:\s*\n((?:\s+-\s+.+\n?)*)/m);
+79 -2
View File
@@ -168,6 +168,15 @@ export interface CodeChunkOptions {
largeChunkThresholdTokens?: number;
fallbackChunkSizeWords?: number;
fallbackOverlapWords?: number;
/**
* Hard upper bound (estimated tokens) on any single emitted chunk. A node
* the AST splitter can't break up (a giant object/array literal, a single
* huge assignment, a massive template literal) would otherwise be emitted
* whole and rejected by the embedder ("input exceeds context length").
* Chunks over this budget are recursively re-split. Default 2000 fits the
* smallest common embedder context (e.g. nomic-embed-text, 2048).
*/
maxChunkTokens?: number;
}
/**
@@ -549,6 +558,7 @@ export function parseWithTimeout(
}
const DEFAULT_CHUNKER_TIMEOUT_MS = 30_000;
const DEFAULT_MAX_CHUNK_TOKENS = 2000;
function resolveChunkerTimeoutMs(): number {
const raw = process.env.GBRAIN_CHUNKER_TIMEOUT_MS;
@@ -706,9 +716,9 @@ export async function chunkCodeTextFull(
}
if (chunks.length === 0) {
return { chunks: fallbackChunks(source, filePath, language, opts), edges: rawEdges };
return { chunks: capOversizedChunks(fallbackChunks(source, filePath, language, opts), filePath, language, opts), edges: rawEdges };
}
return { chunks: mergeSmallSiblings(chunks, chunkTarget), edges: rawEdges };
return { chunks: capOversizedChunks(mergeSmallSiblings(chunks, chunkTarget), filePath, language, opts), edges: rawEdges };
} catch {
return { chunks: fallbackChunks(source, filePath, language, opts), edges: [] };
} finally {
@@ -814,6 +824,73 @@ function buildMergedChunk(group: CodeChunk[], index: number): CodeChunk {
};
}
/**
* Final safety net: guarantee no emitted chunk exceeds the embedder's context
* budget. tree-sitter splitting (splitLargeNode) can only break up a node that
* exposes a `body` with >= 2 named children. A node without one a giant
* object/array literal, a single huge assignment, a massive template literal
* is emitted whole, producing a chunk far larger than the embedder accepts.
* The embedder then rejects it ("input exceeds context length") and the chunk
* is never embedded. Recursively re-split any over-budget chunk; fall back to a
* hard character split for pathological no-whitespace content (e.g. a minified
* one-liner) where word/line splitting can't get under budget.
*/
function capOversizedChunks(
chunks: CodeChunk[],
filePath: string,
language: SupportedCodeLanguage,
opts: CodeChunkOptions,
): CodeChunk[] {
const cap = opts.maxChunkTokens ?? DEFAULT_MAX_CHUNK_TOKENS;
if (!chunks.some((c) => estimateTokens(c.text) > cap)) return chunks;
const out: CodeChunk[] = [];
for (const c of chunks) {
if (estimateTokens(c.text) <= cap) {
out.push({ ...c, index: out.length });
continue;
}
// Strip the structured header ("[Lang] path:N-M symbol\n\n") so the splitter
// works on the raw body; buildChunk re-adds a header to each piece.
const body = c.text.replace(/^\[[^\]]+\] [^\n]+\n\n/, '');
for (const piece of splitToTokenBudget(body, cap, opts)) {
if (!piece.trim()) continue;
out.push(buildChunk({
body: piece,
filePath,
language,
symbolName: c.metadata.symbolName,
symbolType: c.metadata.symbolType,
startLine: c.metadata.startLine,
endLine: c.metadata.endLine,
index: out.length,
parentSymbolPath: c.metadata.parentSymbolPath,
}));
}
}
return out;
}
/** Split `text` into pieces each estimated <= cap tokens. Word/line-aware
* (recursiveChunk) first; a hard character split is the last resort for
* content with no whitespace to break on. */
function splitToTokenBudget(text: string, cap: number, opts: CodeChunkOptions): string[] {
const out: string[] = [];
const pieces = recursiveChunk(text, {
chunkSize: opts.fallbackChunkSizeWords ?? 300,
chunkOverlap: opts.fallbackOverlapWords ?? 50,
}).map((p) => p.text);
for (const piece of pieces) {
if (estimateTokens(piece) <= cap) {
out.push(piece);
continue;
}
// ~3.5 chars/token is a conservative cl100k estimate for source text.
const charBudget = Math.max(1, Math.floor(cap * 3.5));
for (let i = 0; i < piece.length; i += charBudget) out.push(piece.slice(i, i + charBudget));
}
return out;
}
// ---------- Internals ----------
function fallbackChunks(
+33 -1
View File
@@ -620,7 +620,10 @@ export function loadConfig(): GBrainConfig | null {
* size the schema and must be stable across engine connect.
*/
export async function loadConfigWithEngine(
engine: { getConfig(key: string): Promise<string | null | undefined> },
engine: {
getConfig(key: string): Promise<string | null | undefined>;
listConfigKeys?(prefix: string): Promise<string[]>;
},
base?: GBrainConfig | null,
): Promise<GBrainConfig | null> {
// Codex /ship finding #3: when there's no file config AND no env DB URL,
@@ -657,11 +660,31 @@ export async function loadConfigWithEngine(
return undefined;
}
}
async function dbPrefixMap(prefix: string): Promise<Record<string, string> | undefined> {
if (typeof engine.listConfigKeys !== 'function') return undefined;
let keys: string[];
try {
keys = await engine.listConfigKeys(prefix);
} catch {
return undefined;
}
const out: Record<string, string> = {};
for (const key of keys.sort()) {
if (!key.startsWith(prefix)) continue;
const leaf = key.slice(prefix.length);
if (!leaf) continue;
const value = await dbStr(key);
if (value !== undefined) out[leaf] = value;
}
return Object.keys(out).length > 0 ? out : undefined;
}
const dbMultimodal = await dbBool('embedding_multimodal');
const dbMultimodalModel = await dbStr('embedding_multimodal_model');
const dbOcr = await dbBool('embedding_image_ocr');
const dbOcrModel = await dbStr('embedding_image_ocr_model');
const dbProviderBaseUrls = await dbPrefixMap('provider_base_urls.');
// v0.36 (D7) — embedding-column registry merge. Stored as JSON string in
// the config table. Parse + shape-check here; full registry validation
// (regex on keys, type/dim/provider field shapes) runs in the resolver at
@@ -685,6 +708,15 @@ export async function loadConfigWithEngine(
if (merged.embedding_image_ocr_model === undefined && dbOcrModel !== undefined) {
merged.embedding_image_ocr_model = dbOcrModel;
}
if (dbProviderBaseUrls !== undefined) {
const next = { ...(merged.provider_base_urls ?? {}) };
for (const [providerId, baseUrl] of Object.entries(dbProviderBaseUrls)) {
if (next[providerId] === undefined) next[providerId] = baseUrl;
}
if (Object.keys(next).length > 0) {
merged.provider_base_urls = next;
}
}
if (merged.embedding_columns === undefined && dbEmbeddingColumns !== undefined) {
try {
const parsed = JSON.parse(dbEmbeddingColumns);
+22 -5
View File
@@ -54,6 +54,7 @@ import {
import {
generatePerChunkSynopsis,
SYNOPSIS_PROMPT_VERSION,
SYNOPSIS_DOC_MAX_CHARS,
type GeneratePerChunkSynopsisResult,
} from './page-summary.ts';
import {
@@ -103,8 +104,17 @@ function getEmbeddingModelTag(): string {
export function computeCorpusGeneration(args: {
crMode: CRMode;
haikuModel: string;
/**
* Resolved `SYNOPSIS_DOC_MAX_CHARS` for per_chunk_synopsis runs. When
* present, folded into the hash so changes to
* `GBRAIN_SYNOPSIS_DOC_MAX_CHARS` invalidate the prior cache cleanly.
* Omit for `crMode !== 'per_chunk_synopsis'` title / none modes
* don't consult the cap and the field stays out of the hash for
* back-compat with pre-cap embeddings.
*/
synopsisDocMaxChars?: number;
}): string {
return createHash('sha256')
const h = createHash('sha256')
.update(args.crMode)
.update('|')
.update(String(SYNOPSIS_PROMPT_VERSION))
@@ -113,9 +123,11 @@ export function computeCorpusGeneration(args: {
.update('|')
.update(String(TITLE_WRAPPER_VERSION))
.update('|')
.update(getEmbeddingModelTag())
.digest('hex')
.slice(0, 16);
.update(getEmbeddingModelTag());
if (args.synopsisDocMaxChars !== undefined) {
h.update('|doc_cap=').update(String(args.synopsisDocMaxChars));
}
return h.digest('hex').slice(0, 16);
}
/**
@@ -253,7 +265,11 @@ export async function reembedPageWithContextualRetrieval(
args.pageSlug,
args.sourceId,
resolution.mode,
computeCorpusGeneration({ crMode: resolution.mode, haikuModel: args.haikuModel ?? DEFAULT_HAIKU_MODEL }),
computeCorpusGeneration({
crMode: resolution.mode,
haikuModel: args.haikuModel ?? DEFAULT_HAIKU_MODEL,
synopsisDocMaxChars: resolution.mode === 'per_chunk_synopsis' ? SYNOPSIS_DOC_MAX_CHARS : undefined,
}),
);
return { kind: 'skipped', reason: 'no_chunks' };
}
@@ -282,6 +298,7 @@ export async function reembedPageWithContextualRetrieval(
const corpus_generation = computeCorpusGeneration({
crMode: attemptMode,
haikuModel,
synopsisDocMaxChars: attemptMode === 'per_chunk_synopsis' ? SYNOPSIS_DOC_MAX_CHARS : undefined,
});
// ── PHASE 2: single DB transaction ───────────────────────────
+53 -17
View File
@@ -23,14 +23,24 @@
* page coordinate only; legacy NULL-source_markdown_slug rows survive
* because deleteFactsForPage targets source_markdown_slug = slug only.
*
* Empty-fence guard (Codex R2-#7): the phase refuses to do its
* destructive reconciliation pass when legacy rows (row_num IS NULL,
* entity_slug IS NOT NULL) still exist in the brain they're the
* v0.31 hot-memory facts pending the v0_32_2 backfill. Status returns
* `warn` with a hint to run `gbrain apply-migrations --yes`. Without
* the guard, an interrupted upgrade where v0_32_2 hasn't run could
* leave the cycle silently misreporting "0 facts on people/alice"
* while legacy rows linger in the DB.
* Empty-fence guard (Codex R2-#7; #2484): the phase refuses to do its
* destructive reconciliation pass when genuinely-backfillable legacy
* rows still exist `row_num IS NULL` (never fenced) AND `entity_slug`
* resolves to a live page in this source (so the v0_32_2 migration's
* Phase B could fence them). Status returns `warn` with a hint to run
* `gbrain apply-migrations --yes`. Without the guard, an interrupted
* upgrade where v0_32_2 hasn't run could leave the cycle silently
* misreporting "0 facts on people/alice" while legacy rows linger.
*
* The live-page requirement (#2484) is load-bearing: the inline facts
* writer keeps producing `row_num IS NULL, entity_slug IS NOT NULL`
* rows AFTER the migration completes, whenever a resolved slug has no
* fenceable page (slugify-floor / stub-guard-blocked unprefixed slugs).
* Those are structurally unfenceable no page to fence onto, and the
* ledger-complete migration won't re-run so they must NOT gate, or
* the phase jams forever (~16/day observed). Requiring a backing page
* keeps genuine pre-v0.32.2 rows (whose entity page exists) gating
* while excluding the inline-writer's permanent-unfenceable rows.
*/
import type { BrainEngine } from '../engine.ts';
@@ -163,22 +173,48 @@ export async function runExtractFacts(
phantomsMorePending: false,
};
// ── Empty-fence guard (Codex R2-#7) ────────────────────────────
// Pre-check: if any legacy fact rows exist (row_num NULL but
// entity_slug NOT NULL), refuse to run the destructive
// reconciliation pass. The v0_32_2 orchestrator must complete
// first.
// ── Empty-fence guard (Codex R2-#7; #2484) ─────────────────────
// Pre-check: if any genuinely-backfillable legacy fact rows exist,
// refuse to run the destructive reconciliation pass — the v0_32_2
// orchestrator must fence them first.
//
// A row is a real backfill candidate only when `row_num IS NULL`
// (never fenced) AND its `entity_slug` resolves to a LIVE page in
// this source (the migration's Phase B only fences rows whose
// entity_slug maps to a writable page). #2484: the original
// predicate was just `row_num IS NULL AND entity_slug IS NOT NULL`,
// which ALSO matched structurally-unfenceable hot-memory rows the
// inline writer keeps producing post-migration: the legacy DB-only
// fallback (backstop.ts) writes `entity_slug` (a resolved slug, e.g.
// a slugify-floor or stub-guard-blocked unprefixed slug like
// `people-jane-doe`) with `row_num` NULL whenever the slug has no
// fenceable page. Those rows can never satisfy the migration's exit
// condition (no page to fence onto, and `apply-migrations` is a
// ledger-complete no-op for them), so they jammed the phase forever
// — ~16/day, mislabeled "v0.31 pending backfill." We now require a
// live backing page, which both genuine pre-v0.32.2 rows (their
// entity page exists) satisfy and inline-writer unfenceable rows do
// not.
const legacy = await engine.executeRaw<{ n: string }>(
`SELECT COUNT(*) AS n FROM facts WHERE row_num IS NULL AND entity_slug IS NOT NULL`,
`SELECT COUNT(*) AS n
FROM facts f
WHERE f.row_num IS NULL
AND f.entity_slug IS NOT NULL
AND EXISTS (
SELECT 1 FROM pages p
WHERE p.source_id = f.source_id
AND p.slug = f.entity_slug
AND p.deleted_at IS NULL
)`,
);
const legacyCount = parseInt(legacy[0]?.n ?? '0', 10);
result.legacyRowsPending = legacyCount;
if (legacyCount > 0) {
result.guardTriggered = true;
result.warnings.push(
`extract_facts: ${legacyCount} legacy v0.31 fact rows pending fence backfill. ` +
`Run \`gbrain apply-migrations --yes\` to complete v0_32_2 before this phase ` +
`can safely reconcile fence → DB.`,
`extract_facts: ${legacyCount} legacy v0.31 fact rows (entity page present, not yet ` +
`fenced) pending fence backfill. Run \`gbrain apply-migrations --yes\` to complete ` +
`v0_32_2 before this phase can safely reconcile fence → DB.`,
);
return result;
}
+103 -3
View File
@@ -55,6 +55,17 @@ import type { PhaseStatus, CyclePhase } from '../cycle.ts';
*/
export const PROPOSE_TAKES_PROMPT_VERSION = 'v0.36.1.0-tuned-cat15';
/**
* Sentinel claim_text for the tombstone row written when a page extracts
* ZERO gradeable claims. Without a tombstone the idempotency tuple is never
* recorded, so every cycle re-spends an LLM call on unchanged zero-claim
* prose the "unchanged page never re-spends tokens" contract only held
* for pages that produced >=1 claim. The tombstone is inserted with
* status='rejected' so no pending-review query surfaces it as a live
* proposal; its only job is to make the next cycle a cache hit.
*/
export const EMPTY_EXTRACTION_TOMBSTONE_TEXT = '(no gradeable claims)';
/**
* Tuned extractor prompt, validated against the hand-labeled synthetic
* corpus at test/fixtures/calibration/. Measured F1 on first live run
@@ -152,6 +163,8 @@ export interface ProposeTakesResult {
cache_hits: number;
cache_misses: number;
proposals_inserted: number;
/** Idempotency rows written for pages that extracted zero claims. */
tombstones_written: number;
budget_exhausted: boolean;
warnings: string[];
}
@@ -234,7 +247,43 @@ export async function defaultExtractor(
});
// ChatResult.text is already the concatenated text content.
return parseExtractorOutput(result.text);
const takes = parseExtractorOutput(result.text);
// A parse-level `[]` is AMBIGUOUS: it means either "the model genuinely
// found no gradeable claims" OR "the model returned malformed/prose/
// truncated output we couldn't parse." The caller memoizes empty
// extractions with a tombstone, so a transient parse failure would
// PERMANENTLY suppress a page that actually has claims. Only a cleanly
// parsed empty array is a real "no claims" result worth memoizing; treat
// anything else as a transient error and throw, so the phase's catch
// retries the page next cycle (writing no tombstone).
if (takes.length === 0 && !isWellFormedEmptyExtraction(result.text)) {
throw new Error('propose_takes extractor: no parseable takes JSON (transient — retry)');
}
return takes;
}
/**
* True only when `raw` is a cleanly-parseable EMPTY JSON array the
* well-behaved "no gradeable claims" response (the prompt instructs the model
* to return `[]`). Distinguishes a genuine empty extraction (safe to memoize
* via a tombstone) from malformed / prose / truncated output (transient
* must be retried, never tombstoned). Mirrors parseExtractorOutput's
* fence-strip + first-array handling so both agree on what "the model
* returned []" means.
*/
export function isWellFormedEmptyExtraction(raw: string): boolean {
if (!raw || raw.trim().length === 0) return false;
let text = raw.trim();
const fenced = text.match(/^```(?:json)?\s*\n?([\s\S]*?)\n?```$/);
if (fenced) text = (fenced[1] ?? '').trim();
const arrStart = text.indexOf('[');
if (arrStart === -1) return false;
try {
const parsed = JSON.parse(text.slice(arrStart));
return Array.isArray(parsed) && parsed.length === 0;
} catch {
return false;
}
}
/**
@@ -246,6 +295,8 @@ export async function defaultExtractor(
export function parseExtractorOutput(raw: string): ProposedTake[] {
if (!raw || raw.trim().length === 0) return [];
let text = raw.trim();
// Strip <think>...</think> reasoning tags (MiniMax-M3, DeepSeek-R1, etc.).
text = text.replace(/<think>[\s\S]*?<\/think>/g, '').trim();
// Strip markdown code fence wrapper.
const fenced = text.match(/^```(?:json)?\s*\n?([\s\S]*?)\n?```$/);
if (fenced) text = (fenced[1] ?? '').trim();
@@ -258,7 +309,21 @@ export function parseExtractorOutput(raw: string): ProposedTake[] {
try {
parsed = JSON.parse(text.slice(start));
} catch {
return [];
// Fallback: truncate at last ] or } to handle trailing noise (e.g. leftover
// markdown fences after <think> stripping). Try array-closing first.
const sliced = text.slice(start);
const lastArr = sliced.lastIndexOf(']');
const lastObj = sliced.lastIndexOf('}');
const end = Math.max(lastArr, lastObj);
if (end > 0) {
try {
parsed = JSON.parse(sliced.slice(0, end + 1));
} catch {
return [];
}
} else {
return [];
}
}
const arr = Array.isArray(parsed) ? parsed : [parsed];
const out: ProposedTake[] = [];
@@ -314,6 +379,7 @@ class ProposeTakesPhase extends BaseCyclePhase {
cache_hits: 0,
cache_misses: 0,
proposals_inserted: 0,
tombstones_written: 0,
budget_exhausted: false,
warnings: [],
};
@@ -415,6 +481,40 @@ class ProposeTakesPhase extends BaseCyclePhase {
);
result.proposals_inserted += 1;
}
// Memoize the empty case too. A page that extracted zero claims gets
// NO row from the loop above, so without this its idempotency tuple is
// never recorded and the next cycle re-spends an LLM call on unchanged
// prose (the idle-cost bug). Write one tombstone row keyed by the same
// (source, slug, content_hash, prompt_version) tuple. status='rejected'
// keeps it out of any pending-review query; its sole purpose is to make
// the next cycle a cache hit. Only reached on a SUCCESSFUL empty extract
// — the extractor-throw path `continue`s above, so failed pages are
// retried rather than tombstoned.
if (proposals.length === 0) {
await engine.executeRaw(
`INSERT INTO take_proposals
(source_id, page_slug, content_hash, prompt_version, proposal_run_id,
claim_text, kind, holder, weight, domain, dedup_against_fence_rows, model_id, status)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, 'rejected')
ON CONFLICT (source_id, page_slug, content_hash, prompt_version) DO NOTHING`,
[
sourceId,
page.slug,
ch,
promptVersion,
proposalRunId,
EMPTY_EXTRACTION_TOMBSTONE_TEXT,
'fact',
'brain',
0,
null,
JSON.stringify(existingTakes),
opts.model ?? 'claude-sonnet-4-6',
],
);
result.tombstones_written += 1;
}
}
if (opts.reporter) opts.reporter.finish();
@@ -448,7 +548,7 @@ class ProposeTakesPhase extends BaseCyclePhase {
});
return {
summary: `propose_takes: scanned ${result.pages_scanned} pages, ${result.cache_hits} cached, ${result.proposals_inserted} new proposals (run ${proposalRunId})`,
summary: `propose_takes: scanned ${result.pages_scanned} pages, ${result.cache_hits} cached, ${result.proposals_inserted} new proposals, ${result.tombstones_written} empty (run ${proposalRunId})`,
details: { ...result, proposal_run_id: proposalRunId, prompt_version: promptVersion },
status: result.budget_exhausted ? 'warn' : 'ok',
};
+10
View File
@@ -1218,11 +1218,21 @@ export interface BrainEngine {
*
* Uses the `%` trigram operator (GIN-indexed) + the standard `similarity()`
* function. Both engines support pg_trgm (PGLite 0.3+, Postgres always).
*
* `sourceId` constrains the search to a single source and filters out
* soft-deleted pages. Mirrors the same filters `tryFuzzyMatch` in
* `src/core/entities/resolve.ts` got via #1436 (v0.41.13.0). Omit for the
* historical unscoped behavior live-mode callers that already know
* the source should pass it to avoid cross-source slug suggestions that
* get silently dropped at the FK filter downstream. Batch-mode callers
* (e.g. `gbrain extract`) intentionally omit it to build a cross-source
* resolution map.
*/
findByTitleFuzzy(
name: string,
dirPrefix?: string,
minSimilarity?: number,
sourceId?: string,
): Promise<{ slug: string; similarity: number } | null>;
/**
* v0.34.1 (#861 P0 leak seal): `opts.sourceId` / `opts.sourceIds`
+14 -2
View File
@@ -78,7 +78,7 @@ export type FactsBackstopResult =
mode: 'queue';
enqueued: boolean;
queueDepth: number;
skipped?: 'extraction_disabled' | 'queue_overflow' | 'queue_shutdown' | `eligibility_failed:${string}`;
skipped?: 'extraction_disabled' | 'chat_unavailable' | 'queue_overflow' | 'queue_shutdown' | `eligibility_failed:${string}`;
}
| {
mode: 'inline';
@@ -86,7 +86,7 @@ export type FactsBackstopResult =
duplicate: number;
superseded: number;
fact_ids: number[];
skipped?: 'extraction_disabled' | `eligibility_failed:${string}`;
skipped?: 'extraction_disabled' | 'chat_unavailable' | `eligibility_failed:${string}`;
};
interface ParsedPageInput {
@@ -155,6 +155,18 @@ export async function runFactsBackstop(
: { mode: 'inline', inserted: 0, duplicate: 0, superseded: 0, fact_ids: [], skipped };
}
// #3062: no chat gateway → extraction is guaranteed to yield nothing. The
// result was previously byte-identical to a genuine empty extraction
// (inserted: 0, no `skipped`), and queue mode enqueued jobs doomed to
// no-op. Record WHY, warn once per process, and skip the queue entirely.
const { isAvailable, warnUnavailableOnce } = await import('../ai/gateway.ts');
if (!isAvailable('chat')) {
warnUnavailableOnce('chat', 'facts extraction skipped');
return mode === 'queue'
? { mode: 'queue', enqueued: false, queueDepth: 0, skipped: 'chat_unavailable' }
: { mode: 'inline', inserted: 0, duplicate: 0, superseded: 0, fact_ids: [], skipped: 'chat_unavailable' };
}
// --- Mode dispatch ---
if (mode === 'queue') {
// Local patch 2026-06-11: in a one-shot CLI process the in-process queue
+5 -2
View File
@@ -21,7 +21,7 @@
* gateway-down errors are absorbed into NULL-embedding rows.
*/
import { chat, embedOne, isAvailable } from '../ai/gateway.ts';
import { chat, embedOne, isAvailable, warnUnavailableOnce } from '../ai/gateway.ts';
import type { ChatResult } from '../ai/gateway.ts';
import { INJECTION_PATTERNS } from '../think/sanitize.ts';
import { resolveModel } from '../model-config.ts';
@@ -175,7 +175,10 @@ export async function extractFactsFromTurn(input: ExtractInput): Promise<Extract
if (!isAvailable('chat')) {
// No chat gateway → no extraction. Caller still inserts facts via direct
// `gbrain take add` paths.
// `gbrain take add` paths. #3062: warn once per process — silently
// returning [] here made an unauthenticated brain byte-identical to a
// genuine empty extraction across every health surface.
warnUnavailableOnce('chat', 'facts extraction skipped');
return [];
}
+5
View File
@@ -733,6 +733,11 @@ export async function importFromContent(
: computeCorpusGeneration({
crMode: effectiveCRMode,
haikuModel: 'anthropic:claude-haiku-4-5-20251001',
// Inline import-file path never uses per_chunk_synopsis (refuses
// upstream); pass undefined so the doc-cap field stays out of
// the hash here. Per_chunk_synopsis runs through the Minion
// backfill handler which threads SYNOPSIS_DOC_MAX_CHARS through
// the service layer.
});
// Transaction wraps all DB writes. Every per-page tx call carries the
+22 -3
View File
@@ -489,7 +489,22 @@ export async function extractPageLinks(
// text inside `[[...]]` before any `|`), NOT the display alias
// (ref.name = match[2]). `[[struktura|the project]]` must resolve
// `struktura`, not "the project". The display text is for context only.
const matches = await resolver.resolveBasenameMatches(ref.slug);
//
// The literal may be path-qualified (`[[notes/struktura]]`). The FS
// path (resolveSlugAll) strips the dirname before its basename lookup,
// but this path passed the raw literal to an index keyed by final
// segments only — so every slash-containing wikilink outside
// DIR_PATTERN silently resolved to nothing. Query by the final
// segment, then use the written path as a disambiguation filter
// (the analogue of the FS ancestor walk honoring the written path):
// a match must end with the literal, so `[[notes/struktura]]` can
// resolve to `vault/notes/struktura` but never to `wiki/struktura`.
const slashIdx = ref.slug.lastIndexOf('/');
const basename = slashIdx === -1 ? ref.slug : ref.slug.slice(slashIdx + 1);
let matches = await resolver.resolveBasenameMatches(basename);
if (slashIdx !== -1) {
matches = matches.filter(m => m === ref.slug || m.endsWith(`/${ref.slug}`));
}
if (matches.length === 0) continue;
const idx = content.indexOf(ref.slug);
const context = idx >= 0 ? excerpt(content, idx, 240) : ref.name;
@@ -965,10 +980,14 @@ export function makeResolver(
// Step 3: pg_trgm fuzzy title match — both modes. Tries each hint in
// order; first hint with a ≥0.55 similarity match wins. If no hints,
// try the whole pages table.
// try the whole pages table. When opts.sourceId is set, the fuzzy
// search is constrained to that source (and skips soft-deleted pages)
// so cross-source slug suggestions don't get silently dropped at the
// FK filter downstream. Mirrors the same scope fix `tryFuzzyMatch` got
// via #1436.
const searchHints = hints.length > 0 ? hints : [undefined];
for (const hint of searchHints) {
const match = await engine.findByTitleFuzzy(trimmed, hint, 0.55);
const match = await engine.findByTitleFuzzy(trimmed, hint, 0.55, opts.sourceId);
if (match) {
cache.set(cacheKey, match.slug);
return match.slug;
+35 -1
View File
@@ -135,7 +135,16 @@ export function parseMarkdown(
const type = coerceFrontmatterString(frontmatter.type) || (
opts?.activePack ? inferTypeFromPack(filePath, opts.activePack) : inferType(filePath)
);
const title = coerceFrontmatterString(frontmatter.title).trim() || inferTitle(filePath);
// #2446: title precedence is frontmatter `title:` > the body's first H1 >
// the slug/filename-humanized fallback. Slug-based imports (contacts,
// calendar) write a correct `# Heading` but no frontmatter title; without
// the H1 fallback they get junk titles humanized from the slug
// (`Contact 20170928 5 John Defalco`), which also breaks anything keyed on
// the title (e.g. the by-mention gazetteer's first-token bucketing).
const title =
coerceFrontmatterString(frontmatter.title).trim() ||
inferTitleFromBody(body) ||
inferTitle(filePath);
const tags = extractTags(frontmatter);
const slug = coerceFrontmatterString(frontmatter.slug) || inferSlug(filePath);
@@ -602,6 +611,31 @@ function inferTypeWithPrefixes(
return 'concept';
}
/**
* #2446: derive a title from the body's first ATX H1 (`# Heading`).
*
* Returns the trimmed heading text with the leading `# ` and any decorative
* trailing `#` run stripped, or '' if the body has no H1. Only a SINGLE leading
* `#` matches `##`+ (h2 and deeper) are skipped and lines inside a fenced
* code block (```/~~~) are ignored so a `# comment` in a shell snippet can't be
* mistaken for the page title.
*/
function inferTitleFromBody(body: string): string {
let inFence = false;
for (const raw of body.split('\n')) {
const fence = /^\s*(`{3,}|~{3,})/.exec(raw);
if (fence) {
inFence = !inFence;
continue;
}
if (inFence) continue;
// Exactly one leading `#`, then whitespace, then the heading text.
const m = /^#(?!#)\s+(.+?)\s*$/.exec(raw);
if (m) return m[1].replace(/\s+#+\s*$/, '').trim();
}
return '';
}
function inferTitle(filePath?: string): string {
if (!filePath) return 'Untitled';
+5
View File
@@ -24,6 +24,7 @@
*/
const THIRTY_MIN_MS = 30 * 60 * 1000;
const SIXTY_MIN_MS = 60 * 60 * 1000;
const TEN_MIN_MS = 10 * 60 * 1000;
/**
@@ -42,6 +43,10 @@ export const HANDLER_DEFAULT_TIMEOUT_MS: Readonly<Record<string, number>> = {
// few writes. Generous 10-min budget (vs the tight null-default) covers a
// slow gateway without the 30-min loop budget.
chronicle_extract: TEN_MIN_MS,
// Per-page contextual reindex jobs process chunks sequentially with one
// rate-leased LLM synopsis call per chunk; large transcript pages need more
// than the standard 30-min long-job budget.
contextual_reindex_per_chunk: SIXTY_MIN_MS,
};
/**
+8
View File
@@ -87,6 +87,11 @@ function walkMarkdownAndMdxFiles(
for (const entry of entries) {
if (truncated) return;
if (entry.startsWith('.')) continue;
// Skip heavy non-content dirs so the walk doesn't exhaust the time
// budget on dependency/build trees (node_modules can be 50k+ files
// with zero .md). These are never gbrain page sources.
if (entry === 'node_modules' || entry === 'dist' || entry === 'build' ||
entry === '.next' || entry === 'vendor' || entry === 'target') continue;
const full = join(d, entry);
let isDir = false;
try {
@@ -95,6 +100,9 @@ function walkMarkdownAndMdxFiles(
continue;
}
if (isDir) {
// Time check on directory descent too, so a deep dependency-free
// tree still respects the deadline even before any .md is found.
if (Date.now() >= deadlineMs) { truncated = true; return; }
walk(full);
continue;
}
+37 -3
View File
@@ -1153,7 +1153,11 @@ async function runAutoLink(
// Live-mode resolver: per-put throwaway cache, pg_trgm + optional search.
// Issue #972 (codex [P1]): pass sourceId so basename resolution stays
// within this page's source — no cross-source basename edges.
// within this page's source — no cross-source basename edges. Also scopes
// the fuzzy fallback (findByTitleFuzzy) to the same source the put_page is
// targeting — without it, cross-source slug suggestions get silently dropped
// at the FK filter and the link looks like it failed to resolve. Twin of
// #1436's `tryFuzzyMatch` fix.
const resolver = makeResolver(engine, { mode: 'live', sourceId: opts?.sourceId });
// Issue #972: opt-in bare-wikilink basename resolution. Off by default.
const globalBasename = await isGlobalBasenameEnabled(engine);
@@ -1384,7 +1388,11 @@ const list_pages: Operation = {
params: {
type: { type: 'string', description: 'Filter by page type' },
tag: { type: 'string', description: 'Filter by tag' },
limit: { type: 'number', description: 'Max results (default 50)' },
limit: { type: 'number', description: 'Max results (default 50; remote callers are capped at 100)' },
offset: {
type: 'number',
description: 'Skip first N rows (pagination). Engine-supported since PageFilters gained offset; previously accepted at the CLI and silently dropped.',
},
// v0.29 — surface filter that already exists on PageFilters.
updated_after: {
type: 'string',
@@ -1411,10 +1419,36 @@ const list_pages: Operation = {
// were ignored at this op handler and the engine returned every source's
// pages indiscriminately.
const scope = sourceScopeOpts(ctx);
// The 100-row cap exists to protect remote MCP/OAuth transports from
// unbounded result dumps. Local CLI callers (ctx.remote === false — the
// same trust boundary that already bypasses scope enforcement, see the
// Operation.scope doc above) own the machine, and a full enumeration is a
// legitimate local operation, so an explicit limit above 100 is honored.
// Anything that is not strictly `false` stays remote/untrusted (defense
// in depth, matching the ctx.remote contract).
const requestedLimit = p.limit as number | undefined;
const isLocal = ctx.remote === false;
const limit = isLocal
? clampSearchLimit(requestedLimit, 50, Number.MAX_SAFE_INTEGER)
: clampSearchLimit(requestedLimit, 50, 100);
if (!isLocal && requestedLimit !== undefined && Number.isFinite(requestedLimit) && requestedLimit > limit) {
// Loud clamp, parity with the three search paths ("search limit clamped
// from N to 100"). logger.warn goes to stderr — `list` stdout is
// tab-separated and consumed by scripts, so it must stay clean.
ctx.logger.warn(`[gbrain] Warning: list limit clamped from ${requestedLimit} to ${limit}; use offset to paginate`);
}
// Thread offset through — PageFilters has supported it all along; the op
// layer just never passed it, so `--offset` was accepted and ignored.
const requestedOffset = p.offset as number | undefined;
const offset =
requestedOffset !== undefined && Number.isFinite(requestedOffset) && requestedOffset > 0
? Math.floor(requestedOffset)
: undefined;
const pages = await ctx.engine.listPages({
type: p.type as any,
tag: p.tag as string,
limit: clampSearchLimit(p.limit as number | undefined, 50, 100),
limit,
offset,
includeDeleted: (p.include_deleted as boolean) === true,
updated_after: typeof p.updated_after === 'string' ? p.updated_after : undefined,
sort,
+116
View File
@@ -0,0 +1,116 @@
/**
* Shared orphan-reporting exclusion policy.
*
* These are pages where "no inbound links" is expected and should not count
* against health. Keep this in core so the CLI orphan report and engine health
* dashboard cannot drift.
*
* Defaults are GBrain-wide conventions only. Brain-specific exclusions
* (private folder names, one-off fixture slugs) belong in the brain's own
* config, not here:
*
* gbrain config set orphans.exclude_prefixes "my-private-folder/,archive/"
* gbrain config set orphans.exclude_slugs "some-one-off-page"
*/
const AUTO_SUFFIX_PATTERNS = ['/_index', '/log'];
const PSEUDO_SLUGS = new Set(['_atlas', '_index', '_stats', '_orphans', '_scratch', 'claude']);
const RAW_SEGMENT = '/raw/';
const DENY_PREFIXES = [
'output/',
'dashboards/',
'scripts/',
'templates/',
'_templates/',
'openclaw/config/',
'extracts/',
];
const FIRST_SEGMENT_EXCLUSIONS = new Set([
'scratch',
'thoughts',
'catalog',
'entities',
'raw',
'atoms',
'skills',
'dreaming',
'daily',
]);
const ROOT_DATE_SLUG = /^\d{4}-\d{2}-\d{2}(?:-.+)?$/;
function isAgentWorkspaceConvention(slug: string): boolean {
if (!slug.startsWith('agents/')) return false;
if (slug.includes('/memory/dreaming/')) return true;
return /^agents\/[^/]+\/(?:agents|identity|soul|tools|user|heartbeat|dreams|dormant)$/.test(slug);
}
/** Per-brain additions to the convention defaults (from config). */
export interface OrphanPolicyOverrides {
excludePrefixes?: string[];
excludeSlugs?: string[];
}
/** Config keys for per-brain orphan exclusions (comma-separated values). */
export const ORPHAN_EXCLUDE_PREFIXES_KEY = 'orphans.exclude_prefixes';
export const ORPHAN_EXCLUDE_SLUGS_KEY = 'orphans.exclude_slugs';
function parseList(value: string | null): string[] {
if (!value) return [];
return value.split(',').map(s => s.trim()).filter(Boolean);
}
/**
* Load per-brain orphan exclusions from the brain config table. Callers with
* an engine in hand (getHealth, `gbrain orphans`) pass the result as the
* second argument to shouldExcludeFromOrphanReporting.
*/
export async function loadOrphanPolicyOverrides(
engine: { getConfig(key: string): Promise<string | null> },
): Promise<OrphanPolicyOverrides> {
const [prefixes, slugs] = await Promise.all([
engine.getConfig(ORPHAN_EXCLUDE_PREFIXES_KEY),
engine.getConfig(ORPHAN_EXCLUDE_SLUGS_KEY),
]);
return { excludePrefixes: parseList(prefixes), excludeSlugs: parseList(slugs) };
}
export function shouldExcludeFromOrphanReporting(
slug: string,
overrides?: OrphanPolicyOverrides,
): boolean {
if (PSEUDO_SLUGS.has(slug)) return true;
for (const suffix of AUTO_SUFFIX_PATTERNS) {
if (slug.endsWith(suffix)) return true;
}
if (slug.includes(RAW_SEGMENT)) return true;
if (slug.includes('/daily/')) return true;
for (const prefix of DENY_PREFIXES) {
if (slug.startsWith(prefix)) return true;
}
const firstSegment = slug.split('/')[0];
if (FIRST_SEGMENT_EXCLUSIONS.has(firstSegment)) return true;
if (ROOT_DATE_SLUG.test(slug)) return true;
if (slug.startsWith('_brain-')) return true;
if (isAgentWorkspaceConvention(slug)) return true;
if (overrides) {
if (overrides.excludeSlugs?.includes(slug)) return true;
for (const prefix of overrides.excludePrefixes ?? []) {
if (slug.startsWith(prefix)) return true;
}
}
return false;
}
+36 -1
View File
@@ -44,6 +44,33 @@ const HAIKU_MAX_TOKENS = 200;
/** Default model when caller doesn't override. Resolves through the gateway. */
const DEFAULT_SYNOPSIS_MODEL = 'anthropic:claude-haiku-4-5-20251001';
/**
* Hard cap on `documentText` length (chars) before send.
*
* 2026-05-25 fix wave: small local chat models (Gemma 4 E2B, Qwen3 4B) get
* dramatically slower on long contexts even with 131K-token windows declared.
* A 73K-char page synopsis on Gemma 4 E2B takes 60-120s, exceeding the
* worker's default 30s `lockDuration` and tripping `lock-lost` errors.
*
* Truncate to a budget that fits a small model's effective throughput while
* preserving enough document context for the synopsis to be useful. Truncates
* the TAIL because the head (title, frontmatter, intro) carries the
* document-level anchor the synopsis needs.
*
* Override per workload via `GBRAIN_SYNOPSIS_DOC_MAX_CHARS`. Default 32768
* (~8K tokens at 4 chars/tok) keeps small-model synopsis under ~30s.
* Anthropic Haiku is unaffected at this cap; bump higher when running
* frontier models if you want richer document anchoring.
*/
export const SYNOPSIS_DOC_MAX_CHARS = (() => {
const env = process.env.GBRAIN_SYNOPSIS_DOC_MAX_CHARS;
if (env && /^\d+$/.test(env)) {
const n = parseInt(env, 10);
if (n >= 512 && n <= 1_048_576) return n;
}
return 32768;
})();
/**
* Synopsis prompt version. Folded into corpus_generation so prompt edits
* invalidate prior embeddings via the v0.40.3.0 query_cache.page_generations
@@ -188,11 +215,19 @@ function buildUserPrompt(
documentText: string,
chunkText: string,
): string {
// Tail-truncate `documentText` to `SYNOPSIS_DOC_MAX_CHARS` so small local
// chat models don't stall on >100KB pages. Head preserved (title block,
// frontmatter, intro paragraphs carry the document-level anchor).
let trimmedDoc = documentText;
if (documentText.length > SYNOPSIS_DOC_MAX_CHARS) {
trimmedDoc = documentText.slice(0, SYNOPSIS_DOC_MAX_CHARS) +
`\n\n[... ${documentText.length - SYNOPSIS_DOC_MAX_CHARS} chars truncated for synopsis budget ...]`;
}
return [
`<page_title>${pageTitle}</page_title>`,
'',
'<full_document>',
documentText,
trimmedDoc,
'</full_document>',
'',
'<chunk>',
+74 -28
View File
@@ -57,6 +57,8 @@ import { finalizeLastSeen } from './chronicle/last-seen.ts';
import { computeAnomaliesFromBuckets } from './cycle/anomaly.ts';
import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts';
import { buildSourceFactorCase, buildHardExcludeClause, buildVisibilityClause, buildRecencyComponentSql, buildBestPerPagePoolCte, buildOrFallbackWebsearchQuery } from './search/sql-ranking.ts';
import { shouldExcludeFromOrphanReporting, loadOrphanPolicyOverrides } from './orphan-policy.ts';
import { LINK_EXTRACTOR_VERSION_TS } from './link-extraction.ts';
import {
normalizeEngineColumn,
buildVectorCastFragment,
@@ -1058,6 +1060,16 @@ export class PGLiteEngine implements BrainEngine {
RETURNING id, source_id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at, effective_date, effective_date_source, import_filename, source_kind, source_uri, ingested_via, ingested_at`,
[sourceId, slug, page.type, pageKind, page.title, page.compiled_truth, page.timeline || '', JSON.stringify(frontmatter), hash, effectiveDate, effectiveDateSource, importFilename, chunkerVersion, sourcePath, sourceKind, sourceUri, ingestedVia, ingestedAt]
);
// PGLite can return zero rows from INSERT ... ON CONFLICT DO UPDATE ...
// RETURNING in no-op/trigger edge cases, which made rowToPage(undefined)
// throw "undefined is not an object (evaluating 'row.deleted_at')" and
// skip the file during sync. The row WAS written, so re-read instead of
// crashing.
if (rows.length === 0) {
const reread = await this.getPage(slug, { sourceId });
if (reread) return reread;
throw new Error(`putPage: RETURNING produced no row for ${sourceId}/${slug}`);
}
return rowToPage(rows[0] as Record<string, unknown>);
}
@@ -2322,6 +2334,10 @@ export class PGLiteEngine implements BrainEngine {
// v0.40.3.0 D24 NULL→non-NULL race fix mirrors postgres-engine.ts. Two writers
// racing on the same chunk previously raced last-write-wins; the fix lets the
// fresher `embedded_at` win in the text-unchanged branch.
//
// Code-chunk metadata columns follow the same chunk_text-gated CASE pattern as `embedding`
// (#769). Re-chunk trusts EXCLUDED outright; pure re-embed COALESCEs so a caller carrying
// only embedding-shaped fields doesn't clobber metadata to NULL.
await this.db.query(
`INSERT INTO content_chunks ${cols} VALUES ${rowParts.join(', ')}
ON CONFLICT (page_id, chunk_index) DO UPDATE SET
@@ -2345,14 +2361,14 @@ export class PGLiteEngine implements BrainEngine {
THEN EXCLUDED.embedded_at
ELSE content_chunks.embedded_at
END,
language = EXCLUDED.language,
symbol_name = EXCLUDED.symbol_name,
symbol_type = EXCLUDED.symbol_type,
start_line = EXCLUDED.start_line,
end_line = EXCLUDED.end_line,
parent_symbol_path = EXCLUDED.parent_symbol_path,
doc_comment = EXCLUDED.doc_comment,
symbol_name_qualified = EXCLUDED.symbol_name_qualified,
language = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.language ELSE COALESCE(EXCLUDED.language, content_chunks.language) END,
symbol_name = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.symbol_name ELSE COALESCE(EXCLUDED.symbol_name, content_chunks.symbol_name) END,
symbol_type = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.symbol_type ELSE COALESCE(EXCLUDED.symbol_type, content_chunks.symbol_type) END,
start_line = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.start_line ELSE COALESCE(EXCLUDED.start_line, content_chunks.start_line) END,
end_line = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.end_line ELSE COALESCE(EXCLUDED.end_line, content_chunks.end_line) END,
parent_symbol_path = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.parent_symbol_path ELSE COALESCE(EXCLUDED.parent_symbol_path, content_chunks.parent_symbol_path) END,
doc_comment = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.doc_comment ELSE COALESCE(EXCLUDED.doc_comment, content_chunks.doc_comment) END,
symbol_name_qualified = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.symbol_name_qualified ELSE COALESCE(EXCLUDED.symbol_name_qualified, content_chunks.symbol_name_qualified) END,
modality = EXCLUDED.modality,
embedding_image = COALESCE(EXCLUDED.embedding_image, content_chunks.embedding_image)`,
params
@@ -2906,22 +2922,41 @@ export class PGLiteEngine implements BrainEngine {
name: string,
dirPrefix?: string,
minSimilarity: number = 0.55,
sourceId?: string,
): Promise<{ slug: string; similarity: number } | null> {
// Inline threshold comparison instead of `SET LOCAL pg_trgm.similarity_threshold`.
// The GUC only scopes to the current transaction and pglite auto-commits each
// .query() call, so the SET LOCAL would be a no-op. Using similarity() >= $N
// directly gives predictable behavior. Tie-breaker: sort by slug so re-runs
// pick the same winner.
//
// `sourceId` + `deleted_at IS NULL` mirror the filters `tryFuzzyMatch` in
// `src/core/entities/resolve.ts` got via #1436 (v0.41.13.0). Without them,
// fuzzy resolution could suggest cross-source slugs that the caller then
// silently drops at the FK filter — making it look like the match failed
// when in fact it picked the wrong page.
const prefixPattern = dirPrefix ? `${dirPrefix}/%` : '%';
const { rows } = await this.db.query(
`SELECT slug, similarity(title, $1) AS sim
FROM pages
WHERE similarity(title, $1) >= $3
AND slug LIKE $2
ORDER BY sim DESC, slug ASC
LIMIT 1`,
[name, prefixPattern, minSimilarity]
);
const { rows } = sourceId
? await this.db.query(
`SELECT slug, similarity(title, $1) AS sim
FROM pages
WHERE similarity(title, $1) >= $3
AND slug LIKE $2
AND source_id = $4
AND deleted_at IS NULL
ORDER BY sim DESC, slug ASC
LIMIT 1`,
[name, prefixPattern, minSimilarity, sourceId]
)
: await this.db.query(
`SELECT slug, similarity(title, $1) AS sim
FROM pages
WHERE similarity(title, $1) >= $3
AND slug LIKE $2
ORDER BY sim DESC, slug ASC
LIMIT 1`,
[name, prefixPattern, minSimilarity]
);
if (rows.length === 0) return null;
const row = rows[0] as { slug: string; sim: number };
return { slug: row.slug, similarity: row.sim };
@@ -5207,15 +5242,10 @@ export class PGLiteEngine implements BrainEngine {
(SELECT count(*) FROM pages) as page_count,
(SELECT count(*) FROM content_chunks WHERE embedded_at IS NOT NULL)::float /
GREATEST((SELECT count(*) FROM content_chunks), 1)::float as embed_coverage,
(SELECT count(*) FROM pages p
WHERE p.updated_at < (SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id)
) as stale_pages,
-- Bug 11 orphan = islanded (no inbound AND no outbound).
-- See BrainHealth.orphan_pages docstring; docs updated to match this.
(SELECT count(*) FROM pages p
WHERE NOT EXISTS (SELECT 1 FROM links l WHERE l.to_page_id = p.id)
AND NOT EXISTS (SELECT 1 FROM links l WHERE l.from_page_id = p.id)
) as orphan_pages,
0 as stale_pages,
-- Bug 11 orphan = islanded (no inbound AND no outbound). The raw
-- list is filtered in TS using the shared orphan-reporting policy.
0 as orphan_pages,
(SELECT count(*) FROM links l
WHERE NOT EXISTS (SELECT 1 FROM pages p WHERE p.id = l.to_page_id)
) as dead_links,
@@ -5240,10 +5270,20 @@ export class PGLiteEngine implements BrainEngine {
LIMIT 5
`);
const { rows: islandedRows } = await this.db.query(`
SELECT p.slug
FROM pages p
WHERE NOT EXISTS (SELECT 1 FROM links l WHERE l.to_page_id = p.id)
AND NOT EXISTS (SELECT 1 FROM links l WHERE l.from_page_id = p.id)
`);
const r = h as Record<string, unknown>;
const pageCount = Number(r.page_count);
const embedCoverage = Number(r.embed_coverage);
const orphanPages = Number(r.orphan_pages);
const stalePages = await this.countStalePagesForExtraction({ versionTs: LINK_EXTRACTOR_VERSION_TS });
const orphanOverrides = await loadOrphanPolicyOverrides(this);
const orphanPages = (islandedRows as { slug: string }[])
.filter(row => !shouldExcludeFromOrphanReporting(row.slug, orphanOverrides)).length;
const deadLinks = Number(r.dead_links);
const linkCount = Number(r.link_count);
const pagesWithTimeline = Number(r.pages_with_timeline);
@@ -5271,7 +5311,7 @@ export class PGLiteEngine implements BrainEngine {
return {
page_count: pageCount,
embed_coverage: embedCoverage,
stale_pages: Number(r.stale_pages),
stale_pages: stalePages,
orphan_pages: orphanPages,
missing_embeddings: Number(r.missing_embeddings),
brain_score: brainScore,
@@ -5826,6 +5866,11 @@ export class PGLiteEngine implements BrainEngine {
params.push(escaped);
prefixCondition = `AND p.slug LIKE $${params.length} ESCAPE '\\'`;
}
// TIM-37: exclude briefing pages from their own Brain Pulse. See the
// matching block in postgres-engine.ts getRecentSalience() for context.
const excludeBriefings = !(slugPrefix && slugPrefix.startsWith('briefings'))
? `AND p.slug NOT LIKE 'briefings/%'`
: '';
params.push(limit);
const limitParam = `$${params.length}`;
@@ -5861,6 +5906,7 @@ export class PGLiteEngine implements BrainEngine {
LEFT JOIN takes t ON t.page_id = p.id AND t.active = TRUE
WHERE GREATEST(p.updated_at, COALESCE(p.salience_touched_at, p.updated_at)) >= $1::timestamptz
${prefixCondition}
${excludeBriefings}
GROUP BY p.id
ORDER BY score DESC
LIMIT ${limitParam}`,
+68 -30
View File
@@ -67,6 +67,8 @@ import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts';
import { buildSourceFactorCase, buildHardExcludeClause, buildVisibilityClause, buildRecencyComponentSql, buildBestPerPagePoolCte, buildOrFallbackWebsearchQuery } from './search/sql-ranking.ts';
import { DEFAULT_EMBEDDING_MODEL, DEFAULT_EMBEDDING_DIMENSIONS } from './ai/defaults.ts';
import { DELETE_BATCH_SIZE } from './engine-constants.ts';
import { shouldExcludeFromOrphanReporting, loadOrphanPolicyOverrides } from './orphan-policy.ts';
import { LINK_EXTRACTOR_VERSION_TS } from './link-extraction.ts';
function escapeSqlStringLiteral(value: string): string {
return value.replace(/'/g, "''");
@@ -2473,6 +2475,13 @@ export class PostgresEngine implements BrainEngine {
// - new is fresher (embedded_at > existing.embedded_at) → take new
// - otherwise → keep existing (slower writer with stale embedding loses)
// Mirrored in pglite-engine.ts; pinned by test/e2e/concurrent-embed-race.test.ts.
//
// Code-chunk metadata columns (language / symbol_name / symbol_type / line range /
// parent_symbol_path / doc_comment / symbol_name_qualified) follow the SAME chunk_text-gated
// CASE pattern as `embedding` (#769). Re-chunk (chunk_text changed) trusts EXCLUDED outright;
// pure re-embed (chunk_text unchanged) COALESCEs so a caller that only carries embedding
// doesn't clobber metadata to NULL. Without this, every embed --stale pass nuked code-def's
// primary index for thousands of chunks at once.
await sql.unsafe(
`INSERT INTO content_chunks ${cols} VALUES ${rows.join(', ')}
ON CONFLICT (page_id, chunk_index) DO UPDATE SET
@@ -2496,14 +2505,14 @@ export class PostgresEngine implements BrainEngine {
THEN EXCLUDED.embedded_at
ELSE content_chunks.embedded_at
END,
language = EXCLUDED.language,
symbol_name = EXCLUDED.symbol_name,
symbol_type = EXCLUDED.symbol_type,
start_line = EXCLUDED.start_line,
end_line = EXCLUDED.end_line,
parent_symbol_path = EXCLUDED.parent_symbol_path,
doc_comment = EXCLUDED.doc_comment,
symbol_name_qualified = EXCLUDED.symbol_name_qualified,
language = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.language ELSE COALESCE(EXCLUDED.language, content_chunks.language) END,
symbol_name = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.symbol_name ELSE COALESCE(EXCLUDED.symbol_name, content_chunks.symbol_name) END,
symbol_type = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.symbol_type ELSE COALESCE(EXCLUDED.symbol_type, content_chunks.symbol_type) END,
start_line = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.start_line ELSE COALESCE(EXCLUDED.start_line, content_chunks.start_line) END,
end_line = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.end_line ELSE COALESCE(EXCLUDED.end_line, content_chunks.end_line) END,
parent_symbol_path = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.parent_symbol_path ELSE COALESCE(EXCLUDED.parent_symbol_path, content_chunks.parent_symbol_path) END,
doc_comment = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.doc_comment ELSE COALESCE(EXCLUDED.doc_comment, content_chunks.doc_comment) END,
symbol_name_qualified = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.symbol_name_qualified ELSE COALESCE(EXCLUDED.symbol_name_qualified, content_chunks.symbol_name_qualified) END,
modality = EXCLUDED.modality,
embedding_image = COALESCE(EXCLUDED.embedding_image, content_chunks.embedding_image)`,
params as Parameters<typeof sql.unsafe>[1],
@@ -3073,6 +3082,7 @@ export class PostgresEngine implements BrainEngine {
name: string,
dirPrefix?: string,
minSimilarity: number = 0.55,
sourceId?: string,
): Promise<{ slug: string; similarity: number } | null> {
const sql = this.sql;
// Use the `similarity()` function directly with an explicit threshold
@@ -3085,15 +3095,33 @@ export class PostgresEngine implements BrainEngine {
// Tie-breaker: sort by slug after similarity so re-runs return the
// same winner when multiple pages score equally (prevents churn
// in put_page auto-link reconciliation).
//
// `sourceId` + `deleted_at IS NULL` mirror the filters `tryFuzzyMatch`
// in `src/core/entities/resolve.ts` got via #1436 (v0.41.13.0). Without
// them, fuzzy resolution could suggest cross-source slugs that the
// caller then silently drops at the FK filter in
// `operations.ts:reconcileLinks` (the `allSlugs` filter) — making it
// look like the match failed when in fact it picked the wrong page.
const prefixPattern = dirPrefix ? `${dirPrefix}/%` : '%';
const rows = await sql`
SELECT slug, similarity(title, ${name}) AS sim
FROM pages
WHERE similarity(title, ${name}) >= ${minSimilarity}
AND slug LIKE ${prefixPattern}
ORDER BY sim DESC, slug ASC
LIMIT 1
`;
const rows = sourceId
? await sql`
SELECT slug, similarity(title, ${name}) AS sim
FROM pages
WHERE similarity(title, ${name}) >= ${minSimilarity}
AND slug LIKE ${prefixPattern}
AND source_id = ${sourceId}
AND deleted_at IS NULL
ORDER BY sim DESC, slug ASC
LIMIT 1
`
: await sql`
SELECT slug, similarity(title, ${name}) AS sim
FROM pages
WHERE similarity(title, ${name}) >= ${minSimilarity}
AND slug LIKE ${prefixPattern}
ORDER BY sim DESC, slug ASC
LIMIT 1
`;
if (rows.length === 0) return null;
const row = rows[0] as { slug: string; sim: number };
return { slug: row.slug, similarity: row.sim };
@@ -5313,11 +5341,9 @@ export class PostgresEngine implements BrainEngine {
async getHealth(): Promise<BrainHealth> {
const sql = this.sql;
// Bug 11 doc-drift fix — orphan_pages means "islanded" (no inbound AND
// no outbound links), aligning both engines with the user-facing
// definition. The type comment previously said "no inbound" but the
// SQL required both — docs now match code so users can trust the
// number. A hub page that links out to many but has no back-references
// is working as intended, not an orphan.
// no outbound links). The raw islanded list is filtered through the same
// policy as `gbrain orphans` so convention pages do not count against
// dashboard health.
const [h] = await sql`
WITH entity_pages AS (
SELECT id, slug FROM pages WHERE type IN ('person', 'company')
@@ -5326,13 +5352,8 @@ export class PostgresEngine implements BrainEngine {
(SELECT count(*) FROM pages) as page_count,
(SELECT count(*) FROM content_chunks WHERE embedded_at IS NOT NULL)::float /
GREATEST((SELECT count(*) FROM content_chunks), 1)::float as embed_coverage,
(SELECT count(*) FROM pages p
WHERE p.updated_at < (SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id)
) as stale_pages,
(SELECT count(*) FROM pages p
WHERE NOT EXISTS (SELECT 1 FROM links l WHERE l.to_page_id = p.id)
AND NOT EXISTS (SELECT 1 FROM links l WHERE l.from_page_id = p.id)
) as orphan_pages,
0 as stale_pages,
0 as orphan_pages,
(SELECT count(*) FROM links l
WHERE NOT EXISTS (SELECT 1 FROM pages p WHERE p.id = l.to_page_id)
) as dead_links,
@@ -5356,9 +5377,18 @@ export class PostgresEngine implements BrainEngine {
LIMIT 5
`;
const islandedRows = await sql<{ slug: string }[]>`
SELECT p.slug
FROM pages p
WHERE NOT EXISTS (SELECT 1 FROM links l WHERE l.to_page_id = p.id)
AND NOT EXISTS (SELECT 1 FROM links l WHERE l.from_page_id = p.id)
`;
const pageCount = Number(h.page_count);
const embedCoverage = Number(h.embed_coverage);
const orphanPages = Number(h.orphan_pages);
const stalePages = await this.countStalePagesForExtraction({ versionTs: LINK_EXTRACTOR_VERSION_TS });
const orphanOverrides = await loadOrphanPolicyOverrides(this);
const orphanPages = islandedRows.filter(row => !shouldExcludeFromOrphanReporting(row.slug, orphanOverrides)).length;
const deadLinks = Number(h.dead_links);
const linkCount = Number(h.link_count);
const pagesWithTimeline = Number(h.pages_with_timeline);
@@ -5386,7 +5416,7 @@ export class PostgresEngine implements BrainEngine {
return {
page_count: pageCount,
embed_coverage: embedCoverage,
stale_pages: Number(h.stale_pages),
stale_pages: stalePages,
orphan_pages: orphanPages,
missing_embeddings: Number(h.missing_embeddings),
brain_score: brainScore,
@@ -6142,6 +6172,13 @@ export class PostgresEngine implements BrainEngine {
const prefixCondition = slugPrefix
? sql`AND p.slug LIKE ${slugPrefix.replace(/[\\%_]/g, (c) => '\\' + c) + '%'} ESCAPE '\\'`
: sql``;
// TIM-37: exclude briefing pages from their own Brain Pulse. The cron
// briefing writes to 90_Briefings/, gets re-ingested, and would otherwise
// top tomorrow's salience as pure self-reference. Suppress unless the
// caller explicitly asked for the briefings/ prefix.
const excludeBriefings = !(slugPrefix && slugPrefix.startsWith('briefings'))
? sql`AND p.slug NOT LIKE 'briefings/%'`
: sql``;
// v0.29.1: third score term via buildRecencyComponentSql. Default
// 'flat' = v0.29.0 behavior (1 / (1 + days_old)). 'on' opts into the
// per-prefix decay map (concepts/ evergreen, daily/ aggressive, etc.).
@@ -6175,6 +6212,7 @@ export class PostgresEngine implements BrainEngine {
LEFT JOIN takes t ON t.page_id = p.id AND t.active = TRUE
WHERE GREATEST(p.updated_at, COALESCE(p.salience_touched_at, p.updated_at)) >= ${boundaryIso}::timestamptz
${prefixCondition}
${excludeBriefings}
GROUP BY p.id
ORDER BY score DESC
LIMIT ${limit}
+10 -4
View File
@@ -93,7 +93,13 @@ import { resolveLrSchedule } from './lr-schedule.ts';
import { preflight, formatPreflightReport } from './preflight.ts';
import { isRejected, loadRejectedBuffer, makeRejectedEntry, saveRejectedBuffer } from './rejected-buffer.ts';
import { runReflect, runOneShotRewrite, describeJudges } from './reflect.ts';
import { acceptCandidate, bestPath, revertAllPending, skillPath, writeProposed } from './version-store.ts';
import {
acceptCandidate,
proposedPath as proposedFilePath,
revertAllPending,
skillPath,
writeProposed,
} from './version-store.ts';
import { runValidationGate, scoreSkillOnTasks } from './validate-gate.ts';
import { ROLLOUT_SUCCESS_THRESHOLD } from './types.ts';
import type { SkillOptOpts, EditOp, RunReceipt, BenchmarkTask } from './types.ts';
@@ -702,9 +708,9 @@ async function runOptimizationLoop(
// to the catch's assignment values only (it can't prove the async callback ran).
const finalOutcome = outcome as 'accepted' | 'no_improvement' | 'aborted' | 'errored';
if (!mutateDecision.mutate && finalOutcome === 'accepted') {
// best.md was written by writeProposed() in the accept branch (no-mutate
// path); it doubles as proposed.md for human review. SKILL.md untouched.
proposedPath = bestPath(skillsDir, skillName);
// writeProposed() emitted both the best pointer and the stable review
// artifact in the accept branch. SKILL.md remains untouched.
proposedPath = proposedFilePath(skillsDir, skillName);
} else if (mutateDecision.mutate) {
mutatedSkillFile = finalOutcome === 'accepted';
}
+15 -9
View File
@@ -23,6 +23,7 @@
*
* history.json
* best.md
* proposed.md
* versions/
* v0001_e1_s1.md
* v0002_e1_s2.md
@@ -52,6 +53,10 @@ export function bestPath(skillsDir: string, skillName: string): string {
return path.join(skilloptDir(skillsDir, skillName), 'best.md');
}
export function proposedPath(skillsDir: string, skillName: string): string {
return path.join(skilloptDir(skillsDir, skillName), 'proposed.md');
}
export function skillPath(skillsDir: string, skillName: string): string {
return path.join(skillsDir, skillName, 'SKILL.md');
}
@@ -171,17 +176,18 @@ export function acceptCandidate(input: AcceptInput): AcceptResult {
}
/**
* Write the candidate to `best.md` (which doubles as `proposed.md`) WITHOUT
* touching SKILL.md or the history ledger. Used by the `--no-mutate` /
* bundled-without-allow paths: the optimizer found a better candidate but the
* caller opted out of in-place mutation, so we surface it for human review.
* Returns the path written. Atomic (.tmp + rename).
* Write the candidate to both `best.md` and `proposed.md` WITHOUT touching
* SKILL.md or the history ledger. `best.md` remains the optimizer's current
* best pointer; `proposed.md` is the stable human-review artifact promised by
* `--no-mutate`. Returns the proposal path. Each write is atomic (.tmp + rename).
*/
export function writeProposed(skillsDir: string, skillName: string, candidateText: string): string {
const p = bestPath(skillsDir, skillName);
fs.mkdirSync(path.dirname(p), { recursive: true });
atomicWrite(p, candidateText);
return p;
const best = bestPath(skillsDir, skillName);
const proposed = proposedPath(skillsDir, skillName);
fs.mkdirSync(path.dirname(best), { recursive: true });
atomicWrite(best, candidateText);
atomicWrite(proposed, candidateText);
return proposed;
}
/**
+8 -1
View File
@@ -195,7 +195,14 @@ export class SupabaseStorage implements StorageBackend {
throw new Error(`Supabase signed URL failed: ${res.status} ${body}`);
}
const result = await res.json() as { signedURL: string };
return `${this.projectUrl}${result.signedURL}`;
// Supabase returns `signedURL` relative to the Storage API root, e.g.
// "/object/sign/<bucket>/<path>?token=...". Prepend projectUrl + "/storage/v1"
// (not just projectUrl) or the link 404s. Tolerate an already-absolute URL or a
// value that already carries the /storage/v1 prefix.
const signed = result.signedURL;
if (/^https?:\/\//.test(signed)) return signed;
if (signed.startsWith('/storage/v1')) return `${this.projectUrl}${signed}`;
return `${this.projectUrl}/storage/v1${signed.startsWith('/') ? '' : '/'}${signed}`;
}
async getUrl(path: string): Promise<string> {
+35 -1
View File
@@ -553,6 +553,40 @@ export async function runThink(
};
}
/**
* Strip a "## Gaps" section from an answer body.
*
* `think` returns gaps in the structured `gaps` array, which the CLI and the
* persisted synthesis page render exactly once. The system prompt also used to
* ask for a "Gaps" section inside the answer prose, so a model that still emits
* one would make the output show "## Gaps" twice once from the prose, once
* from the structured array. This removes the prose section so the structured
* array stays the single source of truth.
*
* Matches a heading line `## Gaps` (level 2-6, case-insensitive) and removes it
* through the next heading of the same-or-higher level, or end of string.
* Returns the input unchanged when there is no such section.
*/
export function stripGapsSection(answer: string): string {
if (!answer) return answer;
const lines = answer.split('\n');
let start = -1;
let level = 0;
for (let i = 0; i < lines.length; i++) {
const m = /^(#{2,6})\s+gaps\s*$/i.exec(lines[i]);
if (m) { start = i; level = m[1].length; break; }
}
if (start === -1) return answer;
let end = lines.length;
for (let i = start + 1; i < lines.length; i++) {
const h = /^(#{1,6})\s+\S/.exec(lines[i]);
if (h && h[1].length <= level) { end = i; break; }
}
const kept = [...lines.slice(0, start), ...lines.slice(end)].join('\n');
// Drop trailing blank lines left by removing a trailing section.
return kept.replace(/\s+$/, '');
}
/**
* Persist a synthesis page + its evidence. Returns the saved slug.
* Synthesis pages are written under `synthesis/<slugified-question>-<date>.md`.
@@ -582,7 +616,7 @@ export async function persistSynthesis(
const body = [
`# ${result.question}`,
'',
result.answer,
stripGapsSection(result.answer),
'',
result.gaps.length > 0 ? '## Gaps\n\n' + result.gaps.map(g => `- ${g}`).join('\n') : '',
].filter(Boolean).join('\n');
+6 -6
View File
@@ -52,19 +52,19 @@ Hard rules:
rather than asserting it as established. Confidence is part of the data.
- If two takes contradict (different holders, opposite claims), surface BOTH in a "Conflicts"
section. Never silently pick one.
- If you cannot answer because the brain doesn't contain the relevant data, say so in the
"Gaps" section. List the specific missing pieces. Do not make up answers.
- If the brain doesn't contain data needed to answer, do NOT make it up. Record each
missing piece in the structured "gaps" array (below), not as a section in the answer prose.
- Never instruct the user (no "you should" / "I recommend X"). The brain reports; the user decides.
- Output MUST be valid JSON matching the schema below. No prose outside JSON.
Output schema:
{
"answer": "<markdown body. Inline citations like [slug#row] or [slug]. Sections: Answer, Conflicts (optional), Gaps>",
"answer": "<markdown body. Inline citations like [slug#row] or [slug]. Sections: Answer, Conflicts (optional). Do NOT add a Gaps section here — gaps belong in the gaps array.>",
"citations": [
{"page_slug": "people/alice-example", "row_num": 3, "citation_index": 1},
{"page_slug": "companies/acme-example", "row_num": null, "citation_index": 2}
],
"gaps": ["specific missing data point 1", "specific missing data point 2"]
"gaps": ["a specific, self-contained missing-or-stale data point, citing the [slug] where relevant", "another specific gap"]
}
The "row_num" field is required for take citations and MUST be null for page-only citations.`;
@@ -83,7 +83,7 @@ export function buildThinkSystemPrompt(opts: ThinkSystemPromptOpts = {}): string
lines.push(`\nThis is a temporal question. Order key claims chronologically when it helps the reader.`);
}
if (opts.willSave) {
lines.push(`\nThis synthesis will be persisted as a brain page. Aim for completeness — cover Answer, Conflicts, and Gaps thoroughly.`);
lines.push(`\nThis synthesis will be persisted as a brain page. Aim for completeness — cover the Answer and any Conflicts thoroughly, and list every missing piece in the structured "gaps" array.`);
}
if (opts.withCalibration) {
lines.push(
@@ -92,7 +92,7 @@ export function buildThinkSystemPrompt(opts: ThinkSystemPromptOpts = {}): string
lines.push(`- Name both the user's PRIOR (default reasoning) AND the COUNTER-PRIOR from their hedged-domain self.`);
lines.push(`- Reference active bias tags by name when relevant ("this fits the over-confident-geography pattern").`);
lines.push(`- Do NOT silently substitute the debiased answer. ALWAYS surface both priors transparently.`);
lines.push(`- Track-record sentences belong in a "Calibration" section in the answer body, between Conflicts and Gaps.`);
lines.push(`- Track-record sentences belong in a "Calibration" section in the answer body, after the Conflicts section (if present).`);
}
return lines.join('\n');
}
+16 -15
View File
@@ -63,25 +63,26 @@ interface PluginCtx {
[key: string]: unknown;
}
export function register(api: PluginApi) {
api.registerContextEngine(ENGINE_ID, (ctx: PluginCtx) => {
const hostResolver =
typeof ctx.resolveEntities === 'function'
? ctx.resolveEntities
: typeof ctx.brainQuery === 'function'
? ctx.brainQuery
: undefined;
return createGBrainContextEngine({
workspaceDir: ctx.workspaceDir,
resolveEntities: hostResolver,
});
});
}
const entry: PluginEntry = {
id: 'gbrain-context-engine',
name: 'GBrain Context Engine',
description: 'Deterministic temporal/spatial context injection on every turn',
register(api: PluginApi) {
api.registerContextEngine(ENGINE_ID, (ctx: PluginCtx) => {
const hostResolver =
typeof ctx.resolveEntities === 'function'
? ctx.resolveEntities
: typeof ctx.brainQuery === 'function'
? ctx.brainQuery
: undefined;
return createGBrainContextEngine({
workspaceDir: ctx.workspaceDir,
resolveEntities: hostResolver,
});
});
},
register,
};
export default entry;
+36
View File
@@ -34,6 +34,7 @@ import {
resetGateway,
embed,
splitByTokenBudget,
capBatchItems,
isTokenLimitError,
__setEmbedTransportForTests,
__getShrinkStateForTests,
@@ -151,6 +152,41 @@ describe('splitByTokenBudget (pure helper)', () => {
});
});
describe('capBatchItems (hard COUNT cap helper)', () => {
test('batch at or under the cap is returned as a single batch (no copy of contents)', () => {
const texts = ['a', 'b', 'c'];
expect(capBatchItems(texts, 3)).toEqual([texts]);
expect(capBatchItems(texts, 10)).toEqual([texts]);
});
test('oversized batch splits into chunks of at most maxItems', () => {
const texts = Array.from({ length: 100 }, (_, i) => `t${i}`);
const result = capBatchItems(texts, 32);
expect(result.map(b => b.length)).toEqual([32, 32, 32, 4]);
expect(result.every(b => b.length <= 32)).toBe(true);
});
test('exact multiple splits evenly with no trailing empty batch', () => {
const texts = Array.from({ length: 64 }, (_, i) => `t${i}`);
expect(capBatchItems(texts, 32).map(b => b.length)).toEqual([32, 32]);
});
test('order is preserved across the split (concatenation round-trips)', () => {
const texts = Array.from({ length: 70 }, (_, i) => `t${i}`);
expect(capBatchItems(texts, 32).flat()).toEqual(texts);
});
test('maxItems <= 0 is a no-op (single batch) — never produces empty/infinite batches', () => {
const texts = ['a', 'b', 'c'];
expect(capBatchItems(texts, 0)).toEqual([texts]);
expect(capBatchItems(texts, -5)).toEqual([texts]);
});
test('empty input returns a single empty batch', () => {
expect(capBatchItems([], 32)).toEqual([[]]);
});
});
describe('isTokenLimitError (pure helper)', () => {
test('matches Voyage error format', () => {
expect(isTokenLimitError(VOYAGE_TOKEN_LIMIT_ERROR)).toBe(true);
+7 -1
View File
@@ -29,7 +29,7 @@
* the bug made you believe was sufficient.
*/
import { describe, test, expect, beforeEach } from 'bun:test';
import { describe, test, expect, beforeEach, afterAll } from 'bun:test';
import {
chat,
configureGateway,
@@ -37,6 +37,12 @@ import {
__setGenerateTextTransportForTests,
} from '../../src/core/ai/gateway.ts';
// R5 shard hygiene: leave no configured gateway past the file boundary.
afterAll(() => {
resetGateway();
__setGenerateTextTransportForTests(null);
});
describe('gbrain#2490 — Anthropic cache breakpoint placement', () => {
beforeEach(() => {
resetGateway();
+7 -1
View File
@@ -16,7 +16,7 @@
* `generateText` import via Bun's module-replace pattern.
*/
import { describe, test, expect, beforeEach, mock } from 'bun:test';
import { describe, test, expect, beforeEach, mock, afterAll } from 'bun:test';
import {
configureGateway,
resetGateway,
@@ -30,6 +30,12 @@ import { parseModelId, resolveRecipe, assertTouchpoint } from '../../src/core/ai
import { AIConfigError } from '../../src/core/ai/errors.ts';
import { listRecipes, getRecipe } from '../../src/core/ai/recipes/index.ts';
// R5 shard hygiene: leave no configured gateway past the file boundary.
afterAll(() => {
resetGateway();
__setGenerateTextTransportForTests(null);
});
describe('chat touchpoint — recipe registry', () => {
test('all six chat-capable providers ship a chat touchpoint with supports_subagent_loop', () => {
const expected = ['anthropic', 'openai', 'google', 'deepseek', 'groq', 'together'];
@@ -16,7 +16,7 @@
* nothing), and config `provider_chat_options` overrides the derived key
*/
import { describe, test, expect, beforeEach } from 'bun:test';
import { describe, test, expect, beforeEach, afterAll } from 'bun:test';
import {
chat,
configureGateway,
@@ -25,6 +25,12 @@ import {
__setGenerateTextTransportForTests,
} from '../../src/core/ai/gateway.ts';
// R5 shard hygiene: leave no configured gateway past the file boundary.
afterAll(() => {
resetGateway();
__setGenerateTextTransportForTests(null);
});
describe('openAIPromptCacheKey — derivation', () => {
test('same system + same tools → identical stable key (sticky routing)', () => {
const a = openAIPromptCacheKey({ system: 'SYS', toolNames: ['search', 'put_page'] });
@@ -28,8 +28,8 @@ describe('v0.32 #779: no_batch_cap suppresses the missing-max_batch_tokens warni
resetGateway();
});
test('Ollama, LiteLLM, llama-server all declare no_batch_cap: true', () => {
for (const id of ['ollama', 'litellm', 'llama-server']) {
test('Ollama, LiteLLM declare no_batch_cap: true', () => {
for (const id of ['ollama', 'litellm']) {
const r = getRecipe(id);
expect(r, `${id} not registered`).toBeDefined();
expect(
@@ -39,6 +39,16 @@ describe('v0.32 #779: no_batch_cap suppresses the missing-max_batch_tokens warni
}
});
test('llama-server declares a hard item-count cap (max_batch_items: 32)', () => {
// llama.cpp enforces a request-COUNT cap equal to its launch --batch-size
// (default 32); declaring max_batch_items both bounds batches AND suppresses
// the missing-max_batch_tokens warning. Replaces the prior no_batch_cap flag.
const r = getRecipe('llama-server');
expect(r, 'llama-server not registered').toBeDefined();
expect(r!.touchpoints.embedding?.max_batch_items).toBe(32);
expect(r!.touchpoints.embedding?.no_batch_cap).toBeUndefined();
});
test('configureGateway does NOT warn for ollama/litellm/llama-server', () => {
warnSpy.mockClear();
resetGateway();
+41
View File
@@ -0,0 +1,41 @@
/**
* Ollama Matryoshka dims passthrough.
*
* Several embedding models served via Ollama (Qwen3-Embedding family) support
* Matryoshka truncation through the `dimensions` field on /v1/embeddings.
* Without this passthrough, gbrain ignores user-selected reduced dims and the
* provider returns its native size, causing dim-mismatch failures against
* brains configured for smaller widths.
*/
import { describe, expect, test } from 'bun:test';
import { dimsProviderOptions } from '../../src/core/ai/dims.ts';
describe('dims: ollama Matryoshka models', () => {
test('qwen3-embedding:4b threads dimensions=1536', () => {
expect(dimsProviderOptions('openai-compatible', 'qwen3-embedding:4b', 1536))
.toEqual({ openaiCompatible: { dimensions: 1536 } });
});
test('qwen3-embedding:0.6b threads dimensions=512', () => {
expect(dimsProviderOptions('openai-compatible', 'qwen3-embedding:0.6b', 512))
.toEqual({ openaiCompatible: { dimensions: 512 } });
});
test('qwen3-embedding:8b threads dimensions=2048', () => {
expect(dimsProviderOptions('openai-compatible', 'qwen3-embedding:8b', 2048))
.toEqual({ openaiCompatible: { dimensions: 2048 } });
});
test('bare qwen3-embedding (no quant tag) also recognized', () => {
expect(dimsProviderOptions('openai-compatible', 'qwen3-embedding', 1024))
.toEqual({ openaiCompatible: { dimensions: 1024 } });
});
test('unrelated openai-compat model returns undefined (regression guard)', () => {
expect(dimsProviderOptions('openai-compatible', 'nomic-embed-text', 768))
.toBeUndefined();
expect(dimsProviderOptions('openai-compatible', 'mxbai-embed-large', 1024))
.toBeUndefined();
});
});
+35
View File
@@ -167,6 +167,41 @@ describe('buildPlan — diff against completed + installed VERSION', () => {
});
});
describe('force-retry escape hatch', () => {
test("complete then retry-latest → pending and buildPlan lists the version as pending", () => {
const idx = indexCompleted([
{ version: '0.11.0', status: 'complete' },
{ version: '0.11.0', status: 'retry' },
]);
expect(statusForVersion('0.11.0', idx)).toBe('pending');
const plan = buildPlan(idx, '0.11.1', '0.11.0');
expect(plan.pending.map(m => m.version)).toEqual(['0.11.0']);
expect(plan.applied).toEqual([]);
expect(plan.partial).toEqual([]);
expect(plan.wedged).toEqual([]);
});
test('complete then stray partial without retry → still complete', () => {
const idx = indexCompleted([
{ version: '0.11.0', status: 'complete' },
{ version: '0.11.0', status: 'partial' },
]);
expect(statusForVersion('0.11.0', idx)).toBe('complete');
});
test('retry followed by a newer complete → complete', () => {
const idx = indexCompleted([
{ version: '0.11.0', status: 'complete' },
{ version: '0.11.0', status: 'retry' },
{ version: '0.11.0', status: 'complete' },
]);
expect(statusForVersion('0.11.0', idx)).toBe('complete');
});
});
// v0.36.1.x (cherry-pick #1062): list, dry-run, and "all migrations up to
// date" paths must exit 0 so shell scripts gating on the exit code work.
// Pre-fix, these `return` statements left the CLI dispatcher's implicit
+47
View File
@@ -0,0 +1,47 @@
/**
* Structural regression for the backlinks Minion handler default.
*
* Backlinks jobs submitted with an EMPTY payload (the syncembedbacklinks
* chains enqueued after every ingestion) must run as 'check', never 'fix'.
* The pre-fix handler inverted the default (`=== 'check' ? 'check' : 'fix'`),
* so every routine post-ingestion job rewrote tracked brain pages with
* generated "Referenced in" timeline bullets contradicting the documented
* intent in src/core/cycle.ts (runPhaseBacklinks): "Maintenance cycles must
* not rewrite tracked brain pages with generated 'Referenced in' timeline
* bullets."
*
* Source-grep is the right tool here (see fix-wave-structural.test.ts): the
* handler dynamically imports runBacklinksCore and walks a real repo dir, so
* a behavioral test would require heavy mocking that hides the regression
* behind a test seam. The rule is "this specific default must stay 'check'".
*/
import { describe, test, expect } from 'bun:test';
import { readFileSync } from 'fs';
describe('backlinks Minion handler — empty payload defaults to check, not fix', () => {
const src = readFileSync('src/commands/jobs.ts', 'utf8');
// Isolate the backlinks register block so assertions can't accidentally
// match another handler's action parsing.
const blockMatch = src.match(
/worker\.register\('backlinks',[\s\S]*?runBacklinksCore\(\{[\s\S]*?\}\);/
);
test('the backlinks handler block exists', () => {
expect(blockMatch).not.toBeNull();
});
test("default action is 'check' (explicit opt-in required for 'fix')", () => {
const block = blockMatch![0];
expect(block).toMatch(
/job\.data\.action\s*===\s*'fix'\s*\?\s*'fix'\s*:\s*'check'/
);
});
test('the inverted (fix-by-default) shape stays absent', () => {
const block = blockMatch![0];
expect(block).not.toMatch(
/job\.data\.action\s*===\s*'check'\s*\?\s*'check'\s*:\s*'fix'/
);
});
});
+34
View File
@@ -6,6 +6,7 @@ import {
checkResolvable,
parseResolverEntries,
extractDelegationTargets,
extractTriggers,
} from "../src/core/check-resolvable.ts";
const SKILLS_DIR = join(import.meta.dir, "..", "skills");
@@ -195,6 +196,39 @@ describe("parseResolverEntries", () => {
});
});
describe("extractTriggers", () => {
const LF_FRONTMATTER =
"---\nname: query\ndescription: Test\ntriggers:\n - \"what do we know\"\n - \"tell me about\"\ntools:\n - search\n---\n\n# Body\n";
test("parses triggers from LF-terminated frontmatter", () => {
const triggers = extractTriggers(LF_FRONTMATTER);
expect(triggers).toEqual(["what do we know", "tell me about"]);
});
test("parses triggers from CRLF-terminated frontmatter (Windows checkouts)", () => {
// Regression: `core.autocrlf=true` is the Windows default. Without
// CRLF→LF normalization, every Windows skill is reported as a false
// mece_gap warning because the `^---\n` regex never matches `---\r\n`.
const crlf = LF_FRONTMATTER.replace(/\n/g, "\r\n");
const triggers = extractTriggers(crlf);
expect(triggers).toEqual(["what do we know", "tell me about"]);
});
test("returns [] when frontmatter is missing", () => {
expect(extractTriggers("# Just a body, no frontmatter\n")).toEqual([]);
});
test("returns [] when triggers field is absent from frontmatter", () => {
const fm = "---\nname: query\ndescription: Test\ntools:\n - search\n---\n";
expect(extractTriggers(fm)).toEqual([]);
});
test("strips surrounding quotes from trigger values", () => {
const fm = "---\nname: x\ntriggers:\n - \"double quoted\"\n - 'single quoted'\n - unquoted\n---\n";
expect(extractTriggers(fm)).toEqual(["double quoted", "single quoted", "unquoted"]);
});
});
describe("checkResolvable — real skills directory", () => {
const report = checkResolvable(SKILLS_DIR);
+39 -1
View File
@@ -8,13 +8,15 @@
//
// PGLite-only: in-memory engine, no DATABASE_URL needed.
import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { loadConfigWithEngine, type GBrainConfig } from '../src/core/config.ts';
import {
__setRerankTransportForTests,
configureGateway,
getEmbeddingModel,
getMultimodalModel,
rerank,
resetGateway,
} from '../src/core/ai/gateway.ts';
import type { AIGatewayConfig } from '../src/core/ai/types.ts';
@@ -52,10 +54,16 @@ afterAll(async () => {
beforeEach(async () => {
resetGateway();
__setRerankTransportForTests(null);
// Clear any prior config rows so tests are independent. setConfig with
// empty string is treated as undefined by loadConfigWithEngine (per
// dbStr semantics), so this is safe to call between tests.
await engine.setConfig('embedding_multimodal_model', '');
await engine.setConfig('provider_base_urls.llama-server-reranker', '');
});
afterEach(() => {
__setRerankTransportForTests(null);
});
describe('cli connectEngine — embedding_multimodal_model DB→gateway plumbing', () => {
@@ -122,4 +130,34 @@ describe('cli connectEngine — embedding_multimodal_model DB→gateway plumbing
expect(getEmbeddingModel()).toBe('openai:text-embedding-3-large');
expect(getMultimodalModel()).toBeUndefined();
});
test('DB-set provider_base_urls.llama-server-reranker flows to gateway.rerank URL', async () => {
await engine.setConfig('provider_base_urls.llama-server-reranker', 'http://127.0.0.1:8091/v1');
const baseConfig: GBrainConfig = {
engine: 'pglite',
embedding_model: 'openai:text-embedding-3-large',
embedding_dimensions: 1536,
};
const merged = await loadConfigWithEngine(engine, baseConfig);
configureGateway(buildGatewayConfig(merged!));
let capturedUrl = '';
__setRerankTransportForTests(async (url) => {
capturedUrl = url;
return new Response(JSON.stringify({ results: [{ index: 0, relevance_score: 0.9 }] }), {
status: 200,
headers: { 'content-type': 'application/json' },
});
});
await rerank({
query: 'q',
documents: ['d'],
model: 'llama-server-reranker:qwen3-reranker-4b',
});
expect(capturedUrl).toBe('http://127.0.0.1:8091/v1/rerank');
});
});
+2 -1
View File
@@ -10,7 +10,7 @@
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { configureGateway } from '../src/core/ai/gateway.ts';
import { configureGateway, resetGateway } from '../src/core/ai/gateway.ts';
import { runPhaseConsolidate } from '../src/core/cycle/phases/consolidate.ts';
let engine: PGLiteEngine;
@@ -37,6 +37,7 @@ beforeAll(async () => {
afterAll(async () => {
await engine.disconnect();
resetGateway();
});
beforeEach(async () => {
+2 -1
View File
@@ -8,7 +8,7 @@
*/
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { configureGateway } from '../src/core/ai/gateway.ts';
import { configureGateway, resetGateway } from '../src/core/ai/gateway.ts';
import { checkFederationHealth } from '../src/commands/doctor.ts';
let engine: PGLiteEngine;
@@ -33,6 +33,7 @@ beforeAll(async () => {
afterAll(async () => {
await engine.disconnect();
resetGateway();
});
beforeEach(async () => {
+4
View File
@@ -20,6 +20,10 @@ import {
const skip = !hasDatabase();
const describeE2E = skip ? describe.skip : describe;
if (skip) {
console.log('Skipping E2E doctor --progress-json tests (DATABASE_URL not set)');
}
const CLI = join(import.meta.dir, '..', '..', 'src', 'cli.ts');
describeE2E('gbrain doctor --progress-json (E2E)', () => {
+53
View File
@@ -172,6 +172,59 @@ describe('issue #972 — DB-source (gbrain extract links --source db)', () => {
expect(strk!.link_type).toBe('wikilink_basename');
});
test('flag ON → path-qualified wikilink outside DIR_PATTERN resolves via DB path', async () => {
// `[[notes/struktura]]` — `notes` is not in DIR_PATTERN, so the ref
// reaches the generic pass with its dirname intact. Regression: the DB
// path queried the basename index with the raw literal (which is keyed
// by final segments only), so path-qualified wikilinks outside
// DIR_PATTERN silently produced zero edges while the FS path resolved
// the identical content.
await engine.putPage('notes/struktura', {
type: 'concept' as any, title: 'Struktura Notes',
compiled_truth: '', timeline: '',
});
await engine.putPage('concepts/knowledge-graph', {
type: 'concept', title: 'Knowledge Graph',
compiled_truth: 'Background in [[notes/struktura]].', timeline: '',
});
await engine.setConfig('link_resolution.global_basename', 'true');
await runExtract(engine, ['links', '--source', 'db']);
const outLinks = await engine.getLinks('concepts/knowledge-graph');
const strk = outLinks.find(l => l.to_slug === 'notes/struktura');
expect(strk).toBeDefined();
expect(strk!.link_type).toBe('wikilink_basename');
expect(strk!.link_source).toBe('wikilink-resolved');
});
test('path-qualified wikilink never attaches to a basename-only sibling', async () => {
// Both notes/struktura and wiki/struktura exist. The author wrote
// `[[notes/struktura]]` — the written path must exclude wiki/struktura
// (a bare `[[struktura]]` would legitimately match both).
await engine.putPage('notes/struktura', {
type: 'concept' as any, title: 'Struktura Notes',
compiled_truth: '', timeline: '',
});
await engine.putPage('wiki/struktura', {
type: 'concept' as any, title: 'Struktura Wiki',
compiled_truth: '', timeline: '',
});
await engine.putPage('concepts/x', {
type: 'concept', title: 'X',
compiled_truth: 'See [[notes/struktura]].', timeline: '',
});
await engine.setConfig('link_resolution.global_basename', 'true');
await runExtract(engine, ['links', '--source', 'db']);
const outLinks = await engine.getLinks('concepts/x');
const basenameLinks = outLinks
.filter(l => l.link_type === 'wikilink_basename')
.map(l => l.to_slug);
expect(basenameLinks).toEqual(['notes/struktura']);
});
test('flag OFF → no basename edges via DB path (back-compat)', async () => {
await engine.putPage('projects/struktura', {
type: 'project', title: 'Struktura',
+71 -5
View File
@@ -29,9 +29,15 @@ afterAll(async () => {
});
async function truncateAll() {
for (const t of ['content_chunks', 'links', 'tags', 'raw_data', 'timeline_entries', 'page_versions', 'ingest_log', 'pages']) {
for (const t of ['content_chunks', 'links', 'tags', 'raw_data', 'timeline_entries', 'page_versions', 'ingest_log', 'config', 'pages']) {
await (engine as any).db.exec(`DELETE FROM ${t}`);
}
// Re-seed the two config keys this file touches back to their documented
// defaults (both default to ON). This makes every test deterministic even if
// an earlier test threw before its finally restored auto_link/auto_timeline,
// and even though absent-key already resolves truthy via isAuto*Enabled.
await engine.setConfig('auto_link', 'true');
await engine.setConfig('auto_timeline', 'true');
}
function makeContext(): OperationContext {
@@ -77,10 +83,12 @@ describe('E2E graph quality (v0.10.1 pipeline)', () => {
await runExtract(engine, ['links', '--source', 'db']);
await runExtract(engine, ['timeline', '--source', 'db']);
// Verify graph populated.
// Verify graph populated. Concrete floors derived from the seeded fixtures:
// resolvable entity refs: alice->acme, bob->acme, standup->alice, standup->bob = 4
// timeline lines: alice(2) + bob(1) + acme(1) + standup(1) = 5
const stats = await engine.getStats();
expect(stats.link_count).toBeGreaterThan(0);
expect(stats.timeline_entry_count).toBeGreaterThan(0);
expect(stats.link_count).toBeGreaterThanOrEqual(4);
expect(stats.timeline_entry_count).toBeGreaterThanOrEqual(5);
// Verify typed link inference.
const aliceLinks = await engine.getLinks('people/alice');
@@ -91,7 +99,16 @@ describe('E2E graph quality (v0.10.1 pipeline)', () => {
const bobAcme = bobLinks.find(l => l.to_slug === 'companies/acme');
expect(bobAcme?.link_type).toBe('invested_in');
// The standup meeting references both Alice and Bob as attendees. Assert the
// exact attendee edges are present and typed 'attended' (a plain .every()
// would silently pass if a meeting->company edge were misclassified or if the
// attendee edges were missing entirely).
const meetingLinks = await engine.getLinks('meetings/standup');
const attended = new Set(
meetingLinks.filter(l => l.link_type === 'attended').map(l => l.to_slug),
);
expect(attended.has('people/alice')).toBe(true);
expect(attended.has('people/bob')).toBe(true);
expect(meetingLinks.every(l => l.link_type === 'attended')).toBe(true);
});
@@ -118,7 +135,9 @@ Attendees: [Alice](people/alice). Discussed [Acme](companies/acme).
// The response should include auto_links results.
expect((result as any).auto_links).toBeDefined();
const autoLinks = (result as any).auto_links;
expect(autoLinks.created).toBeGreaterThan(0);
// The page references exactly two seeded, resolvable targets (Alice + Acme),
// so exactly two links are created.
expect(autoLinks.created).toBe(2);
expect(autoLinks.errors).toBe(0);
// Verify links actually exist in DB.
@@ -283,6 +302,53 @@ Mention of [Alice](people/alice).
expect(paths[0].link_type).toBe('works_at');
});
test('graph-query traversal: direction out and both, plus depth:2 multi-hop', async () => {
// Seed a 2-hop chain: alice -works_at-> acme -partnered_with-> beta.
await engine.putPage('people/alice', { type: 'person', title: 'Alice', compiled_truth: '', timeline: '' });
await engine.putPage('companies/acme', { type: 'company', title: 'Acme', compiled_truth: '', timeline: '' });
await engine.putPage('companies/beta', { type: 'company', title: 'Beta', compiled_truth: '', timeline: '' });
await engine.addLink('people/alice', 'companies/acme', '', 'works_at');
await engine.addLink('companies/acme', 'companies/beta', '', 'partnered_with');
// direction:'out' from alice, depth 1 -> only the first hop.
const out1 = await engine.traversePaths('people/alice', { direction: 'out', depth: 1 });
expect(out1.length).toBe(1);
expect(out1[0].from_slug).toBe('people/alice');
expect(out1[0].to_slug).toBe('companies/acme');
expect(out1[0].depth).toBe(1);
// depth:2 -> both hops, depths 1 and 2.
const out2 = await engine.traversePaths('people/alice', { direction: 'out', depth: 2 });
const out2Edges = new Set(out2.map(p => `${p.from_slug}->${p.to_slug}@${p.depth}`));
expect(out2Edges.has('people/alice->companies/acme@1')).toBe(true);
expect(out2Edges.has('companies/acme->companies/beta@2')).toBe(true);
expect(out2.length).toBe(2);
// direction:'both' from acme depth 1 -> sees the inbound edge from alice AND
// the outbound edge to beta. Edges keep their natural from->to orientation.
const both = await engine.traversePaths('companies/acme', { direction: 'both', depth: 1 });
const bothEdges = new Set(both.map(p => `${p.from_slug}->${p.to_slug}`));
expect(bothEdges.has('people/alice->companies/acme')).toBe(true);
expect(bothEdges.has('companies/acme->companies/beta')).toBe(true);
});
test('graph-query cycle safety: A->B->A terminates and returns bounded results', async () => {
await engine.putPage('people/alice', { type: 'person', title: 'Alice', compiled_truth: '', timeline: '' });
await engine.putPage('people/bob', { type: 'person', title: 'Bob', compiled_truth: '', timeline: '' });
// Create a 2-cycle: alice -> bob -> alice.
await engine.addLink('people/alice', 'people/bob', '', 'knows');
await engine.addLink('people/bob', 'people/alice', '', 'knows');
// High depth must NOT loop forever; the visited-set guard bounds the walk.
const paths = await engine.traversePaths('people/alice', { direction: 'out', depth: 100 });
const edges = new Set(paths.map(p => `${p.from_slug}->${p.to_slug}`));
// Both edges of the cycle are reachable exactly once.
expect(edges.has('people/alice->people/bob')).toBe(true);
expect(edges.has('people/bob->people/alice')).toBe(true);
// Bounded: there are only two edges in the graph, so no path explosion.
expect(paths.length).toBe(2);
});
test('search backlink boost: well-connected pages rank higher', async () => {
// Create 3 pages all matching a search term, but with different inbound link counts.
await engine.putPage('topic/popular', {
+4
View File
@@ -21,6 +21,10 @@ import { hasDatabase, setupDB, teardownDB, getEngine, getConn } from './helpers.
const skip = !hasDatabase();
const describeE2E = skip ? describe.skip : describe;
if (skip) {
console.log('Skipping E2E JSONB roundtrip tests (DATABASE_URL not set)');
}
describeE2E('E2E: JSONB roundtrip — v0.12.1 reliability wave', () => {
beforeAll(async () => { await setupDB(); });
afterAll(async () => { await teardownDB(); });
+1
View File
@@ -56,6 +56,7 @@ describe('E2E: MCP Tool Generation', () => {
expect(names).toContain('get_health');
expect(names).toContain('sync_brain');
expect(names).toContain('file_upload');
expect(names).toContain('find_orphans');
});
test('MCP server module can be imported', async () => {
+64 -7
View File
@@ -175,6 +175,15 @@ describeE2E('E2E: Search', () => {
for (const [query, score] of Object.entries(scores)) {
console.log(` "${query}": ${(score * 100).toFixed(0)}%`);
}
// Guard value: every known-item query must surface at least one ground-truth
// doc in the top 5. This is a deliberately loose floor (not a tuned P@5
// threshold) — it catches a total keyword-retrieval regression without
// breaking on every scoring/fixture tweak. Without it this test asserted
// nothing and a 0%-precision result passed silently.
for (const [query, score] of Object.entries(scores)) {
expect(score).toBeGreaterThan(0);
}
});
});
@@ -205,10 +214,22 @@ describeE2E('E2E: Links', () => {
}, 30_000);
test('traverse_graph finds connected pages', async () => {
// Links should already be added from prior test in this describe block
const graph = await callOp('traverse_graph', { slug: 'people/sarah-chen', depth: 2 }) as any;
// Self-contained: do not depend on a prior test's add_link. add_link is
// idempotent (ON CONFLICT DO NOTHING), so re-adding here is safe whether or
// not the round-trip test ran first, and the test no longer false-passes or
// false-fails based on describe-block ordering.
await callOp('add_link', {
from: 'people/sarah-chen',
to: 'companies/novamind',
link_type: 'founded',
});
const graph = await callOp('traverse_graph', { slug: 'people/sarah-chen', depth: 2 }) as any[];
expect(Array.isArray(graph)).toBe(true);
expect(graph.length).toBeGreaterThanOrEqual(1);
// Content assertion, not just shape: the linked company must be reachable.
const reachable = graph.map((n: any) => n.slug ?? n.to_slug ?? n.to_page_slug);
expect(reachable).toContain('companies/novamind');
});
test('remove_link removes the link', async () => {
@@ -469,8 +490,14 @@ describeE2E('E2E: Admin', () => {
test('get_health returns valid structure', async () => {
const health = await callOp('get_health') as any;
expect(health).toBeDefined();
expect(typeof health.page_count).toBe('number');
expect(typeof health.embed_coverage).toBe('number');
// Value bounds, not just types: page_count must match the fixture inventory
// and embed_coverage is a 0..1 fraction (src/commands/doctor.ts multiplies
// by 100 and compares to 0.9). Type-only checks let embed_coverage: -9999
// through; these catch a genuinely broken health payload.
expect(health.page_count).toBe(16);
expect(Number.isFinite(health.embed_coverage)).toBe(true);
expect(health.embed_coverage).toBeGreaterThanOrEqual(0);
expect(health.embed_coverage).toBeLessThanOrEqual(1);
});
});
@@ -488,7 +515,17 @@ describeE2E('E2E: Chunks & Resolution', () => {
test('get_chunks returns chunks for imported page', async () => {
const chunks = await callOp('get_chunks', { slug: 'people/sarah-chen' }) as any[];
expect(chunks.length).toBeGreaterThan(0);
expect(chunks[0].chunk_text).toBeTruthy();
// Content + ordering, not just truthiness (a whitespace-only chunk is truthy):
// every chunk has real text and a numeric index, the indexes are
// non-decreasing in return order, and the page's own name appears somewhere.
for (const c of chunks) {
expect(typeof c.chunk_text).toBe('string');
expect(c.chunk_text.trim().length).toBeGreaterThan(0);
expect(typeof c.chunk_index).toBe('number');
}
const indexes = chunks.map((c: any) => c.chunk_index);
expect(indexes).toEqual([...indexes].sort((x, y) => x - y));
expect(chunks.some((c: any) => c.chunk_text.includes('Sarah'))).toBe(true);
}, 30_000);
test('resolve_slugs finds partial match', async () => {
@@ -662,9 +699,29 @@ describeE2E('E2E: file_list LIMIT enforcement', () => {
}, 30_000);
test('file_list without slug also respects LIMIT 100', async () => {
// The 150 rows from the previous test are still in the DB
// Self-sufficient: seed our own >100 rows rather than relying on the
// previous test's 150 rows surviving in the DB. A bun reorder, a focused
// `-t` run, or a failure mid-insert in the prior test would otherwise leave
// this asserting against an indeterminate row count.
const sql = getConn();
const seedSlug = 'test-limit-noslug';
await sql`
INSERT INTO pages (slug, title, type, compiled_truth, frontmatter)
VALUES (${seedSlug}, ${'Test Limit NoSlug'}, ${'note'}, ${'body'}, ${'{}'}::jsonb)
ON CONFLICT (source_id, slug) DO NOTHING
`;
for (let i = 0; i < 120; i++) {
await sql`
INSERT INTO files (page_slug, filename, storage_path, mime_type, size_bytes, content_hash, metadata)
VALUES (${seedSlug}, ${'nf-' + String(i).padStart(3, '0') + '.txt'}, ${seedSlug + '/nf-' + i + '.txt'}, ${'text/plain'}, ${100}, ${'nhash-' + i}, ${'{}'}::jsonb)
ON CONFLICT (storage_path) DO NOTHING
`;
}
const total = await sql`SELECT count(*)::int AS n FROM files`;
expect(Number(total[0].n)).toBeGreaterThan(100); // cap is actually exercised
const files = await callOp('file_list', {}) as any[];
expect(files.length).toBeLessThanOrEqual(100);
expect(files.length).toBe(100);
});
});
+149 -111
View File
@@ -78,6 +78,19 @@ function freshTempHome(label: string) {
return dir;
}
// Restore HOME/PATH to the captured originals. Called from each test's
// `finally` so a throw mid-test can never leave HOME/PATH pointed at a temp
// dir for the rest of the bun process (which would silently break unrelated
// suites that read HOME). PATH keeps the shim prepended because the
// module-level shim install is what subsequent tests in this suite rely on;
// afterAll does the final teardown to the pristine origPath.
function restoreHomePath() {
if (origHome === undefined) delete process.env.HOME;
else process.env.HOME = origHome;
if (origPath === undefined) delete process.env.PATH;
else process.env.PATH = `${fakeBinDir}:${origPath ?? ''}`;
}
beforeAll(() => {
if (SKIP) {
console.log('[migration-flow.e2e] DATABASE_URL not set — skipping.');
@@ -100,6 +113,15 @@ afterAll(() => {
beforeEach(() => {
if (SKIP) return;
// Robust restore: if a prior test threw before its own finally ran (or
// before afterAll), HOME/PATH could still point at a dead temp dir. Reset
// them to the captured originals at the start of every test so a throw in
// one test can never leak a temp HOME/PATH into sibling suites that read
// them. freshTempHome() re-points HOME per test immediately after this.
if (origHome === undefined) delete process.env.HOME;
else process.env.HOME = origHome;
if (origPath === undefined) delete process.env.PATH;
else process.env.PATH = `${fakeBinDir}:${origPath ?? ''}`;
try { if (tmp) rmSync(tmp, { recursive: true, force: true }); } catch { /* best-effort */ }
});
@@ -114,144 +136,160 @@ const COMMON_OPTS = {
describeE2E('E2E: v0.11.0 orchestrator against live Postgres', () => {
test('fresh install flow: schema → smoke → prefs → host-rewrite → completed', async () => {
tmp = freshTempHome('fresh');
const result = await v0_11_0.orchestrator(COMMON_OPTS);
try {
const result = await v0_11_0.orchestrator(COMMON_OPTS);
// Orchestrator returns a structured result (status is `complete` when
// no pending-host-work TODOs fired, `partial` otherwise).
expect(result.version).toBe('0.11.0');
expect(['complete', 'partial']).toContain(result.status);
// Orchestrator returns a structured result (status is `complete` when
// no pending-host-work TODOs fired, `partial` otherwise).
expect(result.version).toBe('0.11.0');
expect(['complete', 'partial']).toContain(result.status);
// Phase D: preferences.json exists with 0o600 + mode=pain_triggered.
const prefsPath = join(tmp, '.gbrain', 'preferences.json');
expect(existsSync(prefsPath)).toBe(true);
expect(statSync(prefsPath).mode & 0o777).toBe(0o600);
const prefs = loadPreferences();
expect(prefs.minion_mode).toBe('pain_triggered');
expect(prefs.set_at).toBeTruthy();
expect(prefs.set_in_version).toBeTruthy();
// Phase D: preferences.json exists with 0o600 + mode=pain_triggered.
const prefsPath = join(tmp, '.gbrain', 'preferences.json');
expect(existsSync(prefsPath)).toBe(true);
expect(statSync(prefsPath).mode & 0o777).toBe(0o600);
const prefs = loadPreferences();
expect(prefs.minion_mode).toBe('pain_triggered');
expect(prefs.set_at).toBeTruthy();
expect(prefs.set_in_version).toBeTruthy();
// Bug 3 (v0.14.2) — orchestrator no longer writes completed.jsonl.
// The runner (apply-migrations.ts) persists the result after the
// orchestrator returns. A direct orchestrator call in E2E leaves the
// ledger empty; the runner path is tested separately in
// test/apply-migrations.test.ts + test/migration-resume.test.ts.
const completed = loadCompletedMigrations();
const v0110Entries = completed.filter(e => e.version === '0.11.0');
expect(v0110Entries.length).toBe(0);
// Bug 3 (v0.14.2) — orchestrator no longer writes completed.jsonl.
// The runner (apply-migrations.ts) persists the result after the
// orchestrator returns. A direct orchestrator call in E2E leaves the
// ledger empty; the runner path is tested separately in
// test/apply-migrations.test.ts + test/migration-resume.test.ts.
const completed = loadCompletedMigrations();
const v0110Entries = completed.filter(e => e.version === '0.11.0');
expect(v0110Entries.length).toBe(0);
// Phase F is skipped per COMMON_OPTS — autopilot should NOT have been
// installed on this host.
expect(result.autopilot_installed).toBe(false);
// Phase F is skipped per COMMON_OPTS — autopilot should NOT have been
// installed on this host.
expect(result.autopilot_installed).toBe(false);
} finally {
restoreHomePath();
}
}, 60_000);
test('idempotent rerun: second invocation is a safe no-op', async () => {
tmp = freshTempHome('rerun');
const first = await v0_11_0.orchestrator(COMMON_OPTS);
expect(['complete', 'partial']).toContain(first.status);
try {
const first = await v0_11_0.orchestrator(COMMON_OPTS);
expect(['complete', 'partial']).toContain(first.status);
const second = await v0_11_0.orchestrator(COMMON_OPTS);
expect(['complete', 'partial']).toContain(second.status);
const second = await v0_11_0.orchestrator(COMMON_OPTS);
expect(['complete', 'partial']).toContain(second.status);
// Bug 3 (v0.14.2) — orchestrator does not write completed.jsonl, so
// repeated direct invocations don't accumulate ledger entries. Assert
// the preferences state stays stable (the real idempotency signal for
// this orchestrator is "running again doesn't corrupt preferences").
expect(loadPreferences().minion_mode).toBe('pain_triggered');
const completed = loadCompletedMigrations();
expect(completed.filter(e => e.version === '0.11.0').length).toBe(0);
// Bug 3 (v0.14.2) — orchestrator does not write completed.jsonl, so
// repeated direct invocations don't accumulate ledger entries. Assert
// the preferences state stays stable (the real idempotency signal for
// this orchestrator is "running again doesn't corrupt preferences").
expect(loadPreferences().minion_mode).toBe('pain_triggered');
const completed = loadCompletedMigrations();
expect(completed.filter(e => e.version === '0.11.0').length).toBe(0);
} finally {
restoreHomePath();
}
}, 90_000);
test('host rewrite: builtin handlers auto-rewritten, non-builtins queued as JSONL TODOs', async () => {
tmp = freshTempHome('host-rewrite');
// Fixture: AGENTS.md + cron/jobs.json with a mix of gbrain-builtin and
// non-builtin handlers.
const claudeDir = join(tmp, '.claude');
mkdirSync(claudeDir, { recursive: true });
writeFileSync(
join(claudeDir, 'AGENTS.md'),
'# Test AGENTS.md\n\nSome existing content referencing sessions_spawn routing.\n',
);
mkdirSync(join(claudeDir, 'cron'), { recursive: true });
writeFileSync(
join(claudeDir, 'cron', 'jobs.json'),
JSON.stringify({
jobs: [
{ schedule: '*/5 * * * *', kind: 'agentTurn', skill: 'sync' }, // builtin
{ schedule: '0 */30 * * *', kind: 'agentTurn', skill: 'ea-inbox-sweep' }, // non-builtin
{ schedule: '*/10 * * * *', kind: 'agentTurn', skill: 'embed' }, // builtin
{ schedule: '0 8 * * *', kind: 'agentTurn', skill: 'morning-briefing' }, // non-builtin
],
}, null, 2) + '\n',
);
try {
// Fixture: AGENTS.md + cron/jobs.json with a mix of gbrain-builtin and
// non-builtin handlers.
const claudeDir = join(tmp, '.claude');
mkdirSync(claudeDir, { recursive: true });
writeFileSync(
join(claudeDir, 'AGENTS.md'),
'# Test AGENTS.md\n\nSome existing content referencing sessions_spawn routing.\n',
);
mkdirSync(join(claudeDir, 'cron'), { recursive: true });
writeFileSync(
join(claudeDir, 'cron', 'jobs.json'),
JSON.stringify({
jobs: [
{ schedule: '*/5 * * * *', kind: 'agentTurn', skill: 'sync' }, // builtin
{ schedule: '0 */30 * * *', kind: 'agentTurn', skill: 'ea-inbox-sweep' }, // non-builtin
{ schedule: '*/10 * * * *', kind: 'agentTurn', skill: 'embed' }, // builtin
{ schedule: '0 8 * * *', kind: 'agentTurn', skill: 'morning-briefing' }, // non-builtin
],
}, null, 2) + '\n',
);
const result = await v0_11_0.orchestrator(COMMON_OPTS);
const result = await v0_11_0.orchestrator(COMMON_OPTS);
// Builtins rewritten in place; non-builtins left alone.
const cronAfter = JSON.parse(readFileSync(join(claudeDir, 'cron', 'jobs.json'), 'utf-8'));
expect(cronAfter.jobs[0].kind).toBe('shell'); // sync (builtin)
expect(cronAfter.jobs[0].cmd).toContain('gbrain jobs submit sync');
expect(cronAfter.jobs[1].kind).toBe('agentTurn'); // ea-inbox-sweep (non-builtin)
expect(cronAfter.jobs[2].kind).toBe('shell'); // embed (builtin)
expect(cronAfter.jobs[3].kind).toBe('agentTurn'); // morning-briefing (non-builtin)
// Builtins rewritten in place; non-builtins left alone.
const cronAfter = JSON.parse(readFileSync(join(claudeDir, 'cron', 'jobs.json'), 'utf-8'));
expect(cronAfter.jobs[0].kind).toBe('shell'); // sync (builtin)
expect(cronAfter.jobs[0].cmd).toContain('gbrain jobs submit sync');
expect(cronAfter.jobs[1].kind).toBe('agentTurn'); // ea-inbox-sweep (non-builtin)
expect(cronAfter.jobs[2].kind).toBe('shell'); // embed (builtin)
expect(cronAfter.jobs[3].kind).toBe('agentTurn'); // morning-briefing (non-builtin)
// files_rewritten counts the 2 builtin rewrites.
expect(result.files_rewritten).toBeGreaterThanOrEqual(2);
// files_rewritten counts the 2 builtin rewrites.
expect(result.files_rewritten).toBeGreaterThanOrEqual(2);
// pending_host_work counts the 2 non-builtin TODOs.
expect(result.pending_host_work).toBe(2);
// pending_host_work counts the 2 non-builtin TODOs.
expect(result.pending_host_work).toBe(2);
// Status is "partial" because non-builtin TODOs remain.
expect(result.status).toBe('partial');
// Status is "partial" because non-builtin TODOs remain.
expect(result.status).toBe('partial');
// AGENTS.md got the marker injected.
const agentsMdAfter = readFileSync(join(claudeDir, 'AGENTS.md'), 'utf-8');
expect(agentsMdAfter).toContain('gbrain:subagent-routing v0.11.0');
expect(agentsMdAfter).toContain('skills/conventions/subagent-routing.md');
// AGENTS.md got the marker injected.
const agentsMdAfter = readFileSync(join(claudeDir, 'AGENTS.md'), 'utf-8');
expect(agentsMdAfter).toContain('gbrain:subagent-routing v0.11.0');
expect(agentsMdAfter).toContain('skills/conventions/subagent-routing.md');
// JSONL TODO file written under ~/.gbrain/migrations/.
const jsonlPath = join(tmp, '.gbrain', 'migrations', 'pending-host-work.jsonl');
expect(existsSync(jsonlPath)).toBe(true);
const lines = readFileSync(jsonlPath, 'utf-8').split('\n').filter(l => l.trim());
expect(lines.length).toBe(2);
const todos = lines.map(l => JSON.parse(l));
const handlers = todos.map(t => t.handler).sort();
expect(handlers).toEqual(['ea-inbox-sweep', 'morning-briefing']);
for (const todo of todos) {
expect(todo.type).toBe('cron-handler-needs-host-registration');
expect(todo.status).toBe('pending');
expect(todo.manifest_path).toContain('cron/jobs.json');
// JSONL TODO file written under ~/.gbrain/migrations/.
const jsonlPath = join(tmp, '.gbrain', 'migrations', 'pending-host-work.jsonl');
expect(existsSync(jsonlPath)).toBe(true);
const lines = readFileSync(jsonlPath, 'utf-8').split('\n').filter(l => l.trim());
expect(lines.length).toBe(2);
const todos = lines.map(l => JSON.parse(l));
const handlers = todos.map(t => t.handler).sort();
expect(handlers).toEqual(['ea-inbox-sweep', 'morning-briefing']);
for (const todo of todos) {
expect(todo.type).toBe('cron-handler-needs-host-registration');
expect(todo.status).toBe('pending');
expect(todo.manifest_path).toContain('cron/jobs.json');
}
} finally {
restoreHomePath();
}
}, 90_000);
test('resumable: partial run → orchestrator re-run → complete', async () => {
tmp = freshTempHome('resumable');
// Simulate a stopgap-written partial entry BEFORE running the orchestrator.
mkdirSync(join(tmp, '.gbrain', 'migrations'), { recursive: true });
writeFileSync(
join(tmp, '.gbrain', 'migrations', 'completed.jsonl'),
JSON.stringify({
version: '0.11.0',
status: 'partial',
apply_migrations_pending: true,
mode: 'pain_triggered',
source: 'fix-v0.11.0.sh',
ts: new Date().toISOString(),
}) + '\n',
);
try {
// Simulate a stopgap-written partial entry BEFORE running the orchestrator.
mkdirSync(join(tmp, '.gbrain', 'migrations'), { recursive: true });
writeFileSync(
join(tmp, '.gbrain', 'migrations', 'completed.jsonl'),
JSON.stringify({
version: '0.11.0',
status: 'partial',
apply_migrations_pending: true,
mode: 'pain_triggered',
source: 'fix-v0.11.0.sh',
ts: new Date().toISOString(),
}) + '\n',
);
// Orchestrator re-running on a partial → should succeed (schema apply
// and smoke are idempotent; prefs are preserved from the partial
// record; host-rewrite runs its safe-skip pass). Per Bug 3 (v0.14.2),
// the orchestrator itself doesn't append to completed.jsonl — the
// runner does. The stopgap's partial entry stays unchanged here.
const result = await v0_11_0.orchestrator(COMMON_OPTS);
expect(['complete', 'partial']).toContain(result.status);
// Orchestrator re-running on a partial → should succeed (schema apply
// and smoke are idempotent; prefs are preserved from the partial
// record; host-rewrite runs its safe-skip pass). Per Bug 3 (v0.14.2),
// the orchestrator itself doesn't append to completed.jsonl — the
// runner does. The stopgap's partial entry stays unchanged here.
const result = await v0_11_0.orchestrator(COMMON_OPTS);
expect(['complete', 'partial']).toContain(result.status);
const completed = loadCompletedMigrations();
const v0110 = completed.filter(e => e.version === '0.11.0');
// Just the stopgap partial — orchestrator doesn't add its own entry.
expect(v0110.length).toBe(1);
expect(v0110[0].status).toBe('partial');
expect(v0110[0].source).toBe('fix-v0.11.0.sh');
const completed = loadCompletedMigrations();
const v0110 = completed.filter(e => e.version === '0.11.0');
// Just the stopgap partial — orchestrator doesn't add its own entry.
expect(v0110.length).toBe(1);
expect(v0110[0].status).toBe('partial');
expect(v0110[0].source).toBe('fix-v0.11.0.sh');
} finally {
restoreHomePath();
}
}, 90_000);
});
+14 -4
View File
@@ -94,7 +94,7 @@ describeE2E('E2E: Minions resilience (OpenClaw real-world patterns)', () => {
}, 30_000);
// --- 2. Runaway handler: ignores AbortSignal, dead-lettered by handleTimeouts ---
test('runaway handler: ignores AbortSignal, handleTimeouts dead-letters in <2s', async () => {
test('runaway handler: ignores AbortSignal, handleTimeouts dead-letters it', async () => {
const { a, b } = await makeEngines();
try {
const queue = new MinionQueue(a);
@@ -133,8 +133,14 @@ describeE2E('E2E: Minions resilience (OpenClaw real-world patterns)', () => {
worker.stop();
await startP;
// Correctness gate: the job MUST be dead-lettered with the timeout reason.
// We intentionally do NOT assert a wall-clock upper bound (deadAt - started):
// on a loaded CI runner the stall/timeout sweep cadence varies, and the only
// thing that matters is that the runaway job terminates as 'dead'. The 3s poll
// deadline above is the real timeout — if the sweep is too slow, finalStatus
// stays '' and this toBe('dead') fails loudly.
expect(finalStatus).toBe('dead');
expect(deadAt - started).toBeLessThan(2000);
void deadAt; // retained for debugging; no timing assertion (flake-prone)
const final = await queue.getJob(job.id);
expect(final?.error_text).toMatch(/timeout exceeded/i);
@@ -304,7 +310,7 @@ describeE2E('E2E: Minions resilience (OpenClaw real-world patterns)', () => {
}, 60_000);
// --- 5. Cascade kill under load: cancelJob aborts all live descendants ---
test('cascade kill: cancelJob on parent aborts 10 live children within 2s', async () => {
test('cascade kill: cancelJob on parent aborts 10 live children', async () => {
const { a, b } = await makeEngines();
try {
const queue = new MinionQueue(a);
@@ -374,8 +380,12 @@ describeE2E('E2E: Minions resilience (OpenClaw real-world patterns)', () => {
worker.stop();
await startP;
// Correctness gate: all 10 cooperative handlers observed the abort and the
// DB shows every descendant + root cancelled. We do NOT assert a wall-clock
// upper bound on cancelElapsed — the 3s abort poll deadline above already
// bounds the wait, and asserting a tighter time flakes on shared runners.
expect(abortedChildren.size).toBe(10);
expect(cancelElapsed).toBeLessThan(3000);
void cancelElapsed; // retained for debugging; no timing assertion (flake-prone)
// DB truth: every descendant + root is 'cancelled'
const conn = getConn();
+23 -3
View File
@@ -78,7 +78,11 @@ describeE2E('v0.18.0 multi-source — Postgres schema shape (fresh install)', ()
);
expect(rows.length).toBe(1);
expect(rows[0].is_nullable).toBe('NO');
expect(String(rows[0].column_default)).toContain('default');
// Postgres renders a TEXT DEFAULT 'default' literal as `'default'::text`.
// Assert the exact stored expression rather than a loose substring so a
// drift in the schema DEFAULT (e.g. a different sentinel source id) fails
// here instead of silently passing.
expect(String(rows[0].column_default)).toBe("'default'::text");
});
test('composite UNIQUE pages(source_id, slug) replaces global UNIQUE(slug)', async () => {
@@ -292,6 +296,18 @@ describeE2E('v0.18.0 multi-source — cascade delete covers every dependent row'
`INSERT INTO files (source_id, page_id, filename, storage_path, content_hash)
VALUES ('cascadetest', ${aliceId}, 'alice.pdf', 'cascadetest/people/alice/alice.pdf', 'fh1')`,
);
const aliceFile = await conn.unsafe(
`SELECT id FROM files WHERE source_id = 'cascadetest' AND storage_path = 'cascadetest/people/alice/alice.pdf'`,
);
const aliceFileId = aliceFile[0].id as number;
// file_migration_ledger row keyed on the file (FK file_id ON DELETE
// CASCADE). Removing the source cascades sources → files → ledger.
await conn.unsafe(
`INSERT INTO file_migration_ledger (file_id, storage_path_old, storage_path_new, status)
VALUES (${aliceFileId}, 'cascadetest/people/alice/alice.pdf', 'cascadetest/people/alice/alice.pdf', 'pending')
ON CONFLICT (file_id) DO NOTHING`,
);
// Sanity: everything exists
expect((await conn.unsafe(`SELECT COUNT(*)::int AS n FROM pages WHERE source_id = 'cascadetest'`))[0].n).toBe(2);
@@ -299,6 +315,7 @@ describeE2E('v0.18.0 multi-source — cascade delete covers every dependent row'
expect((await conn.unsafe(`SELECT COUNT(*)::int AS n FROM timeline_entries WHERE page_id = ${aliceId}`))[0].n).toBe(1);
expect((await conn.unsafe(`SELECT COUNT(*)::int AS n FROM links WHERE from_page_id = ${aliceId}`))[0].n).toBe(1);
expect((await conn.unsafe(`SELECT COUNT(*)::int AS n FROM files WHERE source_id = 'cascadetest'`))[0].n).toBe(1);
expect((await conn.unsafe(`SELECT COUNT(*)::int AS n FROM file_migration_ledger WHERE file_id = ${aliceFileId}`))[0].n).toBe(1);
// Remove the source.
// v0.26.5: populated sources require --confirm-destructive; --yes alone is rejected.
@@ -310,6 +327,7 @@ describeE2E('v0.18.0 multi-source — cascade delete covers every dependent row'
expect((await conn.unsafe(`SELECT COUNT(*)::int AS n FROM timeline_entries WHERE page_id = ${aliceId}`))[0].n).toBe(0);
expect((await conn.unsafe(`SELECT COUNT(*)::int AS n FROM links WHERE from_page_id = ${aliceId}`))[0].n).toBe(0);
expect((await conn.unsafe(`SELECT COUNT(*)::int AS n FROM files WHERE source_id = 'cascadetest'`))[0].n).toBe(0);
expect((await conn.unsafe(`SELECT COUNT(*)::int AS n FROM file_migration_ledger WHERE file_id = ${aliceFileId}`))[0].n).toBe(0);
// The sources row itself is gone.
const src = await conn.unsafe(`SELECT id FROM sources WHERE id = 'cascadetest'`);
@@ -378,8 +396,10 @@ describeE2E('v0.18.0 multi-source — sync --source routes through sources table
test('performSync with no sourceId falls back to global sync.repo_path', async () => {
const engine = getEngine();
// Global config is still '/some/other/default/path' from the
// previous test. Without --source, performSync uses it.
// Self-contained: set the global config this test depends on directly
// instead of inheriting the side effect of the previous test. Without
// --source, performSync must read this global key.
await engine.setConfig('sync.repo_path', '/some/other/default/path');
let err: Error | null = null;
try {
await performSync(engine, {});
+23
View File
@@ -112,4 +112,27 @@ describe('v0.29 E2E — getRecentSalience (Garry test)', () => {
const rows = await engine.getRecentSalience({ days: 7, slugPrefix: 'nope/does-not-exist/' });
expect(rows).toEqual([]);
});
// TIM-37: the daily briefing writes to the vault and re-ingests as
// `briefings/<date>`. Without this filter the briefing itself would top
// every subsequent Brain Pulse — self-reference with no signal.
describe('TIM-37 — briefings excluded from their own Brain Pulse', () => {
test('default query hides briefings/* slugs', async () => {
await engine.putPage('briefings/2026-05-19', {
type: 'note',
title: 'Daily Briefing — 2026-05-19',
compiled_truth: 'Auto-generated cron briefing.',
});
const rows = await engine.getRecentSalience({ days: 7, limit: 50 });
expect(rows.some(r => r.slug.startsWith('briefings/'))).toBe(false);
});
test('explicit slugPrefix=briefings/ still returns them', async () => {
const rows = await engine.getRecentSalience({ days: 7, slugPrefix: 'briefings/' });
expect(rows.length).toBeGreaterThan(0);
for (const r of rows) {
expect(r.slug.startsWith('briefings/')).toBe(true);
}
});
});
});
+15 -2
View File
@@ -128,6 +128,17 @@ describe('SearchResult fields', () => {
expect(r.chunk_index).toBeDefined();
expect(typeof r.chunk_index).toBe('number');
});
test('empty keyword query returns a defined array without throwing', async () => {
const results = await engine.searchKeyword('');
expect(Array.isArray(results)).toBe(true);
});
test('zero vector search returns a defined array without throwing', async () => {
const zeroVector = new Float32Array(1536);
const results = await engine.searchVector(zeroVector);
expect(Array.isArray(results)).toBe(true);
});
});
describe('detail parameter', () => {
@@ -145,9 +156,11 @@ describe('detail parameter', () => {
});
test('detail=low on vector search filters to compiled_truth', async () => {
// Use a timeline-direction embedding — with detail=low, should get no results
// or only compiled_truth results
// Use a timeline-direction embedding — detail=low filters to compiled_truth.
// Vector search returns every chunk with an embedding (ordered by distance),
// so the seeded compiled_truth chunks are non-empty and ALL compiled_truth.
const results = await engine.searchVector(basisEmbedding(1), { detail: 'low' });
expect(results.length).toBeGreaterThan(0);
for (const r of results) {
expect(r.chunk_source).toBe('compiled_truth');
}
+4 -4
View File
@@ -39,6 +39,7 @@ import { runSkillOpt } from '../../src/core/skillopt/orchestrator.ts';
import {
bestPath,
loadHistory,
proposedPath,
skillPath,
} from '../../src/core/skillopt/version-store.ts';
import { loadRejectedBuffer } from '../../src/core/skillopt/rejected-buffer.ts';
@@ -741,7 +742,7 @@ describe('skillopt T3 — F11 held-out gate, ablation opts, no-DB-pollution', ()
} finally { fixture.cleanup(); }
});
test('--no-mutate writes proposed.md (best.md), leaves SKILL.md untouched', async () => {
test('--no-mutate writes proposed.md and best.md, leaves SKILL.md untouched', async () => {
const fixture = setupFixture(SKILL_PEOPLE_ONLY, CITATIONS_BENCHMARK);
try {
installStub({
@@ -753,10 +754,9 @@ describe('skillopt T3 — F11 held-out gate, ablation opts, no-DB-pollution', ()
const result = await runOnce(fixture, { noMutate: true });
expect(result.outcome).toBe('accepted');
expect(result.mutatedSkillFile).toBe(false);
expect(result.proposedPath).toBeDefined();
// proposed.md (best.md) exists and carries the improvement.
expect(fs.existsSync(result.proposedPath!)).toBe(true);
expect(result.proposedPath).toBe(proposedPath(fixture.skillsDir, SKILL));
expect(fs.readFileSync(result.proposedPath!, 'utf8')).toContain('## Citations');
expect(fs.readFileSync(bestPath(fixture.skillsDir, SKILL), 'utf8')).toContain('## Citations');
// SKILL.md on disk is UNCHANGED (still People-only).
const skill = fs.readFileSync(skillPath(fixture.skillsDir, SKILL), 'utf8');
expect(skill).not.toContain('## Citations');
+11 -3
View File
@@ -73,7 +73,7 @@ describeE2E('E2E: Check-Update', () => {
expect(stdout).toContain('--json');
});
test('handles no-releases gracefully (current repo state)', async () => {
test('check-update --json contract holds regardless of real release state', async () => {
const proc = Bun.spawn(['bun', 'run', 'src/cli.ts', 'check-update', '--json'], {
cwd: new URL('../..', import.meta.url).pathname,
stdout: 'pipe',
@@ -84,8 +84,16 @@ describeE2E('E2E: Check-Update', () => {
expect(exitCode).toBe(0);
const output = JSON.parse(stdout);
// With no releases, should return false and an error
expect(output.update_available).toBe(false);
// Don't pin update_available to a literal value — the repo may or may not
// have a published release. Assert the JSON shape instead.
expect(typeof output.update_available).toBe('boolean');
expect(output.current_version).toBe(VERSION);
if (output.latest_version != null) {
expect(typeof output.latest_version).toBe('string');
}
if (output.release_url != null) {
expect(typeof output.release_url).toBe('string');
}
});
test('version comparison wiring works end-to-end', () => {
+104
View File
@@ -803,3 +803,107 @@ describe('embedAllStale --source threading (D7)', () => {
expect((firstCallOpts as { sourceId?: string }).sourceId).toBe('media-corpus');
});
});
// ────────────────────────────────────────────────────────────────
// Code metadata preservation across re-embed (regression for #769)
// ────────────────────────────────────────────────────────────────
//
// gbrain v0.30.1 and earlier silently clobbered code-chunk metadata
// (language, symbol_name, symbol_type, start_line, end_line,
// parent_symbol_path, doc_comment, symbol_name_qualified) on every
// re-embed pass. The chunker populated those columns at import time,
// but embed.ts loaded chunks via getChunks then mapped them to a
// stripped ChunkInput carrying only 5 fields. upsertChunks then
// OVERWROTE (not COALESCEd) the metadata columns from EXCLUDED, so
// re-embed wiped them to NULL. End result on a real brain: 4875 code
// pages, 47866 chunks, all with NULL language/symbol_name/symbol_type;
// code-def returned 0 hits across every indexed repo.
//
// All three runEmbed paths (--stale autopilot, --all, --slugs) must
// thread metadata through the re-upsert. Tests below assert that the
// engine.upsertChunks call carries the same metadata it loaded.
describe('runEmbed preserves code-chunk metadata across re-embed (regression for #769)', () => {
const fullCodeChunk = {
chunk_index: 0,
chunk_text: '[Java] foo/Bar.java:10-20 method baz',
chunk_source: 'compiled_truth' as const,
embedded_at: null,
token_count: 12,
language: 'java',
symbol_name: 'baz',
symbol_type: 'function',
start_line: 10,
end_line: 20,
parent_symbol_path: ['Bar'],
doc_comment: 'does the thing',
symbol_name_qualified: 'Bar.baz',
};
function metadataOf(chunk: any) {
return {
language: chunk.language,
symbol_name: chunk.symbol_name,
symbol_type: chunk.symbol_type,
start_line: chunk.start_line,
end_line: chunk.end_line,
parent_symbol_path: chunk.parent_symbol_path,
doc_comment: chunk.doc_comment,
symbol_name_qualified: chunk.symbol_name_qualified,
};
}
test('--stale (autopilot path) carries code metadata into upsertChunks', async () => {
const stale = [{
slug: 'code-page',
chunk_index: 0,
chunk_text: fullCodeChunk.chunk_text,
chunk_source: 'compiled_truth',
model: null,
token_count: 12,
}];
let upsertChunkArgs: any[] | null = null;
const engine = mockEngine({
countStaleChunks: async () => 1,
listStaleChunks: async () => stale,
getChunks: async () => [fullCodeChunk],
upsertChunks: async (_slug: string, chunks: any[]) => { upsertChunkArgs = chunks; },
});
await runEmbed(engine, ['--stale']);
expect(upsertChunkArgs).not.toBeNull();
expect(upsertChunkArgs!).toHaveLength(1);
expect(metadataOf(upsertChunkArgs![0])).toEqual(metadataOf(fullCodeChunk));
});
test('--all (full re-embed) carries code metadata into upsertChunks', async () => {
let upsertChunkArgs: any[] | null = null;
const engine = mockEngine({
listPages: async () => [{ slug: 'code-page' }],
getChunks: async () => [fullCodeChunk],
upsertChunks: async (_slug: string, chunks: any[]) => { upsertChunkArgs = chunks; },
});
await runEmbed(engine, ['--all']);
expect(upsertChunkArgs).not.toBeNull();
expect(upsertChunkArgs!).toHaveLength(1);
expect(metadataOf(upsertChunkArgs![0])).toEqual(metadataOf(fullCodeChunk));
});
test('--slugs (per-page embed) carries code metadata into upsertChunks', async () => {
let upsertChunkArgs: any[] | null = null;
const engine = mockEngine({
getPage: async () => ({ slug: 'code-page', compiled_truth: 'x', timeline: '' }),
getChunks: async () => [fullCodeChunk],
upsertChunks: async (_slug: string, chunks: any[]) => { upsertChunkArgs = chunks; },
});
await runEmbed(engine, ['--slugs', 'code-page']);
expect(upsertChunkArgs).not.toBeNull();
expect(upsertChunkArgs!).toHaveLength(1);
expect(metadataOf(upsertChunkArgs![0])).toEqual(metadataOf(fullCodeChunk));
});
});
+2 -1
View File
@@ -15,7 +15,7 @@
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { configureGateway } from '../src/core/ai/gateway.ts';
import { configureGateway, resetGateway } from '../src/core/ai/gateway.ts';
import {
detectRegressions,
computeDriftScore,
@@ -45,6 +45,7 @@ beforeAll(async () => {
afterAll(async () => {
await engine.disconnect();
resetGateway();
});
beforeEach(async () => {
+2 -2
View File
@@ -14,7 +14,7 @@ import { readFileSync } from 'fs';
import { join } from 'path';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { hybridSearch } from '../src/core/search/hybrid.ts';
import { __setEmbedTransportForTests } from '../src/core/ai/gateway.ts';
import { __setEmbedTransportForTests, resetGateway } from '../src/core/ai/gateway.ts';
import { parseQuestionsJsonl, runRetrievalQuality, evaluateGate, type SearchFn } from '../src/eval/retrieval-quality/harness.ts';
import type { ChunkInput } from '../src/core/types.ts';
@@ -60,7 +60,7 @@ beforeAll(async () => {
});
afterAll(async () => {
__setEmbedTransportForTests(null);
resetGateway();
await engine.disconnect();
});
+100
View File
@@ -351,6 +351,106 @@ describe('runExtractFacts — empty-fence guard (Codex R2-#7)', () => {
expect(r.guardTriggered).toBe(false);
expect(r.factsInserted).toBe(1);
});
// ── #2484: structurally-unfenceable hot-memory rows ───────────
// The inline facts writer (backstop.ts) keeps producing
// `row_num IS NULL, entity_slug IS NOT NULL` rows AFTER the v0_32_2
// migration completes: when a resolved slug has no fenceable page
// (slugify-floor / stub-guard-blocked unprefixed slugs like
// `wingman` or `people-jane-doe`), it falls through to a DB-only
// insert with row_num NULL. The OLD guard predicate
// (`row_num IS NULL AND entity_slug IS NOT NULL`) matched these and
// jammed the phase forever (~16/day) — they can never be fenced (no
// page to fence onto; the ledger-complete migration won't re-run).
// The fix requires a LIVE backing page, so these rows no longer gate.
test('#2484: unfenceable inline-writer rows (entity_slug set, NO backing page) do NOT trigger the guard', async () => {
// Two unfenceable rows whose entity_slug has no page row at all.
// eslint-disable-next-line @typescript-eslint/no-explicit-any
await (engine as any).db.query(
`INSERT INTO facts (source_id, entity_slug, fact, kind, visibility, notability,
valid_from, source, confidence)
VALUES
('default', 'wingman', 'handoff note A', 'fact', 'private', 'medium', now(), 'mcp:extract_facts', 1.0),
('default', 'people-jane-doe', 'handoff note B', 'fact', 'private', 'medium', now(), 'mcp:extract_facts', 1.0)`,
);
// A real page with a fence that SHOULD reconcile (proves the phase
// converges past the guard rather than early-returning).
await putPage('people/alice', FACT_FENCE(
`| 1 | real fenced fact | fact | 1.0 | world | high | 2026-01-01 | | s | |`,
));
const r = await runExtractFacts(engine, { slugs: ['people/alice'] });
// Guard must NOT trip — the unfenceable rows are permanent by
// construction, not a migration blocker.
expect(r.guardTriggered).toBe(false);
expect(r.legacyRowsPending).toBe(0);
// The phase ran its reconcile pass (did not early-return).
expect(r.factsInserted).toBe(1);
// The unfenceable rows survive untouched (still row_num NULL).
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const survivors = await (engine as any).db.query(
`SELECT entity_slug FROM facts WHERE row_num IS NULL ORDER BY entity_slug`,
);
expect(survivors.rows.map((x: { entity_slug: string }) => x.entity_slug))
.toEqual(['people-jane-doe', 'wingman']);
});
test('#2484: a genuine legacy row WITH a backing page still triggers the guard (discriminator stays sharp)', async () => {
// Same shape as the unfenceable row above (row_num NULL, entity_slug
// set) — the ONLY difference is a live backing page exists, so the
// migration's Phase B could fence it. This MUST still gate.
await putPage('people/bob', FACT_FENCE(
`| 1 | fence fact | fact | 1.0 | world | high | 2026-01-01 | | s | |`,
));
// eslint-disable-next-line @typescript-eslint/no-explicit-any
await (engine as any).db.query(
`INSERT INTO facts (source_id, entity_slug, fact, kind, visibility, notability,
valid_from, source, confidence)
VALUES ('default', 'people/bob', 'genuine legacy claim', 'fact', 'private', 'medium',
now(), 'mcp:put_page', 1.0)`,
);
const r = await runExtractFacts(engine, { slugs: ['people/bob'] });
expect(r.guardTriggered).toBe(true);
expect(r.legacyRowsPending).toBe(1);
expect(r.factsInserted).toBe(0);
expect(r.factsDeleted).toBe(0);
expect(r.warnings.some(w => w.includes('apply-migrations'))).toBe(true);
});
test('#2484: a soft-deleted backing page makes its legacy row unfenceable (does NOT gate)', async () => {
// Page exists then gets soft-deleted (deleted_at set). The migration
// can't fence onto a deleted page, so the row must not gate.
await putPage('people/carol', FACT_FENCE(
`| 1 | live fact | fact | 1.0 | world | high | 2026-01-01 | | s | |`,
));
// eslint-disable-next-line @typescript-eslint/no-explicit-any
await (engine as any).db.query(
`INSERT INTO facts (source_id, entity_slug, fact, kind, visibility, notability,
valid_from, source, confidence)
VALUES ('default', 'people/carol', 'orphaned legacy claim', 'fact', 'private', 'medium',
now(), 'mcp:put_page', 1.0)`,
);
// Soft-delete the page.
// eslint-disable-next-line @typescript-eslint/no-explicit-any
await (engine as any).db.query(
`UPDATE pages SET deleted_at = now() WHERE slug = 'people/carol' AND source_id = 'default'`,
);
// Reconcile a DIFFERENT live page so the phase has work to do.
await putPage('people/dave', FACT_FENCE(
`| 1 | dave fact | fact | 1.0 | world | high | 2026-01-01 | | s | |`,
));
const r = await runExtractFacts(engine, { slugs: ['people/dave'] });
expect(r.guardTriggered).toBe(false);
expect(r.legacyRowsPending).toBe(0);
expect(r.factsInserted).toBe(1);
});
});
describe('runExtractFacts — multi-source isolation', () => {
+26
View File
@@ -209,6 +209,32 @@ describe('gbrain extract --stale', () => {
expect(usRows[0]?.eq).toBe(true);
});
test('REGRESSION: page with updated_at BEFORE LINK_EXTRACTOR_VERSION_TS clears (no permanent-stale loop)', async () => {
// The v112 watermark column ships with no backfill, so every pre-existing
// page starts NULL-stale — and most pre-date the version bump. Pre-fix,
// extractStaleFromDB stamped links_extracted_at = read updated_at; for a
// page edited before LINK_EXTRACTOR_VERSION_TS the stamp landed BELOW the
// version threshold, so the version arm (links_extracted_at < versionTs)
// re-flagged it stale forever — an infinite re-extract loop that never
// cleared the lag (observed: 97% of pages stuck permanently).
await engine.putPage('people/alice', personPage('Alice'));
await engine.putPage('companies/acme', companyPage('Acme', '[Alice](people/alice) leads [Acme](companies/acme).'));
// Backdate every page to BEFORE the extractor version timestamp.
await engine.executeRaw(`UPDATE pages SET updated_at = '2020-01-01T00:00:00Z'`);
expect(await engine.countStalePagesForExtraction({ versionTs: LINK_EXTRACTOR_VERSION_TS })).toBe(2);
await runExtract(engine, ['--stale']);
// Fixed: stamp = GREATEST(read updated_at, versionTs) → lifts old pages to
// the threshold so the version arm clears, while a real future edit still
// advances updated_at past the stamp (CDX-1 race protection preserved).
expect(await engine.countStalePagesForExtraction({ versionTs: LINK_EXTRACTOR_VERSION_TS })).toBe(0);
// Second run must ALSO find 0 — the defining symptom of the bug was that it
// never converged.
await runExtract(engine, ['--stale']);
expect(await engine.countStalePagesForExtraction({ versionTs: LINK_EXTRACTOR_VERSION_TS })).toBe(0);
});
test('CDX-4 (D2): a link-flush throw aborts the sweep and leaves pages UNSTAMPED', async () => {
await engine.putPage('people/alice', personPage('Alice'));
await engine.putPage('companies/acme', companyPage('Acme', '[Alice](people/alice) founded [Acme](companies/acme).'));
+32
View File
@@ -136,6 +136,38 @@ describe('extractTimelineFromContent', () => {
expect(entries).toHaveLength(1);
});
it('does not split on hyphens inside markdown link targets', () => {
const content = `- **2025-03-18** | Referenced in [Alice](../people/alice-example.md)`;
const entries = extractTimelineFromContent(content, 'companies/acme-example');
expect(entries).toHaveLength(1);
expect(entries[0].source).toBe('markdown');
expect(entries[0].summary).toBe('Referenced in [Alice](../people/alice-example.md)');
});
it('does not split on spaced dashes inside link labels', () => {
const content = `- **2025-03-18** | Referenced in [Deals — Q1 Review](../deals/q1-review.md)`;
const entries = extractTimelineFromContent(content, 'companies/acme-example');
expect(entries).toHaveLength(1);
expect(entries[0].source).toBe('markdown');
expect(entries[0].summary).toBe('Referenced in [Deals — Q1 Review](../deals/q1-review.md)');
});
it('splits on the first spaced dash outside links', () => {
const content = `- **2025-03-18** | [Board notes](../meetings/2025-03-18-board.md) — Approved the hire`;
const entries = extractTimelineFromContent(content, 'test');
expect(entries).toHaveLength(1);
expect(entries[0].source).toBe('[Board notes](../meetings/2025-03-18-board.md)');
expect(entries[0].summary).toBe('Approved the hire');
});
it('keeps delimiterless bullet lines whole instead of dropping them', () => {
const content = `- **2025-03-18** | Imported from legacy tracker`;
const entries = extractTimelineFromContent(content, 'test');
expect(entries).toHaveLength(1);
expect(entries[0].source).toBe('markdown');
expect(entries[0].summary).toBe('Imported from legacy tracker');
});
it('extracts inline citation format entries', () => {
const content = `Closed the seed round with fund-a leading. [Source: board meeting notes, 2025-04-02]`;
const entries = extractTimelineFromContent(content, 'deals/acme-seed');
+5 -5
View File
@@ -83,16 +83,16 @@ describe('put_page facts backstop', () => {
'note/substantive',
`---\ntype: note\ntitle: Substantive\n---\n${'this is some real content with meaningful claims. '.repeat(10)}`,
);
// Either queued (gateway configured) or skipped due to gateway absence
// is acceptable; we only insist the gating doesn't reject on the
// happy path.
// Either queued (gateway configured) or skipped: 'chat_unavailable'
// (#3062: the reset gateway has no chat credential, and the backstop
// now records that instead of enqueueing a job doomed to no-op) is
// acceptable; we only insist the gating doesn't reject on the happy path.
expect(result).toBeDefined();
const r = result!;
if ('queued' in r) {
expect(r.queued).toBe(true);
} else {
// 'backstop_error' or 'queue_shutdown' would be a real failure.
expect(r.skipped).toMatch(/^(queue_shutdown|backstop_error)?$/);
expect(r.skipped).toBe('chat_unavailable');
}
});
+89
View File
@@ -25,8 +25,11 @@ import {
resetGateway,
__setChatTransportForTests,
getChatModel,
unavailableReason,
warnUnavailableOnce,
} from '../src/core/ai/gateway.ts';
import { extractFactsFromTurn } from '../src/core/facts/extract.ts';
import { runFactsBackstop } from '../src/core/facts/backstop.ts';
beforeEach(() => {
resetGateway();
@@ -147,3 +150,89 @@ describe('facts extract — silent-no-op regression (v0.31.6 bug class)', () =>
expect(chatCalled).toBe(true); // ← THE bug-class assertion
});
});
// #3062 — an unauthenticated chat gateway must be DIAGNOSABLE, not a
// silent success-shaped no-op. Three surfaces pinned here:
// 1. unavailableReason() names the missing auth_env key + model.
// 2. warnUnavailableOnce() writes exactly one stderr warning per process.
// 3. runFactsBackstop() records skipped: 'chat_unavailable' (and queue
// mode declines to enqueue a job that is guaranteed to no-op).
describe('#3062 — chat-unavailable is diagnosable, not silent', () => {
test('unavailableReason names the missing auth env key and the model', () => {
configureGateway({
chat_model: 'anthropic:claude-sonnet-4-6',
env: {},
});
expect(isAvailable('chat')).toBe(false);
const reason = unavailableReason('chat');
expect(reason).toContain('ANTHROPIC_API_KEY');
expect(reason).toContain('anthropic:claude-sonnet-4-6');
});
test('unavailableReason is null when the touchpoint is available', () => {
configureGateway({
chat_model: 'anthropic:claude-sonnet-4-6',
env: { ANTHROPIC_API_KEY: 'sk-ant-test' },
});
expect(unavailableReason('chat')).toBeNull();
});
test('warnUnavailableOnce warns exactly once per process per touchpoint', () => {
configureGateway({
chat_model: 'anthropic:claude-sonnet-4-6',
env: {},
});
const seen: string[] = [];
const orig = console.warn;
// eslint-disable-next-line no-console
console.warn = (msg: unknown) => { seen.push(String(msg)); };
try {
warnUnavailableOnce('chat', 'facts extraction skipped');
warnUnavailableOnce('chat', 'facts extraction skipped');
} finally {
// eslint-disable-next-line no-console
console.warn = orig;
}
expect(seen).toHaveLength(1);
expect(seen[0]).toContain('ANTHROPIC_API_KEY');
});
test('runFactsBackstop records skipped: chat_unavailable instead of a success-shaped empty result', async () => {
configureGateway({
chat_model: 'anthropic:claude-sonnet-4-6',
env: {},
});
// The gate fires before any engine use beyond the kill-switch config
// read, so a getConfig stub suffices — no PGLite needed.
const stubEngine = { getConfig: async () => null } as unknown as import('../src/core/engine.ts').BrainEngine;
const page = {
slug: 'note/eligible',
type: 'note' as const,
compiled_truth: 'this is some real content with meaningful claims. '.repeat(10),
frontmatter: {},
};
const inline = await runFactsBackstop(page, {
engine: stubEngine,
sourceId: 'default',
sessionId: null,
source: 'mcp:put_page',
mode: 'inline',
});
expect(inline).toEqual({
mode: 'inline', inserted: 0, duplicate: 0, superseded: 0, fact_ids: [],
skipped: 'chat_unavailable',
});
const queued = await runFactsBackstop(page, {
engine: stubEngine,
sourceId: 'default',
sessionId: null,
source: 'sync:import',
mode: 'queue',
});
expect(queued).toEqual({
mode: 'queue', enqueued: false, queueDepth: 0,
skipped: 'chat_unavailable',
});
});
});
+108
View File
@@ -403,6 +403,77 @@ describe('extractPageLinks', () => {
expect(candidates).toEqual([]);
});
test('path-qualified wikilink outside DIR_PATTERN queries by final segment', async () => {
// `[[notes/struktura]]` (dir not in DIR_PATTERN) falls to the generic
// pass. The resolver's basename index is keyed by final path segments,
// so the lookup must strip the dirname — mirroring the FS path
// (resolveSlugAll). Regression: the raw literal was passed through,
// which never matched, so these links silently dropped.
const seen: string[] = [];
const resolver: SlugResolver = {
resolve: async () => null,
resolveBasenameMatches: async (name) => {
seen.push(name);
return name === 'struktura' ? ['notes/struktura'] : [];
},
};
const { candidates } = await extractPageLinks(
'concepts/x', 'See [[notes/struktura]].',
{}, 'concept', resolver, { globalBasename: true },
);
expect(seen).toContain('struktura');
expect(seen).not.toContain('notes/struktura');
expect(candidates.map(c => c.targetSlug)).toEqual(['notes/struktura']);
expect(candidates[0].linkType).toBe('wikilink_basename');
expect(candidates[0].linkSource).toBe('wikilink-resolved');
});
test('path-qualified wikilink keeps only matches ending with the written path', async () => {
// The written path disambiguates: `[[notes/struktura]]` must never
// attach to `wiki/struktura` even though both share the basename.
const resolver: SlugResolver = {
resolve: async () => null,
resolveBasenameMatches: async (name) =>
name === 'struktura' ? ['notes/struktura', 'wiki/struktura'] : [],
};
const { candidates } = await extractPageLinks(
'concepts/x', 'See [[notes/struktura]].',
{}, 'concept', resolver, { globalBasename: true },
);
expect(candidates.map(c => c.targetSlug)).toEqual(['notes/struktura']);
});
test('path-qualified wikilink matches a deeper real slug by path suffix', async () => {
// The page lives at vault/notes/struktura; the author wrote the shorter
// tail `[[notes/struktura]]`. Suffix matching connects them, while the
// basename-only sibling `wiki/struktura` stays excluded.
const resolver: SlugResolver = {
resolve: async () => null,
resolveBasenameMatches: async (name) =>
name === 'struktura' ? ['vault/notes/struktura', 'wiki/struktura'] : [],
};
const { candidates } = await extractPageLinks(
'concepts/x', 'See [[notes/struktura]].',
{}, 'concept', resolver, { globalBasename: true },
);
expect(candidates.map(c => c.targetSlug)).toEqual(['vault/notes/struktura']);
});
test('path-qualified self-link is dropped like the bare form', async () => {
// `[[notes/struktura]]` written on notes/struktura itself must not
// produce a self-loop (same guard as the bare `[[own-tail]]` case).
const resolver: SlugResolver = {
resolve: async () => null,
resolveBasenameMatches: async (name) =>
name === 'struktura' ? ['notes/struktura'] : [],
};
const { candidates } = await extractPageLinks(
'notes/struktura', 'See [[notes/struktura]].',
{}, 'concept', resolver, { globalBasename: true },
);
expect(candidates).toEqual([]);
});
test('bare wikilink resolution does not interfere with DIR_PATTERN wikilinks', async () => {
// 2b refs (people/alice) take the verb-inferred type;
// 2c refs (struktura) take wikilink_basename. Same call.
@@ -1165,6 +1236,43 @@ describe('makeResolver — fallback chain', () => {
const out = await r.resolveBasenameMatches!('struktura');
expect(out.sort()).toEqual(['notes/struktura', 'struktura']);
});
test('opts.sourceId is forwarded to findByTitleFuzzy (twin of #1436 fix)', async () => {
// Captures every (name, dirPrefix, minSimilarity, sourceId) call so we
// can assert the resolver threads sourceId through. Without the wire-up,
// findByTitleFuzzy would be called with sourceId=undefined and the SQL
// could return cross-source slug suggestions that the FK filter
// downstream silently drops.
const calls: Array<{ name: string; dirPrefix?: string; minSimilarity?: number; sourceId?: string }> = [];
const engine = {
async getPage() { return null; },
async findByTitleFuzzy(name: string, dirPrefix?: string, minSimilarity?: number, sourceId?: string) {
calls.push({ name, dirPrefix, minSimilarity, sourceId });
return null;
},
async searchKeyword() { return []; },
} as unknown as BrainEngine;
const r = makeResolver(engine, { mode: 'batch', sourceId: 'src-a' });
await r.resolve('Alice Example', 'people');
expect(calls.length).toBeGreaterThan(0);
expect(calls.every(c => c.sourceId === 'src-a')).toBe(true);
});
test('opts.sourceId omitted → findByTitleFuzzy receives undefined (back-compat)', async () => {
const calls: Array<{ sourceId?: string }> = [];
const engine = {
async getPage() { return null; },
async findByTitleFuzzy(_name: string, _dirPrefix?: string, _min?: number, sourceId?: string) {
calls.push({ sourceId });
return null;
},
async searchKeyword() { return []; },
} as unknown as BrainEngine;
const r = makeResolver(engine, { mode: 'batch' });
await r.resolve('Alice Example', 'people');
expect(calls.length).toBeGreaterThan(0);
expect(calls.every(c => c.sourceId === undefined)).toBe(true);
});
});
describe('FRONTMATTER_LINK_MAP integrity', () => {
+23
View File
@@ -32,6 +32,29 @@ describe('lintContent', () => {
expect(issues.some(i => i.rule === 'code-fence-wrap')).toBe(true);
});
test('no false positive: page CONTAINS an inner ```markdown code block', () => {
// Real-world case: a docs/SKILL page that shows a markdown example inline.
// Before this fix, the detector used the /m flag so ^/$ matched start/end
// of any line, which fired on any file that simply contained a ```markdown
// line. But fixContent's regex has no /m flag and can only strip whole-file
// wrappers, so the issue was reported as "fixable: true" yet never fixed.
const content =
'---\ntitle: Skill\n---\n\n# Skill\n\nExample input shape:\n\n' +
'```markdown\n# Inner page\nContent.\n```\n\nThat ends the example.\n';
const issues = lintContent(content, 'test.md');
expect(issues.filter(i => i.rule === 'code-fence-wrap')).toHaveLength(0);
});
test('no false positive: multiple inner ```markdown blocks', () => {
// Documentation pages frequently include several markdown examples.
const content =
'---\ntitle: Examples\n---\n\n# Examples\n\nFirst:\n\n' +
'```markdown\nfoo\n```\n\nSecond:\n\n' +
'```markdown\nbar\n```\n\nDone.\n';
const issues = lintContent(content, 'test.md');
expect(issues.filter(i => i.rule === 'code-fence-wrap')).toHaveLength(0);
});
test('detects placeholder dates', () => {
const content = '---\ntitle: Test\ntype: person\ncreated: YYYY-MM-DD\n---\n\n# Test';
const issues = lintContent(content, 'test.md');
+132
View File
@@ -0,0 +1,132 @@
/**
* list_pages clamp local-trust + offset threading op-level coverage.
*
* Pins (upstream draft "gbrain list silently clamps --limit to 100"):
* - Local callers (ctx.remote === false) get an explicit limit above 100
* honored full enumeration is a legitimate local operation.
* - Remote callers keep the 100-row DoS cap, and the clamp is now LOUD:
* exactly one logger.warn (stderr, never stdout) naming both numbers.
* - Defaults unchanged: no limit 50 rows for both local and remote.
* - `offset` threads through to the engine (PageFilters supported it all
* along; the op layer dropped it, so `--offset` was silently ignored).
*/
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { operationsByName } from '../src/core/operations.ts';
import type { OperationContext } from '../src/core/operations.ts';
const SEED_COUNT = 120; // must exceed the remote cap (100) and the default (50)
let engine: PGLiteEngine;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
for (let i = 0; i < SEED_COUNT; i++) {
// Zero-padded slugs → sort:'slug' gives a deterministic order for the
// offset assertions regardless of insert timestamps.
await engine.putPage(`listclamp/page-${String(i).padStart(3, '0')}`, {
type: 'note',
title: `Page ${i}`,
compiled_truth: 'body',
});
}
});
afterAll(async () => {
if (engine) await engine.disconnect();
});
function mkCtx(overrides: Partial<OperationContext> = {}): {
ctx: OperationContext;
warnings: string[];
} {
const warnings: string[] = [];
const ctx = {
engine,
config: {} as any,
logger: {
info: () => {},
warn: (msg: string) => warnings.push(msg),
error: () => {},
} as any,
dryRun: false,
remote: false,
...overrides,
} as OperationContext;
return { ctx, warnings };
}
const op = () => operationsByName['list_pages'];
describe('list_pages — local callers escape the 100-row clamp', () => {
test('remote=false with limit 100000 returns every page', async () => {
const { ctx, warnings } = mkCtx({ remote: false });
const rows = (await op().handler(ctx, { limit: 100000 })) as any[];
expect(rows.length).toBe(SEED_COUNT);
expect(warnings.length).toBe(0);
});
test('remote=false default (no limit) is still 50 — default unchanged', async () => {
const { ctx } = mkCtx({ remote: false });
const rows = (await op().handler(ctx, {})) as any[];
expect(rows.length).toBe(50);
});
});
describe('list_pages — remote callers keep the cap, loudly', () => {
test('remote=true with limit 100000 returns 100 and warns once with both numbers', async () => {
const { ctx, warnings } = mkCtx({ remote: true });
const rows = (await op().handler(ctx, { limit: 100000 })) as any[];
expect(rows.length).toBe(100);
expect(warnings.length).toBe(1);
expect(warnings[0]).toContain('list limit clamped from 100000 to 100');
});
test('remote=true with limit <= 100 does not warn', async () => {
const { ctx, warnings } = mkCtx({ remote: true });
const rows = (await op().handler(ctx, { limit: 60 })) as any[];
expect(rows.length).toBe(60);
expect(warnings.length).toBe(0);
});
test('anything not strictly remote===false is treated as remote (defense in depth)', async () => {
// ctx.remote contract: consumers treat non-false as untrusted even if the
// type is bypassed via cast.
const { ctx, warnings } = mkCtx({ remote: undefined as any });
const rows = (await op().handler(ctx, { limit: 100000 })) as any[];
expect(rows.length).toBe(100);
expect(warnings.length).toBe(1);
});
});
describe('list_pages — offset threads through (regression: was silently ignored)', () => {
test('offset shifts the window under sort=slug', async () => {
const { ctx } = mkCtx({ remote: false });
const all = (await op().handler(ctx, { limit: 100000, sort: 'slug' })) as any[];
const paged = (await op().handler(ctx, { limit: 10, offset: 5, sort: 'slug' })) as any[];
expect(paged.length).toBe(10);
expect(paged.map(r => r.slug)).toEqual(all.slice(5, 15).map(r => r.slug));
});
test('offset near the end truncates the page', async () => {
const { ctx } = mkCtx({ remote: false });
const rows = (await op().handler(ctx, {
limit: 100000,
offset: SEED_COUNT - 7,
sort: 'slug',
})) as any[];
expect(rows.length).toBe(7);
});
test('garbage offset (negative / NaN) is ignored, not fatal', async () => {
const { ctx } = mkCtx({ remote: false });
const neg = (await op().handler(ctx, { limit: 10, offset: -5, sort: 'slug' })) as any[];
const nan = (await op().handler(ctx, { limit: 10, offset: NaN, sort: 'slug' })) as any[];
const base = (await op().handler(ctx, { limit: 10, sort: 'slug' })) as any[];
expect(neg.map(r => r.slug)).toEqual(base.map(r => r.slug));
expect(nan.map(r => r.slug)).toEqual(base.map(r => r.slug));
});
});
+29
View File
@@ -9,6 +9,7 @@ import { loadConfigWithEngine, type GBrainConfig } from '../src/core/config.ts';
interface FakeEngine {
getConfig(key: string): Promise<string | null | undefined>;
listConfigKeys?(prefix: string): Promise<string[]>;
}
function makeEngine(map: Record<string, string | null | undefined>): FakeEngine {
@@ -16,6 +17,9 @@ function makeEngine(map: Record<string, string | null | undefined>): FakeEngine
async getConfig(key: string) {
return map[key];
},
async listConfigKeys(prefix: string) {
return Object.keys(map).filter(key => key.startsWith(prefix));
},
};
}
@@ -92,6 +96,31 @@ describe('loadConfigWithEngine (Phase 4 / F3)', () => {
expect(merged?.embedding_image_ocr).toBe(true);
});
test('DB provider_base_urls.<provider> fills the gateway base URL map', async () => {
const base: GBrainConfig = { engine: 'pglite' };
const engine = makeEngine({
'provider_base_urls.llama-server-reranker': 'http://127.0.0.1:8091/v1',
});
const merged = await loadConfigWithEngine(engine, base);
expect(merged?.provider_base_urls?.['llama-server-reranker']).toBe('http://127.0.0.1:8091/v1');
});
test('provider_base_urls merge is per-provider: file value wins and DB fills siblings', async () => {
const base: GBrainConfig = {
engine: 'pglite',
provider_base_urls: {
'llama-server-reranker': 'http://file.example/v1',
},
};
const engine = makeEngine({
'provider_base_urls.llama-server-reranker': 'http://db.example/v1',
'provider_base_urls.openrouter': 'http://openrouter.example/v1',
});
const merged = await loadConfigWithEngine(engine, base);
expect(merged?.provider_base_urls?.['llama-server-reranker']).toBe('http://file.example/v1');
expect(merged?.provider_base_urls?.openrouter).toBe('http://openrouter.example/v1');
});
test('engine.getConfig throwing is non-fatal — file/env config still returned', async () => {
const base: GBrainConfig = {
engine: 'pglite',
+62
View File
@@ -0,0 +1,62 @@
import { describe, expect, test } from 'bun:test';
import {
extractCycleFreshnessSourceIds,
parseMaintainArgs,
} from '../src/commands/maintain.ts';
import type { Check } from '../src/commands/doctor.ts';
describe('maintain args', () => {
test('defaults to dry-run unless --safe is explicit', () => {
expect(parseMaintainArgs([])).toMatchObject({
safe: false,
dryRun: true,
json: false,
});
});
test('--safe enables mutating safe mode', () => {
expect(parseMaintainArgs(['--safe', '--json'])).toMatchObject({
safe: true,
dryRun: false,
json: true,
});
});
test('--dry-run wins over --safe', () => {
expect(parseMaintainArgs(['--safe', '--dry-run'])).toMatchObject({
safe: true,
dryRun: true,
});
});
});
describe('cycle freshness source extraction', () => {
test('extracts stale source ids from doctor messages', () => {
const checks: Check[] = [
{
name: 'cycle_freshness',
status: 'fail',
message: "Source 'brain-sync-remote-teffur' last cycled 40h ago. Run `gbrain dream --source <id>`.",
},
{
name: 'cycle_freshness',
status: 'fail',
message: "Source 'wiki' last cycled 25h ago. Source 'wiki' last cycled 25h ago.",
},
];
expect(extractCycleFreshnessSourceIds(checks)).toEqual([
'brain-sync-remote-teffur',
'wiki',
]);
});
test('ignores ok and unrelated checks', () => {
const checks: Check[] = [
{ name: 'cycle_freshness', status: 'ok', message: "Source 'fresh' last cycled recently." },
{ name: 'frontmatter_integrity', status: 'warn', message: "Source 'wiki' has frontmatter issues." },
];
expect(extractCycleFreshnessSourceIds(checks)).toEqual([]);
});
});
+44
View File
@@ -343,3 +343,47 @@ describe('issue #1939 — non-string frontmatter coercion', () => {
expect(parsed.title).toBe('A Normal Title');
});
});
// issue #2446 — when frontmatter has no `title:`, prefer the body's first H1
// over the slug/filename-humanized fallback. Slug-based imports (contacts,
// calendar) carry a correct `# Heading` but no frontmatter title; humanizing
// the slug leaks date/id tokens and loses casing (`Defalco` vs `DeFalco`).
describe('issue #2446 — body H1 fallback for missing frontmatter title', () => {
test('no frontmatter title uses the body H1, not the slug-humanized junk', () => {
const md = '---\ntype: person\n---\n\n# John DeFalco\n\nNotes about John.\n';
const parsed = parseMarkdown(md, 'people/contact-20170928-5-john-defalco.md');
expect(parsed.title).toBe('John DeFalco');
// The slug-derived junk title must NOT win.
expect(parsed.title).not.toBe('Contact 20170928 5 John Defalco');
});
test('no frontmatter title and no H1 falls back to the inferred slug title', () => {
const md = '---\ntype: note\n---\n\njust body prose, no heading\n';
const parsed = parseMarkdown(md, 'people/alice-example.md');
expect(parsed.title).toBe('Alice Example');
});
test('frontmatter title wins over a body H1 (no regression)', () => {
const md = '---\ntitle: Frontmatter Wins\n---\n\n# Body Heading\n\nbody\n';
const parsed = parseMarkdown(md, 'people/some-slug.md');
expect(parsed.title).toBe('Frontmatter Wins');
});
test('h2 is not treated as the title; first real H1 is used', () => {
const md = '---\ntype: note\n---\n\n## Subsection First\n\n# The Real Title\n\nbody\n';
const parsed = parseMarkdown(md, 'notes/x.md');
expect(parsed.title).toBe('The Real Title');
});
test('a # inside a fenced code block is not mistaken for the title', () => {
const md = '---\ntype: note\n---\n\n```sh\n# this is a shell comment, not a heading\n```\n\n# Actual Heading\n';
const parsed = parseMarkdown(md, 'notes/x.md');
expect(parsed.title).toBe('Actual Heading');
});
test('trailing closing hashes are stripped from the H1', () => {
const md = '---\ntype: note\n---\n\n# Closed ATX Heading #\n\nbody\n';
const parsed = parseMarkdown(md, 'notes/x.md');
expect(parsed.title).toBe('Closed ATX Heading');
});
});
+7
View File
@@ -354,6 +354,13 @@ describe('MinionQueue: #1737 per-handler default timeout', () => {
expect(sub.timeout_ms).toBe(30 * 60 * 1000);
});
test('contextual per-chunk reindex gets the 60-min default', async () => {
const job = await queue.add('contextual_reindex_per_chunk', { page_slug: 'large-transcript' }, undefined, {
allowProtectedSubmit: true,
});
expect(job.timeout_ms).toBe(60 * 60 * 1000);
});
test('explicit timeout_ms always wins over the default', async () => {
const job = await queue.add('embed-backfill', { sourceId: 'x' }, { timeout_ms: 5000 });
expect(job.timeout_ms).toBe(5000);
+17
View File
@@ -0,0 +1,17 @@
import { describe, expect, it } from 'bun:test';
import { readFileSync } from 'fs';
import { join } from 'path';
describe('root OpenClaw plugin manifest', () => {
it('declares the id required by OpenClaw plugin installs', () => {
const manifest = JSON.parse(readFileSync(join(import.meta.dir, '..', 'openclaw.plugin.json'), 'utf8'));
const entrySource = readFileSync(join(import.meta.dir, '..', 'src', 'openclaw-context-engine.ts'), 'utf8');
const entryId = entrySource.match(/id:\s*'([^']+)'/)?.[1];
expect(manifest.id).toBe(entryId);
expect(manifest.configSchema).toBeDefined();
expect(typeof manifest.configSchema).toBe('object');
expect(manifest.contracts?.contextEngines).toContain('gbrain-context');
expect(entrySource).toContain('export function register');
});
});
+56
View File
@@ -186,11 +186,67 @@ describe('shouldExclude — orphan filter regression (preserve curation)', () =>
expect(shouldExclude('entities/anonymous')).toBe(true);
expect(shouldExclude('atoms/fact-123')).toBe(true);
expect(shouldExclude('skills/gbrain-operations')).toBe(true);
expect(shouldExclude('dreaming/light/2026-07-20')).toBe(true);
expect(shouldExclude('daily/2026-07-20')).toBe(true);
expect(shouldExclude('agent-openclaw/daily/2026-07-20')).toBe(true);
});
test('workspace convention slugs are excluded', () => {
expect(shouldExclude('_brain-conventions')).toBe(true);
expect(shouldExclude('_templates/decision')).toBe(true);
expect(shouldExclude('extracts/2026-06-30/takes.proposed/round-single')).toBe(true);
expect(shouldExclude('2026-07-20')).toBe(true);
expect(shouldExclude('2026-07-20-qa-sweep')).toBe(true);
expect(shouldExclude('agents/arya/identity')).toBe(true);
expect(shouldExclude('agents/arya/memory/dreaming/deep/2026-07-20')).toBe(true);
});
test('regular slugs are NOT excluded', () => {
expect(shouldExclude('people/alice')).toBe(false);
expect(shouldExclude('companies/acme')).toBe(false);
expect(shouldExclude('writing/post-1')).toBe(false);
expect(shouldExclude('agents/arya/qa-reports/launch-review')).toBe(false);
});
});
describe('getHealth orphan_pages uses shared exclusion policy', () => {
test('excluded convention islands do not count against health', async () => {
await engine.putPage('_templates/decision', {
type: 'template', title: 'Decision', compiled_truth: 'template', timeline: '', frontmatter: {},
});
await engine.putPage('skills/arya/source-check', {
type: 'concept', title: 'Skill', compiled_truth: 'skill', timeline: '', frontmatter: {},
});
await engine.putPage('agents/arya/identity', {
type: 'note', title: 'Identity', compiled_truth: 'identity', timeline: '', frontmatter: {},
});
await engine.putPage('people/alice', {
type: 'person', title: 'Alice', compiled_truth: 'real island', timeline: '', frontmatter: {},
});
const health = await engine.getHealth();
expect(health.orphan_pages).toBe(1);
});
test('per-brain config overrides (orphans.exclude_*) also apply to health', async () => {
await engine.putPage('my-private-folder/secret-ref', {
type: 'note', title: 'Ref', compiled_truth: 'ref', timeline: '', frontmatter: {},
});
await engine.putPage('one-off-fixture-page', {
type: 'note', title: 'Fixture', compiled_truth: 'fixture', timeline: '', frontmatter: {},
});
await engine.putPage('people/alice', {
type: 'person', title: 'Alice', compiled_truth: 'real island', timeline: '', frontmatter: {},
});
expect((await engine.getHealth()).orphan_pages).toBe(3);
await engine.setConfig('orphans.exclude_prefixes', 'my-private-folder/');
await engine.setConfig('orphans.exclude_slugs', 'one-off-fixture-page');
expect((await engine.getHealth()).orphan_pages).toBe(1);
await engine.unsetConfig('orphans.exclude_prefixes');
await engine.unsetConfig('orphans.exclude_slugs');
});
});
+38
View File
@@ -66,6 +66,10 @@ describe('shouldExclude', () => {
expect(shouldExclude('templates/meeting-note')).toBe(true);
});
test('excludes deny-prefix: _templates/', () => {
expect(shouldExclude('_templates/meeting-note')).toBe(true);
});
test('excludes deny-prefix: openclaw/config/', () => {
expect(shouldExclude('openclaw/config/agent')).toBe(true);
});
@@ -86,10 +90,44 @@ describe('shouldExclude', () => {
expect(shouldExclude('entities/product-hunt')).toBe(true);
});
test('excludes first-segment: skills, dreaming, and daily', () => {
expect(shouldExclude('skills/arya/source-check')).toBe(true);
expect(shouldExclude('dreaming/light/2026-07-20')).toBe(true);
expect(shouldExclude('daily/2026-07-20')).toBe(true);
expect(shouldExclude('agent-openclaw/daily/2026-07-20')).toBe(true);
});
test('excludes root date logs and agent workspace conventions', () => {
expect(shouldExclude('_brain-conventions')).toBe(true);
expect(shouldExclude('2026-07-20')).toBe(true);
expect(shouldExclude('2026-07-20-qa-sweep')).toBe(true);
expect(shouldExclude('agents/arya/identity')).toBe(true);
expect(shouldExclude('agents/arya/memory/dreaming/deep/2026-07-20')).toBe(true);
});
test('excludes generated extracts', () => {
expect(shouldExclude('extracts/2026-06-30/takes.proposed/round-single')).toBe(true);
});
test('brain-specific exclusions come from config overrides, not global defaults', () => {
// No baked-in defaults for these:
expect(shouldExclude('my-private-folder/some-secret-ref.md')).toBe(false);
expect(shouldExclude('one-off-fixture-page')).toBe(false);
// The per-brain config plane (orphans.exclude_prefixes / exclude_slugs):
const overrides = {
excludePrefixes: ['my-private-folder/'],
excludeSlugs: ['one-off-fixture-page'],
};
expect(shouldExclude('my-private-folder/some-secret-ref.md', overrides)).toBe(true);
expect(shouldExclude('one-off-fixture-page', overrides)).toBe(true);
expect(shouldExclude('people/jane-doe', overrides)).toBe(false);
});
test('does NOT exclude a normal content page', () => {
expect(shouldExclude('companies/acme')).toBe(false);
expect(shouldExclude('people/jane-doe')).toBe(false);
expect(shouldExclude('projects/gbrain')).toBe(false);
expect(shouldExclude('agents/arya/qa-reports/launch-review')).toBe(false);
});
test('does NOT exclude a page ending with log-like text that is not /log', () => {
+8 -2
View File
@@ -218,15 +218,21 @@ describe('progress reporter', () => {
test('only one process-level signal handler installed across many reporters', () => {
// Baseline: one handler already installed by prior tests in this file.
const installedBefore = __signalHandlerInstalledForTest();
// liveReporters is module-global, so a reporter left running by ANOTHER
// test file in the same shard shows up here. Assert the DELTA (these 50
// lifecycles leak nothing) instead of an absolute zero — the absolute
// form flaked whenever shard composition changed and an unrelated file
// held a live reporter across this test.
const liveBefore = __liveReporterCountForTest();
const { stream } = sink(false);
for (let i = 0; i < 50; i++) {
const p = createProgress({ mode: 'json', stream, minIntervalMs: 0, minItems: 1 });
p.start(`phase_${i}`, 1);
p.finish();
}
// After 50 reporter lifecycles, still exactly one handler and zero leaked live entries.
// After 50 reporter lifecycles, still exactly one handler and no new live entries.
expect(__signalHandlerInstalledForTest()).toBe(installedBefore || true);
expect(__liveReporterCountForTest()).toBe(0);
expect(__liveReporterCountForTest()).toBe(liveBefore);
});
test('startHeartbeat() fires heartbeats and stop() clears', async () => {
+142 -2
View File
@@ -14,18 +14,25 @@
* - parseExtractorOutput unit tests for the raw JSON parser
*/
import { describe, test, expect } from 'bun:test';
import { describe, test, expect, afterAll } from 'bun:test';
import {
runPhaseProposeTakes,
parseExtractorOutput,
contentHash,
hasCompleteFence,
extractExistingTakesForDedup,
isWellFormedEmptyExtraction,
PROPOSE_TAKES_PROMPT_VERSION,
EMPTY_EXTRACTION_TOMBSTONE_TEXT,
type ProposeTakesExtractor,
type ProposedTake,
} from '../src/core/cycle/propose-takes.ts';
import { configureGateway, resetGateway } from '../src/core/ai/gateway.ts';
// R5 shard hygiene: leave no configured gateway past the file boundary.
afterAll(() => {
resetGateway();
});
import { BudgetMeter } from '../src/core/cycle/budget-meter.ts';
import type { OperationContext } from '../src/core/operations.ts';
import type { BrainEngine } from '../src/core/engine.ts';
@@ -59,7 +66,15 @@ function buildMockEngine(opts: {
if (existing.has(key)) return [{ id: 1 } as unknown as T];
return [];
}
// INSERT — return nothing
// INSERT into take_proposals — persist the idempotency key so a
// subsequent cycle observes a cache hit, mirroring the real unique
// index on (source_id, page_slug, content_hash, prompt_version).
if (sql.includes('INSERT INTO take_proposals')) {
const [sourceId, slug, ch, pv] = params ?? [];
existing.add(`${sourceId}|${slug}|${ch}|${pv}`);
return [];
}
// Other writes — return nothing.
return [];
},
} as unknown as BrainEngine;
@@ -160,6 +175,72 @@ describe('parseExtractorOutput', () => {
const out = parseExtractorOutput(raw);
expect(out[0]!.domain).toBe('macro');
});
test('strips <think> reasoning tags before parsing (MiniMax-M3, DeepSeek-R1)', () => {
const raw = '<think>Analyzing the prose... I see several claims.</think>\n\n```json\n[{"claim_text":"X","kind":"take","holder":"brain","weight":0.5}]\n```';
const out = parseExtractorOutput(raw);
expect(out).toHaveLength(1);
expect(out[0]!.claim_text).toBe('X');
});
test('strips multiple <think> blocks', () => {
const raw = '<think>First thought.</think>\n<tool_call>...</tool_call>\n<think>Second thought.</think>\n\n[{"claim_text":"Y","kind":"bet","holder":"brain","weight":0.7}]';
const out = parseExtractorOutput(raw);
expect(out).toHaveLength(1);
});
test('handles trailing noise after JSON (leftover fences)', () => {
const raw = '<think>done</think>\n```json\n[{"claim_text":"Z","kind":"take","holder":"brain","weight":0.6}]\n```\n';
const out = parseExtractorOutput(raw);
expect(out).toHaveLength(1);
expect(out[0]!.claim_text).toBe('Z');
});
});
// ─── isWellFormedEmptyExtraction ────────────────────────────────────
// Guards the tombstone against permanently memoizing a transient parse
// failure as "no claims". Only a cleanly-parsed empty array counts as a
// genuine empty extraction; malformed/prose/truncated output must not.
describe('isWellFormedEmptyExtraction', () => {
test('true for a clean empty array (the well-behaved "no claims" response)', () => {
expect(isWellFormedEmptyExtraction('[]')).toBe(true);
expect(isWellFormedEmptyExtraction(' [] ')).toBe(true);
expect(isWellFormedEmptyExtraction('[ ]')).toBe(true);
});
test('true for a fenced empty array', () => {
expect(isWellFormedEmptyExtraction('```json\n[]\n```')).toBe(true);
});
test('true for leading prose then an empty array', () => {
expect(isWellFormedEmptyExtraction('No gradeable claims.\n\n[]')).toBe(true);
});
test('false for empty / whitespace output (transient, must retry)', () => {
expect(isWellFormedEmptyExtraction('')).toBe(false);
expect(isWellFormedEmptyExtraction(' \n ')).toBe(false);
});
test('false for prose-only / non-JSON output (transient, must retry)', () => {
expect(isWellFormedEmptyExtraction('There are no gradeable claims here.')).toBe(false);
expect(isWellFormedEmptyExtraction('null')).toBe(false);
});
test('false for malformed / truncated JSON (transient, must retry)', () => {
expect(isWellFormedEmptyExtraction('[')).toBe(false);
expect(isWellFormedEmptyExtraction('[{"claim_text":"x"')).toBe(false);
});
test('false for a NON-empty array (has content — not an empty extraction)', () => {
expect(isWellFormedEmptyExtraction('[{"claim_text":"x","kind":"take","holder":"brain","weight":0.5}]')).toBe(false);
// Parseable but claim-less array is ambiguous garbage → not a genuine empty.
expect(isWellFormedEmptyExtraction('[{"foo":"bar"}]')).toBe(false);
});
test('false for an empty object (model ignored the array-format instruction)', () => {
expect(isWellFormedEmptyExtraction('{}')).toBe(false);
});
});
// ─── contentHash ────────────────────────────────────────────────────
@@ -436,3 +517,62 @@ New prose appended here.`;
}
});
});
// ─── Empty-extraction memoization (idle-cost fix) ───────────────────
// A page that yields zero gradeable claims must still record an
// idempotency row, or every cycle re-spends an LLM call on unchanged
// prose. Regression guard for the "empty result never memoized" bug.
describe('runPhaseProposeTakes — empty extraction memoization', () => {
test('zero-claim page writes a tombstone row (proposals_inserted stays 0)', async () => {
const pages = [buildPage({ slug: 'test/embed-probe', body: '# probe\njust a test, nothing to grade.' })];
const { engine, captured } = buildMockEngine({ pages });
const extractor: ProposeTakesExtractor = async () => [];
const result = await runPhaseProposeTakes(buildCtx(engine), { extractor });
const details = result.details as Record<string, unknown>;
expect(details.cache_misses).toBe(1);
expect(details.proposals_inserted).toBe(0);
expect(details.tombstones_written).toBe(1);
const inserts = captured.filter(c => c.sql.includes('INSERT INTO take_proposals'));
expect(inserts).toHaveLength(1);
// Tombstone carries the sentinel claim_text and an out-of-queue status.
expect(inserts[0]!.params[5]).toBe(EMPTY_EXTRACTION_TOMBSTONE_TEXT); // claim_text
expect(inserts[0]!.sql).toContain("'rejected'");
});
test('unchanged zero-claim page is a cache hit next cycle (no repeat LLM call)', async () => {
const pages = [buildPage({ slug: 'test/embed-probe', body: '# probe\njust a test, nothing to grade.' })];
const { engine } = buildMockEngine({ pages });
let extractorCalls = 0;
const extractor: ProposeTakesExtractor = async () => {
extractorCalls++;
return [];
};
// Cycle 1: cache miss → LLM call → tombstone written.
const r1 = await runPhaseProposeTakes(buildCtx(engine), { extractor });
expect(extractorCalls).toBe(1);
expect((r1.details as Record<string, unknown>).cache_misses).toBe(1);
expect((r1.details as Record<string, unknown>).tombstones_written).toBe(1);
// Cycle 2: same unchanged page → cache hit → extractor NOT called again.
const r2 = await runPhaseProposeTakes(buildCtx(engine), { extractor });
expect(extractorCalls).toBe(1); // the whole point: no re-spend
expect((r2.details as Record<string, unknown>).cache_hits).toBe(1);
expect((r2.details as Record<string, unknown>).cache_misses).toBe(0);
});
test('extractor error does NOT write a tombstone (page retried next cycle)', async () => {
const pages = [buildPage({ slug: 'wiki/x', body: 'some prose' })];
const { engine, captured } = buildMockEngine({ pages });
const extractor: ProposeTakesExtractor = async () => {
throw new Error('LLM timeout');
};
const result = await runPhaseProposeTakes(buildCtx(engine), { extractor });
expect((result.details as Record<string, unknown>).tombstones_written).toBe(0);
expect(captured.filter(c => c.sql.includes('INSERT INTO take_proposals'))).toHaveLength(0);
});
});
+78
View File
@@ -199,6 +199,84 @@ describe('check-test-isolation.sh', () => {
});
});
describe('R5 — gateway configuration requires resetGateway teardown (#3066)', () => {
it('flags configureGateway without resetGateway in a teardown hook', () => {
const r = runLintIn([
{
path: 'gateway-leak.test.ts',
contents:
`import { beforeAll, test, expect } from 'bun:test';\n` +
`import { configureGateway } from '../src/core/ai/gateway.ts';\n` +
`beforeAll(() => { configureGateway({ env: {} }); });\n` +
`test('x', () => expect(1).toBe(1));\n`,
},
]);
expect(r.status).toBe(1);
expect(r.stdout).toContain('R5');
expect(r.stdout).toContain('gateway-leak.test.ts');
});
it('flags __setEmbedTransportForTests without resetGateway', () => {
const r = runLintIn([
{
path: 'transport-leak.test.ts',
contents:
`import { afterAll, test, expect } from 'bun:test';\n` +
`import { __setEmbedTransportForTests } from '../src/core/ai/gateway.ts';\n` +
`__setEmbedTransportForTests(async () => ({ embeddings: [] }));\n` +
`afterAll(() => { /* disconnect only */ });\n` +
`test('x', () => expect(1).toBe(1));\n`,
},
]);
expect(r.status).toBe(1);
expect(r.stdout).toContain('R5');
});
it('passes when resetGateway is called and an afterAll hook exists', () => {
const r = runLintIn([
{
path: 'gateway-clean.test.ts',
contents:
`import { beforeAll, afterAll, test, expect } from 'bun:test';\n` +
`import { configureGateway, resetGateway } from '../src/core/ai/gateway.ts';\n` +
`beforeAll(() => { configureGateway({ env: {} }); });\n` +
`afterAll(() => { resetGateway(); });\n` +
`test('x', () => expect(1).toBe(1));\n`,
},
]);
expect(r.status).toBe(0);
});
it('a comment mentioning resetGateway() does not satisfy the rule', () => {
const r = runLintIn([
{
path: 'gateway-comment-leak.test.ts',
contents:
`import { beforeAll, afterAll, test, expect } from 'bun:test';\n` +
`import { configureGateway } from '../src/core/ai/gateway.ts';\n` +
`// a co-sharded test that calls resetGateway() would clear this\n` +
`beforeAll(() => { configureGateway({ env: {} }); });\n` +
`afterAll(() => { /* disconnect only */ });\n` +
`test('x', () => expect(1).toBe(1));\n`,
},
]);
expect(r.status).toBe(1);
expect(r.stdout).toContain('R5');
});
it('a test name mentioning "configureGateway (" does not trigger the rule', () => {
const r = runLintIn([
{
path: 'gateway-prose.test.ts',
contents:
`import { test, expect } from 'bun:test';\n` +
`test('works WITHOUT configureGateway (reads registry)', () => expect(1).toBe(1));\n`,
},
]);
expect(r.status).toBe(0);
});
});
describe('scope', () => {
it('skips *.serial.test.ts files entirely', () => {
const r = runLintIn([
+2 -2
View File
@@ -6,7 +6,7 @@
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
import { __setEmbedTransportForTests } from '../../src/core/ai/gateway.ts';
import { __setEmbedTransportForTests, resetGateway } from '../../src/core/ai/gateway.ts';
import { runSearchDiagnose } from '../../src/commands/search-diagnose.ts';
import type { ChunkInput } from '../../src/core/types.ts';
@@ -35,7 +35,7 @@ beforeAll(async () => {
await engine.setPageAliases('projects/mingtang', 'default', ['hall of light']);
});
afterAll(async () => { __setEmbedTransportForTests(null); await engine.disconnect(); });
afterAll(async () => { resetGateway(); await engine.disconnect(); });
describe('search diagnose', () => {
test('alias query: trace shows alias match + hybrid rank 1', async () => {
+2 -1
View File
@@ -23,7 +23,7 @@
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
import { configureGateway } from '../../src/core/ai/gateway.ts';
import { configureGateway, resetGateway } from '../../src/core/ai/gateway.ts';
import type { ChunkInput } from '../../src/core/types.ts';
let engine: PGLiteEngine;
@@ -95,6 +95,7 @@ beforeAll(async () => {
afterAll(async () => {
await engine.disconnect();
resetGateway();
});
describe('searchVector per-page max-pool (T1)', () => {

Some files were not shown because too many files have changed in this diff Show More