* fix(security): fence the dedup-resolved slug under the caller's own confinement
put_page's resolved-slug re-check tested `ctx.auth.boundSlugPrefixes` only.
The delegated submit_agent -> subagent context carries `viaSubagent` +
`allowedSlugPrefixes` but no `auth`, so a slug-bound client holding `agent`
scope could delegate a write and have importFromContent's dedup pre-check
redirect it onto a page outside its grant — where the disk write-through
then re-rendered the victim's file with the caller's provenance.
The re-check now applies whichever confinement the caller is actually
under (OAuth binding and/or subagent allow-list / legacy namespace) via
`slugOutsideCallerFence`, which composes the existing match rules rather
than re-deriving them. Dedup returns status 'skipped' before any DB write,
so the throw still rolls nothing back. The denial does not name the
resolved slug (slug-enumeration oracle) and reads "your write scope",
since either confinement can trigger it.
Reported privately by Aleksei Razsadin.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test: coverage for the OAuth in-fence redirect and the missing-subagentId guard
* v0.42.73.2 fix(security): fence dedup-resolved writes to the caller's own write scope
VERSION + package.json + CHANGELOG for 0.42.73.2.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs: state that the write fence follows a delegated write
---------
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* v0.42.73.1 revert(ci): remove the PR gate — it cannot function on this repository
The gate needs an ANTHROPIC_API_KEY Actions secret for its verdict and
read-write workflow permissions to post a comment or set a label. This
repository grants neither, and both are admin-only, so it can only ever skip.
It also caused a real incident before that was understood: a read-only token
turned every API call into a 403, the code treated that as a crash, and the
check went red on an outside contributor's PR four times with no comment
explaining why. v0.42.73.0 fixed the crash, but a check that runs on every PR
and can never reach a verdict does not earn a place in the repo.
Removes the workflow, the script, its type surface, and its test file. The code
is preserved in git history at v0.42.73.0. If it is ever restored, the
mechanical half (intent/screenshot policy, title rule, red flags) should render
to the Actions job summary rather than a comment — that needs no token
permission and no API key.
CONTRIBUTING.md's intent-paragraph and screenshot requirement is unchanged and
stands as written; it is enforced by maintainers reading PRs, as before.
typecheck clean, verify 34/34, llms bundles regenerated.
* v0.42.73.1 revert(docs): withdraw the human-intent-paragraph + screenshot contribution requirement
This reverts commit 6d1232d5a6 (v0.42.72.1).
CONTRIBUTING.md, both issue templates, and the pull-request template return
to their pre-2026-08-02 state. VERSION/package.json/CHANGELOG keep moving
forward (0.42.73.1); the v0.42.72.1 CHANGELOG entry stays as historical
record, and the 0.42.73.1 entry now describes the withdrawal.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
osv-scan passed on the last two release PRs and fails on this one, so these are
newly published, not tolerated debt: 2 High + 4 Medium across fast-uri (3.1.4 →
3.1.5), hono (4.12.25 → 4.12.34), ip-address (10.1.1 → 10.3.1), and admin's
postcss (8.5.10 → 8.5.23). All four are transitive and pinned through the
overrides block, so the fix is the pin, not a dependency addition.
Resolved: fast-uri@3.1.5, hono@4.13.0, ip-address@10.4.0, postcss@8.5.25.
The gate's first live runs put a red X on an outside contributor's PR five
times over, with no comment explaining why. This repo's GITHUB_TOKEN is
read-only, so every comment and label call returned 403; the throw reached the
top-level handler as exit 2.
The gate is advisory. A repository permission problem is an operator condition,
never a statement about the PR under review. 401/403/404 from GitHub now emit a
loud operator-facing warning naming both blockers (workflow permissions and the
missing ANTHROPIC_API_KEY secret) and exit 0. A genuine outage or a bug in here
still fails visibly at exit 2. Pinned by four tests including the entry-handler
routing; mutation-tested.
* fix(envelope importer): honor the envelope's own counts, and stop a second import destroying the first
Two ways this importer lost conversation history at exit 0 with zero stderr
bytes. Both reproduced against output from the real memvelope converter, not
hand-authored envelopes — the blind spot that let them through review.
F1. envelope-v0 makes meta.conversation_count and meta.message_count mandatory:
the envelope states its own totals. The script read neither. A file declaring
353 conversations and 9412 messages that actually held one of each imported
one, exited 0, and printed a receipt internally consistent with the reduced
output. The receipt also counted only pages, never messages, so every
message-level loss was invisible by construction — a conversation arriving with
one turn instead of forty still writes exactly one page.
Each declared count is now checked on its own, before the first write. A count
that disagrees with the contents, or that is present but is not a non-negative
integer, refuses the import (exit 2). A count that is simply absent cannot be
compared, so the envelope still imports — but stderr names the field whose half
of the check was skipped, because an unchecked import must never look like a
checked one. The receipt reports messages as well as pages, and says so out
loud when messages read exceed messages written.
F2. filesWritten was per-process and the default outDir is a fixed literal, so
a second import into the same directory clobbered the first with no read-back
and no warning. The trigger is the positional fallback conv-N, which fires
whenever c.id is not a non-empty string — and the spec is explicit that id is
string | null and that a converter MUST NOT synthesize one, so null is the
CONFORMING shape, not malformed input. The reference converter emits it for any
ChatGPT export lacking conversation_id and id. Filenames then key off array
position rather than identity, and two unrelated exports both put their first
conversation at conv-1.
Target files are now examined before anything is written. Byte-identical
content is a re-import; a page this importer wrote from the SAME conversation
id is a refreshed export legitimately updating its own page. Anything else — a
foreign file, or one of ours whose conversation id cannot be matched — refuses
the import (exit 2) and writes nothing. Refusing rather than disambiguating
keeps the importer from inventing an identity the envelope declined to state,
and the remedy is a different output directory.
Both checks run before the first write, so a refused import leaves no partial
output to be mistaken for a whole one, and does not even create the directory.
Verified against all 12 golden fixtures from the memvelope reference converter
and 7 envelopes freshly produced by running that converter over synthetic
ChatGPT and Claude exports: every one imports at exit 0 with zero stderr bytes,
and all 19 produce byte-identical pages to the previous script. The guards do
not false-positive on legitimate producer output.
Known limit, documented in the header: the target-file check is check-then-
write, not atomic. Two simultaneous imports into one directory can both pass it
— measured 19 refused / 21 raced over 40 trials, against 0 / 40 before.
17 new tests; the 13 existing tests are unchanged and still pass. No new
dependencies: package.json and bun.lock are byte-identical.
* fix(envelope importer): close the three holes the adversarial pass found in the guards
An eight-agent refutation round attacked every claim made for the previous
commit. Four claims survived untouched; three did not, and all three were in
code that commit introduced. Each is reproduced, then closed, with tests.
Identity was compared on the TRIMMED conversation id, because that is what the
filename slugs and what the frontmatter recorded. So two ids differing only by
surrounding whitespace — both copied verbatim by the reference converter, both
schema-valid — looked like one conversation to the guard, and the second import
destroyed the first at exit 0 with zero stderr bytes. The surviving page then
recorded an id that appeared nowhere in the envelope that wrote it. The
frontmatter now records the id verbatim, as the spec requires, and identity is
matched raw. The filename still slugs the trimmed form, so no page's bytes
change for any id without surrounding whitespace.
The identity scan required a page to start with exactly `---\n`. A page THIS
IMPORTER WROTE that later picked up CRLF line endings or a UTF-8 BOM — a git
checkout with core.autocrlf, a cross-platform sync, an editor save — was
therefore reclassified as foreign, and one such page refused the ENTIRE
envelope at exit 2. The previous script treated the same mutation as a harmless
overwrite, so the guard had turned a cosmetic byte change into an unrecoverable
block. Both are normalized away before the scan. The message for a file that
genuinely cannot be recognized no longer asserts "was not written by this
importer" — a claim this code cannot make, and one that was false for exactly
the pages it was being printed about.
The new message-delta warning announced "N message(s) in the envelope are not on
disk" whenever pages collided. Duplicate ids are conforming input — the spec has
merging never deduplicate — so converting an old export together with a newer
one, which is what the memvelope CLI tells users to do, fired it routinely while
every unique turn was on disk. It now states what it knows: the overwritten
pages carried N messages that are not on disk, and if those were earlier copies
the surviving page may already hold them. The raw tally stays, since hiding it
is what made message-level loss invisible to begin with.
Two header claims were false as written and are corrected: message text is
byte-verbatim for 18 of the 19 producer envelopes, not all 19 — the
lone-surrogate fixture writes U+FFFD, unchanged from before and out of scope —
and sequential coverage is no longer described as complete. Three further limits
are now stated rather than left implicit: an id-less conversation cannot be
refreshed in place (refused, not applied — the deliberate trade, since the same
ambiguity resolved the other way is the defect being fixed); a conversation
whose created_at moves orphans its earlier page rather than updating it; and a
hand-written lookalike is indistinguishable from a page this importer wrote.
Rejected after reproducing it: a "silent loss" via a duplicate top-level
conversations key in the JSON text. JSON.parse keeps the last such key, which is
language semantics, is identical on the previous script, and cannot be produced
by JSON.stringify.
36 tests pass, 0 fail. The 13 pre-existing tests remain byte-identical as the
first 306 lines of the file. All 19 real-producer envelopes still produce
byte-identical pages. package.json and bun.lock unchanged.
* fix(envelope importer): write pages gbrain's own conversation parser can read
Every page this importer wrote declared `type: conversation` — which opens the
gate to conversation-facts extraction, chronicle eligibility and the
conversation_format_coverage check — and then presented a turn header
**Assistant** (2025-11-02T14:22:51.000Z · m2):
matching none of the 17 built-in patterns in the conversation parser. The
extractor parsed zero messages, incremented `pages_skipped`, and said nothing.
Pages were stored and searchable; no fact was ever extracted from any of them.
Measured on two throwaway PGLite brains fed the same 13 conversations, one
written each way: `conversation-parser scan` goes from 13/13 `no_match` with 0
messages to 13/13 `imessage-slack` with 33; `extract-conversation-facts
--dry-run` goes from "Skipped 13 page(s)" to every page segmenting and reaching
the extractor; `doctor` conversation_format_coverage goes from warn "13/13 ...
match NO built-in pattern" to ok "13 pages: imessage-slack=13".
The turn header is now `**Me** (2025-11-02 14:22):`, the one shape that parser
reads. 24-hour, not 12-hour-with-AM/PM: both match and both were measured to
reconstruct all 24 hours exactly, so the tie is broken on the fact that 24-hour
is a substring of the envelope's own `ts` (no hour arithmetic, so the 12/0
boundary cannot be got wrong) and sorts chronologically within a day.
That header can carry a wall clock and nothing else, so per-message identity
moves to frontmatter as an array of maps:
messages:
- id: "m1"
ts: "2025-11-02T14:22:51.000Z"
An array, not a map keyed by id — a map discards order and collapses the
duplicate ids the spec permits. Consumers index it by position.
Every scalar is JSON-encoded, so every timestamp is quoted. Unquoted, js-yaml
reads an RFC 3339 scalar as a JS Date: microseconds truncate, a +05:30 offset
normalises away, and gbrain's own coerceFrontmatterString slices a Date to 10
characters — the time of day gone. It is sticky, too. The suite fails if a
timestamp is ever emitted unquoted, and carries a sentinel proving that guard
fires.
Also drops the `---` rule between turns: it is a non-blank line matching no
pattern, so the parser appended it to the preceding message and every extracted
text ended `...\n---`.
Verified over all 12 golden fixtures from the memvelope reference converter
(exit 0, zero stderr, 33/33 messages, every id and ts byte-identical after the
round trip) and over fresh envelopes built by running that converter over
synthetic vendor exports. 52 tests green, up from 36.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(envelope importer): build the offset-shifted header clock without Date.UTC
Self-review of the branch, before the refutation round. `headerClock()` is the
only place in this script that does arithmetic on a timestamp, and its offset
branch had two ways to emit a header that is off — or that does not parse at
all.
`Date.UTC` applies MakeFullYear: a year of 0..99 is read as 1900+y. The shape
regex accepts any four digits, so `0050-01-01T00:30:00+05:30` was rendered as
1949-12-31 — the page moved by nineteen hundred years, silently, at exit 0.
Built with the UTC setters instead, which do not remap.
And `getUTCFullYear()` returns `49`, not `0049`. The pattern's regex requires
`\d{4}`, so an unpadded year emits a header that matches nothing and the turn is
appended to its neighbour as a continuation — one message where there were two.
The year is now padded like every other field.
Neither is reachable from the reference converter, which emits `Z`. Both are
reachable from a conforming envelope: the spec types `ts` as any date-time
string, and RFC 3339 offsets are legal.
18 new tests pin the clock on its own: every Z / offset / designator-less form,
both day-boundary crossings, `+0530` without a colon, the sub-100 year in both
branches, and the six unusable-`ts` shapes that must fall back to the
conversation's date at midnight rather than fabricate a clock or drop a turn.
Three more hostile ids added — U+2028, U+2029 and U+0085 are line breaks in
YAML 1.1 and JSON.stringify emits all three raw, so they are the sharpest
version of the injection the quoting exists to stop. js-yaml 3.14 does not
honor them as breaks; measured, not assumed.
72 tests pass, 0 fail. Adjacent suites (conversation-parser, markdown,
extract-conversation-facts, doctor backlog): 380 pass, 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs(envelope importer): name the built-in pattern count the header claims
'None of the built-in patterns' is a claim about a number the reader cannot
see. There are 17, and `gbrain conversation-parser list-builtins` is how to
count them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(envelope importer): stop the conversation title manufacturing a turn
An adversarial round against the new format. Three claims in the file header
were wrong or overstated, and one of them was a defect.
The H1 heading is the only place a third-party string reaches the BODY — every
frontmatter value is JSON-escaped, the heading was interpolated raw. That was
merely untidy while nothing parsed the body. Now that the turn headers are
legible, a title carrying a newline manufactures a TURN, and one that lands
AHEAD of every real one, so `messages[0]` in frontmatter names content the user
never sent and every id after it is off by one. Measured: a 2-message
conversation titled "Real Title\n\n**Me** (2020-01-01 09:00): INJECTED" parses
to 3 turns, the first attributed to 2020. The heading is flattened to one line;
the verbatim title, newlines and all, is still recorded in frontmatter.
The empty-title fallback also disagreed with itself — "Untitled conversation"
in frontmatter, "Conversation" in the heading, two names for the same missing
thing, and parseMarkdown prefers the body's H1 when frontmatter has no title.
One fallback now.
Corrected in the header, because a false claim there is worse than a limit:
"the frontmatter array still holds exactly the real turns, so the two can be
reconciled by count" was FALSE. parse.ts picks one pattern per page, scored on
the first 10 body lines, and only re-scores full-body under 0.3 — so a pasted
Slack or Telegram snippet inside one message only has to win that window, and
the length of the real conversation is irrelevant. With four `**[09:0N] Name:**`
lines quoted inside message 1: 40 real turns are replaced by 4 fabricated
speakers at fabricated times, at exit 0, with phase regex_match, so
pages_skipped stays 0 and doctor's conversation_format_coverage reports OK. And
at 4 real turns against 4 pasted lines the COUNTS AGREE while every speaker and
timestamp is fabricated — which is exactly the check the old wording offered as
the remedy. The table is in the header now, with what count-comparison does and
does not catch. Closing it needs a parse.ts change, not an importer change.
"every timestamp is QUOTED" was true only for conforming input. JSON quotes
strings; an envelope whose `ts` is a number emits `ts: 1762093371000` unquoted.
That is a YAML integer, not a Date, so it is lossless and carries no truncation
hazard — but the claim as written was wrong, and the behavior is now pinned by
test rather than described.
Two more limits stated: minute resolution is the parser's ceiling, not this
format's — every branch of buildIso hardcodes :00 and no built-in captures
seconds, so two turns in the same minute collide (claude-basic m1 15:02:00 and
m2 15:02:31 both parse to 15:02:00Z) and frontmatter `ts` is the only
full-resolution value on the page. And the array survives a gbrain rewrite
semantically, not textually: serializeMarkdown re-emits `id: "m1"` as `id: m1`.
81 tests pass, 0 fail. Adjacent suites: 389 pass, 0 fail. All 12 golden
fixtures still import at exit 0 with zero stderr.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(envelope importer): range-check the per-message clock, and arm two guards
Second adversarial round. The shape regex counts digits; it does not know a
calendar. `2025-99-99T99:99:00Z` was copied straight into a turn header — and
`imessage-slack` MATCHES that header, so gbrain stored an instant no calendar
contains. `2025-02-30` is the sharper one: it yields a VALID JS Date, silently
shifted to March 2. The script already validates `created_at` before it reaches
a header, with a comment explaining why; the per-message clock is the same
untrusted surface and is used far more often. It is validated now, by round
trip — a date that does not survive its own UTC round trip was never a date.
Also removes a provably unreachable `Number.isFinite` check: over the whole
space the shape regex admits, the constructed instant is always finite.
Two guards had no test that fires. Both are now pinned:
- The `created_at` validation. Dropping it let a hostile value break the turn
header it was supposed to anchor, and the whole suite stayed green.
- The determinism guard — the reason TS_SHAPE exists rather than
`new Date(string)`, which parses a designator-less date-time as LOCAL time.
The one test that touched it only caught the mutation because this box is
America/New_York; on a UTC runner it passed. TZ is now pinned explicitly and
the case runs under four zones including UTC and Pacific/Kiritimati.
94 tests pass, 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs(envelope importer): correct five claims the adversarial round falsified
None of these change behavior. All five were statements in the file header that
an independent pass could not reproduce, and a false claim in a header is worse
than a limitation, because a reader has no way to tell.
"the field is read nowhere outside builtins.ts", of `time_format`, was flatly
false: src/commands/conversation-parser.ts reads it twice. Both are display —
`list-builtins` prints it — and the operative half stands, since parse.ts never
reads it and converts off the captured AM/PM group instead. Said that way now.
"turns averaging more than ~19 non-blank lines fall below the floor" was
optimistic by one line: measured line by line, 18 parses (0.0526) and 19 does
not (0.0499), because the H1 is in the denominator too.
"a fallback when `ts` is null" understated the condition. It fires for any
string this script will not read a clock out of — a date with no time, a basic-
format 20251102T142251Z, an impossible 2025-02-30 — silently preferring the
conversation's date over a value that looked like a time.
"Every emitted scalar is JSON-encoded" was false for the four keys the script
writes itself (`type:`, `origin:`, `date: null`, `messages: []`), which are
literals under its own control rather than envelope data.
"collapses the duplicate ids the spec permits" cited the wrong half of the
spec: duplicates are tolerated for CONVERSATION ids; message ids are positional
and unique within their conversation. The real argument for an array is that
order IS the join. And since a conforming envelope's message id is derivable
from its index, `ts` is the only genuinely new value the array carries — worth
saying rather than implying more is being rescued than is.
Also: the STATUS block quoted `gbrain conversation-parser scan` as producing a
13-page aggregate; it takes one slug and has no aggregate form. The numbers
were reproduced per-slug, so the receipt now shows the command that produced
them.
And two tests changed by this branch still used `page.split('---')[1]` to reach
the frontmatter — the exact idiom this branch's own helper documents as unsound,
since it cuts on the substring anywhere including inside a quoted value. Both
now go through that helper.
94 tests pass, 0 fail. Adjacent suites: 402 pass, 0 fail. All 12 golden
fixtures still exit 0 with zero stderr and 33/33 messages parsed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(envelope importer): break a duplicate id on updated_at, not array position
A merged re-export kept the STALE copy, deterministically.
The memvelope CLI's own USAGE tells users to pass every downloaded export
at once; folder expansion sorts by filename (cli/convert.mjs expandInputs)
and SPEC.md rule 8 forbids the converter from re-sorting conversations
afterwards. Every automatic duplicate-namer a browser or OS gives a second
download of `conversations.json` inserts a character that sorts below `.`
— ` (1)`, `(1)`, `-1`, ` 2` — so the RE-EXPORT lands first in
`conversations[]` and the ORIGINAL lands last. Resolving the collision by
array index therefore threw away the newer copy every time, not half the
time.
Reproduced with an envelope built by the reference converter from two
ChatGPT downloads named the way a browser names them: 4-turn re-export at
index 0, 2-turn original at index 1, and the 2-turn page reached disk at
exit 0.
`updated_at` is a required conversation key in envelope-v0, is populated
by both vendor paths of the reference converter, and was read by nothing
here. It now decides, compared as an INSTANT (a `+05:30` value sorts above
a `Z` value it precedes) at full sub-second resolution (the converter
emits milliseconds always).
The tiebreak applies only when BOTH copies carry an orderable value.
Equal, absent, non-string or unparseable on either side falls back to
array order — a missing timestamp is not evidence of being older, and
preferring the copy that has one is a guess dressed as a rule. `created_at`
is not a secondary key: it is identical in both copies of a re-export.
The collision warning is not quieter — it is louder. It now names which
copy survived, on what basis, and with both timestamps, because
"overwriting the earlier page" is false whenever the tiebreak fires. The
run summary says "discarded" rather than "overwritten" for the same
reason: the losing copy is now sometimes never written at all.
`updated_at` is still written to no page. That is a separate, tracked gap.
Receipts: 107 pass / 0 fail (94 pre-existing, unweakened; 13 new, red-first
against cd88d1de at 96 pass / 11 fail). Over all 12 golden fixtures from
the reference converter plus the repo's sample, output pages, stdout,
stderr and exit code are byte-identical to cd88d1de — this fires only on a
filename collision.
* docs(envelope importer): name the extraction path, the UTC day, and drop a contradicted key claim
R2. The header promised `type: conversation` "keeps pages eligible for
conversation-facts extraction ... after sync" and stopped there, which
reads as "sync extracts facts from these." It does not. The accepting
path is `gbrain extract-conversation-facts`, run by hand, and its
autopilot wrapper — the `conversation_facts_backfill` cycle phase — is
opt-in and OFF by default. Both names verified against upstream/master
before writing them: src/cli.ts:1964 dispatches the command,
src/core/cycle/conversation-facts-backfill.ts:4-6 and :34 document the
phase as "Default OFF" with `enabled (false)`, and doctor.ts:3373 reads
an absent key as disabled. What the type actually buys is admission to
`ALLOWED_TYPES`, and that is now what the header says.
R3. The page `date` and every turn header are the UTC calendar day, not
the user's, and the limits block did not say so. Reproduced on this box:
a conversation held at 19:30 on Sunday 2 November 2025 in California is
`created_at: "2025-11-03T03:30:00.000Z"`, and the page comes out
`date: "2025-11-03"` — a Monday — with a Monday turn header. The output
is byte-identical under TZ=UTC, America/Los_Angeles, Asia/Kolkata,
Pacific/Kiritimati, Pacific/Midway and Europe/London (one sha across all
six), so this is the format's day, not the importing machine's.
Named as unfixable, with the reason rather than an apology: envelope-v0
renders every timestamp as `...Z` and SPEC.md rule 3 says why — "The
source's true offset is unknowable, and UTC is the only
machine-independent choice." A source offset is normalised away before
this script opens the file, and the vendor fields the reference
converter reads carry none to preserve: over the 22 conversation-level
timestamps in its own input fixtures, 16 are bare unix-epoch numbers, 4
end in `Z`, 2 have no designator, and 0 carry a numeric offset. No fix
is invented; the one available guess (the importing machine's zone)
would break determinism.
R4. Deleted the claim that "the id is the natural key," which the header
disclaimed 127 lines later in its own identity limit. The key is the
PAIR — date and id name the file — and the surviving limit already says
so correctly. The same claim appeared a second time as an in-code
comment at the filename site ("the date only leads as a ... prefix; the
id carries uniqueness"); deleting one statement of a false claim and
leaving its twin is a half-correction, so both went. Flagged as a
deviation: R4 as written names the header only.
The bullet R4 emptied also still described the pre-R1 duplicate-id
behaviour, so it is rewritten to the behaviour that now ships.
Doc-only: 107 pass / 0 fail, unchanged.
* fix(envelope importer): correct what the adversarial gate falsified in R1-R4
Six independent refuters plus a completeness critic ran against 81deb83d.
Two of my own corrections were themselves false, one broke a repo-wide CI
gate, and one real defect nobody had measured turned up. Everything here
was re-derived on this box before it was written.
BLOCKER — test/fixtures/memvelope/merged-re-export.mve.json shipped with
no trailing newline, taking scripts/check-trailing-newline.sh from green
to red. It is wired into `bun run check:all`, `bun run check:newlines` and
three call sites in scripts/ci-local.sh, so the branch could not pass the
project's own local CI. Now `trailing-newline check: ok (1547 files)`.
BLOCKER — the R2 paragraph claimed `ALLOWED_TYPES` "requires"
`type: conversation`, "so a page typed anything else is not merely un-run,
it is ineligible." False. ALLOWED_TYPES (extract-conversation-facts.ts:142)
admits six types — conversation, meeting, slack, email, imessage,
imessage-daily — and :569 defaults to the whole list. A new false claim
about a named upstream constant, in a branch whose purpose is deleting
false claims. Rewritten to say what the type actually buys: admission, not
a trigger, and one admission among six.
Also in R2: "extracts no facts from any of them until one of those two is
invoked" is unsafe as an absolute. facts/eligibility.ts ORs the type test
with RESCUE_SLUG_PREFIXES = ['meetings/','personal/','daily/'], so a page
written into an outDir syncing under one of those IS picked up by a plain
sync — subject to the 80-char MIN_BODY_CHARS floor, which nobody in the
gate noticed either. Both named; the default outDir is unaffected and now
says so.
MAJOR — R4's replacement sentence, "Two conversations carrying the SAME id
land on one filename", is false and reinstated the exact contradiction R4
removed. Verified: one id at two created_at dates writes TWO files and
zero collisions. The filename is the PAIR.
MAJOR — the same bullet stated the fallback condition backwards ("when
neither carries an orderable one"). The code (`a === null || b === null ||
a === b`) falls back when EITHER side is unorderable, or when the two are
equal, and the function's own doc said so correctly 460 lines down.
MAJOR — the R3 census was wrong by four in both figures: 22/16 should be
26/20. chatgpt-split.json is an array of PARTS, each an array of
conversations, and my counter walked the parts as if they were
conversations and scored zero. Recounted with the level flattened: 13
conversations, 26 conversation-level timestamps, 20 bare unix-epoch, 4 Z,
2 no-designator, 0 numeric offset. The conclusion is unharmed and slightly
stronger. Also narrowed "not fixable, here or upstream" to "not fixable
HERE" — a converter CAN hold a source offset and this format drops it by
design, which is a decision, not an impossibility.
MINOR (found independently by four of six refuters) — the message-delta
warning still said "the overwritten page(s)", the exact word this branch
deleted from the summary line one commit earlier for being false whenever
the tiebreak keeps the earlier copy. Now "discarded" in both places.
R1's own limits, all newly documented rather than newly introduced:
- when only ONE copy carries an orderable `updated_at` the rule changes
nothing and the stale copy still wins. `updated_at: null` is
producer-reachable (both normalizers end in `|| null`) though it is 0
of 13 in the reference corpus. Refusing to promote a copy for merely
HAVING a timestamp is the deliberate choice; the header now states
what it costs instead of implying coverage the rule does not have.
The argument is also repaired: both copies come from ONE converter
run, so the asymmetry is the vendor's and its direction is
unrecoverable — not the weaker "an older converter" reasoning.
- the reduction is PAIRWISE. With every copy orderable it is a true
maximum in all six permutations; with an unorderable copy in the
middle transitivity is lost and [later, absent, earlier] keeps
`earlier`. Now documented and pinned by a 7-case test.
- the tiebreak is WITHIN one envelope. Across runs, check 2 still lets
an older export refresh a newer page at exit 0 with no warning at all.
Test-file note, stated plainly: exactly ONE pre-existing test was touched
— 'the message-delta warning ties itself to the ... pages' — whose
assertion moved from 'overwritten page(s) carried' to 'discarded page(s)
carried' to track the deliberate message change. Its claim is unchanged
and nothing was loosened. Run against the untouched cd88d1de script in a
detached worktree, the shipped test file is 100 pass / 14 fail: that one
renamed pre-existing test plus 13 of the 20 new ones.
Receipts: 114 pass / 0 fail. Adjacent suites (conversation-parser x6,
markdown x3, extract-conversation-facts, doctor backlog) 301 pass / 0
fail. Over all 12 golden fixtures from the reference converter plus the
repo's sample, pages, stdout, stderr and exit code remain byte-identical
to cd88d1de.
NOT fixed here, escalated instead: a message body containing any of
gbrain's four timeline sentinels truncates the conversation at
markdown.ts before parse.ts runs. Reproduced against a pristine
`git archive upstream/master`: five sentinel forms each take a 4-turn
conversation to 2 parsed turns with the tail reclassified into the page's
timeline, at exit 0 with stdout reporting "4 message(s)"; two controls
stay clean. The re-grade graded this class NOT-A-DEFECT on a different
measurement and the packet puts its drop table out of scope, so this is
the RE's call, not a change smuggled in here.
* test(envelope importer): make the offset guard discriminate, cover the leap-second ceiling
The test named "an offset-bearing updated_at is compared as an instant, not
lexically" asserted only `kept === 'second'`. Array order keeps the second copy
too, so the assertion could not tell "compared as an instant" from "not compared
at all". Measured on the parent commit: dropping the offset alternation from
UPDATED_AT_SHAPE — so an offset-bearing value stops parsing and the pair falls
back to array order — leaves the file at 114 pass / 0 fail. The guard was
decorative against exactly the failure it is named for.
It now runs both offset signs and asserts the VERDICT STRING, which names the
branch that decided. The negative-offset case additionally makes the
offset-bearing copy win from array position 0, an outcome the fallback cannot
produce. Both cases fail under that mutant, and under the sign-arithmetic
mutant that the old test did already catch.
The `s > 60` leap-second ceiling had no test at all. Three cases now pin it.
Two of them pin the ROLL ACROSS THE MINUTE BOUNDARY rather than inside it:
23:59:60Z IS 2027-01-01T00:00:00.000Z, so a copy carrying each of those two
strings is a TIE and falls to array order, and 23:59:60Z outranks
23:59:59.999Z by one millisecond. Asserting only that 23:59:60Z beats
23:59:59Z would have pinned nothing but f(60) > f(59) — measured: making the
seconds field contribute milliseconds instead of seconds (`s * 1000` -> `s`)
destroys the date roll and still passes such a test at 117/0. The third case
is 23:59:61Z, the only value in the file rejected solely by the ceiling.
NOT CLOSED, so nobody reads this as a mutation-clean function: two sibling
guards in `updatedAtInstant` remain decorative on this commit. Deleting either
`if (h > 23 || mi > 59) return null` or the offset-range `if (oh > 23 || om >
59) return null` leaves the file green. They are the same class; they are not
this commit's scope.
Header prose only, no behavior change:
- The Output-layout paragraph said the filename is the (date, id) PAIR. Both
halves are slugged, so two DISTINCT ids reach one filename and the tiebreak
then discards one of two UNRELATED conversations while stderr calls the id
"not unique". Stated on the SLUG, not the raw id, because raw ids predict
nothing either way: `AbC-123`/`abc-123` differ at character 1 and collide,
while sixty `-` plus `a` and sixty `-` plus `b` agree on all of their first
60 characters and do not collide — `slug` strips leading/trailing `-`
before it truncates. Check 2 below already named this class; it REFUSES
what the tiebreak resolves.
- The header explains at length why check 2 refuses to resolve an id-less
`conv-1` across runs, then the code resolves that same name WITHIN one
envelope. Both sides are now stated together, with the asymmetry named:
the within-run path is loud and has `updated_at` on both copies, which is
exactly the evidence check 2 lacks. Also widened — any id that SLUGS to
`conv-N` collides with the positional name, and any id that slugs to EMPTY
enters that namespace while still recording a non-null id in frontmatter.
- "two different values" -> "two different INSTANTS": two different strings
can name one instant (`09:00:00Z` and `14:30:00+05:30`) and fall back.
- The collision block's own comment still glossed "mapping to the same
filename" as "carrying duplicate ids" — the equation the header now
corrects. Fixed in place.
Suite: 114 pass / 0 fail before, 118 pass / 0 fail after.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(envelope importer): remove a duplicate helper tsc rejects
Two functions were both named writeEnvelope. Bun hoists the later declaration,
so the earlier raw-envelope form was already dead at runtime - every call was
executing the fields form, which rebuilds the envelope from .conversations and
.meta and therefore handles both call shapes. All 118 tests passing under bun
is the proof; tsc's TS2393 was the only honest complaint.
Deleting the dead function keeps runtime behavior byte-identical. Removing it
also unmasked the two type mismatches TS2393 had hidden: envelopeWith declared
a return type of unknown for a value with a concrete shape, and writeEnvelope's
parameter did not admit the memvelope key that full-envelope call sites pass.
Both signatures now state what the values actually are.
bun run verify: 2/34 failing, identical to upstream/master's own baseline
(check:skill-brain-first and check:conversation-parser fail on the untouched
base as well).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(envelope importer): recognize our own pages after gbrain re-serializes them
The pre-file sweep proved existingPageIdentity() accepted only this importer's
own JSON.stringify output. gbrain rewrites pages it holds - export --dir, the
DB-only restore path, and put_page write-through all re-emit frontmatter
through gray-matter, which writes a UUID as a plain unquoted scalar and an
all-digit or boolean-looking id single-quoted. 16 of 17 id shapes, and 13 of
13 golden-fixture ids, came back in a form JSON.parse rejects - so the page
turned foreign, a later refresh refused the whole envelope at exit 2, and the
error's remedy advised deleting gbrain's own copy, provenance and body edits
included.
idScalar() now reads the three shapes a YAML round trip produces: our own
JSON, YAML single-quoted (doubled-quote escape), and plain scalars. YAML
null shapes and the empty string stay foreign, exactly as before - refusal
remains the answer for anything not confidently recognized.
Red-first: both new tests (plain-scalar UUID, single-quoted digits) fail on
the parent and pass at this commit. 120 pass / 0 fail; typecheck clean;
bun run verify at the repo baseline.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Sean Gearin <sean@virgilknows.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Three defects a second blind reviewer found that survived round 5. All three
reproduced against cf7a1616 before the fix and are pinned after it.
1. The fence scanner still falsely closed compliant descriptions. CommonMark 4.5
forbids a backtick in a BACKTICK fence's info string, so ```foo`bar is an
ordinary paragraph. The scanner opened a fence on it, never found a closer,
and stripped the body to EOF:
body = '```foo`bar' + three sentences of real prose + an image embed
before: intentWordCount 0, misses [missing_intent, missing_screenshot]
after: intentWordCount 52, misses []
Same class as round 5's closing-length bug and the same cost — opening a
block CommonMark would not open deletes the author's prose exactly the way
closing one late did. Tilde fences keep the permissive rule (4.5 restricts
backtick fences only).
2. The spend guard hashed less than the model consumes. hashInputs covered
title, modelBody, head.sha, exemption and policy ids — but the payload also
carries the changed-file list and the diff, and the workflow degrades the
diff to a marker line when the API 406s on a huge one. So a run that
classified with no diff cached a diff-blind verdict, and the next run — real
diff in hand, same title/body/sha — matched the hash and was served that
verdict permanently (measured: anthropicCalls 1 across both runs). The
assembled payload is now folded in as a fixed-width digest, inside the JSON
tuple where quoting still makes a forged boundary impossible. Round 5's
property is re-pinned: a policy fix past the model's body cap invalidates
even when the payload is byte-identical.
3. The workflow's ${{ }}-in-run scanner missed a legal commented block header.
`run: | # shell block` is valid YAML — js-yaml puts the following
interpolation in the script — but the header fell through to the
single-line branch, which captured `| # shell block` as the whole command
and never looked at the block body. The rule protecting against shell
injection reported clean over an interpolating workflow. The scanner now
handles a trailing comment on the header, and scans the comment text too
rather than leaving itself a hiding place. Guarded against js-yaml's actual
parse, not against the scanner's own opinion.
Also corrected one overstated claim in the file's security block: a BARE url in
a sanitized string still autolinks under GFM. That is a self-labelled link and
the deliberate stopping point — the escaping targets the masking characters so
it cannot forge `` or `[click to approve](…)` — but "no live
links" was too strong for what the code does.
Self-audit: all 13 interpolations into the sticky comment are either literals in
this file or pass sanitizeModelText/sanitizeList; the only unsanitized one
(titleCheck.reason) is a hardcoded literal and renders inside a code span. Every
rendered statement matches behavior the code performs. The four realistic-human
fixtures and the three new fence cases all PASS; empty, "fixes bug", the
unfilled template, ten words of lorem and a fenced-away screenshot all still
FAIL.
test/pr-gate-workflow.test.ts 158 -> 163 pass, 0 fail. Six mutations (the
CommonMark rule, the tilde exemption, the payload digest, the runGate wiring,
the scanner's comment group, the header-comment scan) each fail at least one
test. typecheck, actionlint, verify (34/34) green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A blind reviewer rejected this branch on false positives and the repro held up.
The gate's intent check deleted every list and blockquote LINE before counting
words, then measured what was left against an arbitrary 40. Real people write
their story as bullets. Measured on the four realistic-human descriptions from
the repro:
description before after floor 40 floor 20
own prose written as four bullets 0 55 CLOSE PASS
short non-native-English paragraph 38 38 CLOSE PASS
specific first-person bug report 34 34 CLOSE PASS
mostly a stack trace + a real explanation 28 27 CLOSE PASS
The file's own exemption rationale already says a check that is red on every
release is a check somebody disables inside a week. Red on every terse-but-
genuine contribution dies the same way, and it costs real contributors on the
way there — worse than the forgery risks rounds 1-3 chased.
- intentWordCount strips the list/quote MARKER and keeps the author's words.
Still stripped: fenced code, indented code (new — the other spelling of a
fence, guarded so it never eats a list continuation), headings, HTML
comments, raw HTML, bare URLs, inline code, link/image syntax, and the
template's own bold prompts. An untouched pull_request_template.md still
scores 0, pinned against the real file on disk.
- INTENT_MIN_WORDS 40 -> 20, documented as a floor against empty/boilerplate-
only descriptions, not a quality bar. CONTRIBUTING.md documents no word count
at all. Empty, "fixes bug", the unfilled template, 10 words of lorem and an
indented log paste all still fail.
- The fix-it comment no longer tells contributors to reopen a PR that was never
closed — the gate has no close call in it. It now says what actually happens:
editing the description re-runs the check (the workflow triggers on `edited`),
the PR stays open, a maintainer decides.
- sanitizeModelText backslash-escapes `[` and `]`. Escaping HTML killed the
<details> forgery, but Markdown needs no angle brackets: an image embed
renders a green "APPROVED" picture and a link renders a live phishing target
inside a close-lane comment. The file asserted nothing attacker-controlled
reaches Markdown unescaped; that is now true.
- A label-API failure during a NEUTRAL run no longer throws to exit 2 with no
comment. NEUTRAL has no verdict to record, so it logs, still posts, exits 0 —
and the comment stops claiming labels were cleared when they were not. The
verdict path keeps failing loudly on purpose (ordering note).
Four verbatim human descriptions are pinned as PASSING forever. Eight mutations
(floor, list markers, quote markers, indented code, markdown escaping, the
reopen copy, the NEUTRAL try/catch, the label claim) each fail at least one test.
test/pr-gate-workflow.test.ts 145 -> 158 pass, 0 fail. typecheck, actionlint,
verify (34/34) green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Four findings from a second independent blind review, all reproduced against
HEAD before the fix and all pinned by a mutation-tested case.
- sanitizeModelText escapes &, < and > after the existing comment/mention/
block-marker stripping, so raw HTML from any dynamic string (model output,
mechanical flag details built from PR filenames, neutralReason, the
policy-exempt note) renders as literal text. A <details open><summary>MERGE
LANE — approved</summary> string is a forged verdict inside a close-lane
comment; it now renders as <details.
- hasScreenshot requires a markdown image whose URL looks like a URL or path,
an <img> carrying a non-empty src=, or a bare paste URL, and strips HTML
comments before scanning. ``, `<img alt=proof>` and an image
hidden inside `<!-- -->` no longer clear it. Documented in the code as a
FLOOR against zero-effort submissions, not proof.
- stripCodeFences is a line scanner following the CommonMark rule that a
closing fence must be the same character and at least as long as the
opening one. The old backreference read ```` as a NEW opening fence and
stripped to EOF, so a compliant body documenting fence syntax lost its
intent paragraph and was closed. The scanner is also linear, retiring the
superlinear-backtracking hazard the 16KB cap was sized against.
- Labels are reconciled BEFORE the sticky comment carrying the cached state.
Written the other way, one transient label-API 500 left stale/missing/
duplicate labels forever: the rerun short-circuited on the persisted state
and returned success without repairing them.
Also:
- hashInputs covers what the run consumes — the truncated model body plus the
mechanical policy outcome — so an edit past the 6KB model cap no longer
mints a new hash and buys an identical paid call, while a policy fix landing
past that cap still invalidates the cached verdict.
- The test's YAML block-scalar scanner accepts indentation indicators in both
legal orders (`>2-` as well as `|-2`), with a guard-the-guard case proving
it sees a `run: >2-` block interpolating attacker-controlled text.
- A block at the top of the script states plainly that this gate is a triage
signal and a reviewer checklist, not an authorization boundary; one line of
the sticky comment footer says the same to the contributor.
145 pass / 0 fail (was 126), typecheck clean, actionlint clean, verify 34/34.
Blind review round 2 rejected the branch. Five findings, all fixed.
BLOCKING 1 — the verdict was forgeable through a PR filename.
renderComment wrote mechanical red-flag details raw, and two of them
interpolate filenames (adds_recipe, deletes_tests). git allows a newline
inside a filename and JS `[^/]` matches one, so RECIPE_RE's anchors were
decorative: a PR adding `src/core/ai/recipes/x\n## PR Gate — ...\ncc
@octocat\n<!-- ...state... -->\nz.ts` put that text verbatim into the bot's
comment — forged heading, live third-party mention, and on the NEUTRAL
render (which writes no state block of its own) parseState() returned the
ATTACKER's block, so the next run hit the spend guard and silently skipped
the verdict with no label and exit 0. Three layers:
(a) flag details go through sanitizeList, exactly like the model's
strings. Audited every other interpolation into the comment; the rest
are literals in the file, Number()-coerced, or already sanitized.
(b) every path regex spells its segment class [^/\n], not [^/] —
RECIPE_RE, SOURCE_EXT_RE and the test-path check.
(c) parseState reads the state block only off line 2 of a marker-leading
comment (where renderComment writes it) and STATE_RE is whole-line
anchored. A block anywhere else is somebody else's text.
BLOCKING 2 — no exemption, so every release PR was close-lane.
Measured: 40 of the last 40 merged PRs would be close-lane on
missing_screenshot, including every /ship release PR. A check that is red
on every release gets switched off within a week, and then it filters
nothing. The #3745 policy exists to filter INCOMING OUTSIDE CONTRIBUTIONS;
release automation cannot take a screenshot of itself. It is now waived for
OWNER/MEMBER/COLLABORATOR, bot authors and drafts — the usefulness verdict,
the title rule and every mechanical red flag still run, and the sticky
comment says the check was skipped. author_association / draft / user.type
are read from the pr.json the workflow already fetches: no new API call, one
source of truth. `draft` is the one author-settable input, so
ready_for_review joins the trigger list and the exemption is folded into the
spend-guard hash — the draft-era verdict cannot be reused after the flip.
Stated as a deliberate decision in both the workflow header and the script.
3 — DOWNGRADE_FLAG_IDS omitted deletes_tests, adds_symlink and
adds_node_modules, so a PR deleting test/e2e/engine-parity.test.ts kept
merge-lane and a green check on the strength of its prose. All three added.
The old test used deletes_tests as its example of a NON-downgrading flag;
rewritten to pin the stronger invariant instead — every id detectRedFlags
can emit is a downgrade trigger (derived from the detector, so a new flag
fails until it is classified on purpose), and the set is still an allowlist
(an unrecognized id changes nothing).
4 — FENCE_RE backtracks superlinearly on a hostile body: 65KB of backticks
(GitHub's max body length) measured 8.2s across the two policy scans on a
pull_request_target runner. stripCodeFences now caps the scan at 16KB —
same input, 0.40s. Tradeoff documented at the constant: the intent
paragraph and the screenshot both sit near the top in practice (the PR
template puts them in the first two sections, and the model payload already
caps the same body at 6KB), so a real contributor is not judged on a
truncated tail.
Verified: test/pr-gate-workflow.test.ts 126 pass / 0 fail (was 95),
typecheck clean, actionlint clean, check-privacy /
check-no-tracked-symlinks / check-progress-to-stdout /
check-bun-test-timeout / check-key-files-current-state all exit 0. Each new
pin was mutation-tested against the pre-fix behavior: all six mutants fail
the suite.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three fixes on the strict PR usefulness gate (#3698), plus the merge that
brings in the policy it enforces.
1. The mechanical policy check now outlives the model. The
ANTHROPIC_API_KEY guard used to sit above detectPolicyMisses, so a PR
with no intent paragraph and no screenshot got a green NEUTRAL skip
whenever the key was absent or Anthropic was down — "wait for a 500"
was a documented way past the one hard requirement. The #3745 branch
now sits above the key guard and the spend guard: a policy miss is
close-lane + the friendly fix-it comment + exit 1 with no API
dependency at all. A compliant PR that hits a missing key or a dead
API keeps the round-1 NEUTRAL behavior unchanged (loud comment,
::warning::, exit 0, stale gate:* labels cleared) — and the NEUTRAL
comment now says plainly that the *usefulness verdict* did not run,
while still reporting the title check and mechanical red flags it was
able to compute without a model.
2. Merged origin/master, which carries #3745's CONTRIBUTING.md section
and .github/pull_request_template.md. No conflicts: this branch never
touched VERSION / package.json / CHANGELOG.md, so master's 0.42.72.1
carried through untouched — the feature branch adds no version bump.
The test's inlined pull_request_template fallback (only needed while
the branch predated the merge) is gone; it now reads the real file, so
growing the template's own prose past the 40-word bar fails here
instead of silently letting an untouched template through. The
CONTRIBUTING_URL deep link is pinned against a GitHub-style slug of
every heading in the merged CONTRIBUTING.md, with the slugger itself
pinned so it cannot "pass" against an anchor GitHub never generates.
3. hashInputs joined its three fields with literal NUL bytes, which made
grep treat the whole of scripts/pr-gate.mjs as binary — any future
grep-based CI guard over that file would have matched nothing and
passed silently. Replaced with JSON.stringify of the tuple: still
unforgeable (each field is quoted and escaped), still stable by
construction, and printable. `grep -c hashInputs scripts/pr-gate.mjs`
now returns 2 instead of nothing. Existing sticky-comment state hashes
are invalidated once, costing one re-verdict per open PR.
Tests: 95 pass / 0 fail in test/pr-gate-workflow.test.ts. The no-API-key
policy-miss case was verified to fail against the pre-fix ordering.
Verified live against the Anthropic API: HTTP 200, strict JSON, all seven
required keys, merge-lane on a compliant fixture.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CONTRIBUTING.md (#3745) requires every PR to carry a paragraph the author
wrote themselves and a screenshot of gbrain actually in use. The gate now
checks both, mechanically, before it spends anything on review.
What it checks (no LLM, both exported for testing):
- hasScreenshot: markdown ``, a bare user-images.githubusercontent
or github.com/user-attachments/assets URL, or an <img> tag. Anything inside
a fenced code block does not count — pasting the syntax is not attaching
the picture.
- hasIntentParagraph: >= 40 words of prose left after stripping fenced code,
blockquotes, list items, headings, HTML comments, links and the PR
template's own boilerplate. Per-character scripts are tokenized per
character, so a paragraph written in Chinese counts as one.
The two lane consequences:
- missing_screenshot OR missing_intent forces close-lane from any recommended
lane (exit 1) and skips the model call entirely — closed without review is
the documented consequence, so there is nothing to spend a review on. The
sticky comment leads with what is missing, how to fix it, and the reopen
path; both misses are also recorded in the existing "Mechanical downgrades
applied" section.
- The model's new advisory intent_authenticity verdict forces
needs-maintainer (exit 0) when it reads "ai_generated", and never
close-lane on that signal alone. The comment says only that a maintainer
will read the paragraph personally; the model's reasoning is consumed and
never published at the contributor.
Rubric + strict-JSON schema gain intent_authenticity and
intent_authenticity_reason, with explicit instructions that rough grammar,
terseness and non-native English are evidence of a HUMAN and that "unclear"
is the answer whenever the evidence is not clear-cut.
test/pr-gate-workflow.test.ts: 63 -> 88 tests, all green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two independent blind reviews of #3698 (one APPROVE, one REJECT). Every
confirmed finding from the reject side, plus the cheap hardening both
reviews flagged.
BLOCKING
1. Sticky-comment hijack. upsertStickyComment adopted ANY comment containing
the marker, so a contributor could pre-post `<!-- gbrain-pr-gate -->`, have
the gate PATCH it, then edit it into a fake green verdict. isOwnComment()
now requires user.type === 'Bot' AND login === 'github-actions[bot]' AND
the body to START WITH the marker; anything else gets a fresh comment.
2. LLM output injected as raw Markdown. Model-produced reasons[] and
reviewer_checklist[] reached the comment unescaped — PR-body-driven
injection could forge headings, a second marker, and live @mentions.
sanitizeModelText()/sanitizeList() are now the single choke point in
renderComment(): HTML comments stripped, mentions zero-width-broken,
leading block markers removed, newlines collapsed, 300 chars per string,
8 entries per list, both caps self-marking.
3. Lane was purely model-decided. A well-written feature pitch could talk
itself into merge-lane. The model now RECOMMENDS; applyMechanicalDowngrades
forces merge-lane -> needs-maintainer on any of: workflow edits, a new
package.json dependency, a new src/core/ai/recipes/ provider file, new
KNOWN_CONFIG_KEYS entries, >40 changed files, >400 net source lines outside
test/, or a src/ change with no test file touched (#3665). The sticky
comment reports them under "Mechanical downgrades applied".
4. The test file did not pin what it claimed. Added: no gh pr checkout /
git fetch / refs/pull / pull/*/head in any spelling; exact permissions
key->value map plus a single-permissions-block assertion so no job-level
grant re-widens contents; the ${{ }}-in-run scanner now covers folded
(`run: >`) and chomped blocks, with a guard-the-guard test; and mocked
end-to-end runGate() runs for close-lane exit 1, marker hijack, sanitizer,
truncation, refusal routing, NEUTRAL label clearing, label swap, and the
spend guard.
5. Version-first title regex rejected the documented suffix form.
`v0.31.1.1-fixwave fix: ...` now passes. VERSION_AT_END_RE no longer
false-positives on `chore: bump zod (3.25.76)`: it fires only on a
v-prefixed or 4-segment trailing version, i.e. this project's own shape.
6. Refusal fail-open. stop_reason=refusal exhausted retries into a green
NEUTRAL — a deterministic way to dodge the red X. Refusal and
schema-invalid output now route to needs-maintainer with an explicit
note; only transport failure stays NEUTRAL. Refusal also short-circuits
the retry loop, since retrying a deterministic refusal only burns spend.
ALSO
7. persist-credentials: false on the checkout step.
8. Dropped pull-requests:write. Everything the script calls is the issues
API (comments, label create, label add/remove), so issues:write is the
only grant that is actually needed.
9. Spend guard for the edited/synchronize amplification: the LLM call is
skipped when sha256(title+body+head_sha) matches the hash recorded in the
previous sticky comment's state block, and the stored lane's exit code is
reused.
10. NEUTRAL runs now clear every gate:* label instead of leaving a stale
verdict behind.
Verified: bun test test/pr-gate-workflow.test.ts 63 pass / 0 fail,
bun run typecheck clean, actionlint clean, check-privacy /
check-no-tracked-symlinks / check-progress-to-stdout /
check-bun-test-timeout clean, plus a live Anthropic smoke of the exact
request shape (HTTP 200, valid strict JSON, injection attempt rejected).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Runs on pull_request_target against master. PR code is never checked out
or executed: metadata + a 120KB-capped diff come from the GitHub API only;
the base-repo checkout provides the rubric script. All interpolations are
env-bound; actions SHA-pinned; permissions limited to contents:read +
pull-requests:write + issues:write; per-PR concurrency with cancel.
scripts/pr-gate.mjs (no deps, fetch only) classifies via the strict
usefulness rubric (claude-sonnet-5, strict JSON schema, thinking disabled;
sampling params omitted — rejected on this model), posts one sticky
comment (<!-- gbrain-pr-gate -->), applies exactly one gate:* label, and
exits 1 only on close-lane. Missing ANTHROPIC_API_KEY or API failure after
2 retries NEUTRAL-skips loudly (comment + warning, exit 0). Mechanical
(no-LLM) checks: version-first title rule and diff red flags (>40 files,
node_modules, symlinks, workflow edits force needs-maintainer, new
package.json deps, deleted tests).
test/pr-gate-workflow.test.ts pins the security invariants and unit-tests
the exported title rule + red-flag detector (import side-effect guarded).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
runImport printed its informational lines ('Found N markdown files' and four
siblings) through console.log unconditionally, so `gbrain import <dir> --json`
emitted them ahead of the payload and stdout did not parse as JSON. A consumer
parsing stdout reads that as zero imports while its own bookkeeping records the
files as ingested, so the next run skips them permanently.
Route those five lines through an `info` helper that switches to stderr when
--json is set. The lines are relocated, not removed: human mode is byte-for-byte
unchanged, and the JSON payload itself is untouched. This is the contract
CLAUDE.md already states ('Stdout stays clean for data output (--json
payloads)') and that import.ts's own progress comment repeats.
scripts/check-progress-to-stdout.sh only greps for process.stdout.write('\\r…),
so this class was never guarded; the new spawn-level test pins it.
installHelper's "already current" fast path ran chmodSync(helperPath,
0o755) unconditionally, before the dryRun check further down — so
`gbrain sources harden --dry-run` mutated the helper's permissions even
though dry-run is documented as a pure preview. This is the 5th
instance of the #3594 class (see #3692 for the sibling fix to the
step-1 pull, same function).
Gate the chmod with `if (!dryRun)`, mirroring the guard pattern already
used in installLocalHook's own "already current" branch. Non-dry-run
behavior is unchanged.
Adds a regression pair to test/brain-repo-durability.serial.test.ts:
one asserting dry-run leaves a drifted-permission helper untouched,
and a control asserting the real run still restores the exec bit.
Two cycle phases still carried the label-vs-actual defect that
propose_takes shed in v0.42.62 (and that the #2805 review noted was worth
tracking so it isn't lost):
- grade_takes hardcoded 'claude-sonnet-4-6' into judge_model_id, the
evidence signature, and budget metering, while the default judge call
passed NO model hint and rode the gateway's chat_model. On any brain
with a non-default chat_model, the verdict cache and telemetry recorded
a model that never ran.
- calibration_profile persisted TIER_DEFAULTS.reasoning to model_id but
never passed a model to the patterns generator at all, so the recorded
model and the executed one were unrelated.
Fix, matching the propose_takes convention: each phase resolves ONE
string via explicit override > getChatModel(), and that string drives the
chat call, the cache key, and the stored id.
- grade_takes: the judge hint gets the FULL provider-prefixed string; the
stored judge_model_id and evidence signature keep the historical bare
tail. Stock installs are unchanged: getChatModel() defaults to
'anthropic:claude-sonnet-4-6', whose tail equals the old hardcoded
value, so no verdict cache invalidates. A genuinely different
chat_model invalidates, which is correct — the judge really changed.
- calibration_profile: getChatModel() is provider-prefixed, preserving
the #2451 contract, and its default IS the old TIER_DEFAULTS.reasoning
value — stock behavior unchanged. The generator now receives the same
resolved string that is persisted to model_id.
Tests: regression per phase pinning configured-chat-model routing (full
string to the call, bare tail to the cache key for grade_takes, full
string persisted for calibration); all existing suites pass unchanged,
pinning the no-change-on-stock property. 112 tests across the four
affected suites; typecheck clean.
Co-authored-by: Paolo Belcastro <p3ob7o@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The comment listed `gbrain integrity --dry-run` as a subcommand, but
runIntegrity dispatches on `check`/`auto` and parses --dry-run inside
the auto path, so that form exits 1 with "Unknown subcommand". The
runtime help already shows --dry-run indented under `auto [options]`;
only the source comment disagreed.
Addresses #3738.
The dead-link branch in cmdAuto called logSkip() (writing to
integrity.log.jsonl) but incremented bucketReview, the same counter fed by
the bare-tweet path's appendReview() calls (which write to
integrity-review.md). The printed "Review queue" summary line therefore
included findings that were never written to the review file, so
`gbrain integrity review` silently disagreed with `integrity auto`'s own
summary.
Give dead-link findings their own counter (bucketDeadLink) and print it as
a separate line. No change to bare-tweet bucketing or CLI surface.
retype mapping_rules' existing path_filter matches pages.source_path,
which is only populated for pages synced from a git repo. Pages ingested
via the put_page MCP tool (or any write path that doesn't go through
sync) have source_path = NULL, so path_filter can never disambiguate
them — a same-from_type retype rule targeting a slug prefix has no way
to address this class of page at all.
slug_filter adds an independent, orthogonal LIKE filter on pages.slug
(the field that's always populated), combinable with path_filter via AND
when both are given. Wired through the full call path: schema validation
(manifest-v1.ts), the RetypeRule interface + probeRule/applyRetypeRule
(retype.ts), and the pack-manifest → RetypeRule conversion in the
unify-types job handler (unify-types-handler.ts) — the last one matters
because a field only present in the zod schema but not carried through
that conversion would validate fine yet silently no-op at execution time.
- manifest-v1.ts: add optional slug_filter to RetypeMappingRuleSchema
- retype.ts: slug_filter on RetypeRule; probeRule + applyRetypeRule both
apply `AND slug LIKE $N` when present, independent of path_filter
- unify-types-handler.ts: carry rule.slug_filter through the
pack-mapping-rule → RetypeRule conversion
- tests: skips-outside-slug_filter, matches-despite-NULL-source_path
(the motivating case), path_filter+slug_filter combined via AND
320/320 existing schema-pack tests pass; 3 new tests added; tsc --noEmit
clean.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
Effective 2026-08-02. Every issue and PR must carry a paragraph the author
wrote themselves explaining why they are opening it, and a screenshot showing
gbrain actually in use in that situation. AI-generated or AI-polished intent
text is not accepted; AI assistance for the code is still fine. Missing either
one means closed without review, reopenable once added.
Stated in CONTRIBUTING.md and pre-filled in both issue templates plus a new
pull request template so the fields are in front of the author.
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Registering an OAuth client with --bound-slug-prefixes now makes the write boundary real: writes outside the bound prefixes are refused on every op that can name a page, and ops that write by something other than a slug are refused outright rather than left unfenced. Deny-by-default at dispatch, so a write op added later is refused to bound clients until it is explicitly fenced.
Adds docs/integrations/qm-harness.md (gbrain as the company brain for a qm deployment) with a roster-driven provisioning script and deployment templates, plus a Known limitations section stating plainly that this is a write boundary and not a privacy boundary.
Five review rounds, including three clean-room passes by codex gpt-5.6-sol and Claude Fable 5 against an instruction-stripped tree.
The first Release run (30698650484) failed before building anything: the
build job re-ran the entire unit suite serially on the release runner —
a different environment from the sharded Test workflow that had already
gated the exact SHA at merge — and died on ambient-env tests. The review
of #3573 predicted this ('a flaky test now blocks the release'). The
build job now compiles and smoke-tests the binary (--version must match
the VERSION file), which validates the actual artifact — something the
test suite never did. The Test workflow remains the code gate.
Review hardening from the #3573 gatekeeper pass: a ${{ }} inside run: is
shell injection by construction; bind through env instead. Not
attacker-reachable today (VERSION comes from master), correct anyway.
CI caught what the composite review's watermark 'fix' actually did: the
stamp path clamps links_extracted_at up to the watermark
(GREATEST(updated_at, versionTs), extract.ts:1835), so a future watermark
stamps every page at 2026-08-02 and masks any concurrent edit until then —
the exact race D4 guards — and makes the doctor lag check warn on fresh
brains. The limitation the future date tried to cover (stamps written by
pre-wave code after the watermark) is inherent: no fixed watermark covers
code that keeps running past it. Documented instead.
Two cross-PR findings from the wave-i hostile review (Codex xhigh):
1. The #3560-ungated bare-path pass scanned inside [[...]] spans, so a
dir-qualified wikilink's lowercase prefix (its parent page) became a
spurious 'markdown' edge whenever the parent existed. Wikilink spans
are now masked with equal-length blanks before pass 2; discriminating
test added (fails without the mask).
2. LINK_EXTRACTOR_VERSION_TS was midnight today with a strict-< staleness
predicate, so same-day stamps from pre-wave code read as fresh and
never re-extracted. Bumped to 2026-08-02T00:00:00Z.
Plus two comment corrections from both reviewers: upgrade.ts's X1 hook
rationale (stale after #3085) and PGLiteEngine.transaction's tx-engine
db-proxy hazard for #3613's searchVector wrapper.
In-wave interaction with #3560: extractPageLinks now also emits raw-literal and bare-path candidates that downstream existence checks drop; the tests' whole-list toEqual predates that. Filter to linkSource='wikilink-resolved' — the PR's resolution + no-cross-dir-leak claims are still fully pinned.
Master's #1072 test asserted bare qwen3-embedding@1024 threads dimensions:1024;
wave PR #3699 suppresses the param when the request equals the native width
(1024 for the bare id), so the same input now correctly returns undefined —
pinned by dims-qwen3-native.test.ts. Moved this test to 512 to keep its actual
intent (bare id recognized as Matryoshka-capable) without contradicting the
suppression. Third composition defect of the wave; caught by CI shard 8.
#3574 flipped the unify-types worker default to dry-run (jobs.ts:2221,
apply: data.apply ?? false) and updated the architecture docs, but three
agent-facing surfaces still presented the bare submit as the Apply step:
skills/schema-unify/SKILL.md 'Phase 3: Apply', skills/conventions/
schema-evolution.md, and README.md. Because #3545 also edited SKILL.md in
this wave, each PR looked self-consistent in isolation — only the composed
branch shipped a playbook whose apply step silently retypes nothing and
never flips the active pack. Skills distribute downstream via the skillpack,
so this would have propagated. Found by an independent cross-PR review pass.
#3691's regression test used llama-server as an 'unknown provider' example.
#3541 (same wave) prices ollama/llama-server at $0 via FREE_LOCAL_CHAT_PROVIDERS,
so that example is now priceable and the assertion inverted. Swapped in groq —
the paid-but-unpriced case #3691's own description cites — and added the
positive assertion that free local providers keep their cap enforced.
Non-string values interpolated into the template literal printed
'[object Object]' (e.g. provider_base_urls). Objects now render via
JSON.stringify; objects under a sensitive key redact to '***' like
their string counterparts.
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
The real plugin-load e2e test dynamically imports 'openclaw/plugin-sdk'.
TypeScript still resolves and type-checks that bare specifier via upward
node_modules resolution, so 'bunx tsc --noEmit' on a clean checkout could
fail (TS2339 on sdk.registerContextEngine) or pass depending on whichever
undeclared openclaw package happened to exist in an ancestor directory.
The existing @ts-ignore only covered the import line, not the property
access on the following line.
Cast the awaited import to a local structural interface declaring the
one member the test uses (registerContextEngine, optional). TypeScript
never consults the ambient module's types for the property access, so
typecheck output is identical regardless of ancestor node_modules state.
The @ts-ignore stays on the import statement itself and stays @ts-ignore
(not @ts-expect-error) because whether TS2307 fires there is itself
ambient-dependent. Runtime behavior is unchanged: the cast erases at
compile time and the export's presence is still verified at runtime.
Fixes#2729
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
The phase was calling runSchemaSuggestPhase(engine, { dryRun }) with no
sourceId, so it silently fell back to 'default' on every source's dream
cycle -- same bug class as upstream #1586 (synthesize) and #2666
(patterns/synthesize), just an undiscovered instance for this phase.
Confirmed live: schema-events audit log shows only source=default across
41 entries this week despite calendar/mail/mem/social cycles all running
the phase.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain. 19 of 62 PRs cleared that bar.
Applying a schema mutation batch was not atomic: a failure partway through left earlier mutations permanently written. Reproduced on disk — a failure at index 2 left mutation 0 applied with no way to tell from the pack's state that it was half-done. The fix validates the whole batch first and writes once, which makes partial application impossible by construction rather than by careful ordering.
Verified before merge: the failure was reproduced by injecting one rather than reasoning about it; the PR's own tests fail when the fix is reverted; typecheck clean; MERGEABLE/CLEAN at 22/22 on the current base after batches 1-4 landed.
Sequenced last deliberately — it collides with #3531 on docs/architecture/KEY_FILES.md and with #3667 on src/core/operations.ts, both of which landed earlier today.
Known gap, recorded rather than hidden: lock contention under concurrent writers was reasoned about, not stress-tested.
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain. 19 of 62 PRs cleared that bar.
voyage-4-large was billed at the voyage-3-large rate — $0.18 against a published $0.12 — so every cost estimate using it was wrong by 50%. Corrected in the canonical table only, per CLAUDE.md's rule that every other pricing table is a derived view, and the drift guard passes. Rate checked against the live vendor page.
Verified before merge: the PR's own tests fail when the production change is reverted; typecheck clean; MERGEABLE/CLEAN with 22/22 checks on the current base after batches 1-3 landed.
Known gap, recorded rather than hidden: this PR previously failed the JSONB parity guard on a 32-commit-stale base. I rebased it onto current master and re-ran rather than accepting 'flaky' — the guard passes on the real base, 22/22 green.
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain. 19 of 62 PRs cleared that bar.
Mechanically applies merged #3191's `dropInvalidConcurrentIndex` to the 10 remaining historical migrations that still had the broken DO-block form. Migration ordering and numbering are untouched — this only changes how each guards its own index creation.
Verified before merge: the PR's own tests fail when the production change is reverted; typecheck clean; MERGEABLE/CLEAN with 22/22 checks on the current base after batches 1-3 landed.
Known gap, recorded rather than hidden: verified by sequence inspection and the migration suite rather than by replaying all 120 migrations against every engine.
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain. 19 of 62 PRs cleared that bar.
The npm package named `gbrain` is an unrelated squatted package, so `npm install gbrain` gives users something that is not this project. Adds doctor detection that classifies real checkouts correctly, fails open, and is try/catch'd throughout. Classification rests on the bin-shape marker since this repo has no `repository` field — verified e2e.
Verified before merge: the PR's own tests fail when the production change is reverted; typecheck clean; MERGEABLE/CLEAN with 22/22 checks on the current base after batches 1-3 landed.
Known gap, recorded rather than hidden: the remediation commands were not executed against a real global install, and the Windows `which -a` path is unexercised.
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain. 19 of 62 PRs cleared that bar.
Code files containing raw NUL bytes hard-failed UTF-8 encoding on import, so they silently never indexed. Sanitized at the single choke point both callers route through, with offsets kept in one coordinate space, and exercised end-to-end on a real engine.
Verified before merge: the PR's own tests fail when the production change is reverted; typecheck clean; MERGEABLE/CLEAN with 22/22 checks on the current base after batches 1-3 landed.
Known gap, recorded rather than hidden: follow-up to file: reindex-code hash ping-pongs on NUL-containing files — reproduced, bounded, and causes no data loss.
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain. 19 of 62 PRs cleared that bar.
The google embedding recipe declared no batch caps, so it rode the no-cap fast path with error backstops shaped for Voyage and OpenAI. Caps verified by behavioral probe — 40 texts split into 3 sub-batches matching the declared math. Sequenced after #3531, which touched the same recipe file.
Verified before merge: the PR's own tests fail when the production change is reverted; typecheck clean; MERGEABLE/CLEAN with 22/22 checks on the current base after batches 1-3 landed.
Known gap, recorded rather than hidden: Gemini's actual 20k limit was taken from vendor docs rather than a live call; being wrong in either direction is bounded by the cap itself.
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain. 19 of 62 PRs cleared that bar.
Full-cycle maintenance jobs were stamped with the outer 600s timeout instead of the 30-minute handler anchor — a regression from #3338 that killed long cycles mid-run. Fixed with a named `fullCycleTimeoutMs` derived from the handler anchors, which now fail loudly rather than silently defaulting; reverting fails 3 of 8 tests.
Verified before merge: the PR's own tests fail when the production change is reverted; typecheck clean; MERGEABLE/CLEAN on the current base after batches 1 and 2 landed, not a stale one.
Known gap, recorded rather than hidden: the '38 dead cycles in 24h' figure from the description was not reproduced; the stamp arithmetic was verified by code inspection.
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain. 19 of 62 PRs cleared that bar.
The docs and recipe pinned `X_BEARER_TOKEN` while the resolver only ever read `X_API_BEARER_TOKEN` — so no single name worked and the integration could not be configured by following its own documentation. Renamed the dead documented side; reverting fails exactly 2 of the 3 new tests.
Verified before merge: the PR's own tests fail when the production change is reverted; typecheck clean; MERGEABLE/CLEAN on the current base after batches 1 and 2 landed, not a stale one.
Known gap, recorded rather than hidden: no live X API call was made.
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain. 19 of 62 PRs cleared that bar.
The back-link validator compared bare slugs, so a same-slug page in another source masked a genuinely missing reverse edge — silent under-reporting in exactly the multi-source setup where it matters. Now keyed on the full 4-tuple, per the `(source_id, slug)` uniqueness invariant. Verified on real Docker Postgres with 28/28 parity, which the PR itself had skipped.
Verified before merge: the PR's own tests fail when the production change is reverted; typecheck clean; MERGEABLE/CLEAN on the current base after batches 1 and 2 landed, not a stale one.
Known gap, recorded rather than hidden: remote MCP serialization of the additive Link fields is untested; the fields are additive JSON.
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain. 19 of 62 PRs cleared that bar.
`integrations show` printed `[missing]` for config-plane keys that the runtime gateway resolves perfectly well — so the status display disagreed with reality and sent people hunting for a problem that did not exist. Fixed with a single `secretEnv()` helper at all four read sites, preserving precedence. The spawn environment is deliberately left unchanged, which is the correct posture. Sequenced after #3531, which refactored the `buildGatewayConfig` internals this consumes.
Verified before merge: the PR's own tests fail when the production change is reverted; typecheck clean; MERGEABLE/CLEAN on the current base after batches 1 and 2 landed, not a stale one.
Known gap, recorded rather than hidden: the full-suite env-mutation interaction was not run locally; CI shards are green.
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain. 19 of 62 PRs cleared that bar.
One-character regex fix with a real user-visible effect: "what do **I** know about X" was classified as a general query while the you/we phrasings were correctly classified as entity queries. The new alternation is a strict superset, so no previously-matching phrasing changes, and stubbing the old regex back fails at the exact assertion. Closes verified issue #3615.
Verified before merge: the PR's own tests fail when the production change is reverted; typecheck clean; MERGEABLE/CLEAN on the current base after batch 1 landed, not a stale one.
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain. 19 of 62 PRs cleared that bar.
The shared AbortSignal arrived already aborted, which made the 2s embed floor dead code and silently degraded hybrid search to keyword-only — users got results that looked complete and were not. Fixed with a fresh AbortSignal.timeout(remaining) at the single shared seam; stubbing the old behavior back fails exactly the new test. This also closes verified issue #2028.
Verified before merge: the PR's own tests fail when the production change is reverted; typecheck clean; MERGEABLE/CLEAN on the current base after batch 1 landed, not a stale one.
Known gap, recorded rather than hidden: the DATABASE_URL e2e claims in the description were not re-run, though no SQL is touched.
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain. 19 of 62 PRs cleared that bar.
Transcripts-first work ordering permanently starved the doctor-visible page backlog whenever the budget cap bit — the pages never got reached. Page-first interleave at the single merge point, with spend proven order-independent, and stubbing the old ordering back fails 4 of 5 tests. Landing first among the extract-atoms.ts PRs, so #3691 and #3654 rebase onto it.
Verified before merge: the PR's own tests fail when the production change is reverted; typecheck clean; MERGEABLE/CLEAN on the current base after batch 1 landed, not a stale one.
Known gap, recorded rather than hidden: no real-LLM budget run; the identical error path was driven synthetically.
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain. 19 of 62 PRs cleared that bar.
dashscope and google keys now fold into the gateway env — verified live end-to-end — and the retired gemini-1.5-pro default is swept from 9 files. Reverting the change fails 11 of 98 tests at fixed seams, and the budget-cap claim in the description reproduced. Landing first in the gateway/config-key seam, so #3648 rebases onto it.
Verified before merge: the PR's own tests fail when the production change is reverted; typecheck clean; MERGEABLE/CLEAN on the current base after batch 1 landed, not a stale one.
Known gap, recorded rather than hidden: no live provider call was made; the gemini retirement was taken from issue history rather than a vendor check. Minor follow-up to file: deriveEnvKey('google_api_key') yields a dead GOOGLE_API_KEY in the minion shell-inherit path.
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain.
19,497 false `works_at` edges were being asserted from bare people/→companies/ directory adjacency. This is the exact fix prescribed when #3495 was closed — bare `mentions` plus an extractor version bump — and stubbing the old behavior back fails the new test. Retroactive cleanup of already-written rows is explicitly out of scope; filing that follow-up.
Verified before merge: the PR's own tests fail when the production change is reverted (11 of the previous 32 PRs failed exactly there — one had 7 of 8 new tests passing on master); typecheck clean; MERGEABLE/CLEAN with 22/22 checks green on the current base, not a stale one.
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain.
doctor's probe-health "Latest" tail-picked the oldest cross-week event instead of the newest. Chronological sort at the reader seam, matching the writer's own documented contract, with all 14 consumers audited. Follow-up to file: `doctor.ts:1017` `self_upgrade_health` has the identical bug class.
Verified before merge: the PR's own tests fail when the production change is reverted (11 of the previous 32 PRs failed exactly there — one had 7 of 8 new tests passing on master); typecheck clean; MERGEABLE/CLEAN with 22/22 checks green on the current base, not a stale one.
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain.
Both #3013 defects reproduced live: `sync --dry-run --timeout` reported "unsupported command", and a bare `--timeout 60` was mis-scaled to 60ms. Fixed with a per-command dispatch gate plus timeout handback to its two owners, with every `cliOpts.timeoutMs` consumer audited. Landing first in the cli.ts and sync.ts clusters.
Verified before merge: the PR's own tests fail when the production change is reverted (11 of the previous 32 PRs failed exactly there — one had 7 of 8 new tests passing on master); typecheck clean; MERGEABLE/CLEAN with 22/22 checks green on the current base, not a stale one.
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain.
`GBRAIN_FTS_LANGUAGE` was absent from the query-cache key, so a language switch served stale pre-switch rows. The hash now folds it (v14→15) with all five pin sites updated; reverting the fix fails 4 of 15 tests at the exact claimed step. Landing first in the knobs_hash cluster — the constant is single-writer, so #3617 rebases onto this and takes 16.
Verified before merge: the PR's own tests fail when the production change is reverted (11 of the previous 32 PRs failed exactly there — one had 7 of 8 new tests passing on master); typecheck clean; MERGEABLE/CLEAN with 22/22 checks green on the current base, not a stale one.
Adversarial review: survived two independent refuters — the only PR of 32 reviewed this way to do so.
The bug: `gbrain reindex-frontmatter` and `gbrain backfill <kind>` were 100% dead on PGLite. cli.ts takes the data-dir lock, the command modules built a second engine on the same dir, and acquireLock never reaps a live PID — 30s timeout, exit 1, with the error naming the waiting process itself as the holder. Reproduced on the parent commit at 33.2s; passes in 5.0s with the fix. Root-cause fix at the dispatch layer, not a softening of the lock, and the sibling census confirmed these were the only two affected callers.
Postgres path verified before merge (it was the review's one open gap, since the bug is PGLite-only and all verification had gone there while the change itself is connection-teardown ownership). Against real Postgres 16 + pgvector: reindex-frontmatter and all three registered backfills exit 0 with zero residual connections, zero advisory locks, and zero cycle-lock rows — byte-identical output and identical teardown to master on the same database, confirming the change is behavior-neutral there.
Merged tree re-verified after rebase: typecheck clean, pglite-lock + reindex-frontmatter 16 pass, llms bundle fresh, 23/23 CI green.
Export AdminSseResponse, HttpServerLifecycle, and SignalSource from
serve-http.ts so test fakes can reference them. Cast structural fakes
through `as unknown as T` where the fake return types (EventEmitter,
plain object) cannot structurally match the full Node/Express originals.
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Mastra Code (anthropic/claude-opus-4-6) <noreply@mastra.ai>
* docs: design engine dynamic-import reconciliation
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(engine): reconcile dynamic import hardening
Co-Authored-By: Claude <noreply@anthropic.com>
* test(engine): guard dynamic import policy
* docs: plan engine dynamic-import reconciliation
Record the approved TDD sequence for selective engine-path hardening,
repository guard wiring, documentation, and local verification. Preserve
the no-version-bump and no-publication boundaries for the remaining work.
Co-Authored-By: Claude <noreply@anthropic.com>
* docs(engine): record static import invariant
* fix(engine): parse block comments in import guard
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(engine): parse dynamic imports with TypeScript
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(engine): close import guard bypasses
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(engine): close parser guard edge cases
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(engine): aggregate parser diagnostics
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(engine): bound dynamic import marker directive
Require the line-level opt-out marker to be standalone inside real comment trivia so negated or incidental longer tokens cannot authorize an import. Preserve the existing general marked-line contract and pin it with focused regression coverage.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(engine): close Unicode marker boundary bypasses
Treat Unicode identifier continuations as marker-token characters and inspect adjacent text by code point so supplementary-plane characters cannot turn longer comment tokens into approvals.\n\nCo-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
Review follow-up to #3549.
An envelope is a third-party file, so interpolating source_provider raw let a
provider string carrying a newline close the scalar and inject arbitrary
frontmatter keys. title: on the line above was already quoted; source: now
matches it.
memvelope_conversation_id emitted the literal string undefined when a
conversation carried no id, which asserts a value rather than reporting
absence: every id-less conversation claims the same id, so anything grouping
or deduping on that key merges unrelated pages. The key is now omitted.
Filename and frontmatter read one hasId predicate so they cannot disagree
about whether an id exists. Tests add both cases; the injection case parses
emitted frontmatter with js-yaml rather than substring-matching it.
One Markdown page per conversation from an envelope-v0 file (format spec:
github.com/memvelope/memvelope), written into a directory gbrain sync
ingests. Zero dependencies, deterministic, no network; does not call gbrain.
Filenames are date + conversation id (collision-proof natural key; duplicate
ids overwrite their own file and warn on stderr). Frontmatter carries
type: conversation, source provider, conversation id, and origin. Bodies keep
message-id citations per speaker turn.
Ships as script + test + fixture only; usage and verification steps live in
the script header.
bun ignores bunfig.toml's timeout key, and beforeAll/beforeEach hooks do
NOT inherit a test's third-arg timeout — a bare `bun test` gives every
hook the 5000ms default even when all tests in the file declare 30s+.
Measured on bun 1.3.14: a 6s hook dies at ~5001ms with the signature
`(unnamed) [5001ms] ... hook timed out` (the #3545 jsonb-parity CI
failure); both `beforeAll(fn, ms)` and the CLI `--timeout` flag are
enforced hook budgets (kills observed at exactly the configured ms).
Fixes:
- e2e.yml (jsonb-parity, tier1, tier2) and release.yml ran bare
`bun test`; they now pass --timeout=60000 like every scripts/ runner.
- test/e2e/jsonb-roundtrip.test.ts (the #2339 double-encode guard, which
only real Postgres can surface) additionally carries per-hook 60s
budgets so a bare local run can't flake either — same pattern as its
sibling op-checkpoint-jsonb-parity.test.ts.
- scripts/check-bun-test-timeout.sh: CI guard (run from test.yml's
verify job) failing any future bare `bun test` in workflows/scripts.
- scripts/run-e2e.sh: correct the comment claiming --timeout is
per-test-only (it covers hooks; the outer gtimeout exists for
sync-blocking WASM hangs where no timer can fire).
Proof: with Postgres paused for 6s during setupDB's connect, the
unfixed file fails at 5001.81ms with the exact CI signature; the fixed
file passes the identical condition (5 pass, 6.57s). 396 slow
before-hooks across 362 test files lack per-hook budgets; all of them
run through --timeout-passing invocations after this change, enforced
by the new guard.
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Closes the paths #3382 left open (its author said it narrowed the issue
rather than closing it):
1. checkCycleFreshness iterates EVERY local_path source, so an install
that nightly-dreams one vault via --dir showed a permanent FAIL for
every other federated source — and for any source added minutes ago.
'Never completed a full cycle' is now a WARN with the dream/autopilot
hint; a source that HAS cycled and then went stale still escalates
through the 6h warn / 24h fail thresholds (the regression signal the
check exists for). This is the reporter's actual case: the permanent
red eroded doctor's signal until real staleness hid inside it.
2. resolveSourceForDir's exact-match lookup had no archived filter and
no ORDER BY, so an archived (or duplicate) alias of the same path
could shadow the active source; dream's archived guard then refused
the stamp and the ACTIVE source stayed unstamped forever. The lookup
now excludes archived rows and orders deterministically, matching
the canonical-path fallback's posture. The fallback's fail-closed
ambiguity handling is deliberately unchanged.
3. #3382's own regression test (ii) was environment-sensitive: it
assumed unsetting OPENAI_API_KEY/ANTHROPIC_API_KEY makes the embed
phase fail, which is false wherever another embedding provider
resolves (the cycle then reports 'clean' and the test flips). It now
fails the sync phase against a vanished checkout — deterministic on
every machine, same property pinned (a genuinely failing enabled
phase must prevent the stamp).
New pins fail on unmodified master and pass here: never-cycled→warn
(x2, doctor) and the archived-alias shadow (dream --dir stamp).
Fixes#2540
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
master's rename loop swallows updateSlug failures with an empty catch
("treat as add"), and updateSlug returns void — so a zero-row UPDATE
(old slug absent) and a thrown collision are both invisible. Either way
the run falls through to importFile at the new path while the old row
stays behind live: slug occupied, 0 chunks after the next embed pass,
page count unchanged. A rename that didn't rename, with no trace.
The fix reconciles the duplicate:
- updateSlug returns the number of rows moved in both engines (a
zero-row UPDATE does not throw; the count is the only way to see it).
- When the cheap rename didn't move a row AND the destination
demonstrably materialized — imported, or an errorless skip AT the new
slug (NOT an identity-dedup skip against the old row, which would mean
nothing landed and deleting the old row would destroy the only copy) —
the stale row is located positively by source_path = from and deleted.
No source_path match → nothing is deleted (code-strategy imports don't
populate source_path and fall back safely to leaving the row).
- A failed reconcile delete records a <rename:…> sentinel: the failure
gate hard-blocks the bookmark, the auto-skip valve can never
chronic-skip it (which would bank the duplicate permanently after a
multi-run outage), and the rename is not checkpointed — the next run
retries the same diff and clears the sentinel on convergence.
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
* fix(search): stop boosting compiled_truth at default detail (#3430)
COMPILED_TRUTH_BOOST = 2.0 is applied AFTER RRF normalization, and RRF's whole
dynamic range over a 100-deep pool is 1/60 -> 1/160 (a factor of 2.67). So a
2.0x multiplier consumes roughly three quarters of the range: break-even is
`2/(60+r) >= 1/60`, i.e. r <= 60, which means ANY boosted chunk inside the
first 60 ranks outranks an unboosted rank-1 chunk. That is a categorical
filter, not a tilt.
Measured against master's own rrfFusion, with the correct answer in a
fenced_code chunk at vector rank 0:
compiled_truth chunks in pool | final rank | in top-20
10 | 10 | yes
20 | 20 | NO
40 | 40 | NO
80 | 59 | NO
With the boost off the answer stays at rank 0 in every case.
The gate was spelled `detail !== 'high'` -- written as though `high` were the
special case. The documented contract in src/core/operations.ts is
"low (compiled truth only), medium (default, all with dedup), high (all
chunks)", which makes LOW the special one: `low` already restricts to
compiled_truth, so a boost there is a no-op among equals, while `medium` and
`high` are both meant to see everything. So the default detail was silently
compiled-truth-only, contradicting the op's own description.
Three changes:
1. The three fusion call sites now route through a named predicate,
`shouldBoostCompiledTruth(detail)`, returning true only for 'low'.
Extracted rather than left inline precisely because an inline expression is
only reachable through a full hybridSearch round trip -- which is why the
inversion went unnoticed. The predicate is directly unit-testable.
2. KNOBS_HASH_VERSION 13 -> 14. Results are cached AFTER fusion, so rows
ranked under the old semantics would otherwise be served under the new ones
for the whole TTL (3600s default). One-time miss spike on upgrade.
3. test/search-compiled-truth-boost-scope.test.ts pins both the mapping and
the arithmetic, and documents the displacement it prevents.
Verified the tests discriminate: stubbing the OLD predicate body into master
(so the failure is behavioral rather than a missing export) gives 4 fail /
3 pass; with the fix, 7 pass. typecheck clean, verify 32/32, and 144 pass /
0 fail across the search + fusion + cache suites.
* fix(test): update the three remaining KNOBS_HASH_VERSION pins to 14 (#3430)
Missed in the first pass because I ran a targeted set of test files instead of
the full suite. CI shards 3, 8 and 10 caught them:
test/search/knobs-hash-reranker.test.ts:67
test/cross-modal-phase1.test.ts:139,149
test/search-alias-resolved-boost.test.ts:93
Each carries the running history of why the version moved, so each gets the
13→14 rationale appended rather than just the number swapped. No pins at 13
remain anywhere in test/.
---------
Co-authored-by: Garry Tan <garrytan@gmail.com>
getStats() has excluded soft-deleted pages since v0.26.5, but getHealth()
kept counting raw pages rows: page_count, the islanded/orphan scan, the
entity_pages CTE (link/timeline coverage denominators), and most_connected
all included deleted pages, so brain_score never moved when a user
soft-deleted pages. Repro: 50 pages, soft-delete 40 -> getStats 10 vs
getHealth 50, orphan_pages 50, brain_score byte-identical.
Fix: every page-scoped count in getHealth now filters deleted_at IS NULL,
identically in both engines (engine-parity SQL shapes match).
Deliberate boundary: chunk/link storage counts (embed_coverage,
missing_embeddings, link_count, dead_links) stay raw until the purge phase
runs — matching getStats' documented posture — and destructive-removal
counts (#2235) deliberately keep counting all rows. stale_pages already
filtered via buildStalePagesWhere.
Test: test/health-soft-delete.test.ts — 3 of 4 tests fail behaviorally on
unmodified master (page_count 10 vs 4, orphan_pages 8 vs 0, link_coverage
0.5 vs 1), all pass with the fix.
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* fix(test): make resetGateway restore the test baseline instead of unconfiguring (#3554)
bunfig.toml's legacy-embedding-preload pins the gateway once at process
start to openai:text-embedding-3-large @ 1536, but resetGateway() wiped
that pin to _config = null. The next test file's beforeAll engine-connect
then reconfigured from the SHIPPED default (zembed-1 @ 1280) before the
preload's per-test beforeEach could restore anything, and every 1536-d
fixture in that file failed with `expected 1280 dimensions, not 1536`.
Which file pairs collided depended on shard bin-packing, so adding ANY
test file reshuffled the mines (this is what blocks #3545).
Fix: the preload registers its config as a reset baseline via a new
test-only seam (__setGatewayResetBaselineForTests); resetGateway() clears
all module state as before, then re-applies the baseline. All 93 existing
resetGateway() call sites get the correct behavior with zero edits.
Production is untouched: nothing in src/ calls resetGateway() or the
setter, so the baseline is never registered outside tests and
resetGateway() still fully unconfigures there.
Five tests genuinely need an unconfigured gateway (no_gateway_config
diagnosis, isAvailable=false, the #2590 cold-gateway path, the registry
builtin-default tier); they switch to the new __unconfigureGatewayForTests.
Two files' hand-rolled "restore the legacy pin in afterAll/finally"
workarounds for this exact bug are now redundant and simplified away.
Guard test (test/ai/gateway-reset-baseline.test.ts) pins the contract:
1536/openai immediately after resetGateway(), transports still cleared,
hard-unconfigure still available.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(test): restore preload's GBRAIN_AUDIT_DIR instead of deleting it (#3554 sibling)
Same bug class as the gateway fix in this PR: state set once by a bunfig
preload (audit-dir-preload's scratch GBRAIN_AUDIT_DIR), wiped by one
file's cleanup, blast radius decided by shard bin-packing. In shard 6,
test/minions-shell.test.ts (position 12) unconditionally deleted the var
in afterAll; test/audit/audit-dir-preload.test.ts (position 93) then
found it undefined and failed 3 tests — and every file in between wrote
audit fixtures toward the operator's real ~/.gbrain/audit/.
Fix: capture the prior value at file load and conditionally restore it,
the same inline save/restore pattern 11 sibling files already use.
test/e2e/skill-brain-first.test.ts had the identical unconditional
delete in afterEach; fixed the same way. Sweep of every
`delete process.env.GBRAIN_AUDIT_DIR` in test/ confirms all remaining
sites are conditional restores.
Ordered-pair proof (minions-shell.test.ts then
audit/audit-dir-preload.test.ts, one process): 3 fail on master,
43/43 pass with this fix.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* fix(cli): stop parseOpArgs hanging on a non-TTY stdin with no input (#3513)
parseOpArgs read stdin for stdin-capable ops via readFileSync(0),
assuming non-TTY implies piped content. In a non-TTY with no piped
input — a CI step, a cron job, an agent harness that inherits a non-TTY
stdin without ever writing to it — that call never returns.
The stdin fill moves out of parseOpArgs into an async applyStdinParam
with a bounded read (readStdinBounded), called by the op dispatch right
after arg parsing:
- TTY: skipped, as before.
- Regular file / /dev/null (fstat says not a pipe/socket): readFileSync
returns without blocking — `gbrain put x < file` and `< /dev/null`
behave exactly as before (empty-but-readable still yields '').
- FIFO/socket: stream-read with a deadline on the FIRST byte only
(default 5000ms, GBRAIN_STDIN_TIMEOUT_MS overrides). Real pipes
(`echo foo | gbrain put x`, heredocs) deliver their first byte in
milliseconds; once any data arrives the deadline lifts and the read
drains to EOF, so slow producers keep working. An empty pipe that
closes yields ''. A pipe that never delivers a byte times out, the
param stays unset, and the existing required-param usage error fails
fast with exit 1.
Regression tests spawn the real CLI with a held-open, never-written
pipe (hangs 20s+ on pre-fix code; exits ~1s fixed) plus parity cases
for data pipes, /dev/null, and empty closed pipes, and a subprocess
driver pinning content preservation through applyStdinParam.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(build): restore the executable bit on src/cli.ts
check:cli-exec requires mode 100755; the edit in this branch landed it as
100644, failing `bun run verify` (1/32) on an otherwise-green PR. Mode only,
no content change.
* fix(cli): keep the R4-pinned stdin branch shape in applyStdinParam (#3513)
Shard 10's R4 regression pin (test/cycle/regression-pr-wave-r1-r2-r4.test.ts,
protecting PR #1325's Windows /dev/stdin → fd 0 fix) asserts three source
literals in src/cli.ts: `readFileSync(0, ...)`, the `op.cliHints?.stdin` +
`MAX_STDIN = 5_000_000` branch, and the `!process.stdin.isTTY` gate. The
bounded-read refactor kept the first two but inverted the TTY gate into a
positive early-return, dropping the pinned `!process.stdin.isTTY` spelling.
Restore the original branch shape inside applyStdinParam (guard + read +
cap + assign), unchanged semantics. The #1325 protection itself was never
at risk: no '/dev/stdin' anywhere, readFileSync(0) remains the read for
non-pipe stdin, and pipes drain through process.stdin (fd 0, cross-platform).
The pin now passes unmodified. A comment marks the shape as R4-pinned so
the next refactor doesn't trip it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
The `__all__` sentinel had two spellings and only one worked: as a
per-call source_id param, resolveRequestedScope understood it; as
--source __all__ or GBRAIN_SOURCE=__all__, SOURCE_ID_RE (which forbids
underscores) made all three resolver entry points throw. makeContext's
blanket catch then silently fell back to sourceId 'default' — making
the documented span-everything flag STRICTLY NARROWER than passing no
flag at all, because the catch also discarded the #2561/#3242 federated
widening:
unqualified read (federated brain) -> {"sourceIds":["default","src-a","src-b"]}
--source __all__ (pre-fix) -> {"sourceId":"default"}
--source __all__ (post-fix) -> {} (spans the brain)
Fix:
- src/core/source-id.ts: export ALL_SOURCES = '__all__'. SOURCE_ID_RE
itself is NOT loosened — it still guards source creation, lock ids,
and path joins, and its underscore rejection is what makes the
sentinel collision-free.
- src/core/source-resolver.ts: the explicit and env tiers of
resolveSourceId / resolveSourceIdEngineFree / resolveSourceWithTier
pass the sentinel through verbatim (skipping the regex and
assertSourceExists). Covers --source (#1712/#2289) and
GBRAIN_SOURCE (#2140), local and thin-client alike.
- src/core/operations.ts: sourceScopeOpts — the single choke point every
read-side scope helper delegates to — translates ctx.sourceId ===
ALL_SOURCES into {} for trusted local callers (strictly remote ===
false) and keeps the unsatisfiable literal for remote/untrusted
callers, so the sentinel can never widen past a caller's grant
(fail-closed). A federated grant still wins over the sentinel.
- src/cli.ts: makeContext's catch now rethrows when an explicit
--source was passed — a source that genuinely fails to resolve errors
loudly instead of silently becoming 'default' (the silent fallback is
what turned three bug reports into debugging sessions).
Tests (test/all-sources-sentinel.test.ts) fail on unmodified master
(9/13, behaviorally) and pass with the fix; the 4 that pass on both
sides pin invariants that must hold on both (remote fail-closed
literal, grant precedence, invalid-id rejection).
Closes#1712. #2289 and #2140 were closed as duplicates of it.
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
'backfill' had a fully implemented handler (case 'backfill' dispatching
to commands/backfill.ts) but was missing from the CLI_ONLY set, so
dispatch rejected every invocation with 'Unknown command: backfill'.
Same drift class as #2900 (reconcile-links) and #2035 (calibration).
Also lists backfill in the main --help TOOLS section.
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
shortRunId() truncated run ids to their first 8 chars, so propose_takes
run ids ('propose-<timestamp>-<uuid>') shortened to 'propose-' with a
trailing hyphen. slugifySegment() strips boundary hyphens during repo
sync, so the DB receipt slug and its Git-backed markdown slug disagreed
— writing the receipt through to the system-of-record repo created a
normalized sibling/collision instead of materializing the existing page.
shortRunId now trims boundary hyphens after truncation (invariant:
slugifySegment(shortRunId(x)) === shortRunId(x)), with a non-empty
fallback for pathological all-separator prefixes. All other run-id
families (atoms-, efacts-, concepts-, ecf-) are unchanged.
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
'gbrain takes list' printed 'No takes on list.' even when the brain held
many takes: the CLI had no list subcommand, so cmdList treated the word
'list' as a page slug and looked up a page named 'list'. The failure
read exactly like an empty takes table, so agents concluded there were
no takes and moved on.
'takes list' now lists all active takes (CLI parity with the takes_list
op), prefixing each row with its page slug. 'takes <slug>' unchanged.
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
#3461 — insertChunks stamped DEFAULT_EMBEDDING_MODEL on chunk provenance
whenever the AI gateway was unconfigured: getEmbeddingModel() throws
rather than returning falsy, so the reland's '|| resolvedModel' guard
(e1919fab) was dead code and the catch path kept the compile-time
constant. Both engines now fall back to the brain's own
config.embedding_model row on the throw path, with the compiled default
as last resort. Same-line sibling: the ON CONFLICT clause overwrote
'model' via COALESCE even when the existing vector was preserved —
'model' now mirrors the 'embedding' CASE branch-for-branch so the label
always describes whichever vector wins the upsert. The initSchema
sizing sites drop their dead '||' terms too.
#3507 — every plain re-embed path (embed <slug>, --all, --stale, and
the embed-backfill Minion loop) embedded raw chunk_text, silently
stripping the contextual-retrieval prefixes the sync path applied —
and embed --stale is the NORMAL post-model-migration path. All four
sites now wrap through wrapChunkTextsForStoredMode(), reproducing the
page's STORED convention (pages.contextual_retrieval_mode):
title/per_chunk_synopsis pages get the title-tier prefix, fenced_code
chunks stay unwrapped (D20-T4), unstamped pages stay raw. A fully
re-embedded per_chunk_synopsis page is restamped to 'title' so the
mode column keeps describing the vectors actually in the DB.
Deliberately NOT folding the contextual mode into
currentEmbeddingSignature(): the convention is already recorded
per-page (mode + corpus_generation), pages legitimately differ
per-page so a global signature cannot represent it, and bumping
signature semantics would force a full-corpus re-embed on upgrade.
No KNOBS_HASH_VERSION change (no collision with #3514).
Tests fail on unmodified master (6) and pass with the fix; postgres.js
path verified against a real pgvector instance in addition to the
PGLite suite.
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
facts-absorb performs one LLM extraction call per page — the same shape
as chronicle_extract — but was missing from HANDLER_DEFAULT_TIMEOUT_MS,
so it inherited the tight null-default wall-clock budget and was
dead-lettered mid-generation on slow chat providers. Nothing was
inserted; the page's facts were silently lost.
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Defect A (#3526): the empty-fence guard COUNT had no source_id predicate,
so one pending legacy row in any mounted source jammed extract_facts for
every source in the brain. The count now binds f.source_id = $1 to the
run's sourceId (source-isolation invariant).
Defect B: the guard advised `gbrain apply-migrations --yes`, which is a
proven no-op once the v0.32.2 ledger entry is complete (the runner
classifies it as already-applied and Phase B never re-runs). The warning
(and cycle.ts's phase hint) now gives the drain path verified to work
end-to-end on a real brain: `apply-migrations --force-retry 0.32.2`
then `apply-migrations --yes` (Phase B is idempotent), or forget_fact
per row.
New test proves a pending legacy row in source A does not jam extraction
for source B (fails on master, passes with the fix).
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
gbrain jobs prune --dry-run used to silently drop the flag and run the
destructive default — rows were really deleted while the operator
believed they were previewing.
MinionQueue.prune now takes dryRun: count the would-be-pruned rows
without deleting; the CLI parses --dry-run and labels the output.
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
slugifySegment stripped every character outside [a-z0-9._-] + four CJK
ranges, so filenames in Hebrew, Arabic, Cyrillic, Greek, Thai (and every
other script) collapsed to empty segments. Distinct files then mapped to
the SAME slug (their shared directory prefix) and silently overwrote each
other, last-writer-wins, with import reporting 0 errors.
All three slug grammars move together so sync never emits a slug that
put_page rejects:
- sync.ts SLUGIFY_KEEP_RE + SLUG_SEGMENT_PATTERN — now keep
\p{Ll}\p{Lm}\p{Lo}\p{M}\p{N} (new single-source SLUG_WORD_CHARS in
cjk.ts), with the u flag
- cjk.ts PAGE_SLUG_SEG — rebuilt on SLUG_WORD_CHARS; consumers
(validatePageSlug, SlugRegistry SLUG_RE, dream-cycle SUMMARY_SLUG_RE,
takes-fence HOLDER_REGEX) all gain the required u flag
CJK_SLUG_CHARS is untouched — it also drives the countCJKAwareWords
chunking density heuristic and must not move with slug grammar.
Kebab-casing, lowercasing, Latin accent-strip (cafe), dots/underscores,
path handling, and the NFC re-normalize (NFD macOS filenames converge
with NFC git filenames) are all unchanged and regression-pinned in
test/slug-unicode-scripts.test.ts. No data migration: the collapse was
N-to-1, so only one row per group ever persisted; a normal gbrain sync
recreates the lost pages under their real slugs.
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
The repo publishes zero GitHub releases, so releases/latest is a permanent
404 and fetchLatestRelease() could never succeed — the entire upgrade
notification subsystem was a silent no-op, and refreshUpdateCache() cached
a fabricated up_to_date marker on every failure.
- Resolve the latest version from raw.githubusercontent.com/.../master/VERSION
(same trusted host fetchChangelog already uses). Bounded, shape-gated parse;
handles legacy 3-segment and -suffix channel forms.
- Discriminate network_error from no_releases in the --json error field and
human output.
- Never write up_to_date on a failed check: preserve the last-known-good
marker (mtime bump keeps the TTL throttle) or write nothing.
- Rejected the issue's proposed npm fallback: the gbrain npm package is an
unrelated GPU library (#505) and would produce false upgrade prompts.
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Three layers, verified by 10-run tallies (master: 7 pass/3 fail; branch: 10/10):
1. Deadline inversion: the hook tests' internal 8s poll deadlines sat behind
bun's 5000ms default because no third-arg timeout was passed — and bun
1.3.14 ignores bunfig.toml's `timeout` key, so bare `bun test` runs died
at 5s with the 8s budget unreachable. All three tests now pass 60_000
explicitly; deadlines raised to 30s for loaded-shard headroom. Same class
fixed in test/e2e/jsonb-roundtrip.test.ts (four tests had no explicit
timeout).
2. Root cause of the CI assertion failures: the test's git() helper spawned
git WITHOUT `env: process.env`. Bun snapshots process.env at startup (the
#2747 quirk), so the post-commit hook under test resolved GBRAIN_HOME to
the operator's real ~/.gbrain — writing its log there (polluting the real
brain-push.log every run) while the test polled the temp log. The
LOCAL-ONLY assertion only passed when beforeEach's scaffolding hook push
happened to still be in flight, lose the ref race, and retry after origin
pointed at the dead path — a load-dependent accident. env is now passed
everywhere, making the intended signal deterministic (~1s).
3. index.lock race (the third observed CI form): beforeEach now waits for the
scaffolding commit's detached brain_push to write its terminal log line
before handing the repo to the test, so its pull-rebase fallback can't
take .git/index.lock under the test body's own git calls.
Poll loops untouched — they were already correct; the 150 is the poll interval.
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* v0.42.67.0 fix(build): force LF for shell scripts and route package.json checks through bash
Two independent defects left `bun run test`, `verify`, `ci:local` and
`test:e2e` dead on Windows. All four dispatch through bash.
First, every tracked *.sh is checked out with CRLF. The committed blobs are
clean LF; system-level core.autocrlf=true rewrites them on checkout, and a
strict bash then dies at run-unit-parallel.sh line 23 with
"$'\r': command not found". A root .gitattributes pinning `*.sh text eol=lf`
overrides autocrlf regardless of the contributor's git config.
Second, 33 package.json scripts invoked `scripts/foo.sh` directly, which bun
cannot exec via shebang on Windows. They now go through `bash`, matching the
11 that already did; all 59 tracked *.sh files are bash-shebanged (52
`#!/usr/bin/env bash`, 7 `#!/bin/bash`), so the change is uniform. The five
`scripts/*.ts` entries still run under bun.
Measured on this base, `bun run verify` goes from pass=1 fail=31 to pass=25
fail=7. Every one of the baseline's 29 `command not found: scripts/...`
errors is gone; those were the shebang defect, and they account for the
measured delta.
The line-ending defect is verified structurally rather than by that number,
because the bash on PATH for this measurement tolerates CR and so cannot
exhibit it: under the new attribute all 59 tracked *.sh check out LF-only
(0/59 carry a CR byte, against 59/59 before), and `git add --renormalize .`
is a no-op, confirming the index was always correct and only the working
tree was wrong. Zero content churn.
All 7 residual failures also fail on the pristine baseline: four exceed the
harness's 120s cap (standalone `bun run typecheck` exits 0), and check:wasm,
check:skill-brain-first and check:resolver are pre-existing content or
environment issues. check:resolver is not even a shell script.
No behavior change on Linux or macOS.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs: sync docs to v0.42.67.0
CONTRIBUTING.md gains a Windows section: the `.gitattributes` LF pin makes a
fresh clone correct with no extra steps, working copies cloned earlier need a
one-time `git rm --cached -r . -q && git reset --hard`, and new shell-script
checks must be registered as `bash scripts/<name>.sh`.
docs/TESTING.md records the shell-dispatch convention alongside the command-tier
table, and notes that the table's wallclock figures are Mac numbers: on Windows
`check:privacy`, `check:test-names`, `check:test-isolation` and `typecheck` can
exceed run-verify-parallel.sh's 120s per-check cap while passing on Linux and
macOS. It also flags that the Cygwin bash shipped with Git for Windows tolerates
CRLF where a strict bash does not, so a green local run is not evidence that a
script is CRLF-clean.
CHANGELOG.md's itemized list covers both doc updates.
`bun run build:llms` regenerates byte-identical bundles: docs/TESTING.md is
linked rather than inlined, so llms.txt / llms-full.txt do not move.
`bun test test/build-llms.test.ts` passes 12/12.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
master's empty-fence guard counts soft-expired legacy rows
(row_num IS NULL, expired_at set), so forget_fact — the sanctioned
removal path, which soft-expires rather than deletes — can never drain
the backlog: apply-migrations no-ops (already marked applied) and the
guard stays triggered forever, jamming extract_facts.
Narrowed re-send of #3252-sibling #3234, scoped to exactly what the
maintainer named reviewable: "one predicate plus the
reconcile-preservation guard."
- Guard predicate: the legacy COUNT adds `AND f.expired_at IS NULL`,
so each forget_fact visibly drains the pending counter.
- Reconcile preservation: listExistingFactsForPage excludes soft-expired
legacy rows (they are never fence-owned, so they must neither read as
perpetually stale nor mask a fence row), and the two wipe call sites
pass `preserveExpiredLegacy: true` so deleteFactsForPage keeps the
forget record. The option is the minimal seam for that guard —
implemented identically in both engines (~6 lines each).
Explicitly NOT included from #3234 (per the close review): the
drift-repair lane, the re-runnable migration orchestrator path, and the
race-accounting layer.
Fence-is-canonical semantics are preserved and pinned by test: if the
fence still carries an expired legacy row's claim, the reconcile
reinserts it as a fresh active fence-owned row (legacy DB-only forgets
are documented non-durable; the expired row survives as the audit
record of the forget).
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
Shape requested in #3475's closing review: make capOversizedChunks use a
CJK-aware estimate instead of adding a parallel opt-in cap.
Measurement (vs the Qwen3-Embedding tokenizer, the strict-backend class
from #2826): cl100k matches embedding-family tokenizers on pure-ASCII
source (identical counts on English prose and JSON) but undercounts
MIXED CJK+ASCII chunks — −31% on URL-dense Korean text. The heuristic
fallback (~3.5 chars/token) undercounts CJK ~2.5×.
estimateEmbedTokens(): for chunks containing CJK, max(cl100k, per-char-
class overestimate — CJK 1.0 / other non-ws 0.75 / ws 0.1). ASCII-only
chunks short-circuit to estimateTokens verbatim (bit-identical, pinned);
CJK-DOMINANT text is unchanged too (cl100k already exceeds the weighted
form, so max() returns today's value — pinned). Only mixed-script
chunks, the measured divergence class, estimate higher. Reuses cjk.ts's
existing exports — no new module, no config.
Also: only the empty-AST branch routed its fallback through
capOversizedChunks. The no-language, parse-timeout, no-semantic-nodes
(every JSON/YAML fence — their node types aren't in TOP_LEVEL_TYPES) and
parse-throw branches shipped word-counted chunks unchecked, letting a
14K-char JSON fence emit ~2,700-token chunks past the 2,000 default cap.
Hoist the cap into fallbackChunks so all five emission paths share the
net. Hard-split slice budget becomes 1 char/token for CJK-bearing pieces
(the weighted estimate can reach 1 token/char).
Refs #2826
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
The thin-client branches receive MinionJob rows as parsed JSON off the
MCP wire — every timestamp an ISO string — while formatJob /
formatJobDetail and the stalled-detection comparison hold a Date
contract (locally hydrated by MinionQueue.rowToJob). `jobs get <id>` on
a thin client crashed with "job.started_at.toISOString is not a
function" the moment the remote routing actually worked (unmasked by
the #2951 scratch-engine fix).
Rehydrate once at the unpack boundary via an exported helper that
coerces valid ISO strings to Dates, leaves Dates/nulls/malformed
strings untouched, and preserves the input type. Unit tests +
source-audit pins for both unpack sites.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
On stateless deploys (Docker on EB/K8s/Fly — what the cloud recipes
produce), a container restart wipes federated clones; each is only
re-materialized when that source's next sync job runs. Until then the
v0.41.27.0 git short-circuit cannot probe HEAD at all, and the check
fell through to raw wall-clock age — which no-op syncs never advance —
so every QUIET source read as stale/FAIL right after a restart.
Observed live: 16-source brain, 12 clones gone after a config-update
restart, doctor 70 -> 25-35, monitor alert storm (score < threshold)
while every clone that DID exist was byte-identical to origin HEAD.
Fix: classify the probe three ways (probeSourceGitState: unchanged /
changed / unavailable). 'unavailable' + chunker match borrows the
REMOTE path's newest_content_at lag (v0.41.32.0) — DB-only, no
subprocess — so a quiet source reads healthy while real missed work
(content newer than last sync) still reports stale. 'changed'
(readable clone, HEAD moved / dirty) keeps wall-clock exactly as
before, and a chunker mismatch disables the fallback (D7: a pending
re-chunk is never masked). isSourceUnchangedSinceSync stays as a
boolean facade so source-health.ts is untouched.
Tests: 6 new doctor cases (F1-F6, incl. three-bucket invariant) +
7 probeSourceGitState unit cases; existing suites green
(doctor 90, git-head 21, source-health 28), tsc --noEmit clean.
The no-embedding-provider short-circuit in hybridSearch probed only the
text column's provider. On a multimodal-only install (text embedding
provider absent, a multimodal provider such as Voyage multimodal-3
present), the function returned to the keyword-only path before the
image/unified vector routing ever ran -- so image and unified queries
silently degraded to keyword search (vector_enabled:false) even though a
usable multimodal vector path existed.
Add a willTryMultimodal guard that probes the multimodal embedding
provider (embedding_multimodal_model) so the early-return does not fire
when multimodal vectoring is still possible, and tighten the unified and
image branches' bare aiIsAvailable('embedding') (global-default) checks
to probe the multimodal provider too.
Adds a focused regression test (search-multimodal-no-embed.serial) that
configures a text-provider-absent / multimodal-present install and
asserts image + unified queries reach the multimodal vector path.
Co-authored-by: ElliotDrel <ElliotDrel@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Commit faf5cdba tracked `node_modules -> /tmp/fleet/repo/node_modules`.
That path exists only on the sandbox that produced it, so every other
clone materialized a dangling symlink and `bun install` aborted with
`ENOENT: could not open the "node_modules" directory`. That also broke
`gbrain upgrade` on bun-link installs, which shells out to bun install
and then prints a manual fallback that fails identically.
Three changes:
- Untrack the symlink (`git rm --cached node_modules`).
- Drop the trailing slash from the .gitignore node_modules patterns. A
`node_modules/` pattern matches directories only, which is why a
symlink of the same name was never ignored in the first place.
- Add scripts/check-no-tracked-symlinks.sh, wired into `bun run verify`
and `check:all`. The .gitignore fix alone is not sufficient, since
`git add -f` bypasses it; the guard fails on any mode-120000 entry.
The repo has no legitimate tracked symlinks, so it starts with an
empty allowlist.
Covered by test/no-tracked-symlinks-guard.test.ts, which builds a
throwaway repo containing the exact symlink shape and asserts the guard
exits 1 and names the offender.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
DashScope's OpenAI-compatible /embeddings endpoint rejects requests with
more than 10 input items (documented Model Studio cap). The generic
per-recipe max_batch_items field + gateway capBatchItems pre-split
already exist (#1281); the in-tree dashscope recipe just never declared
the cap, so large embed backfills would send oversized batches and get
rejected server-side. Declare max_batch_items: 10 on the dashscope
embedding touchpoint; max_batch_tokens stays as the aggregate
token-size guard.
Test: pins dashscope's max_batch_items === 10 (+ max_batch_tokens
unchanged) and that 25 items pre-split into groups of at most 10 via
capBatchItems, alongside the existing llama-server cap pin.
No new models or recipes; no error-sniffing/halving recovery — the
pre-split makes the failure unreachable. Item-cap concept credited to
declined community PRs #2643 and #2405.
Also verified (no code change needed): #2103's litellm three-way dead
end is already fixed on master by a25209bb (#2271) via trust_custom_dims.
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Yicong <charlieyiconghuang@gmail.com>
Co-authored-by: Cheng Zijun <robotics.chengzijun@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
extract-conversation-facts extracted nothing on brains that store chat in the
collector's native page types. Two stacked gaps:
- Type routing: the allowlist exact-matched {conversation,meeting,slack,email}
against pages.type and passed each straight to listPages({type}), so
--types slack matched zero rows on a brain carrying slack-dm-day /
slack-thread / email-digest. Add ALLOWED_TYPE_ALIASES + pageTypesForAllowed()
to expand logical -> concrete (canonical name first so consolidated brains are
unaffected), wired into both the single-slug filter and the listPages loop.
- Block-format parsing: the 14 built-in patterns are single-line; the Slack
collector emits a header + indented-body block (`- **Name** (Mon 11:18)` then
body on following lines) that none match -> phase:'no_match', 0 messages, and
the LLM fallback is not wired. Add normalize-block.ts, a strict-no-op pre-pass
in parseConversation that collapses the block into the canonical
`**Name** (HH:MM): body` line the bold-paren-time pattern handles; the
per-message date fills in downstream via fallbackDate. 12h am/pm normalized to
24h; day-of-week dropped.
Verified on a 13.7K-page comms brain whose facts table was empty: a 12-page
Slack sample went 0/12 parsed (no_match) -> 12/12 (regex_match), 103 messages,
13 segments; extraction wrote 58 facts across 16 entities (~$0.09) and
find_trajectory returns a populated points list for a local/owner caller where
it previously returned empty.
Tests: +13 normalize-block (detection, multi-paragraph collapse, 12h->24h,
no-op on canonical, parseConversation integration) + 7 pageTypesForAllowed.
typecheck clean; verify 30/30.
Claude-Session: https://claude.ai/code/session_01E5wtDU4ZLKewXUYkPLQHSy
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
* feat(extract): --infer-dates anchors timeline from a page's content date when its body has none
parseTimelineEntries only reads in-body date lines (`- **YYYY-MM-DD** | ...`).
Comms- and calendar-dominated brains keep the date in frontmatter or the
filename (slug `2026-04-24-...`), so those pages yield zero timeline entries
and find_trajectory stays blind even though the page is firmly dated.
`--infer-dates` (opt-in, DB-source) anchors ONE timeline entry at the page's
already-computed `effective_date` for pages whose body parse returns nothing.
Trustworthy sources only (frontmatter event_date/date/published or the filename
date) — never the `updated_at` fallback. Applied solely on the zero-entry path
so it can never shadow a real in-body timeline.
- new pure helper `deriveTimelineAnchor()` in link-extraction.ts (+6 unit tests)
- `getPage()` now projects effective_date/effective_date_source in BOTH engines
(engine parity)
- on a comms-heavy ~13.7K-page brain this lifts a dry-run timeline yield from 1
to 11,006 entries (timeline coverage 0% -> ~80%)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01E5wtDU4ZLKewXUYkPLQHSy
* docs(extract): correct deriveTimelineAnchor comment — feeds page timeline, not find_trajectory
find_trajectory reads the facts table by entity_slug; the page-level `timeline`
table this helper populates feeds get_timeline + the brain-score timeline_coverage
component instead. Comment-only; no behavior change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01E5wtDU4ZLKewXUYkPLQHSy
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
* feat: CJK entity extraction for Chinese/Japanese/Korean names
- Add hasCJK() / cjkCharCount() detection helpers
- Lower min name length for CJK entities from 4 to 2 chars
- Fix tokenizeTitle() to handle pure CJK titles as single tokens
(was returning [] for CJK-only titles, excluding them from gazetteer)
- Add CJK substring matching pass in findMentionedEntities
- NER extraction works without schema pack (plain mentions fallback)
Verified: gbrain extract links --by-mention creates 27 links from
456 pages with 3 CJK entity pages in gazetteer.
* feat: Chinese link type inference + timeline date formats
Link types:
- CN_FOUNDED_RE: 创立/创办/成立/创建 → founded
- CN_INVESTED_RE: 投资/入股/融资 → invested_in
- CN_ADVISES_RE: 顾问/咨询/指导 → advises
- CN_WORKS_AT_RE: 任职/就职/担任 → works_at
- CN_CITED_RE: 引用/提到/提及 → cited
Timeline:
- TIMELINE_LINE_RE_CN: YYYY年M月D日 | event
- Auto-normalizes to YYYY-MM-DD format
- Falls through to English format if CN doesn't match
* fix: CJK tokenizer uses char-level tokens (reviewer feedback)
Addresses all 4 concerns from review of PR #1637:
1. tokenizeForScan now emits CJK characters as individual tokens
— normal scan path reaches CJK gazetteer entries naturally,
eliminating the separate O(P×C×N) substring fallback pass.
2. tokenizeTitle splits pure CJK titles into individual chars
— e.g. '纳瓦尔' → ['纳','瓦','尔'], matching body-level CJK tokens.
3. Removed O(P×C×N) CJK substring pass — no longer needed.
Performance now O(P × N_tokens) for both ASCII and CJK.
4. Renamed CN_*_RE → ZH_*_RE in link-extraction.ts with a comment
clarifying these are Chinese-only (entity NAME extraction in
by-mention.ts covers CJK scripts, link TYPE extraction is zh only).
Added 12 CJK-specific tests (10 pure + 2 engine integration).
All 51 existing + new tests pass.
* review-repair(#1637): scope CN timeline regex to 年月日, revert off-scope extract-ner no-pack change, cosmetics
- TIMELINE_LINE_RE_CN required only [年-] separators, so non-bold ASCII
dates (- 2020-01-02 - text) started parsing as timeline entries — an
English-default regression. Now requires the 年/月 markers.
- Dropped the dead 'm = cm as any' assignment.
- src/core/extract-ner.ts reverted to origin/master: the no-pack →
plain-mentions walk was off-scope for a CJK PR, duplicated the
existing --by-mention pass, and hardcoded pack_unavailable:false
(breaking the CLI hint).
- by-mention.ts: fixed stray indentation + restored EOF newline.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
* feat(extract): quarantine lane for auto-extracted entities from untrusted input (#160)
extractAndEnrich regex-extracts entity names from arbitrary ingested text
and creates people/ + companies/ stub pages. Those writes are now trust-
gated end to end:
- src/core/extraction-review.ts: new marker module (sibling of
quarantine.ts / embed-skip.ts, frontmatter-key pattern, no migration).
Untrusted-input stubs carry `provenance: auto-extracted` +
`status: unverified`; the shared unverifiedExtractionFragment() is the
single SQL source of truth for every consumer.
- enrichment-service: enrichEntity/enrichEntities/extractAndEnrich take
EnrichmentTrustOptions; only an explicit trusted:true writes
authoritative pages (fail-closed, mirrors the OperationContext.remote
invariant). Also threads sourceId through the write path.
- retrieval: unverified stubs rank as ordinary content — skipped by the
compiled-truth fusion boost (stampUnverifiedExtractions pre-fusion on
all three hybrid paths + keyword-only opt-out) and by the people//
companies/ namespace source-boost (guard inside buildSourceFactorCase,
shared by both engines' search SQL). Results carry `unverified: true`.
New engine method getUnverifiedExtractionPageIds in BOTH engines.
- ops (contract-first): extract_entities (direct write only for
ctx.remote === false + --trusted-extraction; everything else
quarantines), extraction_pending (read, source-scoped list),
extraction_review (owner-only batch promote/reject; promote flips
status to verified keeping provenance for audit, reject soft-deletes).
- doctor: unverified_extractions check warns on stubs older than N days
(default 7) with the exact review commands.
Tests: test/extraction-review.test.ts (PGLite: fail-closed matrix incl.
remote-unset, fusion boost skip, review queue, doctor, hostile-transcript
e2e proving fake entities land quarantined and rank below a verified page
of equal lexical relevance) + test/e2e/extraction-review-postgres.test.ts
(live Postgres parity, verified against pgvector:pg16). sql-ranking
expectations updated to current state. Docs: KEY_FILES + RETRIEVAL +
llms rebuild.
Closes#160
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(extract): close vector-arm source-boost gap + harden extract_entities (#160 review round)
Adversarial review of the quarantine lane found the people//companies/
1.2x source factor still applied to unverified stubs inside searchVector's
pre-LIMIT re-rank (a different multiplier from the fusion-level 2.0x the
lane already cancels — and applied early enough to evict legitimate pages
from the candidate pool, which nothing downstream can restore).
- buildSourceFactorCase gains an optional unverifiedGuardColumn for the
bare-slug re-rank form; both engines' hnsw_candidates CTEs now project
the guard predicate as `unverified_stub` and the factor CASE checks it
first. Wrong "fusion covers the vector arm" comment corrected.
- extract_entities resource guards: 200k-char input cap (loud reject),
200-entity cap surfaced as `truncated` + `entities_found`; the library
extractAndEnrich gets the same default cap. (OperationContext has no
abort signal field — caps are the bound.)
- extraction_review promote is now a targeted JSONB-merge UPDATE instead
of putPage, so non-carried columns (page_kind, content_hash) can't be
reset by the upsert.
- extraction_pending applies buildVisibilityClause (archived-source stubs
no longer list).
- Wording: op description + module header now state the marker-strip
assumption plainly (markers are ordinary frontmatter; the boundary
against wholesale rewrite is put_page write authz) and document the
CREATE-only scope of the lane.
Tests: vector-arm factor-1.0 pinned on BOTH engines (PGLite unit + live
Postgres e2e, identical basis embeddings → score ratio is the factor);
resource-guard test (oversize reject + 300-entity flood capped at 200);
guard-column form pinned in the buildSourceFactorCase unit test.
search/ suite (340), sql-ranking, searchvector-maxpool, title-retrieval-
arm, rrf-source-key, doctor, ops, cli suites all green; JSONB guards clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(migrate): provider-agnostic embedding migration service — the path off ZeroEntropy (#3390)
- gbrain migrate embeddings --to <provider:model> (alias: retrieval-upgrade):
plan + cost preflight, consent gate (--yes / TTY confirm / non-TTY exit 2),
live probe against the target provider before any mutation, env-override
gate, schema dimension transition via the shared runSchemaTransition,
dual-plane config write, NULL-signature-inclusive invalidation, query-cache
purge, resumable re-embed through the standard embed pipeline (single-flight
locks, backoff, pacing, stderr progress). Killed runs resume by re-running
the same command; the NULL-embedding column is the checkpoint.
- #3391 root-cause fix (both engines): countStaleChunks / sumStaleChunkChars /
invalidateStaleSignatureEmbeddings accept includeNullSignature to lift the
v108 grandfather clause; embed --stale warns loudly when a model swap
leaves NULL-signature pages in the old embedding space, and
--include-null-signature re-embeds them. Default sweep behavior unchanged.
- knobs_hash v=12 → v=13 (prov=default legacy callers must not be served
pre-migration cache rows).
- migrate_embeddings op: scope admin, localOnly, hidden cliHints, hard
remote refusal, needs_confirmation without yes=true.
- One-shot post-upgrade ZE-sunset banner (ze_sunset_notice_shown) for brains
resolving to a zeroentropyai:* embedding model or reranker.
- doctor's dimension-mismatch repair hint now names the real command.
- Docs: docs/guides/embedding-migration.md, KEY_FILES entries, spend-controls
gate row. Tests: PGLite unit + full-lifecycle flow (interrupted-run resume),
real-Postgres e2e (pgvector DDL path + #3391 predicate parity).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(test): satisfy check:test-isolation + bump the remaining knobs_hash pins
- test/migrate-embeddings-flow.test.ts → .serial.test.ts: the file holds a
temp GBRAIN_HOME + an installed fake embed transport for its whole
lifecycle (beforeAll → afterAll), which withEnv() can't wrap. This also
fixes the CI shard-pollution failure in
test/ai/recipes-existing-regression.test.ts (that file passes solo on both
master and this branch; the flow test's configureGateway + provider-key
deletion was leaking into it inside the same shard process).
- test/embedding-migration.test.ts: env-override case now uses withEnv().
- Bump the three remaining KNOBS_HASH_VERSION pins to 13
(cross-modal-phase1, search-alias-resolved-boost, search/knobs-hash-reranker).
- Docs + llms bundles follow the test rename.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore(test): wire the new Postgres e2e into the smart e2e selector map
Changes to embed.ts / embedding-migration.ts / retrieval-upgrade-planner.ts /
postgres-engine.ts now trigger test/e2e/migrate-embeddings-postgres.test.ts —
the #3391 stale predicates and runSchemaTransition's DDL path behave
differently on real pgvector than on PGLite, so the smart selector has to know.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(migrate): consult spend.posture in the embedding-migration consent gate
The brief asked the gate to honor spend.posture; it previously didn't read it
at all. Now it does — but deliberately does NOT bypass on tokenmax: posture
waives the spend CEILING, and this gate also guards a destructive schema
rebuild (existing vectors dropped, retrieval degraded until the re-embed
finishes). Under tokenmax the dollar figure is marked informational on stderr
and the confirmation is still asked; --yes stays the single scripted bypass.
Pinned by a new case in the flow test so a later refactor can't quietly turn
posture into a bypass. Guide + spend-controls table updated to match.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* wip: blocker fixes
---------
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Kotlin chunks fine (bundled grammar, symbol-typed chunks) but CALL_CONFIG
had no kotlin entry, so code sync on Kotlin repos produced zero call edges
and code_callers/code_callees/code_blast/code_flow returned empty.
Two grammar quirks made this more than a config row:
- tree-sitter-kotlin defines no fields on call_expression, and
extractCalleeName required calleeFieldName (the interface comment
claimed a text-scan fallback that the code never had). Added an
explicit calleeFirstNamedChild option — the callee is positional
(namedChild(0)) — reusable by any future field-less grammar; corrected
the stale comment.
- receiver calls parse as navigation_expression, unknown to the unwrap
loop. Added a case alongside member_expression (TS) / scoped_identifier
(Rust) that walks to the trailing navigation_suffix identifier, so
receiver.method(...) resolves to the method, not the receiver.
No behavior change for the existing 8 languages: the new callee path only
activates via calleeFirstNamedChild, and navigation_expression does not
occur in the other configured grammars.
Validated on a private production Kotlin codebase (Spring + QueryDSL,
5,143 .kt files): 0 parse errors, 10,621 chunks, 89,279 call edges,
5,586 distinct callees.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(azure): keyless (Entra/AAD) auth for the azure-openai embedding recipe
Subscriptions that enforce `disableLocalAuth` via Azure Policy reject api-key
auth, so the azure-openai recipe was unusable there. Add an Entra path:
- recipes/azure-openai.ts: when AZURE_OPENAI_API_KEY is absent (or
AZURE_OPENAI_USE_ENTRA=1), mint a short-lived AAD bearer token via
`az account get-access-token --resource https://cognitiveservices.azure.com`,
cached ~45min. resolveAuth is sync, so execSync is the seam. Returns an
`Authorization: Bearer …` pair (gateway uses the SDK's native bearer path).
AZURE_OPENAI_API_KEY moves from required → optional.
- config.ts + build-gateway-config.ts: add azure_openai_endpoint /
azure_openai_deployment / azure_openai_use_entra config keys, folded into the
gateway env (same pattern as openai_api_key) so the recipe works in any shell
without per-shell env. Non-secret only; the token is minted at request time.
Caller needs `az login` + the "Cognitive Services OpenAI User" role on the
resource. Verified end-to-end: import + query retrieval against a keyless
Azure OpenAI text-embedding-3-large deployment.
* fix(azure): refresh Entra bearer per request + align recipe tests with keyless auth
The gateway caches model instances with auth baked in at instantiation, so
the AAD token minted in resolveAuth would go stale after ~1h in long-running
processes. The recipe's existing api-version fetch wrapper now re-sets the
Authorization header from the TTL-cached token on every request in Entra
mode. Adds a test seam (__setEntraTokenForTests) so unit tests never shell
out to az, and updates test/ai/recipe-azure-openai.test.ts for the
required->optional AZURE_OPENAI_API_KEY move.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(azure): non-null assert api key in key mode (typecheck)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: joncules <jon.in.christ@gmail.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat: support OpenRouter API key in config
* fixup: dedupe openrouter_api_key vs master, drop no-op compile-guard test
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(gateway): constrain query expansion JSON key to "queries"
The expansion prompt asks the model to "Rewrite the search query below
into 3-4 different, related queries" without naming the JSON key.
On OpenAI-compatible endpoints that don't enforce a strict JSON schema
server-side (e.g. DeepSeek, many self-hosted gateways), the model
picks the prompt-salient noun and emits {"rewrites": [...]}, which
fails ExpansionSchema ({ queries: string[] }) validation. The catch
block only warns for AIConfigError, so the schema-validation failure
silently falls back to [query] and expansion is effectively disabled.
Verified on two providers: oMLX serving Qwen3.6-35B-A3B-6bit at
http://127.0.0.1:8888/v1 and deepseek-v4-flash at
https://api.deepseek.com/v1. With the prompt constraint, both return
{"queries": [...]} and gbrain query latency increases by ~150 ms
(the expansion inference), confirming expansion now runs end-to-end.
Refs #1156
(cherry picked from commit 132973039c)
* fix(gateway): expand() falls back to generateText for openai-compat providers
generateObject() with a Zod schema uses the response_format
json_schema mode, which most openai-compatible providers do not
support. When the provider rejects structured outputs, the expansion
silently returns only the original query — no error, no log, just
degraded retrieval quality.
For openai-compatible recipes, use generateText() with a JSON prompt
and parse the response manually. Native providers (Anthropic, OpenAI,
Google) keep the existing generateObject() path. This fixes silent
expansion failure for all openai-compatible providers: Zhipu/GLM,
DeepSeek, Groq, Together, Ollama, and any future recipe using the
openai-compatible implementation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
(cherry picked from commit 0e271961c0)
* refactor(ai): lift parseLlmJson into a leaf util
parseLlmJson lived in conversation-parser/llm-base.ts, which imports chat from the gateway. The gateway needs the same tolerant decoder for its expansion fallback, so importing it back would create a dependency cycle and pull the conversation-parser base into the gateway's module graph.
Move the function to src/core/llm-json.ts, a leaf with no provider or gateway imports, and re-export it from llm-base.ts so existing importers (llm-fallback, llm-polish) and its test keep their import path unchanged. Behavior-preserving.
* feat(ai/gateway): structured-output opt-in + capability-aware expansion fallback
Unifies two cherry-picked fixes (preserved in this branch's history) under a single capability flag and one expand() path:
- #1158 (im4saken): names the required "queries" key in the expansion prompt.
- #1618 (punksterlabs): falls back to generateText for openai-compatible providers.
Adds ChatTouchpoint.supports_structured_outputs (default false) and threads it into createOpenAICompatible's supportsStructuredOutputs at the chat and expansion build sites via recipeSupportsStructuredOutputs().
expand() now routes three ways:
- Native providers (Anthropic, OpenAI, Google) use generateObject unchanged.
- openai-compatible recipes that opt into structured outputs request a strict json_schema and fall back to the text path if it is rejected at call time, so a mis-declared capability never drops expansion.
- Every other openai-compatible recipe skips the json_schema attempt and parses the model's text directly, which removes the AI SDK warning and the silent degradation.
parseExpansionResponse() recovers the queries through a tolerant JSON decode plus schema validation, replacing the inline regex parse.
Net: fixes the silent expansion failure for every openai-compatible backend (the #1618 case), keeps the named-key prompt (closes the gap in #1156 that #1158 addresses), and adds strict structured outputs for backends that support them, which the always-generateText approach cannot reach.
Tests: capability gating across recipes plus a synthetic opt-in recipe; schemaless recovery from clean, fenced, and prose-wrapped JSON; null on non-JSON and schema-violating output.
---------
Co-authored-by: im4saken <280051114+im4saken@users.noreply.github.com>
Co-authored-by: Allwin Agnel <allwin.agnel@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
PR #2917 shipped the security-CI trio (OSV-Scanner, Semgrep CE SAST,
release-binary attestations) but landed no contributor/user-facing docs.
This adds the functional posture notes the issues asked for:
- SECURITY.md: "Automated security scanning" section — what runs, when,
and the gh attestation verify commands for release binaries (#2142
item 4).
- CONTRIBUTING.md: PR-side note that Semgrep is advisory/non-blocking
while the baseline is tuned (#2272 item 5), plus when OSV-Scanner and
actionlint fire on a PR.
No workflow changes: the audit found all three workflows already on
master, green, SHA-pinned, least-privilege, with the reusable-workflow
caller-permission superset already granted.
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: maxpetrusenkoagent <max.petrusenko.agent@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
DeepSeek retired `deepseek-chat` and `deepseek-reasoner` on 2026-07-24;
both map to `deepseek-v4-flash` (non-thinking / thinking mode). Recipe
model lists, context window (1M), providers-test example, and canonical
pricing updated; legacy `deepseek:deepseek-chat` pricing row kept so
historical usage/audit rows still price.
Reported by @W4RW1CK in #1255.
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
resolveSourceForDir matched two path SPELLINGS: --dir goes through
resolve(), while sources.local_path stores whatever spelling the source
was registered with. Neither side is canonicalized, so a source
registered through a symlink but dreamt via the real path (or vice
versa) never matched, no source was derived, the #1869 freshness stamp
never landed, and doctor's cycle_freshness stayed permanently stale.
On an exact-match miss, retry with realpathSync applied to both sides.
Archived sources are excluded (dream already refuses to stamp them) and
an ambiguous canonical match fails closed rather than picking an
arbitrary id.
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
* fix(synthesize): dedupe across corpus moves
* fix(synthesize): dedupe legacy CHUNKED completions; keep plain-completed suppression
Repairs three gaps in the corpus-move dedupe (v2 content-hash keys):
1. Legacy chunked completions now suppress v2 resubmission. The scan
previously matched only keys ending ':<hash16>' (legacy single-chunk),
so every transcript synthesized under the pre-v2 chunked family
'dream:synth:<path>:<hash16>:c<i>of<n>' re-ran as a full paid v2
synthesis after upgrade. findLegacyCompletion now also matches the
chunked family, counting a transcript as done only when the FULL
chunk set c0..c(n-1) completed; partial sets fall through to a fresh
v2 run (reason: already_synthesized_legacy_chunked for full sets).
2+3. Legacy suppression reverts to plain status='completed', dropping the
result->>'stop_reason' = 'end_turn' filter. This restores the pre-v2
cost-safe semantics (queue-level idempotency blocks re-submission of
completed jobs regardless of stop_reason, pinned in test/minions.test.ts)
and sidesteps the double-encoded-jsonb result rows the naive ->> read
missed. The tightening was not documented as intended in the PR.
Tests: legacy chunked full-set suppression + partial-set resubmission;
double-encoded jsonb result row still recognized.
Co-authored-by: zsimovanforgeops <justin@caddolandworks.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Forge (Ron) <forge@zsimovan.dev>
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
sources.local_path is machine-specific state in a brain-wide table. Any
brain whose sources were registered from more than one machine — or a
sanctioned setup mid-migration (topologies.md Topology 2, or the
system-of-record git flow before every repo is cloned) — has sources
whose checkout is not present on the machine running sync --all. Each
surfaced as a hard failure and forced rc=1 every run; on one observed
fleet that was 12 phantom failures per hour, training operators to
ignore the exit code.
--missing-path skip classifies them honestly: ⊘ in the human aggregate,
status skipped_missing_path + local_path in the --json envelope, new
skipped_count, excluded from error_count and the rc=1 gate. Using the
flag outside --all warns instead of silently no-oping.
Default stays fail: on a single-machine brain a missing local_path
usually means an unmounted volume or deleted checkout, and silently
skipping would hide data loss. Skip is explicit opt-in.
Pure helpers (parseMissingPathMode, partitionMissingPathSources)
exported and unit-tested in the sync-all-parallel style — no DB, no fs.
Docs: sync --help, docs/TESTING.md inventory, KEY_FILES.md sync entry.
CHANGELOG/VERSION deliberately untouched per the release process.
Co-authored-by: Ziggy <lazyclaw137@gmail.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Lazydayz137 <Lazydayz137@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(sources): recover corrupted config shapes (#3401)
Use one canonical normalizer for nested string and array-shaped source configs across federation reads, config writes, archive/restore, and doctor remediation.\n\nFixes #3401\nFixes #3402\nFixes #3403
Signed-off-by: arisgysel-design <arisgysel-design@users.noreply.github.com>
* fix(sources): bind restoreSource federated patch via ::text::jsonb (#2339 class)
restoreSource bound a JS JSON string to a bare $1::jsonb placeholder;
postgres.js double-encodes that into a jsonb string scalar, so on the
Postgres engine the coerced object || string-scalar concat evaluates as
array-concat and restore RE-CORRUPTS the exact config shape this PR
repairs. PGLite masks the bug (its driver parses the bind natively).
Fix: bind through $1::text::jsonb per the repo JSONB rule.
Adds the DATABASE_URL-gated Postgres regression
(test/e2e/restore-source-config-jsonb-postgres.test.ts): seeds a
corrupted string-scalar config, runs archive -> restore, asserts
jsonb_typeof(config) = 'object' with the federated flag applied and
pre-existing keys preserved. Verified red on the bare ::jsonb bind
(config became a jsonb array) and green on the fix against a real
pgvector Postgres; skips cleanly without DATABASE_URL.
Co-authored-by: arisgysel-design <arisgysel-design@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Signed-off-by: arisgysel-design <arisgysel-design@users.noreply.github.com>
Co-authored-by: arisgysel-design <arisgysel-design@users.noreply.github.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(onboard): resolve pack checks with file config
* test(onboard): sandbox GBRAIN_HOME in pre-existing pack-check tests
The fix routes checkPackUpgradeAvailable/checkTypeProliferation through
loadConfigFileOnly(), so the file's pre-existing tests now read the real
~/.gbrain/config.json and fail on any machine whose config sets
schema_pack. Wrap them in withEnv({ GBRAIN_HOME: emptyHome(), ... }),
matching the new test's idiom.
Co-authored-by: javieraldape <javieraldape@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: gbrain-contrib <gbrain-contrib@example.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(exports): expose runThink synthesis via gbrain/think subpath
The think synthesis pipeline (runThink, stripGapsSection, persistSynthesis,
maxOutputTokensFor + the ThinkResult/ParsedCitation types) lives in
src/core/think/index.ts but is not reachable through the public exports
map. Downstream consumers importing `gbrain/think` fail to resolve it, and
no other exported entrypoint re-exports runThink.
Add `./think` to package.json exports and extend the public-exports
contract test (count 20 -> 21; new EXPECTED_EXPORTS row with runtime
canaries runThink + stripGapsSection). Test passes 38/38.
Left the VERSION / package.json version / CHANGELOG / llms bumps to the
maintainer /ship flow to avoid colliding with the version-queue allocator.
* fix(ci): bump public-exports guard baseline to 21 for gbrain/think
The new ./think subpath grows the exports map to 21 entries;
scripts/check-exports-count.sh still pinned EXPECTED_COUNT=20 and
exits 1 on growth, failing CI.
Co-authored-by: mnemonik-dev <dev@mnemonik.xyz>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
#2775 reported that `gbrain init --migrate-only` fails with
`column "event_page_id" does not exist` on PGLite brains predating
migration v121, because PGLiteEngine#initSchema() replayed the embedded
schema blob (which indexes timeline_entries.event_page_id) before
runMigrations() could add the column.
That ordering bug was already fixed on master by #2735 (which resolved
the Postgres-side report of the same bug, #2724) via a forward-reference
bootstrap probe in both pglite-engine.ts and postgres-engine.ts, with
coverage in test/bootstrap.test.ts and
test/schema-bootstrap-coverage.test.ts.
Add a regression test at the actual CLI-facing entry point
(runMigrateOnlyCore, what `gbrain init --migrate-only` calls) against a
downgraded pre-v121 brain, closing the gap between the existing
engine-method-level tests and the command users actually run. Verified
this test fails with the exact reported error when the bootstrap probe
is neutralized, and passes with it in place.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
The ollama recipe declared a single `default_dims: 768` (nomic-embed-text's
width) while serving models spanning 384..4096. Every non-nomic model
resolved to 768, so `gbrain init --embedding-model ollama:bge-m3` built a
768-wide `content_chunks.embedding` column for a model that emits 1024. The
schema looked fine and only failed at first insert with
`expected 768 dimensions, not 1024`.
Adds an optional `model_dims` map to `EmbeddingTouchpoint` and an
`embeddingDimsForModel()` resolver that prefers the per-model entry and falls
back to `default_dims`. The ollama recipe declares real widths for the models
it lists; bge-m3 is added to that list. The three `init` call sites that read
`default_dims` now resolve per model.
Partial by design: unlisted models still fall back to `default_dims`, and
`trust_custom_dims` keeps an explicit `--embedding-dimensions` override
working. `user_provided_models` recipes (litellm, llama-server) still resolve
to 0, so they continue to require explicit dimensions.
Verified end to end against an OpenAI-compatible stub standing in for Ollama,
using an isolated GBRAIN_HOME:
before: config 768, content_chunks.embedding vector(768), insert fails
after: config 1024, content_chunks.embedding vector(1024), insert succeeds
* fix(patterns): make reflections/patterns slug sub-paths configurable
gatherReflections()'s SQL WHERE clause and the pattern-page write slug
were hardcoded to wiki/personal/reflections/ and wiki/personal/patterns/
respectively. A prior fix (#2415/#2939) made the leading namespace root
configurable via dream.synthesize.output_root, but the personal/reflections
and personal/patterns sub-path segments stayed pinned literals, so brains
whose schema has no personal/ nesting (e.g. a flat meetings/ tree) could
not point the phase at their own compiled_truth source.
Adds two new config keys:
- dream.patterns.source_slug_prefix (default: <output_root>/personal/reflections)
- dream.patterns.output_slug_prefix (default: <output_root>/personal/patterns)
Both default to the exact literal the code previously hardcoded, so
existing installs see no behavior change. A custom output_slug_prefix is
also added to the subagent's put_page allow-list, since the filing-rules
JSON globs only remap the wiki/personal/patterns/* literal by output_root
and would otherwise reject writes to a differently-shaped output path.
Updated test/cycle-patterns.test.ts's scope-filter assertions to match;
added coverage for the two new config keys and the allow-list addition.
* fix(patterns): drain PGLite subagent job inline (no worker claims it)
runPhasePatterns submitted a subagent job via queue.add() and waited on
it via waitForCompletion, but on PGLite there is no separate Minions
worker process (the embedded data-dir holds an exclusive file lock;
'gbrain jobs work' refuses to start against it). synthesize.ts already
has runPgliteSubagentsInline to drive the claim -> run -> complete loop
inline for exactly this reason; patterns.ts never called it, so a real
(non-dry-run) invocation against a PGLite brain always hung until
subagentWaitTimeoutMs (default 35 min) with the job stuck in 'waiting'.
Exports runPgliteSubagentsInline from synthesize.ts (was test-only via
__testing) and calls it from patterns.ts with the same private
per-run childQueueName derivation synthesize.ts uses, so the inline
drain never claims unrelated 'default'-queue jobs a Postgres worker
owns.
Updated test/cycle-patterns-child-outcome.test.ts's #2782 regression
test: its premise (no worker running with a 1ms wait timeout, so the
job never completes and waitForCompletion genuinely times out) is
exactly the scenario this fix addresses. With the inline drain, a fake
ANTHROPIC_API_KEY test fixture now gets claimed and actually attempted,
failing fast and landing the job in 'dead' rather than staying
uncompleted until a timeout. The #2782 status-reflects-outcome contract
the test exists to pin is unchanged (any non-'complete' outcome with
zero writes still surfaces as status 'fail'); updated the expected
outcome/error code to match the outcome that now actually occurs.
* feat(think): surface usage/cost_usd in --json output
think's own cost was previously unsurfaced anywhere: not in this CLI's
own --json output, not in budget_ledger (nothing in src/core/think/*.ts
ever writes to it), and invisible to a wrapping caller's own token
accounting since the LLM call think makes is its own, separate API
call from anything the caller's session tracks.
runThink() already captured result.usage.{input_tokens,output_tokens}
from the underlying client.create() call but discarded it. Adds
usage/cost_usd to ThinkResult, populates usage on the real-LLM-call
path (undefined on the no-client/stub paths, matching how synthesisOk
already distinguishes those), and computes cost_usd in think.ts's CLI
handler via the existing canonicalLookup() pricing table (same pattern
brain-score-recommendations.ts's estimateAnthropicCost already uses).
Extracted the multiply-and-sum into a small exported computeThinkCostUsd
for direct unit testing. Also appends the cost to the human-readable
footer.
Verified live: gbrain think --json against a real anchor returned
usage:{input_tokens:3271,output_tokens:1490}, cost_usd:0.0536, matching
Opus pricing ($5/$25 per MTok) by hand calculation.
Two robustness fixes to `gbrain autopilot --install`/`--status`, hardening #3305.
1. Universal bun PATH (extends #3305). The install-generated wrapper
(~/.gbrain/autopilot-run.sh) execs the `#!/usr/bin/env bun` gbrain shim, so
bun must be on PATH under cron/systemd/launchd's minimal env. #3305 hardcodes
`$HOME/.bun/bin`, which only covers the default bun.sh installer. Hosts where
bun lives elsewhere (Homebrew, npm -g, Docker /usr/local/bin, custom
BUN_INSTALL, nix) still die with `env: bun: No such file or directory`,
leaving a stale lock that stalls the nightly cycle. Fix: bake the dir of the
actually-running bun (dirname(process.execPath)) onto PATH at install time,
~/.bun/bin kept as fallback, single-quote-escaped, empty execPath guarded.
2. `--status` false negative. showStatus() checked crontab.includes('gbrain
autopilot'), but --install writes a line calling the wrapper
`.../autopilot-run.sh` — no such substring. So `--status` reported
installed:false on every wrapper-based Linux host. Fix: also match
'autopilot-run.sh'.
Tests: test/autopilot-install.test.ts — universal-form + runtime-derivation +
wrapper-detection assertions (fail-before/pass-after verified).
* v0.42.67.0 fix(agent): provider-neutral help + one truthiness parser for the gateway-loop toggle (#2753)
The gbrain agent help described --model as Anthropic-only and named only
ANTHROPIC_API_KEY. It also overclaimed that any recipe works and that MCP
submitters get permission_denied.
Reviewing that turned up a live mismatch: the doctor accepted true/1/yes/on
for agent.use_gateway_loop, the subagent worker accepted only true/1. So
config set ... yes reported healthy and still refused the job. Both now share
isConfigTruthy() in src/core/config.ts.
Item 1 of the issue (registering the key) is already on master, so this scopes
to the help text, the parser, and the regression test.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* drop VERSION/package.json/CHANGELOG bump — contributor PRs in this repo do not carry it
Checked precedent on my own merged PRs (#3253, #3248, #3241, #3236): none
touch VERSION, package.json or CHANGELOG. The version-first title + 5-file
sync rule in CLAUDE.md is the maintainer ship flow, not the contributor path.
Carrying the bump here would just hand the maintainer a guaranteed conflict
on every merge.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* skillify: make the Phase 0 gate fail closed
The gate only rejected when all three answers were no, but each
criterion's parenthetical reads as individually disqualifying
("One-off work != skill"). A one-line alias used once answers
No/No/Yes and runs the entire pipeline - up to 9 frontier eval
calls, four test layers, resolver wiring - and gets certified
properly skilled.
Any single no now stops the run, with the forbidden follow-on
work enumerated so executors cannot rationalize past it.
* skillify: add an upper-bound scope check to Phase 0
Phase 0 only guarded the lower bound (one-off, trivial), so an
entire multi-feature subsystem answered yes to all three checks
and became one mega-skill. In that shape the cross-modal eval
diagnoses the problem (every model says split it) but no phase
can act on the advice - decomposition is not a file edit - so
the only path is ship-with-KNOWN_GAPS, and Phase 4 then locks
the below-bar scope in with tests: the exact tests-cement-
mediocrity outcome the eval gate exists to prevent.
Multi-intent targets now stop in Phase 0 with a proposed split
and a question about which target to skillify first.
The check asks about the set of intents rather than the
existence of a trigger phrase, because check 3 is existential
and any one phrase ("ship it") makes a subsystem answer yes.
Anthropic released Claude Opus 5, at the same $5/$25 pricing tier as
Opus 4.8. Neither the chat recipe allowlist nor CANONICAL_PRICING knew
about it, so operators could not opt into it via models.tier.deep /
models.default without gbrain rejecting the id.
- src/core/ai/recipes/anthropic.ts: add claude-opus-5 to the models list.
- src/core/model-pricing.ts: add anthropic:claude-opus-5 { input: 5.00,
output: 25.00 } (plus cache rates, matching Opus 4.8's ratios).
- src/core/takes-quality-eval/pricing.ts: add it to SUPPORTED_MODELS so
eval takes-quality run --budget-usd doesn't reject it during preflight.
- Refreshed the stale pricing-verification date and the Opus list in
docs/architecture/KEY_FILES.md.
- Tests: pinned-value regression in test/model-pricing.test.ts, recipe
membership in test/anthropic-model-ids.test.ts, budget-pricing coverage
in test/eval-takes-quality-pricing.test.ts.
Scope: registration only. TIER_DEFAULTS / DEFAULT_ALIASES /
DEFAULT_CHAT_MODEL are untouched — default-routing bumps are the
separate, already-open #2858; this just makes the id valid/priced for
operators who opt in explicitly.
* chore(ci): refresh GitHub Actions SHA pins (checkout v4, action-gh-release v2)
Pre-ship pin staleness check per docs/RELEASING.md: both floating major
tags moved upstream; pins updated to the current tag commits.
* v0.42.65.0 chore(release): 92 verified fixes since v0.42.64.0 — changelog + version bump
Aggregates everything merged to master since the v0.42.64.0 bump commit:
community fixes, credited takeovers, batch re-lands, CI hardening, and
maintainer-approved features. Net commit list excludes revert pairs.
No new schema migrations.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(deps): clear OSV-flagged transitive dependencies via override floors
Raise the existing security-floor overrides so the lockfile resolves
patched versions of three transitive packages flagged by the OSV scan
(@hono/node-server, fast-uri, body-parser). None are on gbrain's own
runtime path (@hono/node-server is only referenced by the MCP SDK's
optional hono transport, which gbrain does not load); the floors keep
the dependency scan green. MCP/OAuth unit tests pass against the
resolved versions.
* chore(release): fold #3110 into the v0.42.65.0 entry (93 net changes)
* v0.42.66.0 chore(release): 54 verified fixes since v0.42.65.0 — changelog + version bump
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
list_pages clamps limit to max 100 (default 50) — deliberate server
protection, pinned in test/search-limit.test.ts. But the clamp was
SILENT: a caller whose limit was defaulted or clamped got a
full-looking array with no signal that rows were dropped, and with the
default updated_desc sort the dropped rows are always the OLDEST —
precisely what exhaustive consumers (audits, scans, backfills) exist
to find. Observed in the field: a source with 212 pages enumerated as
80 visible rows, hiding 26 pages from a compliance scan for days.
Fix, with no response-shape change (MCP consumers still get an array)
and no engine surface change (handler probes limit+1):
- handler probes one row past the effective limit; when the caller's
limit was NOT honored (unset -> default, or clamped to cap) and rows
were dropped, it warns on stderr for local (CLI) callers — same
operator-facing channel as the put_page unknown-type hint, but
without the isTTY gate: scripted callers are exactly the consumers
that cannot detect truncation any other way, and stderr keeps stdout
parseable. An explicit honored limit stays silent (ordinary
pagination), as does a clamped-but-complete result. Remote (MCP)
ctx never writes to stderr.
- LIST_PAGES_DESCRIPTION documents the cap and the exhaustive-listing
recipe (sort=updated_asc + updated_after cursor) — the description
is the signal channel MCP clients actually read.
- regression suite: default-limit truncation warns, honored limit
silent, clamped-but-complete silent, remote silent, and the
documented cursor recipe enumerates a corpus to completion.
Co-authored-by: paul-0320 <paul@ymyd.co.kr>
Co-authored-by: YMYD <53603073+OJ-OnJourney@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
* v0.42.66.0 fix(extract): make conversation backfill outcomes durable (takeover of #3293)
Versioned, snapshot-bound terminal audit rows become the durable authority
for conversation fact backfill completion; checkpoint GC can no longer
repeat completed model work, and best-effort empty results no longer mask
provider/output failures as complete.
Supersedes #3293 (rebased onto current master; only version-trio conflicts).
Co-authored-by: FloridaStyle <danwiggins@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: drop version-trio bump — individual fixes do not carry version bumps (release PRs do)
* merge: reconcile durable-outcome skip accounting with master's LLM fallback tests
The two fallback replay tests from #3371 asserted the legacy checkpoint
pages_skipped counter; under this PR's durable-outcome authority a
completed page is skipped via pages_skipped_completed before any parse.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: FloridaStyle <danwiggins@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(recipes): add reranker touchpoint to OpenRouter (#2164)
OpenRouter's POST /api/v1/rerank is wire-compatible with gateway.rerank()
({query, documents, model} → {results: [{index, relevance_score}]}). This
adds a recipe-only reranker touchpoint declaring four models:
- cohere/rerank-v3.5 (default; $0.001/search)
- cohere/rerank-4-fast ($0.002/search, 32K context)
- cohere/rerank-4-pro ($0.0025/search, SOTA quality)
- nvidia/llama-nemotron-rerank-vl-1b-v2:free (multimodal)
Unlike embedding/chat, the reranker path strictly enforces the models
allowlist — the openai-compat extended-model bypass does not apply. New
rerank models must be added to this recipe before they can be called.
The cost_per_1m_tokens_usd value is a pseudo-rate for the budget tracker's
chars/4 heuristic — Cohere bills per-search, not per-token. At ~4K chars
the estimated cost is in the right ballpark.
Recipe-only change; no gateway or search-layer modifications. gateway
auto-concatenates path → .../api/v1/rerank.
Adds hermetic unit test (test/openrouter-reranker-recipe.test.ts) covering
shape, models, default_model, path, max_payload_bytes, default_timeout_ms,
and cost field. No DB, no env mutation — survives the parallel 8-shard
fan-out.
Verified: bun run verify (30/30 green); 285 targeted recipe+rerank+budget
tests pass.
Co-authored-by: Hippityy <Hippityy@users.noreply.github.com>
* test(facts): pin gateway to 1536d in facts-engine.test.ts beforeAll
Shard-composition hermeticity fix. The legacy preload's beforeEach only
re-applies the 1536-d gateway default before each TEST, not before a
file's beforeAll — so when the previous file in the shard resets the
gateway in its teardown (e.g. test/providers-test-model-base-url.test.ts
via afterEach), this file's initSchema() sized facts.embedding at the
1280-d production default and the 1536-d fixture inserts threw
'expected 1280 dimensions, not 1536' (CI shard 1 failure on #3302).
Same pattern as test/consolidate-valid-until.test.ts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Ryan Xie <64182766+Hippityy@users.noreply.github.com>
Co-authored-by: Hippityy <Hippityy@users.noreply.github.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
When link_resolution.global_basename is enabled, extend basename-index
resolution to frontmatter link fields (FRONTMATTER_LINK_MAP), mirroring the
body bare-wikilink path added in #972.
Problem: a bare-title wikilink in a frontmatter list -- e.g.
sources:
- "[[2025-12-25_mentor-extraction]]"
never resolves. SlugResolver.resolve() has no '/' to hit the slug-direct
getPage, and the field's dirHint (sources -> ['source','media']) may name
folders absent from the brain, so the dir-scoped exact + fuzzy steps also
miss. The frontmatter path never consulted resolveBasenameMatches -- that was
wired only for body bare-wikilinks. On a PARA/Obsidian vault this silently
drops the bulk of sources:/related: provenance edges.
Fix: extractFrontmatterLinks takes a globalBasename flag (threaded from
extractPageLinks). On a resolve() miss, unwrap [[ ]] and fall back to
resolver.resolveBasenameMatches -- UNIQUE-MATCH-ONLY, so ambiguous basenames
(archive dupes, generic hubs like _index) stay unresolved rather than create
a wrong edge. Purely additive; resolved frontmatter edges are unchanged.
Scope: covers the db-source extract and live put_page paths (real
makeResolver). The --source fs extract uses an inline resolver without a
basename index, so it gracefully no-ops there (typeof guard).
Tested: 3 new cases (resolves-when-on, ambiguous-stays-unresolved,
gated-off-by-flag); full link-extraction suite green (130 pass).
Co-authored-by: spiky02plateau <155588579+spiky02plateau@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
* fix(onboard): stop repeating the same auto-remediation within a run (#2854)
When the recommendation list is refreshed between remediation steps, a
remediation that doesn't clear its own health signal is reintroduced
under its stable id and attempted again, indefinitely on long runs.
Track attempted recommendation ids for the run and skip re-attempts.
Includes a behavioral regression test: a persistently-stuck signal is
attempted once, the loop terminates, and other remediations still run.
* fix(test): quarantine remediation-run-loop test as serial + complete BrainHealth fixture
Two CI failures, one root cause each:
- verify (check:test-isolation + typecheck): the new test uses mock.module
(R2) so it must live in the *.serial.test.ts quarantine, and the
BrainHealth fixture was missing the now-required linkable_page_count.
- test (6): the top-level mock.module('../src/core/ai/gateway.ts') leaked
into other files in the parallel shard process, flaking
test/ai/adaptive-embed-batch.test.ts. Serial quarantine fixes it —
run-serial-tests.sh executes each serial file in its own bun process.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Sanchal Ranjan <84386862+sanchalr@users.noreply.github.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Dispatch timeout was derived as interval*2 with a 5-minute floor, tuned
for light per-interval work. A full autopilot cycle routinely needs more
than 10 minutes at common intervals, so healthy full cycles were killed
mid-run. Full-cycle dispatch now gets a 30-minute floor; lighter
dispatches keep the interval-derived budget.
Adds a regression test for the full-cycle floor.
Co-authored-by: Sanchal Ranjan <84386862+sanchalr@users.noreply.github.com>
* fix(serve): boot-readiness deadline releases PGLite lock on wedged boot (#3273)
A serve process that wedges mid-boot (e.g. a boot step blocked on an
unreachable upstream) held the PGLite write lock indefinitely — the
post-#2348 lock discipline never steals from a live holder, so every CLI
consumer timed out until the serve PID was manually killed.
runServe (stdio path) now arms a boot-readiness deadline around
startMcpServer: if the transport hasn't connected within
GBRAIN_SERVE_BOOT_TIMEOUT_SECONDS (default 60, 0 disables), it logs the
condition, awaits engine.disconnect() (raced against the existing
5s cleanup deadline so a wedged WASM close can't trap it either), and
exits non-zero so supervisors restart with backoff. A completed boot
clears the timer; the HTTP path is untouched (own lifecycle).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(e2e): 60s hook timeout for jsonb-parity setup/teardown
The #2339 parity guard's beforeAll runs setupDB (full migration chain)
under bun's default 5s hook timeout, which flaked on a slow CI runner
(setupDB hit 5001ms). Other e2e suites already pass explicit hook
timeouts; bring this file in line.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
`sources.config` is a jsonb OBJECT column, but a read→write cycle that
JSON.stringify'd an already-stringified value re-wrapped it into a JSON string
scalar ("{}", "\"{}\"", ...) that grew one layer per write. parseSourceConfig
only unwrapped one layer, so the corruption never healed and federation/ACL
reads saw a string instead of the settings object.
- Add normalizeSourceConfig: a bounded (10-iteration) loop that JSON.parses
while the value is a string and returns {} (with a console.warn) when the
result is not a plain object. All six `UPDATE sources SET config` writers run
their config through it before stringify, converging the stored value back to
a jsonb object on the next write.
- parseSourceConfig now does the same bounded unwrap and warns once when more
than one layer was found (one layer is the normal PGLite path).
- Add a `source_config_shape` doctor check that flags any sources row where
jsonb_typeof(config) <> 'object', with the repair path.
- Unit-test the helper (object passthrough, 1-layer, 5-layer nested, garbage
and over-bound inputs) and the doctor check (mock engine).
Co-authored-by: 1alessio <alessio.sulpizi@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
gbrain list --limit 100000 silently returned 100 rows (default 50) with
no warning, and --offset was accepted but dropped at the op layer even
though PageFilters has supported it all along.
- Local CLI callers (ctx.remote === false, the same trust boundary that
already bypasses scope enforcement) get an explicit limit above 100
honored — full enumeration is a legitimate local operation.
- Remote MCP/OAuth callers keep the 100-row DoS cap, now loud: one
logger.warn (stderr, stdout stays script-clean) with both numbers,
parity with the three search-path clamp warnings.
- offset is declared as a param (so the CLI coerces it to number) and
threaded to engine.listPages for real pagination.
Claude-Session: https://claude.ai/code/session_01Vswwe1y5fQbJWfbaSK3enT
Co-authored-by: Deacon Bot Doctor <deacon@botdoctor.io>
Co-authored-by: deacon-botdoctor <291411030+deacon-botdoctor@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The idempotency row is only written inside `for (const p of proposals)`, so a
page that extracts ZERO gradeable claims never records an idempotency tuple
and is re-sent to the LLM on every cycle forever. The docstring's "unchanged
page never re-spends tokens" contract only holds for pages that produce >=1
claim; a page that legitimately has no gradeable claims (or any machine-
generated page) is a perpetual cache miss and re-spends tokens indefinitely.
Fix: when `proposals.length === 0`, write one tombstone row keyed by the same
(source_id, page_slug, content_hash, prompt_version) tuple, with
status='rejected' so it never surfaces in a pending-review query (the pending
index filters status='pending'). Content changes (new content_hash) or a
PROPOSE_TAKES_PROMPT_VERSION bump still miss the tombstone and re-extract. The
extractor-throw path `continue`s before the tombstone, so failed pages are
retried rather than cached.
Guard against a subtle regression: `parseExtractorOutput` returns [] for BOTH
a genuine empty extraction AND malformed/prose/truncated model output, so
naively tombstoning every [] would permanently suppress a page that has claims
but hit a transient parse failure. `defaultExtractor` now throws when the
output is empty-but-not-a-clean-`[]` (new `isWellFormedEmptyExtraction`
predicate), routing transient failures into the existing retry path; only a
cleanly-parsed empty array is memoized.
Adds a `tombstones_written` counter for observability.
Tests: tombstone written on genuine empty extraction; two-cycle idempotency
(no repeat LLM call on an unchanged zero-claim page); extractor error writes
no tombstone; isWellFormedEmptyExtraction discriminates clean-[] from
malformed/prose/non-empty output. propose-takes suite: 36 pass / 0 fail.
Co-authored-by: ivandebot <ivanlanlei@gmail.com>
Co-authored-by: ivandebot <187176982+ivandebot@users.noreply.github.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(search): honor recency decay config on the hybrid path (#2386)
The hybrid recency stage in runPostFusionStages imported
DEFAULT_RECENCY_DECAY directly, so operator overrides via the
GBRAIN_RECENCY_DECAY env var and the gbrain.yml `recency:` section were
honored only on the get_recent_salience SQL path and silently ignored on
the hot hybridSearch path. Non-default vault layouts therefore stayed on
the baked-in defaults / DEFAULT_FALLBACK (90d / 0.5) regardless of
tuning.
Call resolveRecencyDecayMap() (already used by the SQL path) so the
configured decay map reaches the boost stage. Behavior is unchanged when
no override is set — resolveRecencyDecayMap() returns DEFAULT_RECENCY_DECAY.
Adds test/hybrid-recency-config.test.ts asserting the env override
reaches the applied recency factor (fails against the prior wiring).
* test: use withEnv() in hybrid-recency-config test (check-test-isolation R1)
The test-isolation lint (shipped after #2386 was written) rejects raw
process.env mutation in non-serial test files. Wrap the
GBRAIN_RECENCY_DECAY overrides in withEnv() from test/helpers/with-env.ts;
assertions unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Richard Baker <rich@rwbaker.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(frontmatter): derive validate slug from brain root, not absolute path (#2340)
Single-file `frontmatter validate` derived the expected slug from the
absolute path: relative(resolve(target), file) is empty when target IS the
file, so it fell back to `|| file` (the full path), yielding "root/<abs>"
slugs and a false SLUG_MISMATCH. The pre-commit hook from install-hook
validates staged files one-by-one, so this rejected every commit in a
markdown brain (only bypassable with --no-verify).
Walk up to the brain root (nearest .git) and use relative(brainRoot, file)
|| basename(file), matching runAudit/runGenerate and sync/extract. Files
above the root fall back to basename instead of a ../-prefixed slug.
Reopens#565. Present since v0.32.0; reproduced on v0.42.51.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(facts): pin embedding dims in facts-engine — kill the shard-order 1280/1536 flake
facts-engine.test.ts hardcodes Float32Array(1536) vectors (vec()) but lets
initSchema size its vector columns from process-global gateway state
(getEmbeddingDimensions(), default 1280). Whether the file passes depends
on which test files run before it in the shard; adding
test/frontmatter-validate-slug-565.test.ts reshuffled the weight-packed
shards and tripped it on this PR's CI (test (1):
'expected 1280 dimensions, not 1536' in findCandidateDuplicates cosine
ordering).
Same fix + rationale as doctor-hidden-by-search-policy.test.ts (#2801),
engine-find-trajectory.test.ts and cosine-rescore-column.test.ts:
configureGateway(1536) in beforeAll BEFORE initSchema, resetGateway in
afterAll.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: alessioalionco <alessioalionco@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
* fix(autopilot): export ~/.bun/bin onto PATH in cron wrapper (#2013)
The wrapper script that 'gbrain autopilot --install' writes to
~/.gbrain/autopilot-run.sh sources ~/.bashrc to inherit PATH for the
exec'd gbrain binary (which has a '#!/usr/bin/env bun' shebang). The
standard Debian/Ubuntu ~/.bashrc ships a non-interactive guard that
returns early when bash is launched non-interactively (cron, launchd,
systemd) — so PATH exports operators add to ~/.bashrc never reach the
wrapper subprocess.
The result: the wrapper dies silently with 'env: bun: No such file or
directory', leaves a stale lockfile, and every subsequent cron tick
hits the lockfile and bails. The nightly dream cycle hangs waiting on
a worker that never comes back, and the wrapper's own 10-min
stale-lock window is the only thing that can recover it.
This bites every operator whose bashrc is the standard distro default
(which is the default), and there is no warning at install time.
Fix: prepend ~/.bun/bin to PATH directly in the wrapper, so it is
self-contained regardless of which init file the OS loaded. Add a
regression test alongside the existing zshenv/zshrc source-order test
(v0.36.1.x #966) so this class of bug stays caught.
* fix(test): scrub real agent-fork name from regression comment (privacy check)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: klampatech <73077262+klampatech@users.noreply.github.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Idempotency was keyed on atom rows alone — a page the LLM judges
un-atomizable leaves no row, so it re-entered the discovery window every
run. Two production consequences: --drain false-stopped with
no_progress once the window head was mostly zero-yield pages (remaining
frozen while batches report +0), and every nightly re-spent extraction
budget on the same pages.
Fix:
- After a SUCCESSFUL chat call that parses to zero atoms, stamp the
source page with frontmatter.atoms_scan_hash = contentHash16. LLM
failures take the catch path and stay retryable.
- discoverExtractablePages + countExtractAtomsBacklog (both variants)
exclude pages whose stamp matches the CURRENT content hash prefix —
content edits re-eligibilize, mirroring atom-row staleness semantics.
- Drain no_progress now recounts the backlog on a zero-atom batch and
only stops when it genuinely didn't shrink — tombstoning IS progress.
Tests: +2 pure-loop drain cases (shrinking backlog continues / flat
backlog stops) and +3 PGLite integration cases (stamp + exclusion /
content-change re-eligibility / failed chat does not stamp).
29 pass / 0 fail across the two files; tsc clean.
Co-authored-by: 陈源泉 <84364275+ChenyqThu@users.noreply.github.com>
Co-authored-by: 陈源泉 <chenyuanquan@chenyuanquandeMac-mini.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
Adds the bold-time-dash built-in pattern: **Speaker** HH:MM <dash> text
(em dash, en dash, or ASCII hyphen), valid 24-hour times only, date from
page frontmatter/date headings, multi-line continuation bodies.
Opt-in score_continuations_as_body scoring keeps long multiline messages
parseable while preserving the sparse-prose false-positive floor (needs
two anchors or a first-line anchor before candidate-only scoring kicks in).
Hardens validatePatternEntry to reject non-integer / out-of-range capture
indexes including text_group. Adds maintainer doc, JSONL fixtures, and
adversarial coverage.
Takeover of #3289 (fork branch went CONFLICTING against master on the
version trio); code applied 3-way, version/CHANGELOG bump dropped per
fleet release convention.
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: FloridaStyle <daniel.wiggins@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
On machines with neither gtimeout nor timeout on PATH, run-verify-parallel.sh
and run-unit-parallel.sh fall back to a bg-pid + sleep-watchdog cap. Both
read $? only after tearing the watchdog down (kill + wait on cap_pid), so the
sentinel .exit files recorded the killed watchdog's status — 143 — instead of
the check/shard's own exit code. Every run reported total failure (verify:
pass=0 fail=31; unit: rc=143 per shard) while every per-check/shard log
showed success.
Capture rc immediately after `wait $pid` in both scripts, and reap the
watchdog's sleep child (pkill -P, children-first — the same orphan quirk the
heartbeat cleanup documents) so the fallback stops leaking one sleep per
check/shard.
Regression tests force the fallback branch hermetically on any host via a
curated PATH with no timeout binaries: the verify dispatcher runs from a
tempdir copy with a stubbed `bun`, pinning exit 0 + all-zero sentinels when
checks pass and the check's own rc (not 143) when one fails; the unit wrapper
runs real two-shard fixture passes, pinning rc=0 sentinels and a real
failure's rc=1.
Co-authored-by: paul-0320 <paul@ymyd.co.kr>
Co-authored-by: YMYD <53603073+OJ-OnJourney@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(embed): stamp gateway-resolved model in content_chunks.model, not compiled default (#2846)
upsertChunks fell back to the compile-time DEFAULT_EMBEDDING_MODEL
('zeroentropyai:zembed-1') when a ChunkInput carried no explicit `model`.
The embed pipeline (src/commands/embed.ts) builds ChunkInputs without a
`model` field, so rows whose vectors were produced by the config-resolved
model (e.g. openai:text-embedding-3-large) were mislabeled with the
hardcoded default — corrupting the provenance that signature-drift
staleness and dimension-migration logic depend on.
Both engines now resolve the gateway's runtime embedding model once per
upsert and use it as the fallback, mirroring the existing resolve-then-
default pattern used for schema sizing. Regression test added (pglite);
verified via negative control that it fails against the old fallback.
This is a write-path change (upsertChunks), not a search-path change, so
retrieval eval replay is not applicable.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* test: Lane A.7 pins gateway-resolved chunk model, not compiled default
#2846 changed upsertChunks' fallback from DEFAULT_EMBEDDING_MODEL to the
gateway-resolved runtime model. Lane A.7 still pinned the old fallback,
and the test preload (test/helpers/legacy-embedding-preload.ts) pins the
gateway to openai:text-embedding-3-large for every test process — so the
original #2846 landing failed this test deterministically and got batch-
reverted. The test now asserts the resolved model (the intended #2846
semantics) while keeping the CDX2-4 bare-literal regression guard.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: SailorJoe6 <SailorJoe6@Gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
DashScope's OpenAI-compatible rerank endpoint lives at
{base}/compatible-api/v1/reranks — PLURAL leaf, different base path from
the embedding surface (compatible-mode). Reusing llama-server-reranker
against DashScope forces users to hand-patch the recipe's '/rerank' leaf
in node_modules, which every upgrade silently reverts (and llama.cpp
genuinely serves singular /rerank, so changing that recipe would break
real llama.cpp users).
New dedicated recipe rides the v0.40.6.1 recipe-pluggable reranker path:
- id dashscope-rerank, base_url_default compatible-api/v1 (intl), ZE wire
- path '/reranks', default_timeout_ms 30s, 5MB payload ceiling
- models: only qwen3-rerank (live-verified 200; gte-rerank-v2 is rejected
by the compat surface with 'Unsupported model for OpenAI compatibility
mode', so it is deliberately not listed)
- separate recipe (not a reranker touchpoint on dashscope) because
provider_base_urls is keyed by recipe id and the two capabilities need
different prefixes — same topology as llama-server vs
llama-server-reranker
Tests: recipe shape smoke mirroring recipe-llama-server-reranker.test.ts
(path/timeout/payload pins, /v1/v1 concat guard, auth resolve, sibling
recipe isolation). bun test test/ai/: 322 pass / 0 fail.
Co-authored-by: Yicon <charlieyiconghuang@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The four `hybridSearch — reranker enabled (reorder)` cases stub the gateway
at 1536 dims (DIMS). Since v0.36.3.0 hybridSearch resolves the embedding
column via loadConfig(), whose precedence is
cfg.embedding_dimensions > gateway dims > default. On any machine whose
~/.gbrain/config.json sets embedding_dimensions to something other than 1536
(e.g. text-embedding-3-small at 1280), the real config outranks the stub: the
1536-d stub vector fails the gateway dim check, the error is swallowed, search
falls back to keyword-only, and the reranker never runs (rerankerFn gets 0
docs, rerank_score undefined). Green in CI only because a fresh runner has no
config file — deterministic red on a contributor's machine.
Fix (test-only): isolate GBRAIN_HOME to an empty tmpdir in beforeAll so
loadConfig() returns null and the stub's dims win, then restore it and clean
up in afterAll. Same idiom as emptyHome() in
test/ai/gateway-probe-chat-model.test.ts.
Verified with a planted ~/.gbrain/config.json at 1280 dims: 2 pass / 4 fail
before, 6 pass / 0 fail after; still green with no config file. typecheck clean.
Fixes#1527
Co-authored-by: Willisbest <132954469+Willisbest@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Reland of #2639, reverted with its batch in 68e4cebd. getHealth's
entity_pages CTE and the top-linked-pages query only match the legacy
'person' and 'company' types, so brains using the gbrain-base-v2 pack's
'entity' type report 0% entity link/timeline coverage in `gbrain health`.
Add 'entity' to both queries in both engines (PGLite + Postgres, in
lockstep per the engine-parity rule).
Reland fix (the batch-red root cause): the original PR's test expected
the entities/project-x page to appear in orphan_pages, but #3023's shared
orphan-reporting policy (landed before #2639 merged) excludes the
'entities' first segment from orphan reporting, so the test failed on
master. Orphan expectations now account for the policy exclusion.
Co-authored-by: Tyler Robinson <tylr.rob@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Replace the strictly sequential per-chunk synopsis loop with a bounded
sliding worker pool (existing runSlidingPool helper). Results land in
chunk order via index-addressed writes; code chunks still bypass the
wrapper; embedding remains one page-level batch after all synopses.
New knob GBRAIN_CONTEXTUAL_CHUNK_CONCURRENCY, default 4, clamped to
[1,16]; 1 reproduces the prior sequential behavior exactly. Each chunk
task still acquires/releases the global synopsis rate-lease, which
remains the cross-worker governor; the lease id now travels from
acquire to release instead of shared mutable state, and lease waits
are abort-responsive.
At 20-45s per synopsis call, a 120-chunk transcript page previously
needed 60-90+ min wall time and routinely outlived job timeouts.
Co-authored-by: spiky02plateau <155588579+spiky02plateau@users.noreply.github.com>
statusForVersion short-circuited on any 'complete' entry before checking
the trailing 'retry' marker, so --force-retry appended an inert row and a
version marked complete with zero work done could never be re-run without
hand-editing completed.jsonl. Check retry-latest first: an explicit
--force-retry now yields 'pending' even past an earlier 'complete', while
a stray 'partial' after 'complete' still cannot regress the version.
Co-authored-by: spiky02plateau <155588579+spiky02plateau@users.noreply.github.com>
SupabaseStorage.getSignedUrl built the download URL as `${projectUrl}${signedURL}`,
but Supabase's sign API returns `signedURL` relative to the Storage API root
(/object/sign/<bucket>/<path>?token=...), so the generated link dropped /storage/v1
and returned 404. Now prepends `${projectUrl}/storage/v1`, tolerating an
already-absolute URL or a value that already carries the prefix. `gbrain files
signed-url` links resolve again.
Co-authored-by: FloridaStyle <daniel.wiggins@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Reasoning models (MiniMax-M3, DeepSeek-R1, etc.) return <think>...</think>
tags in the content field before the actual JSON output. This caused
parseExtractorOutput to fail in two ways:
1. The fence regex /^\`\`\`(json)?...$/ requires the fence at text start;
<think> preceding it prevents matching, so the raw text (with trailing
fences) hits JSON.parse and throws.
2. When think tags contain [ or { characters, indexOf finds them inside
the reasoning block instead of the actual JSON array.
Changes:
- Strip <think>...</think> tags before any parsing (covers all reasoning models)
- Add JSON.parse fallback: truncate at last ] or } to handle trailing
noise (leftover markdown fences after stripping)
Tests: 28/28 pass (3 new cases for think tags + trailing noise).
Co-authored-by: qaz8545355 <603191978@qq.com>
Co-authored-by: qaz8545355 <junjun@openclaw.local>
The empty-fence guard counted every `row_num IS NULL AND entity_slug IS NOT NULL`
row as a pending v0_32_2 backfill, but the inline facts writer keeps producing
rows of exactly that shape post-migration: when a resolved slug has no fenceable
page (slugify-floor / stub-guard-blocked unprefixed slugs like `wingman`,
`people-jane-doe`), backstop.ts falls through to a DB-only insert with row_num
NULL. Those rows are structurally unfenceable — no page to fence onto, and the
ledger-complete migration won't re-run — so they jammed the phase forever
(~16/day observed) and the warning advised a no-op `apply-migrations --yes`.
Discriminator: a row is a genuine backfill candidate only if its entity_slug
resolves to a LIVE page in the same source (EXISTS in `pages` with deleted_at
NULL) — mirroring the migration's Phase B, which only fences slugs that map to
a writable page. Genuine pre-v0.32.2 rows (their entity page exists) still gate;
inline-writer unfenceable rows no longer do. Warning text updated to name the
"entity page present, not yet fenced" condition.
Regression tests pin both sides: unfenceable rows (no page / soft-deleted page)
do NOT gate and the phase converges; a legacy row WITH a backing page still
gates. Fails pre-fix, passes post-fix.
(#2484)
Reland note: original merge (53c90869) was batch-reverted (4b6cf32c) —
the guard-semantics change broke test/phantom-redirect.test.ts
'round 2 P1: legacy-row guard fires BEFORE phantom-redirect pass',
which seeded a legacy row WITHOUT a backing page and expected the
guard to fire. Under the new (intended) semantics such a row is
structurally unfenceable and must NOT gate. Fixed by seeding a live
backing page for the legacy row, preserving what the test pins
(guard fires before the phantom-redirect pass).
Co-authored-by: Javier Aldape <javieraldape@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Code legitimately contains tiktoken special-token strings (e.g. CLIP/GPT tokenizers embed the literal <|endoftext|>). The default encode() uses disallowed_special='all' and THROWS on those, crashing reindex-code on valid source files. Re-encode treating them as ordinary text (allowed=[], disallowed=[]); heuristic fallback if even that fails. A token COUNT needs no special-token semantics.
Co-authored-by: Jim Tang <jimruitang@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(write-through): guard mkdir against EEXIST on Bun+Windows
* fix(budget): resolve non-Anthropic model pricing via canonical table under --max-cost
* fix(enrich): exclude dream-generated pages from thin candidates
---------
Co-authored-by: nguyenchiviet <40517873+nguyenchiviet@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
OpenRouter (and potentially other proxy providers) expose OpenAI's
text-embedding-3 models with a provider prefix in the model ID, e.g.
`openai/text-embedding-3-large` rather than bare `text-embedding-3-large`.
`dimsProviderOptions()` checks `modelId.startsWith('text-embedding-3')`
which fails for the prefixed form, so the `dimensions` parameter is never
sent. The upstream provider returns its native dimensionality (3072 for
-large) instead of the configured value (e.g. 1536), causing an immediate
"dim mismatch" error on first embed.
The default OpenRouter embedding (`text-embedding-3-small` at 1536d)
masked this because its native size happens to match the default config.
The bug surfaces when using `-large`, or `-small` with a non-1536 dim
(512, 768, 1024 — all listed in the recipe's `dims_options`).
Fix: strip the provider prefix before the `startsWith` check. The full
prefixed ID is preserved in the error message for user clarity.
Co-authored-by: Noetherly <280958447+noetherly@users.noreply.github.com>
* feat(subagent): claude-cli MessagesClient adapter (baseline, no tool use)
Closes#334 (partially — text-only baseline; tool use lands in the next
commit on this branch).
Adds a MessagesClient adapter that shells out to `claude --print
--output-format json --model <model>` instead of the Anthropic SDK. When
`GBRAIN_USE_CLAUDE_CLI=1` is set, the subagent worker registers the adapter
in place of the SDK client; the default path (Anthropic SDK with
ANTHROPIC_API_KEY) is unchanged when the env var is unset or set to
anything else.
The benefit is that Claude Max subscribers can run Minions subagents
against their existing OAuth subscription, no ANTHROPIC_API_KEY needed.
New: src/core/minions/handlers/claude-cli-adapter.ts
- Implements the MessagesClient interface exported from subagent.ts.
- Strips provider prefixes (`anthropic:`, `litellm:`) from the model id
because `claude --print` only accepts CLI-native aliases (`sonnet`,
`opus`, `haiku`, or the bare `claude-*-N-M` form).
- Flattens the Anthropic messages array into a single text prompt for
claude-cli stdin. Tool blocks (tool_use / tool_result) are stringified
as placeholders so multi-turn conversations stay coherent in this
baseline; native tool_use round-tripping is the follow-up commit.
- Spawns claude with stdio piped, captures stdout, parses the
`{type:"result", subtype:"success", result, usage, ...}` JSON envelope,
and returns it as a properly shaped Anthropic.Message with
`stop_reason: 'end_turn'`.
- Token totals propagate from the claude usage block so the subagent
handler's `ctx.updateTokens()` reports usable numbers.
- AbortSignal is wired through to SIGTERM the child so the subagent loop's
cancellation path stays correct.
Modified: src/commands/jobs.ts (worker registration)
- Conditionally constructs a MessagesClient via the new adapter when
GBRAIN_USE_CLAUDE_CLI=1.
- Passes it into makeSubagentHandler({ engine, client: subagentClient }).
- Logs `[minion worker] subagent routing via claude-cli (GBRAIN_USE_CLAUDE_CLI=1)`
on startup so the env var status is operator-visible.
Limitations of this commit (addressed in the follow-up):
- Tool use is not yet supported. Tools in params.tools are ignored; the
adapter returns a single text block with stop_reason='end_turn'.
- Token counts come from claude-cli's reporting and may not match the
Anthropic API's accounting precisely (especially for cache tiers).
Original design from #334; this commit preserves that author's attribution.
The follow-up commits on this branch carry the tool-use implementation.
* feat(subagent): tool use + context isolation + convention rename on top of jarvisdoes baseline
Builds on the previous commit (jarvisdoes's #334 baseline) by adding three
things the upstream issue called out as gaps or that surfaced during review:
1. Tool use support via system-prompt-instructed JSON emission.
2. Context isolation flags so claude-cli does not load operator-level
CLAUDE.md, skills, and local project context into every subagent call.
3. Env var rename from GBRAIN_USE_CLAUDE_CLI=1 to
GBRAIN_SUBAGENT_PROVIDER=claude-cli to match the existing
GBRAIN_<noun>_<role>=<value> convention used by GBRAIN_CHAT_MODEL,
GBRAIN_EMBEDDING_MODEL, GBRAIN_EXPANSION_MODEL.
## Tool use
The MessagesClient interface returns Anthropic.Message objects whose
content array may include tool_use blocks. The subagent handler filters
those blocks and dispatches each tool, so any backend that produces
correctly shaped tool_use blocks gets the same loop behavior as the
Anthropic SDK.
The adapter injects a system-prompt addendum describing the tool registry
plus an emission protocol:
<use_tools>
[{"id": "...", "name": "...", "input": {...}}, ...]
</use_tools>
After the response comes back, extractToolCalls() scans for the block,
parses the JSON (tolerant of optional ```json fencing), and converts each
entry into a tool_use content block. Multiple parallel tool calls in one
turn are supported via the array shape; this is the exact case that
breaks today on the codex-proxy / litellm GPT-5.x bridge where parallel
tool-call response IDs get dropped.
Defensive fallbacks:
- Malformed JSON inside the block: drop to text-only, stop_reason='end_turn'.
- Unterminated <use_tools> (no close tag): drop to text-only.
- Model omits id field: adapter synthesizes a toolu_claude_cli_<rand> id.
- Empty response: still hand the subagent loop a well-formed content
array so the .filter chain does not crash.
## Context isolation
claude-cli auto-discovers CLAUDE.md from cwd upward and injects the
operator's skills + plugins + auto-memory into the default system prompt.
On a real install that is ~42-65k tokens of contamination per subagent
call, with both cost and behavioral consequences (the subagent picks up
the operator's coding conventions, opinions, and preferences).
The maximum suppression that still preserves OAuth / Claude Max
subscription auth is:
- Spawn from a dedicated clean cwd (tmpdir-based) so LOCAL CLAUDE.md
auto-discovery has nothing to find. -13k tokens on a real gbrain
install where CLAUDE.md is substantial.
- --disable-slash-commands so skill resolution does not pull in
/skill-name handlers.
- --system-prompt <gbrain prompt> so the default system prompt is
replaced rather than appended to.
The --bare flag would also strip user-level ~/.claude/CLAUDE.md but it
forces ANTHROPIC_API_KEY auth, defeating the whole point of this adapter.
The remaining ~42k cached tokens from user-level instructions are
accepted as a cost-trivial trade-off because the Max subscription absorbs
the per-call cost. Behavioral contamination is mitigated by gbrain's
strong per-call system prompt overriding any operator-level drift.
## Env var rename
Surveyed all ~140 GBRAIN_* env vars in src/. The codebase uses three
patterns: GBRAIN_NO_<feature> (negative toggles), GBRAIN_<noun>_<role>
=<value> (routing keys), GBRAIN_ALLOW_<feature> (permissive toggles).
GBRAIN_USE_* does not appear anywhere except jarvisdoes's original
commit; it would introduce a fourth pattern.
GBRAIN_SUBAGENT_PROVIDER=claude-cli aligns with the routing-keys family
and is value-extensible — adding codex-cli / meridian-proxy / etc. later
means a new value, not a new env var. The scope ('SUBAGENT_*') is also
unambiguous about which calls the toggle covers; GBRAIN_USE_CLAUDE_CLI
was silent on whether it applied to all gbrain LLM calls or only the
subagent path.
Unknown values are rejected with a fail-fast error message naming the
two valid values rather than silently falling through to the default.
## Tests
New file: test/claude-cli-adapter.test.ts — 12 tests, 33 assertions:
- Text-only round trip (single text block, usage propagation, end_turn).
- Provider prefix stripping ('anthropic:claude-sonnet-4-6' -> 'claude-sonnet-4-6').
- Single tool_use parsing.
- Multiple parallel tool calls in one block (the case that triggered
the codex-proxy regression).
- Fenced JSON inside <use_tools> block.
- Model-omitted id gets synthesized to toolu_claude_cli_<rand>.
- Malformed JSON falls back to text.
- Unterminated block falls back to text.
- AbortSignal SIGTERMs the child.
- Error envelope rejected with informative message.
- Non-JSON output rejected with raw-output excerpt in the error.
- argv + cwd assertion: --disable-slash-commands + --system-prompt are
present and cwd is the dedicated tmpdir.
Tests use a POSIX shell stub at GBRAIN_CLAUDE_CLI_BIN that emits a
scripted --output-format json envelope, so the suite runs without
claude-cli installed and without API credits.
* feat(ai): claude-cli recipe with native gateway integration (supersedes #334 baseline)
Replaces the MessagesClient adapter + GBRAIN_USE_CLAUDE_CLI=1 env-var
gate from the previous commit on this branch with a proper gateway recipe.
The recipe path gives per-call routing as a native capability: a model
string like `claude-cli:claude-sonnet-4-6` lands here while a sibling
`litellm:gpt-5.4` continues through the litellm-proxy / codex-proxy path
in the same worker. No global env-var switch, no agent.use_gateway_loop
bypass, no MessagesClient injection at jobs.ts worker startup.
The previous commit on this branch (jarvisdoes baseline) is preserved
in the history for #334 authorship attribution. Its functional changes
are backed out here because the recipe pattern is gbrain's established
integration seam; introducing a parallel MessagesClient + env-var path
would have created two routing mechanisms competing for the same job.
New: src/core/ai/recipes/claude-cli.ts
- Recipe declaration: id 'claude-cli', tier 'native', implementation
'claude-cli', chat-only (no embedding or expansion touchpoints).
- Models: claude-opus-4-7, claude-sonnet-4-6, claude-haiku-4-5-20251001.
- supports_tools and supports_subagent_loop both true.
- supports_prompt_cache false because the CLI handles caching internally
and does not surface cache_control via the standard control plane.
- auth_env.required is the empty array because the CLI owns auth (OAuth
session managed by `claude login`).
- Friendly aliases mirror the `anthropic` recipe: `sonnet`, `haiku`,
`opus` and the same legacy-id rewrites for back-compat with stale
config strings.
New: src/core/ai/providers/claude-cli-language-model.ts
- ClaudeCliLanguageModel class implementing the ai-sdk LanguageModelV2
interface.
- doGenerate: renders the ai-sdk prompt array into a system text + user
text, injects the use_tools protocol instructions when tools are
present, spawns `claude --print --output-format json --model <X>
--disable-slash-commands --system-prompt <gbrain prompt>` from a
dedicated tmpdir (contamination suppression: no local CLAUDE.md
auto-discovery), parses the JSON envelope, extracts <use_tools>
blocks, and returns ai-sdk-shaped LanguageModelV2Content (text +
tool-call parts with stringified-JSON input matching the V2 contract).
- Tolerates fenced JSON inside use_tools blocks, malformed JSON
(falls back to text), missing close tag (falls back to text),
model-omitted ids (synthesizes toolu_claude_cli_<rand>).
- Parallel tool calls in one block round-trip cleanly: this is the
case that drops IDs on the litellm + codex-proxy bridge today.
- AbortSignal SIGTERMs the child for proper cancellation.
- doStream throws not-supported (gateway.toolLoop is non-streaming).
Modified: src/core/ai/gateway.ts
- Adds case 'claude-cli' to instantiateChat (returns ClaudeCliLanguageModel).
- Adds case 'claude-cli' to instantiateExpansion (same wrapper, reserved
for a future expansion touchpoint declaration).
- Adds case 'claude-cli' to instantiateEmbedding (throws, no embedding
model, mirrors the native-anthropic path).
- Lazy require() at the call site keeps the gateway module load cheap
for users who never use the claude-cli path.
Modified: src/core/ai/recipes/index.ts
- Registers `claudeCli` in the ALL[] array next to `anthropic`.
Modified: src/core/ai/types.ts
- Adds 'claude-cli' to the Implementation union so the gateway switch
is exhaustive at compile time.
Reverted: src/commands/jobs.ts
- Drops the GBRAIN_USE_CLAUDE_CLI=1 env-var gate the prior commit
added. Routing now happens at the gateway based on the model string.
Deleted: src/core/minions/handlers/claude-cli-adapter.ts
- The MessagesClient adapter is superseded by the recipe + LanguageModelV2
path. Two routing mechanisms competing for the same job would have
forced users to reason about which one wins; the recipe is the single
source of truth.
New file: test/claude-cli-recipe.test.ts (16 tests, 46 assertions):
- Recipe registration: getRecipe returns chat-only Recipe; aliases map
short names (sonnet/haiku/opus) to canonical model ids.
- Text round trip: single text content block, usage propagation, stop
finish reason.
- Provider prefix stripping.
- Single tool-call parsing.
- Multiple parallel tool calls in one block.
- Fenced JSON inside the block.
- Model-omitted id synthesizes toolu_claude_cli_<rand>.
- Malformed JSON falls back to text + stop reason.
- Unterminated block falls back to text + stop reason.
- Tools offered but model declines: returns text-only with stop reason
so the gateway-loop treats it as a final answer rather than wedging
for tool calls that never come.
- AbortSignal SIGTERMs the child.
- is_error envelope rejected.
- Non-JSON output rejected.
- doStream throws.
- argv + cwd assertion: --print, --disable-slash-commands,
--system-prompt are present and cwd is the dedicated tmpdir.
Tests use a POSIX shell stub at GBRAIN_CLAUDE_CLI_BIN so the suite runs
without claude-cli installed and without API credits.
End-to-end smoke verified against a real `claude --print --model haiku`
invocation: model emitted `<use_tools>` block with toolu_add_001 +
{"a":12,"b":30}, adapter parsed back into a `tool-call` content block,
finishReason 'tool-calls'.
* feat(ai/claude-cli): harden subagent isolation, env scrub, verbose + stdin robustness
Four defensive fixes to the claude-cli provider so a subagent call behaves
identically regardless of the host's ambient Claude Code config:
- Agent isolation: pass `--tools ''` and `--strict-mcp-config` so the subprocess
runs as a raw LLM with no built-in tools and no inherited user MCP servers.
Without `--strict-mcp-config`, each call boots the user's MCP servers (including
gbrain's own), causing recursion plus PGLite single-writer lock contention.
- Env scrub: drop ANTHROPIC_API_KEY / ANTHROPIC_AUTH_TOKEN / ANTHROPIC_BASE_URL
from the child env so the CLI authenticates via its own OAuth subscription
session. An inherited API key silently flips billing to per-token API usage,
the exact setup this recipe exists to replace.
- Verbose-mode compat: with `"verbose": true` in ~/.claude/settings.json,
`--print --output-format json` emits an event array instead of a bare result
object. Tolerate both shapes and select the result event.
- stdin robustness: handle the child stdin 'error' event and wrap write/end so a
missing binary (ENOENT) or early child death (EPIPE) rejects cleanly instead of
crashing the worker with an unhandled error.
Adds unit coverage for the env scrub, the isolation argv, and the verbose event
array. Verified against claude CLI 2.1.x.
* test(ai/claude-cli): cover verbose-array no-result + missing-binary reject paths
Two error branches in the hardened claude-cli provider had no coverage: the
verbose event-array path when no result event is present, and a missing binary
surfacing as a clean spawn-failed rejection. The missing-binary case is the
deterministic form of the stdin/EPIPE robustness; a synchronous stdin-write
throw is not reliably triggerable in a unit test, so the real ENOENT path the
handlers defend is exercised instead. Both reuse the existing shell-stub harness.
---------
Co-authored-by: Brett <brettdavies@users.noreply.github.com>
Co-authored-by: jarvisdoes <258486803+jarvisdoes@users.noreply.github.com>
Co-authored-by: Marco Maldonado <34176133+loweaxerium@users.noreply.github.com>
queue.add() with an idempotency_key returns any existing row regardless
of status. This means dead jobs (exhausted retries from a transient
provider outage) permanently block re-submission of the same work —
even after the underlying issue is fixed.
Fix: when the existing row is dead or cancelled, NULL its
idempotency_key (preserving the row for audit) and fall through to the
INSERT path so a fresh job can be created.
Affects dream synthesize children that died during provider migrations
(429 rate-limit on old Anthropic proxy, tool-results-missing on old
OpenRouter). 45 dead children were blocking re-synthesis of transcripts
in production.
Includes 4 new tests covering dead, cancelled, completed, and active
status interactions with idempotency dedup.
Co-authored-by: Rafael Reis <57492577+rafaelreis-r@users.noreply.github.com>
Co-authored-by: Rafael Reis <rafael.reis@contabilizei.com.br>
`gbrain doctor --remediation-plan` printed two consecutive lines that
contradicted each other when the brain was below target AND the target
was unreachable with autonomous remediation:
Brain score: 45/100 → target 90
Target unreachable: max with autonomous remediation is 70/100.
No remediations needed. Brain is at target.
The second sentence hid the real next step (configure the prereqs that
would lift `max_reachable_score`) and made the brain look healthy when it
was not.
Fix: gate the "Brain is at target" line on `brain_score_current >=
targetScore`. When the plan is empty AND the brain is below target, the
"Target unreachable" line above is already the user-facing explanation;
the `Blocked checks` block below surfaces the manual gap.
Extracted `renderRemediationPlanLines(plan, targetScore): string[]` as a
pure helper alongside `runRemediationPlan` so the regression coverage
asserts on the rendered output directly rather than mocking
`console.log`. `runRemediationPlan` now joins the lines verbatim through
console.log; behavior is byte-identical for every case other than the
fixed contradiction.
Five regression tests cover: unreachable-and-below-target (the bug
case), reachable-and-at-target, exact-target, below-target-with-plan,
unreachable-with-partial-plan. 38 tests across the adjacent doctor test
files stay green; `bun run typecheck` clean.
Co-authored-by: Brett <brettdavies@users.noreply.github.com>
synthesize-concepts.ts's design comment says extract_atoms stamps a
`concepts:` frontmatter field on each atom and :92 consumes ONLY that
field — but the extractor never wrote it, so the atoms → concepts
pipeline was dead end-to-end: every cycle reported "synthesize_concepts:
skipped — no atoms with concept refs" no matter how many atoms
accumulated (696 page-derived atoms / 0 with concepts on our production
brain before an external backfill).
Fix, all on the extractor side (no synthesize change needed):
- EXTRACT_PROMPT asks for `concepts` (1-3 kebab-case TOPIC labels) with
an explicit reuse-over-coinage instruction — labels must cluster,
since synthesize_concepts only materializes groups of >=2.
- parseAtomsResponse validates labels (kebab regex, max 3, drop
invalid; empty -> undefined).
- The putPage frontmatter write stamps `concepts` alongside lesson /
source_quote.
Tests: 4 parse cases + an end-to-end regression that goes extractor ->
real frontmatter -> synthesize_concepts' OWN DB query path -> concept
page. The existing tests fed synthesize via the `_atoms` seam, which is
exactly how this gap survived.
Validated in production ahead of this PR by stamping the same shape
externally: the next synthesize_concepts run wrote 33 concept pages
(T2=7/T3=26) from 60 stamped atoms, zero failures.
Co-authored-by: 陈源泉 <84364275+ChenyqThu@users.noreply.github.com>
Co-authored-by: 陈源泉 <chenyuanquan@chenyuanquandeMac-mini.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(doctor): raw-source persistence guarantee — warn-only v1 (#1978)
Every synthesized/derived page (dream_generated:true or type:synthesis)
must carry a raw trace or an explicit exemption. v1 is warn-only:
- New doctor check `raw_provenance` (brain category) flags synthesized
pages with none of: raw_trace/raw_source/source_uri/raw_trace_exempt
frontmatter, an attached raw_data row, or synthesis_evidence rows.
- Dream synthesize now stamps `raw_source: <transcript path>` into each
written page's frontmatter via the existing #2569 provenance stamp.
- Dream-cycle summary index pages and extract receipts carry an explicit
`raw_trace_exempt: true` + reason (no source document of their own).
No write path is blocked; fail-closed enforcement is the v2 escalation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(doctor): exclude soft-deleted pages from raw_provenance check
Sibling frontmatter checks (quarantined_pages, flagged_pages) filter
deleted_at IS NULL; without it a deleted synthesized page keeps warning
(and its slug keeps being named) through the 72h recovery window with
no way to clear the warn.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
dream.drift.enabled has gated an unwired scaffold since v0.28: runPhaseDrift
had zero call sites, the resolved model + BudgetMeter were discarded
(void modelId; void meter), and no operator-readable output existed.
- Wire 'drift' as a CyclePhase (default OFF via dream.drift.enabled),
ordered after the calibration trio and before embed so the report page
gets embedded same-cycle. PHASE_SCOPE=global, cycle-lock coordinated,
--once (--phase drift --once) bypasses the gate for one run.
- Implement the LLM judge: soft-band candidates (weight 0.3-0.85, active,
unresolved, fresh timeline evidence) are judged against their page's
recent timeline entries via gateway chat; BudgetMeter-gated
(dream.drift.budget, default $1), capped by dream.drift.max_per_cycle
(default 20). Judge model resolves models.drift -> reasoning tier ->
sonnet fallback (unchanged from scaffold).
- Report-only v1: judged candidates land on a reports/drift-<date> page.
dream.drift.auto_update mutates NOTHING; the flag state is recorded in
the report for operators.
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(auth): admin-gated rescope surface for DCR clients stuck on default scope (#1914)
DCR clients self-register with source_id='default' + federated_read=['default']
and the registration comment promised 'rescope via the CLI later' — but no
rescope surface existed. Adds:
- GBrainOAuthProvider.rescopeClient(clientId, { sourceId?, federatedRead? }):
single-statement COALESCE update, canonical source-id validation
(assertValidSourceId), FK-backed existence check on the write source,
friendly errors for pre-v60/v61 schemas and unknown clients. Takes effect
for already-issued tokens because verifyAccessToken re-reads oauth_clients.
- gbrain auth rescope-client <client_id> [--source S] [--federated-read a,b]
(trusted local CLI).
- POST /admin/api/rescope-client (requireAdmin), mirroring the existing
register-client / revoke-client admin endpoints.
Deliberately does NOT let clients self-widen scope (options a/b from the
issue) — fail-closed trust invariant.
Fixes#1914
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(auth): rescope-client admin endpoint returns 400 (not 500) for nonexistent write source
The FK-translated 'Source "x" does not exist' error is a client error;
map it to 400 like the sibling validation failures. ('No OAuth client
found' is matched first, so the 404 path is unaffected.)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Pages ingested into a config.federated=true source were invisible to
normal reads: get_page/list_pages scoped to the scalar resolved source
('default'), while the fully UNSCOPED resolve_slugs leaked every
source's slugs — the reporter's exact observation matrix.
- federatedSearchScope now backs get_page, list_pages and resolve_slugs
(not just search/query), so the unqualified read surface shares one
visibility set: grant > federated set > scalar source. resolve_slugs
gains the missing sourceScopeOpts-family scoping (leak sealed).
- The widening gate is now field-presence instead of ctx.remote:
localFederatedSourceIds is populated only by server-side transports
(never from caller params), so trust stays fail-closed while the
stdio MCP transport (no GBRAIN_SOURCE) and the legacy HTTP token
path (no operator-set permissions.source_id grant) can opt their
unqualified callers into the operator-configured federated set.
Tokens WITH a grant, per-call source_id, and OAuth allowedSources
all still win and never widen.
- gbrain sync now attributes its ingest-log row to the synced source
instead of the shared 'default' bucket (attribution sub-bug).
No engine SQL changes: getPage/listPages/resolveSlugs already accept
sourceIds[] in both engines (#1393/#876).
Fixes#3242
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Both engines' searchTakes used whole-string trigram similarity
(claim % query), which structurally cannot pass the 0.3 threshold for a
short keyword against a 100-200 char claim — keyword search returned
zero results on real brains. Switch the predicate to word similarity
(query <% claim) and rank by word_similarity(query, claim), in both
postgres-engine and pglite-engine per the engine-parity invariant.
Holder allow-list and source-scope filters unchanged.
Regression test: single-word query must match a long claim containing
it (fails under the old predicate).
Fixes#3267
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
OAuth/source scoping only guards the serve --http path; a container
sharing Docker's default bridge with the brain's Postgres can open a
direct DB session without a token. Adds a 'Co-located Docker workloads'
subsection to docs/mcp/DEPLOY.md with the operator checklist, a
trust-boundary paragraph in SECURITY.md, and an ops note + cross-link
in the company-brain tutorial.
Fixes#3270
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Rebase of #3279 onto current master: the whoami oauth shape gains
source_id (AuthInfo.sourceId, null when absent) and federated_read
(AuthInfo.allowedSources, [] when absent) — read-only self-introspection
that widens no grant. Re-applied against the post-#3091 description
string (stdio transport shape preserved) and merged the grant tests
into the current whoami.test.ts alongside the stdio cases.
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: boundless-forest <boundless-forest@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Agent tools (Claude Code, Codex, …) support a *.local.md counterpart to the
committed CLAUDE.md/AGENTS.md for personal, per-clone instruction overrides that
load after the committed file and are meant to stay uncommitted.
gbrain already ignores other workspace-local agent artifacts (.context/,
.claude/), so this fills the remaining gap for the two root-level override files.
Explicit filenames rather than a *.local.md glob, matching the existing
commented, specific style.
doctor and the get_health CLI surface printed two different timeline
metrics under one ambiguous 'timeline' label: the entity-scoped
timeline_coverage fraction (eligible entity pages with a timeline entry)
and the whole-brain timeline_coverage_score brain-score component (all
pages with a timeline entry, 0-15). Different numerators AND
denominators, indistinguishable in output.
Label-only fix, scoring unchanged:
- graph_coverage check: 'entity timeline coverage N%'
- brain_score breakdown: 'timeline density (all pages) N/15'
- get_health CLI: 'Timeline coverage (entity pages)' plus a new
'Timeline density (all pages): N/15' line when the score is present
Adds test/doctor-timeline-metric-labels-2298.test.ts pinning the
denominator semantics, the rendered doctor messages, and the CLI
guard matrix (fails on master, passes here).
Takeover of #2761.
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: TurgutKural <TurgutKural@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(thin-client): map --source/GBRAIN_SOURCE/.gbrain-source onto source_id for routed ops (#2098)
The thin-client route short-circuits before makeContext, so the 6-tier
source resolution never ran and `gbrain query --source X` against a
remote brain sent the unknown `source` key verbatim — the server op
ignored it and searched unscoped.
applyThinClientSourceScope now runs the engine-free tiers (flag → env →
dotfile; DB-backed tiers need an engine, and the server's grant scoping
covers the rest) and sets the op's source_id wire param. Ops declaring
their own `source` param are untouched; an explicit --source on an op
with no source_id wire param errors loudly instead of silently dropping;
explicit --source-id/--all-sources on the wire win over ambient tiers.
Fixes#2098
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(thin-client): use withEnv() instead of direct process.env mutation (test-isolation R1)
CI verify failed on check:test-isolation — thin-client-source-scope.test.ts
mutated process.env.GBRAIN_SOURCE via beforeEach/afterEach. Wrapped each
test body in withEnv() from test/helpers/with-env.ts instead.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(thin-client): keep ambient scope out of get_skill's non-scope source_id param
get_skill's source_id is a mode switch (host catalog vs brain-resident-pack
lookup), not a read-scope filter. Ambient GBRAIN_SOURCE / .gbrain-source
injection would silently reroute 'gbrain skill <name>' on thin clients to
getResidentSkillDetail. Exclude it via NON_SCOPE_SOURCE_ID_OPS; explicit
--source-id still passes through, explicit --source errors with a hint.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
* chore(ci): refresh GitHub Actions SHA pins (checkout v4, action-gh-release v2)
Pre-ship pin staleness check per docs/RELEASING.md: both floating major
tags moved upstream; pins updated to the current tag commits.
* v0.42.65.0 chore(release): 92 verified fixes since v0.42.64.0 — changelog + version bump
Aggregates everything merged to master since the v0.42.64.0 bump commit:
community fixes, credited takeovers, batch re-lands, CI hardening, and
maintainer-approved features. Net commit list excludes revert pairs.
No new schema migrations.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(deps): clear OSV-flagged transitive dependencies via override floors
Raise the existing security-floor overrides so the lockfile resolves
patched versions of three transitive packages flagged by the OSV scan
(@hono/node-server, fast-uri, body-parser). None are on gbrain's own
runtime path (@hono/node-server is only referenced by the MCP SDK's
optional hono transport, which gbrain does not load); the floors keep
the dependency scan green. MCP/OAuth unit tests pass against the
resolved versions.
* chore(release): fold #3110 into the v0.42.65.0 entry (93 net changes)
---------
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(links): resolve [[wikilink]] + slug-path frontmatter values; keep frontmatter links fresh on the incremental cycle
Takeover/rebase of two community PRs:
PR #1983 — frontmatter link fields never resolved Obsidian-style values:
- makeResolver step 1's strict slug regex rejected digit-leading folders
(90-people/nicolai) and nested paths (a/b/c); broadened to any slug-shaped
value with an EXACT getPage match only (no fuzzy, no false positives).
- extractFrontmatterLinks resolved "[[dir/slug]]" verbatim; new anchored
unwrapWikilink() strips wholly-wrapped [[...]] (and |alias/#heading/^block)
before resolution. Bare values pass through unchanged.
- Same broadened slug-shape applied to the fs-path synthetic resolver in
extractLinksFromFile (exact Set membership guards it), so the fs
frontmatter path resolves PARA-numbered slugs too.
PR #2434 — the cycle's incremental extract (extractForSlugs) extracted body
links only, so externally-edited YAML (sources:/related:) edges drifted
stale. Adds an includeFrontmatter opt (threaded as a param after sourceId,
which master added in #1747/#1503 after the PR was cut), gated by the new
config key autopilot.incremental_extract_include_frontmatter (default off,
preserves body-only behavior).
Tests: unwrapWikilink unit coverage, broadened-resolver + end-to-end
frontmatter cases in test/link-extraction.test.ts; fs-resolver digit-leading
case in test/extract.test.ts; incremental gate off/on cases in
test/extract-incremental.test.ts.
Co-authored-by: spiky02plateau <spiky02plateau@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(cycle): honor DB-plane config for incremental_extract_include_frontmatter
The gate read loadConfig() (file/env plane) only, but the documented enable
command — gbrain config set autopilot.incremental_extract_include_frontmatter
true — writes the DB plane via engine.setConfig, so the feature could never be
turned on the documented way (silent no-op, #2120 class). Now the file plane
wins when the key is present there; otherwise the DB plane is consulted,
matching the autopilot.auto_drain.* read pattern.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: spiky02plateau <spiky02plateau@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
Adds a `perplexity` embedding recipe (OpenAI-compatible at
https://api.perplexity.ai/v1, auth via PERPLEXITY_API_KEY only — never an
OPENAI_API_KEY fallback) covering pplx-embed-v1-0.6b and pplx-embed-v1-4b.
Perplexity's /embeddings endpoint diverges from OpenAI's wire shape in two
places that break the AI SDK adapter, handled by a new perplexityCompatFetch
shim (mirrors the Voyage/ZeroEntropy pattern incl. the two-layer OOM caps):
- encoding_format only accepts base64_int8/base64_binary; the SDK's 'float'
default is forced to 'base64_int8' outbound.
- The response embedding is base64-encoded signed int8 components (natively
quantized); decoded to number[] inbound so the SDK's Zod schema validates.
Cosine similarity is scale-invariant, so raw int8 components rank correctly.
Flexible dims (Matryoshka-style 128..native max: 1024 for 0.6b, 2560 for 4b)
validate fail-loud in dims.ts + the init preflight; `dimensions` is
Perplexity's native field so no wire translation is needed. default_dims is
1024 (works on a plain vector column for both models); the 4b model's full
2560 width rides the existing halfvec (>2000 dims) storage/ANN path. Pricing
entries land in embedding-pricing.ts.
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(schema): make bundled pack inspection truthful (#2029)
Two live bugs:
- parseYamlMini had no block-scalar support, so a 'description: |' swallowed
every following top-level key — the active gbrain-recommended pack loaded
with 0 page types. Add parseBlockScalar for |/|-/|+ and >/>-/>+ in both
mapping and sequence-sibling positions.
- The bundled-pack list was hand-copied in three places (operations.ts had 2
names, mutate.ts had 3, load-active.ts had 7). New single registry
src/core/schema-pack/bundled.ts carries all 7 shipped packs; every
consumer derives from it.
Takeover of #2029, rebased onto master (schema.ts hunks already landed).
Co-authored-by: JiraiyaETH <JiraiyaETH@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(minions): subagent default client resolves config-stored Anthropic key (#2048)
The legacy subagent path constructed a bare new Anthropic() (env-only), so
launchd/MCP workers whose key lives in gbrain config (anthropic_api_key)
failed auth. anthropic-key.ts now exports resolveAnthropicKey() (env first,
then config; hasAnthropicKey delegates) and makeSubagentHandler passes it as
apiKey.
Partial takeover of #2048 — only the auth patch; the path patches were
superseded by the outputRoot mechanism (#2415).
Co-authored-by: JiraiyaETH <JiraiyaETH@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(schema): point bundled-registry test at bundled.ts source of truth
The bundled pack list moved from load-active.ts to bundled.ts in the
truthful-inspection refactor; the T4 registry test still grepped
load-active.ts source. Assert BUNDLED_PACK_NAMES directly instead.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(schema): block scalars keep '#' as literal content
Inside a YAML block scalar '#' is content, not a comment; parseBlockScalar
was routing lines through stripComment/isBlank, truncating descriptions
like 'see issue #2029' and blanking comment-looking lines. Use the raw
line inside the scalar. Adds a pinning test.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: JiraiyaETH <JiraiyaETH@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(cycle): propose_takes skips cleanly when the chat provider is unavailable + narrow page projection
Takeover of PR #1979 by @shawnduggan. The original PR gated on a
hardcoded ANTHROPIC_API_KEY heuristic (modelNeedsAnthropicKey defaulting
to true), which master deliberately removed elsewhere — it misclassified
non-Anthropic stacks and fought the tier-config model resolution. This
lands the intent the master-blessed way: probe the RESOLVED chat model
(opts.model ?? getChatModel()) via probeChatModel — same semantics as
patterns.ts / think/index.ts — and skip the phase cheaply when the
provider can't run. Injected extractors are never gated.
Also keeps the PR's uncontested half: load proposal candidates with a
narrow projection (slug, source_id, compiled_truth) instead of
listPages' SELECT p.*, preserving sourceIds > sourceId scope precedence
and updated_desc ordering.
Co-authored-by: shawnduggan <shawnduggan@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(brainstorm): price cost preview against the configured chat model + models.brainstorm.judge config key
Takeover of PR #1855 by @starm2010, shrunk to the brainstorm-only
portion (the cycle-phase hunks are superseded by the resolveModel-in-
phase approach already on master). The cost preview + hard cost ceiling
previously always priced anthropic:claude-sonnet-4-6 even when the
configured chat_model (which the gateway actually runs) was something
else; modelStr now resolves override → config.chat_model → fallback.
The judge phase honors a new models.brainstorm.judge config key when no
--judge-model flag is passed, resolved in the orchestrator so every
caller (brainstorm, lsd, eval-brainstorm) benefits.
Co-authored-by: starm2010 <starm2010@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(test): use withEnv()/emptyHome() in propose-takes no-key tests
check:test-isolation R1 flagged direct process.env mutation in the two
new no-key tests. Swap the hand-rolled save/mutate/restore for the
canonical withEnv() helper (+ emptyHome() for the hermetic GBRAIN_HOME).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(jobs/autopilot): interval persistence, --lock-duration flag, dead-jobs doctor check, deployment-shape docs
Four backlog items in the jobs/autopilot workers, locks & installers area:
- #2794: `gbrain autopilot --install` silently dropped `--interval`. The
installer now parses + validates it, persists it to config
(autopilot.interval), and threads it into the wrapper's exec line; a
later flag-less --install regenerates the wrapper from the persisted
value, and the daemon run path falls back to the same config key.
- #1014: new `--lock-duration MS` flag (env: GBRAIN_LOCK_DURATION) on
`gbrain jobs work` and `gbrain jobs supervisor` to tune the worker
stall-lock window (and so the lockDuration x max_stalled wall-clock
dead-letter cap). Validated like --health-interval (integer >= 1000ms);
the supervisor propagates it to the spawned worker via buildWorkerArgs;
shown in the worker startup banner.
- Takeover of PR #1185 (@ethanbeard): `gbrain integrations doctor` now
surfaces dead minion jobs as a cross-cutting [queue] check. Reworked
from the original: consumes a new machine-readable `gbrain jobs list
--json` surface instead of screen-scraping the human table (long job
names shift the columns), and scopes to a 24h finished_at window so one
ancient dead job can't flag ISSUES forever (parity with main doctor's
queue checks).
- #631: documented the production deployment shape for autopilot vs jobs
supervisor in docs/guides/minions-deployment.md — recommend the
`autopilot --no-worker` + `jobs supervisor` split, warn against running
both worker lanes, and cross-link the --no-worker liveness probe.
Co-authored-by: ethanbeard <ethanbeard@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(autopilot): make #2794 wrapper-script tests hermetic — fake gbrain on PATH
writeWrapperScript calls resolveGbrainCliPath(), which shells out to
`which gbrain` and throws on CI runners where no gbrain binary is
installed. The two new --interval threading tests failed only in CI
(dev machines have gbrain on PATH). Prepend a fake executable to PATH
for the describe block so resolution is deterministic everywhere.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: ethanbeard <ethanbeard@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(init): explicit --embedding-model overrides the persisted --no-embedding sentinel (#2301)
Pre-fix, once ~/.gbrain/config.json carried embedding_disabled: true (the
--no-embedding deferred-setup sentinel), every re-init silently re-deferred
embedding: resolveAIOptions honored the sentinel BEFORE the explicit
--embedding-model flag and never cleared noEmbedding, and the persistence
merge carried the sentinel forward via ...existingFile. Both recovery paths
were dead ends — `gbrain config set embedding_model` is hard-refused
(schema-sizing field), and re-init hit the sentinel.
Fix:
- resolveAIOptions: an explicit --embedding-model / --model flag clears the
sentinel-derived noEmbedding (explicit --no-embedding on the same
invocation still wins — that branch runs after).
- initPGLite + initPostgres persistence: a resolved (model, dims) tuple
drops the stale embedding_disabled key instead of inheriting it.
- assertEmbeddingEnabled message no longer recommends the refused
`gbrain config set embedding_model` command; the working re-init recipe
leads.
Test: test/e2e/init-reinit-after-deferred.test.ts — deferred init then
re-init with an explicit model recovers (sentinel gone, model persisted);
bare re-init still honors the sentinel.
Fixes#2301
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* review: scrub remaining hard-refused `config set embedding_model` advice from init deferred-setup messages
The PR fixed the recovery recipe in assertEmbeddingEnabled but the
deferred-setup lines in initPGLite/initPostgres and the fail-loud
defer hint still pointed users at the Lane C.2 hard-refused command.
Point all three at the working re-init recipe instead.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(dream): stamp path-derived sources so --dir runs land cycle freshness (#1869)
gbrain dream --dir <path> (and the configured sync.repo_path fallback)
never wrote last_source_cycle_at / last_full_cycle_at because runCycle's
stamp gate reads opts.sourceId and dream only set it from --source.
Doctor's cycle_freshness stayed perpetually stale on path-scoped brains.
Fix at the command level: dream derives the source id from the resolved
brain dir via resolveSourceForDir (now exported from cycle.ts) and passes
it as opts.sourceId. runCycle's stamp/lock semantics are untouched, so
legacy global callers (autopilot-global-maintenance runs GLOBAL_PHASES
with a brainDir and no sourceId) cannot falsely stamp per-source
freshness — the flaw that sank the runCycle-wide variant in PR #2549.
A derived match on an archived source is skipped (mirrors the explicit
--source archived guard).
Takeover of #2549.
Co-authored-by: javieraldape <javieraldape@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(autopilot): close the engine on SIGTERM/SIGINT instead of hard-exiting (#1872)
systemctl stop (SIGTERM) previously hard-exited autopilot without ever
closing the engine. On PGLite the cycle steps run INLINE in the autopilot
process, so a mid-write exit kills WASM Postgres with the WAL dirty and
can corrupt the brain.
Now both exit paths close the engine first:
- autopilot's own shutdown() (SIGINT + internal stops like max_crashes /
cycle-failure-cap) aborts the in-flight inline cycle via an
AbortController threaded into runCycle, drains it briefly, and awaits
engine.disconnect() before process.exit(0).
- process-cleanup's SIGTERM handler (installed at cli.ts module load,
exits within its 3s cleanup deadline) reaches the same closeEngine via
a registered 'autopilot-engine-close' cleanup callback.
PGLite's disconnect() drains the pending query and checkpoints before
closing; a second call is a no-op, so both paths firing is safe.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(dream): conform dream-dir-source-stamp to canonical PGLite isolation pattern
check:test-isolation R3/R4 flagged the new test file: engine was created
in beforeEach (outside beforeAll) and never disconnected in afterAll.
Switch to the canonical shared-engine pattern (beforeAll create,
beforeEach resetPgliteState, afterAll disconnect) per
test/helpers/reset-pglite.ts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: javieraldape <javieraldape@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
orphan_ratio's denominator is swamped on auto_chronicle brains by the
machine-generated chronicle events (life/events/<day>-<hash>, written
per eligible event) — no inbound links by design. The shipped policy
already excludes raw/atoms/skills/dreaming/daily and extracts/, but
life/events/ was still counted; on a 1,657-page auto_chronicle brain it
was ~72% of the orphan mass, enough to pin the ratio red.
Add 'life/events/' to DENY_PREFIXES — a scoped prefix, NOT the whole
`life/` first-segment, so human-authored life/diary/ (gbrain capture
--type diary) stays IN the denominator. Same shipped hardcoded-class
mechanism as the existing entries; not the #2215 user-config route
(closed not_planned). Knowledge classes (concepts/people/notes/projects)
also stay in, so genuine graph decay still trips.
Regression in test/orphans-pure-fn.test.ts: life/events/ now excluded
(fails before, passes after); life/diary/ and concepts//notes//projects/
pinned as still-counted. doctor's orphan_ratio uses the same shouldExclude
path (getOrphansData; local + doctor-remote MCP), covered transitively.
zerank-2 is the default reranker under search_mode: tokenmax, but had no
pricing entry — any --max-cost-capped rerank call TX2 hard-failed in
BudgetTracker.reserve() with "no pricing entry".
Adding the entry to EMBEDDING_PRICING alone (the issue's suggested fix)
does not resolve this: lookupPricing()'s rerank branch in
budget-tracker.ts never consulted that table at all, only
ANTHROPIC_PRICING and the FREE_LOCAL_RERANK_PROVIDERS zero-price set.
Verified by reproducing the hard-fail with only the pricing-table entry
added and confirming it still threw.
Fix: add the $0.025/1M-token entry (docs/ai-providers/zeroentropy.md)
and wire the rerank branch to fall back to lookupEmbeddingPrice, reusing
the existing provider:model-keyed table instead of duplicating a third
pricing surface.
Addresses the report in #3223.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
gbrain migrate's per-page copy loop had no failure handling at all: a page
write that threw (e.g. a NOT-NULL column with no protection against the
JS `undefined` postgres.js's UNDEFINED_VALUE guard rejects) crashed the
whole command outright, with no per-page accounting and no way to tell
which page caused it.
Two changes:
- Normalize `undefined` column values to explicit `null` at the migrate
copy boundary before calling putPage. PGLite can hand back `undefined`
for a column that is legitimately NULL/empty; postgres.js rejects a raw
`undefined` bound parameter but accepts `null` fine. This is the root
cause behind the report: a page whose title/compiled_truth/type came
back `undefined` threw mid-insert.
- Wrap the per-page copy in try/catch: failures are tracked (slug +
reason), excluded from the resume manifest's completed_slugs (so a
retry picks them back up), and the run ends with a non-zero exit
verdict + an honest "N copied, M failed" summary instead of a bare
crash or a false "N/N copied" success.
Fixing this properly also required making the pre-existing resume
manifest actually usable without --force (a matching manifest now
bypasses the non-empty-target guard instead of demanding a wipe that
would orphan already-copied pages), always resetting the manifest on
--force regardless of whether the target looked empty, persisting the
manifest before the copy loop starts (so a run where every page fails
after its row lands still leaves a resumable manifest on disk), only
flipping the active config to the target once the migration is fully
clean, and skipping link-copy for slugs known to have failed above
(avoiding an FK-violation crash on the next phase).
Addresses the report in #3194 (reported by @hbohlen).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Addresses the report in #2662: buildGatewayConfig folded openai_api_key,
anthropic_api_key, zeroentropy_api_key and openrouter_api_key from
~/.gbrain/config.json into the gateway env, but not voyage_api_key. In
launchd/daemon/MCP contexts (no process-env export), multimodal/image
embeds with Voyage failed silently even though config.json looked complete.
- build-gateway-config.ts: fold voyage_api_key -> VOYAGE_API_KEY, mirroring
the existing zeroentropy/openrouter fold (process.env still wins).
- config.ts: add the voyage_api_key file-plane field to GBrainConfig and
KNOWN_CONFIG_KEYS.
- brain-score-recommendations.ts: HOSTED_EMBED_KEY_CONFIG now maps
VOYAGE_API_KEY -> voyage_api_key so doctor/autopilot judge a config-keyed
Voyage brain as usable instead of dispatching a doomed embed job.
- autopilot.ts: the HOSTED_EMBED_KEY_CONFIG producer closure now resolves
hosted keys via the same file-plane source (loadConfigFileOnly) doctor
already uses, instead of the DB plane (engine.getConfig) - the DB plane
is never threaded into buildGatewayConfig for these fields, so reading it
here would let a DB-only key report "configured" while the gateway still
has no key. This also tightens the pre-existing openai/zeroentropy path,
not just voyage.
- Tests: fold + env-precedence tests in build-gateway-config.test.ts,
HOSTED_EMBED_KEY_CONFIG map test, and a real end-to-end regression in
brain-score-recommendations.test.ts through loadConfigFileOnly() and
buildGatewayConfig() with an actual temp config.json.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(frontmatter): stop treating YAML comments inside the fence as markdown headings
autoFixFrontmatter's MISSING_CLOSE repair walked lines from the opening
`---` and broke out of the scan on the first `#`-prefixed line, treating
it as a markdown heading before it ever reached the real closing fence.
A `#` line inside a closed YAML block is a comment, not a heading — but
the scan never got that far, so it inserted a spurious `---` right
before the comment and split valid frontmatter in two, pushing the real
keys (title, pubDate, ...) into the document body.
This is the same bug PR #2153 fixed in the parseMarkdown validator, but
autoFixFrontmatter in brain-writer.ts is a separate reimplementation of
the same MISSING_CLOSE logic that PR never touched. Because
parseMarkdown's validator now parses this shape cleanly, autoFixFrontmatter
is only reachable when some other fixable error (SLUG_MISMATCH, NULL_BYTES,
etc.) also fires on the same file — a common real-world case (e.g. a
renamed file with a stale slug: field) that still corrupts otherwise-valid
frontmatter today.
Fix: scan the full zone for the closing `---` first; only fall back to
the heading-shaped-line heuristic when no closer is found at all.
Addresses the report in #3225. Thanks to @WilliamCourterWelch for the
clear repro and for catching this via git diff before it reached a live
site.
Tests: 4 new regression cases in test/brain-writer.test.ts covering a
YAML comment before the close, comment-only frontmatter, a `#` inside a
quoted string value, and a comment co-occurring with an unrelated real
fix (SLUG_MISMATCH) — confirmed all 3 corruption-covering cases fail
against the pre-fix code (stash/red/restore) and pass after the fix.
The pre-existing genuinely-missing-closer case is unchanged.
bun test test/brain-writer.test.ts test/markdown-validation.test.ts
test/markdown.test.ts test/lint-frontmatter.test.ts
test/doctor-frontmatter-partial.test.ts test/frontmatter-cli.test.ts
-> 122 pass / 0 fail. bun run typecheck -> clean. Full suite intentionally
not run locally (targeted scope per contribution norms); CI covers it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(frontmatter): swap non-exercising regression case per codex review
The quoted-string test (title: "Chapter #1 recap") never exercised the
fixed branch — the heading regex is line-anchored on the trimmed line,
so a `#` mid-string never matched before or after the fix. Replace it
with an indented `#` line inside a YAML block scalar, which does hit
the same closer-first-scan code path as the other regression cases
with a different real-world shape.
bun test test/brain-writer.test.ts -> 27 pass / 0 fail. bun run
typecheck -> clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
extract-atoms-drain's runBatch discarded runPhaseExtractAtoms's per-item
failures/status, so a batch where EVERY provider call errored collapsed to
{extracted: 0, skipped: 0} — indistinguishable from a legitimate no-op. The
drain loop reported status: 'ok' regardless, the Minion handler returned
normally, and the worker marked the durable job complete while the backlog
sat untouched with no retry ever applied.
- runBatch now derives providerFailure from the same counts the phase
already returns (failures.length > 0 && transcripts_processed +
pages_processed === 0 — every attempted item errored, zero succeeded).
Partial success (>=1 item processed) is unaffected.
- The pure loop surfaces this as status/stopped = 'provider_failure',
breaking immediately (same hot-loop guard as no_progress) instead of
letting a final remaining===0 recount silently overwrite it to 'drained'.
- The extract-atoms-drain Minion handler throws when it sees
status === 'provider_failure', so the worker's ordinary failJob path
(attempt+backoff, dead-letter on exhaustion) takes over. The
LockUnavailableError -> deferred path is unchanged.
- autopilot's auto-drain submission bumps max_attempts from 1 to 3 (queue
default) — with the handler now actually throwing, max_attempts:1 meant
the first provider blip dead-lettered instantly with no backoff attempt.
Tests: pure-loop provider_failure propagation (incl. the remaining===0
precedence case), runPhaseExtractAtoms's all-items-fail counts contract,
and source-shape guards on the handler throw + autopilot max_attempts.
Full suite deferred to CI per repo convention (targeted run: 104 pass / 0
fail across the touched + adjacent extract-atoms/drain/autopilot files;
`bun run typecheck` clean).
Two rounds of codex review (gpt-5.6-sol, high effort): round 1 flagged
autopilot's max_attempts:1 and the stopped-precedence bug (both fixed
above); round 2 confirmed no new issues.
Thanks to @aaronkhawkins for the detailed report. Addresses the report in
#3218.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(sync): report partial (pull_failed) instead of up_to_date when git pull fails with zero imports (#3068)
A warn-and-continue internal git pull failure (e.g. a local-path origin
rejected by protocol.file.allow=never) combined with a zero-import run
previously reported `up_to_date`, exited 0, and bumped the last_sync_at
freshness heartbeat. A permanently-failing pull was therefore invisible
forever: doctor's sync_freshness never fired and every scheduled sync
looked clean while the source silently went stale.
Now, when the pull failed and the run imported nothing, sync returns
`partial` with the new reason `pull_failed`, leaves last_commit AND
last_sync_at untouched (so staleness monitoring fires), and prints a
dedicated non-success message. The fall-through-to-working-tree design
is unchanged: local commits still import when the remote is unreachable,
and the anchor still advances over commits that were actually imported.
Addresses the report in #3068.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(sync): surface pull_failed to CLI exit codes, sync --all JSON, and the cycle phase (#3068 review round)
Codex review round 1 follow-ups:
- Single-source `gbrain sync` sets exit code 1 on partial/pull_failed
(timeout-class partials keep exit 0 — they converge on retry; a failing
pull does not).
- `sync --all` exits 1 when any source reports pull_failed, and the
--json envelope carries the per-source partial `reason`.
- The autopilot cycle's sync phase maps partial/pull_failed to `warn`
with a dedicated summary and a `syncReason` detail, so a scheduled
cycle no longer reports a clean run over a wedged source.
- The regression test now isolates GBRAIN_HOME to a temp dir so the
first full sync cannot touch the real sync-failure ledger.
- Current-state docs: KEY_FILES.md sync.ts entry + TESTING.md inventory
describe the pull_failed contract and the new test.
Addresses the report in #3068.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(sync): route pull_failed exit through the owned verdict channel; make the regression test serial (#3068 review round 2)
Codex review round 2 follow-ups:
- Single-source exit now uses setCliExitVerdict(1) instead of a raw
process.exitCode assignment, which the CLI teardown deliberately
ignores (PGLite's Emscripten runtime clobbers process.exitCode
mid-run; the owned channel in src/core/cli-force-exit.ts is the only
trusted verdict). Pinned by test/cli-exit-verdict-pin.test.ts.
- The regression test is renamed to *.serial.test.ts because it pins
GBRAIN_HOME for the whole file (scripts/check-test-isolation.sh R1);
docs updated to the new name.
Addresses the report in #3068.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Keep each page excerpt within the existing fixed budget while selecting the window with the strongest query-term coverage. Preserve leading truncation for callers without a matching question.
* fix(migrate): drop invalid CONCURRENTLY-build remnants without a DO block
Migration v66 (embed_stale_partial_index) pre-drops an invalid index left
over from a previously interrupted CREATE INDEX CONCURRENTLY using
DO $$ BEGIN ... EXECUTE 'DROP INDEX CONCURRENTLY IF EXISTS ...'; END $$.
Postgres rejects CONCURRENTLY from any function/EXECUTE context, so the
guard's EXISTS check passes but the EXECUTE inside it always throws
"DROP INDEX CONCURRENTLY cannot be executed from a function" -- the
migration only fails on brains carrying an invalid-index leftover.
Add dropInvalidConcurrentIndex(): the validity probe runs as a plain
application-level SELECT, and the DROP runs as its own top-level
runMigration call instead of inside a DO block. Fixes#1178.
* fix(migrate): address codex review — schema-safe index resolution + OID-based no-op assertion
- dropInvalidConcurrentIndex(): resolve indexName via to_regclass() (search_path
resolution, same as the unqualified DROP that follows) instead of matching
pg_class.relname bare, which could hit a same-named index in a different
schema on a non-default search_path.
- e2e test: the no-op re-run case now compares index OID before/after, not
just validity -- validity alone wouldn't catch a spurious drop+recreate.
The conversation-fact extractor renders turns as `${speaker} (${ts}): ${text}`
and its `confidence` field scores confidence-in-the-CLAIM, not confidence-in-
WHO-said-it. So a first-person self-assertion from an anonymous speaker
("Speaker A: I'm joining Acme") could come back with the anonymous label echoed
as `entity` — a confident attribution to a person we cannot identify. That
label is then stored verbatim as the fact's `entity_slug` (the batch insert
path does no canonicalization), polluting entity-scoped queries and the
top_entities aggregation, or misattributing the claim.
Add a deterministic gate (`isUnknownSpeakerLabel`) at the single candidate-loop
choke point that nulls ONLY that self-referential attribution, plus one
EXTRACTOR_SYSTEM rule telling the model not to guess a name for anonymous
first-person turns. Third-person entities from the same turn ("Acme raised $5M"
-> entity=acme) and named-speaker attributions are untouched. The fact itself
is always preserved; only the bad attribution is dropped.
The P1 entry asked for fail-closed semantics in resolveTakesSourceId
(src/commands/takes.ts). That landed in #2973 (merged 2026-07-20):
the function now delegates straight to resolveSourceId with no
catch-and-fallback, so an unresolvable explicit source throws instead
of silently restoring the pre-#2698 unscoped cross-source write path.
Regression tests for the invalid-source path shipped in the same PR.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
discoverGitRoot() probes `git rev-parse --show-toplevel` to locate the repo
root; a miss is expected/routine (a non-git-yet brain dir, a scratch dir) and
is either self-healed via auto git-init or surfaced as a friendlier Error.
Node's execFileSync writes the child's stderr straight to the parent's real
stderr by default unless an explicit `stdio` array is given, so every routine
probe miss dumped git's raw "fatal: not a git repository ..." line into
gbrain's operator logs -- indistinguishable from an actual crash to an
operator grepping logs for "fatal:" as a crash signature.
Add an opt-in `silenceStderr` param to the shared `git()` helper (sets
`stdio: ['ignore', 'pipe', 'pipe']`, which disables the implicit
passthrough-to-parent-stderr behavior) and pass it only from
discoverGitRoot's internal probe call. Every other `git()` call site is
unchanged, so unexpected-failure visibility elsewhere is preserved.
Related to #2964, which added the auto-recovery this probe feeds.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
`getMemoryUsage()` in src/core/backoff.ts computes 1 - freemem()/totalmem(),
where Node's `os.freemem()` returns Linux's `MemFree`. `MemFree` excludes the
page cache, which the kernel grows aggressively in any environment that reads
files (i.e. essentially all containers). On a healthy 4 GB Linux container with
~1.4 GB MemAvailable, MemFree is routinely ~100 MB, so `getMemoryUsage()`
reports 96-97% used and `waitForCapacity()` rejects every job with:
Throttle timeout: system overloaded after 20 attempts (~600s).
Load: ..%, Memory: 97%
even though the host has plenty of usable memory.
Linux exposes `MemAvailable` in `/proc/meminfo` precisely as the kernel's
estimate of memory available for new allocations without swapping (it factors
in reclaimable page cache). This is what `htop` and `free -h` show as
"available". Using it removes the false positive entirely.
Behaviour:
- On Linux (when /proc/meminfo is readable): use 1 - MemAvailable/MemTotal.
- Anywhere else (macOS, Windows, sandboxed envs without /proc): unchanged
fallback to 1 - freemem()/totalmem().
Scope is intentionally minimal — data correctness only. An env override like
GBRAIN_MEMORY_STOP_PCT would also be reasonable but is out of scope here.
Co-authored-by: Kevin Hsu <kevinhsu.ecofirst@gmail.com>
mechanical.test.ts shells out to `gbrain init --non-interactive`,
`gbrain import`, and similar commands via Bun.spawnSync. The four
`cliEnv()` helpers in this file forward `process.env` unchanged, so
`gbrain init` ends up calling saveConfig() against the developer's real
$HOME/.gbrain/config.json, overwriting their production database_url
with the test container's URL on every `bun run test:e2e` invocation.
Sibling test/e2e/migration-flow.test.ts already solved this with a
module-level temp HOME and an afterAll restore. Mirror that pattern in
mechanical.test.ts.
Verified by md5'ing ~/.gbrain/config.json before and after running the
Setup Journey, Init Edge Cases, Schema Idempotency, RLS Verification,
Doctor Command, and Parallel Import describe blocks — config hash is
identical pre and post (26 passing tests, 0 failures, 0 mutations to
the user's real config).
Co-authored-by: Seth Armbrust <setharmbrust@seth.local>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(recipes/x-to-brain): use /users/by/username for app-only bearer health check
Takeover of #2343. /users/me requires user-context OAuth and always fails
under the app-only bearer the recipe collects. Health check + setup curls
now use /users/by/username/$X_HANDLE, with X_HANDLE declared in secrets
so the installer prompts for it. Recipe version 0.8.1 -> 0.8.2.
Co-authored-by: ethanbeard <ethanbeard@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(cycle): bound propose_takes with per-call timeout + phase deadline
Takeover of #2262. The extractor's gateway.chat call had no abortSignal, so
one stalled provider socket could pin the phase for the 300s gateway default
per page; the nightly wrapper then SIGTERMed the whole phase mid-run. Each
extractor call is now bounded at 90s (per-page failure already logs a warning
and continues), and the page loop carries a 30-min wall-clock deadline that
breaks cleanly into a partial result with deadline_hit:true + warn status.
Unlike the original PR, the default pageLimit stays at 100 — shrinking it to
30 was an unrelated product-knob change that would permanently cut nightly
take coverage.
Co-authored-by: tschew72 <tschew72@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(capture): make fallback title truncation explicit and astral-safe
Takeover of #2310. deriveTitle's silent .slice(0, 80) could split an astral
surrogate pair mid-character and gave no signal the title was cut. Truncation
is now codepoint-aware and appends an ellipsis (still capped at 80 codepoints).
Unlike the original PR, this stays a three-line change: no whitespace
normalization of every derived title, no word-boundary heuristics.
Co-authored-by: xd-Neji <xd-Neji@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(doctor): clear extract_atoms raw source-holder backlog + normalize pooler direct-URL overrides
Takeover of #2242, split to the two concerns that survive review:
- extract_atoms: exclude source pages whose frontmatter declares a raw
payload pointer from discovery AND the doctor backlog count (shared SQL
fragment so they can't drift). Extraction on these yields zero atoms, so
no atom row is ever written and they re-enter the backlog every cycle —
a permanent no-progress doctor blocker.
- connection-manager: a direct-URL override (opts/env) that still points at
the Supavisor TRANSACTION pooler (port 6543, usually a copy-paste of the
primary URL) is normalized to the real direct host via deriveDirectUrl.
Session-mode pooler overrides (port 5432) pass through — they are a
legitimate direct-ish target, which the original PR would have nulled out.
Dropped from the original PR: orphan-reporting atom exclusions (master
already excludes atoms/ and raw/ first segments plus /raw/ segments in
src/commands/orphans.ts) and the drain dry-run status tweak.
Co-authored-by: benjonp <benjonp@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(cycle): record propose_takes deadline break as a halt in the extract rollup
A deadline-hit run breaks the page loop mid-list — same posture as budget
exhaustion — but the rollup still counted it as a completed round with no
halt, hiding chronic never-finishing nightly runs from extract-status/
doctor. Treat deadline_hit like budget_exhausted in the rollup deltas;
deadline test now pins halt=1 / completed=0.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: ethanbeard <ethanbeard@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: tschew72 <tschew72@users.noreply.github.com>
Co-authored-by: xd-Neji <xd-Neji@users.noreply.github.com>
Co-authored-by: benjonp <benjonp@users.noreply.github.com>
* fix(cycle): drain PGLite synth subagents inline (takeover of #2699)
PGLite holds an exclusive file lock on its embedded data-dir, so no
separate Minions worker can serve the subagent children the synthesize
phase enqueues — they sat in 'waiting' until waitForCompletion timed
out. Drain a private per-run child queue inline (claim → run →
complete/fail, plus the promote/stall/timeout housekeeping a worker
would perform). No-op on Postgres, where children stay on the shared
'default' queue.
Rebased onto the reworked synthesize (config.subagentTimeoutMs, #1586
source scoping): the inline job context now carries deadlineAtMs from
the claim-time timeout_at stamp, and opts.yieldDuringPhase is ticked on
a 60s keepalive while each child runs so the 5-min cycle lock TTL
refreshes across long (up to 30-min) children.
Co-authored-by: TheRealMrSystem <TheRealMrSystem@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(lint): --exclude flag for mixed-content repos (takeover of #2649)
Adds --exclude=a,b (and LintOpts.exclude) so mixed-content repos can
skip software trees and repo metafiles by basename when collecting
pages. The only built-in default is node_modules — vendored dependency
trees are never knowledge pages; dot/underscore entries were already
skipped by the walk.
Diverges from #2649 deliberately: the original hardcoded an opinionated
default list (README.md, CHANGELOG.md, CLAUDE.md, test/ dirs at any
depth, plus fork-specific filenames), which silently changed lint
counts for every existing repo. Those are repo policy — pass --exclude.
Co-authored-by: ryangu00 <ryangu00@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(cycle): enforce per-job timeout_ms in the PGLite inline subagent drain
The inline drain claimed children with deadlineAtMs derived from timeout_at
but never armed the worker's timeout timer — and the handleTimeouts sweep
only runs between jobs, so nothing could stop a child that blew past its
30-min timeout_ms. A hung LLM call wedged the drain loop (and the whole
cycle) indefinitely, with the 60s keepalive refreshing the cycle lock
forever. Worker.ts parity: arm a timer from the claim-time timeout_at
stamp, abort ctx.signal on fire, and dead-letter (never delayed-retry)
timed-out children, mirroring handleTimeouts' stall→retry / timeout→dead
split.
Regression test: a child with timeout_ms=100 whose handler only ends on
ctx.signal abort is dead-lettered with 'timeout exceeded'; pre-fix the
test hangs to its 30s timeout.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: TheRealMrSystem <TheRealMrSystem@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: ryangu00 <ryangu00@users.noreply.github.com>
Takeover of #2525, rebased onto current master. getHealth() in both engines
now computes orphan_pages and the timeline component over a linkable_pages
CTE driven by the same constants the orphans audit uses
(src/core/linkable-scope.ts), so one doctor report can no longer show a 19%
orphan_ratio next to a no-orphans score implying ~70%. Master's newer
first-segment exclusions (raw, atoms, skills) are folded into the shared
scope so the orphans audit loses nothing in the move.
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: pabloglzg <pabloglzg@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Takeover of #1988 (OpenRouter prompt caching), reimplemented on current
master: supports_prompt_cache may now be a per-model-id predicate; the
OpenRouter recipe marks openai/* chat and anthropic/claude-* routes
cacheable. Claude routes get an explicit cache_control on the system
content block via the recipe compat fetch shim (OpenRouter's documented
per-block format, not a top-level body field), signaled through a private
in-process marker header instead of the promptCacheKey sentinel that now
collides with the real OpenAI prompt_cache_key derivation. Cache reads on
OpenAI-compatible routes surface via the SDK's cachedInputTokens.
Root fix for #1135: deepseek, groq, and together now declare expansion
touchpoints (their expansion path is the same plain OpenAI-compatible
languageModel call as chat), so an explicit expansion_model pointed at
them no longer silently yields zero expansion.
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: tmchow <tmchow@users.noreply.github.com>
Co-authored-by: warkcod <warkcod@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
putPage lowercases slugs via validateSlug, but upsertChunks queried
pages by the caller's raw slug — so a mixed-case slug through
importFromContent created the page row, then failed the chunk upsert
with 'Page not found' and rolled back the whole import.
Normalize via validateSlug at importFromContent entry and inside
_upsertChunksOnce on BOTH engines (postgres + pglite parity).
Takeover of #855, rebased onto current master shapes (batchRetry
wrapper / _upsertChunksOnce, rewritten importFromContent opts block).
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Kage18 <Kage18@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
A source registered with `gbrain sources add --federated` was invisible to
an unqualified `gbrain search`/`gbrain query`: the local CLI always emitted
a scalar {sourceId} scope, and nothing on the read path ever consulted
sources.config.federated — contradicting docs/guides/multi-source-brains.md
('Source participates in unqualified gbrain search results').
Fix, at the trusted-local boundary only:
- src/cli.ts makeContext resolves the source WITH its tier and, when the
tier is non-explicit (local_path / brain_default / sole_non_default /
seed_default), computes ctx.localFederatedSourceIds = [resolved source,
...other config.federated=true sources] (archived excluded).
- New federatedSearchScope (operations.ts) delegates to
resolveRequestedScope, then widens an unqualified trusted-local scalar
scope to that set. Used by the search + query handlers only.
- Expansion NEVER applies when ctx.remote !== false (fail-closed source
isolation), when a per-call source_id/__all__ is passed, when an OAuth
grant (allowedSources) is present, or when --source/GBRAIN_SOURCE/dotfile
named the source explicitly.
Deliberately NOT inside sourceScopeOpts: code-intel ops reject multi-source
scopes (resolveCodeIntelScope) and non-search reads keep their scalar
behavior. Cache contamination is already handled — cacheScopeKey folds
sourceIds sets into the query-cache key.
Fixes#2561
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
- #1035: importFromContent preserves an existing page's type when incoming
frontmatter omits an explicit type: field. Explicit type stays an override;
absence means preserve; new pages still path-infer. The existing-page fetch
moved above the content-hash compute so a no-op re-put stays a hash-match
skip. Root-cause fix covers put_page, sync, capture — every caller.
- #1284: sync's end-of-run auto-embed no longer receives slugs deleted in the
same run (embedPage threw 'Page not found' per deleted slug and serr-logged
noise on every rename/delete sync). pagesAffected stays the full manifest
for extract/report paths; a slug deleted then re-imported in the same run
stays embeddable.
- #1196: gbrain serve --http now runs doctor's embedding_width_consistency
check at startup and prints a loud stderr banner (with the paste-ready
recipe + GBRAIN_EMBEDDING_MODEL/DIMENSIONS hint) when the resolved width
diverges from the brain's vector(N) column — the stateless-container
fallthrough that broke every write. Fail-open; reads unaffected.
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Missing ZEROENTROPY_API_KEY threw AIConfigError from auth resolution, which
rerank.ts recorded as reason 'unknown' — and doctor's reranker_health had no
unknown bucket, so it reported ok while every rerank silently failed open.
- gateway.rerank wraps AIConfigError from applyResolveAuth as
RerankError(reason: 'auth') before any HTTP call.
- checkRerankerHealth warns on >=3 'unknown' failures in the 7-day window
(covers historical pre-fix audit rows), with a ZEROENTROPY_API_KEY setup
hint when the error summary points at a missing key.
- Tests: RerankError(auth) classification, applyReranker fail-open + audit
reason, doctor warn on repeated unknowns.
Takeover of #2070.
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: maxpetrusenkoagent <maxpetrusenkoagent@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
fetchCountRows and detectDeadPrefixes in src/core/schema-pack/stats.ts
swallowed EVERY engine error into empty results, so any real failure
printed 'Total pages: 0' + a vacuous 100% coverage on a populated brain.
Both catches now swallow only isUndefinedTableError (pre-init brain,
missing pages table) and rethrow everything else. Four regression tests:
real non-zero count on a populated PGLite brain, rethrow on non-missing-
table errors in both catch sites, and the missing-table degrade path.
Takeover of #2493.
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: javieraldape <javieraldape@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
resolveTodayTasks only matched a bare '## Today' heading and bold-prefixed
'- [ ] **task**' lines, while the daily-task-manager skill's documented
Output Format writes '## P1 — Today' with plain '- [ ] task' lines — so
documented writes surfaced zero tasks in live context.
Reader now accepts both heading forms and both line forms, two-step: the
legacy bold prefix extracts just the task name (dropping trailing metadata),
falling back to the plain full-line form.
Salvaged from PR #2188 (reader-side half). The skill-doc rewrites in that PR
are dropped: master #2938 kept ops/ synced and made put_page write-through
durable, so the 'gbrain get/put ops/tasks' docs are correct as-is. The PR's
single-regex line matcher is replaced with the two-step match because its
alternation captured '**name** — metadata' verbatim for bold lines.
Takeover of #2188. Fixes#2186.
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: caioribeiroclw-pixel <caioribeiroclw-pixel@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
On macOS/Windows (case-folding filesystems), the write-through rename
silently clobbered a differently-cased file already occupying the target
path (uncontrolled repo files like README.md vs slug readme, or unicode
normalization variants between slugs). Refuse with
skipped: 'case_insensitive_collision' when the path exists on disk but no
exactly-named directory entry does; exact-case updates fall through and
case-sensitive filesystems are unaffected.
Fixes#2831
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The Dream cycle disables sync's inline extraction and routes changed
slugs through extractForSlugs, which flushed link/timeline batches but
never stamped links_extracted_at — so incrementally extracted pages
stayed permanently visible to `extract --stale` / doctor.
Collect processedRefs per successfully processed page and stamp them
via stampExtracted (best-effort) after both batch flushes, non-dry-run
mode 'all' only. Source-id threading from the original PR #2637 already
landed on master via #1503/#1747, so this rebase carries only the
missing watermark stamp plus regression tests.
Takeover of #2637.
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: JavanC <JavanC@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Adds GBRAIN_OAUTH_TOKEN_RATE_LIMIT_MAX and
GBRAIN_OAUTH_TOKEN_RATE_LIMIT_WINDOW_MS to tune the /token
client_credentials limiter (default unchanged: 50 req / 15 min).
Invalid, zero, or negative values fall back to the default.
Takeover of #2501 (mechanical rebase onto master after #2625 shifted
the surrounding context in serve-http.ts). Fixes#2463.
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: techtony2018 <techtony2018@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The cycle's embed phase called runEmbedCore with no output suppression, so
the '[dry-run] Would embed ...' / 'Embedded N chunks ...' slog summaries
landed on stdout ahead of the JSON CycleReport, breaking the documented
stdout-clean-for-JSON contract (docs/progress-events.md).
Adds EmbedOpts.quiet gating the human stdout summary slog sites in
embed.ts (embedPage, embedAll, embedAllStale); the cycle's runPhaseEmbed
sets quiet: true since it reports counts via its own PhaseResult. Errors
and warnings still go to stderr regardless.
Takeover of #854 (same approach, reimplemented on current master — the
original patch predates the slog migration and the widened
embedAll/embedAllStale signatures). Regression test ported from #854.
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Kage18 <Kage18@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Takeover of #2161. runRemediation ignored onboard-check extras in three
places: the pre-flight plan, the initial recommendation build, and the D7
mid-run recheck that rebuilds recs after every completed step. The --check
path threaded extras correctly, so `gbrain onboard --apply --auto` reported
"Nothing to do" when the only remediable work came from onboard checks —
and even with the first two sites fixed (the original PR diff), any plan
with 2+ steps dropped all remaining extras after step 1 via the recheck.
- Add RemediationOpts.extraRemediations; thread it through the pre-flight
plan, initial recs, and the mid-run recheck.
- Recheck filters extras to ids not already processed this run: extras
carry static status:'remediable', so unfiltered threading would resubmit
completed extras forever.
- Wire the CLI --auto path (onboard.ts) AND the MCP run_onboard auto path
(operations.ts), which already computed the scope-filtered allowedExtras
and then dropped it.
- Regression test: extras-only plan on an empty brain runs BOTH extras
exactly once and terminates (serial file: mock.module queue stub +
GBRAIN_HOME tmpdir).
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: brettdavies <brettdavies@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(autopilot,eval): nightly quality probe enable path works end-to-end + wire conversation-parser probe
Takeover of #2629 and #2630 (rebased onto master; dropped the
test/engine-find-trajectory.test.ts hunk both PRs carried — master
already ships the equivalent gateway-dims fix).
#2629 — nightly quality probe enable path:
- autopilot + doctor read the probe flag dual-plane (DB config row from
'gbrain config set' wins, ~/.gbrain/config.json fallback) via new
resolveProbeEnabled/resolveProbeMaxUsd helpers
- resolveRepoRoot prefers the gbrain package root where the committed
fixture lives, not the brain repoPath
- rate_limited skips no longer write an audit row every autopilot cycle
- eval-longmemeval strips 'provider:' recipe ids before raw Anthropic SDK
calls and emits the gold answer for downstream judges
- cross-modal batch folds the gold answer into the judge task; probe
passes QA-shaped dimensions instead of the agent-response rubric
- DEFAULT_SLOTS slot A moves to openai:gpt-5.2 (gpt-4o left the recipe);
new consistency test pins every default slot to its recipe
#2630 — conversation-parser nightly probe wire-up:
- autopilot step 4.6 invokes runConversationParserNightlyProbe (dual-plane
flag + D10 tokenmax mode-gate, package-root fixtures, 24h gate, audit
trail via new src/core/audit-parser-probe.ts)
- doctor's conversation_parser_probe_health replaces the hardcoded
'Skipped' stub with a real pure-function check over the audit trail
Co-authored-by: p3ob7o <p3ob7o@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(pricing): add openai:gpt-5.2 canonical entry for the new default slot A
DEFAULT_SLOTS slot A moved to openai:gpt-5.2, which had no CANONICAL_PRICING
entry — estimateCost silently dropped slot A from the --max-usd pre-flight
and est_cost_usd audit rows (~1/3 under-count on the default panel). Rates
from the OpenAI recipe chat touchpoint (verified 2026-04-20). Also refresh
the --slot-a-model help text default and pin a pricing-presence assertion
in the DEFAULT_SLOTS consistency test.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: p3ob7o <p3ob7o@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
- phaseBFenceFacts now queries legacy rows FIRST and dirty-checks only
the source_ids it will actually write into. Zero fenceable rows (or
rows scoped to clean sources) no longer fail on an unrelated dirty
source. Targeted-dirty-source refusal unchanged. Fixes#927.
- apply-migrations now prints each failed phase's name + detail to
stderr alongside 'reported status=failed', instead of burying the
actionable message in the ledger. Fixes#921.
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Two backlog fixes:
- init (#1058): loadConfig() returns null on a cold install (no config.json
AND no DATABASE_URL), short-circuiting before its env merge — so
GBRAIN_EMBEDDING_MODEL / GBRAIN_EMBEDDING_DIMENSIONS /
GBRAIN_EXPANSION_MODEL / GBRAIN_CHAT_MODEL were silently ignored and
Tier-3 detection auto-picked by API key instead. resolveAIOptions' config
seed now falls back to those env vars directly when loadConfig() is null
(new exported helper seedAIOptionsFromConfig, env-injectable for tests).
- whoami (#1061): the stdio MCP dispatch is remote/untrusted by design but
has no per-token auth (local pipe), so whoami threw unknown_transport on
the primary stdio surface. The stdio dispatch now marks
ctx.transport = 'stdio' and whoami returns {transport: 'stdio', scopes: []}
for it. Trust posture unchanged: remote stays true, the marker is never
used for trust decisions, and an unmarked auth-less remote context still
throws (fail-closed preserved).
Co-authored-by: Garry Tan <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
MiniMax's /v1/embeddings endpoint is not OpenAI-compatible: it requires
texts (not input) plus a type field and returns {vectors} instead of
{data:[{embedding}]}. The recipe shipped no transport shim, so every
embed call failed with an invalid-params error, and it declared no chat
touchpoint, so assertTouchpoint blocked gbrain think even though
MiniMax chat is genuinely OpenAI-compatible.
Fix (takeover of #2882, corrected):
- minimaxCompatFetch via the DeepSeek-style compat.fetch seam (keeps
cfg.base_urls overrides working; no new env var), gated on the
/embeddings path so chat requests/responses pass through untouched.
- Response rewrite parses via resp.clone() and rebuilds with fresh
headers — never returns a body-consumed Response (the flaw in #2882's
wrapper, which broke every non-streaming chat completion).
- chat touchpoint with the /v1/models list from #1977.
Fixes#1977
Co-authored-by: Garry Tan <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: ArthurHeung <ArthurHeung@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
deriveDirectUrl() swaps the Supabase pooler host to db.<ref>.supabase.co:5432,
which is IPv6-only without the paid IPv4 add-on. On IPv4-only networks the
direct pool could never connect, and initDirectPool()'s throw killed
'gbrain init --url' and migrations with ENOTFOUND/ECONNREFUSED.
getDirectPool() now classifies network-unreachable errors (ENOTFOUND,
ECONNREFUSED, ENETUNREACH, EHOSTUNREACH, ETIMEDOUT, CONNECT_TIMEOUT) via the
new isNetworkUnreachableError(), self-activates the kill-switch, logs one
stderr line pointing at GBRAIN_DIRECT_DATABASE_URL / GBRAIN_DISABLE_DIRECT_POOL,
and returns the read pool. Auth/SQL errors still throw (misconfig, not
unreachability). The failed pool is ended via endPoolBounded so it can't
leak sockets into the now-continuing process.
Also surfaces the kill-switch + override envs in the init.ts IPv6 warnings
and docs/guides/live-sync.md (they were previously undocumented outside
connection-manager.ts).
Fixes#1641
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The zhipu recipe was embedding-only, so models.tier.subagent=zhipu:glm-5.1
threw "does not offer a chat touchpoint" — while the error hint falsely
listed zhipu (and dashscope/minimax, also embedding-only) among providers
with chat.
- zhipu recipe: add a chat touchpoint (glm-5.1 family, supports_tools +
supports_subagent_loop; no Anthropic-style prompt cache on the
OpenAI-compat path, so the loop runs with the degraded:no_caching warn).
openai-compat tier means newer GLM ids pass without a recipe edit.
- capabilities.ts: compute the "Known providers with chat" hint from the
recipe registry instead of a hardcoded list, so it can never drift into
naming chat-less providers again.
- Declines the originally requested models.anthropic_compatible_prefixes
config: v0.38's recipe-driven capability gate already replaced the
Anthropic-only enforcement, so a recipe chat touchpoint is the whole fix.
Fixes#1157
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Master already widened slugifySegment (sync.ts) and validatePageSlug
(operations.ts) to CJK in v0.32.7, but the other two validators #782
targeted stayed ASCII-only: SlugRegistry's SLUG_RE rejected any CJK
desiredSlug from BrainWriter, and synthesize.ts's SUMMARY_SLUG_RE (whose
comment claimed it was kept in sync with validatePageSlug) rejected CJK
output roots.
Hoist the segment grammar into cjk.ts as PAGE_SLUG_SEG and compose all
three regex sites from it, so the four slug validators share one grammar.
Each site keeps its own shape (SlugRegistry's >=2-segment dir/name form,
validatePageSlug's case-insensitive flag).
Scope stays CJK (matching v0.32.7), not full \p{L} Unicode as #782
proposed — all-scripts slugs (lookalike/RTL spoofing) is a maintainer
policy call.
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: tamagodo-fu <tamagodo-fu@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Takeover of #2510: migrations v40 (facts) and v55 (query_cache)
unconditionally created HNSW indexes with the configured embedding
dimension, so `gbrain init` with embedding_dimensions above pgvector's
per-type HNSW caps (vector 2000 / halfvec 4000) failed with
"column cannot have more than 4000 dimensions for hnsw index".
- vector-index.ts: add PGVECTOR_HNSW_HALFVEC_MAX_DIMS + hnswMaxDimsForType
- migrate.ts v40/v55: emit the HNSW index only when dims fit the cap,
otherwise a comment noting exact scans remain available
- embedding-dim-check.ts: buildFactsAlterRecipe skips the reindex step
above the cap for the same reason
- tests: 4096d init round-trip on PGLite (columns exist, indexes
skipped) + recipe-skip unit test
Drops the unrelated context-engine.ts interface change and the
tsconfig.json strictFunctionTypes=false hunk from #2510; typecheck is
clean without them.
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: javieraldape <javieraldape@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(calibration): resolve owner holder via config (default 'self'), fixes#2464
Takeover of #2467 (rebased onto master). consolidate writes owner takes
with holder='self' while calibration-profile, calibration CLI/op, think's
calibration block, emotional-weight, and doctor's calibration_freshness
all defaulted to a hardcoded 'garry' — so getScorecard returned 0
resolved and the calibration profile never built on non-upstream brains.
New src/core/owner-holder.ts is the single source of truth:
resolveOwnerHolder({override, configValue}) = override >
emotional_weight.user_holder config > 'self'. All six call sites route
through it; doctor's freshness SQL is parameterized ().
Upgrade note: upstream-owner brains with historical holder='garry'
profiles should `gbrain config set emotional_weight.user_holder garry`
to keep reading them.
Co-authored-by: devty <devty@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(calibration): replace real-name holder fixture with charlie-example placeholder
Privacy iron rule: no real people's names in checked-in code. The sanctioned
placeholder mapping uses people/charlie-example.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: devty <devty@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
The help text (gbrain check-backlinks <check|fix> [dir]) promised a
positional directory argument, but runBacklinks only parsed --dir and
defaulted to cwd, so the walker ran from the wrong root and could hit
EPERM on unreadable sibling dirs.
Extract parseBacklinksArgs: positional [dir] is now honored, --dir still
overrides it, --dry-run preserved, and a --dir flag missing its value
falls back to the positional dir instead of picking up undefined.
Takeover of #852 (rebased onto master past the findBacklinkGaps dedupe
test block). Fixes#485.
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Kage18 <Kage18@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
doctor.ts pushes runAllOnboardChecks results into the checks list, but the
7 onboard check names (embed_staleness, entity_link_coverage,
timeline_coverage, takes_count, dangling_aliases, pack_upgrade_available,
type_proliferation) were never added to doctor-categories.ts, so every
doctor run emitted an 'unknown check name' stderr warn per onboard check.
Registers the 5 data-quality names under BRAIN and the 2 schema-pack names
under META (alphabetical order preserved), and widens the drift-guard test
to scan src/core/onboard/checks.ts alongside src/commands/doctor.ts so
future onboard checks can't drift uncategorized.
Takeover of #1839, rebased onto master (keeps master's timeline_dedup_index).
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: mvanhorn <mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Title precedence is now frontmatter title: > body's first ATX H1 > the
slug/filename-humanized fallback. Slug-based imports (contacts, calendar)
carry a correct # Heading but no frontmatter title; without the H1 fallback
they got junk titles humanized from the slug. The H1 scan skips h2+ and
lines inside fenced code blocks, and strips closed-ATX trailing hashes.
Takeover of #2495.
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: javieraldape <javieraldape@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Three fixes in the same leak class (a remote caller reading or writing
outside its granted sources), for multi-source / multi-tenant brains:
1. dispatchToolCall now refuses remote calls that arrive without a
resolved sourceId (missing_source_scope error envelope) instead of
silently falling back to the shared 'default' source. Every shipped
transport already passes sourceId explicitly (serve-http from the
OAuth client row, http-transport from the legacy token grant, stdio
from GBRAIN_SOURCE); reaching the fallback remotely always meant a
programmatic caller skipped scope resolution — the bug class behind
#1924 / #1371. Trusted local callers (remote === false) keep the
historical fallback. Direct-dispatch tests updated to carry an
explicit sourceId, matching the real transport contract.
2. log_ingest threads ctx.sourceId (same pattern as get_chunks /
get_page), so ingest events are attributed to the caller's source
instead of piling into 'default'. Engines already accept
entry.source_id (v0.31.2).
3. get_ingest_log is source-scoped for remote callers via the
linkReadScopeOpts collapse rule (scalar grant → [scalar]; federated
grant → granted array); it previously returned the whole brain's
ingest log to any read-scoped remote client, and ingest summaries can
carry another source's private context. Trusted local callers keep
the whole-brain view.
Tests: dispatch guard (refuse remote-without-source, keep local
fallback, guard ordering after op lookup) and end-to-end ingest-log
attribution + scoping over the real dispatch path, on PGLite.
gbrain init --no-embedding writes embedding_disabled: true as a
deferred-setup sentinel, and init/import/embed honor it via
assertEmbeddingEnabled. sync's embed credential preflight (v0.41.6.0 D1)
only checked the --no-embed CLI flag, so every gbrain sync on a keyless
deferred-setup brain exited 1 demanding <PROVIDER>_API_KEY — including
orchestrated callers (gstack /sync-gbrain) that never pass --no-embed.
embed-preflight.ts's own skip protocol documents that the sentinel is
owned upstream of the credential check; this wires that contract into
sync by deriving noEmbed from CLI args + config in one exported pure
helper (resolveNoEmbed), covered by test/sync-no-embed-sentinel.test.ts.
Co-authored-by: Gawie van Blerk <gawie.vanblerk@emeraldlife.co.za>
A page write is not 'done' until it is readable back. After the import
transaction commits, verify the page resolves via getPage and its
content_hash matches what was just written. On mismatch or miss, fail
LOUDLY instead of reporting success, and record the failure in
ingest_log (best-effort) so it is durable and agent-inspectable rather
than a transient stderr message.
This catches the silent-desync class: the page file exists on disk (or
the git commit landed) but the DB index never picked the write up —
the operation previously reported success while the page stayed
invisible to every read path (get_page, search, query) until someone
noticed the gap manually.
Guard applies to both importFromContent (markdown) and importCodeFile.
Tests: new write-verify-guard suite (hermetic PGLite) covering the
happy path, index-miss, stale-hash, ingest_log record, and the put_page
operation surface; import-file.test.ts mock upgraded to simulate a
readable DB (writes are read-backable), matching the new guard.
PRJ-2026-032
Co-authored-by: merlin-drizzyenterprises[bot] <144527811+merlin-drizzyenterprises[bot]@users.noreply.github.com>
list_pages clamps limit to max 100 (default 50) — deliberate server
protection, pinned in test/search-limit.test.ts. But the clamp was
SILENT: a caller whose limit was defaulted or clamped got a
full-looking array with no signal that rows were dropped, and with the
default updated_desc sort the dropped rows are always the OLDEST —
precisely what exhaustive consumers (audits, scans, backfills) exist
to find. Observed in the field: a source with 212 pages enumerated as
80 visible rows, hiding 26 pages from a compliance scan for days.
Fix, with no response-shape change (MCP consumers still get an array)
and no engine surface change (handler probes limit+1):
- handler probes one row past the effective limit; when the caller's
limit was NOT honored (unset -> default, or clamped to cap) and rows
were dropped, it warns on stderr for local (CLI) callers — same
operator-facing channel as the put_page unknown-type hint, but
without the isTTY gate: scripted callers are exactly the consumers
that cannot detect truncation any other way, and stderr keeps stdout
parseable. An explicit honored limit stays silent (ordinary
pagination), as does a clamped-but-complete result. Remote (MCP)
ctx never writes to stderr.
- LIST_PAGES_DESCRIPTION documents the cap and the exhaustive-listing
recipe (sort=updated_asc + updated_after cursor) — the description
is the signal channel MCP clients actually read.
- regression suite: default-limit truncation warns, honored limit
silent, clamped-but-complete silent, remote silent, and the
documented cursor recipe enumerates a corpus to completion.
Co-authored-by: YMYD <53603073+OJ-OnJourney@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
On machines with neither gtimeout nor timeout on PATH, run-verify-parallel.sh
and run-unit-parallel.sh fall back to a bg-pid + sleep-watchdog cap. Both
read $? only after tearing the watchdog down (kill + wait on cap_pid), so the
sentinel .exit files recorded the killed watchdog's status — 143 — instead of
the check/shard's own exit code. Every run reported total failure (verify:
pass=0 fail=31; unit: rc=143 per shard) while every per-check/shard log
showed success.
Capture rc immediately after `wait $pid` in both scripts, and reap the
watchdog's sleep child (pkill -P, children-first — the same orphan quirk the
heartbeat cleanup documents) so the fallback stops leaking one sleep per
check/shard.
Regression tests force the fallback branch hermetically on any host via a
curated PATH with no timeout binaries: the verify dispatcher runs from a
tempdir copy with a stubbed `bun`, pinning exit 0 + all-zero sentinels when
checks pass and the check's own rc (not 143) when one fails; the unit wrapper
runs real two-shard fixture passes, pinning rc=0 sentinels and a real
failure's rc=1.
Co-authored-by: YMYD <53603073+OJ-OnJourney@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
When the recommendation list is refreshed between remediation steps, a
remediation that doesn't clear its own health signal is reintroduced
under its stable id and attempted again, indefinitely on long runs.
Track attempted recommendation ids for the run and skip re-attempts.
Includes a behavioral regression test: a persistently-stuck signal is
attempted once, the loop terminates, and other remediations still run.
Dispatch timeout was derived as interval*2 with a 5-minute floor, tuned
for light per-interval work. A full autopilot cycle routinely needs more
than 10 minutes at common intervals, so healthy full cycles were killed
mid-run. Full-cycle dispatch now gets a 30-minute floor; lighter
dispatches keep the interval-derived budget.
Adds a regression test for the full-cycle floor.
upsertChunks fell back to the compile-time DEFAULT_EMBEDDING_MODEL
('zeroentropyai:zembed-1') when a ChunkInput carried no explicit `model`.
The embed pipeline (src/commands/embed.ts) builds ChunkInputs without a
`model` field, so rows whose vectors were produced by the config-resolved
model (e.g. openai:text-embedding-3-large) were mislabeled with the
hardcoded default — corrupting the provenance that signature-drift
staleness and dimension-migration logic depend on.
Both engines now resolve the gateway's runtime embedding model once per
upsert and use it as the fallback, mirroring the existing resolve-then-
default pattern used for schema sizing. Regression test added (pglite);
verified via negative control that it fails against the old fallback.
This is a write-path change (upsertChunks), not a search-path change, so
retrieval eval replay is not applicable.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
`sources.config` is a jsonb OBJECT column, but a read→write cycle that
JSON.stringify'd an already-stringified value re-wrapped it into a JSON string
scalar ("{}", "\"{}\"", ...) that grew one layer per write. parseSourceConfig
only unwrapped one layer, so the corruption never healed and federation/ACL
reads saw a string instead of the settings object.
- Add normalizeSourceConfig: a bounded (10-iteration) loop that JSON.parses
while the value is a string and returns {} (with a console.warn) when the
result is not a plain object. All six `UPDATE sources SET config` writers run
their config through it before stringify, converging the stored value back to
a jsonb object on the next write.
- parseSourceConfig now does the same bounded unwrap and warns once when more
than one layer was found (one layer is the normal PGLite path).
- Add a `source_config_shape` doctor check that flags any sources row where
jsonb_typeof(config) <> 'object', with the repair path.
- Unit-test the helper (object passthrough, 1-layer, 5-layer nested, garbage
and over-bound inputs) and the doctor check (mock engine).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Issue #2298: 'gbrain doctor' surfaced two distinct timeline
metrics under the same user-facing 'timeline' label:
1. Entity timeline coverage (graph_coverage metric)
- numerator: eligible entity pages WITH a timeline entry
- denominator: eligible entity pages
- 0-1 fraction, surfaced by graph_coverage check
2. Whole-brain timeline density (brain-score 0-15 component)
- numerator: all pages WITH a timeline entry
- denominator: all pages
- 0-15 scale, surfaced by brain_score breakdown
These have DIFFERENT numerators/denominators. The old single
'timeline X%' label let a reader mistake the entity-scoped
percentage for whole-brain density.
Presentation/contract clarity only — scoring formula,
health weights, takes, source routing, extraction UNCHANGED.
- doctor.ts graph_coverage: 'timeline X%' -> 'entity timeline coverage X%'
- doctor.ts brain_score: 'timeline X/15' -> 'timeline density (all pages) X/15'
- cli.ts get_health: 'Timeline coverage (entity pages)' ->
'Timeline density (all pages): X/15 (whole-brain brain-score component)'
Test: test/doctor-timeline-metric-labels-2298.test.ts uses a
synthetic in-memory PGLite fixture (NO private EriadorMu data):
4 total pages, 2 eligible entity pages, 1 entity page with a
timeline entry, 1 total page with a timeline entry.
Expected: entity coverage 1/2 = 50%; whole-brain density
1/4 -> round(25% * 15) = 4/15. Asserts the two metrics
render with distinct scoped labels and the brain-score component
is explicitly whole-brain (no 'entity' in its label). 5/5 pass.
Addresses #2298
extractStaleFromDB still used the pre-#972 `includeFrontmatter ? resolver :
nullResolver` ternary. The synthetic resolver has no resolveBasenameMatches,
so the gate in extractPageLinks skipped the issue-#972 bare-wikilink pass
regardless of link_resolution.global_basename — the sweep stamped every page
as extracted while silently dropping its [[bare-name]] links. Same brain,
same pages: `extract --stale` created 0 links where `extract links --source
db` created 218.
- Always pass the real batch resolver; gate passes via extractPageLinks opts
({ skipFrontmatter: !includeFrontmatter, globalBasename }), mirroring
extractLinksFromDB — including the codex-[P1] sourceId scoping.
- Bump LINK_EXTRACTOR_VERSION_TS (documented protocol) so pages stamped by
the broken sweep re-flag stale and re-extract under the fixed logic.
- Regression tests: bare wikilink resolves on --stale with the flag ON;
still drops with the flag OFF (back-compat). The #1768 fixture now derives
its updated_at from LINK_EXTRACTOR_VERSION_TS instead of a hardcoded date,
so future version bumps can't silently flip its version arm.
Fixes bug 1 + bug 3 of #2576. Bug 2 (DIR_PATTERN gaps) is a separate
whitelist design call, intentionally not addressed here.
Co-authored-by: YMYD <53603073+OJ-OnJourney@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
An empty --content (most commonly a non-interactive caller that meant
file input — put has no --file flag — so the missing --content fell
back to reading empty stdin) silently blanked existing pages. put_page
now rejects an empty/whitespace-only body over an existing non-empty
page with invalid_params, pointing at `gbrain capture --file PATH
--slug SLUG` for file input; allow_empty: true (CLI: --allow-empty)
opts into an intentional blank. The guard read is scoped to the exact
(source_id, slug) row the write targets; new-slug creates and
soft-deleted-page overwrites stay allowed.
Co-authored-by: Matthew Thompson <matthew@symmetric-consulting.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
DashScope's OpenAI-compatible rerank endpoint lives at
{base}/compatible-api/v1/reranks — PLURAL leaf, different base path from
the embedding surface (compatible-mode). Reusing llama-server-reranker
against DashScope forces users to hand-patch the recipe's '/rerank' leaf
in node_modules, which every upgrade silently reverts (and llama.cpp
genuinely serves singular /rerank, so changing that recipe would break
real llama.cpp users).
New dedicated recipe rides the v0.40.6.1 recipe-pluggable reranker path:
- id dashscope-rerank, base_url_default compatible-api/v1 (intl), ZE wire
- path '/reranks', default_timeout_ms 30s, 5MB payload ceiling
- models: only qwen3-rerank (live-verified 200; gte-rerank-v2 is rejected
by the compat surface with 'Unsupported model for OpenAI compatibility
mode', so it is deliberately not listed)
- separate recipe (not a reranker touchpoint on dashscope) because
provider_base_urls is keyed by recipe id and the two capabilities need
different prefixes — same topology as llama-server vs
llama-server-reranker
Tests: recipe shape smoke mirroring recipe-llama-server-reranker.test.ts
(path/timeout/payload pins, /v1/v1 concat guard, auth resolve, sibling
recipe isolation). bun test test/ai/: 322 pass / 0 fail.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The four `hybridSearch — reranker enabled (reorder)` cases stub the gateway
at 1536 dims (DIMS). Since v0.36.3.0 hybridSearch resolves the embedding
column via loadConfig(), whose precedence is
cfg.embedding_dimensions > gateway dims > default. On any machine whose
~/.gbrain/config.json sets embedding_dimensions to something other than 1536
(e.g. text-embedding-3-small at 1280), the real config outranks the stub: the
1536-d stub vector fails the gateway dim check, the error is swallowed, search
falls back to keyword-only, and the reranker never runs (rerankerFn gets 0
docs, rerank_score undefined). Green in CI only because a fresh runner has no
config file — deterministic red on a contributor's machine.
Fix (test-only): isolate GBRAIN_HOME to an empty tmpdir in beforeAll so
loadConfig() returns null and the stub's dims win, then restore it and clean
up in afterAll. Same idiom as emptyHome() in
test/ai/gateway-probe-chat-model.test.ts.
Verified with a planted ~/.gbrain/config.json at 1280 dims: 2 pass / 4 fail
before, 6 pass / 0 fail after; still green with no config file. typecheck clean.
Fixes#1527
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
getHealth's entity_pages CTE and the top-linked-pages query only match the
legacy 'person' and 'company' types, so brains using the gbrain-base-v2
pack's 'entity' type report 0% entity link/timeline coverage in `gbrain
health` even when doctor's graph_coverage shows real coverage. Add 'entity'
to both queries in both engines (PGLite + Postgres, in lockstep per the
engine-parity rule) and extend the getHealth graph-metrics test with an
entity-typed page.
Validation: bun test test/pglite-engine.test.ts --test-name-pattern 'getHealth graph metrics' (5 pass).
Replace the strictly sequential per-chunk synopsis loop with a bounded
sliding worker pool (existing runSlidingPool helper). Results land in
chunk order via index-addressed writes; code chunks still bypass the
wrapper; embedding remains one page-level batch after all synopses.
New knob GBRAIN_CONTEXTUAL_CHUNK_CONCURRENCY, default 4, clamped to
[1,16]; 1 reproduces the prior sequential behavior exactly. Each chunk
task still acquires/releases the global synopsis rate-lease, which
remains the cross-worker governor; the lease id now travels from
acquire to release instead of shared mutable state, and lease waits
are abort-responsive.
At 20-45s per synopsis call, a 120-chunk transcript page previously
needed 60-90+ min wall time and routinely outlived job timeouts.
statusForVersion short-circuited on any 'complete' entry before checking
the trailing 'retry' marker, so --force-retry appended an inert row and a
version marked complete with zero work done could never be re-run without
hand-editing completed.jsonl. Check retry-latest first: an explicit
--force-retry now yields 'pending' even past an earlier 'complete', while
a stray 'partial' after 'complete' still cannot regress the version.
gbrain list --limit 100000 silently returned 100 rows (default 50) with
no warning, and --offset was accepted but dropped at the op layer even
though PageFilters has supported it all along.
- Local CLI callers (ctx.remote === false, the same trust boundary that
already bypasses scope enforcement) get an explicit limit above 100
honored — full enumeration is a legitimate local operation.
- Remote MCP/OAuth callers keep the 100-row DoS cap, now loud: one
logger.warn (stderr, stdout stays script-clean) with both numbers,
parity with the three search-path clamp warnings.
- offset is declared as a param (so the CLI coerces it to number) and
threaded to engine.listPages for real pagination.
Claude-Session: https://claude.ai/code/session_01Vswwe1y5fQbJWfbaSK3enT
Co-authored-by: deacon-botdoctor <291411030+deacon-botdoctor@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
SupabaseStorage.getSignedUrl built the download URL as `${projectUrl}${signedURL}`,
but Supabase's sign API returns `signedURL` relative to the Storage API root
(/object/sign/<bucket>/<path>?token=...), so the generated link dropped /storage/v1
and returned 404. Now prepends `${projectUrl}/storage/v1`, tolerating an
already-absolute URL or a value that already carries the prefix. `gbrain files
signed-url` links resolve again.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Reasoning models (MiniMax-M3, DeepSeek-R1, etc.) return <think>...</think>
tags in the content field before the actual JSON output. This caused
parseExtractorOutput to fail in two ways:
1. The fence regex /^\`\`\`(json)?...$/ requires the fence at text start;
<think> preceding it prevents matching, so the raw text (with trailing
fences) hits JSON.parse and throws.
2. When think tags contain [ or { characters, indexOf finds them inside
the reasoning block instead of the actual JSON array.
Changes:
- Strip <think>...</think> tags before any parsing (covers all reasoning models)
- Add JSON.parse fallback: truncate at last ] or } to handle trailing
noise (leftover markdown fences after stripping)
Tests: 28/28 pass (3 new cases for think tags + trailing noise).
Co-authored-by: qaz8545355 <junjun@openclaw.local>
The idempotency row is only written inside `for (const p of proposals)`, so a
page that extracts ZERO gradeable claims never records an idempotency tuple
and is re-sent to the LLM on every cycle forever. The docstring's "unchanged
page never re-spends tokens" contract only holds for pages that produce >=1
claim; a page that legitimately has no gradeable claims (or any machine-
generated page) is a perpetual cache miss and re-spends tokens indefinitely.
Fix: when `proposals.length === 0`, write one tombstone row keyed by the same
(source_id, page_slug, content_hash, prompt_version) tuple, with
status='rejected' so it never surfaces in a pending-review query (the pending
index filters status='pending'). Content changes (new content_hash) or a
PROPOSE_TAKES_PROMPT_VERSION bump still miss the tombstone and re-extract. The
extractor-throw path `continue`s before the tombstone, so failed pages are
retried rather than cached.
Guard against a subtle regression: `parseExtractorOutput` returns [] for BOTH
a genuine empty extraction AND malformed/prose/truncated model output, so
naively tombstoning every [] would permanently suppress a page that has claims
but hit a transient parse failure. `defaultExtractor` now throws when the
output is empty-but-not-a-clean-`[]` (new `isWellFormedEmptyExtraction`
predicate), routing transient failures into the existing retry path; only a
cleanly-parsed empty array is memoized.
Adds a `tombstones_written` counter for observability.
Tests: tombstone written on genuine empty extraction; two-cycle idempotency
(no repeat LLM call on an unchanged zero-claim page); extractor error writes
no tombstone; isWellFormedEmptyExtraction discriminates clean-[] from
malformed/prose/non-empty output. propose-takes suite: 36 pass / 0 fail.
Co-authored-by: ivandebot <187176982+ivandebot@users.noreply.github.com>
The empty-fence guard counted every `row_num IS NULL AND entity_slug IS NOT NULL`
row as a pending v0_32_2 backfill, but the inline facts writer keeps producing
rows of exactly that shape post-migration: when a resolved slug has no fenceable
page (slugify-floor / stub-guard-blocked unprefixed slugs like `wingman`,
`people-jane-doe`), backstop.ts falls through to a DB-only insert with row_num
NULL. Those rows are structurally unfenceable — no page to fence onto, and the
ledger-complete migration won't re-run — so they jammed the phase forever
(~16/day observed) and the warning advised a no-op `apply-migrations --yes`.
Discriminator: a row is a genuine backfill candidate only if its entity_slug
resolves to a LIVE page in the same source (EXISTS in `pages` with deleted_at
NULL) — mirroring the migration's Phase B, which only fences slugs that map to
a writable page. Genuine pre-v0.32.2 rows (their entity page exists) still gate;
inline-writer unfenceable rows no longer do. Warning text updated to name the
"entity page present, not yet fenced" condition.
Regression tests pin both sides: unfenceable rows (no page / soft-deleted page)
do NOT gate and the phase converges; a legacy row WITH a backing page still
gates. Fails pre-fix, passes post-fix.
(#2484)
Co-authored-by: Javier Aldape <javieraldape@Javiers-Laptop.local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`gbrain schema stats` reported "Total pages: 0" on populated brains because
fetchCountRows wrapped its count query in a bare `catch { return []; }` that
converted EVERY error into zero rows — false 0 pages, false "100% coverage"
(0/0 → vacuous 1.0), and a starved `schema suggest`. A sibling bare catch in
detectDeadPrefixes had the same defect.
Root cause is the masked error, NOT a PGLite query incompatibility: reproduced
the exact COUNT query (COALESCE/NULLIF/GROUP BY/ORDER BY ... NULLS LAST) against
the pinned PGLite 0.4.3 (PG17.5) through the real engine + full schema, plus
PG18 and NULL/empty edge-case data — it returns correct counts every time and
never throws. The issue's "the query is failing on PGLite" premise doesn't
reproduce; the actual failure on the reporter's brain was hidden by the catch
(they could not capture it, consistent with an engine/init-level throw). The
honest fix is to stop hiding it.
Both catches now swallow ONLY the genuine missing-table case via the existing
isUndefinedTableError helper (SQLSTATE 42P01 + PGLite "relation ... does not
exist") and rethrow everything else, so the next occurrence shows the real
error instead of a fake zero. Pre-init/empty-brain behavior is preserved.
Regression: 4 new cases in test/schema-pack-stats.test.ts pin (1) real non-zero
count on a populated PGLite brain, (2) fetchCountRows rethrows a non-missing-
table error, (3) fetchCountRows still degrades to empty on 42P01, (4)
detectDeadPrefixes rethrows via the sibling catch. Each error-surfacing test
verified to fail when its catch is re-broadened.
Co-authored-by: Javier Aldape <javieraldape@Javiers-Laptop.local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Code legitimately contains tiktoken special-token strings (e.g. CLIP/GPT tokenizers embed the literal <|endoftext|>). The default encode() uses disallowed_special='all' and THROWS on those, crashing reindex-code on valid source files. Re-encode treating them as ordinary text (allowed=[], disallowed=[]); heuristic fallback if even that fails. A token COUNT needs no special-token semantics.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
putPage lowercases the slug via validateSlug, but the tag/link/timeline
reconcilers (tx.addTag, addLink, addTimelineEntry) query the slug as passed.
A remote put_page with a capitalized slug (e.g. 'Projects/Team-Wiki/Quarterly-Roadmap')
stored the page under 'projects/team-wiki/quarterly-roadmap', then threw
'addTag failed: page "…" not found' on the existence check, rolling back the
entire write — so the page never persisted under either casing. Any agent
driving the HTTP MCP server (where slugs arrive verbatim) lost every page whose
slug carried a capital letter plus a frontmatter tag.
Normalize the slug once at the top of importFromContent (the shared chokepoint
for MCP put_page and CLI capture) so putPage and every reconciler agree on the
canonical lowercased slug. No-op for disk imports (already slugifyPath output),
idempotent with putPage's own validateSlug call. Engine-agnostic, so PGLite and
Postgres move together.
Adds test/put-page-mixed-case-slug-tags.test.ts pinning the regression on PGLite.
* fix(write-through): guard mkdir against EEXIST on Bun+Windows
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(budget): resolve non-Anthropic model pricing via canonical table under --max-cost
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(enrich): exclude dream-generated pages from thin candidates
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
When link_resolution.global_basename is enabled, extend basename-index
resolution to frontmatter link fields (FRONTMATTER_LINK_MAP), mirroring the
body bare-wikilink path added in #972.
Problem: a bare-title wikilink in a frontmatter list -- e.g.
sources:
- "[[2025-12-25_mentor-extraction]]"
never resolves. SlugResolver.resolve() has no '/' to hit the slug-direct
getPage, and the field's dirHint (sources -> ['source','media']) may name
folders absent from the brain, so the dir-scoped exact + fuzzy steps also
miss. The frontmatter path never consulted resolveBasenameMatches -- that was
wired only for body bare-wikilinks. On a PARA/Obsidian vault this silently
drops the bulk of sources:/related: provenance edges.
Fix: extractFrontmatterLinks takes a globalBasename flag (threaded from
extractPageLinks). On a resolve() miss, unwrap [[ ]] and fall back to
resolver.resolveBasenameMatches -- UNIQUE-MATCH-ONLY, so ambiguous basenames
(archive dupes, generic hubs like _index) stay unresolved rather than create
a wrong edge. Purely additive; resolved frontmatter edges are unchanged.
Scope: covers the db-source extract and live put_page paths (real
makeResolver). The --source fs extract uses an inline resolver without a
basename index, so it gracefully no-ops there (typeof guard).
Tested: 3 new cases (resolves-when-on, ambiguous-stays-unresolved,
gated-off-by-flag); full link-extraction suite green (130 pass).
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The hybrid recency stage in runPostFusionStages imported
DEFAULT_RECENCY_DECAY directly, so operator overrides via the
GBRAIN_RECENCY_DECAY env var and the gbrain.yml `recency:` section were
honored only on the get_recent_salience SQL path and silently ignored on
the hot hybridSearch path. Non-default vault layouts therefore stayed on
the baked-in defaults / DEFAULT_FALLBACK (90d / 0.5) regardless of
tuning.
Call resolveRecencyDecayMap() (already used by the SQL path) so the
configured decay map reaches the boost stage. Behavior is unchanged when
no override is set — resolveRecencyDecayMap() returns DEFAULT_RECENCY_DECAY.
Adds test/hybrid-recency-config.test.ts asserting the env override
reaches the applied recency factor (fails against the prior wiring).
Single-file `frontmatter validate` derived the expected slug from the
absolute path: relative(resolve(target), file) is empty when target IS the
file, so it fell back to `|| file` (the full path), yielding "root/<abs>"
slugs and a false SLUG_MISMATCH. The pre-commit hook from install-hook
validates staged files one-by-one, so this rejected every commit in a
markdown brain (only bypassable with --no-verify).
Walk up to the brain root (nearest .git) and use relative(brainRoot, file)
|| basename(file), matching runAudit/runGenerate and sync/extract. Files
above the root fall back to basename instead of a ../-prefixed slug.
Reopens#565. Present since v0.32.0; reproduced on v0.42.51.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
OpenRouter (and potentially other proxy providers) expose OpenAI's
text-embedding-3 models with a provider prefix in the model ID, e.g.
`openai/text-embedding-3-large` rather than bare `text-embedding-3-large`.
`dimsProviderOptions()` checks `modelId.startsWith('text-embedding-3')`
which fails for the prefixed form, so the `dimensions` parameter is never
sent. The upstream provider returns its native dimensionality (3072 for
-large) instead of the configured value (e.g. 1536), causing an immediate
"dim mismatch" error on first embed.
The default OpenRouter embedding (`text-embedding-3-small` at 1536d)
masked this because its native size happens to match the default config.
The bug surfaces when using `-large`, or `-small` with a non-1536 dim
(512, 768, 1024 — all listed in the recipe's `dims_options`).
Fix: strip the provider prefix before the `startsWith` check. The full
prefixed ID is preserved in the error message for user clarity.
* feat(subagent): claude-cli MessagesClient adapter (baseline, no tool use)
Closes#334 (partially — text-only baseline; tool use lands in the next
commit on this branch).
Adds a MessagesClient adapter that shells out to `claude --print
--output-format json --model <model>` instead of the Anthropic SDK. When
`GBRAIN_USE_CLAUDE_CLI=1` is set, the subagent worker registers the adapter
in place of the SDK client; the default path (Anthropic SDK with
ANTHROPIC_API_KEY) is unchanged when the env var is unset or set to
anything else.
The benefit is that Claude Max subscribers can run Minions subagents
against their existing OAuth subscription, no ANTHROPIC_API_KEY needed.
New: src/core/minions/handlers/claude-cli-adapter.ts
- Implements the MessagesClient interface exported from subagent.ts.
- Strips provider prefixes (`anthropic:`, `litellm:`) from the model id
because `claude --print` only accepts CLI-native aliases (`sonnet`,
`opus`, `haiku`, or the bare `claude-*-N-M` form).
- Flattens the Anthropic messages array into a single text prompt for
claude-cli stdin. Tool blocks (tool_use / tool_result) are stringified
as placeholders so multi-turn conversations stay coherent in this
baseline; native tool_use round-tripping is the follow-up commit.
- Spawns claude with stdio piped, captures stdout, parses the
`{type:"result", subtype:"success", result, usage, ...}` JSON envelope,
and returns it as a properly shaped Anthropic.Message with
`stop_reason: 'end_turn'`.
- Token totals propagate from the claude usage block so the subagent
handler's `ctx.updateTokens()` reports usable numbers.
- AbortSignal is wired through to SIGTERM the child so the subagent loop's
cancellation path stays correct.
Modified: src/commands/jobs.ts (worker registration)
- Conditionally constructs a MessagesClient via the new adapter when
GBRAIN_USE_CLAUDE_CLI=1.
- Passes it into makeSubagentHandler({ engine, client: subagentClient }).
- Logs `[minion worker] subagent routing via claude-cli (GBRAIN_USE_CLAUDE_CLI=1)`
on startup so the env var status is operator-visible.
Limitations of this commit (addressed in the follow-up):
- Tool use is not yet supported. Tools in params.tools are ignored; the
adapter returns a single text block with stop_reason='end_turn'.
- Token counts come from claude-cli's reporting and may not match the
Anthropic API's accounting precisely (especially for cache tiers).
Original design from #334; this commit preserves that author's attribution.
The follow-up commits on this branch carry the tool-use implementation.
* feat(subagent): tool use + context isolation + convention rename on top of jarvisdoes baseline
Builds on the previous commit (jarvisdoes's #334 baseline) by adding three
things the upstream issue called out as gaps or that surfaced during review:
1. Tool use support via system-prompt-instructed JSON emission.
2. Context isolation flags so claude-cli does not load operator-level
CLAUDE.md, skills, and local project context into every subagent call.
3. Env var rename from GBRAIN_USE_CLAUDE_CLI=1 to
GBRAIN_SUBAGENT_PROVIDER=claude-cli to match the existing
GBRAIN_<noun>_<role>=<value> convention used by GBRAIN_CHAT_MODEL,
GBRAIN_EMBEDDING_MODEL, GBRAIN_EXPANSION_MODEL.
## Tool use
The MessagesClient interface returns Anthropic.Message objects whose
content array may include tool_use blocks. The subagent handler filters
those blocks and dispatches each tool, so any backend that produces
correctly shaped tool_use blocks gets the same loop behavior as the
Anthropic SDK.
The adapter injects a system-prompt addendum describing the tool registry
plus an emission protocol:
<use_tools>
[{"id": "...", "name": "...", "input": {...}}, ...]
</use_tools>
After the response comes back, extractToolCalls() scans for the block,
parses the JSON (tolerant of optional ```json fencing), and converts each
entry into a tool_use content block. Multiple parallel tool calls in one
turn are supported via the array shape; this is the exact case that
breaks today on the codex-proxy / litellm GPT-5.x bridge where parallel
tool-call response IDs get dropped.
Defensive fallbacks:
- Malformed JSON inside the block: drop to text-only, stop_reason='end_turn'.
- Unterminated <use_tools> (no close tag): drop to text-only.
- Model omits id field: adapter synthesizes a toolu_claude_cli_<rand> id.
- Empty response: still hand the subagent loop a well-formed content
array so the .filter chain does not crash.
## Context isolation
claude-cli auto-discovers CLAUDE.md from cwd upward and injects the
operator's skills + plugins + auto-memory into the default system prompt.
On a real install that is ~42-65k tokens of contamination per subagent
call, with both cost and behavioral consequences (the subagent picks up
the operator's coding conventions, opinions, and preferences).
The maximum suppression that still preserves OAuth / Claude Max
subscription auth is:
- Spawn from a dedicated clean cwd (tmpdir-based) so LOCAL CLAUDE.md
auto-discovery has nothing to find. -13k tokens on a real gbrain
install where CLAUDE.md is substantial.
- --disable-slash-commands so skill resolution does not pull in
/skill-name handlers.
- --system-prompt <gbrain prompt> so the default system prompt is
replaced rather than appended to.
The --bare flag would also strip user-level ~/.claude/CLAUDE.md but it
forces ANTHROPIC_API_KEY auth, defeating the whole point of this adapter.
The remaining ~42k cached tokens from user-level instructions are
accepted as a cost-trivial trade-off because the Max subscription absorbs
the per-call cost. Behavioral contamination is mitigated by gbrain's
strong per-call system prompt overriding any operator-level drift.
## Env var rename
Surveyed all ~140 GBRAIN_* env vars in src/. The codebase uses three
patterns: GBRAIN_NO_<feature> (negative toggles), GBRAIN_<noun>_<role>
=<value> (routing keys), GBRAIN_ALLOW_<feature> (permissive toggles).
GBRAIN_USE_* does not appear anywhere except jarvisdoes's original
commit; it would introduce a fourth pattern.
GBRAIN_SUBAGENT_PROVIDER=claude-cli aligns with the routing-keys family
and is value-extensible — adding codex-cli / meridian-proxy / etc. later
means a new value, not a new env var. The scope ('SUBAGENT_*') is also
unambiguous about which calls the toggle covers; GBRAIN_USE_CLAUDE_CLI
was silent on whether it applied to all gbrain LLM calls or only the
subagent path.
Unknown values are rejected with a fail-fast error message naming the
two valid values rather than silently falling through to the default.
## Tests
New file: test/claude-cli-adapter.test.ts — 12 tests, 33 assertions:
- Text-only round trip (single text block, usage propagation, end_turn).
- Provider prefix stripping ('anthropic:claude-sonnet-4-6' -> 'claude-sonnet-4-6').
- Single tool_use parsing.
- Multiple parallel tool calls in one block (the case that triggered
the codex-proxy regression).
- Fenced JSON inside <use_tools> block.
- Model-omitted id gets synthesized to toolu_claude_cli_<rand>.
- Malformed JSON falls back to text.
- Unterminated block falls back to text.
- AbortSignal SIGTERMs the child.
- Error envelope rejected with informative message.
- Non-JSON output rejected with raw-output excerpt in the error.
- argv + cwd assertion: --disable-slash-commands + --system-prompt are
present and cwd is the dedicated tmpdir.
Tests use a POSIX shell stub at GBRAIN_CLAUDE_CLI_BIN that emits a
scripted --output-format json envelope, so the suite runs without
claude-cli installed and without API credits.
* feat(ai): claude-cli recipe with native gateway integration (supersedes #334 baseline)
Replaces the MessagesClient adapter + GBRAIN_USE_CLAUDE_CLI=1 env-var
gate from the previous commit on this branch with a proper gateway recipe.
The recipe path gives per-call routing as a native capability: a model
string like `claude-cli:claude-sonnet-4-6` lands here while a sibling
`litellm:gpt-5.4` continues through the litellm-proxy / codex-proxy path
in the same worker. No global env-var switch, no agent.use_gateway_loop
bypass, no MessagesClient injection at jobs.ts worker startup.
The previous commit on this branch (jarvisdoes baseline) is preserved
in the history for #334 authorship attribution. Its functional changes
are backed out here because the recipe pattern is gbrain's established
integration seam; introducing a parallel MessagesClient + env-var path
would have created two routing mechanisms competing for the same job.
New: src/core/ai/recipes/claude-cli.ts
- Recipe declaration: id 'claude-cli', tier 'native', implementation
'claude-cli', chat-only (no embedding or expansion touchpoints).
- Models: claude-opus-4-7, claude-sonnet-4-6, claude-haiku-4-5-20251001.
- supports_tools and supports_subagent_loop both true.
- supports_prompt_cache false because the CLI handles caching internally
and does not surface cache_control via the standard control plane.
- auth_env.required is the empty array because the CLI owns auth (OAuth
session managed by `claude login`).
- Friendly aliases mirror the `anthropic` recipe: `sonnet`, `haiku`,
`opus` and the same legacy-id rewrites for back-compat with stale
config strings.
New: src/core/ai/providers/claude-cli-language-model.ts
- ClaudeCliLanguageModel class implementing the ai-sdk LanguageModelV2
interface.
- doGenerate: renders the ai-sdk prompt array into a system text + user
text, injects the use_tools protocol instructions when tools are
present, spawns `claude --print --output-format json --model <X>
--disable-slash-commands --system-prompt <gbrain prompt>` from a
dedicated tmpdir (contamination suppression: no local CLAUDE.md
auto-discovery), parses the JSON envelope, extracts <use_tools>
blocks, and returns ai-sdk-shaped LanguageModelV2Content (text +
tool-call parts with stringified-JSON input matching the V2 contract).
- Tolerates fenced JSON inside use_tools blocks, malformed JSON
(falls back to text), missing close tag (falls back to text),
model-omitted ids (synthesizes toolu_claude_cli_<rand>).
- Parallel tool calls in one block round-trip cleanly: this is the
case that drops IDs on the litellm + codex-proxy bridge today.
- AbortSignal SIGTERMs the child for proper cancellation.
- doStream throws not-supported (gateway.toolLoop is non-streaming).
Modified: src/core/ai/gateway.ts
- Adds case 'claude-cli' to instantiateChat (returns ClaudeCliLanguageModel).
- Adds case 'claude-cli' to instantiateExpansion (same wrapper, reserved
for a future expansion touchpoint declaration).
- Adds case 'claude-cli' to instantiateEmbedding (throws, no embedding
model, mirrors the native-anthropic path).
- Lazy require() at the call site keeps the gateway module load cheap
for users who never use the claude-cli path.
Modified: src/core/ai/recipes/index.ts
- Registers `claudeCli` in the ALL[] array next to `anthropic`.
Modified: src/core/ai/types.ts
- Adds 'claude-cli' to the Implementation union so the gateway switch
is exhaustive at compile time.
Reverted: src/commands/jobs.ts
- Drops the GBRAIN_USE_CLAUDE_CLI=1 env-var gate the prior commit
added. Routing now happens at the gateway based on the model string.
Deleted: src/core/minions/handlers/claude-cli-adapter.ts
- The MessagesClient adapter is superseded by the recipe + LanguageModelV2
path. Two routing mechanisms competing for the same job would have
forced users to reason about which one wins; the recipe is the single
source of truth.
New file: test/claude-cli-recipe.test.ts (16 tests, 46 assertions):
- Recipe registration: getRecipe returns chat-only Recipe; aliases map
short names (sonnet/haiku/opus) to canonical model ids.
- Text round trip: single text content block, usage propagation, stop
finish reason.
- Provider prefix stripping.
- Single tool-call parsing.
- Multiple parallel tool calls in one block.
- Fenced JSON inside the block.
- Model-omitted id synthesizes toolu_claude_cli_<rand>.
- Malformed JSON falls back to text + stop reason.
- Unterminated block falls back to text + stop reason.
- Tools offered but model declines: returns text-only with stop reason
so the gateway-loop treats it as a final answer rather than wedging
for tool calls that never come.
- AbortSignal SIGTERMs the child.
- is_error envelope rejected.
- Non-JSON output rejected.
- doStream throws.
- argv + cwd assertion: --print, --disable-slash-commands,
--system-prompt are present and cwd is the dedicated tmpdir.
Tests use a POSIX shell stub at GBRAIN_CLAUDE_CLI_BIN so the suite runs
without claude-cli installed and without API credits.
End-to-end smoke verified against a real `claude --print --model haiku`
invocation: model emitted `<use_tools>` block with toolu_add_001 +
{"a":12,"b":30}, adapter parsed back into a `tool-call` content block,
finishReason 'tool-calls'.
* feat(ai/claude-cli): harden subagent isolation, env scrub, verbose + stdin robustness
Four defensive fixes to the claude-cli provider so a subagent call behaves
identically regardless of the host's ambient Claude Code config:
- Agent isolation: pass `--tools ''` and `--strict-mcp-config` so the subprocess
runs as a raw LLM with no built-in tools and no inherited user MCP servers.
Without `--strict-mcp-config`, each call boots the user's MCP servers (including
gbrain's own), causing recursion plus PGLite single-writer lock contention.
- Env scrub: drop ANTHROPIC_API_KEY / ANTHROPIC_AUTH_TOKEN / ANTHROPIC_BASE_URL
from the child env so the CLI authenticates via its own OAuth subscription
session. An inherited API key silently flips billing to per-token API usage,
the exact setup this recipe exists to replace.
- Verbose-mode compat: with `"verbose": true` in ~/.claude/settings.json,
`--print --output-format json` emits an event array instead of a bare result
object. Tolerate both shapes and select the result event.
- stdin robustness: handle the child stdin 'error' event and wrap write/end so a
missing binary (ENOENT) or early child death (EPIPE) rejects cleanly instead of
crashing the worker with an unhandled error.
Adds unit coverage for the env scrub, the isolation argv, and the verbose event
array. Verified against claude CLI 2.1.x.
* test(ai/claude-cli): cover verbose-array no-result + missing-binary reject paths
Two error branches in the hardened claude-cli provider had no coverage: the
verbose event-array path when no result event is present, and a missing binary
surfacing as a clean spawn-failed rejection. The missing-binary case is the
deterministic form of the stdin/EPIPE robustness; a synchronous stdin-write
throw is not reliably triggerable in a unit test, so the real ENOENT path the
handlers defend is exercised instead. Both reuse the existing shell-stub harness.
---------
Co-authored-by: jarvisdoes <258486803+jarvisdoes@users.noreply.github.com>
Co-authored-by: Marco Maldonado <34176133+loweaxerium@users.noreply.github.com>
queue.add() with an idempotency_key returns any existing row regardless
of status. This means dead jobs (exhausted retries from a transient
provider outage) permanently block re-submission of the same work —
even after the underlying issue is fixed.
Fix: when the existing row is dead or cancelled, NULL its
idempotency_key (preserving the row for audit) and fall through to the
INSERT path so a fresh job can be created.
Affects dream synthesize children that died during provider migrations
(429 rate-limit on old Anthropic proxy, tool-results-missing on old
OpenRouter). 45 dead children were blocking re-synthesis of transcripts
in production.
Includes 4 new tests covering dead, cancelled, completed, and active
status interactions with idempotency dedup.
Co-authored-by: Rafael Reis <rafael.reis@contabilizei.com.br>
OpenRouter's POST /api/v1/rerank is wire-compatible with gateway.rerank()
({query, documents, model} → {results: [{index, relevance_score}]}). This
adds a recipe-only reranker touchpoint declaring four models:
- cohere/rerank-v3.5 (default; $0.001/search)
- cohere/rerank-4-fast ($0.002/search, 32K context)
- cohere/rerank-4-pro ($0.0025/search, SOTA quality)
- nvidia/llama-nemotron-rerank-vl-1b-v2:free (multimodal)
Unlike embedding/chat, the reranker path strictly enforces the models
allowlist — the openai-compat extended-model bypass does not apply. New
rerank models must be added to this recipe before they can be called.
The cost_per_1m_tokens_usd value is a pseudo-rate for the budget tracker's
chars/4 heuristic — Cohere bills per-search, not per-token. At ~4K chars
the estimated cost is in the right ballpark.
Recipe-only change; no gateway or search-layer modifications. gateway
auto-concatenates path → .../api/v1/rerank.
Adds hermetic unit test (test/openrouter-reranker-recipe.test.ts) covering
shape, models, default_model, path, max_payload_bytes, default_timeout_ms,
and cost field. No DB, no env mutation — survives the parallel 8-shard
fan-out.
Verified: bun run verify (30/30 green); 285 targeted recipe+rerank+budget
tests pass.
Co-authored-by: Hippityy <Hippityy@users.noreply.github.com>
`gbrain doctor --remediation-plan` printed two consecutive lines that
contradicted each other when the brain was below target AND the target
was unreachable with autonomous remediation:
Brain score: 45/100 → target 90
Target unreachable: max with autonomous remediation is 70/100.
No remediations needed. Brain is at target.
The second sentence hid the real next step (configure the prereqs that
would lift `max_reachable_score`) and made the brain look healthy when it
was not.
Fix: gate the "Brain is at target" line on `brain_score_current >=
targetScore`. When the plan is empty AND the brain is below target, the
"Target unreachable" line above is already the user-facing explanation;
the `Blocked checks` block below surfaces the manual gap.
Extracted `renderRemediationPlanLines(plan, targetScore): string[]` as a
pure helper alongside `runRemediationPlan` so the regression coverage
asserts on the rendered output directly rather than mocking
`console.log`. `runRemediationPlan` now joins the lines verbatim through
console.log; behavior is byte-identical for every case other than the
fixed contradiction.
Five regression tests cover: unreachable-and-below-target (the bug
case), reachable-and-at-target, exact-target, below-target-with-plan,
unreachable-with-partial-plan. 38 tests across the adjacent doctor test
files stay green; `bun run typecheck` clean.
Idempotency was keyed on atom rows alone — a page the LLM judges
un-atomizable leaves no row, so it re-entered the discovery window every
run. Two production consequences: --drain false-stopped with
no_progress once the window head was mostly zero-yield pages (remaining
frozen while batches report +0), and every nightly re-spent extraction
budget on the same pages.
Fix:
- After a SUCCESSFUL chat call that parses to zero atoms, stamp the
source page with frontmatter.atoms_scan_hash = contentHash16. LLM
failures take the catch path and stay retryable.
- discoverExtractablePages + countExtractAtomsBacklog (both variants)
exclude pages whose stamp matches the CURRENT content hash prefix —
content edits re-eligibilize, mirroring atom-row staleness semantics.
- Drain no_progress now recounts the backlog on a zero-atom batch and
only stops when it genuinely didn't shrink — tombstoning IS progress.
Tests: +2 pure-loop drain cases (shrinking backlog continues / flat
backlog stops) and +3 PGLite integration cases (stamp + exclusion /
content-change re-eligibility / failed chat does not stamp).
29 pass / 0 fail across the two files; tsc clean.
Co-authored-by: 陈源泉 <chenyuanquan@chenyuanquandeMac-mini.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
synthesize-concepts.ts's design comment says extract_atoms stamps a
`concepts:` frontmatter field on each atom and :92 consumes ONLY that
field — but the extractor never wrote it, so the atoms → concepts
pipeline was dead end-to-end: every cycle reported "synthesize_concepts:
skipped — no atoms with concept refs" no matter how many atoms
accumulated (696 page-derived atoms / 0 with concepts on our production
brain before an external backfill).
Fix, all on the extractor side (no synthesize change needed):
- EXTRACT_PROMPT asks for `concepts` (1-3 kebab-case TOPIC labels) with
an explicit reuse-over-coinage instruction — labels must cluster,
since synthesize_concepts only materializes groups of >=2.
- parseAtomsResponse validates labels (kebab regex, max 3, drop
invalid; empty -> undefined).
- The putPage frontmatter write stamps `concepts` alongside lesson /
source_quote.
Tests: 4 parse cases + an end-to-end regression that goes extractor ->
real frontmatter -> synthesize_concepts' OWN DB query path -> concept
page. The existing tests fed synthesize via the `_atoms` seam, which is
exactly how this gap survived.
Validated in production ahead of this PR by stamping the same shape
externally: the next synthesize_concepts run wrote 33 concept pages
(T2=7/T3=26) from 60 stamped atoms, zero failures.
Co-authored-by: 陈源泉 <chenyuanquan@chenyuanquandeMac-mini.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The wrapper script that 'gbrain autopilot --install' writes to
~/.gbrain/autopilot-run.sh sources ~/.bashrc to inherit PATH for the
exec'd gbrain binary (which has a '#!/usr/bin/env bun' shebang). The
standard Debian/Ubuntu ~/.bashrc ships a non-interactive guard that
returns early when bash is launched non-interactively (cron, launchd,
systemd) — so PATH exports operators add to ~/.bashrc never reach the
wrapper subprocess.
The result: the wrapper dies silently with 'env: bun: No such file or
directory', leaves a stale lockfile, and every subsequent cron tick
hits the lockfile and bails. The nightly dream cycle hangs waiting on
a worker that never comes back, and the wrapper's own 10-min
stale-lock window is the only thing that can recover it.
This bites every operator whose bashrc is the standard distro default
(which is the default), and there is no warning at install time.
Fix: prepend ~/.bun/bin to PATH directly in the wrapper, so it is
self-contained regardless of which init file the OS loaded. Add a
regression test alongside the existing zshenv/zshrc source-order test
(v0.36.1.x #966) so this class of bug stays caught.
The conversation_format_coverage check recommended `gbrain config set
conversation_parser.llm_fallback_enabled true`, but that config key is dead
(never read) — see #1890. The recommendation is a no-op and misleads users into
thinking a fallback will kick in. Drop it; keep the actionable `scan` hint.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Backlinks Minion jobs submitted with an empty payload (the
sync→embed→backlinks chains enqueued after every ingestion) defaulted to
action='fix', rewriting tracked brain pages with generated "Referenced in"
timeline bullets on every routine run — 129 vault files polluted in one
day on our production brain before we traced it.
This contradicts the documented intent in src/core/cycle.ts
(runPhaseBacklinks): "Maintenance cycles must not rewrite tracked brain
pages with generated 'Referenced in' timeline bullets. [...] the legacy
filesystem fixer remains available explicitly via `gbrain check-backlinks
fix`." — the jobs-worker handler simply inverted that default.
Fix: default to 'check'; 'fix' requires explicit opt-in via
'{"action":"fix"}' (the documented submit shape) or
`gbrain check-backlinks fix`. Both explicit paths are unchanged.
Adds a structural regression test (fix-wave-structural.test.ts precedent)
pinning the default, since the handler dynamically imports
runBacklinksCore and walks a real repo dir — a behavioral test would
require mocking that hides the regression behind a test seam.
Co-authored-by: Valentin Ferriere <valentin@v-labs.fr>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`extractStaleFromDB` stamped `links_extracted_at` with each page's read
`updated_at` (the D4 race-fix). But the stale predicate also flags
`links_extracted_at < LINK_EXTRACTOR_VERSION_TS`. Any page last edited
BEFORE the version timestamp got stamped below the threshold, so the
version arm re-flagged it stale on every run — an infinite re-extract
loop that never cleared the lag.
Since the v112 watermark column ships with no backfill, every
pre-existing page starts stale, and most pre-date the version bump. In
practice this left ~97% of pages permanently stale: `extract --stale`
reported "done" each run but `links_extraction_lag` never dropped.
Fix: stamp `GREATEST(read updated_at, versionTs)`. Old pages lift to the
threshold so the version arm clears; a real future edit still advances
`updated_at` past the stamp, so the CDX-1 edited-after-stamp race
protection is preserved.
Adds a regression test: a page with `updated_at` before
LINK_EXTRACTOR_VERSION_TS must clear after extract AND stay clear on a
second run (the existing tests only used now()-dated pages, so the
old-page case was uncovered).
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(drift): skip node_modules/dist/build in multi-source drift walk
The drift walker recursed into node_modules (50k+ files in RN/Astro repos),
exhausting the time budget before completing, so multi_source_drift always
reported 'walk hit limit/timeout' on real projects. Skip heavy non-content
dirs + add a deadline check on directory descent.
* fix(integrity): skip inline-code spans + [Source:] citations in bare-tweet detection
Recipe/doc pages that show the CORRECT citation format inline (e.g.
`Tweeted about {topic} [Source: X, @handle, date]`) were false-flagged.
The fenced-code skip didn't cover inline backticks; add inline-code
stripping + an explicit-citation exemption.
---------
Co-authored-by: Son Le <tuanson1200@gmail.com>
* test(e2e): drop flaky wall-clock bounds in minions-resilience
The runaway-dead-letter and cascade-kill tests asserted tight real-clock
upper bounds (<2000ms, <3000ms) on top of already-complete terminal-state
checks. Those bounds carry no correctness signal — the dead/cancelled status
and abortedChildren==10 assertions fully prove behavior — and flake on loaded
CI runners where the stall/timeout sweep cadence varies. Removed both bounds,
kept the diagnostic values, de-promised the test titles.
* test(e2e): kill order-dependence + no-op assertions in mechanical
- traverse_graph: self-contained (re-adds its own idempotent link) and asserts
the linked company is reachable, instead of depending on a prior it() and
only checking array shape.
- file_list-without-slug: seeds its own >100 rows instead of relying on the
previous test's 150 surviving in the DB; asserts the cap is exercised.
- precision@5: add a loose floor (every known-item query surfaces >=1 truth doc
in top-5) so a 0% retrieval regression no longer passes silently.
- get_health: assert value bounds (page_count==16, embed_coverage 0..1) not just
typeof; get_chunks: assert non-empty text, numeric non-decreasing chunk_index,
and that the page name appears, instead of toBeTruthy on chunks[0].
* test(e2e): strengthen graph/search quality assertions + close coverage gaps
graph-quality: truncate+reseed 'config' in truncateAll (kills config leak where a
setConfig test throwing before its finally bleeds into later tests); replace
toBeGreaterThan(0) link/timeline floors with fixture-derived minimums; assert exact
attendee slugs are 'attended' instead of a vacuous .every; pin autoLinks.created to
the provable 2 (Alice+Acme); add direction out/both + depth:2 multi-hop and a
cycle-safety (A->B->A terminates) test.
search-quality: fix the vacuous detail=low vector test; assert pedro returns >=2
chunks; assert detail=high includes the timeline chunk; add empty-query and
zero-vector no-throw edge tests.
* test(e2e): self-contain multi-source sync test + assert ledger cascade
Break the sequential dependency where 'performSync no sourceId' relied on a prior
test writing sync.repo_path — it now sets its own config. Add the missing
file_migration_ledger COUNT(*)==0 cascade assertion. Tighten the source_id default
check from toContain('default') to exact "'default'::text".
* test(e2e): make migration-flow HOME/PATH swap throw-safe
The suite repoints process.env.HOME/PATH to a temp dir and only restored them in
afterAll, so a mid-test throw left HOME dead for the rest of the bun process and
silently broke sibling suites. Wrap each test body in try/finally restore + a
defensive restore at the top of beforeEach.
* test(e2e): loud-skip jsonb-roundtrip + doctor-progress
Both skipped silently with no DATABASE_URL, giving zero signal the regression guard
never ran. Add the console.log skip line matching the sibling e2e files.
* test(e2e): robust check-update contract + find_orphans tool coverage
upgrade: the 'no-releases' test hard-asserted update_available===false, which flips
to failing the moment the repo has a real release. Assert the JSON contract shape
(boolean update_available, current_version===VERSION, typed optional fields) instead.
mcp: add find_orphans to the asserted generated tool names.
splitLargeNode can only break up a node that exposes a `body` with >= 2
named children. A node without one -- a giant object/array literal, a single
huge assignment, a massive template literal -- is emitted whole. On real
source that yields a chunk far larger than the embedder's context window; the
embedder then rejects it ("input exceeds context length") and it is never
embedded. Example: a 372 KB service file produced 113 chunks, one a single
281 KB (~70k-token) node -> permanently unembedded.
Add a final safety-net pass (capOversizedChunks) that recursively re-splits
any chunk over a token budget (default 2000, configurable via maxChunkTokens),
with a hard character split as a last resort for no-whitespace content
(minified one-liners). Normal files are untouched.
Verified: that 372 KB file now yields 188 chunks, max ~1.5k tokens, zero
oversized; a small file is unchanged.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
gbrain think printed "## Gaps" twice: the synthesis prompt asked the model for a Gaps section inside the answer body AND a separate structured gaps array, then both render paths printed both — the CLI human output (src/commands/think.ts) and the --save page (persistSynthesis in src/core/think/index.ts).
Make the structured gaps array the single source. The prompt now routes gaps into the array, not an answer-body section. New exported stripGapsSection(answer) defensively removes any "## Gaps" section a model still emits (any heading level, case-insensitive, bounded by the next same/higher heading); both render sites call it, so the dedup is structural rather than dependent on the model obeying the prompt.
Adds test/think-gaps.test.ts (hermetic): strip helper across heading levels / case / no-section / mid-document / false-match, the one-render-only repro, and the prompt contract.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
PGLite can return zero rows from INSERT ... ON CONFLICT DO UPDATE ...
RETURNING in no-op/trigger edge cases. The previous code called
rowToPage(rows[0]) unconditionally, so rows[0] was undefined and
rowToPage threw "undefined is not an object (evaluating 'row.deleted_at')",
which aborted the import and silently skipped the file during sync.
getPage() already has the empty-rows guard; putPage() was missing the
parallel one. The row was in fact written by the upsert, so re-read it
via getPage() instead of crashing. On a real monorepo index this
recovered ~19% of files (985/5148) that were failing to embed.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
DEF_TYPES listed only canonical symbol-type names (function, class, interface, ...). But normalizeSymbolType in the code chunker canonicalizes only some tree-sitter node types and lets the rest fall through type.replace(/_/g, ' '). So method_declaration is stored as 'method declaration', struct_specifier as 'struct specifier', protocol_declaration as 'protocol declaration'. None were in DEF_TYPES, so code-def returned 0 hits for every method, constructor, field, C struct, and Swift protocol. The plain 'struct' entry never matched either. Add the fallthrough definition forms. Read-path only; no reindex needed (0 -> N on existing indexes).
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
The code-fence-wrap detector in lintContent used the /m multiline flag, so
^/$ matched start/end of any line. The rule fired on any page that simply
contained a ```markdown code block, not only pages wrapped end-to-end.
The matching fixer in fixContent has no /m flag, so it can only strip
whole-file wrappers. Result: detected issues were marked fixable: true,
yet fixContent could never strip them. `gbrain dream` reported
"0 fix(es) applied, N remaining" perpetually for the rule.
Drops the /m flag from the detector so detector and fixer stay in sync.
Whole-file wrapper detection is preserved; inner code blocks no longer
trigger the rule.
Real-world impact: a brain with 5 docs pages containing markdown examples
(skill READMEs, decision registry, journal templates) reports 5 phantom
"fixable: true" issues every dream cycle, never converging. After this
fix, the dream-cycle lint phase reports only real-and-unfixable issues
(missing frontmatter, missing title/type) which is the intended behavior.
Two regression tests added in test/lint.test.ts:
- Page contains a single inner ```markdown block
- Page contains multiple inner ```markdown blocks
Both assert no code-fence-wrap issue is reported. The existing
"detects wrapping code fences" test (true-positive case) continues to
pass; total tests in the file are 18 -> 20.
Co-authored-by: Thomas Chung <thomaschung@macbookair.lan>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
`findByTitleFuzzy` on both `postgres-engine.ts` and `pglite-engine.ts`
has no `source_id` filter and no `deleted_at IS NULL` filter. `tryFuzzyMatch`
in `src/core/entities/resolve.ts` got both of those filters via #1436
(v0.41.13.0) for exactly the reasons that apply to its sibling here:
fuzzy resolution can suggest cross-source slug candidates that the
caller then silently drops at the FK filter (or worse, picks a
soft-deleted page).
This is the missing twin of #1436. In multi-source brains, the live-mode
auto-link resolver invoked from `put_page` (`operations.ts:937`) calls
`engine.findByTitleFuzzy` with no scope. When two sources contain pages
with similar titles (`people/alice-example` on `source-a`,
`people/alice-other` on `source-b`), the fuzzy lookup can return the
wrong-source slug, which then fails the downstream `allSlugs` /
`addLink` FK filter — the link silently doesn't get created, and from
the caller's view the resolver "failed" even though the page existed
under the right source.
Reproducible with a 2-source PGLite setup + identical-title pages on
both sides; the fuzzy call returns a slug whose `source_id` doesn't
match the put_page caller's source.
- 2-source brains: auto-links between same-title-different-source pages
now resolve under the caller's source instead of the wrong neighbor.
- Soft-deleted pages can no longer be returned as fuzzy candidates
(mirroring the resolve.ts fix from #1436).
- 1-source brains: no behavior change. `sourceId` is optional; when
omitted the SQL takes the pre-existing unscoped path.
- `engine.ts`: add optional 4th `sourceId` param to the
`findByTitleFuzzy` interface + JSDoc explaining the scope semantics.
- `postgres-engine.ts` / `pglite-engine.ts`: implement the param via a
conditional SQL branch that adds `AND source_id = $N AND
deleted_at IS NULL` when `sourceId` is set; existing query path
unchanged when omitted.
- `link-extraction.ts`: add optional `sourceId` to `makeResolver` opts,
forward to `findByTitleFuzzy` in step 3 of the resolve chain.
- `operations.ts`: pass `opts?.sourceId` to `makeResolver` from the
live-mode put_page resolver (the place that already knows the
caller's source).
- New unit tests in `test/link-extraction.test.ts` (2 cases):
- `opts.sourceId` is forwarded to `findByTitleFuzzy` when set.
- `opts.sourceId` omitted → `findByTitleFuzzy` receives `undefined`
(back-compat).
- `bun run typecheck` clean.
- `bun test test/link-extraction.test.ts test/entity-resolve.test.ts
test/operations.test.ts test/extract.test.ts` — 145/145 pass.
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
* fix(models): dispatch subcommand reads args[0] not args[1]
`gbrain models doctor` silently fell through to the read view
instead of running the reachability probe.
`runModels` checks `args[1] === 'doctor'`, but the caller —
`handleCliOnly(command, subArgs)` in `src/cli.ts:113` — passes
`subArgs` (the leading command token already stripped). So inside
`runModels`, args[0] is the subcommand. args[1] is undefined.
The doctor probe path has been unreachable from the CLI since the
handleCliOnly refactor. `gbrain models help` happened to work by
falling through to the `--help` flag detection.
Two-char fix: `args[1]` → `args[0]` on both branches of the
ternary.
Verified by manual probe — `gbrain models doctor` now prints
"Model reachability probe:" with per-model results (real production
brain, 4 touchpoints probed):
```
Model reachability probe:
embedding_config ollama:bge-m3 ok (0ms)
reranker_config (none) ok (0ms)
chat lmstudio:mistralai/magistral-small-2509 unknown (5012ms)
[chat(lmstudio:mistralai/magistral-small-2509)] probe timed out after 5s
expansion lmstudio:google/gemma-4-e2b ok (535ms)
Summary: 3/4 reachable.
```
RECOVERY REBUILD 2026-05-26 of original 20ed0eee.
* fix: honor --help before doctor dispatch to avoid running probes on `models doctor --help`
Codex review of #1428 flagged that the args[1]→args[0] rewrite
regressed `gbrain models doctor --help` into running network
probes instead of printing usage. The original args[1]-shaped
ternary happened to dodge this by always falling through to the
args.includes('--help') branch when args[1] === 'doctor' was
false; the new args[0] code checks doctor first, so --help no
longer wins.
Reorder ternary: `hasHelp` is computed FIRST from
(--help / -h / args[0] === 'help'), then the sub is hasHelp ?
'help' : args[0] === 'doctor' ? 'doctor' : 'read'.
Addresses codex review P2 on PR #1428.
---------
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
The HTTP MCP server's 401 responses missed the `resource_metadata`
parameter in the WWW-Authenticate header. MCP authorization spec
(2025-06-18 draft §5.1) and RFC 9728 require:
WWW-Authenticate: Bearer resource_metadata="<url>"
MCP-aware OAuth clients (claude.ai, Cursor, etc.) use that URL to find
the authorization-server discovery doc without the user manually
configuring the issuer. Pre-fix the header shipped only `Bearer
error="invalid_token", error_description="..."` and MCP clients silently
failed to begin the OAuth flow — symptom on claude.ai's UI was "Couldn't
reach the MCP server" even when discovery + /token + /register all
responded 200 individually.
The `requireBearerAuth` middleware in @modelcontextprotocol/sdk's
BearerAuthMiddlewareOptions already supports a `resourceMetadataUrl`
parameter. Two call sites (`/mcp` and `/ingest`) now pass it.
Verified against a real claude.ai connector attempt: pre-fix the
connector showed "Couldn't reach the MCP server" with no OAuth redirect.
Post-fix the connector successfully begins the authorization flow.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
llama.cpp's llama-server rejects /v1/embeddings requests with more inputs
than its launch --batch-size (default 32): "batch size 100 > maximum allowed
batch size 32". gbrain sends batches of 100, so any page with >32 chunks fails
to embed, and embed --stale then trips the Postgres statement_timeout retrying
the doomed batches. The existing token-based protection (max_batch_tokens)
can't bound item count — N tiny chunks fit under any token budget.
Add an optional max_batch_items count cap to EmbeddingTouchpoint, enforced as a
hard re-split after the token split in embed(), and set it to 32 on the
llama-server recipe (replacing no_batch_cap: true, which wrongly assumed
llama.cpp has no per-request item cap). A declared item cap also suppresses the
missing-max_batch_tokens startup warning.
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
The cron daily briefing writes 90_Briefings/<date>.md, which gets
re-ingested on the next sync and then dominates tomorrow's
getRecentSalience output as pure self-reference (observed: top
result score 0.9956, everyone else clustered at 0.587).
Filter `p.slug LIKE 'briefings/%'` out of getRecentSalience in both
the PG and PGLite engines. Suppressed by default; callers can still
opt in by passing `slugPrefix: 'briefings/'` (or `--kind briefings/`
from the CLI). search and list_pages are unaffected.
Co-authored-by: CTO <cto@timelycare.local>
Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
On Windows, `core.autocrlf=true` is the default and SKILL.md files are
checked out with CRLF line endings. `extractTriggers` used regexes
anchored to `\n` (`/^---\n.../` and `/^triggers:\s*\n.../`), which
never matched `\r\n`, so the parser returned `[]` for every skill.
Result: `gbrain doctor --fast --json` on Windows reported every skill
not in `OVERLAP_WHITELIST` (39 of 42) as a false `mece_gap` warning —
even though `skill_conformance` in the same run reported "42/42 skills
pass". CI runs Ubuntu-only so the divergence never surfaced.
Fix: normalize CRLF → LF at the top of `extractTriggers`. Single-line
change preserves existing LF behavior. Function is now exported so the
test can target it directly.
Tests: added `describe("extractTriggers")` block covering LF input,
CRLF input (regression case), missing frontmatter, missing triggers
field, and quote-stripping. All 30 tests in `check-resolvable.test.ts`
pass.
Verified locally on Windows: `gbrain doctor --fast --json` now reports
`resolver_health: ok, 42 skills, all reachable` (health_score 90 → 95).
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
The signal-handler test asserted an absolute liveReporters===0 on a
module-global set, so any other test file in the shard holding a live
reporter flaked it — it red-flagged ~12 unrelated PR runs and one master
push in two days, purely as a function of shard composition. The delta
form pins the same claim (50 reporter lifecycles leak nothing).
The 15-minute shard timeout cancelled 13 fully-passing runs under
parallel PR load (PGLite WASM cold-starts stretch shards); the
test-status gate then reported the cancellations as failures.
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Qwen3-Embedding family on Ollama supports Matryoshka truncation via the
'dimensions' field on /v1/embeddings. Without this passthrough, gbrain
ignores user-selected reduced dims and the provider returns its native
size, causing dim-mismatch errors against brains configured for narrower
widths (e.g. existing 1536-dim brains).
Matches by bare name 'qwen3-embedding' or any tag variant
'qwen3-embedding:0.6b' / ':4b' / ':8b'.
Native dims: 0.6B=1024, 4B=2560, 8B=4096. All MRL-truncatable.
5 new tests; full AI suite 137/137 green.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
Add the OpenClaw-required top-level id to openclaw.plugin.json, export a
direct register(api) entrypoint from src/openclaw-context-engine.ts, add a
manifest regression test, and document that skillpack harvest must preserve
OpenClaw-native manifest fields (id, configSchema, contracts).
llms bundles regenerated (bun run build:llms) — no content drift.
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Filip <FilipHarald@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Takeover of #2719 (fork head; rebased onto origin/master).
- writeProposed now writes both best.md (current-best pointer) and
proposed.md (stable human-review artifact); returns the proposal path.
- Orchestrator reports the real proposed.md path for accepted --no-mutate runs.
- Tutorial updated; llms bundles regenerated (no content drift — tutorial
is not inlined in the bundle).
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Ziyang Guo <121015044+RerankerGuo@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Point the OpenClaw and Hermes anchors in the "Have your agent install
it" section at their real upstream repos; the previous openclawagents
org URLs 404. Regenerated llms-full.txt to match.
Takeover of #1961 (fork branch) rebased onto current master.
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: jessems <jessems@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Ports #3015 by @jdewoski-cmd onto current master:
- src/core/orphan-policy.ts centralizes the orphan-reporting exclusion
convention so `gbrain orphans`, doctor's orphan_ratio, and both engines'
getHealth orphan_pages can no longer drift.
- getHealth stale_pages now uses the link-extractor stale watermark
(countStalePagesForExtraction) so health agrees with what `gbrain extract
--stale` will actually process.
- New `gbrain maintain` command: dry-run by default, `--safe` applies only
the conservative runbook actions (DB-backed stale extraction + source-scoped
dream cycles for doctor cycle_freshness findings), `--json` for structured
before/action/after reports. Frontmatter mutations, schema-pack upgrades,
and semantic hub links stay review-only by design.
Changed from the original PR: the shared defaults carried slugs specific to
the contributor's own brain ('josa-secrets/', '*-ga4-property-id.md',
'*-josa-test', literal 'welcome'/'untitled' fixtures). Global defaults now
carry only GBrain-wide conventions; brain-specific exclusions move to a new
per-brain config plane the policy reads through loadOrphanPolicyOverrides:
gbrain config set orphans.exclude_prefixes "my-private-folder/,archive/"
gbrain config set orphans.exclude_slugs "some-one-off-page"
Both engines' getHealth and the orphans command thread the overrides;
tests cover the neutral defaults, the override plane, and health parity.
Also registered `maintain` in CLI_ONLY_SELF_HELP so `gbrain maintain --help`
reaches the command's own usage block.
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: jdewoski-cmd <jdewoski@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The generic wikilink pass (issue #972) forwarded the raw literal to
resolveBasenameMatches, whose index is keyed by final path segments —
so [[notes/struktura]] (any dir outside DIR_PATTERN) silently produced
zero edges from `extract links --source db` and put_page auto-link,
while the FS extractor resolves the identical content (resolveSlugAll
strips the dirname before its basename lookup).
Query by the literal's final segment, then keep only matches whose slug
ends with the written path — [[notes/struktura]] can resolve to
vault/notes/struktura but never attach to wiki/struktura. Bare literals
are untouched. Flag-gated by link_resolution.global_basename as before.
Co-authored-by: YMYD <53603073+OJ-OnJourney@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
Closes#769. Every re-embed pass clobbered code-chunk metadata
(language, symbol_name, symbol_type, start_line, end_line,
parent_symbol_path, doc_comment, symbol_name_qualified) to NULL,
disabling code-def queries across thousands of indexed chunks.
Two complementary fixes:
embed.ts — three re-upsert call sites (embedPage, embedAll
non-stale, embedAllStale autopilot path) build ChunkInputs from
loaded chunks; they were stripping the 8 metadata fields. New
preserveCodeMetadata helper threads those fields through
consistently. Integrated cleanly with v0.34.4.0's cursor-paginated
--stale hardening — the wrap sits inside the worker function
between embedBatchWithBackoff and engine.upsertChunks.
postgres-engine.ts + pglite-engine.ts — upsertChunks ON CONFLICT
clause OVERWROTE metadata columns from EXCLUDED. Asymmetric vs the
embedding/embedded_at columns which already used a chunk_text-gated
CASE pattern (re-chunk → trust EXCLUDED, re-embed → COALESCE
preserve). Applied the same pattern to all 8 metadata columns.
Three regression tests in test/embed.serial.test.ts cover --stale
(autopilot), --all, and --slugs paths. Each loads a chunk with
full metadata, runs runEmbed, and asserts engine.upsertChunks
receives the metadata round-tripped. Coexists with master's D5
embedBatchWithBackoff test block.
Backfill required after deploy: \`gbrain sync --strategy code
--force --source <id>\` per code source to re-populate metadata via
the chunker. Without backfill, existing NULL columns stay NULL —
re-embed alone never produces metadata, only the chunker does.
Originally landed as part of PR #768 (the wave that bundled #767 +
fix; this PR carries the #769 fix alone with no scope overlap.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
* feat(synopsis): tail-truncate documentText for small-model chat handlers
Small local chat models (Gemma 4 E2B, Qwen3 4B) get dramatically
slower on long contexts even at 131K declared windows. A 73K-char
page synopsis on Gemma 4 E2B takes 60-120s, exceeding the worker's
default 30s `lockDuration` and tripping `lock-lost` errors.
Add `SYNOPSIS_DOC_MAX_CHARS` env-overridable cap (default 32768
chars, ~8K tokens) applied in `buildUserPrompt`. Truncate the TAIL
so the head (title, frontmatter, intro) preserves the document-level
anchor the synopsis needs.
Anthropic Haiku is unaffected at this cap; bump via
`GBRAIN_SYNOPSIS_DOC_MAX_CHARS` for frontier models that want
richer document anchoring.
Belt-and-suspenders companion to commit 0aaff691 (--lock-duration
flag on the worker). Combined: bumping lock TTL gives the handler
more time, AND truncating doc cap makes the handler complete faster.
Either alone helps; both together get the synopsis backfill running
reliably on small local LLMs.
Verified: real 1383-chunk personal brain backfill at
GBRAIN_SYNOPSIS_MODEL=lmstudio:google/gemma-4-e2b +
GBRAIN_SYNOPSIS_DOC_MAX_CHARS=16384 +
`gbrain jobs work --concurrency 4 --lock-duration 300000`
transitions from "lock-lost on every transcript page" to "no
deaths, no stalls, steady throughput."
RECOVERY REBUILD 2026-05-26 of original ac213aa6.
* fix: fold SYNOPSIS_DOC_MAX_CHARS into corpus_generation hash
Codex review of #1427 flagged that changing GBRAIN_SYNOPSIS_DOC_MAX_CHARS
shifts the synopsis prompt + downstream embeddings for long documents
but was NOT folded into the computeCorpusGeneration hash. Pages
re-embedded with a different cap would retain the same
corpus_generation, defeating the v0.40.3.0 D27 P1-5 cache invalidation
contract.
Three changes:
1. Export SYNOPSIS_DOC_MAX_CHARS from src/core/page-summary.ts
2. computeCorpusGeneration accepts optional synopsisDocMaxChars param.
When set, folded into hash via '|doc_cap=<N>'. Omitted for
non-synopsis modes (title / none don't consult the cap) so existing
pre-PR caches stay valid for those.
3. Service-layer call sites (2 in contextual-retrieval-service.ts)
pass SYNOPSIS_DOC_MAX_CHARS when attemptMode/resolution.mode is
per_chunk_synopsis, undefined otherwise.
4. import-file.ts inline path passes undefined (per_chunk_synopsis
refused upstream there).
One-time effect: per_chunk_synopsis pages re-embedded post-PR get a
NEW corpus_generation including the cap. v0.40.3.0 query_cache.page_generations
contract auto-invalidates cached query results on first re-embed.
Future cap changes track correctly.
Addresses codex review P2 on PR #1427.
---------
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-07-21 16:50:43 -07:00
689 changed files with 54375 additions and 4236 deletions
All notable changes to GBrain will be documented in this file.
## [0.42.73.2] - 2026-08-05
**A write that deduplication redirects onto an existing page is now checked against the write scope of whoever asked for it.** When the same content arrives under a new slug, gbrain recognises it and points the write at the page that already holds it. That redirected target is now tested against the caller's own scope — under whichever mechanism confines that caller. One of the two mechanisms was consulted at that point; both are now.
Nothing changes for local CLI use, or for clients that hold unrestricted write access — neither was ever scope-confined. A confined caller whose write dedups onto a page **inside** its own scope keeps working exactly as before; that redirect is a feature and it is preserved, with a regression test to keep it that way. A confined caller whose write dedups onto a page **outside** its scope now gets `permission_denied`, with the remedy in the message: drop the `id:` frontmatter field, or change the content, to write a new page under your own prefix. The denial does not name the page the write resolved to.
Recommended for any brain served over HTTP to scope-restricted clients.
### To take advantage of v0.42.73.2
```bash
gbrain upgrade
```
Nothing to configure. Existing clients keep their scopes unchanged, and no re-registration is needed.
### For contributors
Reported privately by an external security researcher, who supplied a fix and a regression test with it. The version that shipped composes the two existing scope-matching rules into a single predicate rather than restating either one, so the check at the door and the check after a redirect cannot drift apart; the audit the report prompted closed the same gap on one further caller path.
## [0.42.73.1] - 2026-08-05
**Removes the PR gate that v0.42.73.0 added, and reverts the v0.42.72.1 contribution-policy change it enforced.** The gate cannot function on this repository, and it caused a real incident before that was understood.
The gate needed two things this repository does not grant it: an `ANTHROPIC_API_KEY` Actions secret for its verdict, and read-write workflow permissions to post a comment or set a label. Without them it can only skip. Worse, on its first live runs a read-only token turned every API call into a 403, the code treated that as a crash, and the check went red on an outside contributor's pull request four times with no comment explaining why. That was fixed in v0.42.73.0, but a check that runs on every pull request and can never reach a verdict does not earn its place in the repository.
The v0.42.72.1 contribution policy is also withdrawn: the human-written intent paragraph and gbrain-in-use screenshot are no longer required on issues and pull requests. `CONTRIBUTING.md`, both issue templates, and the pull-request template return to their pre-2026-08-02 state, and issues and PRs are reviewed on their content by maintainers, as before.
The code is preserved in git history at v0.42.73.0 and can be restored if the repository ever grants those permissions. If it is restored, the mechanical half — the intent and screenshot check, the version-first title rule, the red flags — should render to the Actions job summary instead of a comment, because that needs no token permission and no API key.
### To take advantage of v0.42.73.1
```bash
gbrain upgrade
```
Nothing to change. Everything else v0.42.73.0 shipped — the five contributed correctness fixes, `slug_filter`, and the four dependency pins that cleared six CVEs — is unaffected and stays.
## [0.42.73.0] - 2026-08-04
**Every incoming pull request now gets a verdict before anyone reads it — and five contributed fixes for silent wrong answers.**
**The PR gate.** Open a pull request against gbrain and an automated check now posts a single verdict comment within a minute: **merge-lane**, **close-lane**, or **needs-maintainer**, with its reasons and a checklist of what a human reviewer should verify for that specific diff. It also checks mechanically that the description carries the human-written intent paragraph and the screenshot of gbrain in use that `CONTRIBUTING.md` requires, and that the title leads with its version.
It is deliberately **advisory** — a triage signal and a reviewer checklist, not an authorization boundary. A green verdict is not permission to merge; a maintainer still decides. Pull-request code is never checked out or executed: the verdict comes from the description and the diff read through the API. Maintainer, bot, and draft pull requests are exempt from the intent-and-screenshot floor only (release automation cannot screenshot itself); they still receive the full verdict. Where the rubric can be argued with, the decision is taken away from it: a merge-lane recommendation is downgraded automatically when a diff adds a dependency, a new provider recipe, or new config keys, edits workflows, deletes a test, exceeds 40 files or 400 net source lines, or changes `src/` without touching a single test.
**Your import output parses again.** `gbrain import <dir> --json` printed five informational lines to stdout ahead of the JSON payload, so anything parsing that output read zero imports while its own bookkeeping recorded the files as ingested — and the next run skipped them permanently. Those lines now go to stderr under `--json`; human output is byte-for-byte unchanged.
**`sources harden --dry-run` no longer changes anything.** It reset the helper's executable bit before reaching the dry-run check, so a documented preview quietly mutated permissions.
**Telemetry records the model that actually ran.** Two nightly-cycle phases wrote a hardcoded or unrelated model name into their verdict cache, evidence signature, and spend metering while the gateway ran whatever chat model you configured. On any brain with a non-default model, the recorded history was fiction.
**`gbrain integrity` stops contradicting itself.** Dead-link findings were counted in the "Review queue" total but written to a different file, so `integrity review` disagreed with `integrity auto`'s own summary. They now get their own line.
**Retype rules can address API-ingested pages.** Mapping rules could only filter on a file path, which is empty for every page written through `put_page` — so no rule could target that whole class. A new `slug_filter` filters on the slug instead, and combines with the path filter when both are given.
Also: the `integrity` source comment no longer documents a `--dry-run` subcommand form that exits with an error.
### To take advantage of v0.42.73.0
```bash
gbrain upgrade
gbrain import <dir> --json | jq . # now parses
gbrain integrity auto # dead links reported separately
```
Nothing to configure for the gate — it runs on pull requests to this repository. If you maintain a fork and want it, the workflow needs an `ANTHROPIC_API_KEY` secret; without one it skips loudly rather than blocking anyone.
### For contributors
The gate went through six rounds against two independent blind reviewers, each judging cold. The findings that changed the design most were not exploits but false positives: a code fence that swallowed the rest of a description, an explanation written as bullet points scoring zero words, a word floor stricter than the published policy, and a comment telling contributors to reopen a pull request that was never closed. Those four descriptions are now permanent regression fixtures — a gate that insults a first-time contributor is worse than no gate. Two properties are deliberate and documented rather than fixed: the mechanical floor is a floor (a determined author clears it in seconds), and a bare URL in a cited reason still autolinks.
Contributed by @YiconZiwei (#2655), @time-attack (#3764, #3759, #3726, #3751, #3739, and the gate groundwork in #3573/#3698).
## [0.42.72.1] - 2026-08-02
**Every issue and pull request now needs a human-written paragraph and a screenshot of gbrain actually being used.**
Effective immediately, opening an issue or a PR requires two things from you personally: a paragraph you wrote yourself saying why you're opening it — what you were doing, what went wrong or what you needed, why it matters — and a screenshot of your terminal, agent session, or logs showing the real situation. Rough grammar is fine and preferred over polish. AI-generated or AI-polished intent text is not accepted; the paragraph is the human part. AI assistance for the *code* is still welcome.
Issues and PRs missing either are closed without review, and can be reopened once both are added. Scrub private names, companies, keys, and brain contents from screenshots before attaching — a redacted screenshot is fine, a missing one is not.
The requirement is stated in `CONTRIBUTING.md` and pre-filled in the bug-report and feature-request issue templates plus a new pull-request template, so the fields are in front of you when you open one.
## [0.42.72.0] - 2026-08-01
**Per-person write isolation inside a shared source, and a guide for putting gbrain behind a multi-user agent harness.**
Until now, `--source` was the only write boundary: a client could write anywhere inside the source it was scoped to, and keeping each person in their own folder was a convention the agent had to honor by itself. Registering a client with `--bound-slug-prefixes` now makes that boundary real. Writes outside the bound prefixes are refused by the server, on every op that can name a page.
**Adding a binding to an existing client narrows it on purpose.** Ops that write by something other than a page slug can't be confined to a prefix, so a bound client is refused them outright rather than left with an unfenced path: `extract_entities`, `extract_facts`, `forget_fact`, `ontology_propose`, `sources_add`/`sources_remove`, and `POST /ingest`. `put_page`'s automatic fact extraction is skipped for the same reason — it writes to entity pages the caller never named. Reads are unaffected, and unbound clients behave exactly as before. The gate keys on "anything that is not a plain read", so an op added in a future release is refused to bound clients until it is explicitly fenced.
Both prefix spellings work: the `wiki/agents/alice/*` glob that `submit_agent` bindings already use, and the plainer `emp-alice/` form. Change a binding in place with `gbrain auth rescope-client <id> --bound-slug-prefixes <p1,p2|none>` — existing tokens pick it up on their next request, so no secret rotation is needed when someone joins or leaves a team.
**New guide: [gbrain as the company brain for a qm deployment](docs/integrations/qm-harness.md).** qm is a multiplayer agent harness where each employee and each channel gets an isolated agent scope. The guide covers the whole path — one central `gbrain serve --http`, the thin-client binary baked into the sandbox image, one OAuth client per scope, and a roster-driven provisioning script that converges the brain to a list of people and channels. It also states plainly what the model does *not* give you: within a shared source, reads stay source-granular, so prefix isolation is a write boundary, not a privacy boundary.
gbrain upgrade # or: bun install -g gbrain@0.42.72.0
gbrain apply-migrations --yes # required: the fence refuses writes it cannot evaluate
Verify it took, from a client holding that credential — the first write should succeed and the second should be refused:
```bash
gbrain put partners/alice-example/notes/test --content "mine"
gbrain put partners/bob-example/notes/test --content "not mine"
```
## [0.42.71.0] - 2026-08-01
**GBrain now publishes real releases. Every version bump from here on lands on the [Releases page](https://github.com/garrytan/gbrain/releases) with organized notes and downloadable binaries — and binary self-update finally works.**
Until now the repo had no releases at all: `gbrain check-update` could tell you a new version existed, but `gbrain self-upgrade` downloaded from an empty releases API and failed every time, and anyone trying to follow what shipped had to read raw commit history. That's what people have been (rightly) complaining about.
From this release forward, every version bump automatically:
- **Tags the commit** (`v0.42.71.0`) so versions are addressable in git.
- **Publishes a GitHub Release** whose notes are that version's CHANGELOG entry — the same organized, user-facing writeup, not a commit dump.
- **Attaches compiled binaries** for macOS (arm64) and Linux (x64), so `gbrain self-upgrade` and fresh binary installs work without a toolchain.
The pipeline is idempotent: a partial release (tag exists, assets incomplete) is repaired on the next run instead of wedging. It runs post-merge, so a flaky release build can never turn master red. Releases for today's two fix waves (v0.42.69.0 and v0.42.70.0) have been backfilled with their CHANGELOG notes so the Releases page tells the whole story of the day; binaries attach from v0.42.71.0 onward.
### To take advantage of v0.42.71.0
```bash
gbrain check-update # now resolves against real releases
gbrain self-upgrade # now actually downloads a binary
```
Or browse https://github.com/garrytan/gbrain/releases for organized per-version notes.
### For contributors
`docs/RELEASING.md` gains the release-publication section; `scripts/changelog-entry.sh` extracts a version's CHANGELOG section (used for release notes — keep entries under the standard `## [X.Y.Z.W]` headers and they publish verbatim). The workflow keeps all actions SHA-pinned, tightens top-level permissions to `contents: read` with write scoped to the release job only, and env-binds all interpolations.
Contributed by @time-attack (#3573, closing #3521).
## [0.42.70.0] - 2026-08-01
**Community fix wave two: 18 contributed fixes. The headline: several things you asked gbrain to do were being quietly ignored — and now they aren't.**
**`--brain` now actually routes.** The documented `gbrain query "X" --brain media-team` parsed the flag and then ran against your host brain anyway. It now routes to the named brain, and an unknown brain name fails loudly instead of silently answering from the wrong database.
**`sync --dry-run` no longer touches anything.** A dry run could pull from the remote and — if your sync strategy had changed — delete indexed pages before the "dry run" early-return was reached. Previews are now read-only, full stop.
**`apply-migrations --yes` applies.** It previously warned that your schema was behind and then printed "All migrations up to date" with exit 0. If you have wedged brains that upgrade never healed, this was why.
**Links between your pages resolve the way you write them.** Dir-qualified wikilinks with raw Obsidian names (`[[wiki/entities/AI 3.0]]`) now resolve to the sync-slugified page; references in non-whitelisted directories are no longer silently dropped; and a scan bug that could add an edge to a *parent* page you never referenced was caught in the wave's composite review and fixed before shipping.
**Windows and self-hosters.** Markdown files keep LF endings so frontmatter parsers stop mis-reading on Windows checkouts; the archive-crawler path gate no longer denies every real Windows path (and no longer fail-opens on NTFS case-insensitivity); a chat-synopsis tier that was hardcoded to one provider now follows your configured models; vector search asks the index for as many candidates as it was told to consider.
**Quieter, more honest infrastructure.** `serve --http` no longer leaves an orphan holding the database lock after Ctrl-C; a minion child that fails to launch settles immediately instead of hanging its slot; doctor gains checks for content-hash duplicates, undeclared database-only pages, stale heartbeats, and a tamper-evident manifest for the skills directory; federated reads respect per-source isolation settings in two more paths; and the security docs were rewritten to describe fixes without cataloguing attack surface.
gbrain upgrade
gbrain extract --stale # re-extracts links under the fixed resolver
gbrain doctor # includes the new silent-failure checks
```
If your brain uses `link_resolution.global_basename` and was populated before this release, a small number of superseded `wikilink_basename` edges can linger beside their newer typed replacements after re-extraction (edge writes are append-only by design). `gbrain reconcile-links` cleans them up; they are harmless to queries that dedup on target.
### For contributors
The composite review of this wave (two independent max-effort review passes over the combined branch) caught two interaction defects that per-PR review could not: the ungated bare-path scanner reading inside wikilink spans, and an extraction watermark set to a date that same-day stamps would already outrun. Both were fixed in the wave with discriminating tests. One reviewed-and-approved PR was deliberately held out: it conflicts semantically with its author's own sibling PR in this wave, and choosing between their two path-resolution mechanisms is the author's call.
**A community fix wave: 22 contributed fixes, most of them for work your brain was quietly not doing.**
The theme of this release is silent failure. A nightly cycle that reported `ok` while extracting nothing. An `embed` run that left a whole page unsearchable because one chunk in it failed, then exited 0. A health metric that recommended the same step forever because it counted one thing and the fix measured another. None of these looked broken from the outside, which is exactly why they lasted.
**If you run a local or non-Anthropic model, atom extraction was doing nothing.** With a cost cap set, any model absent from the pricing tables made the first work item hard-fail, which latched a budget flag and skipped every remaining item — while the phase still reported success. Local models (`ollama`, `llama-server`) now price at $0, because local inference costs electricity rather than tokens, so their caps stay enforceable. Genuinely unpriced paid providers still skip, but loudly now instead of silently.
**`gbrain embed` no longer lets one bad chunk darken an entire page,** and it exits non-zero when embeddings actually fail. If you have a cron wrapping `gbrain embed`, a brain holding permanently un-embeddable content will now turn that cron red. That is the intended change — it was previously green while silently incomplete.
**Non-Latin and diacritic names now survive mention extraction.** The by-mention tokenizer matched ASCII letters and digits only, so `Đà Nẵng` shredded into one- and two-character fragments and never matched anything. Names in Vietnamese, and any script outside ASCII, are now tokenized properly.
**Self-hosted embedding backends work.** Fixed-dimension OpenAI-compatible servers that reject an explicit `dimensions` parameter no longer get sent one when the requested width already matches the model's native width. A vector search on the embedded database also now asks the index for as many candidates as it was told to consider, instead of silently truncating the pool to the driver default.
**Multi-source brains route correctly in two more places.** A programmatic `sync_brain` call now syncs the source it was handed rather than the global default, and entity slug resolution keeps its path separators instead of flattening `people/alice-example` into an id no page can hold.
**Safer default on a destructive migration.** Submitting the type-unification job without an explicit `apply` now previews instead of applying. If you have that command in a runbook, add `"apply":true` — the playbooks and README were updated to show it.
Also: interrupted imports keep their tail instead of losing progress below the next 100-file boundary; `gbrain init --help` prints its own help instead of a stub; `doctor` stops reporting Windows drive paths as missing files under WSL and bounds its embedding health probe instead of retrying a permanent auth failure three times; cycle lock-release and stamp-write failures are visible instead of swallowed; and references to a `gbrain install` command that never existed are gone from the docs.
### To take advantage of v0.42.69.0
```bash
gbrain upgrade # or: bun install -g gbrain@0.42.69.0
gbrain doctor # confirms the health metric now converges
gbrain embed --stale # exits non-zero if anything is genuinely un-embeddable
```
If you use a local chat model for the nightly cycle, re-run it once and check that atoms actually land:
If you have `gbrain jobs submit unify-types` in a runbook or script, add `"apply":true` to its `--params` or it will now preview only.
### For contributors
Two defects existed only in the *combination* of otherwise-sound fixes, and were caught by reviewing the composed branch rather than the individual changes:
- `isModelPriceable` was introduced with a test asserting `llama-server` is unpriced, while a second fix in the same wave priced `llama-server` at $0. Together the assertion inverted. Reconciled by using a genuinely unpriced provider in the test and pinning the positive case: free local providers are priceable at $0, so their caps stay enforced.
- The type-unification default flipped to dry-run, but three agent-facing playbooks still presented a bare submit as the apply step. Because a second fix in the same wave also edited one of those files, each change looked self-consistent alone. Skills ship downstream via the skillpack, so this would have propagated a playbook whose apply step silently did nothing.
One reviewed fix was deliberately held back: extending the inline subagent drain to Postgres composes badly with this wave's minion connection-recovery work, since the drain calls the same queue operations without the new recovery path and can strand a child job in a per-run queue no worker will claim.
**If you run `gbrain reindex-frontmatter` or `gbrain backfill` on the default embedded database, they now work. Until this release both failed every time, after waiting 30 seconds.**
The embedded database allows one process at a time, and holds a lock to enforce it. These two commands opened a second connection to the same database from inside the process that already held that lock, then waited for a lock that could never be released — because the thing holding it was the waiting process itself. The wait ran its full 30 seconds and the command exited with an error naming a blocking process that was, in fact, itself. Both commands now reuse the connection that is already open.
Nothing changes for brains on Postgres, where a second connection was always allowed.
## To take advantage of v0.42.68.1
Nothing to undo — the commands failed without writing anything. Just run whichever you needed:
```bash
gbrain reindex-frontmatter
```
## [0.42.67.0] - 2026-07-28
**If you develop GBrain on Windows, the test and check commands now actually run. Until this release they were quietly doing almost nothing.**
`bun run test`, `bun run verify`, `bun run ci:local` and `bun run test:e2e` all hand off to shell scripts, and on Windows that hand-off was broken in two separate places. The commands did not stop with an obvious error. They reported a result, so a run could look finished when barely any of the checks had actually inspected anything. On a clean Windows clone, `bun run verify` got 1 check to pass and 31 to fail. It now gets 25 to pass and 7 to fail, and none of the 7 are caused by this change.
The first problem was line endings. Git for Windows installs with `core.autocrlf=true`, which rewrites shell scripts to Windows line endings when you clone or check out. Bash refuses to run those, so a script died on its second line before doing any work. The scripts stored in the repository were always correct; only the copy on your disk was wrong. A new `.gitattributes` pins every `.sh` file to Unix line endings at checkout, no matter how your Git is configured.
The second problem was how the checks were started. Thirty three of them pointed straight at a `.sh` file. On macOS and Linux the shell reads the `#!/usr/bin/env bash` line at the top of the script and runs it correctly. Bun on Windows does not do that, so those commands failed the moment they were called. They now go through `bash` explicitly, the same way the other eleven were already written.
Nothing changes for macOS and Linux. No stored file content moves, and no check behaves differently on those platforms.
## To take advantage of v0.42.67.0
Only Windows contributors need to do anything, and only once. `.gitattributes` applies at checkout time, so shell scripts already sitting on your disk keep their old line endings until you refresh them.
1. **Refresh the working copy** from the repository root:
```bash
git rm --cached -r . -q
git reset --hard
```
2. **Confirm bash can read the scripts:**
```bash
bash -n scripts/run-unit-parallel.sh
```
Silence means it worked. `$'\r': command not found` means step 1 did not take effect.
3. **Run the gate:**
```bash
bun run verify
```
### Itemized changes
- New root `.gitattributes` pins `*.sh text eol=lf`, so shell scripts check out with Unix line endings regardless of the contributor's `core.autocrlf` setting. All 59 tracked `.sh` files were already stored with Unix endings, so `git add --renormalize .` reports nothing to do and no stored content changes.
- `package.json` now routes the remaining 33 `.sh` check commands through `bash`, matching the 11 that already did. Every tracked `.sh` file carries a bash shebang (52 `#!/usr/bin/env bash` and 7 `#!/bin/bash`), so the treatment is uniform across all of them.
- The five `scripts/*.ts` entries still run under bun and are untouched.
- `CONTRIBUTING.md` gains a Windows section covering the one-time working-copy refresh and the `bash scripts/<name>.sh` convention for new checks.
- `docs/TESTING.md` records how the test commands dispatch through bash, and notes that three tree-walking checks plus `typecheck` can exceed the 120s per-check cap on Windows while passing on Linux and macOS.
## [0.42.66.1] - 2026-07-27
### Fixed
- `gbrain doctor` now treats embedding columns wider than pgvector's HNSW limit as healthy exact-scan configurations instead of prescribing an index PostgreSQL cannot build.
- Local CI now passes an empty Docker mount list correctly and compiles the embedded-WASM smoke binary from container-local storage on Docker Desktop.
## [0.42.66.0] - 2026-07-24
**54 verified fixes from the community backlog: background enrichment stops wasting money on dead pages, autopilot stops killing its own healthy runs, and search respects your settings.**
This release is the second big sweep through the open pull-request backlog, with every change reviewed and tested individually before merging. The theme is trust in the background machinery. The overnight "dream" cycle now remembers which pages produced nothing and stops re-reading them every night, meters its small-model calls against your spend caps, and keeps claim proposals from silently overwriting each other. Long consolidation runs get a 30-minute deadline instead of being killed at 10 minutes mid-work. A wedged server boot now releases its database lock instead of blocking every later command.
Search behaves the way you configured it: the recency-decay setting now actually applies to hybrid search, a local `list_pages` call returns as many rows as you asked for, and when a listing is cut short it says so instead of looking complete. Slack conversation exports parse cleanly, with an optional AI fallback for formats the parser does not know.
New provider recipes: DashScope reranking, OpenRouter reranking, and a claude-cli recipe for dispatching subagents through the gateway.
## To take advantage of v0.42.66.0
`gbrain upgrade` should do this automatically. One schema migration ships in this release (v125, take-proposal idempotency); it is idempotent and needs no manual action.
1. **Upgrade and verify:**
```bash
gbrain upgrade
gbrain doctor
gbrain stats
```
2. **If `gbrain doctor` warns about a partial migration**, run the orchestrator manually:
```bash
gbrain apply-migrations --yes
```
3. **If any step fails,** please file an issue at https://github.com/garrytan/gbrain/issues with the output of `gbrain doctor` and `~/.gbrain/upgrade-errors.jsonl` if it exists.
### Itemized changes
#### Dream cycle, takes, and spend control
- Pages whose extraction yields zero claims are memoized, so the cycle stops re-spending on them every night. (#2514, #3319, contributed by @ivandebot)
- Zero-yield pages are tombstoned so `extract_atoms` stops rediscovering them. (#2144, #3304, contributed by @ChenyqThu)
- `extract_atoms` Haiku calls are metered against the cost gate. (#2371, #3329, contributed by @TheRealMrSystem)
- `extract_atoms` stamps concepts so `synthesize_concepts` has material to work with. (#2123, #3308, contributed by @ChenyqThu)
- `extract_facts` requires a live backing page, not just a non-NULL entity slug. (#2497, #3321, contributed by @javieraldape)
- Multi-claim pages keep every proposal instead of only the first (migration v125 makes the idempotency key per claim). (#3297, contributed by @rp-agent-bot)
- Superseding a take now queries the active row first. (#3275, contributed by @arisgysel-design)
- Takes keyword search matches words inside long claims via `word_similarity`. (#3267)
- Dream-generated orphan pages stay scoped to their source. (#2368, #3344, contributed by @snvtac)
- Drift detection is wired into the dream cycle, report-only for now. (#2653, #3317)
#### Autopilot, jobs, and serve
- Full consolidation cycles get a 30-minute timeout floor; lighter dispatches keep the interval-derived budget. (#2852, #3338, contributed by @sanchalr)
- The cron wrapper exports `~/.bun/bin` onto PATH so autopilot survives minimal environments. (#2013, #3305, contributed by @klampatech)
- Dead or cancelled jobs no longer block idempotent re-submission. (#2253, #3306, contributed by @rafaelreis-r)
- Contextual reindex jobs get a default timeout. (#2611, #3323, contributed by @spiky02plateau)
- Onboarding stops repeating the same auto-remediation within a single run. (#2854, #3342, contributed by @sanchalr)
- A wedged `gbrain serve` boot hits a readiness deadline and releases the PGLite lock. (#3335)
#### Search, retrieval, and health
- The recency-decay config is honored on the hybrid search path. (#2386, #3312, contributed by @rwbaker)
- `list_pages` honors explicit limits for local callers, warns on remote clamping, and threads `offset`. (#2591, #3322, contributed by @deacon-botdoctor)
- Truncated `list_pages` results say so instead of silently capping. (#2865, #3341, contributed by @paul-0320)
- Negative metrics no longer invert trajectory regression signals. (#2621, #3324, contributed by @morluto)
- Per-chunk synopsis generation in contextual retrieval is concurrency-bounded. (#2628, #3326, contributed by @spiky02plateau)
- Graph health metrics count `entity` pages. (#2639, #3330, contributed by @tylr-r)
#### Ingestion, extraction, and links
- Conversation parsing gains an opt-in LLM fallback for unknown formats. (#2247, #3371, contributed by @danwiggins)
- Normalized Slack markdown parses into conversations. (#3289, #3372, contributed by @danwiggins)
- Conversation backfill outcomes are durable, so completed pages skip on the next run. (#3293, #3373, contributed by @danwiggins)
- Reference-style wikilinks are recognized during extraction. (#2071, #3303, contributed by @mzkarami)
- `[[wikilink]]` frontmatter values resolve via global basename lookup. (#2406, #3313, contributed by @spiky02plateau)
- `CLAUDE.local.md` / `AGENTS.local.md` are gitignored. (#3290, contributed by @igbymyboy)
- The hybrid-reranker integration test isolates `GBRAIN_HOME`. (#1527, #3327, contributed by @Willisbest)
- Test-shard scripts capture the real exit code before watchdog teardown in the no-timeout fallback. (#2864, #3340, contributed by @paul-0320)
## [0.42.65.0] - 2026-07-23
**A large maintenance release: 93 verified fixes and small features merged since v0.42.64.0, most of them community contributions.**
If you use gbrain day to day, this release makes the boring parts trustworthy. Importing and syncing notes is safer: a failed pull no longer pretends everything is up to date, imported pages are read back after writing to confirm they landed, and a page with real content can no longer be silently overwritten by an empty one. Search answers get better inputs: the think command now picks excerpts that actually match your question, and results respect your federated source settings. Background enrichment (the "dream" cycle) wastes less money and retries properly when an AI provider is down. Spending caps now fail closed, so a billing hiccup can never turn into an uncapped spend. And `gbrain doctor` is quieter, with several false alarms removed and real problems (like an embedding backlog with no worker running) now flagged.
More AI providers work out of the box, including OpenRouter prompt caching, MiniMax and Zhipu GLM recipes, Ollama Matryoshka embedding dimensions, and llama-server batch limits.
## To take advantage of v0.42.65.0
`gbrain upgrade` should do this automatically. No new schema migrations ship in this release.
1. **Upgrade and verify:**
```bash
gbrain upgrade
gbrain doctor
gbrain stats
```
2. **If `gbrain doctor` reports new findings after upgrading,** that is the quieter, more accurate check set working as intended. Each finding names its fix.
3. **If any step fails,** please file an issue at https://github.com/garrytan/gbrain/issues with the output of `gbrain doctor` and `~/.gbrain/upgrade-errors.jsonl` if it exists.
### Itemized changes
#### Security
- MCP source scoping for remote callers got a hardening pass, so agent-facing connections stay confined to the sources they were granted. (#2881, contributed by @spinsirr)
- Paid MCP spend accounting is now atomic and fails closed, and resolver spend is recorded before a cap error is raised, so caps cannot be raced past or undercounted. (#3203, #3204, contributed by @caterpillarC15)
- The OAuth token endpoint rate limit on the HTTP server is now configurable via env for deployments behind shared IPs. (#3114, contributed by @time-attack)
- `WWW-Authenticate` responses now carry `resource_metadata` per the MCP spec and RFC 9728, so conforming clients can discover the auth server. (#1410, contributed by @rayers)
#### Search, retrieval, and think
- `think` selects query-relevant excerpts instead of generic ones. (#3197, contributed by @Y0lan)
- Unqualified local CLI `search`/`query` now honors `sources.config.federated` read visibility. (#2561, #3141, contributed by @time-attack)
- Email citation metadata is projected into search results. (#2873, contributed by @amtagrwl)
- The `think` Gaps section renders once instead of twice. (#1662, contributed by @howwohmm)
- Fuzzy entity lookup threads the caller's source scope and skips soft-deleted entities. (#1508, contributed by @tim404x)
- `code-def` surfaces method, constructor, field, and struct definitions, not just top-level symbols. (#1628, contributed by @rayers)
- Briefing pages are excluded from their own Brain Pulse salience. (#1202, contributed by @rwbaker)
- Reranker calls with missing auth are classified as configuration errors before falling back. (#2059, #3139, contributed by @time-attack)
#### Import, sync, and ingestion
- A failed git pull with zero imports reports `partial (pull_failed)` instead of `up_to_date`. (#3068, #3253, contributed by @Masashi-Ono0611)
- Imports run a post-write read-back verification with a durable ingest-log record. (#2869, contributed by @Andredsouza1984)
- `put` refuses to overwrite a non-empty page with empty content. (#2708, contributed by @symmetric-matthew)
- `putPage` restores soft-deleted rows instead of colliding with them. (#2779, contributed by @RerankerGuo)
- Mixed-case slugs are normalized before chunk upsert, ending duplicate-chunk churn. (#430, #3143, contributed by @time-attack)
- Imports fall back to the body H1 for the title when frontmatter lacks `title:`. (#2446, #3072, contributed by @time-attack)
- YAML comments inside the frontmatter fence are no longer treated as markdown headings. (#3225, #3247, contributed by @Masashi-Ono0611)
- Write-through guards case-insensitive filesystem collisions before the atomic write. (#2831, #3119, contributed by @time-attack)
- Path-qualified wikilinks outside the known directory pattern resolve on the DB/put_page path. (#2866, contributed by @paul-0320)
- CJK slugs are supported in the slug registry and dream-cycle summary slugs. (#782, #738, #3083, contributed by @time-attack)
- Three ingest/sync/serve singleton fixes: page-type round-trip, deleted-slug embed noise, and a stateless width guard. (#3140, contributed by @time-attack)
- Sync honors the `embedding_disabled` sentinel as an implicit `--no-embed`. (#2879, contributed by @gawievanblerk)
- Verified sync head sentinels are cleared correctly. (#2734, contributed by @symmetric-matthew)
- Resumed syncs report the pinned commit they actually landed on. (#3202, contributed by @caterpillarC15)
- The expected `discover_git_root` probe failure stays off stderr. (#3232, contributed by @Masashi-Ono0611)
- `extract --stale` runs the real resolver so basename resolution reaches stale pages, and clears pre-version-bump pages. (#2576, #2717, contributed by @paul-0320; #1791, contributed by @Nazim22)
- Oversized code chunks are capped so they stay embeddable, and code-chunk metadata survives re-embeds. (#1675, contributed by @lubosxyz; #769, #1232, contributed by @rayers)
#### Background cycle, dream, and facts
- Path-derived dream sources are stamped, and the engine closes cleanly on autopilot shutdown. (#3178, contributed by @time-attack)
- All-provider-failed atom drains propagate so durable jobs retry instead of silently dropping work. (#3218, #3248, contributed by @Masashi-Ono0611)
- Atom extraction raises `maxTokens` and case-normalizes `atom_type` for Gemini models. (#3211, contributed by @alexey-metaengage)
- The conversation extractor gates anonymous-speaker self-attribution instead of guessing. (#3228, contributed by @asenkovskiy)
- Incremental dream extraction stamps its watermark so re-runs stop reprocessing. (#2636, #3115, contributed by @time-attack)
- `dream --dry-run --json` keeps stdout clean of embed summaries. (#394, #3109, contributed by @time-attack)
- Synthesized dream pages require a self-contained opening summary. (#2770, contributed by @Masashi-Ono0611)
- PGLite inline synth subagent drains complete, and `lint` gains `--exclude`. (#2699, #2649, #3162, contributed by @time-attack)
- Live context reads the documented "P1 Today" heading form with plain checkbox tasks, matching the daily-task-manager skill's output format. (#2186, #3124, contributed by @time-attack)
- Queued AI jobs refresh gateway config at execution time instead of using a stale snapshot. (#2125, contributed by @maxpetrusenkoagent)
- `brainstorm`/`propose_takes` honor configured models: cost preview uses the configured model, the judge reads its config key, provider probes are skipped when unneeded, and page projection is narrowed. (#3120, contributed by @time-attack)
- Backlog hardening wave: x-to-brain health check, propose_takes deadlines, capture title truncation, extract_atoms backlog handling, and pooler direct-URL routing. (#3165, contributed by @time-attack)
- `skillopt` emits `proposed.md` in no-mutate mode. (#2635, #3182, contributed by @time-attack)
- Nightly quality probe enable path and conversation-parser probe are wired up. (#2629, #2630, #3094, contributed by @time-attack)
#### Doctor, health, and maintenance
- New safe maintenance automation with a shared orphan-exclusion policy, so routine cleanup runs without risking linked content. (#3015, #3023, contributed by @time-attack)
- `orphan_ratio` excludes the chronicle volume under `life/events/`. (#2264, #3214, contributed by @asenkovskiy)
- `brain_score` orphan/timeline components use the orphans-audit linkable scope. (#3155, contributed by @time-attack)
- Entity timeline coverage is measured separately from whole-brain density. (#2761, contributed by @TurgutKural)
- Doctor flags embed backfills queued with no worker running. (#2696, contributed by @javieraldape)
- Two doctor false-positive/timeout fixes: the drift walk skips `node_modules`, and the bare-tweet check skips inline code and cited lines. (#1772, contributed by @sonlndv)
- A dead `llm_fallback_enabled` recommendation is dropped from conversation format coverage. (#1903, contributed by @ElliotDrel)
- Skill triggers with CRLF line endings parse on Windows. (#1149, contributed by @samporter-31)
- Onboard check names are registered in doctor categories, ending unknown-check warnings, and onboard-check remediations survive the `--apply --auto` path. (#3075, #3097, contributed by @time-attack)
- Dead slug prefixes are counted by slug. (#2697, contributed by @RerankerGuo)
- The backlinks worker defaults to check, not fix, and `check-backlinks` honors its positional directory argument. (#1853, contributed by @choomz; #3076, contributed by @time-attack)
- Calibration resolves the owner holder via config, defaulting to `self`. (#3077, contributed by @time-attack)
- Memory throttling on Linux reads `/proc/meminfo` MemAvailable. (#556, contributed by @chengzehsu)
#### AI providers and gateway
- OpenRouter gets family-scoped prompt caching, and query expansion works on chat-capable openai-compat recipes. (#3152, contributed by @time-attack)
- MiniMax recipe: embedding wire-shape compat fetch plus a chat touchpoint. (#1977, #3089, contributed by @time-attack)
- The Zhipu recipe gains a chat touchpoint so GLM subagents work. (#1157, #3084, contributed by @time-attack)
- Tier-configured models reach the recipe allowlist, Anthropic model lists are refreshed, tier resolutions are registered, and probe labels are honest. (#2800, contributed by @p3ob7o)
- Provider base URL config merges from the DB. (#1676, contributed by @TheLordArgus)
- The gateway falls back to the pooler when the derived direct host is unreachable. (#1641, #3088, contributed by @time-attack)
- Config-plane `voyage_api_key` folds into `VOYAGE_API_KEY` like the other hosted keys. (#3236, contributed by @Masashi-Ono0611)
- The `zeroentropyai:zerank-2` reranker has a pricing entry so the budget tracker can meter it. (#3223, #3233, contributed by @Masashi-Ono0611)
- llama-server embedding batches are capped at its 32-input request limit. (#1281, contributed by @mmekkaoui)
- Matryoshka dimensions thread through for Qwen3-Embedding on Ollama. (#1072, contributed by @mgandal)
- `init` seeds AI options from env on cold install, and `whoami` reports the stdio transport. (#3091, contributed by @time-attack)
- The `models` dispatch subcommand reads its first argument correctly. (#1428, contributed by @BenjaminDSmithy)
- Synopsis generation tail-truncates document text for small-model chat handlers. (#1427, contributed by @BenjaminDSmithy)
- The contradiction judge token cap is raised for thinking models. (#3210, contributed by @alexey-metaengage)
#### Schema, migrations, and storage engines
- Engine migration counts and surfaces per-page copy failures instead of silently advancing. (#3241, contributed by @Masashi-Ono0611)
- Invalid `CONCURRENTLY`-build index remnants are dropped without a DO block. (#3191, contributed by @Masashi-Ono0611)
- Unsupported large-dimension HNSW indexes are skipped instead of failing schema setup. (#1734, #3080, contributed by @time-attack)
- The v0.32.2 migration dirty-check scopes to targeted sources and surfaces failed phase detail. (#3093, contributed by @time-attack)
- Schema packs merge the full `extends` chain and `borrow_from` into the resolved manifest. (#1749, #3181, contributed by @time-attack)
- The schema-pack stats catch-all is narrowed so masked errors surface instead of fake zero-page counts. (#2466, #3133, contributed by @time-attack)
- Bundled schema-pack inspection reports the pack actually shipped in the binary, and minion subagent auth resolves through config. (#3110, contributed by @time-attack)
- PGLite `putPage` guards against zero-row RETURNING. (#1649, contributed by @alexhawkins)
#### MCP server and CLI surface
- `list_pages` rows include `source_id`. (#3209, contributed by @alexey-metaengage)
- Running CLI commands while `gbrain serve` (MCP) holds the brain now notifies about the conflict instead of failing confusingly. (#3243, contributed by @fdefitte)
- The OpenClaw plugin manifest entry is declared so the plugin loads. (#2551, #3185, contributed by @time-attack)
#### For contributors
- CI scanner roots are normalized on macOS. (#3198, contributed by @caterpillarC15)
- CI shard timeout raised to 22 minutes plus a delta-assert reporter leak test. (#3231, contributed by @time-attack)
- E2E suite hardening: flaky tests, no-op assertions, and cross-test coupling removed. (#1704, contributed by @auroracapital)
- `mechanical.test.ts` isolates `$HOME` so the E2E suite stops clobbering user config. (#434, contributed by @lloydarmbrust)
- The lint code-fence-wrap detector and fixer regex now agree. (#1597, contributed by @chungty)
- README project links for OpenClaw and Hermes are corrected. (#1961, #3179, contributed by @time-attack)
- A completed TODOS entry is dropped. (#3229, contributed by @Masashi-Ono0611)
@@ -65,14 +65,24 @@ This is the difference between a search engine and a brain. Search finds the pag
## Install
> [!WARNING]
> **GBrain is NOT distributed on npm.** The npm package named `gbrain` is an unrelated
> package with no connection to this project. Do not run `npm install -g gbrain` or
> `bun add -g gbrain` — you'll get something else, and it can shadow the real binary on
> your PATH. Install and upgrade ONLY via the documented paths below
> (`bun install -g github:garrytan/gbrain`, or `git clone` + `bun install && bun link`).
> If you already ran the npm install by mistake: `npm uninstall -g gbrain` /
> `bun remove -g gbrain`, then reinstall from GitHub. `gbrain doctor` detects a
> shadowing npm install and prints the fix.
GBrain is designed to be installed and operated by an AI agent. The fastest path is to have your agent do it for you. The CLI and MCP paths below are for people who want to wire it up themselves.
### Have your agent install it (recommended)
If you don't already have an AI agent platform running, start with one of these. Both are designed to read GBrain's install protocol and execute it:
- **[Hermes](https://github.com/NousResearch/hermes-agent)** — deploy on [Railway](https://github.com/praveen-ks-2001/hermes-agent-template) (one click)
Then paste this into your agent:
@@ -206,7 +216,7 @@ Most personal-knowledge tools force one fixed layout: their idea of "notes" + "p
**gbrain doesn't have a fixed layout.** It ships with bundled schema packs and lets you author your own when none fit:
- **`gbrain-base-v2`** (default as of v0.41.22) — 15-type DRY/MECE canonical taxonomy (14 canonical + `note` catch-all): `person`, `company`, `media`, `tweet`, `social-digest`, `analysis`, `atom`, `concept`, `source`, `deal`, `email`, `slack`, `writing`, `project`, `note`. Subtypes/format/origin pushed to frontmatter. The taxonomy that responds to issue #1479.
- **`gbrain-base`** (legacy, v0.41 and earlier brains) — the original 24-type layout. Stays bundled for back-compat; brains on it can upgrade via `gbrain onboard --check --explain` → `gbrain jobs submit unify-types --allow-protected --params '{"target_pack":"gbrain-base-v2"}'`.
- **`gbrain-base`** (legacy, v0.41 and earlier brains) — the original 24-type layout. Stays bundled for back-compat; brains on it can upgrade via `gbrain onboard --check --explain` → `gbrain jobs submit unify-types --allow-protected --params '{"target_pack":"gbrain-base-v2","apply":true}'` (omit `"apply":true` for a dry-run preview — that is the default).
- **`gbrain-recommended`** — extends `gbrain-base` with the 13 additional directories from `docs/GBRAIN_RECOMMENDED_SCHEMA.md` (source, place, trip, conversation, personal, civic, project, etc.). Activate with `gbrain schema use gbrain-recommended`.
- **Your own pack** — `gbrain schema detect` clusters your actual filesystem into proposed types, `gbrain schema suggest` runs an LLM pass over them, and `gbrain schema review-candidates --apply` promotes the ones you like. Three commands and the brain knows your shape. Authoring a successor pack (declares `migration_from:` so existing brains can opt in): see [`docs/architecture/pack-upgrade-mechanism.md`](docs/architecture/pack-upgrade-mechanism.md).
v0.40.7.0 emitter does NOT preserve comments. Authors who care
@@ -2549,7 +2580,7 @@ contributor traps.
- [ ] **v0.37.x: Adopt `resolveDefaultHeaders` for Together / Groq / other attribution-bearing recipes.** v0.37.6.0's `default_headers` / `resolveDefaultHeaders` seam is generic — any recipe whose provider benefits from app-attribution headers can opt in. Together and Groq both have rankings/analytics tied to per-app headers. Add their respective attribution headers to each recipe, similar to OR's `HTTP-Referer` + `X-OpenRouter-Title`. No type-system or gateway changes needed; just `default_headers` blocks on the existing recipes plus `<PROVIDER>_REFERER` / `<PROVIDER>_TITLE` env vars in their `auth_env.optional`. Filed during v0.37.6.0 eng review as a D4 generalization opportunity.
- [] **v0.37.x: Guard cli.ts `main()` so importing `buildGatewayConfig` doesn't print help.** v0.37.6.0 exported `buildGatewayConfig` from `src/cli.ts` for test access. Importing it triggers the file's top-level `main()` which prints help to stdout during tests — functionally harmless (tests pass) but noisy. Fix: wrap `main()` in `if (import.meta.main)` so it only runs when cli.ts is the entry point, not when imported. Touches one line; trivial. Filed during v0.37.6.0 implementation.
- [x] **v0.37.x: Guard cli.ts `main()` so importing `buildGatewayConfig` doesn't print help.** v0.37.6.0 exported `buildGatewayConfig` from `src/cli.ts` for test access. Importing it triggers the file's top-level `main()` which prints help to stdout during tests — functionally harmless (tests pass) but noisy. Fix: wrap `main()` in `if (import.meta.main)` so it only runs when cli.ts is the entry point, not when imported. Touches one line; trivial. Filed during v0.37.6.0 implementation.
## v0.37.4.0 pgGraph CI scaffolding follow-ups (v0.37.x+)
For shared / large / multi-machine deployments (a team or company brain with multiple users hitting one server over HTTP MCP with OAuth scoping per user), follow the dedicated walkthrough: **[Tutorial: set up GBrain as your company brain](tutorials/company-brain.md)**.
API keys live in `~/.gbrain/config.json` (file plane) or env vars (`OPENAI_API_KEY`, `ZEROENTROPY_API_KEY`, `VOYAGE_API_KEY`, `ANTHROPIC_API_KEY`). Set via CLI:
API keys live in `~/.gbrain/config.json` (file plane) or env vars (`OPENAI_API_KEY`,`OPENROUTER_API_KEY`,`ZEROENTROPY_API_KEY`, `VOYAGE_API_KEY`, `ANTHROPIC_API_KEY`). Set via CLI:
@@ -19,6 +19,32 @@ Seven test command tiers, each with a clear scope:
| `bun run test:e2e` | Real Postgres E2E. Requires Docker + `DATABASE_URL`. Sequential. | ~5-10min | Pre-ship; nightly. |
| `bun run check:all` | The historical pre-check scripts (22, chained sequentially in package.json). Overlaps `verify` heavily but is NOT a superset — `verify`'s `CHECKS` array in `scripts/run-verify-parallel.sh` (~30 entries incl. typecheck) is the authoritative gate; `check:all` keeps a few local-only extras (trailing-newline, exports-count, no-legacy-getconnection). | ~10s | Local-only sweep for the extras. |
### Shell dispatch and Windows
All four of `test`, `verify`, `ci:local` and `test:e2e` hand off to shell scripts
under `scripts/`, so every `check:*` entry in `package.json` invokes its script as
`bash scripts/<name>.sh` instead of relying on the shebang — bun on Windows cannot
exec a `.sh` directly. Add a new shell-script check with that same prefix. The
`scripts/*.ts` entries run under bun and take no prefix.
The scripts must also be on disk with Unix line endings. A strict bash (WSL, Linux
CI, macOS) rejects CRLF and dies on the script's first meaningful line; the Cygwin
bash that ships with Git for Windows tolerates it, so a green local run is not by
itself evidence that a script is CRLF-clean.
The root `.gitattributes` pins `*.sh text eol=lf`, which overrides the
`core.autocrlf=true` default that Git for Windows installs. It pins `*.md` the
same way, because the frontmatter readers anchor on a `---` fence followed by a
Unix line ending and a CRLF checkout makes a document parse as having no
frontmatter, silently. Working copies cloned
before those pins need a one-time `git rm --cached -r . -q && git reset --hard` to
pick them up; see the Windows section of `CONTRIBUTING.md`.
Wallclock figures in the table above are from a Mac dev box. Windows is
substantially slower because each check pays full process-creation cost, and three
plus `typecheck` can exceed the 120s per-check cap in `run-verify-parallel.sh`
there even though they pass on Linux and macOS.
### CI vs local: intentionally divergent file sets
- **CI matrix** (`.github/workflows/test.yml`) runs `scripts/test-shard.sh` across 10 matrix shards partitioned by weight-aware LPT bin-packing (`scripts/sharding.ts`) and INCLUDES `*.slow.test.ts` (the two outlier slow files run as dedicated jobs alongside the matrix). CI EXCLUDES `*.serial.test.ts` from the shards and runs them in a dedicated job via `bun run test:serial`, one bun process per file — keeping serial files out of the shard processes is what preserves the `mock.module` quarantine (a top-level mock in one file leaks into every other file sharing its process). `bun run verify` gets its own job too. CI is the ground truth for "did everything pass."
@@ -46,6 +72,15 @@ If a shard wedges (per-shard `GBRAIN_TEST_SHARD_TIMEOUT` cap, default 600s), the
- `tests/heavy/*.sh` → ops-shape shell scripts. Cost minutes per run; NOT in default `bun test`. Run via `bun run test:heavy` or scheduled nightly via `.github/workflows/heavy-tests.yml`. Examples: pg_upgrade matrix (boot legacy brain → walk to head), RSS budget gate (measure peak worker RSS vs committed baseline), read-latency-under-sync (p50/p95/p99 under concurrent writer load), sync lock regression (N concurrent syncs assert 1 winner + N-1 lock-busy + zero leaked `gbrain_cycle_locks` rows). See `tests/heavy/README.md` for when to add a script here vs `*.slow.test.ts`. Files prefixed with `_` (e.g. `tests/heavy/_build_legacy_fixtures.sh`) are helpers/libs invoked by sibling tests — the runner skips them.
- `test/fuzz/*.test.ts` → property-based fuzz harness. Pure-validator targets in `pure-validators.test.ts` are guarded by `scripts/check-fuzz-purity.sh` (in `bun run verify`), which `bun build --target=bun` bundles each target and greps the resulting bundle for banned transitive imports (`node:fs`, `node:child_process`, engine modules). Anything that fails the guard moves to `mixed-validators.test.ts` (still property-tested, but no purity guarantee) or `filesystem-validators.test.ts` (fs-backed, uses temp dirs). Fuzz tests run in the default `bun test` loop because they're fast (~3s for ~12 properties × 1000 runs each).
### Skills-manifest freshness guard
`skills/skills.lock.json` is a committed sha256 inventory of every bundled file under
`skills/` (tamper evidence, not signatures — see `src/core/skills-integrity.ts`).
Any change under `skills/` must regenerate it: `bun run scripts/generate-skills-manifest.ts`.
`scripts/check-skills-manifest-fresh.sh` (`bun run check:skills-manifest`, wired into
`bun run verify`) regenerates to a tmp file and diffs, failing CI on drift; at runtime
`gbrain doctor` reports the same drift as a warn-only `skills_manifest_integrity` check.
### Test-isolation lint and helpers
The cross-file flake class is enforced statically by `scripts/check-test-isolation.sh`, wired into `bun run verify` and `bun run check:all`. Rules (non-serial unit files only; `*.serial.test.ts` and `test/e2e/*` are skipped):
@@ -189,8 +224,10 @@ Unit tests and what they cover:
- `test/postgres-engine.test.ts` — `statement_timeout` scoping: `sql.begin` + `SET LOCAL` shape, source-level grep guardrail against a reintroduced bare `SET statement_timeout`.
- `test/sync.test.ts` — sync logic + regression guard asserting top-level `engine.transaction` is not called.
- `test/sync-pull-failed-anchor.serial.test.ts` — #3068 regression: a failed internal `git pull` (local-path origin vs `protocol.file.allow=never`) with zero imports returns `partial`/`pull_failed` (not `up_to_date`), freezes `last_commit` + `last_sync_at`, recovers after a manual pull; fall-through import of local commits preserved. Serial: pins `GBRAIN_HOME` to a temp dir for the whole file.
- `test/sync-parallel.test.ts` — PGLite-routed coverage of the bookmark gate under concurrency, head-drift gate, vanished-file failure capture, PGLite-stays-serial, and the `gbrain-sync` writer-lock contract.
- `test/sync-all-missing-path.test.ts` — `sync --all --missing-path <fail|skip>` pure helpers: `parseMissingPathMode` (default fail, explicit values, loud rejection of bad/dangling values, never swallows a following flag) and `partitionMissingPathSources` (classification driven only by the injected pathExists predicate — no fs; null `local_path` passes through runnable; order preserved).
- `test/sync-failures.test.ts` — `classifyErrorCode` regex coverage for all 12 codes against literal production message strings from `markdown.ts` and `import-file.ts`; `summarizeFailuresByCode` sort + pre-classified-honor; `recordSyncFailures` code-field persistence; `acknowledgeSyncFailures``AcknowledgeResult` shape + backfill on legacy entries.
- `test/doctor.test.ts` — doctor command; assertions that `jsonb_integrity` scans the four JSONB write sites and `markdown_body_completeness` is present.
@@ -108,8 +109,8 @@ Every primitive ships with a documented rollback:
| Operation | Rollback |
|-----------|----------|
| Retype | `frontmatter.legacy_type = <original>` preserved on every page (D8). One SQL UPDATE restores types: `UPDATE pages SET type = frontmatter->>'legacy_type' WHERE frontmatter ? 'legacy_type'`. |
| Page-to-link | Source page soft-deleted with 72h TTL. `gbrain pages restore <slug>` within 72h. Link row stays harmless if source restored. |
| Page-to-alias | Source page soft-deleted with 72h TTL. `gbrain pages restore <slug>` within 72h. Alias row stays harmless (or `DELETE FROM slug_aliases WHERE alias_slug = <slug>` to clean up). |
| Page-to-link | Source page soft-deleted with 72h TTL. `gbrain restore <slug>` within 72h. Link row stays harmless if source restored. |
| Page-to-alias | Source page soft-deleted with 72h TTL. `gbrain restore <slug>` within 72h. Alias row stays harmless (or `DELETE FROM slug_aliases WHERE alias_slug = <slug>` to clean up). |
| Active-pack flip | `gbrain schema use gbrain-base` reverses the flip. |
## What if my brain doesn't fit?
@@ -127,7 +128,8 @@ For brains with substantial custom types that deserve their own canonical
2. Edit your fork to add page_types + mapping_rules covering your
custom domain.
3. Target your fork: `gbrain jobs submit unify-types --allow-protected
@@ -183,6 +183,6 @@ This also means the best AI agent setups will be open source by default. Closed,
Software distribution reimagined: the package is a markdown file, the runtime is a sufficiently smart model, the package manager is your AI agent, and the app store is a git repo.
@@ -46,10 +46,10 @@ on user_shares_media(url_or_file):
# Step 4: Extract and cross-reference entities
for person in transcript.mentioned_people:
gbrain add_link <slug> <person_slug>
gbrain add_link <person_slug> <slug>
gbrain add_timeline_entry <person_slug> \
--entry "Discussed in {video_title}: {what_was_said}" \
gbrain link <slug> <person_slug>
gbrain link <person_slug> <slug>
gbrain timeline-add <person_slug> {date} \
"Discussed in {video_title}: {what_was_said}" \
--source "YouTube: {url}"
# PATTERN 2: Social Media Bundles
@@ -80,8 +80,8 @@ on user_shares_media(url_or_file):
# Extract entities and cross-reference
for entity in bundle.mentioned_entities:
gbrain add_link <slug> <entity_slug>
gbrain add_link <entity_slug> <slug>
gbrain link <slug> <entity_slug>
gbrain link <entity_slug> <slug>
# PATTERN 3: PDFs and Documents
elif media.type == "pdf" or media.type == "document":
@@ -109,8 +109,8 @@ on user_shares_media(url_or_file):
"""
for entity in document.mentioned_entities:
gbrain add_link <slug> <entity_slug>
gbrain add_link <entity_slug> <slug>
gbrain link <slug> <entity_slug>
gbrain link <entity_slug> <slug>
# Always sync after ingestion
gbrain sync
@@ -127,7 +127,7 @@ on user_shares_media(url_or_file):
## How to Verify
1. Ingest a YouTube video. Run `gbrain get media/youtube/{slug}`. Confirm the page has: the agent's analysis (not just a summary), key quotes with speaker attribution, and the full diarized transcript.
2. Run `gbrain get_links media/youtube/{slug}`. Confirm back-links exist to brain pages for every person and company mentioned in the video.
2. Run `gbrain call get_links '{"slug": "media/youtube/{slug}"}'`. Confirm back-links exist to brain pages for every person and company mentioned in the video.
3. Pick a person mentioned in the video. Run `gbrain get <person_slug>`. Confirm their timeline has a new entry referencing the video with specific context.
4. Ingest a tweet. Confirm the brain page includes the thread context, linked article summaries, and entity cross-references -- not just the tweet text.
5. Run `gbrain search "{topic_from_video}"`. Confirm the media page appears in search results (verifies the content is indexed and searchable).
# Flag contradictions -- don't silently resolve them
# Step 7: Cross-reference the graph
gbrain add_link <person_slug> <company_slug> # person -> company
gbrain add_link <company_slug> <person_slug> # company -> person
gbrain add_link <person_slug> <deal_slug> # person -> deal
gbrain link <person_slug> <company_slug> # person -> company
gbrain link <company_slug> <person_slug> # company -> person
gbrain link <person_slug> <deal_slug> # person -> deal
# Every entity page links to every other entity page that references it
# People page sections (not a LinkedIn profile -- a living portrait):
@@ -94,8 +94,8 @@ on enrich(entity, trigger):
## How to Verify
1. Enrich a Tier 1 person. Run `gbrain get <slug>` and confirm the page has Executive Summary, State, What They Believe, Contact, and Timeline sections populated from multiple sources.
2. Run `gbrain get_raw_data <slug>`. Confirm raw API responses are stored with `sources.{provider}.fetched_at` timestamps.
3. Run `gbrain get_links <slug>`. Confirm cross-reference links exist to the person's company page, deal pages, and related entities.
2. Run `gbrain call get_raw_data '{"slug": "<slug>"}'`. Confirm raw API responses are stored with `sources.{provider}.fetched_at` timestamps.
3. Run `gbrain call get_links '{"slug": "<slug>"}'`. Confirm cross-reference links exist to the person's company page, deal pages, and related entities.
4. Check a page that was enriched AND has a user-written Assessment. Confirm the Assessment section was preserved, not overwritten by API data.
5. Try to re-enrich the same person. Confirm the system checks the `fetched_at` timestamp and skips if less than a week old.
gbrain link <slug> <entity_slug> # meeting -> entity
gbrain link <entity_slug> <slug> # entity -> meeting
# Step 6: Sync so new pages are immediately searchable
gbrain sync
@@ -73,7 +73,7 @@ on new_meeting_transcript(meeting):
1. After ingesting a meeting, run `gbrain get meetings/{date}-{slug}`. Confirm the page has the agent's analysis above the bar and the full diarized transcript below it.
2. For each attendee, run `gbrain get <attendee_slug>`. Check that their timeline has a new entry referencing the meeting with specific insights (not just "attended meeting").
3. Pick a company mentioned in the meeting. Run `gbrain get <company_slug>`. Confirm a timeline entry exists referencing what was discussed about the company.
4. Run `gbrain get_links meetings/{date}-{slug}`. Verify back-links exist to all attendee and entity pages.
4. Run `gbrain call get_links '{"slug": "meetings/{date}-{slug}"}'`. Verify back-links exist to all attendee and entity pages.
5. Run `gbrain search "{meeting_topic}"`. Confirm the meeting page appears in search results (verifies sync ran).
@@ -110,11 +110,11 @@ on nightly_schedule("02:00"):
## How to Verify
1. Send a message mentioning a person with a brain page. Confirm the agent detects the entity and adds a timeline entry to their page (`gbrain get_timeline <slug>`).
1. Send a message mentioning a person with a brain page. Confirm the agent detects the entity and adds a timeline entry to their page (`gbrain timeline <slug>`).
2. Ask the agent about someone in the brain. Confirm it runs `gbrain search` or `gbrain get` BEFORE reaching for external APIs (check the tool call order).
3. Write a new page with `gbrain put`, then immediately run `gbrain search` for it. Confirm it appears in results (verifies sync ran).
4. Run `gbrain doctor`. Confirm it returns a health report with database status, page count, and any flagged issues.
5. After a dream cycle runs, check a page that had unlinked entity mentions. Confirm new links were added (`gbrain get_links <slug>`).
5. After a dream cycle runs, check a page that had unlinked entity mentions. Confirm new links were added (`gbrain call get_links '{"slug": "<slug>"}'`).
---
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*
# Step 3: Cross-link to everything that shaped the thinking
for entity in idea.influences:
gbrain add_link originals/{slug} <entity_slug>
gbrain add_link <entity_slug> originals/{slug}
gbrain link originals/{slug} <entity_slug>
gbrain link <entity_slug> originals/{slug}
# Step 4: Sync
gbrain sync
@@ -79,7 +79,7 @@ on user_message(message):
1. Generate an original idea in conversation (e.g., "I call this the 'ambition debt' problem -- every year you delay going big, the compound interest works against you"). Confirm a new page appears at `brain/originals/ambition-debt` with `gbrain get originals/ambition-debt`.
2. Check that the page uses the user's exact phrasing for the title and slug -- not a sanitized version.
3. Run `gbrain get_links originals/ambition-debt`. Confirm cross-links exist to related people, meetings, or other originals.
3. Run `gbrain call get_links '{"slug": "originals/ambition-debt"}'`. Confirm cross-links exist to related people, meetings, or other originals.
4. Express a take on someone else's idea (e.g., "I think Thiel's contrarian question is wrong because..."). Confirm it goes to `originals/` (synthesis is original), not `concepts/`.
5. Run `gbrain search "ambition debt"`. Confirm the originals page appears in search results and is discoverable.
| [qm Harness](qm-harness.md) | gbrain as the company brain for a qm (multi-user agent harness) deployment — central HTTP MCP, per-scope clients, roster provisioning, write fencing |
## How to Read a Recipe
@@ -69,6 +70,12 @@ health_checks: # typed DSL to verify the integration is working
@@ -103,7 +103,7 @@ For GCP service-account / Vertex AI auth (production deployments), see the v0.32
### OpenRouter
Single OpenAI-compatible API for fan-out to OpenAI, Anthropic, Google, DeepSeek, Meta Llama, Qwen, and dozens of other hosted providers. One key, many models. Set `OPENROUTER_API_KEY`and use `openrouter:<provider>/<model>` (e.g. `openrouter:openai/gpt-5.2`, `openrouter:anthropic/claude-sonnet-4.6`).
Single OpenAI-compatible API for fan-out to OpenAI, Anthropic, Google, DeepSeek, Meta Llama, Qwen, and dozens of other hosted providers. One key, many models. Set `OPENROUTER_API_KEY`or `openrouter_api_key` in `~/.gbrain/config.json`, then use `openrouter:<provider>/<model>` (e.g. `openrouter:openai/gpt-5.2`, `openrouter:anthropic/claude-sonnet-4.6`).
**Embedding**: `openai/text-embedding-3-small` (1536d default, Matryoshka shrink to 512/768/1024). OR's embedding catalog also includes `text-embedding-3-large`, `google/gemini-embedding-2-preview`, `qwen/qwen3-embedding-8b`, `bge-m3` — opt in via `--embedding-model openrouter:<id>`. Pricing matches the upstream provider (OR adds a small markup).
description: Search and write the company knowledge brain. Use for any question about the org, people, projects, decisions, or history, and to persist durable knowledge beyond this scope's notebook.
---
# gbrain — the company brain
This sandbox has the `gbrain` CLI connected (thin-client) to the org's central
brain. It is the deep, indexed, cross-source memory: org docs, shared channel
knowledge, and every agent's durable notes. Your scope's own notebook stays the
fast per-turn memory; the brain is where knowledge outlives a scope and becomes
searchable by everyone entitled to it.
## First-run setup (once per sandbox — skip if `gbrain remote doctor` passes)
Your scope's brain credentials arrive via the deployment's secret handoff
(keychain entry or one-time secret drop named `gbrain`). Then:
```bash
gbrain init --mcp-only \
--issuer-url "https://brain.<org>.com" \
--mcp-url "https://brain.<org>.com/mcp" \
--oauth-client-id "<client id from the handoff>" \
--oauth-client-secret "<client secret from the handoff>"
gbrain whoami # must succeed before using any other command
```
Pass the secret with `--oauth-client-secret`, not via `GBRAIN_REMOTE_CLIENT_SECRET`:
an env-sourced secret is deliberately NOT written to `~/.gbrain/config.json`, so
every later command would fail with "No client_secret available" once the
variable is out of scope. The flag persists it to the config file on this
sandbox's durable disk, which is what the tool's credential capture expects.
Do not run `gbrain remote doctor` — it needs `admin` scope, which your client
does not have (by design). `gbrain whoami` is the read-scope health check.
## Reading (do this liberally)
```bash
gbrain search "who decided X and why" # hybrid semantic + keyword search
gbrain get <slug> # read one page
gbrain query "question" --json # search tuned for agent consumption
```
You can read: the shared agent-memory source, org read-only sources (wiki,
handbook), and everything under them. Reads are isolation-enforced server-side;
you only ever see sources your client is entitled to.
## Writing (durable knowledge only, under YOUR prefixes)
Your client is write-fenced to slug prefixes — your own namespace plus the
channels you belong to. Writes outside them are rejected server-side.
```bash
# personal durable memory (your namespace):
gbrain put emp-<your-slug>/people/jane-example --content "..."
# shared channel knowledge (channels you are in):
gbrain put chan-eng/decisions/2026-08-database-choice --content "..."
```
Conventions:
- Write conclusions and durable facts, not chat transcripts. One page per
entity/decision/topic; update the page rather than appending near-duplicates.
- Markdown with YAML frontmatter; the brain chunks, embeds, and links it.
- Cross-reference liberally: `gbrain link <from> <to>` (from must be in your
namespace; linking TO any readable page is fine).
- When you learn something channel-relevant in personal work, mirror the
conclusion into the channel prefix with a `(said in <where>)` provenance
note.
## When to reach for the brain
- Any question about the org, a person, a project, a decision, or history →
`gbrain search` FIRST, then answer.
- You produced knowledge with value beyond this conversation → `gbrain put`.
line="${line%$'\r'}"# a CRLF roster would otherwise yield 'emp-alice\r/' prefixes that fence everything out
[ -z "${line//[[:space:]]/}"]&&continue
# shellcheck disable=SC2086 — deliberate word split; globbing is off (set -f above)
set -- $line
["$#" -le 3]|| die "roster line $lineno: too many fields ('$line'). Channels are ONE comma-separated field with no spaces: 'employee alice eng,product'"
case"$1" in
channel)
require_slug channel "${2:-}"
CHANNELS="$CHANNELS$2"
;;
employee)
require_slug employee "${2:-}"
case"$EMPLOYEES" in *"$2:"*) die "roster line $lineno: employee '$2' listed twice";;esac
if[ -n "${3:-}"];then
for c in ${3//,/ };do require_slug "channel-reference""$c";done
fi
EMPLOYEES="$EMPLOYEES$2:${3:-}"
;;
*) die "roster line $lineno: unknown entry type '$1' (expected 'channel' or 'employee')";;
|| die "register-client failed for '$slug' (output withheld: it can contain a secret). Re-run the command by hand to see why."
client_id=$(echo"$out"| sed -n 's/.*Client ID:[[:space:]]*\(gbrain_cl_[^[:space:]]*\).*/\1/p'| head -1)
secret=$(echo"$out"| sed -n 's/.*Client Secret:[[:space:]]*\(gbrain_cs_[^[:space:]]*\).*/\1/p'| head -1)
if[ -z "$client_id"]||[ -z "$secret"];then
# The client may well have been created — dying silently would strand a
# live credential nobody can find. Say so WITHOUT echoing the captured
# output: it contains the freshly minted secret, and this path ends up
# in CI logs.
die "could not parse client id/secret for '$slug' from register-client output (output withheld: it contains a secret). A client MAY have been created; check \`gbrain auth list\` and revoke any stray 'qm-emp-$slug'."
`gbrain extract-conversation-facts` stores page-level outcomes in `facts` so
bulk runs, autopilot, and `gbrain doctor` can distinguish finished work from
retryable work without adding another state table.
This is completion authority, not ordinary extracted knowledge. The authority
is deliberately narrow: a marker is valid only for the exact page or transcript
snapshot that was parsed, and only after every required operation succeeded.
## Outcome protocol
The current protocol is v2. Its source names are versioned so rows written by
older best-effort implementations cannot suppress a corrective replay.
| Outcome | `facts.source` | Meaning |
|---|---|---|
| Complete | `cli:extract-conversation-facts:terminal:v2` | Every eligible segment was extracted and inserted successfully, the input remained unchanged, and the terminal write succeeded. |
| Scanned, not extractable | `cli:extract-conversation-facts:non-extractable:v2` | A recognized input was scanned successfully but contained no eligible multi-message segment. |
| Unfinished | no matching v2 outcome | Work is pending, failed, was not recognized, changed during extraction, or has only a legacy marker. |
The non-extractable outcome is intentionally separate from completion. It does
not claim that knowledge facts were extracted. CLI counters, cycle details, and
doctor output preserve that distinction.
## Snapshot identity
Every v2 marker binds `source_session` to the parser input snapshot:
```text
<outcome-source>:<page-slug>:<version-token>
```
There are two token forms.
### Database-backed page body
For pages parsed from `compiled_truth` and `timeline`, the token is:
```text
page-<pages.content_hash>-<effective-date>
```
`content_hash` covers title, type, compiled truth, timeline, and frontmatter.
The effective-date suffix covers the remaining date input used by parsing. This
identity does not depend on JavaScript's millisecond timestamp precision, so two
writes within one PostgreSQL millisecond still produce different tokens when
parser input changes. A legacy page with a null content hash uses a computed
SHA-256 fallback and is verified in-process by both extraction and doctor.
### Raw transcript sidecar
When frontmatter contains `raw_transcript`, the source text lives outside the
page row and may change without changing `pages.updated_at`. Its token is:
```text
sidecar-<SHA-256>
```
The digest covers the exact body given to the parser plus parser-relevant page
metadata: title, type, frontmatter, and effective date. Selection recomputes
the digest before skipping work. A sidecar-only edit therefore reopens the page.
`gbrain doctor` cannot read sidecars in its SQL aggregate, so it enumerates those
pages in bounded batches and calls the same canonical verifier used by
extraction. Doctor and extraction therefore agree after sidecar-only edits.
## Selection and locking
Bulk extraction follows this sequence:
1. Enumerate candidate pages in bounded batches.
2. Filter candidates with matching v2 outcomes.
3. Apply `--limit` to the remaining pages that actually need work.
4. Acquire the source-and-slug advisory lock.
5. Re-fetch the page under that lock.
6. Recompute and recheck the snapshot-bound outcome.
7. Prepare one immutable parser snapshot and process it.
8. Re-fetch and recompute the snapshot before writing an outcome.
The pre-lock check avoids parser, filesystem, and model work for ordinary
completed pages. The under-lock refetch prevents a stale enumeration object
from becoming the certified input. The final comparison prevents an edit that
happens during model or insertion work from receiving a marker for old content.
An edit can occur after the final comparison and before marker insertion. That
is still safe because the marker contains the old version token. Future
selection compares the token, not marker creation time, and reopens the page.
Single-page `--slug` runs use the same under-lock path.
## Strict extraction success
The general `extractFactsFromTurn` API remains best-effort for interactive
callers. It historically returns an empty array for both a legitimate zero-fact
@@ -51,6 +51,18 @@ When storage configuration is present, `gbrain sync` automatically manages `.git
- Skipped when the repo is a git submodule (`.git` is a file, not a directory) — submodule .gitignore changes don't survive parent updates. A warning explains.
- Skipped entirely when `GBRAIN_NO_GITIGNORE=1` is set (escape hatch for shared-repo setups where a maintainer wants gbrain to leave .gitignore alone).
- Failures (write permission denied, etc.) are caught and logged, never crash sync.
- Warns when a configured collector's declared output dir (recipe `output_paths`
frontmatter) sits inside a `db_only` path: gitignored files never appear in the
git-walking sync diff, and `gbrain import` honors `.gitignore` too — the
collector would run green while nothing reaches the DB. The
`db_only_collector_collision` doctor check surfaces the same trap.
Related doctor coverage: `undeclared_db_only_pages` warns about DB pages with no
backing file that sit outside every declared `db_only` path. The engine's own
# Engine Dynamic-Import Reconciliation Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Reconstruct the missing engine-path static-import hardening, preserve the four load-bearing lazy gateway fallbacks, and prevent unreviewed dynamic imports from returning.
**Architecture:** Make the 13 safe engine/migration import statements static and leave only four line-marked `ai/gateway.ts` imports inside their existing soft-failure `try/catch` boundaries. Enforce that current state with a repository-anchored Bash wrapper delegating to a fail-closed TypeScript AST scanner, a hermetic Bun regression test, package/verify wiring, and current-state architecture documentation.
**Tech Stack:** TypeScript compiler API, Bun test runner, Bash, Git, generated llms documentation bundles.
## Global Constraints
- Reconstruct directly on branch `claude/kind-meitner-330c90`, based on investigated `origin/master` commit `6136e139972a5449630b4f47f5ed7b4cbe5b811b` plus design commit `d7f52d8c`.
- Do not merge or cherry-pick `48ada48f`, `248bfe55`, `ef4cf7a8`, or either historical branch wholesale.
- Do not modify `VERSION`, `CHANGELOG.md`, `TODOS.md`, or release metadata; this is a no-version-bump reconciliation.
- Keep all four `await import('./ai/gateway.ts')` calls lazy: PGLite and Postgres `initSchema`, plus both `_upsertChunksOnce` methods.
- Every allowed lazy gateway line must carry `engine-dynamic-import-ok`; there is no file-level exemption.
- Preserve the stronger gateway rationale: the static closure is large, and eager module evaluation would occur outside the local `try/catch`, potentially converting a recoverable configuration/import failure into a module-load-time hard failure.
- Describe the hoists as engine-path hardening. Do not claim every dynamic import deterministically causes a Windows crash; system-wide commit exhaustion confounded prior measurements.
- Keep shared PGLite/Postgres behavior in parity.
- Invoke repository shell scripts through `bash` in `package.json`.
- Capture complete test/check output to workspace-local `.context/*.txt` files before inspecting it; never pipe a test command directly through `head` or `tail`.
- Use `git log -G`, not `git log -S`, for any additional dynamic-to-static import history work.
- Keep every implementation and verification commit local. Do not push, create a PR, comment upstream, or otherwise publish without explicit user approval after local completion.
- Before editing any affected function, run GBrain `code_blast` and `code_callers` for that symbol and inspect any disambiguation candidates.
---
## File Map
- Create `scripts/check-engine-dynamic-import.sh` — repository-anchored Bash wrapper for default and explicit input routing.
- Create `scripts/check-engine-dynamic-import.ts` — TypeScript AST policy scanner for runtime `import()` expressions, parse/read failures, and exact-line comment-trivia opt-outs.
- Consumes: shell positional arguments `FILE...`; without arguments, the guard scans the three repository files.
- Produces: `scripts/check-engine-dynamic-import.sh [FILE...]`, exit `0` when every runtime dynamic import is allowed and exit `1` after reporting every `file:line:text` violation plus every read/parse error on stderr.
- Produces: one line-level opt-out token, `engine-dynamic-import-ok`, accepted only in real comment trivia on the same physical line as the deliberately lazy import.
- Fails closed on missing/unreadable inputs, TypeScript parse diagnostics, and scanner/process failures; comments, strings, templates, regex literals, and type-position `import(...)` syntax are not runtime imports.
- [ ] **Step 1: Record call-graph blast radius before touching functions**
First call `sources_list` and select the source whose registered path is this gbrain checkout. Then run `code_blast` and `code_callers` for these qualified symbols with that exact `source_id`, following `did_you_mean`/`candidates` when a method name is ambiguous:
Use `depth: 5`, `max_nodes: 200`, and `limit: 100`. Expected: no caller requires a signature or behavior change; the patch only changes module binding time and retains all local fallback/error handling.
Create `test/scripts/check-engine-dynamic-import.test.ts` as a hermetic subprocess suite. The completed 22-test surface covers:
- unmarked runtime `import()` rejection, including bare and trivia-separated forms;
- same-line markers in real line or multiline block-comment trivia;
- rejection of markers on prior lines or inside strings, templates, and module paths;
- comments and comment-like delimiters inside strings, templates, and regex literals;
- live code after same-line or multiline block comments close;
- CRLF input and complete multi-file violation aggregation;
- missing/readable mixed inputs and TypeScript parse diagnostics;
- default repository anchoring when invoked from a foreign Git repository;
- the reconciled three-file source scan plus package/parallel-verifier wiring.
Use the TypeScript parser rather than a partial lexical reimplementation. On Windows, set the test default to 30 seconds because each case launches Git Bash and Bun, whose startup can exceed Bun's 5-second per-test default.
- [ ] **Step 3: Run the test to prove the pre-implementation red state**
```bash
bun test test/scripts/check-engine-dynamic-import.test.ts > .context/engine-dynamic-import-red.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit 0
```
Expected: non-zero Bun result captured inside the log. At minimum, the `exists` assertion fails because `scripts/check-engine-dynamic-import.sh` does not exist. Read `.context/engine-dynamic-import-red.txt`; do not infer the result from a truncated pipeline.
- [ ] **Step 4: Add the CRLF-safe, fail-closed guard**
Create `scripts/check-engine-dynamic-import.sh` as a thin LF-terminated wrapper. Resolve its own directory first; when no explicit files are passed, anchor the repository with `git -C "$SCRIPT_DIR/.."` and scan the two engines plus `migrate.ts`. Delegate with `exec bun "$SCRIPT_DIR/check-engine-dynamic-import.ts" "${FILES[@]}"` so scanner failures propagate.
Create `scripts/check-engine-dynamic-import.ts` using the TypeScript compiler API:
- read every requested file and aggregate read failures;
- parse as TypeScript and aggregate parse diagnostics;
- walk the AST for `CallExpression`s whose expression is `ImportKeyword`;
- locate all marker occurrences in the full source and use `ts.getTokenAtPosition` to admit only occurrences outside AST tokens (real comment trivia), recording their physical source lines;
- require each runtime import's line to have an admitted marker or report its original `file:line:text`;
- print every read/parse error and every violation before exiting nonzero.
This preserves CRLF line accounting, ignores comment/literal/type-only false positives, catches every legal runtime `import()` shape the TypeScript parser recognizes, rejects marker spoofing, and fails closed.
- [ ] **Step 5: Run the guard test to prove the source-tree midpoint is still red**
```bash
bun test test/scripts/check-engine-dynamic-import.test.ts > .context/engine-dynamic-import-midpoint.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit 0
```
Expected: the synthetic violation, marker, comments, and CRLF cases pass. The default repository scan fails and reports all 17 current imports: 13 unmarked safe candidates plus the four not-yet-marked gateway calls.
- [ ] **Step 6: Hoist the three safe PGLite import statements**
Replace the existing `retry.ts` import and add the ontology/recency imports near the top of `src/core/pglite-engine.ts`:
```ts
// Engine-path imports stay static unless a call site carries an explicit
// engine-dynamic-import-ok justification. The gateway is the only current
// exception because its local try/catch preserves a soft fallback.
import {
withRetry,
BULK_RETRY_OPTS,
resolveBulkRetryOpts,
computeNextDelay,
isRetryableConnError,
type BatchAuditSite,
} from './retry.ts';
import {
valueHash,
normalizeDimension,
isNovelDimension,
} from './chronicle/ontology.ts';
import {
resolveRecencyDecayMap,
DEFAULT_FALLBACK,
} from './search/recency-decay.ts';
```
Delete only these three in-method destructuring imports, leaving their uses unchanged:
Expected: exactly four lines, all importing `./ai/gateway.ts` and all carrying `engine-dynamic-import-ok`; no match in `src/core/migrate.ts`.
- [ ] **Step 13: Run focused behavior tests**
```bash
bun test test/chronicle-ontology.test.ts test/chronicle-ontology-ops.test.ts test/recency-decay.test.ts test/core/retry.test.ts test/retry-matcher.test.ts test/audit/pool-recovery-audit.test.ts test/migrate-retry.test.ts test/timeline-dedup-repair.test.ts > .context/engine-dynamic-import-focused.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`. If Windows resource pressure aborts the process, record the exact exit code and rerun the failing file alone; do not relabel an infrastructure abort as a source pass.
- [ ] **Step 14: Commit the source invariant locally**
Expected: one local commit with the guard wiring and its regression assertions.
---
### Task 3: Document the current-state invariant
**Files:**
- Modify: `CLAUDE.md`
- Modify: `docs/architecture/KEY_FILES.md`
- Regenerate: `llms.txt`
- Regenerate: `llms-full.txt`
**Interfaces:**
- Consumes: the four-marked-import source state and the `check:engine-dynamic-import` package surface.
- Produces: current-state contributor guidance and fresh generated documentation bundles.
- [ ] **Step 1: Add the cross-cutting invariant to `CLAUDE.md`**
Add this bullet under “Cross-cutting invariants” near the other language/filesystem guards:
```md
- **Engine-live paths use static imports by default.** In
`src/core/pglite-engine.ts`, `src/core/postgres-engine.ts`, and
`src/core/migrate.ts`, helper modules are top-level imports. The only current
exceptions are the four `ai/gateway.ts` lookups in both engines'
`initSchema()` and `_upsertChunksOnce()` methods; each remains lazy inside a
local `try/catch` because the gateway has a large provider/config closure and,
more importantly, eager evaluation would occur before the catch and could
turn a recoverable default/config-row fallback into a module-load failure.
Every exception carries `engine-dynamic-import-ok` on the import line.
`scripts/check-engine-dynamic-import.sh` enforces the rule. For history, use
`git log -G'await[[:space:]]+import\\('`, not `git log -S`: a dynamic-to-static
rewrite can preserve the searched token while changing its context.
```
Do not add release tags, Windows-crash certainty, or historical branch names.
- [ ] **Step 2: Update the PGLite current-state entry in `KEY_FILES.md`**
Append this current-state sentence to the existing `src/core/pglite-engine.ts` entry, preserving the entry as one bullet:
```md
Engine-path helper dependencies (`retry`, ontology, recency decay) bind statically; the only lazy imports are `ai/gateway.ts` in `initSchema` and `_upsertChunksOnce`, line-marked because their local catches preserve compiled-default and stored-config fallbacks that eager module evaluation would bypass.
```
- [ ] **Step 3: Update the Postgres current-state entry in `KEY_FILES.md`**
Append this sentence to the existing `src/core/postgres-engine.ts` entry:
```md
Retry classifiers, ontology/recency helpers, and disconnect/pool-recovery audit writers bind statically; only the two `ai/gateway.ts` fallback lookups stay lazy and line-marked, in parity with PGLite.
```
- [ ] **Step 4: Update the migration current-state entry in `KEY_FILES.md`**
Append this sentence to the canonical `src/core/migrate.ts` entry (the broad runner entry, not the older v95-specific index note):
```md
`retry-matcher.ts` and `timeline-dedup-repair.ts` are static dependencies because `runMigrations()` executes from live engine initialization; the engine dynamic-import guard scans this file with both engine implementations.
```
Keep all three entries current-state only: no `v0.42.x`, branch, commit, “previously,” or “was/now” narration.
- [ ] **Step 5: Regenerate the llms bundles**
```bash
bun run build:llms > .context/engine-dynamic-import-build-llms.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`; `llms.txt` and/or `llms-full.txt` update according to their configured linked/inlined status. Byte-identical output for a linked source is acceptable; the freshness test is authoritative.
- [ ] **Step 6: Run documentation freshness checks**
```bash
bun test test/build-llms.test.ts > .context/engine-dynamic-import-llms-test.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`.
```bash
bun run check:doc-history > .context/engine-dynamic-import-doc-history.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`; no release-history marker is introduced into current-state reference docs.
bun run typecheck > .context/engine-dynamic-import-typecheck.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`. Report exact diagnostics if the branch or current Windows environment has a pre-existing failure.
- [ ] **Step 3: Run the authoritative verify dispatcher**
```bash
bun run verify > .context/engine-dynamic-import-verify.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`, including `check:engine-dynamic-import`. On Windows, classify any per-check timeout from the complete log instead of treating the aggregate result as a source regression without evidence.
- [ ] **Step 4: Re-run focused tests as an ownership check**
```bash
bun test test/chronicle-ontology.test.ts test/chronicle-ontology-ops.test.ts test/recency-decay.test.ts test/core/retry.test.ts test/retry-matcher.test.ts test/audit/pool-recovery-audit.test.ts test/migrate-retry.test.ts test/timeline-dedup-repair.test.ts > .context/engine-dynamic-import-final-focused.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`; record any infrastructure abort separately and rerun only the named file before classifying it.
- [ ] **Step 5: Run the llms freshness test after all documentation settles**
```bash
bun test test/build-llms.test.ts > .context/engine-dynamic-import-final-llms.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
- Exactly 13 safe `await import(...)` statements are removed.
- Exactly four `ai/gateway.ts` imports remain, all marked on the same line.
- All four gateway imports remain inside their original local `try/catch` fallback boundaries.
- No accessor logic, fallback ordering, SQL, public signature, or engine parity behavior changes.
- The parser-backed guard reports all violations plus read/parse failures, preserves CRLF line accounting, ignores comments/literals/type-only syntax, detects every runtime `import()` call expression, and accepts opt-outs only from real comment trivia on the same physical line.
- The package script invokes the shell guard through Bash; `check:all` invokes that shell guard directly, and the parallel verify dispatcher invokes the package check.
- Documentation is current-state and makes no deterministic Windows-crash claim.
**Observed Windows verification classification:** The authoritative aggregate completed with 25 of 33 checks passing. Individual reruns showed `check:test-names` and `typecheck` green; privacy/isolation exceeded Windows timing budgets; WASM failed in unrelated temporary-symlink setup; eval-glossary was CRLF/LF drift; resolver/brain-first findings predated and did not intersect this branch. The focused aggregate produced 103 pass / 5 fail: three setup-hook timeouts reproduced at the untouched base, and the known `migrate-retry` polling failure reproduced there. Its additional race-status assertion did not reproduce at base, so it remains an unresolved timing-sensitive limitation in untouched code—not evidence of an in-scope defect and not claimed as conclusively pre-existing.
- [ ] **Step 8: Commit the approved plan document locally**
The plan is an approved, tracked execution artifact and must not be left as an uncommitted file after implementation:
git commit -m "docs: plan engine dynamic-import reconciliation"
```
Expected: one local plan commit; no release metadata staged.
- [ ] **Step 9: Inspect final status without publishing**
```bash
git status --short --branch
```
Expected: branch `claude/kind-meitner-330c90` with a clean working tree. No push, PR, upstream comment, or other external side effect.
- [ ] **Step 10: Capture the completed milestone to memory**
Before writing, search MemPalace wing `gbrain` for this exact reconciliation to avoid duplication. Add a verbatim drawer recording exact base/head commits, the 13 hoists, four gateway opt-outs and rationale, guard/test/docs files, every verification command with exit code, and any environment-owned failures. Add a GBrain project timeline entry only if there is an existing relevant gbrain project page; do not create duplicate release metadata.
- [ ] **Step 11: Report the local result and ask separately before publication**
Report:
- exact local commits;
- changed files;
- test/check exit codes;
- any blocked or pre-existing failures;
- confirmation that release files were untouched;
- confirmation that nothing was pushed or published.
Do not run any publication command. Wait for explicit user approval before any push, PR, or upstream interaction.
# Scalar-source Backlink Validation Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Make backlink validation compare exact `(source_id, slug)` endpoint identities while preserving existing scalar, unscoped, and federated link-read semantics.
**Architecture:** Enrich every engine link-read row with the source identity of its joined from, to, and visible origin pages. Pass the validated page's scalar or federated scope into validator context; the backlink validator scopes its initial read consistently, groups targets by exact identity, and accepts only an exact reverse endpoint pair. SQL predicates remain unchanged, so trusted scalar cross-source visibility and federated all-endpoint containment remain intact.
**Tech Stack:** TypeScript, Bun test, PGLite, PostgreSQL/postgres.js.
## Global Constraints
- Use strict red-before-green TDD with duplicate slugs across sources.
- Keep PostgreSQL and PGLite projections in parity.
- Do not change schema or conditional-write conflict semantics.
- Keep deployment, restart, migration, and push actions outside the implementation tasks; a separately authorized release workflow may perform them after verification.
- Capture full test output to files before inspecting it.
---
### Task 1: Pin the backlink false-negative in PGLite
**Files:**
- Modify: `test/writer.test.ts`
**Interfaces:**
- Consumes: `backLinkValidator.validate(PageValidationContext)` and source-qualified `putPage`/`addLink`.
- Produces: regressions for wrong-source reverse rejection, exact reverse acceptance, cross-source pair acceptance, and exact target deduplication.
Create `default` and `team-x` copies of the origin and target, add `(team-x, origin) -> (team-x, target)` plus the wrong reverse `(team-x, target) -> (default, origin)`, validate with `sourceId: 'team-x'`, and require one warning.
- [ ] **Step 2: Run the focused test and verify RED**
```bash
bun test test/writer.test.ts -t "wrong-source reverse" > "$TEMP/backlink-red.txt" 2>&1
```
Expected: assertion failure because current slug-only validation returns zero findings.
- [ ] **Step 3: Add the remaining behavioral regressions after the first red is recorded**
Add tests proving that the exact reverse clears the warning, a legitimate cross-source forward/reverse pair passes, and two destinations sharing one slug but differing by source are validated independently.
### Task 2: Expose exact endpoint identity from both engines
- [ ] **Step 1: Add engine-contract assertions before implementation**
Assert scalar cross-source rows expose `beta -> default`, federated rows expose only in-grant endpoint IDs, `sourceIds` still beats scalar `sourceId`, and an out-of-grant origin has both `origin_slug` and `origin_source_id` null.
- [ ] **Step 2: Run the focused contract tests and verify RED**
```bash
bun test test/get-page-federated-scope.test.ts test/e2e/multi-source-bug-class.test.ts > "$TEMP/link-identity-red.txt" 2>&1
```
Expected: source-ID assertions fail because fields are absent.
- [ ] **Step 3: Extend `Link` and project IDs without changing predicates**
Use this additive contract:
```ts
export interface Link {
from_slug: string;
from_source_id: string;
to_slug: string;
to_source_id: string;
link_type: string;
context: string;
link_source?: string | null;
origin_slug?: string | null;
origin_source_id?: string | null;
origin_field?: string | null;
}
```
In all six branches per engine, project:
```sql
f.source_id AS from_source_id,
t.source_id AS to_source_id,
o.source_id AS origin_source_id
```
Keep every `WHERE` and grant-aware origin `LEFT JOIN` unchanged.
- Produces: optional `PageValidationContext.sourceId` and `sourceIds`, with `sourceIds` taking precedence.
- `runPostWriteLint(..., opts)` accepts the same optional scope and loads the validated page through it.
- [ ] **Step 1: Add a post-write nested-read regression and verify RED**
Validate a non-default page with a wrong-source reverse via `runPostWriteLint(..., { force: true, noLog: true, sourceId: 'team-x' })`; require a backlink warning.
Add `sourceId?`/`sourceIds?` to validation context and lint options. Load pages using `sourceIds` when non-empty, otherwise scalar `sourceId`. Pass the same scope into nested validators. In the put-page success hook, call lint with the already-resolved write source ID.
Initial outbound reads use the validation scope. Deduplicate rows by all four endpoint identity fields so every distinct expected origin remains represented even when targets share a source-qualified identity. Read each target using the federated grant when present, otherwise the target's exact scalar source. Accept only a row matching all four endpoint fields of the expected reverse.
- [ ] **Step 4: Run writer and post-write tests and verify GREEN**
```bash
bun test test/writer.test.ts test/post-write-lint.test.ts > "$TEMP/backlink-green.txt" 2>&1
```
Expected: all tests pass, including the recorded false-negative.
### Task 4: Verify PostgreSQL/PGLite parity and final scope
Compare sorted tuples containing `from_source_id`, `from_slug`, `to_source_id`, `to_slug`, `origin_source_id`, and `origin_slug` for scalar and federated fixtures.
- [ ] **Step 2: Run focused PGLite/source-isolation tests**
```bash
bun test test/writer.test.ts test/post-write-lint.test.ts test/get-page-federated-scope.test.ts test/e2e/multi-source-bug-class.test.ts > "$TEMP/backlink-focused.txt" 2>&1
```
Expected: exit 0.
- [ ] **Step 3: Run PostgreSQL parity when the test database is available**
```bash
bun test test/e2e/engine-parity.test.ts -t "federated sourceIds" --timeout=300000 > "$TEMP/backlink-parity.txt" 2>&1
```
Expected: exit 0; if the configured test database is unavailable, report the exact environmental blocker rather than claiming parity execution.
- [ ] **Step 4: Typecheck and inspect the final diff**
```bash
bun run typecheck > "$TEMP/backlink-typecheck.txt" 2>&1
```
Expected: exit 0. Then run `git diff --check` and confirm no version, schema, migration, deployment, or conditional-write files changed.
Reconcile the overlapping engine dynamic-import changes from:
- `claude/hungry-edison-8bb1cd` at release commits `48ada48f` and `248bfe55`
- `claude/elegant-gates-e5275e` at `ef4cf7a8`
onto a fresh branch from current `origin/master`, without merging or cherry-picking either lineage wholesale and without adding a release/version bump.
## Established state
At investigation time:
- `origin/master` was `6136e139972a5449630b4f47f5ed7b4cbe5b811b`, version `0.42.67.0`.
- Upstream PR #3511 was still open, so trunk did not contain its two `chronicle/ontology.ts` hoists.
- Neither source branch was an ancestor of trunk.
- Trunk contained 17 dynamic imports in the three engine-path files:
- 13 safe-hoist candidates: two ontology imports, nine engine helper/audit imports, and two migration imports.
- Four `ai/gateway.ts` imports, all inside `try/catch` fallback paths.
- `git log -G` showed the separate ontology, helper, migration, and gateway histories. `git log -S` is not suitable for this dynamic-to-static replacement because the relevant token can remain present while its context changes.
- The guard from `ef4cf7a8` passed against that commit but failed against trunk. It also knew about only two gateway opt-outs because two `_upsertChunksOnce` gateway lookups landed later in trunk.
## Selected approach
Reconstruct the intended current state directly on fresh `origin/master`.
Do not merge or cherry-pick either old lineage. Selectively reproduce the desired source changes, adapt the guard to the current four gateway call sites, and write current-state documentation. This avoids importing stale release metadata, stale TODO claims, and unrelated lineage changes.
## Source changes
### Safe static imports
Hoist all 13 safe candidates:
- `src/core/pglite-engine.ts`
- `valueHash`, `normalizeDimension`, `isNovelDimension` from `chronicle/ontology.ts`
- `isRetryableConnError` through the existing `retry.ts` import
- `resolveRecencyDecayMap`, `DEFAULT_FALLBACK` from `search/recency-decay.ts`
- `src/core/postgres-engine.ts`
- the same ontology, retry, and recency helpers
- `isConnectionEndedError` from `retry-matcher.ts`
- `logDbDisconnect` from `audit/db-disconnect-audit.ts`
- `logPoolRecovery` from `audit/pool-recovery-audit.ts`
- `src/core/migrate.ts`
- `isStatementTimeoutError`, `isRetryableConnError` from `retry-matcher.ts`
- `repairTimelineDedupIndex` from `timeline-dedup-repair.ts`
The implementation must keep the two engines in parity where the behavior is shared. Comments should describe current invariants, not repeat an unproven causal claim that these hoists fix the Windows test-runner crash.
### Deliberately lazy gateway imports
Keep all four `await import('./ai/gateway.ts')` call sites lazy:
- PGLite `initSchema`
- PGLite `_upsertChunksOnce`
- Postgres `initSchema`
- Postgres `_upsertChunksOnce`
Each line receives the explicit `engine-dynamic-import-ok` marker and a concise nearby rationale.
The rationale has two parts:
1. The gateway's static closure includes the AI SDK, provider packages, and validation/config machinery, so eager loading would tax engine startup paths that do not otherwise need it.
2. More importantly, each lookup is inside a `try/catch` that preserves a soft fallback (compiled defaults or the brain's stored embedding-model config). Hoisting the module would evaluate it before that catch can run and could convert a recoverable configuration/import failure into a module-load-time hard failure.
The guard must not allow unmarked gateway imports or a broad file-level exemption.
## Guard and wiring
Add `scripts/check-engine-dynamic-import.sh`, adapted from `ef4cf7a8`, with these properties:
- Default scan set:
- `src/core/pglite-engine.ts`
- `src/core/postgres-engine.ts`
- `src/core/migrate.ts`
- Normalize trailing CR before matching so CRLF checkouts cannot bypass the check.
- Ignore comment-only lines.
- Ignore only lines carrying `engine-dynamic-import-ok`.
- Report every unmarked `await import(` with file and line.
- Explain that contributors should prefer a static import and must justify a real opt-out.
- Avoid asserting that every dynamic import deterministically crashes Windows; the measured evidence supports treating the pattern as an engine-path hardening invariant, while box-level commit exhaustion remained a confound in prior runs.
Wire it into:
- `package.json` as `check:engine-dynamic-import`
- `package.json``check:all`
- `scripts/run-verify-parallel.sh`
Follow trunk's current rule that package scripts invoke repository shell scripts through `bash`.
## Regression coverage
Add an automated test for the guard. It must cover:
- A real dynamic import produces exit 1 and is reported.
- A line carrying `engine-dynamic-import-ok` is allowed.
- Line comments and block-comment lines do not produce findings.
- The same violation is caught with CRLF input.
- The default repository scan passes after the source reconciliation.
Use a temporary fixture rather than mutating tracked source files. Keep assertions path-portable.
The pre-fix red demonstration is the exact guard from `ef4cf7a8` run against current trunk: it exits 1 and reports the existing unmarked imports. The post-fix guard and test must pass.
## Documentation policy
Preserve current behavior, not either old release narrative:
- Do not modify `VERSION` or add a release `CHANGELOG.md` entry.
- Do not copy old version headings or completed release TODO blocks.
- Do not retain the old TODO claiming that extracting gateway accessors is necessarily the fix; the lazy imports are deliberately protected by their local soft-failure boundaries.
- Add the cross-cutting no-unmarked-dynamic-import invariant to `CLAUDE.md`.
- Update the current-state entries for `src/core/pglite-engine.ts`, `src/core/postgres-engine.ts`, and `src/core/migrate.ts` in `docs/architecture/KEY_FILES.md` where needed.
- Regenerate `llms.txt` and `llms-full.txt` after the documentation edits.
- Add a TODO only if implementation uncovers a real unresolved action.
Public documentation must use generic language and must not overstate the historical Windows crash causality.
## Verification
Capture full output to files before inspecting summaries. Run, at minimum:
1. The guard regression test.
2. `bash scripts/check-engine-dynamic-import.sh`.
3. Focused tests that exercise the touched engine, migration, retry, audit, and recency modules.
4. `bun run typecheck`.
5. `bun run verify`.
6. `bun run build:llms` followed by `bun test test/build-llms.test.ts`.
7. `git diff --check` and a final clean-status/diff review.
If platform contention or existing Windows suite defects block a broad test, report the exact command, exit code, and ownership classification rather than declaring success from a partial run.
## Git and publication boundary
- Work on `claude/kind-meitner-330c90`, reset locally to the exact investigated `origin/master` base.
- Preserve the previous worktree tip under `claude/kind-meitner-330c90-pre-reconcile`.
- Keep implementation and verification commits local.
- Do not push, create a PR, comment upstream, or otherwise publish without explicit user approval after the local result is complete.
the validator accepts any reverse row whose bare slugs are:
```text
people/target -> concepts/origin
```
That can incorrectly accept a row ending at `(default, concepts/origin)` instead of `(source-a, concepts/origin)`.
The bug is not that scalar `getLinks(slug, { sourceId })` permits cross-source destinations. That behavior is intentional: scalar scope qualifies the near/from endpoint while trusted local callers retain visibility into explicit cross-source edges. The gap is that a returned `Link` does not carry the source identity of either endpoint, so callers cannot distinguish same-slug pages.
## Reproduction and evidence
A deterministic PGLite reproduction creates duplicate `concepts/a` and `people/b` pages in `default` and `team-x`, then adds:
```text
(team-x, concepts/a) -> (team-x, people/b)
(team-x, people/b) -> (default, concepts/a)
```
The second edge is not a valid reverse of the first. Nevertheless:
returns the second row, and the current validator accepts it because `to_slug === 'concepts/a'`.
Both engines implement the same scalar rule: filter `f.slug` and `f.source_id`, join the actual destination by `to_page_id`, and do not filter `t.source_id`. Federated `sourceIds` is a separate branch that constrains all visible endpoints and takes precedence over scalar scope.
## Goals
1. Validate back-links by exact source-qualified endpoint identity.
2. Preserve explicit cross-source links for trusted scalar reads.
3. Preserve federated all-endpoint containment and `sourceIds` precedence.
4. Keep PostgreSQL and PGLite behavior identical.
5. Add strict red-before-green regressions using duplicate slugs across sources.
6. Avoid schema migrations and production operational changes.
## Non-goals
- Changing scalar link reads to same-source-only reads.
- Weakening or widening federated reads.
- Changing link write identity or database schema.
- Refactoring the atomic conditional-write branch.
- Coupling deployment, restart, or migration mechanics to the backlink code change. Release operations are handled separately after verification.
## Chosen approach
Extend the engine `Link` result with endpoint source identities and use those fields in the validator.
```ts
interface Link {
from_slug: string;
from_source_id: string;
to_slug: string;
to_source_id: string;
// existing fields
origin_slug?: string | null;
origin_source_id?: string | null;
}
```
All `getLinks` and `getBacklinks` query branches in PostgreSQL and PGLite will project the source IDs from the pages already joined as `f`, `t`, and `o`. No filtering behavior changes.
This approach is preferred over a dedicated `hasExactLink` method because it keeps source identity attached to the link data everywhere, avoids duplicate engine SQL and per-edge existence queries, and matches existing source-qualified link-write and batch-row contracts.
Validator-only raw SQL is rejected because validators should consume the `BrainEngine` contract rather than bypass it with engine-specific schema knowledge.
## Engine semantics
The existing three read modes remain unchanged.
### Unscoped
`getLinks(slug)` returns rows from all same-slug from-pages across sources. Each row identifies the actual source of both endpoints.
### Scalar source
`getLinks(slug, { sourceId })` matches exactly `(sourceId, slug)` on the from side. A destination may belong to another source, and `to_source_id` reveals that exact identity.
The corresponding scalar `getBacklinks` rule continues to match the exact destination/to-page identity while allowing a cross-source referrer.
### Federated sources
`getLinks(slug, { sourceIds })` continues to constrain from and to endpoints to the grant. The origin join continues to redact an out-of-grant origin. `sourceIds` continues to take precedence over scalar `sourceId`.
Adding source IDs to returned in-grant endpoints does not disclose anything new: the existing result already discloses those pages' slugs and edges. An out-of-grant endpoint remains absent.
## Validator algorithm
The validator receives the source scope associated with the page being validated.
Duplicate edge rows are deduplicated by the full endpoint pair `(from_source_id, from_slug, to_source_id, to_slug)`, not by bare target slug. This preserves separate reverse requirements when multiple same-slug origin pages point to one exact target.
For each target:
1. Read target outbound links using the target's exact scalar source when validation is scalar-scoped.
2. Under federated validation, retain the caller's `sourceIds` grant rather than converting it to scalar scope.
3. Accept only a returned row whose `from_source_id`, `from_slug`, `to_source_id`, and `to_slug` exactly match the expected reverse identity.
4. Emit the existing warning when no exact reverse exists.
This preserves legitimate cross-source pairs. For example:
`PageValidationContext` must carry the relevant scalar or federated source scope. The writer and post-write lint paths must load the page with that scope and pass the same scope to nested validator reads.
This change is scoped to source routing needed by validation. It does not modify conditional-write revision or conflict semantics and must not be applied to the atomic conditional-write branch.
## Testing strategy
### PGLite strict-TDD regression
Add duplicate pages across `default` and a second source, then prove before the production fix that:
1. A forward edge in the second source plus a wrong-source reverse produces a warning.
2. Adding the exact reverse removes the warning.
3. A legitimate cross-source forward/reverse pair passes.
4. Two same-slug destination pages are not collapsed into one target identity.
The first assertion must fail against the pre-fix implementation.
### Engine contract tests
For PGLite and PostgreSQL:
1. Assert link rows expose exact from/to source IDs.
2. Assert scalar reads still return explicit cross-source destinations.
3. Assert federated reads still exclude out-of-grant endpoints.
4. Assert `sourceIds` still takes precedence over scalar `sourceId`.
5. Assert origin source identity is null when the origin is redacted by the federated branch.
### Parity and focused verification
Run:
- the focused backlink validator test;
- source-isolation and federated link tests;
- the Postgres/PGLite parity fixture with a test database;
- related writer/post-write tests;
- `bun run typecheck`.
Capture complete command output to files before inspecting summaries. Do not use production databases or restart the live service.
## Compatibility
The `Link` change is additive at runtime. Existing consumers that read only slug or provenance fields continue to work. TypeScript object literals typed as complete `Link` values may need source fields; if compatibility pressure is high, the source fields can initially be optional in the public type while engine implementations and validator tests require their presence. The preferred contract is required endpoint source IDs because every persisted link always has both pages and therefore both source IDs.
No schema migration is required because source IDs already live on the joined `pages` rows.
## Operational constraints
The implementation phase does not deploy, restart GBrain, run production migrations, or alter the atomic conditional-write branch. Release, migration, and restart operations are a separate verified workflow and do not change this design's engine or validator semantics.
@@ -13,7 +13,7 @@ Step-by-step walkthroughs that take you from zero to a working outcome. Concrete
These are the next tutorials on the roadmap. Open an issue if one of them is the one you need most; that's how we'll prioritize.
- **Set up GBrain for VC dealflow** — the operator's recipe. People pages for founders, companies with typed Facts fence carrying ARR / team-size / runway across dates, meetings auto-ingested, deal pages linking everything. Shows `gbrain whoknows`, `gbrain find_trajectory`, and `gbrain founder scorecard` on real workflows.
- **Set up GBrain for VC dealflow** — the operator's recipe. People pages for founders, companies with typed Facts fence carrying ARR / team-size / runway across dates, meetings auto-ingested, deal pages linking everything. Shows `gbrain whoknows`, `gbrain find-trajectory`, and `gbrain founder scorecard` on real workflows.
- **Migrate your existing vault into GBrain** — for Notion / Obsidian / Roam users with a vault that doesn't match GBrain's default layout. Walks through `gbrain schema detect` → `suggest` → `review-candidates` so the brain learns your shape instead of forcing you to learn its.
@@ -93,7 +93,7 @@ There are two ways to scope teammates' access. They suit different deployment sh
**Model A: separate sources with OAuth scoping (recommended for true multi-user with different AI clients).** What this tutorial walks you through. Each teammate gets their own OAuth client, which carries `--source` + `--federated-read` flags. The brain refuses cross-source reads at the SQL layer; isolation is database-enforced. Each teammate can run their own MCP-aware client (Claude Code, Cursor, their own OpenClaw, etc.) and the scoping holds.
**Model B: one source, directory-based per-person scoping (simpler for one-agent-serves-everyone setups).** The shape I actually run in production: a single source called `default`, with a `partners/<slug>/` convention inside it (e.g. `partners/alice-example/`, `partners/bob-example/`). Each partner gets their own subdirectory holding their personal pages: `partners/alice-example/USER.md`, `partners/alice-example/concepts/`, `partners/alice-example/sources/`, etc. There's no OAuth-enforced isolation; the agent itself enforces "Alice's writes go to her partners/ subdir." This is the right model when ONE agent (yours) serves everyone over Telegram or a single shared interface. It's simpler ops, no per-user OAuth, but the scoping is convention-only.
**Model B: one source, directory-based per-person scoping (simpler for one-agent-serves-everyone setups).** The shape I actually run in production: a single source called `default`, with a `partners/<slug>/` convention inside it (e.g. `partners/alice-example/`, `partners/bob-example/`). Each partner gets their own subdirectory holding their personal pages: `partners/alice-example/USER.md`, `partners/alice-example/concepts/`, `partners/alice-example/sources/`, etc. This is the right model when ONE agent (yours) serves everyone over Telegram or a single shared interface. It's simpler ops, no per-user OAuth. **Write scoping within the shared source can be server-enforced:** register each per-person client with `--bound-slug-prefixes partners/alice-example/` and every slug-mutating write outside that prefix is rejected with `permission_denied` (v0.42.72.0+). Without the binding, the scoping is convention-only (the agent polices itself). Read scoping stays source-granular in both models — within a shared source, everyone entitled to the source can read every folder.
For most company-brain installs (10+ teammates each with their own AI client), Model A is the right starting point. If you're running the fat-agent-serves-everyone pattern from the personal-brain tutorial, Model B is genuinely simpler. You can also mix: separate sources for the obviously-different ones (customer notes vs internal-only) AND a `partners/<slug>/` convention inside the shared source for per-person workspace.
@@ -210,7 +210,7 @@ Each `register-client` command prints a `client_id` and a `client_secret`. Save
A note on the flags:
- `--scopes read,write` lets the client query the brain and write new pages. You can omit `write` for read-only clients (executive summaries, dashboards). The `admin` scope is needed for operational commands like `gbrain remote doctor` and is usually reserved for your own admin client.
- `--source` controls write authority. A client can only write to one source. Within that source, your folder convention from Part 3 keeps each person's writes in their own subfolder.
- `--source` controls write authority. A client can only write to one source. Within that source, your folder convention from Part 3 keeps each person's writes in their own subfolder — and you can make that server-enforced with `--bound-slug-prefixes alice-example/` (v0.42.72.0+): every slug-mutating write op (put_page, delete_page, tags, links, timeline, revert, raw data) outside the bound prefixes is rejected with `permission_denied`. Update the binding later with `gbrain auth rescope-client <id> --bound-slug-prefixes <p1,p2|none>`. **Adding a binding to an existing client narrows it in ways you should expect:** ops that write by something other than a slug (`extract_entities`, `extract_facts`, `forget_fact`, `ontology_propose`, `sources_add`/`sources_remove`) and `POST /ingest` become unavailable to that client, and `put_page`'s automatic fact extraction is skipped — all because none of them can be confined to a prefix. Reads are unaffected. See [the qm-harness guide](../integrations/qm-harness.md) for the full model.
- `--federated-read` controls read scope. A client can read from one or more sources.
### Verify the scoping actually scopes
@@ -484,6 +484,10 @@ Returns a per-source dashboard: when each source last synced, how many pages, ho
The admin dashboard at `https://brain.acme-co.com/admin` shows live request volume, registered OAuth clients, recent activity, and brain stats. Use the admin bootstrap token from Part 4 to log in the first time, then register additional admin users from inside the dashboard.
### If agents run as containers on the same Docker host
OAuth source scoping only guards the HTTP MCP path. If the brain's Postgres and your teammates' agent runtimes are containers on the same Docker host, make sure the agents can't reach Postgres directly over Docker's default bridge network — a direct DB session skips OAuth entirely. Put Postgres on its own user-defined network, publish it loopback-only if at all, and never hand agent containers a `DATABASE_URL`. The copy-paste operator checklist lives in [docs/mcp/DEPLOY.md — Co-located Docker workloads](../mcp/DEPLOY.md#co-located-docker-workloads-self-hosted-postgres).
---
## Part 13: Cost and speed expectations
@@ -550,7 +554,7 @@ What to do next:
- **Wire ingestion** from external systems (Granola, Linear, Slack) using the [ingestion source contract](../skillpack-anatomy.md). Most companies want their meetings auto-ingested so the brain stays current without anyone typing notes.
- **Set up team-specific dashboards** through the admin UI. Each team lead can have their own view of brain health and activity.
- **Explore the rest of the brain layer.**`gbrain whoknows` (find the expert on a topic), `gbrain find_trajectory` (how a metric changed over time), `gbrain founder scorecard` (especially useful for VC and ops teams), the contradiction-detection cycle that surfaces conflicts between different people's notes.
- **Explore the rest of the brain layer.**`gbrain whoknows` (find the expert on a topic), `gbrain find-trajectory` (how a metric changed over time), `gbrain founder scorecard` (especially useful for VC and ops teams), the contradiction-detection cycle that surfaces conflicts between different people's notes.
If you're building in this space (which YC has flagged as the [company-brain category in its Request for Startups](https://www.ycombinator.com/rfs#company-brain)), you might as well build on this. Everything described above is open source, MIT licensed, and what I run in production behind my own AI agents.
@@ -115,21 +115,21 @@ You can use the same keys across multiple agents.
## Step 6: Install GBrain
Once OpenClaw is running:
Once OpenClaw is running, installation is two commands — one in the brain repo, one in the agent workspace:
```bash
gbrain install
# In the BRAIN repo (the git repo that holds your markdown pages):
gbrain init --supabase
# In the AGENT WORKSPACE repo (where OpenClaw runs):
gbrain skillpack scaffold --all
```
This installs:
`gbrain init --supabase` walks a short wizard that asks for your Supabase connection string and creates the schema. You'll get that connection string in Step 7 — read 7a and 7b first so you paste the right one (the transaction pooler, not the direct connection). If you'd rather try things locally before paying for a database, `gbrain init --pglite` gives you a zero-config embedded engine instead; you can migrate to Supabase later with `gbrain migrate --to supabase`.
- About 60 skills
- About 9 skill packs
- Default brain structure
- MCP server configuration
- Supabase connection (for embeddings and search)
`gbrain skillpack scaffold --all` copies the ~43 bundled skills into your agent workspace as first-class files you can edit freely. (The old managed-install model was retired in v0.36.0.0; see `docs/INSTALL.md` if you're upgrading from an older release.)
GBrain populates the brain repo with its default directory structure, skill files, and configuration. From this point, the agent has working memory and access to every skill.
From this point, the agent has working memory and access to every skill.
The sync backfills `page.type = 'meeting'` on all 4000 pages in 1000-row batches. Now:
- `gbrain whoknows "Q3 roadmap discussion"` routes through the meeting type, ranking by `expert_routing` signal (attendees, recency, salience) instead of raw text.
- `gbrain extract-facts` runs on every meeting page automatically (because `extractable: true`), pulling typed facts like `attended_by=alice-example`, `date=2026-05-23`.
- The `extract_facts` cycle runs on every meeting page automatically (because `extractable: true`), pulling typed facts like `attended_by=alice-example`, `date=2026-05-23`.
- The downstream `think` skill can now answer "what did we decide about pricing in the last three roadmap meetings" by querying the meeting graph instead of grep'ing 4000 files.
One command. 4000 pages went from invisible to queryable. The content didn't change. The structure did.
Now `gbrain whoknows "Series A SaaS"` routes through `investor` and `portco` types specifically, not the noisy general type set. `gbrain graph-query alice-example --type intro-from --depth 2` walks two hops of intros to surface "Alice introduced you to Bob who introduced you to Charlie." `gbrain extract-facts` starts producing typed claims from the fence in your deal pages: `(deals/acme-seed, raise=2000000, valuation=15000000, lead=widget-vc, closed_at=2026-05-23)`.
Now `gbrain whoknows "Series A SaaS"` routes through `investor` and `portco` types specifically, not the noisy general type set. `gbrain graph-query alice-example --type intro-from --depth 2` walks two hops of intros to surface "Alice introduced you to Bob who introduced you to Charlie." The `extract_facts` cycle starts producing typed claims from the fence in your deal pages: `(deals/acme-seed, raise=2000000, valuation=15000000, lead=widget-vc, closed_at=2026-05-23)`.
The CRM you've been promising yourself you'll set up next quarter? You just shipped it in 4 commands. It's downstream of your notes, not parallel to them.
@@ -143,7 +143,7 @@ Re-run the same `whoknows` query. Top-3 should shift, because the new type is no
Three things gbrain does that generic note systems can't:
**1. The brain knows the difference between a person and an idea.** Page-type matters at query time. `gbrain whoknows` only considers `expert_routing: true` types. `gbrain extract-facts` only runs on `extractable: true` types. `gbrain graph-query` walks declared link verbs. None of that works on a flat tag system because tags don't have semantics — they're labels. Types are first-class citizens with rules attached.
**1. The brain knows the difference between a person and an idea.** Page-type matters at query time. `gbrain whoknows` only considers `expert_routing: true` types. The `extract_facts` cycle only runs on `extractable: true` types. `gbrain graph-query` walks declared link verbs. None of that works on a flat tag system because tags don't have semantics — they're labels. Types are first-class citizens with rules attached.
**2. Untyped content is invisible content.** If your meetings are typed as `note`, expert routing skips them, facts extraction ignores them, link inference doesn't fire. They exist on disk and they're indexed for text search, but the structural surfaces (whoknows, find_experts, recall, think) treat them as second-class. Adding a type isn't cosmetic; it's structural promotion.
Default path (Bun is required — gbrain is a Bun + TypeScript runtime):
```bash
@@ -1559,14 +1579,24 @@ This is the difference between a search engine and a brain. Search finds the pag
## Install
> [!WARNING]
> **GBrain is NOT distributed on npm.** The npm package named `gbrain` is an unrelated
> package with no connection to this project. Do not run `npm install -g gbrain` or
> `bun add -g gbrain` — you'll get something else, and it can shadow the real binary on
> your PATH. Install and upgrade ONLY via the documented paths below
> (`bun install -g github:garrytan/gbrain`, or `git clone` + `bun install && bun link`).
> If you already ran the npm install by mistake: `npm uninstall -g gbrain` /
> `bun remove -g gbrain`, then reinstall from GitHub. `gbrain doctor` detects a
> shadowing npm install and prints the fix.
GBrain is designed to be installed and operated by an AI agent. The fastest path is to have your agent do it for you. The CLI and MCP paths below are for people who want to wire it up themselves.
### Have your agent install it (recommended)
If you don't already have an AI agent platform running, start with one of these. Both are designed to read GBrain's install protocol and execute it:
- **[Hermes](https://github.com/NousResearch/hermes-agent)** — deploy on [Railway](https://github.com/praveen-ks-2001/hermes-agent-template) (one click)
Then paste this into your agent:
@@ -1700,7 +1730,7 @@ Most personal-knowledge tools force one fixed layout: their idea of "notes" + "p
**gbrain doesn't have a fixed layout.** It ships with bundled schema packs and lets you author your own when none fit:
- **`gbrain-base-v2`** (default as of v0.41.22) — 15-type DRY/MECE canonical taxonomy (14 canonical + `note` catch-all): `person`, `company`, `media`, `tweet`, `social-digest`, `analysis`, `atom`, `concept`, `source`, `deal`, `email`, `slack`, `writing`, `project`, `note`. Subtypes/format/origin pushed to frontmatter. The taxonomy that responds to issue #1479.
- **`gbrain-base`** (legacy, v0.41 and earlier brains) — the original 24-type layout. Stays bundled for back-compat; brains on it can upgrade via `gbrain onboard --check --explain` → `gbrain jobs submit unify-types --allow-protected --params '{"target_pack":"gbrain-base-v2"}'`.
- **`gbrain-base`** (legacy, v0.41 and earlier brains) — the original 24-type layout. Stays bundled for back-compat; brains on it can upgrade via `gbrain onboard --check --explain` → `gbrain jobs submit unify-types --allow-protected --params '{"target_pack":"gbrain-base-v2","apply":true}'` (omit `"apply":true` for a dry-run preview — that is the default).
- **`gbrain-recommended`** — extends `gbrain-base` with the 13 additional directories from `docs/GBRAIN_RECOMMENDED_SCHEMA.md` (source, place, trip, conversation, personal, civic, project, etc.). Activate with `gbrain schema use gbrain-recommended`.
- **Your own pack** — `gbrain schema detect` clusters your actual filesystem into proposed types, `gbrain schema suggest` runs an LLM pass over them, and `gbrain schema review-candidates --apply` promotes the ones you like. Three commands and the brain knows your shape. Authoring a successor pack (declares `migration_from:` so existing brains can opt in): see [`docs/architecture/pack-upgrade-mechanism.md`](docs/architecture/pack-upgrade-mechanism.md).
@@ -2316,7 +2346,7 @@ gbrain schema sync --apply
The sync backfills `page.type = 'meeting'` on all 4000 pages in 1000-row batches. Now:
- `gbrain whoknows "Q3 roadmap discussion"` routes through the meeting type, ranking by `expert_routing` signal (attendees, recency, salience) instead of raw text.
- `gbrain extract-facts` runs on every meeting page automatically (because `extractable: true`), pulling typed facts like `attended_by=alice-example`, `date=2026-05-23`.
- The `extract_facts` cycle runs on every meeting page automatically (because `extractable: true`), pulling typed facts like `attended_by=alice-example`, `date=2026-05-23`.
- The downstream `think` skill can now answer "what did we decide about pricing in the last three roadmap meetings" by querying the meeting graph instead of grep'ing 4000 files.
One command. 4000 pages went from invisible to queryable. The content didn't change. The structure did.
Now `gbrain whoknows "Series A SaaS"` routes through `investor` and `portco` types specifically, not the noisy general type set. `gbrain graph-query alice-example --type intro-from --depth 2` walks two hops of intros to surface "Alice introduced you to Bob who introduced you to Charlie." `gbrain extract-facts` starts producing typed claims from the fence in your deal pages: `(deals/acme-seed, raise=2000000, valuation=15000000, lead=widget-vc, closed_at=2026-05-23)`.
Now `gbrain whoknows "Series A SaaS"` routes through `investor` and `portco` types specifically, not the noisy general type set. `gbrain graph-query alice-example --type intro-from --depth 2` walks two hops of intros to surface "Alice introduced you to Bob who introduced you to Charlie." The `extract_facts` cycle starts producing typed claims from the fence in your deal pages: `(deals/acme-seed, raise=2000000, valuation=15000000, lead=widget-vc, closed_at=2026-05-23)`.
The CRM you've been promising yourself you'll set up next quarter? You just shipped it in 4 commands. It's downstream of your notes, not parallel to them.
@@ -2427,7 +2457,7 @@ Re-run the same `whoknows` query. Top-3 should shift, because the new type is no
Three things gbrain does that generic note systems can't:
**1. The brain knows the difference between a person and an idea.** Page-type matters at query time. `gbrain whoknows` only considers `expert_routing: true` types. `gbrain extract-facts` only runs on `extractable: true` types. `gbrain graph-query` walks declared link verbs. None of that works on a flat tag system because tags don't have semantics — they're labels. Types are first-class citizens with rules attached.
**1. The brain knows the difference between a person and an idea.** Page-type matters at query time. `gbrain whoknows` only considers `expert_routing: true` types. The `extract_facts` cycle only runs on `extractable: true` types. `gbrain graph-query` walks declared link verbs. None of that works on a flat tag system because tags don't have semantics — they're labels. Types are first-class citizens with rules attached.
**2. Untyped content is invisible content.** If your meetings are typed as `note`, expert routing skips them, facts extraction ignores them, link inference doesn't fire. They exist on disk and they're indexed for text search, but the structural surfaces (whoknows, find_experts, recall, think) treat them as second-class. Adding a type isn't cosmetic; it's structural promotion.
@@ -2720,14 +2750,17 @@ GBrain is tuned for the Supabase **Transaction pooler** (port 6543): it
auto-disables prepared statements there and routes `engine.transaction()`
(migrations, DDL, sync imports) to a derived **direct** connection
(`db.<ref>.supabase.co:5432`). That direct host is IPv6-only, so on an
IPv4-only host, reads work but sync **silently skips most pages**. This is the
number one cause of "sync ran but nothing happened."
IPv4-only host it is unreachable. When that happens gbrain now falls back to
the pooler automatically (one stderr warning, then single-pool mode for the
rest of the process) — but the pooler's ~2-min statement timeout can truncate
very long migrations or bulk imports.
Fix: make the direct connection reachable over IPv4. Either set
`GBRAIN_DIRECT_DATABASE_URL` to the **Session pooler** string (port 5432 on the
`pooler.supabase.com` host, IPv4), or enable Supabase's IPv4 add-on. Verify by
running `gbrain sync` and checking that the page count in `gbrain stats` matches
the syncable file count in the repo.
`pooler.supabase.com` host, IPv4), or enable Supabase's IPv4 add-on.
`GBRAIN_DISABLE_DIRECT_POOL=1` skips the direct pool (and the fallback warning)
entirely. Verify by running `gbrain sync` and checking that the page count in
`gbrain stats` matches the syncable file count in the repo.
### The Primitives
@@ -3894,7 +3927,7 @@ All 30 GBrain operations are available remotely, including `sync_brain` and
directory where `gbrain serve` was launched. Symlinks, `..` traversal, and absolute
paths outside cwd are rejected. Page slugs and filenames are allowlist-validated
(alphanumeric + hyphens; no control chars, RTL overrides, or backslashes). Local
"eval:autocut":"bun test test/search/autocut-eval.test.ts",
"test:full":"bun run verify && bash scripts/run-unit-parallel.sh && bun run test:slow && ([ -n \"$DATABASE_URL\" ] && bash scripts/run-e2e.sh || echo '[test:full] skipped E2E (no DATABASE_URL); run docker-compose -f docker-compose.ci.yml up + bun run test:e2e to include' 1>&2)",
description: Twitter timeline, mentions, and keyword monitoring flow into brain pages. Tracks deletions, engagement velocity, OCR on images, and real-time alerts.
category: sense
requires: []
secrets:
- name: X_BEARER_TOKEN
- name: X_API_BEARER_TOKEN
description: X API v2 Bearer token (Basic tier minimum, $200/mo for full archive search)
where: https://developer.x.com/en/portal/dashboard — create a project + app, copy the Bearer Token from "Keys and tokens"
- name: X_HANDLE
description: Your X username without the @ (used for the app-only health check — /users/me requires user-context OAuth, which app-only bearer tokens don't have)
where: Your X profile — the handle in your profile URL, e.g. x.com/yourhandle → yourhandle
console.error('This envelope is truncated, hand-edited, or from a broken converter. Nothing was written. Re-export, or correct the declared counts if the contents are known-good.');
// envelope-v0 requires both fields, so this file is already non-conforming.
// Import it anyway — hand-authored envelopes are useful — but never let an
// unchecked import look identical to a checked one on the way past. Naming
// the missing field matters: with one count present, only half the envelope
// was verified, and the receipt alone cannot show which half.
console.warn(
`warning: envelope declares no ${absent.map((c)=>c.field).join(' and no ')} (envelope-v0 requires both) — integrity check skipped for ${absent.length===2?'conversations and messages':absent[0].field.replace('meta.','').replace('_count','s')}; a truncated envelope would import silently.`,
// Name the decision AND its inputs. "Overwriting the earlier page" was the
// whole message before, and it would now be false half the time — the reader
// has to be able to check which copy survived rather than assume the old
// rule still applies.
constverdict=byUpdatedAt
?`keeping the copy whose updated_at is later (${JSON.stringify(keepLater?rendered.updatedAt:earlier.updatedAt)}) over ${JSON.stringify(keepLater?earlier.updatedAt:rendered.updatedAt)}`
:`updated_at cannot order these two copies (${JSON.stringify(earlier.updatedAt??null)} and ${JSON.stringify(rendered.updatedAt??null)}), so array order decides — overwriting the earlier page`;
console.warn(`warning: filename collision on "${name}" — conversation id ${JSON.stringify(c.id)} is not unique; ${verdict}.`);
// Say what is true — the file was not recognized. Asserting that this
// importer did not write it is a claim this code is in no position to make,
// and it is wrong for any page of ours that has been edited since.
conflicts.push(`${name} — already exists and could not be recognized as a page written by this importer.`);
}elseif(identity.id===null){
// Ours, but written from an id-less conversation, so its filename encodes
// array position rather than identity. An update and a wholly different
// conversation are indistinguishable here; guessing either way risks
// destroying an import.
conflicts.push(`${name} — written by this importer from a conversation with no id, so it cannot be matched to this envelope's conversation. Refusing to guess.`);
}elseif(identity.id!==page.conversationId){
// Distinct ids that slug to one filename (truncation at 60 chars, or
// characters that slug away). Rare, but silently fatal if permitted.
conflicts.push(`${name} — holds conversation ${JSON.stringify(identity.id)}, but this envelope maps ${JSON.stringify(page.conversationId)} to the same filename.`);
}
// Otherwise: same conversation id, different content — a refreshed export
// updating its own page. That is exactly what re-importing is for.
}
if(conflicts.length){
console.error(`refusing to import: ${conflicts.length} target file(s) in ${outDir} would be overwritten with different content.`);
for(constlineofconflicts)console.error(line);
console.error('Nothing was written. Import into a different output directory, or delete the listed file(s) if they are stale.');
console.log(`wrote ${pages.size} markdown page(s) (${messagesWritten} message(s)) to ${outDir} — point gbrain's sync at this directory.`);
if(collisions){
// "Overwritten" would now be false whenever the tiebreak kept the earlier
// copy: that copy is never rewritten and the later one is never written at
// all. "Discarded" is true in both directions, and the per-collision lines
// above already say which copy went.
console.warn(`warning: ${collisions} filename collision(s) — ${collisions} page(s) discarded. Deduplicate conversation ids in the envelope to avoid data loss.`);
}
if(messagesWritten!==actualMessages){
// The page count alone cannot show this: a discarded copy leaves the same one
// file on disk, so only the message tally reveals the turns that went with it.
//
// "Discarded", not "overwritten", for the same reason as the summary line
// above: since the tiebreak can keep the EARLIER copy, the losing copy is
// sometimes never written at any point.
//
// Worded as a fact about the discarded copies, not as an announcement of
// loss. Duplicate ids are conforming input — the spec has merging never
// deduplicate — so converting an old export together with a newer one, which
// is what the memvelope CLI tells users to do, lands here routinely with the
// surviving page already holding every unique turn. An alarm that cries wolf
// on the mainstream path teaches its reader to ignore the one that matters.
console.warn(`warning: the discarded page(s) carried ${actualMessages-messagesWritten} message(s) that are not on disk (${actualMessages} read, ${messagesWritten} written). If they were earlier copies of the same conversation, the surviving page may already contain those turns; if not, this is real loss.`);
- **Title:** first non-empty line of the body, capped at 80 chars.
- **Title:** first non-empty line of the body, capped at 80 chars (truncation appends `…`).
## Output Format
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
