Compare commits

...
Author SHA1 Message Date
Garry Tan 14a2dd81ff fix(deps): bump four override pins past newly-published CVEs
osv-scan passed on the last two release PRs and fails on this one, so these are
newly published, not tolerated debt: 2 High + 4 Medium across fast-uri (3.1.4 →
3.1.5), hono (4.12.25 → 4.12.34), ip-address (10.1.1 → 10.3.1), and admin's
postcss (8.5.10 → 8.5.23). All four are transitive and pinned through the
overrides block, so the fix is the pin, not a dependency addition.

Resolved: fast-uri@3.1.5, hono@4.13.0, ip-address@10.4.0, postcss@8.5.25.
2026-08-04 20:21:55 +07:00
Garry Tan b2732ddd02 fix(ci): a permission failure must never red-X a contributor's PR (live incident)
The gate's first live runs put a red X on an outside contributor's PR five
times over, with no comment explaining why. This repo's GITHUB_TOKEN is
read-only, so every comment and label call returned 403; the throw reached the
top-level handler as exit 2.

The gate is advisory. A repository permission problem is an operator condition,
never a statement about the PR under review. 401/403/404 from GitHub now emit a
loud operator-facing warning naming both blockers (workflow permissions and the
missing ANTHROPIC_API_KEY secret) and exit 0. A genuine outage or a bug in here
still fails visibly at exit 2. Pinned by four tests including the entry-handler
routing; mutation-tested.
2026-08-04 20:21:55 +07:00
Garry Tan dba86d34c9 v0.42.73.0 feat(ci): strict PR usefulness gate + five contributed correctness fixes
Release-only commit: VERSION, package.json, CHANGELOG. All code already on
master. Covers the 8 gate commits (#3794, closes #3698), plus #3764 import
stdout, #3759 dry-run chmod, #3726 cycle model telemetry, #3751 integrity
counter, #3739 doc comment, and #2655 slug_filter.
2026-08-04 10:12:15 +07:00
Garry TanandClaude Fable 5 d610a845a8 fix(ci): gate — CommonMark info-string rule, hash the real model payload, close the YAML scanner gap (blind review round 5)
Three defects a second blind reviewer found that survived round 5. All three
reproduced against cf7a1616 before the fix and are pinned after it.

1. The fence scanner still falsely closed compliant descriptions. CommonMark 4.5
   forbids a backtick in a BACKTICK fence's info string, so ```foo`bar is an
   ordinary paragraph. The scanner opened a fence on it, never found a closer,
   and stripped the body to EOF:

     body = '```foo`bar' + three sentences of real prose + an image embed
     before: intentWordCount 0, misses [missing_intent, missing_screenshot]
     after:  intentWordCount 52, misses []

   Same class as round 5's closing-length bug and the same cost — opening a
   block CommonMark would not open deletes the author's prose exactly the way
   closing one late did. Tilde fences keep the permissive rule (4.5 restricts
   backtick fences only).

2. The spend guard hashed less than the model consumes. hashInputs covered
   title, modelBody, head.sha, exemption and policy ids — but the payload also
   carries the changed-file list and the diff, and the workflow degrades the
   diff to a marker line when the API 406s on a huge one. So a run that
   classified with no diff cached a diff-blind verdict, and the next run — real
   diff in hand, same title/body/sha — matched the hash and was served that
   verdict permanently (measured: anthropicCalls 1 across both runs). The
   assembled payload is now folded in as a fixed-width digest, inside the JSON
   tuple where quoting still makes a forged boundary impossible. Round 5's
   property is re-pinned: a policy fix past the model's body cap invalidates
   even when the payload is byte-identical.

3. The workflow's ${{ }}-in-run scanner missed a legal commented block header.
   `run: | # shell block` is valid YAML — js-yaml puts the following
   interpolation in the script — but the header fell through to the
   single-line branch, which captured `| # shell block` as the whole command
   and never looked at the block body. The rule protecting against shell
   injection reported clean over an interpolating workflow. The scanner now
   handles a trailing comment on the header, and scans the comment text too
   rather than leaving itself a hiding place. Guarded against js-yaml's actual
   parse, not against the scanner's own opinion.

Also corrected one overstated claim in the file's security block: a BARE url in
a sanitized string still autolinks under GFM. That is a self-labelled link and
the deliberate stopping point — the escaping targets the masking characters so
it cannot forge `![APPROVED](…)` or `[click to approve](…)` — but "no live
links" was too strong for what the code does.

Self-audit: all 13 interpolations into the sticky comment are either literals in
this file or pass sanitizeModelText/sanitizeList; the only unsanitized one
(titleCheck.reason) is a hardcoded literal and renders inside a code span. Every
rendered statement matches behavior the code performs. The four realistic-human
fixtures and the three new fence cases all PASS; empty, "fixes bug", the
unfilled template, ten words of lorem and a fenced-away screenshot all still
FAIL.

test/pr-gate-workflow.test.ts 158 -> 163 pass, 0 fail. Six mutations (the
CommonMark rule, the tilde exemption, the payload digest, the runGate wiring,
the scanner's comment group, the header-comment scan) each fail at least one
test. typecheck, actionlint, verify (34/34) green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-04 10:46:09 +08:00
Garry TanandClaude Fable 5 f87488ff36 fix(ci): gate — stop red-Xing genuine contributors; neutralize markdown embeds (blind review round 4)
A blind reviewer rejected this branch on false positives and the repro held up.
The gate's intent check deleted every list and blockquote LINE before counting
words, then measured what was left against an arbitrary 40. Real people write
their story as bullets. Measured on the four realistic-human descriptions from
the repro:

  description                              before  after  floor 40  floor 20
  own prose written as four bullets             0     55  CLOSE      PASS
  short non-native-English paragraph           38     38  CLOSE      PASS
  specific first-person bug report             34     34  CLOSE      PASS
  mostly a stack trace + a real explanation    28     27  CLOSE      PASS

The file's own exemption rationale already says a check that is red on every
release is a check somebody disables inside a week. Red on every terse-but-
genuine contribution dies the same way, and it costs real contributors on the
way there — worse than the forgery risks rounds 1-3 chased.

- intentWordCount strips the list/quote MARKER and keeps the author's words.
  Still stripped: fenced code, indented code (new — the other spelling of a
  fence, guarded so it never eats a list continuation), headings, HTML
  comments, raw HTML, bare URLs, inline code, link/image syntax, and the
  template's own bold prompts. An untouched pull_request_template.md still
  scores 0, pinned against the real file on disk.
- INTENT_MIN_WORDS 40 -> 20, documented as a floor against empty/boilerplate-
  only descriptions, not a quality bar. CONTRIBUTING.md documents no word count
  at all. Empty, "fixes bug", the unfilled template, 10 words of lorem and an
  indented log paste all still fail.
- The fix-it comment no longer tells contributors to reopen a PR that was never
  closed — the gate has no close call in it. It now says what actually happens:
  editing the description re-runs the check (the workflow triggers on `edited`),
  the PR stays open, a maintainer decides.
- sanitizeModelText backslash-escapes `[` and `]`. Escaping HTML killed the
  <details> forgery, but Markdown needs no angle brackets: an image embed
  renders a green "APPROVED" picture and a link renders a live phishing target
  inside a close-lane comment. The file asserted nothing attacker-controlled
  reaches Markdown unescaped; that is now true.
- A label-API failure during a NEUTRAL run no longer throws to exit 2 with no
  comment. NEUTRAL has no verdict to record, so it logs, still posts, exits 0 —
  and the comment stops claiming labels were cleared when they were not. The
  verdict path keeps failing loudly on purpose (ordering note).

Four verbatim human descriptions are pinned as PASSING forever. Eight mutations
(floor, list markers, quote markers, indented code, markdown escaping, the
reopen copy, the NEUTRAL try/catch, the label claim) each fail at least one test.

test/pr-gate-workflow.test.ts 145 -> 158 pass, 0 fail. typecheck, actionlint,
verify (34/34) green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-04 10:46:09 +08:00
Garry Tan ddb39df23a fix(ci): gate — escape comment HTML, tighten screenshot floor, fix fence false-positive, repair label ordering (blind review round 3)
Four findings from a second independent blind review, all reproduced against
HEAD before the fix and all pinned by a mutation-tested case.

- sanitizeModelText escapes &, < and > after the existing comment/mention/
  block-marker stripping, so raw HTML from any dynamic string (model output,
  mechanical flag details built from PR filenames, neutralReason, the
  policy-exempt note) renders as literal text. A <details open><summary>MERGE
  LANE — approved</summary> string is a forged verdict inside a close-lane
  comment; it now renders as &lt;details.
- hasScreenshot requires a markdown image whose URL looks like a URL or path,
  an <img> carrying a non-empty src=, or a bare paste URL, and strips HTML
  comments before scanning. `![proof](x)`, `<img alt=proof>` and an image
  hidden inside `<!-- -->` no longer clear it. Documented in the code as a
  FLOOR against zero-effort submissions, not proof.
- stripCodeFences is a line scanner following the CommonMark rule that a
  closing fence must be the same character and at least as long as the
  opening one. The old backreference read ```` as a NEW opening fence and
  stripped to EOF, so a compliant body documenting fence syntax lost its
  intent paragraph and was closed. The scanner is also linear, retiring the
  superlinear-backtracking hazard the 16KB cap was sized against.
- Labels are reconciled BEFORE the sticky comment carrying the cached state.
  Written the other way, one transient label-API 500 left stale/missing/
  duplicate labels forever: the rerun short-circuited on the persisted state
  and returned success without repairing them.

Also:
- hashInputs covers what the run consumes — the truncated model body plus the
  mechanical policy outcome — so an edit past the 6KB model cap no longer
  mints a new hash and buys an identical paid call, while a policy fix landing
  past that cap still invalidates the cached verdict.
- The test's YAML block-scalar scanner accepts indentation indicators in both
  legal orders (`>2-` as well as `|-2`), with a guard-the-guard case proving
  it sees a `run: >2-` block interpolating attacker-controlled text.
- A block at the top of the script states plainly that this gate is a triage
  signal and a reviewer checklist, not an authorization boundary; one line of
  the sticky comment footer says the same to the contributor.

145 pass / 0 fail (was 126), typecheck clean, actionlint clean, verify 34/34.
2026-08-04 10:46:09 +08:00
Garry TanandClaude Fable 5 88731d8cf3 fix(ci): gate — sanitize mechanical flag details, anchor path regexes, exempt maintainer/bot/draft from the policy check (blind review round 2)
Blind review round 2 rejected the branch. Five findings, all fixed.

BLOCKING 1 — the verdict was forgeable through a PR filename.
renderComment wrote mechanical red-flag details raw, and two of them
interpolate filenames (adds_recipe, deletes_tests). git allows a newline
inside a filename and JS `[^/]` matches one, so RECIPE_RE's anchors were
decorative: a PR adding `src/core/ai/recipes/x\n## PR Gate — ...\ncc
@octocat\n<!-- ...state... -->\nz.ts` put that text verbatim into the bot's
comment — forged heading, live third-party mention, and on the NEUTRAL
render (which writes no state block of its own) parseState() returned the
ATTACKER's block, so the next run hit the spend guard and silently skipped
the verdict with no label and exit 0. Three layers:
  (a) flag details go through sanitizeList, exactly like the model's
      strings. Audited every other interpolation into the comment; the rest
      are literals in the file, Number()-coerced, or already sanitized.
  (b) every path regex spells its segment class [^/\n], not [^/] —
      RECIPE_RE, SOURCE_EXT_RE and the test-path check.
  (c) parseState reads the state block only off line 2 of a marker-leading
      comment (where renderComment writes it) and STATE_RE is whole-line
      anchored. A block anywhere else is somebody else's text.

BLOCKING 2 — no exemption, so every release PR was close-lane.
Measured: 40 of the last 40 merged PRs would be close-lane on
missing_screenshot, including every /ship release PR. A check that is red
on every release gets switched off within a week, and then it filters
nothing. The #3745 policy exists to filter INCOMING OUTSIDE CONTRIBUTIONS;
release automation cannot take a screenshot of itself. It is now waived for
OWNER/MEMBER/COLLABORATOR, bot authors and drafts — the usefulness verdict,
the title rule and every mechanical red flag still run, and the sticky
comment says the check was skipped. author_association / draft / user.type
are read from the pr.json the workflow already fetches: no new API call, one
source of truth. `draft` is the one author-settable input, so
ready_for_review joins the trigger list and the exemption is folded into the
spend-guard hash — the draft-era verdict cannot be reused after the flip.
Stated as a deliberate decision in both the workflow header and the script.

3 — DOWNGRADE_FLAG_IDS omitted deletes_tests, adds_symlink and
adds_node_modules, so a PR deleting test/e2e/engine-parity.test.ts kept
merge-lane and a green check on the strength of its prose. All three added.
The old test used deletes_tests as its example of a NON-downgrading flag;
rewritten to pin the stronger invariant instead — every id detectRedFlags
can emit is a downgrade trigger (derived from the detector, so a new flag
fails until it is classified on purpose), and the set is still an allowlist
(an unrecognized id changes nothing).

4 — FENCE_RE backtracks superlinearly on a hostile body: 65KB of backticks
(GitHub's max body length) measured 8.2s across the two policy scans on a
pull_request_target runner. stripCodeFences now caps the scan at 16KB —
same input, 0.40s. Tradeoff documented at the constant: the intent
paragraph and the screenshot both sit near the top in practice (the PR
template puts them in the first two sections, and the model payload already
caps the same body at 6KB), so a real contributor is not judged on a
truncated tail.

Verified: test/pr-gate-workflow.test.ts 126 pass / 0 fail (was 95),
typecheck clean, actionlint clean, check-privacy /
check-no-tracked-symlinks / check-progress-to-stdout /
check-bun-test-timeout / check-key-files-current-state all exit 0. Each new
pin was mutation-tested against the pre-fix behavior: all six mutants fail
the suite.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-04 10:46:09 +08:00
Garry TanandClaude Fable 5 4a00c31b12 fix(ci): gate policy check survives an API outage; merge master; drop NUL separators
Three fixes on the strict PR usefulness gate (#3698), plus the merge that
brings in the policy it enforces.

1. The mechanical policy check now outlives the model. The
   ANTHROPIC_API_KEY guard used to sit above detectPolicyMisses, so a PR
   with no intent paragraph and no screenshot got a green NEUTRAL skip
   whenever the key was absent or Anthropic was down — "wait for a 500"
   was a documented way past the one hard requirement. The #3745 branch
   now sits above the key guard and the spend guard: a policy miss is
   close-lane + the friendly fix-it comment + exit 1 with no API
   dependency at all. A compliant PR that hits a missing key or a dead
   API keeps the round-1 NEUTRAL behavior unchanged (loud comment,
   ::warning::, exit 0, stale gate:* labels cleared) — and the NEUTRAL
   comment now says plainly that the *usefulness verdict* did not run,
   while still reporting the title check and mechanical red flags it was
   able to compute without a model.

2. Merged origin/master, which carries #3745's CONTRIBUTING.md section
   and .github/pull_request_template.md. No conflicts: this branch never
   touched VERSION / package.json / CHANGELOG.md, so master's 0.42.72.1
   carried through untouched — the feature branch adds no version bump.
   The test's inlined pull_request_template fallback (only needed while
   the branch predated the merge) is gone; it now reads the real file, so
   growing the template's own prose past the 40-word bar fails here
   instead of silently letting an untouched template through. The
   CONTRIBUTING_URL deep link is pinned against a GitHub-style slug of
   every heading in the merged CONTRIBUTING.md, with the slugger itself
   pinned so it cannot "pass" against an anchor GitHub never generates.

3. hashInputs joined its three fields with literal NUL bytes, which made
   grep treat the whole of scripts/pr-gate.mjs as binary — any future
   grep-based CI guard over that file would have matched nothing and
   passed silently. Replaced with JSON.stringify of the tuple: still
   unforgeable (each field is quoted and escaped), still stable by
   construction, and printable. `grep -c hashInputs scripts/pr-gate.mjs`
   now returns 2 instead of nothing. Existing sticky-comment state hashes
   are invalidated once, costing one re-verdict per open PR.

Tests: 95 pass / 0 fail in test/pr-gate-workflow.test.ts. The no-API-key
policy-miss case was verified to fail against the pre-fix ordering.
Verified live against the Anthropic API: HTTP 200, strict JSON, all seven
required keys, merge-lane on a compliant fixture.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-04 10:46:09 +08:00
Garry TanandClaude Fable 5 94662eb1e1 feat(ci): gate enforces the #3745 intent-paragraph + screenshot requirement
CONTRIBUTING.md (#3745) requires every PR to carry a paragraph the author
wrote themselves and a screenshot of gbrain actually in use. The gate now
checks both, mechanically, before it spends anything on review.

What it checks (no LLM, both exported for testing):
- hasScreenshot: markdown `![alt](url)`, a bare user-images.githubusercontent
  or github.com/user-attachments/assets URL, or an <img> tag. Anything inside
  a fenced code block does not count — pasting the syntax is not attaching
  the picture.
- hasIntentParagraph: >= 40 words of prose left after stripping fenced code,
  blockquotes, list items, headings, HTML comments, links and the PR
  template's own boilerplate. Per-character scripts are tokenized per
  character, so a paragraph written in Chinese counts as one.

The two lane consequences:
- missing_screenshot OR missing_intent forces close-lane from any recommended
  lane (exit 1) and skips the model call entirely — closed without review is
  the documented consequence, so there is nothing to spend a review on. The
  sticky comment leads with what is missing, how to fix it, and the reopen
  path; both misses are also recorded in the existing "Mechanical downgrades
  applied" section.
- The model's new advisory intent_authenticity verdict forces
  needs-maintainer (exit 0) when it reads "ai_generated", and never
  close-lane on that signal alone. The comment says only that a maintainer
  will read the paragraph personally; the model's reasoning is consumed and
  never published at the contributor.

Rubric + strict-JSON schema gain intent_authenticity and
intent_authenticity_reason, with explicit instructions that rough grammar,
terseness and non-native English are evidence of a HUMAN and that "unclear"
is the answer whenever the evidence is not clear-cut.

test/pr-gate-workflow.test.ts: 63 -> 88 tests, all green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-04 10:46:09 +08:00
Garry TanandClaude Fable 5 2f65ed8da6 fix(ci): harden the PR gate — comment-ownership, output sanitization, deterministic lane downgrades (blind review round 1)
Two independent blind reviews of #3698 (one APPROVE, one REJECT). Every
confirmed finding from the reject side, plus the cheap hardening both
reviews flagged.

BLOCKING

1. Sticky-comment hijack. upsertStickyComment adopted ANY comment containing
   the marker, so a contributor could pre-post `<!-- gbrain-pr-gate -->`, have
   the gate PATCH it, then edit it into a fake green verdict. isOwnComment()
   now requires user.type === 'Bot' AND login === 'github-actions[bot]' AND
   the body to START WITH the marker; anything else gets a fresh comment.

2. LLM output injected as raw Markdown. Model-produced reasons[] and
   reviewer_checklist[] reached the comment unescaped — PR-body-driven
   injection could forge headings, a second marker, and live @mentions.
   sanitizeModelText()/sanitizeList() are now the single choke point in
   renderComment(): HTML comments stripped, mentions zero-width-broken,
   leading block markers removed, newlines collapsed, 300 chars per string,
   8 entries per list, both caps self-marking.

3. Lane was purely model-decided. A well-written feature pitch could talk
   itself into merge-lane. The model now RECOMMENDS; applyMechanicalDowngrades
   forces merge-lane -> needs-maintainer on any of: workflow edits, a new
   package.json dependency, a new src/core/ai/recipes/ provider file, new
   KNOWN_CONFIG_KEYS entries, >40 changed files, >400 net source lines outside
   test/, or a src/ change with no test file touched (#3665). The sticky
   comment reports them under "Mechanical downgrades applied".

4. The test file did not pin what it claimed. Added: no gh pr checkout /
   git fetch / refs/pull / pull/*/head in any spelling; exact permissions
   key->value map plus a single-permissions-block assertion so no job-level
   grant re-widens contents; the ${{ }}-in-run scanner now covers folded
   (`run: >`) and chomped blocks, with a guard-the-guard test; and mocked
   end-to-end runGate() runs for close-lane exit 1, marker hijack, sanitizer,
   truncation, refusal routing, NEUTRAL label clearing, label swap, and the
   spend guard.

5. Version-first title regex rejected the documented suffix form.
   `v0.31.1.1-fixwave fix: ...` now passes. VERSION_AT_END_RE no longer
   false-positives on `chore: bump zod (3.25.76)`: it fires only on a
   v-prefixed or 4-segment trailing version, i.e. this project's own shape.

6. Refusal fail-open. stop_reason=refusal exhausted retries into a green
   NEUTRAL — a deterministic way to dodge the red X. Refusal and
   schema-invalid output now route to needs-maintainer with an explicit
   note; only transport failure stays NEUTRAL. Refusal also short-circuits
   the retry loop, since retrying a deterministic refusal only burns spend.

ALSO

7. persist-credentials: false on the checkout step.
8. Dropped pull-requests:write. Everything the script calls is the issues
   API (comments, label create, label add/remove), so issues:write is the
   only grant that is actually needed.
9. Spend guard for the edited/synchronize amplification: the LLM call is
   skipped when sha256(title+body+head_sha) matches the hash recorded in the
   previous sticky comment's state block, and the stored lane's exit code is
   reused.
10. NEUTRAL runs now clear every gate:* label instead of leaving a stale
    verdict behind.

Verified: bun test test/pr-gate-workflow.test.ts 63 pass / 0 fail,
bun run typecheck clean, actionlint clean, check-privacy /
check-no-tracked-symlinks / check-progress-to-stdout /
check-bun-test-timeout clean, plus a live Anthropic smoke of the exact
request shape (HTTP 200, valid strict JSON, injection attempt rejected).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-04 10:46:09 +08:00
Garry TanandClaude Fable 5 ca1eed3e95 feat(ci): strict PR usefulness gate — merge-lane/close-lane verdict before any review effort (#3698)
Runs on pull_request_target against master. PR code is never checked out
or executed: metadata + a 120KB-capped diff come from the GitHub API only;
the base-repo checkout provides the rubric script. All interpolations are
env-bound; actions SHA-pinned; permissions limited to contents:read +
pull-requests:write + issues:write; per-PR concurrency with cancel.

scripts/pr-gate.mjs (no deps, fetch only) classifies via the strict
usefulness rubric (claude-sonnet-5, strict JSON schema, thinking disabled;
sampling params omitted — rejected on this model), posts one sticky
comment (<!-- gbrain-pr-gate -->), applies exactly one gate:* label, and
exits 1 only on close-lane. Missing ANTHROPIC_API_KEY or API failure after
2 retries NEUTRAL-skips loudly (comment + warning, exit 0). Mechanical
(no-LLM) checks: version-first title rule and diff red flags (>40 files,
node_modules, symlinks, workflow edits force needs-maintainer, new
package.json deps, deleted tests).

test/pr-gate-workflow.test.ts pins the security invariants and unit-tests
the exported title rule + red-flag detector (import side-effect guarded).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-04 10:46:09 +08:00
Sean Gearin 82fe0216ff fix(import): keep stdout JSON-only under --json (#3637) (#3764)
runImport printed its informational lines ('Found N markdown files' and four
siblings) through console.log unconditionally, so `gbrain import <dir> --json`
emitted them ahead of the payload and stdout did not parse as JSON. A consumer
parsing stdout reads that as zero imports while its own bookkeeping records the
files as ingested, so the next run skips them permanently.

Route those five lines through an `info` helper that switches to stderr when
--json is set. The lines are relocated, not removed: human mode is byte-for-byte
unchanged, and the JSON payload itself is untouched. This is the contract
CLAUDE.md already states ('Stdout stays clean for data output (--json
payloads)') and that import.ts's own progress comment repeats.

scripts/check-progress-to-stdout.sh only greps for process.stdout.write('\\r…),
so this class was never guarded; the new spawn-level test pins it.
2026-08-03 19:55:04 +08:00
Masa fa313f2969 fix(durability): gate installHelper's exec-bit chmod on dryRun (#3736) (#3759)
installHelper's "already current" fast path ran chmodSync(helperPath,
0o755) unconditionally, before the dryRun check further down — so
`gbrain sources harden --dry-run` mutated the helper's permissions even
though dry-run is documented as a pure preview. This is the 5th
instance of the #3594 class (see #3692 for the sibling fix to the
step-1 pull, same function).

Gate the chmod with `if (!dryRun)`, mirroring the guard pattern already
used in installLocalHook's own "already current" branch. Non-dry-run
behavior is unchanged.

Adds a regression pair to test/brain-repo-durability.serial.test.ts:
one asserting dry-run leaves a drifted-permission helper untouched,
and a control asserting the real run still restores the exec bit.
2026-08-03 19:37:18 +08:00
2e554500e2 fix(cycle): grade_takes and calibration_profile record the model they actually run — follow the gateway chat model (#3726)
Two cycle phases still carried the label-vs-actual defect that
propose_takes shed in v0.42.62 (and that the #2805 review noted was worth
tracking so it isn't lost):

- grade_takes hardcoded 'claude-sonnet-4-6' into judge_model_id, the
  evidence signature, and budget metering, while the default judge call
  passed NO model hint and rode the gateway's chat_model. On any brain
  with a non-default chat_model, the verdict cache and telemetry recorded
  a model that never ran.
- calibration_profile persisted TIER_DEFAULTS.reasoning to model_id but
  never passed a model to the patterns generator at all, so the recorded
  model and the executed one were unrelated.

Fix, matching the propose_takes convention: each phase resolves ONE
string via explicit override > getChatModel(), and that string drives the
chat call, the cache key, and the stored id.

- grade_takes: the judge hint gets the FULL provider-prefixed string; the
  stored judge_model_id and evidence signature keep the historical bare
  tail. Stock installs are unchanged: getChatModel() defaults to
  'anthropic:claude-sonnet-4-6', whose tail equals the old hardcoded
  value, so no verdict cache invalidates. A genuinely different
  chat_model invalidates, which is correct — the judge really changed.
- calibration_profile: getChatModel() is provider-prefixed, preserving
  the #2451 contract, and its default IS the old TIER_DEFAULTS.reasoning
  value — stock behavior unchanged. The generator now receives the same
  resolved string that is persisted to model_id.

Tests: regression per phase pinning configured-chat-model routing (full
string to the call, bare tail to the cache key for grade_takes, full
string persisted for calibration); all existing suites pass unchanged,
pinning the no-change-on-stock property. 112 tests across the four
affected suites; typecheck clean.

Co-authored-by: Paolo Belcastro <p3ob7o@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-02 20:26:51 +08:00
Masa 630786dc65 docs(integrity): correct the --dry-run line in the subcommand comment (#3739)
The comment listed `gbrain integrity --dry-run` as a subcommand, but
runIntegrity dispatches on `check`/`auto` and parses --dry-run inside
the auto path, so that form exits 1 with "Unknown subcommand". The
runtime help already shows --dry-run indented under `auto [options]`;
only the source comment disagreed.

Addresses #3738.
2026-08-02 20:10:31 +08:00
Masa d92f0dc86f fix(integrity): stop counting dead-link findings as review-queue entries (#3750) (#3751)
The dead-link branch in cmdAuto called logSkip() (writing to
integrity.log.jsonl) but incremented bucketReview, the same counter fed by
the bare-tweet path's appendReview() calls (which write to
integrity-review.md). The printed "Review queue" summary line therefore
included findings that were never written to the review file, so
`gbrain integrity review` silently disagreed with `integrity auto`'s own
summary.

Give dead-link findings their own counter (bucketDeadLink) and print it as
a separate line. No change to bare-tweet bucketing or CLI surface.
2026-08-02 19:54:49 +08:00
a68379050e feat(schema-pack): add slug_filter to retype mapping rules (#2655)
retype mapping_rules' existing path_filter matches pages.source_path,
which is only populated for pages synced from a git repo. Pages ingested
via the put_page MCP tool (or any write path that doesn't go through
sync) have source_path = NULL, so path_filter can never disambiguate
them — a same-from_type retype rule targeting a slug prefix has no way
to address this class of page at all.

slug_filter adds an independent, orthogonal LIKE filter on pages.slug
(the field that's always populated), combinable with path_filter via AND
when both are given. Wired through the full call path: schema validation
(manifest-v1.ts), the RetypeRule interface + probeRule/applyRetypeRule
(retype.ts), and the pack-manifest → RetypeRule conversion in the
unify-types job handler (unify-types-handler.ts) — the last one matters
because a field only present in the zod schema but not carried through
that conversion would validate fine yet silently no-op at execution time.

- manifest-v1.ts: add optional slug_filter to RetypeMappingRuleSchema
- retype.ts: slug_filter on RetypeRule; probeRule + applyRetypeRule both
  apply `AND slug LIKE $N` when present, independent of path_filter
- unify-types-handler.ts: carry rule.slug_filter through the
  pack-mapping-rule → RetypeRule conversion
- tests: skips-outside-slug_filter, matches-despite-NULL-source_path
  (the motivating case), path_filter+slug_filter combined via AND

320/320 existing schema-pack tests pass; 3 new tests added; tsc --noEmit
clean.

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-02 08:12:46 +08:00
6d1232d5a6 v0.42.72.1 docs(contributing): require a human-written intent paragraph + live screenshot on every issue and PR (#3745)
Effective 2026-08-02. Every issue and PR must carry a paragraph the author
wrote themselves explaining why they are opening it, and a screenshot showing
gbrain actually in use in that situation. AI-generated or AI-polished intent
text is not accepted; AI assistance for the code is still fine. Missing either
one means closed without review, reopenable once added.

Stated in CONTRIBUTING.md and pre-filled in both issue templates plus a new
pull request template so the fields are in front of the author.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-02 07:47:23 +08:00
Sina Matian aff34a428a v0.42.72.0 feat(auth): server-enforced slug-prefix write fence for OAuth clients + qm-harness integration guide (#3712)
Registering an OAuth client with --bound-slug-prefixes now makes the write boundary real: writes outside the bound prefixes are refused on every op that can name a page, and ops that write by something other than a slug are refused outright rather than left unfenced. Deny-by-default at dispatch, so a write op added later is refused to bound clients until it is explicitly fenced.

Adds docs/integrations/qm-harness.md (gbrain as the company brain for a qm deployment) with a roster-driven provisioning script and deployment templates, plus a Known limitations section stating plainly that this is a write boundary and not a privacy boundary.

Five review rounds, including three clean-room passes by codex gpt-5.6-sol and Claude Fable 5 against an instruction-stripped tree.
2026-08-02 07:05:13 +08:00
Garry Tan f07e9dff11 fix(ci): release build gates on the artifact, not a serial re-run of the whole suite
The first Release run (30698650484) failed before building anything: the
build job re-ran the entire unit suite serially on the release runner —
a different environment from the sharded Test workflow that had already
gated the exact SHA at merge — and died on ambient-env tests. The review
of #3573 predicted this ('a flaky test now blocks the release'). The
build job now compiles and smoke-tests the binary (--version must match
the VERSION file), which validates the actual artifact — something the
test suite never did. The Test workflow remains the code gate.
2026-08-01 20:43:58 +08:00
Garry Tan 72ec53bb49 v0.42.71.0 feat(ci): publish GitHub releases on every version bump — organized notes + binaries, self-update unbroken (#3521 #3716) 2026-08-01 19:55:28 +08:00
Garry Tan 0244104b8d fix(ci): env-bind the release-notes version interpolation
Review hardening from the #3573 gatekeeper pass: a ${{ }} inside run: is
shell injection by construction; bind through env instead. Not
attacker-reachable today (VERSION comes from master), correct anyway.
2026-08-01 19:55:28 +08:00
Garry TanandTime Attakc 48618bd4bf feat(ci): publish GitHub releases so binary self-update can work (#3521) (#3573)
Co-Authored-By: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 19:55:28 +08:00
Garry Tan dcaea7fdbb fix(wave): revert extractor watermark to merge-day midnight — a future watermark masks concurrent edits (D4)
CI caught what the composite review's watermark 'fix' actually did: the
stamp path clamps links_extracted_at up to the watermark
(GREATEST(updated_at, versionTs), extract.ts:1835), so a future watermark
stamps every page at 2026-08-02 and masks any concurrent edit until then —
the exact race D4 guards — and makes the doctor lag check warn on fresh
brains. The limitation the future date tried to cover (stamps written by
pre-wave code after the watermark) is inherent: no fixed watermark covers
code that keeps running past it. Documented instead.
2026-08-01 10:06:17 +08:00
Garry Tan c6bfab582e v0.42.70.0 fix: community fix wave two — 18 contributed fixes for dead flags, dry-run safety, link resolution, and Windows support 2026-08-01 10:06:17 +08:00
Garry Tan c5ac3efe9f fix(wave): composite-review findings — mask wikilink interiors from the bare-path scan; future-safe extractor watermark
Two cross-PR findings from the wave-i hostile review (Codex xhigh):
1. The #3560-ungated bare-path pass scanned inside [[...]] spans, so a
   dir-qualified wikilink's lowercase prefix (its parent page) became a
   spurious 'markdown' edge whenever the parent existed. Wikilink spans
   are now masked with equal-length blanks before pass 2; discriminating
   test added (fails without the mask).
2. LINK_EXTRACTOR_VERSION_TS was midnight today with a strict-< staleness
   predicate, so same-day stamps from pre-wave code read as fresh and
   never re-extracted. Bumped to 2026-08-02T00:00:00Z.
Plus two comment corrections from both reviewers: upgrade.ts's X1 hook
rationale (stale after #3085) and PGLiteEngine.transaction's tx-engine
db-proxy hazard for #3613's searchVector wrapper.
2026-08-01 10:06:17 +08:00
Garry Tan 0a6b697070 fixup(#3161): pin #1964 assertions to the wikilink-resolved lane
In-wave interaction with #3560: extractPageLinks now also emits raw-literal and bare-path candidates that downstream existence checks drop; the tests' whole-list toEqual predates that. Filter to linkSource='wikilink-resolved' — the PR's resolution + no-cross-dir-leak claims are still fully pinned.
2026-08-01 10:06:17 +08:00
Garry Tan 873587f831 fixup(#3453): regenerate skills.lock.json against current master skills/
The PR's committed manifest predates master's latest SKILL.md edits; regenerated with the PR's own generator per the check's instructions.
2026-08-01 10:06:17 +08:00
Garry Tan 78391e8b64 fix(extract): resolve cross-directory wikilinks via sync-consistent slugification (#3161)
Co-Authored-By: Garry Tan <garrytan@gmail.com>
2026-08-01 10:06:17 +08:00
Garry TanandTime Attakc f08a51d9de fix(extract): stop silently dropping links from non-whitelisted directories (#2576) (#3560)
Co-Authored-By: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 10:06:17 +08:00
Garry TanandTime Attakc 7cbb99ffef feat(doctor): silent-failure check batch — content-hash duplicates, undeclared DB-only pages, heartbeat staleness, db_only collector collision (#2250 #2784 #2787 #2788) (#3457)
Co-Authored-By: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 10:06:17 +08:00
Garry TanandTime Attakc aa5b9e6e2d feat(doctor): tamper-evident skills/ manifest — generator, doctor drift check, CI freshness guard (#159) (#3453)
Co-Authored-By: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 10:06:17 +08:00
Garry Tan 8e3699156f fix(sources): honor unfederate for unqualified reads (#2928) (#3533)
Co-Authored-By: Garry Tan <garrytan@gmail.com>
2026-08-01 10:06:17 +08:00
Garry Tan cc1783c0e4 fix(cli): make --brain actually route to the named brain (#3576)
Co-Authored-By: Garry Tan <garrytan@gmail.com>
2026-08-01 10:06:17 +08:00
Garry TanandDiego e77a15f0d8 fix(windows): use native path boundaries and assertions (#3578)
Co-Authored-By: Diego <diegodearagao@gmail.com>
2026-08-01 10:06:17 +08:00
Garry TanandYMYD 522cfb032a fix(chunker): measured hard-split budgets + token-aware capByChars — the follow-up invited in #3477's merge review (#3564)
Co-Authored-By: YMYD <paul@ymyd.co.kr>
2026-08-01 10:06:17 +08:00
Garry Tananddialthewolff 273bd0e2be fix: scope residual federated reads (#3550)
Co-Authored-By: dialthewolff <jordan@jcwolff.com>
2026-08-01 10:06:17 +08:00
Garry Tan 7522b90db9 fix(search): implement documented relative since/until durations (#3442) (#3539)
Co-Authored-By: Garry Tan <garrytan@gmail.com>
2026-08-01 10:06:17 +08:00
Garry TanandTime Attakc 42f810c406 fix(search): raise hnsw.ef_search to match the vector candidate request (#3613)
Co-Authored-By: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 10:06:17 +08:00
Garry Tan 1ca1a14f24 fix(apply-migrations): stop reporting 'All migrations up to date' while schema is behind (#3085)
Co-Authored-By: Garry Tan <garrytan@gmail.com>
2026-08-01 10:06:17 +08:00
Garry TanandDiego 64c191b1d7 fix(build): force LF for Markdown too (frontmatter parsers anchor on LF) (#3517)
Co-Authored-By: Diego <diegodearagao@gmail.com>
2026-08-01 10:06:17 +08:00
Garry TanandDiego d49ea83db4 fix(minions): settle the spawn loop when a child never launches (#3536)
Co-Authored-By: Diego <diegodearagao@gmail.com>
2026-08-01 10:06:17 +08:00
Garry Tan d71d50503d fix(serve,import): stop the SIGINT orphan; quote imported frontmatter (#3618)
Co-Authored-By: Garry Tan <garrytan@gmail.com>
2026-08-01 10:06:17 +08:00
Garry TanandDiego a228776b8f docs(security): tighten responsible disclosure language (#3619)
Co-Authored-By: Diego <diegodearagao@gmail.com>
2026-08-01 10:06:17 +08:00
Garry TanandMammad M. c0b28f104d fix(sync): prevent dry-run pull and out-of-strategy page deletion (#3624)
Co-Authored-By: Mammad M. <mammad@digit.az>
2026-08-01 10:06:17 +08:00
Garry Tanandcvillarroel2 61ef727710 fix(contextual): route synopsis through configured models (#3678)
Co-Authored-By: cvillarroel2 <20239888+cvillarroel2@users.noreply.github.com>
2026-08-01 10:06:17 +08:00
Garry Tan 532e591655 fix(test): recipe-ollama-dims pins a truncated width — native-width requests now omit the param (#3699)
Master's #1072 test asserted bare qwen3-embedding@1024 threads dimensions:1024;
wave PR #3699 suppresses the param when the request equals the native width
(1024 for the bare id), so the same input now correctly returns undefined —
pinned by dims-qwen3-native.test.ts. Moved this test to 512 to keep its actual
intent (bare id recognized as Matryoshka-capable) without contradicting the
suppression. Third composition defect of the wave; caught by CI shard 8.
2026-08-01 07:40:48 +08:00
Garry Tan 3acd511b80 v0.42.69.0 fix: community fix wave — 22 contributed fixes for silent-failure paths, local-model support, and multi-source routing 2026-08-01 07:40:48 +08:00
Garry Tan 25e4c0c3b1 fix(skills,docs): unify-types playbooks must pass apply:true after the #3574 default flip
#3574 flipped the unify-types worker default to dry-run (jobs.ts:2221,
apply: data.apply ?? false) and updated the architecture docs, but three
agent-facing surfaces still presented the bare submit as the Apply step:
skills/schema-unify/SKILL.md 'Phase 3: Apply', skills/conventions/
schema-evolution.md, and README.md. Because #3545 also edited SKILL.md in
this wave, each PR looked self-consistent in isolation — only the composed
branch shipped a playbook whose apply step silently retypes nothing and
never flips the active pack. Skills distribute downstream via the skillpack,
so this would have propagated. Found by an independent cross-PR review pass.
2026-08-01 07:40:48 +08:00
Garry Tan ba27a186ec fix(budget): reconcile #3691 and #3541 — free local providers are priceable at $0
#3691's regression test used llama-server as an 'unknown provider' example.
#3541 (same wave) prices ollama/llama-server at $0 via FREE_LOCAL_CHAT_PROVIDERS,
so that example is now priceable and the assertion inverted. Swapped in groq —
the paid-but-unpriced case #3691's own description cites — and added the
positive assertion that free local providers keep their cap enforced.
2026-08-01 07:40:48 +08:00
Garry TanandChristoph 4bb313cd80 feat(by-mention): Unicode-aware word tokenizer — Vietnamese/diacritic name extraction (#3563)
Co-Authored-By: Christoph <astaran@herr-der-ringe-film.de>
2026-08-01 07:40:48 +08:00
Garry Tanand老赵 f39b059ad8 test(cli): cover import side-effect guard (#3581)
Co-Authored-By: 老赵 <273731059@qq.com>
2026-08-01 07:40:48 +08:00
Garry TanandKyzcreig 42c2d56df3 fix(import): checkpoint on a time interval and before preserve, not only every 100 files (#3585)
Co-Authored-By: Kyzcreig <9063726+Kyzcreig@users.noreply.github.com>
2026-08-01 07:40:48 +08:00
Garry TanandVyacheslav Zakharov e3806cf46f fix(test): kill timeout-resistant unit shards (#3631)
Co-Authored-By: Vyacheslav Zakharov <vyacheslav.zakharov@avers.kz>
2026-08-01 07:40:48 +08:00
Garry TanandAlexey a82a83dbc3 fix: health metric that cannot converge, and facts lost to a row_num collision (#3634)
Co-Authored-By: Alexey (CTO) <cto@phrase.local>
2026-08-01 07:40:48 +08:00
Garry TanandBen Young 03de3246f3 fix(budget): price free local chat providers at $0, matching embed and rerank (#3541)
Co-Authored-By: Ben Young <Grimnoth@users.noreply.github.com>
2026-08-01 07:40:48 +08:00
Garry TanandAustin Wilhite 37ad1d2104 fix(cycle): stop extract_atoms silently skipping all work on unpriced models (#3691)
Co-Authored-By: Austin Wilhite <austinw80@gmail.com>
2026-08-01 07:40:48 +08:00
Garry Tanandzenspam aeb75dc839 fix(dims): skip dimensions param when it equals qwen3-embedding native width (#3699)
Co-Authored-By: zenspam <zenspam@gmail.com>
2026-08-01 07:40:48 +08:00
Garry TanandTime Attakc 298b6b01b8 fix(docs): remove references to the nonexistent gbrain install command (#3502) (#3545)
Co-Authored-By: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 07:40:48 +08:00
Garry Tan d6f929bfbd fix(embed): stop one bad chunk darkening a page, and exit non-zero on failures (#3037) (#3532)
Co-Authored-By: Garry Tan <garrytan@gmail.com>
2026-08-01 07:40:48 +08:00
Garry TanandRyan Ayers 9c1a4b8fce fix(cycle): surface stamp-write failure on CycleReport and degrade status (#3504) (#3589)
Co-Authored-By: Ryan Ayers <rayers@dividia.net>
2026-08-01 07:40:48 +08:00
Garry Tan 3523d8fd7e fix(jobs): unify-types worker defaults to dry-run per its handler contract (#1575) (#3574)
Co-Authored-By: Garry Tan <garrytan@gmail.com>
2026-08-01 07:40:48 +08:00
Garry TanandTime Attakc a3000d4630 fix(skillpack): resolve gbrain root from module path when cwd walk fails (#3144)
Co-Authored-By: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 07:40:48 +08:00
Garry TanandBrett 64ad743d98 fix(models): hint the /v1 base-URL suffix when a doctor chat probe 401s on an openai-compatible proxy (#3553)
Co-Authored-By: Brett <brettdavies@users.noreply.github.com>
2026-08-01 07:40:48 +08:00
Garry TanandTime Attakc 9ada48e600 fix(doctor): bound embedding provider health probe (#3364)
Co-Authored-By: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 07:40:48 +08:00
Garry TanandTime Attakc aa05255887 fix(doctor): stop reporting Windows-drive image paths as missing under WSL (#1835) (#3523)
Co-Authored-By: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 07:40:48 +08:00
Garry TanandTime Attakc 241603aab8 fix(dream): honor a configured 0 in synthesize + auto_think config resolution (stop coercing to the default) (#3552)
Co-Authored-By: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 07:40:48 +08:00
Garry TanandTime Attakc 69be8bb707 fix(minions): reconnect the stall timer, stop failJob masking job errors, unify retry matchers (#1720) (#3555)
Co-Authored-By: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 07:40:48 +08:00
Garry TanandTime Attakc 1116a95926 fix(entities): resolveEntitySlug fallback keeps the path separator (#3447) (#3567)
Co-Authored-By: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 07:40:48 +08:00
Garry TanandTime Attakc c8ea38421a fix(cycle): log swallowed lock.release() failures (#1470) (#3572)
Co-Authored-By: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 07:40:48 +08:00
Garry TanandTime Attakc 163a83baa3 fix(operations): sync_brain threads ctx.sourceId to performSync (#2830) (#3568)
Co-Authored-By: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 07:40:48 +08:00
Garry TanandTime Attakc f7295e3308 fix(cli): route init --help to its own usage text (#3652)
Co-Authored-By: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 07:40:48 +08:00
f84bfb57f2 fix(config): render object-valued fields as JSON in config show (#575) (#3575)
Non-string values interpolated into the template literal printed
'[object Object]' (e.g. provider_base_urls). Objects now render via
JSON.stringify; objects under a sensitive key redact to '***' like
their string counterparts.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 05:03:23 +08:00
ad7114f0ad fix(test): make typecheck hermetic to ambient parent openclaw SDK types (#2729) (#3641)
The real plugin-load e2e test dynamically imports 'openclaw/plugin-sdk'.
TypeScript still resolves and type-checks that bare specifier via upward
node_modules resolution, so 'bunx tsc --noEmit' on a clean checkout could
fail (TS2339 on sdk.registerContextEngine) or pass depending on whichever
undeclared openclaw package happened to exist in an ancestor directory.
The existing @ts-ignore only covered the import line, not the property
access on the following line.

Cast the awaited import to a local structural interface declaring the
one member the test uses (registerContextEngine, optional). TypeScript
never consults the ambient module's types for the property access, so
typecheck output is identical regardless of ancestor node_modules state.
The @ts-ignore stays on the import statement itself and stays @ts-ignore
(not @ts-expect-error) because whether TS2307 fires there is itself
ambient-dependent. Runtime behavior is unchanged: the cast erases at
compile time and the export's presence is still verified at runtime.

Fixes #2729

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 05:03:12 +08:00
23003a2163 fix(facts): preserve remote fence writes (#3659)
Co-authored-by: gbrain contributor <contributor@example.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 05:03:00 +08:00
4c0ec60275 fix(cycle): thread cycleSourceId into the schema-suggest phase (#3701)
The phase was calling runSchemaSuggestPhase(engine, { dryRun }) with no
sourceId, so it silently fell back to 'default' on every source's dream
cycle -- same bug class as upstream #1586 (synthesize) and #2666
(patterns/synthesize), just an undiscovered instance for this phase.
Confirmed live: schema-events audit log shows only source=default across
41 entries this week despite calendar/mail/mem/social cycles all running
the phase.

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 05:02:48 +08:00
Masa 13d95ba0ab fix(schema): apply mutation batches atomically so a mid-batch failure leaves the pack untouched (#2581) (#3446)
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain. 19 of 62 PRs cleared that bar.

Applying a schema mutation batch was not atomic: a failure partway through left earlier mutations permanently written. Reproduced on disk — a failure at index 2 left mutation 0 applied with no way to tell from the pack's state that it was half-done. The fix validates the whole batch first and writes once, which makes partial application impossible by construction rather than by careful ordering.

Verified before merge: the failure was reproduced by injecting one rather than reasoning about it; the PR's own tests fail when the fix is reverted; typecheck clean; MERGEABLE/CLEAN at 22/22 on the current base after batches 1-4 landed.

Sequenced last deliberately — it collides with #3531 on docs/architecture/KEY_FILES.md and with #3667 on src/core/operations.ts, both of which landed earlier today.

Known gap, recorded rather than hidden: lock contention under concurrent writers was reasoned about, not stress-tested.
2026-08-01 04:09:12 +08:00
Time Attakc 022a443e9b fix(pricing): correct voyage-4-large rate and add the missing voyage-4 family entries (#3480)
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain. 19 of 62 PRs cleared that bar.

voyage-4-large was billed at the voyage-3-large rate — $0.18 against a published $0.12 — so every cost estimate using it was wrong by 50%. Corrected in the canonical table only, per CLAUDE.md's rule that every other pricing table is a derived view, and the drift guard passes. Rate checked against the live vendor page.

Verified before merge: the PR's own tests fail when the production change is reverted; typecheck clean; MERGEABLE/CLEAN with 22/22 checks on the current base after batches 1-3 landed.

Known gap, recorded rather than hidden: this PR previously failed the JSONB parity guard on a 32-commit-stale base. I rebased it onto current master and re-ran rather than accepting 'flaky' — the guard passes on the real base, 22/22 green.
2026-08-01 03:53:41 +08:00
Masa e7439828f1 fix(migrate): apply the #1178 invalid-index-guard fix to 10 more historical sites (#3192)
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain. 19 of 62 PRs cleared that bar.

Mechanically applies merged #3191's `dropInvalidConcurrentIndex` to the 10 remaining historical migrations that still had the broken DO-block form. Migration ordering and numbering are untouched — this only changes how each guards its own index creation.

Verified before merge: the PR's own tests fail when the production change is reverted; typecheck clean; MERGEABLE/CLEAN with 22/22 checks on the current base after batches 1-3 landed.

Known gap, recorded rather than hidden: verified by sequence inspection and the migration suite rather than by replaying all 120 migrations against every engine.
2026-08-01 03:53:37 +08:00
Time Attakc 5773736c63 fix(doctor,docs): warn that the npm name 'gbrain' is unrelated + detect a shadowing npm install (#505) (#3454)
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain. 19 of 62 PRs cleared that bar.

The npm package named `gbrain` is an unrelated squatted package, so `npm install gbrain` gives users something that is not this project. Adds doctor detection that classifies real checkouts correctly, fails open, and is try/catch'd throughout. Classification rests on the bin-shape marker since this repo has no `repository` field — verified e2e.

Verified before merge: the PR's own tests fail when the production change is reverted; typecheck clean; MERGEABLE/CLEAN with 22/22 checks on the current base after batches 1-3 landed.

Known gap, recorded rather than hidden: the remediation commands were not executed against a real global install, and the Windows `which -a` path is unexercised.
2026-08-01 03:53:33 +08:00
cybernaut6404 56454c6ba8 fix(index): preserve code files containing NUL bytes (#3483)
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain. 19 of 62 PRs cleared that bar.

Code files containing raw NUL bytes hard-failed UTF-8 encoding on import, so they silently never indexed. Sanitized at the single choke point both callers route through, with offsets kept in one coordinate space, and exercised end-to-end on a real engine.

Verified before merge: the PR's own tests fail when the production change is reverted; typecheck clean; MERGEABLE/CLEAN with 22/22 checks on the current base after batches 1-3 landed.

Known gap, recorded rather than hidden: follow-up to file: reindex-code hash ping-pongs on NUL-containing files — reproduced, bounded, and causes no data loss.
2026-08-01 03:53:29 +08:00
alexey-metaengage 63e79838b9 feat(recipes): declare Gemini embedding batch-token budget (#3651)
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain. 19 of 62 PRs cleared that bar.

The google embedding recipe declared no batch caps, so it rode the no-cap fast path with error backstops shaped for Voyage and OpenAI. Caps verified by behavioral probe — 40 texts split into 3 sub-batches matching the declared math. Sequenced after #3531, which touched the same recipe file.

Verified before merge: the PR's own tests fail when the production change is reverted; typecheck clean; MERGEABLE/CLEAN with 22/22 checks on the current base after batches 1-3 landed.

Known gap, recorded rather than hidden: Gemini's actual 20k limit was taken from vendor docs rather than a live call; being wrong in either direction is bounded by the cap itself.
2026-08-01 03:53:25 +08:00
Masa 3062859420 fix(autopilot): derive the full-cycle timeout floor from the handler anchors (#2781) (#3656)
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain. 19 of 62 PRs cleared that bar.

Full-cycle maintenance jobs were stamped with the outer 600s timeout instead of the 30-minute handler anchor — a regression from #3338 that killed long cycles mid-run. Fixed with a named `fullCycleTimeoutMs` derived from the handler anchors, which now fail loudly rather than silently defaulting; reverting fails 3 of 8 tests.

Verified before merge: the PR's own tests fail when the production change is reverted; typecheck clean; MERGEABLE/CLEAN on the current base after batches 1 and 2 landed, not a stale one.

Known gap, recorded rather than hidden: the '38 dead cycles in 24h' figure from the description was not reproduced; the stamp arithmetic was verified by code inspection.
2026-08-01 03:39:20 +08:00
Masa addf03119d fix(recipes): align the X secret name with the resolver — X_API_BEARER_TOKEN (#2789 defect 2) (#3649)
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain. 19 of 62 PRs cleared that bar.

The docs and recipe pinned `X_BEARER_TOKEN` while the resolver only ever read `X_API_BEARER_TOKEN` — so no single name worked and the integration could not be configured by following its own documentation. Renamed the dead documented side; reverting fails exactly 2 of the 3 new tests.

Verified before merge: the PR's own tests fail when the production change is reverted; typecheck clean; MERGEABLE/CLEAN on the current base after batches 1 and 2 landed, not a stale one.

Known gap, recorded rather than hidden: no live X API call was made.
2026-08-01 03:39:16 +08:00
daragao3 dba0ae7b1e fix(validation): qualify backlink endpoint identity (#3667)
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain. 19 of 62 PRs cleared that bar.

The back-link validator compared bare slugs, so a same-slug page in another source masked a genuinely missing reverse edge — silent under-reporting in exactly the multi-source setup where it matters. Now keyed on the full 4-tuple, per the `(source_id, slug)` uniqueness invariant. Verified on real Docker Postgres with 28/28 parity, which the PR itself had skipped.

Verified before merge: the PR's own tests fail when the production change is reverted; typecheck clean; MERGEABLE/CLEAN on the current base after batches 1 and 2 landed, not a stale one.

Known gap, recorded rather than hidden: remote MCP serialization of the additive Link fields is untested; the fields are additive JSON.
2026-08-01 03:39:12 +08:00
Masa 7376c0266e fix(integrations): resolve secrets through buildGatewayConfig's config→env folding (#3648)
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain. 19 of 62 PRs cleared that bar.

`integrations show` printed `[missing]` for config-plane keys that the runtime gateway resolves perfectly well — so the status display disagreed with reality and sent people hunting for a problem that did not exist. Fixed with a single `secretEnv()` helper at all four read sites, preserving precedence. The spawn environment is deliberately left unchanged, which is the correct posture. Sequenced after #3531, which refactored the `buildGatewayConfig` internals this consumes.

Verified before merge: the PR's own tests fail when the production change is reverted; typecheck clean; MERGEABLE/CLEAN on the current base after batches 1 and 2 landed, not a stale one.

Known gap, recorded rather than hidden: the full-suite env-mutation interaction was not run locally; CI shards are green.
2026-08-01 03:39:07 +08:00
Sean Gearin 002ac8050f fix(search): classify first-person "what do I know about X" as an entity query (#3615) (#3616)
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain. 19 of 62 PRs cleared that bar.

One-character regex fix with a real user-visible effect: "what do **I** know about X" was classified as a general query while the you/we phrasings were correctly classified as entity queries. The new alternation is a strict superset, so no previously-matching phrasing changes, and stubbing the old regex back fails at the exact assertion. Closes verified issue #3615.

Verified before merge: the PR's own tests fail when the production change is reverted; typecheck clean; MERGEABLE/CLEAN on the current base after batch 1 landed, not a stale one.
2026-08-01 03:26:22 +08:00
Javier Aldape f75dbb4ed6 fix(search): give query embeds a fresh floored deadline (#3690)
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain. 19 of 62 PRs cleared that bar.

The shared AbortSignal arrived already aborted, which made the 2s embed floor dead code and silently degraded hybrid search to keyword-only — users got results that looked complete and were not. Fixed with a fresh AbortSignal.timeout(remaining) at the single shared seam; stubbing the old behavior back fails exactly the new test. This also closes verified issue #2028.

Verified before merge: the PR's own tests fail when the production change is reverted; typecheck clean; MERGEABLE/CLEAN on the current base after batch 1 landed, not a stale one.

Known gap, recorded rather than hidden: the DATABASE_URL e2e claims in the description were not re-run, though no SQL is touched.
2026-08-01 03:26:18 +08:00
Masa 437889c0bd fix(cycle): interleave transcript/page work items so a budget cap can't starve the doctor-visible page backlog (#3384)
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain. 19 of 62 PRs cleared that bar.

Transcripts-first work ordering permanently starved the doctor-visible page backlog whenever the budget cap bit — the pages never got reached. Page-first interleave at the single merge point, with spend proven order-independent, and stubbing the old ordering back fails 4 of 5 tests. Landing first among the extract-atoms.ts PRs, so #3691 and #3654 rebase onto it.

Verified before merge: the PR's own tests fail when the production change is reverted; typecheck clean; MERGEABLE/CLEAN on the current base after batch 1 landed, not a stale one.

Known gap, recorded rather than hidden: no real-LLM budget run; the identical error path was driven synthetically.
2026-08-01 03:26:12 +08:00
Time Attakc 335e470394 fix(ai): fold dashscope + google keys into gateway env, drop the retired Gemini default (#3500, #3510) (#3531)
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain. 19 of 62 PRs cleared that bar.

dashscope and google keys now fold into the gateway env — verified live end-to-end — and the retired gemini-1.5-pro default is swept from 9 files. Reverting the change fails 11 of 98 tests at fixed seams, and the budget-cap claim in the description reproduced. Landing first in the gateway/config-key seam, so #3648 rebases onto it.

Verified before merge: the PR's own tests fail when the production change is reverted; typecheck clean; MERGEABLE/CLEAN on the current base after batch 1 landed, not a stale one.

Known gap, recorded rather than hidden: no live provider call was made; the gemini retirement was taken from issue history rather than a vendor check. Minor follow-up to file: deriveEnvKey('google_api_key') yields a dead GOOGLE_API_KEY in the minion shell-inherit path.
2026-08-01 03:26:09 +08:00
Masa 7e21e47c15 fix(extract): stop asserting works_at from bare people/->companies/ adjacency (#3466) (#3642)
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain.

19,497 false `works_at` edges were being asserted from bare people/→companies/ directory adjacency. This is the exact fix prescribed when #3495 was closed — bare `mentions` plus an extractor version bump — and stubbing the old behavior back fails the new test. Retroactive cleanup of already-written rows is explicitly out of scope; filing that follow-up.

Verified before merge: the PR's own tests fail when the production change is reverted (11 of the previous 32 PRs failed exactly there — one had 7 of 8 new tests passing on master); typecheck clean; MERGEABLE/CLEAN with 22/22 checks green on the current base, not a stale one.
2026-08-01 03:11:55 +08:00
Paolo Belcastro bf7d706bb4 fix(doctor): probe-health 'Latest' reports the true newest run — sort audit events chronologically (#3366)
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain.

doctor's probe-health "Latest" tail-picked the oldest cross-week event instead of the newest. Chronological sort at the reader seam, matching the writer's own documented contract, with all 14 consumers audited. Follow-up to file: `doctor.ts:1017` `self_upgrade_health` has the identical bug class.

Verified before merge: the PR's own tests fail when the production change is reverted (11 of the previous 32 PRs failed exactly there — one had 7 of 8 new tests passing on master); typecheck clean; MERGEABLE/CLEAN with 22/22 checks green on the current base, not a stale one.
2026-08-01 03:11:51 +08:00
Masa bb69aa8b65 fix(cli): stop rerouting sync --timeout into dispatchReadOnlyCommand (#3013) (#3650)
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain.

Both #3013 defects reproduced live: `sync --dry-run --timeout` reported "unsupported command", and a bare `--timeout 60` was mis-scaled to 60ms. Fixed with a per-command dispatch gate plus timeout handback to its two owners, with every `cliOpts.timeoutMs` consumer audited. Landing first in the cli.ts and sync.ts clusters.

Verified before merge: the PR's own tests fail when the production change is reverted (11 of the previous 32 PRs failed exactly there — one had 7 of 8 new tests passing on master); typecheck clean; MERGEABLE/CLEAN with 22/22 checks green on the current base, not a stale one.
2026-08-01 03:11:47 +08:00
paul-0320 b4a9c7683d fix(search): fold the FTS configuration name into knobs_hash — stop stale rows surviving a reindex-search-vector language switch (#3677)
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain.

`GBRAIN_FTS_LANGUAGE` was absent from the query-cache key, so a language switch served stale pre-switch rows. The hash now folds it (v14→15) with all five pin sites updated; reverting the fix fails 4 of 15 tests at the exact claimed step. Landing first in the knobs_hash cluster — the constant is single-writer, so #3617 rebases onto this and takes 16.

Verified before merge: the PR's own tests fail when the production change is reverted (11 of the previous 32 PRs failed exactly there — one had 7 of 8 new tests passing on master); typecheck clean; MERGEABLE/CLEAN with 22/22 checks green on the current base, not a stale one.
2026-08-01 03:11:42 +08:00
Time Attakc 3c61e25503 fix(reindex-frontmatter): reuse the connected engine instead of self-deadlocking on the PGLite lock (#1963) (#3558)
Adversarial review: survived two independent refuters — the only PR of 32 reviewed this way to do so.

The bug: `gbrain reindex-frontmatter` and `gbrain backfill <kind>` were 100% dead on PGLite. cli.ts takes the data-dir lock, the command modules built a second engine on the same dir, and acquireLock never reaps a live PID — 30s timeout, exit 1, with the error naming the waiting process itself as the holder. Reproduced on the parent commit at 33.2s; passes in 5.0s with the fix. Root-cause fix at the dispatch layer, not a softening of the lock, and the sibling census confirmed these were the only two affected callers.

Postgres path verified before merge (it was the review's one open gap, since the bug is PGLite-only and all verification had gone there while the change itself is connection-teardown ownership). Against real Postgres 16 + pgvector: reindex-frontmatter and all three registered backfills exit 0 with zero residual connections, zero advisory locks, and zero cycle-lock rows — byte-identical output and identical teardown to master on the same database, confirming the change is behavior-neutral there.

Merged tree re-verified after rebase: typecheck clean, pglite-lock + reindex-frontmatter 16 pass, llms bundle fresh, 23/23 CI green.
2026-07-31 10:10:57 -07:00
c6dc0adf26 fix(test): repair typecheck failures in admin-sse and lifecycle tests (#3598, #3599) (#3610)
Export AdminSseResponse, HttpServerLifecycle, and SignalSource from
serve-http.ts so test fakes can reference them. Cast structural fakes
through `as unknown as T` where the fake return types (EventEmitter,
plain object) cannot structurally match the full Node/Express originals.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Mastra Code (anthropic/claude-opus-4-6) <noreply@mastra.ai>
2026-07-29 17:12:08 -07:00
daragao3andClaude 945fed6105 fix(engine): enforce static engine-live import boundaries (#3596)
* docs: design engine dynamic-import reconciliation

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(engine): reconcile dynamic import hardening

Co-Authored-By: Claude <noreply@anthropic.com>

* test(engine): guard dynamic import policy

* docs: plan engine dynamic-import reconciliation

Record the approved TDD sequence for selective engine-path hardening,
repository guard wiring, documentation, and local verification. Preserve
the no-version-bump and no-publication boundaries for the remaining work.

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(engine): record static import invariant

* fix(engine): parse block comments in import guard

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(engine): parse dynamic imports with TypeScript

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(engine): close import guard bypasses

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(engine): close parser guard edge cases

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(engine): aggregate parser diagnostics

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(engine): bound dynamic import marker directive

Require the line-level opt-out marker to be standalone inside real comment trivia so negated or incidental longer tokens cannot authorize an import. Preserve the existing general marked-line contract and pin it with focused regression coverage.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(engine): close Unicode marker boundary bypasses

Treat Unicode identifier continuations as marker-token characters and inspect adjacent text by code point so supplementary-plane characters cannot turn longer comment tokens into approvals.\n\nCo-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-07-29 16:08:08 -07:00
Mikhail Merkulov a175dd0047 fix admin SSE handshake through reverse proxies (#3598) 2026-07-29 16:07:59 -07:00
Mikhail Merkulov 2118f02fc7 fix HTTP server lifecycle retention (#3599) 2026-07-29 16:07:50 -07:00
Sean Gearin a12ab5eabc fix(import): quote frontmatter values and omit absent conversation id (#3600)
Review follow-up to #3549.

An envelope is a third-party file, so interpolating source_provider raw let a
provider string carrying a newline close the scalar and inject arbitrary
frontmatter keys. title: on the line above was already quoted; source: now
matches it.

memvelope_conversation_id emitted the literal string undefined when a
conversation carried no id, which asserts a value rather than reporting
absence: every id-less conversation claims the same id, so anything grouping
or deduping on that key merges unrelated pages. The key is now omitted.

Filename and frontmatter read one hasId predicate so they cannot disagree
about whether an id exists. Tests add both cases; the injection case parses
emitted frontmatter with js-yaml rather than substring-matching it.
2026-07-29 16:07:42 -07:00
Tony Guan e98249a624 fix(files): display zero-byte file sizes (#3608) 2026-07-29 16:07:32 -07:00
Sean Gearin 1057bf4368 feat(import): standalone importer seeding a brain from envelope-v0 chat-history files (#3549)
One Markdown page per conversation from an envelope-v0 file (format spec:
github.com/memvelope/memvelope), written into a directory gbrain sync
ingests. Zero dependencies, deterministic, no network; does not call gbrain.

Filenames are date + conversation id (collision-proof natural key; duplicate
ids overwrite their own file and warn on stderr). Frontmatter carries
type: conversation, source provider, conversation id, and origin. Bodies keep
message-id citations per speaker turn.

Ships as script + test + fixture only; usage and verification steps live in
the script header.
2026-07-29 11:56:20 -07:00
913d2d7f79 fix(test): give slow setup hooks a real timeout budget (#3566)
bun ignores bunfig.toml's timeout key, and beforeAll/beforeEach hooks do
NOT inherit a test's third-arg timeout — a bare `bun test` gives every
hook the 5000ms default even when all tests in the file declare 30s+.
Measured on bun 1.3.14: a 6s hook dies at ~5001ms with the signature
`(unnamed) [5001ms] ... hook timed out` (the #3545 jsonb-parity CI
failure); both `beforeAll(fn, ms)` and the CLI `--timeout` flag are
enforced hook budgets (kills observed at exactly the configured ms).

Fixes:
- e2e.yml (jsonb-parity, tier1, tier2) and release.yml ran bare
  `bun test`; they now pass --timeout=60000 like every scripts/ runner.
- test/e2e/jsonb-roundtrip.test.ts (the #2339 double-encode guard, which
  only real Postgres can surface) additionally carries per-hook 60s
  budgets so a bare local run can't flake either — same pattern as its
  sibling op-checkpoint-jsonb-parity.test.ts.
- scripts/check-bun-test-timeout.sh: CI guard (run from test.yml's
  verify job) failing any future bare `bun test` in workflows/scripts.
- scripts/run-e2e.sh: correct the comment claiming --timeout is
  per-test-only (it covers hooks; the outer gtimeout exists for
  sync-blocking WASM hangs where no timer can fire).

Proof: with Postgres paused for 6s during setupDB's connect, the
unfixed file fails at 5001.81ms with the exact CI signature; the fixed
file passes the identical condition (5 pass, 6.57s). 396 slow
before-hooks across 362 test files lack per-hook budgets; all of them
run through --timeout-passing invocations after this change, enforced
by the new guard.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-07-28 20:08:36 -07:00
f9349ba07f fix(doctor,cycle): stop permanent cycle_freshness FAILs on multi-source installs (#2540) (#3562)
Closes the paths #3382 left open (its author said it narrowed the issue
rather than closing it):

1. checkCycleFreshness iterates EVERY local_path source, so an install
   that nightly-dreams one vault via --dir showed a permanent FAIL for
   every other federated source — and for any source added minutes ago.
   'Never completed a full cycle' is now a WARN with the dream/autopilot
   hint; a source that HAS cycled and then went stale still escalates
   through the 6h warn / 24h fail thresholds (the regression signal the
   check exists for). This is the reporter's actual case: the permanent
   red eroded doctor's signal until real staleness hid inside it.

2. resolveSourceForDir's exact-match lookup had no archived filter and
   no ORDER BY, so an archived (or duplicate) alias of the same path
   could shadow the active source; dream's archived guard then refused
   the stamp and the ACTIVE source stayed unstamped forever. The lookup
   now excludes archived rows and orders deterministically, matching
   the canonical-path fallback's posture. The fallback's fail-closed
   ambiguity handling is deliberately unchanged.

3. #3382's own regression test (ii) was environment-sensitive: it
   assumed unsetting OPENAI_API_KEY/ANTHROPIC_API_KEY makes the embed
   phase fail, which is false wherever another embedding provider
   resolves (the cycle then reports 'clean' and the test flips). It now
   fails the sync phase against a vanished checkout — deterministic on
   every machine, same property pinned (a genuinely failing enabled
   phase must prevent the stamp).

New pins fail on unmodified master and pass here: never-cycled→warn
(x2, doctor) and the archived-alias shadow (dream --dir stamp).

Fixes #2540

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-07-28 19:50:44 -07:00
MasaandTime Attakc e72d93fdb5 fix(sync): reconcile the stale old row when a rename falls back to add (#3056) (#3479)
master's rename loop swallows updateSlug failures with an empty catch
("treat as add"), and updateSlug returns void — so a zero-row UPDATE
(old slug absent) and a thrown collision are both invisible. Either way
the run falls through to importFile at the new path while the old row
stays behind live: slug occupied, 0 chunks after the next embed pass,
page count unchanged. A rename that didn't rename, with no trace.

The fix reconciles the duplicate:

- updateSlug returns the number of rows moved in both engines (a
  zero-row UPDATE does not throw; the count is the only way to see it).
- When the cheap rename didn't move a row AND the destination
  demonstrably materialized — imported, or an errorless skip AT the new
  slug (NOT an identity-dedup skip against the old row, which would mean
  nothing landed and deleting the old row would destroy the only copy) —
  the stale row is located positively by source_path = from and deleted.
  No source_path match → nothing is deleted (code-strategy imports don't
  populate source_path and fall back safely to leaving the row).
- A failed reconcile delete records a <rename:…> sentinel: the failure
  gate hard-blocks the bookmark, the auto-skip valve can never
  chronic-skip it (which would bank the duplicate permanently after a
  multi-run outage), and the rename is not checkpointed — the next run
  retries the same diff and clears the sentinel on convergence.

Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-07-28 19:32:55 -07:00
Time AttakcandGarry Tan 85286a556c fix(search): stop boosting compiled_truth at default detail (#3430) (#3514)
* fix(search): stop boosting compiled_truth at default detail (#3430)

COMPILED_TRUTH_BOOST = 2.0 is applied AFTER RRF normalization, and RRF's whole
dynamic range over a 100-deep pool is 1/60 -> 1/160 (a factor of 2.67). So a
2.0x multiplier consumes roughly three quarters of the range: break-even is
`2/(60+r) >= 1/60`, i.e. r <= 60, which means ANY boosted chunk inside the
first 60 ranks outranks an unboosted rank-1 chunk. That is a categorical
filter, not a tilt.

Measured against master's own rrfFusion, with the correct answer in a
fenced_code chunk at vector rank 0:

  compiled_truth chunks in pool | final rank | in top-20
  10                            | 10         | yes
  20                            | 20         | NO
  40                            | 40         | NO
  80                            | 59         | NO

With the boost off the answer stays at rank 0 in every case.

The gate was spelled `detail !== 'high'` -- written as though `high` were the
special case. The documented contract in src/core/operations.ts is
"low (compiled truth only), medium (default, all with dedup), high (all
chunks)", which makes LOW the special one: `low` already restricts to
compiled_truth, so a boost there is a no-op among equals, while `medium` and
`high` are both meant to see everything. So the default detail was silently
compiled-truth-only, contradicting the op's own description.

Three changes:

1. The three fusion call sites now route through a named predicate,
   `shouldBoostCompiledTruth(detail)`, returning true only for 'low'.
   Extracted rather than left inline precisely because an inline expression is
   only reachable through a full hybridSearch round trip -- which is why the
   inversion went unnoticed. The predicate is directly unit-testable.

2. KNOBS_HASH_VERSION 13 -> 14. Results are cached AFTER fusion, so rows
   ranked under the old semantics would otherwise be served under the new ones
   for the whole TTL (3600s default). One-time miss spike on upgrade.

3. test/search-compiled-truth-boost-scope.test.ts pins both the mapping and
   the arithmetic, and documents the displacement it prevents.

Verified the tests discriminate: stubbing the OLD predicate body into master
(so the failure is behavioral rather than a missing export) gives 4 fail /
3 pass; with the fix, 7 pass. typecheck clean, verify 32/32, and 144 pass /
0 fail across the search + fusion + cache suites.

* fix(test): update the three remaining KNOBS_HASH_VERSION pins to 14 (#3430)

Missed in the first pass because I ran a targeted set of test files instead of
the full suite. CI shards 3, 8 and 10 caught them:

  test/search/knobs-hash-reranker.test.ts:67
  test/cross-modal-phase1.test.ts:139,149
  test/search-alias-resolved-boost.test.ts:93

Each carries the running history of why the version moved, so each gets the
13→14 rationale appended rather than just the number swapped. No pins at 13
remain anywhere in test/.

---------

Co-authored-by: Garry Tan <garrytan@gmail.com>
2026-07-28 19:27:46 -07:00
a8a3b6df9f fix(engine): exclude soft-deleted pages from getHealth counts (#1305) (#3556)
getStats() has excluded soft-deleted pages since v0.26.5, but getHealth()
kept counting raw pages rows: page_count, the islanded/orphan scan, the
entity_pages CTE (link/timeline coverage denominators), and most_connected
all included deleted pages, so brain_score never moved when a user
soft-deleted pages. Repro: 50 pages, soft-delete 40 -> getStats 10 vs
getHealth 50, orphan_pages 50, brain_score byte-identical.

Fix: every page-scoped count in getHealth now filters deleted_at IS NULL,
identically in both engines (engine-parity SQL shapes match).

Deliberate boundary: chunk/link storage counts (embed_coverage,
missing_embeddings, link_count, dead_links) stay raw until the purge phase
runs — matching getStats' documented posture — and destructive-removal
counts (#2235) deliberately keep counting all rows. stale_pages already
filtered via buildStalePagesWhere.

Test: test/health-soft-delete.test.ts — 3 of 4 tests fail behaviorally on
unmodified master (page_count 10 vs 4, orphan_pages 8 vs 0, link_coverage
0.5 vs 1), all pass with the fix.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-07-28 19:14:39 -07:00
316 changed files with 21737 additions and 1689 deletions
+13
View File
@@ -14,3 +14,16 @@
# through bash. `eol=lf` pins the checkout regardless of the user's
# core.autocrlf setting.
*.sh text eol=lf
# Markdown gets the same pin, for a different failure mode: the frontmatter
# parsers anchor on LF. Under a CRLF checkout the opening fence becomes
# "---\r\n", which an LF-only /^---\n/ (or a startsWith("---\n")) does not
# match, so a well-formed document silently parses as having no frontmatter.
# There is no error -- the field just comes back empty. That has surfaced as
# blank skill descriptions, a fixer inserting its banner above the
# frontmatter instead of below it, resolver trigger extraction dropping
# entries, and a generated-doc freshness check reporting every line as
# drifted. The parsers stay CR-tolerant on their own merits (gbrain reads
# Markdown it does not own), but pinning this repo's own .md checkout to LF
# removes the whole class for anyone working here.
*.md text eol=lf
+13
View File
@@ -4,6 +4,19 @@ about: Something isn't working
labels: bug
---
**Why are you opening this? (human-written, required)**
<!-- Write this yourself. Not AI-generated, not AI-polished. What were you
doing, what happened, why does it matter to you? Rough is fine.
Issues/PRs without this are closed unreviewed. -->
**Screenshot of gbrain in use (required)**
<!-- Your terminal / agent session / logs showing the real situation.
Redact private names, keys, and brain contents first. -->
**What happened?**
+13
View File
@@ -4,6 +4,19 @@ about: Suggest an improvement
labels: enhancement
---
**Why are you opening this? (human-written, required)**
<!-- Write this yourself. Not AI-generated, not AI-polished. What were you
doing, what happened, why does it matter to you? Rough is fine.
Issues/PRs without this are closed unreviewed. -->
**Screenshot of gbrain in use (required)**
<!-- Your terminal / agent session / logs showing the real situation.
Redact private names, keys, and brain contents first. -->
**What problem does this solve?**
+17
View File
@@ -0,0 +1,17 @@
**Why are you opening this? (human-written, required)**
<!-- Write this yourself. Not AI-generated, not AI-polished. What were you
doing, what went wrong or what you needed, why it matters to you.
Rough grammar is fine. PRs without this are closed unreviewed. -->
**Screenshot of gbrain in use (required)**
<!-- Your terminal / agent session / logs showing the real need this fixes.
Redact private names, keys, and brain contents first. -->
**What changed**
**How it was tested**
+6 -3
View File
@@ -61,7 +61,10 @@ jobs:
- name: Run JSONB double-encode parity tests on real Postgres
env:
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/gbrain_test
run: bun test test/e2e/op-checkpoint-jsonb-parity.test.ts test/e2e/jsonb-roundtrip.test.ts
# --timeout also raises bun's 5s default hook budget (beforeAll/afterAll
# do NOT inherit a test's third-arg timeout; verified on bun 1.3.x).
# Every runner script in scripts/ passes it; bare invocations must too.
run: bun test --timeout=60000 test/e2e/op-checkpoint-jsonb-parity.test.ts test/e2e/jsonb-roundtrip.test.ts
tier1:
name: Tier 1 (Mechanical)
@@ -88,7 +91,7 @@ jobs:
bun-version: 1.3.13
- run: bun install
- name: Run Tier 1 E2E tests
run: bun test test/e2e/mechanical.test.ts test/e2e/mcp.test.ts
run: bun test --timeout=60000 test/e2e/mechanical.test.ts test/e2e/mcp.test.ts
env:
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/gbrain_test
@@ -155,7 +158,7 @@ jobs:
}
EOF
- name: Run Tier 2 skill tests
run: bun test test/e2e/skills.test.ts test/e2e/zeroentropy-live.test.ts
run: bun test --timeout=60000 test/e2e/skills.test.ts test/e2e/zeroentropy-live.test.ts
env:
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/gbrain_test
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+110
View File
@@ -0,0 +1,110 @@
name: PR Gate
# Strict PR usefulness gate (#3698): classifies every PR to master into
# merge-lane / close-lane / needs-maintainer BEFORE any human review effort.
# Verdict + reviewer checklist land in one sticky comment; exactly one
# gate:* label is applied; close-lane exits 1 (red X = strong signal).
#
# SECURITY MODEL (pull_request_target on a 30k-star public repo):
# - PR code is NEVER checked out or executed. Metadata + diff come from the
# GitHub API only; the diff is capped at 120KB.
# - The checkout below is the BASE repo (master) — rubric/script only.
# NEVER add a `ref:` pointing at the PR head.
# - Attacker-controlled values (title/body/diff) never touch the shell:
# every ${{ }} is env-bound; run: scripts use plain env vars.
# - Only the issues API is used (comments + labels), so issues:write is the
# single write grant; the checkout drops its credentials.
# - The mechanical CONTRIBUTING.md #3745 check (intent paragraph + screenshot)
# runs BEFORE any API dependency, so a PR missing either still lands in
# close-lane during an Anthropic outage — an outage is not a way through.
# - If ANTHROPIC_API_KEY is missing or the API is unreachable on an otherwise
# compliant PR, the script NEUTRAL-skips loudly (sticky comment + warning
# annotation, exit 0) and CLEARS any stale gate:* label — never a silent
# green, never a red X for a missing secret, never a stale verdict. A model
# REFUSAL is not a skip: it routes to needs-maintainer so refusing is not a
# way to dodge the gate.
#
# #3745 EXEMPTION (deliberate — mirrors policyExemption() in
# scripts/pr-gate.mjs): the intent-paragraph + screenshot requirement filters
# INCOMING OUTSIDE CONTRIBUTIONS. It is waived for repo owners / members /
# collaborators, bot authors, and drafts.
# Release automation cannot take a screenshot of itself, and without the
# exemption every /ship release PR lands in close-lane
# (measured: 40 of the last 40 merged PRs) — a check that is red
# on every release gets switched off within a week, and then it filters
# nothing. Exempt PRs still get the FULL usefulness verdict, the title rule and
# every mechanical red flag; only the description requirement is skipped, and
# the sticky comment says so on its own line.
# author_association / draft / user.type are read from the pr.json fetched
# below — GitHub-computed, not author-settable (except `draft`), and already
# on disk, so nothing new is fetched and there is one source of truth.
# `ready_for_review` is in the trigger list precisely because `draft` IS
# author-settable: leaving draft re-runs the gate with the exemption gone, and
# the exemption is folded into the spend-guard hash so the draft-era verdict
# cannot be reused.
# Pinned by test/pr-gate-workflow.test.ts.
on:
pull_request_target:
types: [opened, edited, synchronize, reopened, ready_for_review]
branches: [master]
# issues:write is the ONLY write grant. Everything the script calls is the
# issues API (comments, label create, label add/remove on the PR's issue), so
# pull-requests:write would be a redundant second grant on the same objects.
permissions:
contents: read
issues: write
concurrency:
group: pr-gate-${{ github.event.pull_request.number }}
cancel-in-progress: true
jobs:
gate:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
# Base repo (master) only — provides scripts/pr-gate.mjs.
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
with:
# Nothing here needs git auth after the clone; don't leave a token
# in .git/config for the rest of the job.
persist-credentials: false
- name: Fetch PR metadata + diff (API only — PR code is never checked out)
env:
GH_TOKEN: ${{ github.token }}
REPO: ${{ github.repository }}
PR_NUMBER: ${{ github.event.pull_request.number }}
run: |
set -euo pipefail
mkdir -p "$RUNNER_TEMP/pr-gate"
gh api "repos/${REPO}/pulls/${PR_NUMBER}" > "$RUNNER_TEMP/pr-gate/pr.json"
# First 100 files is enough: red flags key off pr.json's changed_files
# count, and >40 files already flags.
gh api "repos/${REPO}/pulls/${PR_NUMBER}/files?per_page=100" \
> "$RUNNER_TEMP/pr-gate/files.json"
# Diff via the .diff media type; GitHub can 406 on huge diffs —
# degrade to a marker instead of failing the gate.
gh api "repos/${REPO}/pulls/${PR_NUMBER}" \
-H "Accept: application/vnd.github.diff" \
> "$RUNNER_TEMP/pr-gate/pr.diff.full" \
|| printf '[diff unavailable from the GitHub API — too large or unfetchable]\n' \
> "$RUNNER_TEMP/pr-gate/pr.diff.full"
MAX=122880 # 120KB cap
if [ "$(wc -c < "$RUNNER_TEMP/pr-gate/pr.diff.full")" -gt "$MAX" ]; then
head -c "$MAX" "$RUNNER_TEMP/pr-gate/pr.diff.full" > "$RUNNER_TEMP/pr-gate/pr.diff"
printf '\n\n[TRUNCATED: diff capped at 120KB]\n' >> "$RUNNER_TEMP/pr-gate/pr.diff"
else
mv "$RUNNER_TEMP/pr-gate/pr.diff.full" "$RUNNER_TEMP/pr-gate/pr.diff"
fi
rm -f "$RUNNER_TEMP/pr-gate/pr.diff.full"
- name: Gate verdict (sticky comment + label; exit 1 only on close-lane)
env:
GITHUB_TOKEN: ${{ github.token }}
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
GITHUB_REPOSITORY: ${{ github.repository }}
PR_NUMBER: ${{ github.event.pull_request.number }}
run: node scripts/pr-gate.mjs "$RUNNER_TEMP/pr-gate"
+88 -6
View File
@@ -1,14 +1,67 @@
name: Release
# Publishes a GitHub release for every VERSION bump that lands on master:
# tag + title `v<VERSION>`, notes from that version's CHANGELOG.md entry,
# compiled binaries attached (#3521).
#
# Why every bump: `gbrain check-update` resolves the latest version from the
# VERSION file on master, but binary self-update
# (src/core/binary-self-update.ts) downloads assets from `releases/latest`.
# If releases lag VERSION, binary installs are told an upgrade exists that
# self-update cannot apply. Keeping releases/latest == VERSION closes that gap.
#
# Idempotent: the `version` job skips build+release when a release for
# v<VERSION> already exists WITH all expected assets. A half-published release
# (tag exists / assets incomplete) is repaired on the next run — softprops
# updates the existing release in place. Historical 3-segment tags are never
# touched; a new 4-segment VERSION always mints a new tag.
#
# The asset names are a contract with expectedAssetName() in
# src/core/binary-self-update.ts, pinned by test/release-workflow.test.ts.
on:
push:
tags: ['v*']
branches: [master]
paths: [VERSION]
workflow_dispatch: {} # manual first run / backfill of the current VERSION
permissions:
contents: write
contents: read
concurrency:
group: release
cancel-in-progress: false
jobs:
version:
runs-on: ubuntu-latest
outputs:
version: ${{ steps.v.outputs.version }}
exists: ${{ steps.v.outputs.exists }}
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- id: v
name: Read VERSION and check for an existing complete release
env:
GH_TOKEN: ${{ github.token }}
run: |
version="$(tr -d '[:space:]' < VERSION)"
echo "version=$version" >> "$GITHUB_OUTPUT"
# Complete = release exists AND carries every asset the self-updater
# can request. A partial release must NOT short-circuit, so a re-run
# can repair it.
assets="$(gh release view "v$version" --repo "$GITHUB_REPOSITORY" \
--json assets --jq '[.assets[].name] | sort | join(",")' 2>/dev/null || true)"
if [ "$assets" = "gbrain-darwin-arm64,gbrain-linux-x64" ]; then
echo "exists=true" >> "$GITHUB_OUTPUT"
echo "Release v$version already published with all assets — nothing to do."
else
echo "exists=false" >> "$GITHUB_OUTPUT"
fi
build:
needs: version
if: needs.version.outputs.exists == 'false'
strategy:
matrix:
include:
@@ -29,9 +82,19 @@ jobs:
with:
bun-version: 1.3.13
- run: bun install
- run: bun test
- run: bun run verify
# No test re-run here: the Test workflow already gated this exact SHA at
# merge (10 shards + E2E). Re-running the whole suite serially on the
# release runner is a flakier duplicate gate — it blocked the first
# release on ambient-env tests (run 30698650484). The build job's gate
# is the artifact itself: compile, then smoke-test the binary.
- run: bun build --compile --target=${{ matrix.target }} --outfile bin/${{ matrix.artifact }} src/cli.ts
- name: Smoke-test the compiled binary
run: |
chmod +x bin/${{ matrix.artifact }}
out="$(./bin/${{ matrix.artifact }} --version)"
echo "binary reports: $out"
v="$(tr -d '[:space:]' < VERSION)"
case "$out" in *"$v"*) echo "version matches VERSION file" ;; *) echo "binary version '$out' does not contain '$v'" >&2; exit 1 ;; esac
- name: Attest build provenance
uses: actions/attest-build-provenance@0f67c3f4856b2e3261c31976d6725780e5e4c373 # v4.1.1
with:
@@ -42,16 +105,35 @@ jobs:
path: bin/${{ matrix.artifact }}
release:
needs: build
needs: [version, build]
if: needs.version.outputs.exists == 'false'
runs-on: ubuntu-latest
permissions:
contents: write # create the tag + release (scoped to this job only)
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
with:
path: artifacts
- name: Extract CHANGELOG entry for release notes
# env-bound, not inlined into the script: VERSION comes from master so
# it isn't attacker-reachable today, but a `${{ }}` inside `run:` is
# shell injection by construction if that ever changes.
env:
RELEASE_VERSION: ${{ needs.version.outputs.version }}
run: |
v="$RELEASE_VERSION"
if ! bash scripts/changelog-entry.sh "$v" > /tmp/release-notes.md || ! [ -s /tmp/release-notes.md ]; then
echo "See [CHANGELOG.md](https://github.com/${GITHUB_REPOSITORY}/blob/master/CHANGELOG.md) for v$v." > /tmp/release-notes.md
fi
- name: Create release
uses: softprops/action-gh-release@3bb12739c298aeb8a4eeaf626c5b8d85266b0e65 # v2
with:
tag_name: v${{ needs.version.outputs.version }}
name: v${{ needs.version.outputs.version }}
target_commitish: ${{ github.sha }}
body_path: /tmp/release-notes.md
fail_on_unmatched_files: true
files: |
artifacts/gbrain-darwin-arm64/gbrain-darwin-arm64
artifacts/gbrain-linux-x64/gbrain-linux-x64
generate_release_notes: true
+6 -1
View File
@@ -87,7 +87,7 @@ jobs:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
with:
fetch-depth: 0
- uses: gitleaks/gitleaks-action@dcedce43c6f43de0b836d1fe38946645c9c638dc # v2
- uses: gitleaks/gitleaks-action@ff98106e4c7b2bc287b24eaf42907196329070c7 # v2
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
@@ -113,6 +113,11 @@ jobs:
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
- run: bun install
- run: bun run verify
# Guard: no bare `bun test` in workflows/scripts — bun ignores
# bunfig.toml's timeout, and hooks (beforeAll/afterAll) get the 5s
# default regardless of per-test third-arg timeouts. Runs directly
# (not via verify's CHECKS array) to avoid a package.json edit.
- run: bash scripts/check-bun-test-timeout.sh
serial-tests:
# *.serial.test.ts at --max-concurrency=1. Lives in its own runner so
+194
View File
@@ -2,6 +2,200 @@
All notable changes to GBrain will be documented in this file.
## [0.42.73.0] - 2026-08-04
**Every incoming pull request now gets a verdict before anyone reads it — and five contributed fixes for silent wrong answers.**
**The PR gate.** Open a pull request against gbrain and an automated check now posts a single verdict comment within a minute: **merge-lane**, **close-lane**, or **needs-maintainer**, with its reasons and a checklist of what a human reviewer should verify for that specific diff. It also checks mechanically that the description carries the human-written intent paragraph and the screenshot of gbrain in use that `CONTRIBUTING.md` requires, and that the title leads with its version.
It is deliberately **advisory** — a triage signal and a reviewer checklist, not an authorization boundary. A green verdict is not permission to merge; a maintainer still decides. Pull-request code is never checked out or executed: the verdict comes from the description and the diff read through the API. Maintainer, bot, and draft pull requests are exempt from the intent-and-screenshot floor only (release automation cannot screenshot itself); they still receive the full verdict. Where the rubric can be argued with, the decision is taken away from it: a merge-lane recommendation is downgraded automatically when a diff adds a dependency, a new provider recipe, or new config keys, edits workflows, deletes a test, exceeds 40 files or 400 net source lines, or changes `src/` without touching a single test.
**Your import output parses again.** `gbrain import <dir> --json` printed five informational lines to stdout ahead of the JSON payload, so anything parsing that output read zero imports while its own bookkeeping recorded the files as ingested — and the next run skipped them permanently. Those lines now go to stderr under `--json`; human output is byte-for-byte unchanged.
**`sources harden --dry-run` no longer changes anything.** It reset the helper's executable bit before reaching the dry-run check, so a documented preview quietly mutated permissions.
**Telemetry records the model that actually ran.** Two nightly-cycle phases wrote a hardcoded or unrelated model name into their verdict cache, evidence signature, and spend metering while the gateway ran whatever chat model you configured. On any brain with a non-default model, the recorded history was fiction.
**`gbrain integrity` stops contradicting itself.** Dead-link findings were counted in the "Review queue" total but written to a different file, so `integrity review` disagreed with `integrity auto`'s own summary. They now get their own line.
**Retype rules can address API-ingested pages.** Mapping rules could only filter on a file path, which is empty for every page written through `put_page` — so no rule could target that whole class. A new `slug_filter` filters on the slug instead, and combines with the path filter when both are given.
Also: the `integrity` source comment no longer documents a `--dry-run` subcommand form that exits with an error.
### To take advantage of v0.42.73.0
```bash
gbrain upgrade
gbrain import <dir> --json | jq . # now parses
gbrain integrity auto # dead links reported separately
```
Nothing to configure for the gate — it runs on pull requests to this repository. If you maintain a fork and want it, the workflow needs an `ANTHROPIC_API_KEY` secret; without one it skips loudly rather than blocking anyone.
### For contributors
The gate went through six rounds against two independent blind reviewers, each judging cold. The findings that changed the design most were not exploits but false positives: a code fence that swallowed the rest of a description, an explanation written as bullet points scoring zero words, a word floor stricter than the published policy, and a comment telling contributors to reopen a pull request that was never closed. Those four descriptions are now permanent regression fixtures — a gate that insults a first-time contributor is worse than no gate. Two properties are deliberate and documented rather than fixed: the mechanical floor is a floor (a determined author clears it in seconds), and a bare URL in a cited reason still autolinks.
Contributed by @YiconZiwei (#2655), @time-attack (#3764, #3759, #3726, #3751, #3739, and the gate groundwork in #3573/#3698).
## [0.42.72.1] - 2026-08-02
**Every issue and pull request now needs a human-written paragraph and a screenshot of gbrain actually being used.**
Effective immediately, opening an issue or a PR requires two things from you personally: a paragraph you wrote yourself saying why you're opening it — what you were doing, what went wrong or what you needed, why it matters — and a screenshot of your terminal, agent session, or logs showing the real situation. Rough grammar is fine and preferred over polish. AI-generated or AI-polished intent text is not accepted; the paragraph is the human part. AI assistance for the *code* is still welcome.
Issues and PRs missing either are closed without review, and can be reopened once both are added. Scrub private names, companies, keys, and brain contents from screenshots before attaching — a redacted screenshot is fine, a missing one is not.
The requirement is stated in `CONTRIBUTING.md` and pre-filled in the bug-report and feature-request issue templates plus a new pull-request template, so the fields are in front of you when you open one.
## [0.42.72.0] - 2026-08-01
**Per-person write isolation inside a shared source, and a guide for putting gbrain behind a multi-user agent harness.**
Until now, `--source` was the only write boundary: a client could write anywhere inside the source it was scoped to, and keeping each person in their own folder was a convention the agent had to honor by itself. Registering a client with `--bound-slug-prefixes` now makes that boundary real. Writes outside the bound prefixes are refused by the server, on every op that can name a page.
**Adding a binding to an existing client narrows it on purpose.** Ops that write by something other than a page slug can't be confined to a prefix, so a bound client is refused them outright rather than left with an unfenced path: `extract_entities`, `extract_facts`, `forget_fact`, `ontology_propose`, `sources_add`/`sources_remove`, and `POST /ingest`. `put_page`'s automatic fact extraction is skipped for the same reason — it writes to entity pages the caller never named. Reads are unaffected, and unbound clients behave exactly as before. The gate keys on "anything that is not a plain read", so an op added in a future release is refused to bound clients until it is explicitly fenced.
Both prefix spellings work: the `wiki/agents/alice/*` glob that `submit_agent` bindings already use, and the plainer `emp-alice/` form. Change a binding in place with `gbrain auth rescope-client <id> --bound-slug-prefixes <p1,p2|none>` — existing tokens pick it up on their next request, so no secret rotation is needed when someone joins or leaves a team.
**New guide: [gbrain as the company brain for a qm deployment](docs/integrations/qm-harness.md).** qm is a multiplayer agent harness where each employee and each channel gets an isolated agent scope. The guide covers the whole path — one central `gbrain serve --http`, the thin-client binary baked into the sandbox image, one OAuth client per scope, and a roster-driven provisioning script that converges the brain to a list of people and channels. It also states plainly what the model does *not* give you: within a shared source, reads stay source-granular, so prefix isolation is a write boundary, not a privacy boundary.
gbrain upgrade # or: bun install -g gbrain@0.42.72.0
gbrain apply-migrations --yes # required: the fence refuses writes it cannot evaluate
```
To fence an existing client to a folder:
```bash
gbrain auth rescope-client <client_id> --bound-slug-prefixes partners/alice-example/
gbrain auth rescope-client <client_id> --bound-slug-prefixes none # undo
```
Verify it took, from a client holding that credential — the first write should succeed and the second should be refused:
```bash
gbrain put partners/alice-example/notes/test --content "mine"
gbrain put partners/bob-example/notes/test --content "not mine"
```
## [0.42.71.0] - 2026-08-01
**GBrain now publishes real releases. Every version bump from here on lands on the [Releases page](https://github.com/garrytan/gbrain/releases) with organized notes and downloadable binaries — and binary self-update finally works.**
Until now the repo had no releases at all: `gbrain check-update` could tell you a new version existed, but `gbrain self-upgrade` downloaded from an empty releases API and failed every time, and anyone trying to follow what shipped had to read raw commit history. That's what people have been (rightly) complaining about.
From this release forward, every version bump automatically:
- **Tags the commit** (`v0.42.71.0`) so versions are addressable in git.
- **Publishes a GitHub Release** whose notes are that version's CHANGELOG entry — the same organized, user-facing writeup, not a commit dump.
- **Attaches compiled binaries** for macOS (arm64) and Linux (x64), so `gbrain self-upgrade` and fresh binary installs work without a toolchain.
The pipeline is idempotent: a partial release (tag exists, assets incomplete) is repaired on the next run instead of wedging. It runs post-merge, so a flaky release build can never turn master red. Releases for today's two fix waves (v0.42.69.0 and v0.42.70.0) have been backfilled with their CHANGELOG notes so the Releases page tells the whole story of the day; binaries attach from v0.42.71.0 onward.
### To take advantage of v0.42.71.0
```bash
gbrain check-update # now resolves against real releases
gbrain self-upgrade # now actually downloads a binary
```
Or browse https://github.com/garrytan/gbrain/releases for organized per-version notes.
### For contributors
`docs/RELEASING.md` gains the release-publication section; `scripts/changelog-entry.sh` extracts a version's CHANGELOG section (used for release notes — keep entries under the standard `## [X.Y.Z.W]` headers and they publish verbatim). The workflow keeps all actions SHA-pinned, tightens top-level permissions to `contents: read` with write scoped to the release job only, and env-binds all interpolations.
Contributed by @time-attack (#3573, closing #3521).
## [0.42.70.0] - 2026-08-01
**Community fix wave two: 18 contributed fixes. The headline: several things you asked gbrain to do were being quietly ignored — and now they aren't.**
**`--brain` now actually routes.** The documented `gbrain query "X" --brain media-team` parsed the flag and then ran against your host brain anyway. It now routes to the named brain, and an unknown brain name fails loudly instead of silently answering from the wrong database.
**`sync --dry-run` no longer touches anything.** A dry run could pull from the remote and — if your sync strategy had changed — delete indexed pages before the "dry run" early-return was reached. Previews are now read-only, full stop.
**`apply-migrations --yes` applies.** It previously warned that your schema was behind and then printed "All migrations up to date" with exit 0. If you have wedged brains that upgrade never healed, this was why.
**Links between your pages resolve the way you write them.** Dir-qualified wikilinks with raw Obsidian names (`[[wiki/entities/AI 3.0]]`) now resolve to the sync-slugified page; references in non-whitelisted directories are no longer silently dropped; and a scan bug that could add an edge to a *parent* page you never referenced was caught in the wave's composite review and fixed before shipping.
**Windows and self-hosters.** Markdown files keep LF endings so frontmatter parsers stop mis-reading on Windows checkouts; the archive-crawler path gate no longer denies every real Windows path (and no longer fail-opens on NTFS case-insensitivity); a chat-synopsis tier that was hardcoded to one provider now follows your configured models; vector search asks the index for as many candidates as it was told to consider.
**Quieter, more honest infrastructure.** `serve --http` no longer leaves an orphan holding the database lock after Ctrl-C; a minion child that fails to launch settles immediately instead of hanging its slot; doctor gains checks for content-hash duplicates, undeclared database-only pages, stale heartbeats, and a tamper-evident manifest for the skills directory; federated reads respect per-source isolation settings in two more paths; and the security docs were rewritten to describe fixes without cataloguing attack surface.
gbrain upgrade
gbrain extract --stale # re-extracts links under the fixed resolver
gbrain doctor # includes the new silent-failure checks
```
If your brain uses `link_resolution.global_basename` and was populated before this release, a small number of superseded `wikilink_basename` edges can linger beside their newer typed replacements after re-extraction (edge writes are append-only by design). `gbrain reconcile-links` cleans them up; they are harmless to queries that dedup on target.
### For contributors
The composite review of this wave (two independent max-effort review passes over the combined branch) caught two interaction defects that per-PR review could not: the ungated bare-path scanner reading inside wikilink spans, and an extraction watermark set to a date that same-day stamps would already outrun. Both were fixed in the wave with discriminating tests. One reviewed-and-approved PR was deliberately held out: it conflicts semantically with its author's own sibling PR in this wave, and choosing between their two path-resolution mechanisms is the author's call.
Contributed by @time-attack (#3618, #3085, #3539, #3576, #3533, #3453, #3457, #3560, #3161), @daragao3 (#3619, #3536, #3517, #3578), @paul-0320 (#3613, #3564), @cvillarroel2 (#3678), @mamedov (#3624), @dialthewolff (#3550).
## [0.42.69.0] - 2026-08-01
**A community fix wave: 22 contributed fixes, most of them for work your brain was quietly not doing.**
The theme of this release is silent failure. A nightly cycle that reported `ok` while extracting nothing. An `embed` run that left a whole page unsearchable because one chunk in it failed, then exited 0. A health metric that recommended the same step forever because it counted one thing and the fix measured another. None of these looked broken from the outside, which is exactly why they lasted.
**If you run a local or non-Anthropic model, atom extraction was doing nothing.** With a cost cap set, any model absent from the pricing tables made the first work item hard-fail, which latched a budget flag and skipped every remaining item — while the phase still reported success. Local models (`ollama`, `llama-server`) now price at $0, because local inference costs electricity rather than tokens, so their caps stay enforceable. Genuinely unpriced paid providers still skip, but loudly now instead of silently.
**`gbrain embed` no longer lets one bad chunk darken an entire page,** and it exits non-zero when embeddings actually fail. If you have a cron wrapping `gbrain embed`, a brain holding permanently un-embeddable content will now turn that cron red. That is the intended change — it was previously green while silently incomplete.
**Non-Latin and diacritic names now survive mention extraction.** The by-mention tokenizer matched ASCII letters and digits only, so `Đà Nẵng` shredded into one- and two-character fragments and never matched anything. Names in Vietnamese, and any script outside ASCII, are now tokenized properly.
**Self-hosted embedding backends work.** Fixed-dimension OpenAI-compatible servers that reject an explicit `dimensions` parameter no longer get sent one when the requested width already matches the model's native width. A vector search on the embedded database also now asks the index for as many candidates as it was told to consider, instead of silently truncating the pool to the driver default.
**Multi-source brains route correctly in two more places.** A programmatic `sync_brain` call now syncs the source it was handed rather than the global default, and entity slug resolution keeps its path separators instead of flattening `people/alice-example` into an id no page can hold.
**Safer default on a destructive migration.** Submitting the type-unification job without an explicit `apply` now previews instead of applying. If you have that command in a runbook, add `"apply":true` — the playbooks and README were updated to show it.
Also: interrupted imports keep their tail instead of losing progress below the next 100-file boundary; `gbrain init --help` prints its own help instead of a stub; `doctor` stops reporting Windows drive paths as missing files under WSL and bounds its embedding health probe instead of retrying a permanent auth failure three times; cycle lock-release and stamp-write failures are visible instead of swallowed; and references to a `gbrain install` command that never existed are gone from the docs.
### To take advantage of v0.42.69.0
```bash
gbrain upgrade # or: bun install -g gbrain@0.42.69.0
gbrain doctor # confirms the health metric now converges
gbrain embed --stale # exits non-zero if anything is genuinely un-embeddable
```
If you use a local chat model for the nightly cycle, re-run it once and check that atoms actually land:
```bash
gbrain dream --json | jq '.phases[] | select(.name=="extract_atoms")'
```
If you have `gbrain jobs submit unify-types` in a runbook or script, add `"apply":true` to its `--params` or it will now preview only.
### For contributors
Two defects existed only in the *combination* of otherwise-sound fixes, and were caught by reviewing the composed branch rather than the individual changes:
- `isModelPriceable` was introduced with a test asserting `llama-server` is unpriced, while a second fix in the same wave priced `llama-server` at $0. Together the assertion inverted. Reconciled by using a genuinely unpriced provider in the test and pinning the positive case: free local providers are priceable at $0, so their caps stay enforced.
- The type-unification default flipped to dry-run, but three agent-facing playbooks still presented a bare submit as the apply step. Because a second fix in the same wave also edited one of those files, each change looked self-consistent alone. Skills ship downstream via the skillpack, so this would have propagated a playbook whose apply step silently did nothing.
One reviewed fix was deliberately held back: extending the inline subagent drain to Postgres composes badly with this wave's minion connection-recovery work, since the drain calls the same queue operations without the new recovery path and can strand a child job in a per-run queue no worker will claim.
Contributed by @alexey-metaengage (#3652), @time-attack (#3568, #3572, #3567, #3555, #3523, #3144, #3574, #3532, #3545), @brettdavies (#3552, #3553), @mattchronicle (#3364), @rayers (#3589), @zenspam (#3699), @awilhite (#3691), @Grimnoth (#3541), @georgell-ceo (#3634), @Vyacheslav-Zakharov (#3631), @Kyzcreig (#3585), @HammerTech-Z (#3581), @cfeddersen (#3563).
## [0.42.68.1] - 2026-07-30
**If you run `gbrain reindex-frontmatter` or `gbrain backfill` on the default embedded database, they now work. Until this release both failed every time, after waiting 30 seconds.**
The embedded database allows one process at a time, and holds a lock to enforce it. These two commands opened a second connection to the same database from inside the process that already held that lock, then waited for a lock that could never be released — because the thing holding it was the waiting process itself. The wait ran its full 30 seconds and the command exited with an error naming a blocking process that was, in fact, itself. Both commands now reuse the connection that is already open.
Nothing changes for brains on Postgres, where a second connection was always allowed.
## To take advantage of v0.42.68.1
Nothing to undo — the commands failed without writing anything. Just run whichever you needed:
```bash
gbrain reindex-frontmatter
```
## [0.42.67.0] - 2026-07-28
**If you develop GBrain on Windows, the test and check commands now actually run. Until this release they were quietly doing almost nothing.**
+13
View File
@@ -67,6 +67,19 @@ Per-file detail is in `docs/architecture/KEY_FILES.md`.
text, the cast parses it). Guarded by `scripts/check-jsonb-pattern.sh` (template grep) +
`scripts/check-jsonb-params.mjs` (positional AST scanner); the real backstop is the DATABASE_URL-gated
e2e parity tests, since PGLite can't surface the bug. Full rule in `docs/ENGINES.md`.
- **Engine-live paths avoid runtime dynamic `import()` for helper dependencies.** In
`src/core/pglite-engine.ts`, `src/core/postgres-engine.ts`, and
`src/core/migrate.ts`, dependencies previously reached through runtime dynamic
imports use static top-level imports. The only current dynamic-`import()` exceptions
are the four `ai/gateway.ts` lookups in both engines'
`initSchema()` and `_upsertChunksOnce()` methods; each remains lazy inside a
local `try/catch` because the gateway has a large provider/config closure and,
more importantly, eager evaluation would occur before the catch and could
turn a recoverable default/config-row fallback into a module-load failure.
Every exception carries `engine-dynamic-import-ok` on the import line.
`scripts/check-engine-dynamic-import.sh` enforces the rule. For history, use
`git log -G'await[[:space:]]+import\\('`, not `git log -S`: a dynamic-to-static
rewrite can preserve the searched token while changing its context.
- **Engine parity.** `src/core/postgres-engine.ts` and `src/core/pglite-engine.ts` move in
lockstep — a new method/SQL shape lands in BOTH, pinned by `test/e2e/engine-parity.test.ts`.
Forward-referenced columns/indexes go in the bootstrap probe set (guarded by
+32 -4
View File
@@ -1,5 +1,27 @@
# Contributing to GBrain
## Human-authored intent (required, no exceptions)
Effective 2026-08-02, every issue and every pull request must include:
1. **A paragraph you wrote yourself**, explaining why you are opening this.
What you were doing, what went wrong or what you needed, why it matters to
you. AI-generated or AI-polished text is not accepted here — this one
paragraph is the human part. Rough grammar is fine and preferred over
polish.
2. **A screenshot showing gbrain actually being used** in the situation you
are describing — your terminal, your agent session, your logs. Proof the
need is real, not hypothetical.
Issues or PRs without both are closed without review. You may reopen once
they're added.
Scrub anything private before you attach a screenshot: real names, companies,
API keys, brain contents. See the privacy rule in `CLAUDE.md`. A redacted
screenshot is fine; a missing one is not.
AI assistance for the *code* is fine. The intent paragraph is not code.
## Setup
```bash
@@ -19,14 +41,20 @@ The root `.gitattributes` pins `*.sh text eol=lf`, which overrides the
`core.autocrlf=true` that Git for Windows installs by default. A fresh clone is
correct with no extra steps.
If you cloned before that pin existed, your working copy still has the old
Windows line endings and bash will fail with `$'\r': command not found`. Refresh
it once, from the repository root:
`.gitattributes` pins `*.md text eol=lf` for the same reason. The frontmatter
readers anchor on a `---` fence followed by a Unix line ending, so a CRLF
checkout makes a well-formed document parse as having no frontmatter. That
failure is silent: no error, the field just comes back empty.
If you cloned before either pin existed, your working copy still has the old
Windows line endings. Bash will fail with `$'\r': command not found`, and
frontmatter will read as absent. Refresh it once, from the repository root:
```bash
git rm --cached -r . -q
git reset --hard
bash -n scripts/run-unit-parallel.sh # silence means bash can read the scripts
bash -n scripts/run-unit-parallel.sh # silence means bash can read the scripts
git ls-files --eol -- '*.md' | grep -c w/crlf # 0 means Markdown is clean
```
Every `check:*` entry in `package.json` invokes its script as `bash scripts/<name>.sh`
+7
View File
@@ -16,6 +16,13 @@ If you fetched this file by URL without cloning yet, the companion files live at
## Step 1: Install GBrain
> **NEVER install from the npm registry.** GBrain is not distributed on npm; the npm
> package named `gbrain` is an unrelated package. Do NOT run `npm install -g gbrain` or
> `bun add -g gbrain` (note the missing `github:` prefix — that's the trap). The only
> supported sources are `github:garrytan/gbrain` and a git clone, exactly as shown below.
> If an unrelated npm install is already present, remove it first
> (`npm uninstall -g gbrain` / `bun remove -g gbrain`); `gbrain doctor` also detects this.
Default path (Bun is required — gbrain is a Bun + TypeScript runtime):
```bash
+11 -1
View File
@@ -65,6 +65,16 @@ This is the difference between a search engine and a brain. Search finds the pag
## Install
> [!WARNING]
> **GBrain is NOT distributed on npm.** The npm package named `gbrain` is an unrelated
> package with no connection to this project. Do not run `npm install -g gbrain` or
> `bun add -g gbrain` — you'll get something else, and it can shadow the real binary on
> your PATH. Install and upgrade ONLY via the documented paths below
> (`bun install -g github:garrytan/gbrain`, or `git clone` + `bun install && bun link`).
> If you already ran the npm install by mistake: `npm uninstall -g gbrain` /
> `bun remove -g gbrain`, then reinstall from GitHub. `gbrain doctor` detects a
> shadowing npm install and prints the fix.
GBrain is designed to be installed and operated by an AI agent. The fastest path is to have your agent do it for you. The CLI and MCP paths below are for people who want to wire it up themselves.
### Have your agent install it (recommended)
@@ -206,7 +216,7 @@ Most personal-knowledge tools force one fixed layout: their idea of "notes" + "p
**gbrain doesn't have a fixed layout.** It ships with bundled schema packs and lets you author your own when none fit:
- **`gbrain-base-v2`** (default as of v0.41.22) — 15-type DRY/MECE canonical taxonomy (14 canonical + `note` catch-all): `person`, `company`, `media`, `tweet`, `social-digest`, `analysis`, `atom`, `concept`, `source`, `deal`, `email`, `slack`, `writing`, `project`, `note`. Subtypes/format/origin pushed to frontmatter. The taxonomy that responds to issue #1479.
- **`gbrain-base`** (legacy, v0.41 and earlier brains) — the original 24-type layout. Stays bundled for back-compat; brains on it can upgrade via `gbrain onboard --check --explain``gbrain jobs submit unify-types --allow-protected --params '{"target_pack":"gbrain-base-v2"}'`.
- **`gbrain-base`** (legacy, v0.41 and earlier brains) — the original 24-type layout. Stays bundled for back-compat; brains on it can upgrade via `gbrain onboard --check --explain``gbrain jobs submit unify-types --allow-protected --params '{"target_pack":"gbrain-base-v2","apply":true}'` (omit `"apply":true` for a dry-run preview — that is the default).
- **`gbrain-recommended`** — extends `gbrain-base` with the 13 additional directories from `docs/GBRAIN_RECOMMENDED_SCHEMA.md` (source, place, trip, conversation, personal, civic, project, etc.). Activate with `gbrain schema use gbrain-recommended`.
- **Your own pack** — `gbrain schema detect` clusters your actual filesystem into proposed types, `gbrain schema suggest` runs an LLM pass over them, and `gbrain schema review-candidates --apply` promotes the ones you like. Three commands and the brain knows your shape. Authoring a successor pack (declares `migration_from:` so existing brains can opt in): see [`docs/architecture/pack-upgrade-mechanism.md`](docs/architecture/pack-upgrade-mechanism.md).
+18 -23
View File
@@ -34,15 +34,18 @@ enforced structurally by actionlint on every workflow change.
## Remote MCP Security
### ⚠️ Do NOT use open OAuth client registration for remote MCP
### Keep dynamic client registration disabled unless explicitly needed
If you deploy GBrain's MCP server behind an HTTP wrapper with OAuth 2.1
support, **never allow unauthenticated client registration**. An attacker
who discovers your server URL can:
GBrain disables Dynamic Client Registration (DCR) by default. Keep that
default for internet-reachable deployments and pre-register trusted clients
with operator-approved scopes and source access. Enabling DCR lets network
callers create OAuth client records, so use it only when the deployment's
trust model requires self-service registration and browser approval remains
part of the authorization flow.
1. Register a new OAuth client via `POST /register`
2. Use `client_credentials` grant to obtain a bearer token
3. Access all brain data via the MCP tools
Do not enable `--enable-dcr-insecure` on an untrusted network. That option is
reserved for deployments that intentionally allow self-registered
machine-to-machine clients without browser approval.
### Recommended: `gbrain serve --http`
@@ -104,12 +107,10 @@ Auth methods (`--token-endpoint-auth-method`):
- `none` — public PKCE-only client (no secret minted; ChatGPT custom
connector, Claude Code, Cursor)
The validator rejects unknown methods at the registration boundary, and
the same gate applies to the admin endpoint `POST /admin/api/register-client`
and the DCR `POST /register` path. Pre-v0.41.3 the CLI hard-coded
`redirect_uris = []` and `token_endpoint_auth_method = NULL`, forcing
operators to UPDATE `oauth_clients` rows by hand to make claude.ai work
without `--enable-dcr`. That footgun is gone.
The same validator applies to CLI, admin, and DCR registration paths, so
unknown authentication methods are rejected consistently. Browser-based
clients can be configured entirely through the supported CLI flags; operators
do not need to edit OAuth database rows by hand.
### DCR consent default (v0.42.55+)
@@ -186,16 +187,10 @@ When the request `Origin` matches the allowlist, the server echoes it
back in `Access-Control-Allow-Origin` (with `Vary: Origin`). Otherwise no
CORS header is sent and the browser blocks the request.
**v0.41.3:** the same allowlist now gates every OAuth endpoint (`/mcp`,
`/token`, `/authorize`, `/register`, `/revoke`). Pre-v0.41.3 these used
default-wide-open `cors()` middleware, leaking
`Access-Control-Allow-Origin: *` on every response — any web origin could
complete a token exchange from a logged-in operator's browser. The CORS
preflight handler in the legacy bearer transport was also asymmetric
(actual-request path correctly default-deny, but OPTIONS preflight leaked
`Access-Control-Allow-Methods` + `Access-Control-Allow-Headers` to every
Origin); both are now consolidated through a single allowlist-gated path.
A startup stderr WARN fires when `--bind 0.0.0.0` is set without
The same allowlist gates the complete MCP and OAuth HTTP surface. Actual
requests and browser preflight requests use one allowlist-gated policy, so
unlisted origins receive no cross-origin authorization. A startup stderr
warning fires when `--bind 0.0.0.0` is set without
`GBRAIN_HTTP_CORS_ORIGIN`, surfacing the default-deny posture before the
first request.
+1 -1
View File
@@ -2580,7 +2580,7 @@ contributor traps.
- [ ] **v0.37.x: Adopt `resolveDefaultHeaders` for Together / Groq / other attribution-bearing recipes.** v0.37.6.0's `default_headers` / `resolveDefaultHeaders` seam is generic — any recipe whose provider benefits from app-attribution headers can opt in. Together and Groq both have rankings/analytics tied to per-app headers. Add their respective attribution headers to each recipe, similar to OR's `HTTP-Referer` + `X-OpenRouter-Title`. No type-system or gateway changes needed; just `default_headers` blocks on the existing recipes plus `<PROVIDER>_REFERER` / `<PROVIDER>_TITLE` env vars in their `auth_env.optional`. Filed during v0.37.6.0 eng review as a D4 generalization opportunity.
- [ ] **v0.37.x: Guard cli.ts `main()` so importing `buildGatewayConfig` doesn't print help.** v0.37.6.0 exported `buildGatewayConfig` from `src/cli.ts` for test access. Importing it triggers the file's top-level `main()` which prints help to stdout during tests — functionally harmless (tests pass) but noisy. Fix: wrap `main()` in `if (import.meta.main)` so it only runs when cli.ts is the entry point, not when imported. Touches one line; trivial. Filed during v0.37.6.0 implementation.
- [x] **v0.37.x: Guard cli.ts `main()` so importing `buildGatewayConfig` doesn't print help.** v0.37.6.0 exported `buildGatewayConfig` from `src/cli.ts` for test access. Importing it triggers the file's top-level `main()` which prints help to stdout during tests — functionally harmless (tests pass) but noisy. Fix: wrap `main()` in `if (import.meta.main)` so it only runs when cli.ts is the entry point, not when imported. Touches one line; trivial. Filed during v0.37.6.0 implementation.
## v0.37.4.0 pgGraph CI scaffolding follow-ups (v0.37.x+)
+1 -1
View File
@@ -1 +1 @@
0.42.67.0
0.42.73.0
+2 -2
View File
@@ -19,7 +19,7 @@
},
"overrides": {
"@babel/core": "^7.29.6",
"postcss": "^8.5.10",
"postcss": "^8.5.23",
},
"packages": {
"@babel/code-frame": ["@babel/code-frame@7.29.7", "", { "dependencies": { "@babel/helper-validator-identifier": "^7.29.7", "js-tokens": "^4.0.0", "picocolors": "^1.1.1" } }, "sha512-Aup7aUOfpbAUg2ROOJN6Iw5f9DMBlzu0mIkm/malLQFN/YQgO48wCj0Kxa3sEHJvPVFg7siR+qRInwXd2qhQKw=="],
@@ -232,7 +232,7 @@
"picomatch": ["picomatch@4.0.4", "", {}, "sha512-QP88BAKvMam/3NxH6vj2o21R6MjxZUAd6nlwAS/pnGvN9IVLocLHxGYIzFhg6fUQ+5th6P4dv4eW9jX3DSIj7A=="],
"postcss": ["postcss@8.5.19", "", { "dependencies": { "nanoid": "^3.3.12", "picocolors": "^1.1.1", "source-map-js": "^1.2.1" } }, "sha512-Mz8SaolMd8nB+G13WkORcxQKHZ/NE4xXevtkJHVuG+guo9/wYKlIMTKAqGdEmYOXR2ijPjTYNHssizdaVSUNdQ=="],
"postcss": ["postcss@8.5.25", "", { "dependencies": { "nanoid": "^3.3.16", "picocolors": "^1.1.1", "source-map-js": "^1.2.1" } }, "sha512-DTPx3RWSSnWyzLxQnlH0rJP+EW5ekl16ZU4/psbIhA0e53kJfdgaN5vKM+xP7yJtXVu+nfdVFmlgFDEKAe4Pyw=="],
"react": ["react@19.2.5", "", {}, "sha512-llUJLzz1zTUBrskt2pwZgLq59AemifIftw4aB7JxOqf1HY2FDaGDxgwpAPVzHU1kdWabH7FauP4i1oEeer2WCA=="],
+1 -1
View File
@@ -20,6 +20,6 @@
},
"overrides": {
"@babel/core": "^7.29.6",
"postcss": "^8.5.10"
"postcss": "^8.5.23"
}
}
+6 -6
View File
@@ -53,12 +53,12 @@
"overrides": {
"@hono/node-server": "^2.0.5",
"body-parser": "^2.3.0",
"fast-uri": "^3.1.4",
"fast-uri": "^3.1.5",
"fast-xml-builder": "^1.1.7",
"fast-xml-parser": "^5.7.0",
"form-data": "^4.0.6",
"hono": "^4.12.25",
"ip-address": "^10.1.1",
"hono": "^4.12.34",
"ip-address": "^10.3.1",
"js-yaml": "^3.15.0",
"qs": "^6.15.2",
},
@@ -401,7 +401,7 @@
"fast-deep-equal": ["fast-deep-equal@3.1.3", "", {}, "sha512-f3qQ9oQy9j2AhBe/H9VC91wLmKBCCU/gDOnKNAYG5hswO7BLKj09Hc5HYNz9cGI++xlpDCIgDaitVs03ATR84Q=="],
"fast-uri": ["fast-uri@3.1.4", "", {}, "sha512-8JnbkQ4juDyvYs4mgFGQqg4yCYtFDtUtmp2QIQq11ZZe5CFQ5wcqm1rqDgAh/QdMySuBnPzMUiJUNZG5N/AiQw=="],
"fast-uri": ["fast-uri@3.1.5", "", {}, "sha512-gHwA1O9LDIcKunMKhObS/HimwtehO1nPUECKAu5TpKgaO19fcWEl4bliWe1jWxVFvIXztJjjQ4L8XQ1EU9f7Jw=="],
"fast-xml-builder": ["fast-xml-builder@1.3.0", "", { "dependencies": { "path-expression-matcher": "^1.6.2", "xml-naming": "^0.3.0" } }, "sha512-F74cZEdCvuw9P41GAC3rod4X04jjWGM1JPEv/GWSqFTWLsdyMSBMBMlm9Hk3GLBgLBbdBNY8yee0pQh2RBVESQ=="],
@@ -437,7 +437,7 @@
"heic-decode": ["heic-decode@2.1.0", "", { "dependencies": { "libheif-js": "^1.19.8" } }, "sha512-0fB3O3WMk38+PScbHLVp66jcNhsZ/ErtQ6u2lMYu/YxXgbBtl+oKOhGQHa4RpvE68k8IzbWkABzHnyAIjR758A=="],
"hono": ["hono@4.12.30", "", {}, "sha512-emn+JoJjrN9YTpRDS5it/UI2SO9BAE37T6I3d963RxcZ81G9A4pr2SZTEiiaiKbzx+NKRg5BZ89fCL7gCJCUog=="],
"hono": ["hono@4.13.0", "", {}, "sha512-jhunvfHWxd7J5EFfSgH4xsYJzSe/lfqbUCxiyyeaQasUsXeEHXtzVid+7EOGByc5JnFa23SSFL3Y2RV/z1T+eQ=="],
"http-errors": ["http-errors@2.0.1", "", { "dependencies": { "depd": "~2.0.0", "inherits": "~2.0.4", "setprototypeof": "~1.2.0", "statuses": "~2.0.2", "toidentifier": "~1.0.1" } }, "sha512-4FbRdAX+bSdmo4AUFuS0WNiPz8NgFt+r8ThgNWmlrjQjt1Q7ZR9+zTlce2859x4KSXrwIsaeTqDoKQmtP8pLmQ=="],
@@ -447,7 +447,7 @@
"inherits": ["inherits@2.0.4", "", {}, "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ=="],
"ip-address": ["ip-address@10.2.0", "", {}, "sha512-/+S6j4E9AHvW9SWMSEY9Xfy66O5PWvVEJ08O0y5JGyEKQpojb0K0GKpz/v5HJ/G0vi3D2sjGK78119oXZeE0qA=="],
"ip-address": ["ip-address@10.4.0", "", {}, "sha512-oSK96Grm3aP6OrS263xVxbNDGVL7rzBtYdpGqlDG8iQdoenDoTs/nkki+DflYbAEE8Xl6o5YxhxlrKvI3nqKXQ=="],
"ipaddr.js": ["ipaddr.js@1.9.1", "", {}, "sha512-0KI/607xoxSToH7GjN1FfSbLoU0+btTicjsQSWQlh/hZykN8KpmMf7uYwPW3R+akZ6R/w18ZlXSHBYXiYUPO3g=="],
+33
View File
@@ -362,6 +362,39 @@ done
If any SHA differs from what's in the workflow files, update the pin and version comment.
## GitHub releases (binary assets + self-update) — #3521
`.github/workflows/release.yml` publishes a GitHub release automatically for
**every VERSION bump that lands on master** (trigger: push to master touching
`VERSION`, plus `workflow_dispatch` for a manual first run or repair). No
manual tag push is part of the ship flow — the workflow reads `VERSION` (the
single source of truth), mints tag `v<VERSION>` at the pushed commit, titles
the release the same, uses that version's `CHANGELOG.md` entry as the notes
(`scripts/changelog-entry.sh`; falls back to a CHANGELOG link if the entry is
missing), and attaches the compiled binaries.
Why every bump, not selective: `gbrain check-update` resolves the latest
version from `VERSION` on master, while binary self-update
(`src/core/binary-self-update.ts`) downloads assets from `releases/latest`.
Any release that lags `VERSION` tells binary installs an upgrade exists that
self-update cannot apply. `releases/latest` must track `VERSION`.
Invariants:
- **Asset names are a contract.** The build matrix's `artifact:` names must
equal what `expectedAssetName()` in `src/core/binary-self-update.ts`
returns (`gbrain-darwin-arm64`, `gbrain-linux-x64` today). Adding a
platform means updating BOTH plus the version job's completeness check;
`test/release-workflow.test.ts` pins all of it.
- **Idempotent + self-repairing.** The version job skips when a release for
`v<VERSION>` already exists with all expected assets; a partial release
(tag but no release, or missing assets) is completed on re-run. Racing
master pushes queue via the `release` concurrency group — a skipped
intermediate version is fine, latest is what matters.
- **Historical tags are never rewritten.** Old 3-segment versions keep their
history; every new 4-segment `VERSION` mints a fresh tag.
- **Permissions stay scoped.** `contents: write` lives on the release job
only; everything else runs read-only.
## PR descriptions cover the whole branch
+15 -3
View File
@@ -32,9 +32,12 @@ CI, macOS) rejects CRLF and dies on the script's first meaningful line; the Cygw
bash that ships with Git for Windows tolerates it, so a green local run is not by
itself evidence that a script is CRLF-clean.
The root `.gitattributes` pins `*.sh text eol=lf`, which overrides the
`core.autocrlf=true` default that Git for Windows installs. Working copies cloned
before that pin need a one-time `git rm --cached -r . -q && git reset --hard` to
pick it up; see the Windows section of `CONTRIBUTING.md`.
`core.autocrlf=true` default that Git for Windows installs. It pins `*.md` the
same way, because the frontmatter readers anchor on a `---` fence followed by a
Unix line ending and a CRLF checkout makes a document parse as having no
frontmatter, silently. Working copies cloned
before those pins need a one-time `git rm --cached -r . -q && git reset --hard` to
pick them up; see the Windows section of `CONTRIBUTING.md`.
Wallclock figures in the table above are from a Mac dev box. Windows is
substantially slower because each check pays full process-creation cost, and three
@@ -69,6 +72,15 @@ If a shard wedges (per-shard `GBRAIN_TEST_SHARD_TIMEOUT` cap, default 600s), the
- `tests/heavy/*.sh` → ops-shape shell scripts. Cost minutes per run; NOT in default `bun test`. Run via `bun run test:heavy` or scheduled nightly via `.github/workflows/heavy-tests.yml`. Examples: pg_upgrade matrix (boot legacy brain → walk to head), RSS budget gate (measure peak worker RSS vs committed baseline), read-latency-under-sync (p50/p95/p99 under concurrent writer load), sync lock regression (N concurrent syncs assert 1 winner + N-1 lock-busy + zero leaked `gbrain_cycle_locks` rows). See `tests/heavy/README.md` for when to add a script here vs `*.slow.test.ts`. Files prefixed with `_` (e.g. `tests/heavy/_build_legacy_fixtures.sh`) are helpers/libs invoked by sibling tests — the runner skips them.
- `test/fuzz/*.test.ts` → property-based fuzz harness. Pure-validator targets in `pure-validators.test.ts` are guarded by `scripts/check-fuzz-purity.sh` (in `bun run verify`), which `bun build --target=bun` bundles each target and greps the resulting bundle for banned transitive imports (`node:fs`, `node:child_process`, engine modules). Anything that fails the guard moves to `mixed-validators.test.ts` (still property-tested, but no purity guarantee) or `filesystem-validators.test.ts` (fs-backed, uses temp dirs). Fuzz tests run in the default `bun test` loop because they're fast (~3s for ~12 properties × 1000 runs each).
### Skills-manifest freshness guard
`skills/skills.lock.json` is a committed sha256 inventory of every bundled file under
`skills/` (tamper evidence, not signatures — see `src/core/skills-integrity.ts`).
Any change under `skills/` must regenerate it: `bun run scripts/generate-skills-manifest.ts`.
`scripts/check-skills-manifest-fresh.sh` (`bun run check:skills-manifest`, wired into
`bun run verify`) regenerates to a tmp file and diffs, failing CI on drift; at runtime
`gbrain doctor` reports the same drift as a warn-only `skills_manifest_integrity` check.
### Test-isolation lint and helpers
The cross-file flake class is enforced statically by `scripts/check-test-isolation.sh`, wired into `bun run verify` and `bun run check:all`. Rules (non-serial unit files only; `*.serial.test.ts` and `test/e2e/*` are skipped):
File diff suppressed because one or more lines are too long
+3 -2
View File
@@ -56,7 +56,8 @@ that tuple lights up the `pack_upgrade_available` onboard check.
│ gbrain onboard --check --explain shows per-cluster narrative │
│ User reviews; if OK, runs: │
│ gbrain jobs submit unify-types --allow-protected \ │
│ --params '{"target_pack":"gbrain-base-v2"}'
│ --params '{"target_pack":"gbrain-base-v2","apply":true}'
│ (omit "apply":true for a dry-run; that is the default) │
│ (Autopilot never auto-fires this; manual_only) │
└──────────────────────────┬─────────────────────────────────────┘
@@ -229,7 +230,7 @@ add `GBRAIN_AUDIT_FULL=1` (v0.43+ TODO; not yet wired).
- Per-source pack-upgrade (the handler accepts `sourceId` but
`findPackSuccessors` doesn't yet pass it through)
- Cross-brain federated mounts that disagree on canonical packs
- Automatic rollback (today: manual SQL or `gbrain pages restore`)
- Automatic rollback (today: manual SQL or `gbrain restore`)
- LLM-assisted mapping_rules codegen from production data (`gbrain
schema detect-mappings`; deferred to v0.43+)
+1 -1
View File
@@ -214,7 +214,7 @@ gbrain schema downgrade
1. `git revert <merge-commit>` — restores the code.
2. `gbrain schema downgrade --to gbrain-base` — restores config.
3. (Optional) `gbrain pages purge-deleted --older-than 0h` — drops
3. (Optional) `gbrain purge-deleted --older-than 0h` — drops
v0.39-typed pages that no longer have a matching type in the active
pack.
+10 -10
View File
@@ -19,11 +19,13 @@ entire DB from scratch.
This means:
- **Disaster recovery is one command.** If your DB volume corrupts, if
Postgres eats itself, if PGLite's WASM lock wedges — you don't need
a backup. You wipe the DB, re-import from your brain repo, and the
derived state regenerates. v0.32.3 ships `gbrain rebuild
--confirm-destructive` as the documented one-liner.
- **Disaster recovery is a short, boring sequence.** If your DB volume
corrupts, if Postgres eats itself, if PGLite's WASM lock wedges — you
don't need a backup. You wipe the derived tables (on PGLite,
`gbrain reinit-pglite` wipes the whole embedded DB), re-import from
your brain repo with `gbrain sync`, and `gbrain extract all`
regenerates the derived state. See "Disaster recovery" below for the
exact commands.
- **Multi-machine sync is git.** Your brain is a repo. Push from one
machine, pull from another, and the second machine's DB rebuilds on
its next sync. No "back up the database" step.
@@ -146,11 +148,9 @@ The promise the rule makes:
# Snapshot what's there
gbrain stats > /tmp/before.txt
# Wipe and rebuild
gbrain rebuild --confirm-destructive # v0.32.3 — deletes derived tables
# (pages + content_chunks survive
# the CASCADE-safe design)
# OR manually for v0.32.2:
# Wipe and rebuild — delete the derived tables (pages + content_chunks
# survive the CASCADE-safe design), then re-derive from the repo.
# On PGLite, `gbrain reinit-pglite` wipes the whole embedded DB instead.
psql -c 'DELETE FROM facts; DELETE FROM takes; DELETE FROM links; DELETE FROM timeline_entries;'
gbrain sync
gbrain extract all
+6 -4
View File
@@ -76,7 +76,8 @@ gbrain onboard --check --explain # per-cluster narrative dry-run
gbrain jobs submit unify-types \ # PROTECTED + manual_only
--allow-protected \
--params '{"target_pack":"gbrain-base-v2"}'
--params '{"target_pack":"gbrain-base-v2","apply":true}'
# omit "apply":true → dry-run (default)
Handler runs 4 phases:
┌─────────────────────────────────────┐
@@ -108,8 +109,8 @@ Every primitive ships with a documented rollback:
| Operation | Rollback |
|-----------|----------|
| Retype | `frontmatter.legacy_type = <original>` preserved on every page (D8). One SQL UPDATE restores types: `UPDATE pages SET type = frontmatter->>'legacy_type' WHERE frontmatter ? 'legacy_type'`. |
| Page-to-link | Source page soft-deleted with 72h TTL. `gbrain pages restore <slug>` within 72h. Link row stays harmless if source restored. |
| Page-to-alias | Source page soft-deleted with 72h TTL. `gbrain pages restore <slug>` within 72h. Alias row stays harmless (or `DELETE FROM slug_aliases WHERE alias_slug = <slug>` to clean up). |
| Page-to-link | Source page soft-deleted with 72h TTL. `gbrain restore <slug>` within 72h. Link row stays harmless if source restored. |
| Page-to-alias | Source page soft-deleted with 72h TTL. `gbrain restore <slug>` within 72h. Alias row stays harmless (or `DELETE FROM slug_aliases WHERE alias_slug = <slug>` to clean up). |
| Active-pack flip | `gbrain schema use gbrain-base` reverses the flip. |
## What if my brain doesn't fit?
@@ -127,7 +128,8 @@ For brains with substantial custom types that deserve their own canonical
2. Edit your fork to add page_types + mapping_rules covering your
custom domain.
3. Target your fork: `gbrain jobs submit unify-types --allow-protected
--params '{"target_pack":"my-pack"}'`
--params '{"target_pack":"my-pack","apply":true}'` (omit `"apply":true`
for a dry-run preview — that is the default)
Your fork can also declare `migration_from: {pack: gbrain-base-v2,
version: "1.x"}` to register itself as a successor — future agents
+5 -4
View File
@@ -154,10 +154,11 @@ these are the densest source of real bugs in the whole backlog.
`aliases:`), first-H1-title, and basename fallback resolution (path-equality-only gives
~5.5% edge recall on real vaults). Master shipped global-basename (#1388); the alias/
title fallbacks are the still-novel part.
- **Schema-pack-aware link extraction** (#1547, @billy-armstrong) — **OPEN, high.** The
link extractor's `DIR_PATTERN` is a frozen 16-prefix const that ignores pack-declared
`path_prefixes`, so default-pack installs silently lose wikilinks to `person/`,
`writing/`, `wiki/*`. Resolve prefixes from the active pack.
- **Schema-pack-aware link extraction** (#1547, @billy-armstrong) — **RESOLVED via #2576.**
The extractor no longer gates on the frozen `DIR_PATTERN` whitelist: any dir-shaped
path produces a candidate and the persist paths' page-existence checks decide, so
pack-declared directories (`person/`, `writing/`, `wiki/*`, `ops/`) link without a
prefix registry.
- **DB-source extraction** (#1539, @afshaker) — **OPEN, high.** The cycle's extract phase
only walks the filesystem, so DB-resident pages (imported transcripts, remote-DB brains)
never get links/timeline and `brain_score` is capped. Thread `source:'db'`.
+1 -1
View File
@@ -183,6 +183,6 @@ This also means the best AI agent setups will be open source by default. Closed,
Software distribution reimagined: the package is a markdown file, the runtime is a sufficiently smart model, the package manager is your AI agent, and the app store is a git repo.
`gbrain install voice-agent`
`gbrain skillpack scaffold voice-agent`
That's it.
+2 -2
View File
@@ -31,7 +31,7 @@ receipt file from disk and re-renders it. The other modes need the brain.
| `--budget-usd N` | unset | Abort before next call's projected cost would exceed cap. Models without a `pricing.ts` entry fail loud (codex #4). |
| `--source db|fs` | `db` | `fs` is reserved for v0.33+. |
| `--slug-prefix P` | unset | Filter takes to pages whose slug starts with P. |
| `--models a,b,c` | `openai:gpt-4o,anthropic:claude-opus-4-7,google:gemini-1.5-pro` | Comma-separated panel. |
| `--models a,b,c` | `openai:gpt-5.2,anthropic:claude-opus-4-7,google:gemini-2.0-flash` | Comma-separated panel. |
| `--json` | off | Emit the full receipt to stdout. |
## Receipt JSON shape (`schema_version: 1`)
@@ -50,7 +50,7 @@ receipt file from disk and re-renders it. The other modes need the brain.
},
"prompt_sha8": "abcd1234",
"models_sha8": "abcd1234",
"models": ["openai:gpt-4o", "anthropic:claude-opus-4-7", "google:gemini-1.5-pro"],
"models": ["openai:gpt-5.2", "anthropic:claude-opus-4-7", "google:gemini-2.0-flash"],
"cycles_run": 3,
"successes_per_cycle": [3, 3, 2],
"verdict": "pass",
+1 -1
View File
@@ -69,7 +69,7 @@ update_brain_page(slug, new_info, source):
page = gbrain get {slug}
// TIMELINE: always APPEND (never edit existing entries)
gbrain add_timeline_entry {slug} {
gbrain timeline-add {slug} {
date: today,
summary: new_info.summary,
detail: new_info.detail,
+9 -9
View File
@@ -46,10 +46,10 @@ on user_shares_media(url_or_file):
# Step 4: Extract and cross-reference entities
for person in transcript.mentioned_people:
gbrain add_link <slug> <person_slug>
gbrain add_link <person_slug> <slug>
gbrain add_timeline_entry <person_slug> \
--entry "Discussed in {video_title}: {what_was_said}" \
gbrain link <slug> <person_slug>
gbrain link <person_slug> <slug>
gbrain timeline-add <person_slug> {date} \
"Discussed in {video_title}: {what_was_said}" \
--source "YouTube: {url}"
# PATTERN 2: Social Media Bundles
@@ -80,8 +80,8 @@ on user_shares_media(url_or_file):
# Extract entities and cross-reference
for entity in bundle.mentioned_entities:
gbrain add_link <slug> <entity_slug>
gbrain add_link <entity_slug> <slug>
gbrain link <slug> <entity_slug>
gbrain link <entity_slug> <slug>
# PATTERN 3: PDFs and Documents
elif media.type == "pdf" or media.type == "document":
@@ -109,8 +109,8 @@ on user_shares_media(url_or_file):
"""
for entity in document.mentioned_entities:
gbrain add_link <slug> <entity_slug>
gbrain add_link <entity_slug> <slug>
gbrain link <slug> <entity_slug>
gbrain link <entity_slug> <slug>
# Always sync after ingestion
gbrain sync
@@ -127,7 +127,7 @@ on user_shares_media(url_or_file):
## How to Verify
1. Ingest a YouTube video. Run `gbrain get media/youtube/{slug}`. Confirm the page has: the agent's analysis (not just a summary), key quotes with speaker attribution, and the full diarized transcript.
2. Run `gbrain get_links media/youtube/{slug}`. Confirm back-links exist to brain pages for every person and company mentioned in the video.
2. Run `gbrain call get_links '{"slug": "media/youtube/{slug}"}'`. Confirm back-links exist to brain pages for every person and company mentioned in the video.
3. Pick a person mentioned in the video. Run `gbrain get <person_slug>`. Confirm their timeline has a new entry referencing the video with specific context.
4. Ingest a tweet. Confirm the brain page includes the thread context, linked article summaries, and entity cross-references -- not just the tweet text.
5. Run `gbrain search "{topic_from_video}"`. Confirm the media page appears in search results (verifies the content is indexed and searchable).
+9 -9
View File
@@ -49,23 +49,23 @@ on enrich(entity, trigger):
data["contacts"] = google_contacts(entity.email) # Contact data
# Step 5: Store raw data (auditable, re-processable)
gbrain put_raw_data <entity_slug> \
--data '{"sources": {"crustdata": {"fetched_at": "...", "data": {...}}, ...}}'
gbrain call put_raw_data \
'{"slug": "<entity_slug>", "data": {"sources": {"crustdata": {"fetched_at": "...", "data": {...}}, ...}}}'
# Overwrite on re-enrichment, don't append
# Step 6: Write to brain page
if path == "CREATE":
gbrain put <entity_slug> --content "<compiled_truth_from_all_sources>"
gbrain add_timeline_entry <entity_slug> --entry "Page created via enrichment"
gbrain timeline-add <entity_slug> {date} "Page created via enrichment"
elif path == "UPDATE":
# Append timeline, update compiled truth ONLY if materially new
gbrain add_timeline_entry <entity_slug> --entry "Enriched: {new_signal}"
gbrain timeline-add <entity_slug> {date} "Enriched: {new_signal}"
# Flag contradictions -- don't silently resolve them
# Step 7: Cross-reference the graph
gbrain add_link <person_slug> <company_slug> # person -> company
gbrain add_link <company_slug> <person_slug> # company -> person
gbrain add_link <person_slug> <deal_slug> # person -> deal
gbrain link <person_slug> <company_slug> # person -> company
gbrain link <company_slug> <person_slug> # company -> person
gbrain link <person_slug> <deal_slug> # person -> deal
# Every entity page links to every other entity page that references it
# People page sections (not a LinkedIn profile -- a living portrait):
@@ -94,8 +94,8 @@ on enrich(entity, trigger):
## How to Verify
1. Enrich a Tier 1 person. Run `gbrain get <slug>` and confirm the page has Executive Summary, State, What They Believe, Contact, and Timeline sections populated from multiple sources.
2. Run `gbrain get_raw_data <slug>`. Confirm raw API responses are stored with `sources.{provider}.fetched_at` timestamps.
3. Run `gbrain get_links <slug>`. Confirm cross-reference links exist to the person's company page, deal pages, and related entities.
2. Run `gbrain call get_raw_data '{"slug": "<slug>"}'`. Confirm raw API responses are stored with `sources.{provider}.fetched_at` timestamps.
3. Run `gbrain call get_links '{"slug": "<slug>"}'`. Confirm cross-reference links exist to the person's company page, deal pages, and related entities.
4. Check a page that was enriched AND has a user-written Assessment. Confirm the Assessment section was preserved, not overwritten by API data.
5. Try to re-enrich the same person. Confirm the system checks the `fetched_at` timestamp and skips if less than a week old.
+5 -5
View File
@@ -53,7 +53,7 @@ on upcoming_meeting(meeting):
"last_interaction": page.timeline[0], # most recent
"open_threads": page.open_threads,
"relationship_temperature": page.relationship,
"relevant_deals": gbrain get_links <attendee_slug>,
"relevant_deals": gbrain call get_links '{"slug": "<attendee_slug>"}',
}
else:
briefing[attendee] = "No brain page -- consider enriching"
@@ -67,14 +67,14 @@ on inbox_cleared():
for email in processed_emails:
if email.contained_new_information:
# Update the sender's brain page with new signal
gbrain add_timeline_entry <sender_slug> \
--entry "Email re: {subject}. Key info: {extracted_signal}" \
gbrain timeline-add <sender_slug> {date} \
"Email re: {subject}. Key info: {extracted_signal}" \
--source "email from {sender} re {subject}, {date}"
# Update any mentioned entity pages too
for entity in email.mentioned_entities:
gbrain add_timeline_entry <entity_slug> \
--entry "{what_was_said_about_them}" \
gbrain timeline-add <entity_slug> {date} \
"{what_was_said_about_them}" \
--source "email from {sender}, {date}"
# WORKFLOW 4: Scheduling Nudges
+7 -7
View File
@@ -32,15 +32,15 @@ on new_meeting_transcript(meeting):
# Step 3: Propagate to ALL entity pages (MANDATORY -- most agents skip this)
for person in meeting.attendees + meeting.mentioned_people:
gbrain add_timeline_entry <person_slug> \
--entry "Met in '{meeting.title}' on {date}. Key points: ..." \
gbrain timeline-add <person_slug> {date} \
"Met in '{meeting.title}' on {date}. Key points: ..." \
--source "Meeting notes '{meeting.title}', {date}"
# Update their State section if new information surfaced
# Update company pages for each person's company if relevant
for company in meeting.mentioned_companies:
gbrain add_timeline_entry <company_slug> \
--entry "Discussed in '{meeting.title}': {what_was_said}" \
gbrain timeline-add <company_slug> {date} \
"Discussed in '{meeting.title}': {what_was_said}" \
--source "Meeting notes '{meeting.title}', {date}"
# Step 4: Extract action items
@@ -49,8 +49,8 @@ on new_meeting_transcript(meeting):
# Step 5: Back-link everything (bidirectional graph)
for entity in all_entities_mentioned:
gbrain add_link <slug> <entity_slug> # meeting -> entity
gbrain add_link <entity_slug> <slug> # entity -> meeting
gbrain link <slug> <entity_slug> # meeting -> entity
gbrain link <entity_slug> <slug> # entity -> meeting
# Step 6: Sync so new pages are immediately searchable
gbrain sync
@@ -73,7 +73,7 @@ on new_meeting_transcript(meeting):
1. After ingesting a meeting, run `gbrain get meetings/{date}-{slug}`. Confirm the page has the agent's analysis above the bar and the full diarized transcript below it.
2. For each attendee, run `gbrain get <attendee_slug>`. Check that their timeline has a new entry referencing the meeting with specific insights (not just "attended meeting").
3. Pick a company mentioned in the meeting. Run `gbrain get <company_slug>`. Confirm a timeline entry exists referencing what was discussed about the company.
4. Run `gbrain get_links meetings/{date}-{slug}`. Verify back-links exist to all attendee and entity pages.
4. Run `gbrain call get_links '{"slug": "meetings/{date}-{slug}"}'`. Verify back-links exist to all attendee and entity pages.
5. Run `gbrain search "{meeting_topic}"`. Confirm the meeting page appears in search results (verifies sync ran).
---
+6
View File
@@ -59,6 +59,12 @@ streaming progress to stderr. It is idempotent: re-running with the same
language produces identical vectors. `--json` prints a machine-readable
result envelope but still requires `--yes` (or an interactive confirm).
No cache purge is needed. The resolved language is part of the query-cache
key, so rows written under the previous language are unreachable after the
switch — searches read the retokenized index immediately instead of being
served pre-switch results for up to `search.cache.ttl_seconds`. Switching
back reaches the original rows rather than rebuilding them.
## Recipe: accent-insensitive Portuguese (`pt_br`)
Brazilian Portuguese content often mixes accented and unaccented spellings
+3 -3
View File
@@ -91,7 +91,7 @@ first):
6. The seeded `default` source.
So inside `~/.gstack/plans/` on a brain that pinned `gstack` to
`~/.gstack` via `.gbrain-source`, `gbrain put-page` implicitly writes to
`~/.gstack` via `.gbrain-source`, `gbrain put` implicitly writes to
the `gstack` source. Outside any registered directory with no env/dotfile
set, it writes to the default.
@@ -188,10 +188,10 @@ citations keep working.
```bash
# Pass --source explicitly
gbrain put-page topics/ai ... --source wiki
gbrain put topics/ai ... --source wiki
# Or rely on the dotfile / env / CWD match
cd ~/.gstack && gbrain put-page plans/multi-repo ...
cd ~/.gstack && gbrain put plans/multi-repo ...
# → source auto-resolves to gstack
```
+8 -8
View File
@@ -20,8 +20,8 @@ on every_inbound_message(message):
for entity in entities:
existing = gbrain search "{entity.name}"
if existing:
gbrain add_timeline_entry <entity_slug> \
--entry "{what_was_said}" \
gbrain timeline-add <entity_slug> {date} \
"{what_was_said}" \
--source "User, direct message, {timestamp}"
# else: flag for enrichment if important enough
@@ -64,13 +64,13 @@ on nightly_schedule("02:00"):
# The brain COMPOUNDS overnight.
# 5a: Entity sweep -- find unlinked mentions
pages = gbrain list_pages
pages = gbrain list
for page in pages:
mentions = extract_entity_mentions(page.content)
existing_links = gbrain get_links <page.slug>
existing_links = gbrain call get_links '{"slug": "<page.slug>"}'
for mention in mentions:
if mention not in existing_links:
gbrain add_link <page.slug> <mention_slug> # fix broken graph
gbrain link <page.slug> <mention_slug> # fix broken graph
# 5b: Citation audit -- find facts without sources
for page in pages:
@@ -80,7 +80,7 @@ on nightly_schedule("02:00"):
# 5c: Memory consolidation -- update compiled truth from timeline
for page in stale_pages(older_than="7d"):
timeline = gbrain get_timeline <page.slug>
timeline = gbrain timeline <page.slug>
if timeline.has_new_entries_since_last_consolidation:
# Re-synthesize compiled truth from accumulated timeline
updated_truth = consolidate(page.compiled_truth, timeline.new_entries)
@@ -110,11 +110,11 @@ on nightly_schedule("02:00"):
## How to Verify
1. Send a message mentioning a person with a brain page. Confirm the agent detects the entity and adds a timeline entry to their page (`gbrain get_timeline <slug>`).
1. Send a message mentioning a person with a brain page. Confirm the agent detects the entity and adds a timeline entry to their page (`gbrain timeline <slug>`).
2. Ask the agent about someone in the brain. Confirm it runs `gbrain search` or `gbrain get` BEFORE reaching for external APIs (check the tool call order).
3. Write a new page with `gbrain put`, then immediately run `gbrain search` for it. Confirm it appears in results (verifies sync ran).
4. Run `gbrain doctor`. Confirm it returns a health report with database status, page count, and any flagged issues.
5. After a dream cycle runs, check a page that had unlinked entity mentions. Confirm new links were added (`gbrain get_links <slug>`).
5. After a dream cycle runs, check a page that had unlinked entity mentions. Confirm new links were added (`gbrain call get_links '{"slug": "<slug>"}'`).
---
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*
+3 -3
View File
@@ -47,8 +47,8 @@ on user_message(message):
# Step 3: Cross-link to everything that shaped the thinking
for entity in idea.influences:
gbrain add_link originals/{slug} <entity_slug>
gbrain add_link <entity_slug> originals/{slug}
gbrain link originals/{slug} <entity_slug>
gbrain link <entity_slug> originals/{slug}
# Step 4: Sync
gbrain sync
@@ -79,7 +79,7 @@ on user_message(message):
1. Generate an original idea in conversation (e.g., "I call this the 'ambition debt' problem -- every year you delay going big, the compound interest works against you"). Confirm a new page appears at `brain/originals/ambition-debt` with `gbrain get originals/ambition-debt`.
2. Check that the page uses the user's exact phrasing for the title and slug -- not a sanitized version.
3. Run `gbrain get_links originals/ambition-debt`. Confirm cross-links exist to related people, meetings, or other originals.
3. Run `gbrain call get_links '{"slug": "originals/ambition-debt"}'`. Confirm cross-links exist to related people, meetings, or other originals.
4. Express a take on someone else's idea (e.g., "I think Thiel's contrarian question is wrong because..."). Confirm it goes to `originals/` (synthesis is original), not `concepts/`.
5. Run `gbrain search "ambition debt"`. Confirm the originals page appears in search results and is discoverable.
+1 -1
View File
@@ -87,7 +87,7 @@ expect it.
| `version` | string | yes | Your plugin's semver. Informational. |
| `plugin_version` | string | yes | Contract lock. Must equal `"gbrain-plugin-v1"` for v0.15. |
| `subagents` | string | no | Subdir name (default `subagents`). Escape-attempts are rejected. |
| `description` | string | no | Shown in future `gbrain plugin list`. |
| `description` | string | no | Shown in a future plugin-listing command. |
## Subagent definition files
+9 -1
View File
@@ -44,6 +44,7 @@ These require manual setup (no self-installing recipe yet):
|-------|-------------|
| [Credential Gateway](credential-gateway.md) | Set up ClawVisor or Hermes for Gmail, Calendar, Contacts access |
| [Meeting & Call Webhooks](meeting-webhooks.md) | Circleback meeting transcripts + Quo/OpenPhone SMS/calls |
| [qm Harness](qm-harness.md) | gbrain as the company brain for a qm (multi-user agent harness) deployment — central HTTP MCP, per-scope clients, roster provisioning, write fencing |
## How to Read a Recipe
@@ -69,6 +70,12 @@ health_checks: # typed DSL to verify the integration is working
auth_user: "$TWILIO_ACCOUNT_SID"
auth_token: "$TWILIO_AUTH_TOKEN"
label: "Twilio account"
- type: heartbeat_max_age # staleness gate: FAILS `integrations doctor`
max_age: 48h # when the newest heartbeat event is older.
label: "Data freshness" # The other types are point-in-time and stay
# green even when a sense stops producing data.
output_paths: # repo-relative dirs the collector writes files to;
- daily/voice/ # lets doctor/sync warn if one lands in db_only
setup_time: 30 min # estimated time to complete setup
---
@@ -86,7 +93,8 @@ a source install, or the global install copy) are trusted. Recipes discovered at
runtime from `$GBRAIN_RECIPES_DIR` or a cwd-local `./recipes/` are marked untrusted:
they cannot run `command` health checks, cannot run `http` health checks (SSRF
defense), and cannot use the deprecated string health_check form. Untrusted recipes
can still use `env_exists` and `any_of` compositions. To ship a recipe that runs
can still use `env_exists`, `heartbeat_max_age` (reads only the local heartbeat
file — no exec, no network), and `any_of` compositions. To ship a recipe that runs
live checks, contribute it upstream so it becomes package-bundled.
## The Deterministic Collector Pattern
@@ -0,0 +1,79 @@
---
name: gbrain
description: Search and write the company knowledge brain. Use for any question about the org, people, projects, decisions, or history, and to persist durable knowledge beyond this scope's notebook.
---
# gbrain — the company brain
This sandbox has the `gbrain` CLI connected (thin-client) to the org's central
brain. It is the deep, indexed, cross-source memory: org docs, shared channel
knowledge, and every agent's durable notes. Your scope's own notebook stays the
fast per-turn memory; the brain is where knowledge outlives a scope and becomes
searchable by everyone entitled to it.
## First-run setup (once per sandbox — skip if `gbrain remote doctor` passes)
Your scope's brain credentials arrive via the deployment's secret handoff
(keychain entry or one-time secret drop named `gbrain`). Then:
```bash
gbrain init --mcp-only \
--issuer-url "https://brain.<org>.com" \
--mcp-url "https://brain.<org>.com/mcp" \
--oauth-client-id "<client id from the handoff>" \
--oauth-client-secret "<client secret from the handoff>"
gbrain whoami # must succeed before using any other command
```
Pass the secret with `--oauth-client-secret`, not via `GBRAIN_REMOTE_CLIENT_SECRET`:
an env-sourced secret is deliberately NOT written to `~/.gbrain/config.json`, so
every later command would fail with "No client_secret available" once the
variable is out of scope. The flag persists it to the config file on this
sandbox's durable disk, which is what the tool's credential capture expects.
Do not run `gbrain remote doctor` — it needs `admin` scope, which your client
does not have (by design). `gbrain whoami` is the read-scope health check.
## Reading (do this liberally)
```bash
gbrain search "who decided X and why" # hybrid semantic + keyword search
gbrain get <slug> # read one page
gbrain query "question" --json # search tuned for agent consumption
```
You can read: the shared agent-memory source, org read-only sources (wiki,
handbook), and everything under them. Reads are isolation-enforced server-side;
you only ever see sources your client is entitled to.
## Writing (durable knowledge only, under YOUR prefixes)
Your client is write-fenced to slug prefixes — your own namespace plus the
channels you belong to. Writes outside them are rejected server-side.
```bash
# personal durable memory (your namespace):
gbrain put emp-<your-slug>/people/jane-example --content "..."
# shared channel knowledge (channels you are in):
gbrain put chan-eng/decisions/2026-08-database-choice --content "..."
```
Conventions:
- Write conclusions and durable facts, not chat transcripts. One page per
entity/decision/topic; update the page rather than appending near-duplicates.
- Markdown with YAML frontmatter; the brain chunks, embeds, and links it.
- Cross-reference liberally: `gbrain link <from> <to>` (from must be in your
namespace; linking TO any readable page is fine).
- When you learn something channel-relevant in personal work, mirror the
conclusion into the channel prefix with a `(said in <where>)` provenance
note.
## When to reach for the brain
- Any question about the org, a person, a project, a decision, or history →
`gbrain search` FIRST, then answer.
- You produced knowledge with value beyond this conversation → `gbrain put`.
- Something looks wrong (auth errors, empty results you don't expect) →
`gbrain whoami` to confirm which client and scopes you're using, and report
its output.
+258
View File
@@ -0,0 +1,258 @@
#!/usr/bin/env bash
# provision-scopes.sh — roster-driven gbrain provisioning for a qm deployment
# (or any multi-user agent harness with per-person + per-channel scopes).
#
# Reads a roster of channels + employees and converges the brain to it:
# - ensures the shared agent-memory source exists (path-less: agents write
# pages into it over MCP; `gbrain sync` skips it; if the brain host has
# sync.repo_path configured, pages also write through to .sources/<id>/
# on disk for git-backed durability)
# - registers one OAuth client per employee, write-fenced via
# bound_slug_prefixes to emp-<slug>/ plus chan-<c>/ for each channel
# they are in, with federated reads over the memory source + any
# read-only sources you pass
# - re-running after roster edits rescopes existing clients IN PLACE
# (client ids are remembered in the state file; secrets never rotate
# unless you revoke + delete the state row)
#
# Usage:
# provision-scopes.sh roster.tsv \
# [--memory-source agents] [--read-sources org-wiki,handbook] \
# [--budget-usd-per-day 5] [--state-file roster.state.tsv] \
# [--secrets-out new-credentials.tsv] [--gbrain gbrain] [--dry-run]
#
# Roster format (one entry per line; '#' comments and blank lines ignored):
# channel <slug>
# employee <slug> [comma-separated channel slugs]
#
# SECURITY: --secrets-out receives client secrets for NEW registrations,
# written exactly once (gbrain never re-shows them). Deliver each row to its
# scope's sandbox (e.g. via the harness keychain or a one-time secret drop),
# then delete the file.
#
# ponytail: sequential CLI loop, one gbrain invocation per roster row — fine
# to hundreds of employees; batch via the admin API if that ever hurts.
# -f (noglob) is load-bearing, not stylistic: roster lines are word-split
# unquoted below, so without it a line like `employee * eng` would expand
# against the working directory and silently provision a filename as a
# person — i.e. the wrong write fence. Nothing here needs globbing.
set -euf -o pipefail
# Client secrets and the id state file are written by this script; 077 makes
# them 0600 instead of the default 0644. Set before the first file is created.
umask 077
die() { echo "ERROR: $*" >&2; exit 1; }
# Slugs become source ids, client names, AND slug-prefix write fences. The
# fence list is comma-separated, so an unvalidated slug containing a comma
# would inject an EXTRA prefix and hand the client write access to someone
# else's namespace. Fail closed on anything that isn't plain kebab-case.
valid_slug() {
case "$1" in
'') return 1 ;;
-*|*-) return 1 ;;
*[!a-z0-9-]*) return 1 ;;
*) return 0 ;;
esac
}
require_slug() {
valid_slug "$2" || die "roster: invalid $1 slug '$2' (allowed: lowercase a-z, 0-9, interior hyphens)"
}
ROSTER="${1:-}"
[ -n "$ROSTER" ] && [ -f "$ROSTER" ] || die "usage: provision-scopes.sh <roster-file> [flags] (roster not found: '$ROSTER')"
shift
GBRAIN="${GBRAIN:-gbrain}"
MEMORY_SOURCE="agents"
READ_SOURCES=""
BUDGET="5"
STATE_FILE=""
SECRETS_OUT=""
DRY_RUN=0
while [ $# -gt 0 ]; do
case "$1" in
--memory-source) MEMORY_SOURCE="$2"; shift 2 ;;
--read-sources) READ_SOURCES="$2"; shift 2 ;;
--budget-usd-per-day) BUDGET="$2"; shift 2 ;;
--state-file) STATE_FILE="$2"; shift 2 ;;
--secrets-out) SECRETS_OUT="$2"; shift 2 ;;
--gbrain) GBRAIN="$2"; shift 2 ;;
--dry-run) DRY_RUN=1; shift ;;
*) die "unknown flag: $1" ;;
esac
done
STATE_FILE="${STATE_FILE:-${ROSTER}.state.tsv}"
SECRETS_OUT="${SECRETS_OUT:-${ROSTER}.new-credentials.tsv}"
# The roster usually lives in the deployment repo, so the default secrets and
# state paths land there too — one `git add -A` from committing live
# credentials. The STATE file matters as much as the secrets file: it maps
# employee -> client_id, and this script feeds that id straight to
# `rescope-client`, so whoever can write it decides which client receives a
# given employee's write authority. Treat both as privileged infrastructure,
# at the same trust level as the roster itself.
for f in "$SECRETS_OUT" "$STATE_FILE"; do
if git -C "$(dirname "$f")" rev-parse --is-inside-work-tree >/dev/null 2>&1; then
echo "WARN: $f is inside a git work tree. Never commit it;" >&2
echo " gitignore it, or pass --secrets-out/--state-file outside the repo." >&2
fi
done
# A group/world-writable parent directory defeats the symlink and ownership
# checks below: anyone with write access there can swap the file between our
# check and our append. Refuse rather than pretend the checks hold.
for d in "$(dirname "$SECRETS_OUT")" "$(dirname "$STATE_FILE")"; do
perms=$(ls -ld "$d" | awk '{print $1}')
case "$perms" in
?????w*|????????w*) die "refusing to write credentials into a group/world-writable directory: $d ($perms)" ;;
esac
done
# Secure the credential sinks BEFORE anything is appended. umask only governs
# files this script creates; a pre-existing world-readable file would receive
# secrets first and be chmod'ed only afterwards, and a symlink planted at
# either path would redirect them entirely.
for f in "$SECRETS_OUT" "$STATE_FILE"; do
[ -L "$f" ] && die "refusing to write credentials through a symlink: $f"
if [ -e "$f" ]; then
[ -f "$f" ] || die "refusing to write credentials to a non-regular file: $f"
[ -O "$f" ] || die "refusing to write credentials to a file owned by another user: $f"
else
: > "$f"
fi
chmod 600 "$f"
done
run() {
if [ "$DRY_RUN" = 1 ]; then echo "DRY-RUN: $GBRAIN $*" >&2; return 0; fi
# shellcheck disable=SC2086 — $GBRAIN may carry args ("bun run src/cli.ts")
$GBRAIN "$@"
}
state_lookup() { # state_lookup <employee-slug> -> client_id or empty
[ -f "$STATE_FILE" ] || return 0
awk -F'\t' -v s="$1" '$1 == s { print $2; exit }' "$STATE_FILE"
}
# ── Pass 1: parse roster, collect declared channels ─────────────────────────
CHANNELS=""
EMPLOYEES=""
lineno=0
while IFS= read -r line || [ -n "$line" ]; do
lineno=$((lineno + 1))
line="${line%%#*}"
line="${line%$'\r'}" # a CRLF roster would otherwise yield 'emp-alice\r/' prefixes that fence everything out
[ -z "${line//[[:space:]]/}" ] && continue
# shellcheck disable=SC2086 — deliberate word split; globbing is off (set -f above)
set -- $line
[ "$#" -le 3 ] || die "roster line $lineno: too many fields ('$line'). Channels are ONE comma-separated field with no spaces: 'employee alice eng,product'"
case "$1" in
channel)
require_slug channel "${2:-}"
CHANNELS="$CHANNELS $2"
;;
employee)
require_slug employee "${2:-}"
case " $EMPLOYEES " in *" $2:"*) die "roster line $lineno: employee '$2' listed twice" ;; esac
if [ -n "${3:-}" ]; then
for c in ${3//,/ }; do require_slug "channel-reference" "$c"; done
fi
EMPLOYEES="$EMPLOYEES $2:${3:-}"
;;
*) die "roster line $lineno: unknown entry type '$1' (expected 'channel' or 'employee')" ;;
esac
done < "$ROSTER"
# ── Pass 2: ensure the shared memory source exists (path-less) ──────────────
if out=$(run sources add "$MEMORY_SOURCE" --name "agent memory ($MEMORY_SOURCE)" 2>&1); then
echo "source '$MEMORY_SOURCE': created"
else
echo "$out" | grep -q "already registered" || die "sources add failed: $out"
echo "source '$MEMORY_SOURCE': already exists"
fi
# ── Pass 3: converge one client per employee ────────────────────────────────
FED_READ="$MEMORY_SOURCE${READ_SOURCES:+,$READ_SOURCES}"
new_secrets=0
for entry in $EMPLOYEES; do
slug="${entry%%:*}"
chans="${entry#*:}"
prefixes="emp-$slug/"
if [ -n "$chans" ]; then
for c in ${chans//,/ }; do
echo " $CHANNELS " | grep -q " $c " || echo "WARN: employee '$slug' references undeclared channel '$c'" >&2
prefixes="$prefixes,chan-$c/"
done
fi
client_id="$(state_lookup "$slug")"
if [ -n "$client_id" ]; then
# The state file usually sits in the deployment repo, so anyone who can
# edit it could otherwise retarget this privileged rescope at an arbitrary
# client id (e.g. point alice's row at an admin client). Shape-check it.
case "$client_id" in
gbrain_cl_) die "state file: empty client id for '$slug'" ;;
gbrain_cl_*[!a-zA-Z0-9_]*) die "state file: malformed client id for '$slug': $client_id" ;;
gbrain_cl_*) ;;
*) die "state file: client id for '$slug' does not look like a gbrain client: $client_id" ;;
esac
# --source too, so a re-run actually CONVERGES the client to the roster:
# without it, changing --memory-source (or inheriting a state row written
# against an older one) silently leaves the old write source in place
# while the script reports success.
run auth rescope-client "$client_id" --source "$MEMORY_SOURCE" \
--federated-read "$FED_READ" --bound-slug-prefixes "$prefixes" >/dev/null
echo "employee '$slug': rescoped $client_id [write: $prefixes]"
elif [ "$DRY_RUN" = 1 ]; then
echo "employee '$slug': WOULD register qm-emp-$slug [write: $prefixes] [read: $FED_READ]"
continue
else
out=$(run auth register-client "qm-emp-$slug" \
--grant-types client_credentials --scopes "read write" \
--source "$MEMORY_SOURCE" --federated-read "$FED_READ" \
--bound-slug-prefixes "$prefixes" --budget-usd-per-day "$BUDGET" 2>&1) \
|| die "register-client failed for '$slug' (output withheld: it can contain a secret). Re-run the command by hand to see why."
client_id=$(echo "$out" | sed -n 's/.*Client ID:[[:space:]]*\(gbrain_cl_[^[:space:]]*\).*/\1/p' | head -1)
secret=$(echo "$out" | sed -n 's/.*Client Secret:[[:space:]]*\(gbrain_cs_[^[:space:]]*\).*/\1/p' | head -1)
if [ -z "$client_id" ] || [ -z "$secret" ]; then
# The client may well have been created — dying silently would strand a
# live credential nobody can find. Say so WITHOUT echoing the captured
# output: it contains the freshly minted secret, and this path ends up
# in CI logs.
die "could not parse client id/secret for '$slug' from register-client output (output withheld: it contains a secret). A client MAY have been created; check \`gbrain auth list\` and revoke any stray 'qm-emp-$slug'."
fi
printf '%s\t%s\n' "$slug" "$client_id" >> "$STATE_FILE"
printf '%s\t%s\t%s\n' "$slug" "$client_id" "$secret" >> "$SECRETS_OUT"
chmod 600 "$STATE_FILE" "$SECRETS_OUT" 2>/dev/null || true # umask covers new files; this covers pre-existing ones
new_secrets=$((new_secrets + 1))
echo "employee '$slug': registered $client_id [write: $prefixes]"
fi
done
# ── Pass 4: flag offboarded employees ───────────────────────────────────────
# Removing someone from the roster is the highest-stakes edit there is, and
# this script cannot safely revoke on its own (a typo'd roster would nuke live
# credentials). Report instead, with the exact command.
if [ -f "$STATE_FILE" ]; then
while IFS=$'\t' read -r st_slug st_client _rest; do
[ -n "${st_slug:-}" ] || continue
case " $EMPLOYEES " in
*" $st_slug:"*) ;;
*) echo "STALE: '$st_slug' ($st_client) is no longer in the roster but its credentials still work." >&2
echo " Revoke with: $GBRAIN auth revoke-client $st_client" >&2 ;;
esac
done < "$STATE_FILE"
fi
echo
echo "Done. State: $STATE_FILE"
if [ "$new_secrets" -gt 0 ]; then
echo "$new_secrets NEW client secret(s) written to $SECRETS_OUT — deliver to each scope's sandbox, then DELETE the file."
fi
@@ -0,0 +1,10 @@
# Roster for provision-scopes.sh — one line per channel / employee.
# channel <slug>
# employee <slug> [comma-separated channels they belong to]
channel eng
channel product
employee alice-example eng,product
employee bob-example eng
employee carol-example
1 # Roster for provision-scopes.sh — one line per channel / employee.
2 # channel <slug>
3 # employee <slug> [comma-separated channels they belong to]
4 channel eng
5 channel product
6 employee alice-example eng,product
7 employee bob-example eng
8 employee carol-example
@@ -0,0 +1,19 @@
{
"id": "gbrain",
"label": "gbrain company brain",
"advertise": "gbrain",
"hints": [
"Company knowledge brain: searchable, cross-source, persistent.",
"Search it BEFORE answering questions about the org, people, projects, decisions, or history: `gbrain search \"<question>\"`.",
"Write durable knowledge with `gbrain put <slug> --content ...`, only under your own slug prefixes.",
"See the gbrain skill for slug conventions and first-run setup."
],
"auth": {
"check": "gbrain whoami",
"reauth": "gbrain init --mcp-only --force --issuer-url \"$GBRAIN_ISSUER_URL\" --mcp-url \"$GBRAIN_MCP_URL\" --oauth-client-id \"$GBRAIN_CLIENT_ID\" --oauth-client-secret \"$GBRAIN_CLIENT_SECRET\"",
"credentialPaths": [
{ "path": ".gbrain/config.json", "kind": "file" }
]
},
"install": { "binary": "gbrain" }
}
+224
View File
@@ -0,0 +1,224 @@
# qm (multi-user agent harness) — gbrain as the company brain
Connect gbrain to [qm](https://github.com/yc-software/qm) — the multiplayer
agent harness where each employee and each channel gets an isolated agent
scope — so every scope's agent can search and write one shared, indexed,
isolation-enforced company brain. The same recipe fits any harness with
per-person sandboxes that can run a CLI.
**Shape:** one central `gbrain serve --http` (OAuth 2.1) next to qm's core;
the `gbrain` binary baked into qm's sandbox image as a thin client; one OAuth
client per employee, read-fenced by source federation and write-fenced by
`bound_slug_prefixes`. Zero qm code changes — everything lives in the qm
*deployment directory*.
qm's native memory (per-scope notebook) stays as-is for fast per-turn recall.
gbrain adds what qm doesn't have: semantic + hybrid search, cross-scope
knowledge, entity graphs, and durable memory that outlives a scope.
## Topology
| gbrain concept | qm concept |
|---|---|
| one brain (one Postgres/Supabase DB) | the org |
| source `agents` (path-less, shared) | all agent-written memory |
| slug prefix `emp-<slug>/` in `agents` | an employee's personal scope |
| slug prefix `chan-<slug>/` in `agents` | a channel/room scope |
| source `org-wiki` (git-backed, read-only) | company docs |
| OAuth client `qm-emp-<slug>` | one employee's agent identity |
Isolation model:
- **Reads** are source-granular, SQL-enforced (`federated_read`): every
employee client reads `agents` + the read-only sources you grant.
- **Writes** are slug-prefix-granular, server-enforced (`bound_slug_prefixes`,
v0.42.72.0+): a client can only mutate pages under its own `emp-<slug>/`
and its channels' `chan-<x>/` prefixes — on `put_page`, `delete_page`,
`restore_page`, `add_tag`, `remove_tag`, `add_link`/`remove_link`,
`add_timeline_entry`, `revert_version` and `put_raw_data`, plus the
`POST /ingest` webhook route. Not by convention.
- **Every op that is not a plain read is denied unless allow-listed.** Ops
that write by a key other than a slug — `extract_entities` and
`extract_facts` (which mutate `people/*` and `companies/*`), `forget_fact`
(targets a fact by numeric id, across sources), `ontology_propose`, and the
`sources_admin` pair `sources_add`/`sources_remove` — cannot be fenced by
slug, so a bound client gets `permission_denied` at dispatch. The gate keys
on "not a pure read", not on a list of scope strings, so a write op added
later (or one carrying a bespoke scope) is denied until it is explicitly
fenced and added to `CLIENT_FENCED_WRITE_OPS` (`src/core/operations.ts`).
`think` is allow-listed because remote callers cannot persist from it;
`submit_agent` because it enforces this same column itself.
- **Indirect write paths are gated too, not just the ops.** `put_page`'s
facts backstop would otherwise extract entities from the page body and
write fact rows (and a `## Facts` fence on git-backed sources) onto
`people/*` pages the caller never named — the same capability
`extract_facts` is denied for, reached through an in-prefix write. It is
skipped for bound clients. `POST /ingest` is refused outright: its handler
bypasses the op layer *and* discards the source grant for untrusted
payloads, so it would write into the `default` source.
### Known limitations — read these before you rely on the fence
The write fence is a **write** boundary within a source. It is not a privacy
boundary, and it does not make every side effect prefix-clean. As of
v0.42.72.0:
- **`add_link`/`remove_link` fence the `from` endpoint only.** A bound client
can create an edge pointing AT a page it cannot write; the edge's `context`
text surfaces in that page's backlinks and contributes to its search
ranking. Fencing `to` would break legitimate cross-referencing into
`org-wiki`, so this is deliberate — treat inbound-edge context as untrusted
content, the same way you treat page bodies.
- **Reads are source-granular, never prefix-granular.** Everyone entitled to
a source can read every prefix in it. If a scope needs genuine read
privacy, give it its own source.
- **`put_page` can create one reverse graph edge outside the fence.** If a
page body cites a code location (`src/x.ts:42`) and a code page for it
exists *in the same source*, doc↔impl reconciliation adds an edge
originating from that code page. It affects graph/backlink ranking, not
page content. Unreachable in the layout above (the `agents` source is
path-less and holds no code pages); it applies only if you point employee
writes at a code-synced source.
- **A few read ops are still brain-wide** and ignore the federated grant:
`get_recent_salience`, `find_anomalies`, `find_contradictions`, and
`sources_list`/`sources_status` (which expose source ids, paths and URLs).
A read-scoped client can learn facts derived from sources it was not
granted. Pre-existing, not introduced by the fence; if that matters for
your deployment, withhold those tools at the harness layer for now.
- **Reads touch `last_retrieved_at`** on the pages they return, including
pages in read-only sources. Freshness/usage signals are therefore
writable-by-reading; nothing else about the page is.
- **`POST /ingest` writes land in the `default` source** regardless of the
calling client's `source_id`, because the handler discards the source for
untrusted payloads. Bound clients are refused the route outright for this
reason; if you point a webhook integration at it, scope that brain's
`default` source deliberately.
- **Tradeoff to state out loud:** read isolation is per-source, so within the
shared `agents` source every employee can *read* every prefix (including
other employees' `emp-*/`). That matches qm's transparent-by-default,
everything-audited posture. If you need hard read privacy for personal
memory, give those employees their own write source instead of a prefix
(one `sources add emp-<slug>` + `--source emp-<slug>` per client) and keep
channel prefixes in `agents` via a second, channels-only client — at the
cost of two credentials in that sandbox.
## Host setup (the machine running qm's core, or any box its sandboxes can reach)
```bash
# 1. Engine: Postgres/Supabase. PGLite is single-process and cannot serve
# many concurrent sandboxes.
gbrain init --supabase --embedding-model voyage:voyage-4-large
# 2. Modes + gates (publish_* default OFF and fail as silent 403s):
gbrain config set search.mode balanced
gbrain config set mcp.publish_skills true
gbrain config set mcp.publish_advisor true
# 3. Read-only org sources + first sync:
gbrain sources add org-wiki --path ~/brains/org-wiki
gbrain sync --all # cron this
# 4. Serve over HTTP MCP (OAuth 2.1):
gbrain serve --http --bind 0.0.0.0 --port 3131 \
--public-url https://brain.acme-example.com
```
Never hand sandboxes `DATABASE_URL` — direct DB access bypasses OAuth, source
federation, and the write fence entirely.
## Provision scopes from a roster
[`qm-harness-snippets/provision-scopes.sh`](qm-harness-snippets/provision-scopes.sh)
converges the brain to a roster file
([`roster.example.tsv`](qm-harness-snippets/roster.example.tsv)):
```bash
bash provision-scopes.sh roster.tsv --read-sources org-wiki
```
- Creates the path-less `agents` source (agent-written memory needs no git
clone; if the host has `sync.repo_path` configured, pages also write
through to `.sources/agents/` for git-backed durability).
- Registers `qm-emp-<slug>` clients: `--scopes "read write"`,
`--source agents`, `--federated-read agents,org-wiki`,
`--bound-slug-prefixes emp-<slug>/,chan-<a>/,...`, per-day budget.
- **Idempotent:** re-run after every roster edit; existing clients are
`rescope-client`ed in place (channel joins/leaves update the write fence
without rotating secrets).
- New client secrets land once in `<roster>.new-credentials.tsv` — deliver
each row to its scope (qm keychain / one-time secret drop), then delete
the file.
## qm deployment directory
In the org's qm deployment repo (the directory `qm init` produced):
1. **Tool:** copy [`qm-harness-snippets/tool.json`](qm-harness-snippets/tool.json)
to `sandbox/tools/gbrain/tool.json` and drop the compiled `gbrain` binary
beside it (`bun build --compile --outfile gbrain src/cli.ts`, built for
the sandbox image's OS/arch). `auth.credentialPaths` marks
`~/.gbrain/config.json` as the scope's resident credential file;
`auth.check` wires `gbrain whoami` into qm's connector status (read-scope;
see the note below on why `remote doctor` cannot be used here).
2. **Skill:** copy [`qm-harness-snippets/SKILL.md`](qm-harness-snippets/SKILL.md)
to `sandbox/skills/gbrain/SKILL.md` (edit slug conventions to taste).
3. Ship it: `qm sandbox build && qm sandbox publish && qm up`.
Per scope, one-time (agent- or operator-run, credentials from the handoff):
```bash
gbrain init --mcp-only \
--issuer-url https://brain.acme-example.com \
--mcp-url https://brain.acme-example.com/mcp \
--oauth-client-id gbrain_cl_... --oauth-client-secret gbrain_cs_...
gbrain whoami # must succeed
```
Use `--oauth-client-secret`, not `GBRAIN_REMOTE_CLIENT_SECRET`: an env-sourced
secret is deliberately not written to `~/.gbrain/config.json`
(`src/commands/init.ts`), so with the env var alone every later command fails
once it leaves scope — and qm's `sandbox.secretEnv` is org-wide, so there is no
per-scope env to keep it in. With the flag, the credential lands in the config
file on the scope's durable disk and this runs once per scope, ever.
`gbrain remote doctor` is **not** the health check here: `run_doctor` is an
`admin`-scope op and these clients are `read write` on purpose. `gbrain whoami`
is read-scope and reports the client's identity, source, and grants.
## Verify isolation before rollout
From two differently-scoped sandboxes (or two thin-client configs):
```bash
# alice-example (bound to emp-alice-example/, chan-eng/):
gbrain put emp-alice-example/notes/test --content "mine" # OK
gbrain put chan-eng/notes/test --content "shared" # OK
gbrain put emp-bob-example/notes/test --content "not mine" # permission_denied
gbrain put chan-product/notes/test --content "not my channel" # permission_denied
gbrain search "test" # sees agents + org-wiki only
```
## Cost + operations
- `search.mode balanced` (12K token budget, relational retrieval on) is the
right default for a startup fleet; see `docs/guides/search-modes.md` for
the cost matrix before changing it.
- Budgets: `--budget-usd-per-day` is recorded on the client but only enforced
on the `submit_agent` path (`src/core/minions/budget-meter.ts`), which these
`read write` clients cannot reach — so it does **not** cap spend from
ordinary `search`/`put_page` traffic. Treat runaway-agent containment as an
open item: watch the admin SPA (`/admin`) and `gbrain search stats`, and cap
at the model/harness layer.
- Backfills on a live brain: `gbrain embed --stale --pace` (see Pace Mode in
CLAUDE.md / `docs/operations/spend-controls.md`).
## Deliberately deferred
- **qm `MemoryService` decorator** (mirror notebook captures into gbrain,
fan `recall` out and merge, `volunteer_context` push): needs a qm code
change; today's integration is agent-initiated via the CLI + skill.
- **MCP-native attach:** qm pins `strictMcpConfig` with only its in-process
server, so gbrain's MCP-discovered brain-resident skillpacks don't reach
qm agents; the sandbox skill above covers it.
- **Read-side prefix fencing** (hard privacy for `emp-*/` inside a shared
source) — tracked upstream; the roster layout is forward-compatible with
it.
+1 -1
View File
@@ -250,7 +250,7 @@ All 30 GBrain operations are available remotely, including `sync_brain` and
directory where `gbrain serve` was launched. Symlinks, `..` traversal, and absolute
paths outside cwd are rejected. Page slugs and filenames are allowlist-validated
(alphanumeric + hyphens; no control chars, RTL overrides, or backslashes). Local
CLI callers (`gbrain file upload ...`) keep unrestricted filesystem access since
CLI callers (`gbrain files upload ...`) keep unrestricted filesystem access since
the user owns the machine.
## Deployment Options
+12
View File
@@ -51,6 +51,18 @@ When storage configuration is present, `gbrain sync` automatically manages `.git
- Skipped when the repo is a git submodule (`.git` is a file, not a directory) — submodule .gitignore changes don't survive parent updates. A warning explains.
- Skipped entirely when `GBRAIN_NO_GITIGNORE=1` is set (escape hatch for shared-repo setups where a maintainer wants gbrain to leave .gitignore alone).
- Failures (write permission denied, etc.) are caught and logged, never crash sync.
- Warns when a configured collector's declared output dir (recipe `output_paths`
frontmatter) sits inside a `db_only` path: gitignored files never appear in the
git-walking sync diff, and `gbrain import` honors `.gitignore` too — the
collector would run green while nothing reaches the DB. The
`db_only_collector_collision` doctor check surfaces the same trap.
Related doctor coverage: `undeclared_db_only_pages` warns about DB pages with no
backing file that sit outside every declared `db_only` path. The engine's own
derive-phase output prefixes (`life/events/`, `atoms/`, `extracts/`,
`dream-cycle-summaries/`) count as implicitly declared for that check, so healthy
brains stay quiet without adding them to `gbrain.yml`. They are NOT auto-added to
`.gitignore` — only explicitly declared `db_only` dirs are.
Example `.gitignore` addition:
@@ -0,0 +1,690 @@
# Engine Dynamic-Import Reconciliation Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Reconstruct the missing engine-path static-import hardening, preserve the four load-bearing lazy gateway fallbacks, and prevent unreviewed dynamic imports from returning.
**Architecture:** Make the 13 safe engine/migration import statements static and leave only four line-marked `ai/gateway.ts` imports inside their existing soft-failure `try/catch` boundaries. Enforce that current state with a repository-anchored Bash wrapper delegating to a fail-closed TypeScript AST scanner, a hermetic Bun regression test, package/verify wiring, and current-state architecture documentation.
**Tech Stack:** TypeScript compiler API, Bun test runner, Bash, Git, generated llms documentation bundles.
## Global Constraints
- Reconstruct directly on branch `claude/kind-meitner-330c90`, based on investigated `origin/master` commit `6136e139972a5449630b4f47f5ed7b4cbe5b811b` plus design commit `d7f52d8c`.
- Do not merge or cherry-pick `48ada48f`, `248bfe55`, `ef4cf7a8`, or either historical branch wholesale.
- Do not modify `VERSION`, `CHANGELOG.md`, `TODOS.md`, or release metadata; this is a no-version-bump reconciliation.
- Keep all four `await import('./ai/gateway.ts')` calls lazy: PGLite and Postgres `initSchema`, plus both `_upsertChunksOnce` methods.
- Every allowed lazy gateway line must carry `engine-dynamic-import-ok`; there is no file-level exemption.
- Preserve the stronger gateway rationale: the static closure is large, and eager module evaluation would occur outside the local `try/catch`, potentially converting a recoverable configuration/import failure into a module-load-time hard failure.
- Describe the hoists as engine-path hardening. Do not claim every dynamic import deterministically causes a Windows crash; system-wide commit exhaustion confounded prior measurements.
- Keep shared PGLite/Postgres behavior in parity.
- Invoke repository shell scripts through `bash` in `package.json`.
- Capture complete test/check output to workspace-local `.context/*.txt` files before inspecting it; never pipe a test command directly through `head` or `tail`.
- Use `git log -G`, not `git log -S`, for any additional dynamic-to-static import history work.
- Keep every implementation and verification commit local. Do not push, create a PR, comment upstream, or otherwise publish without explicit user approval after local completion.
- Before editing any affected function, run GBrain `code_blast` and `code_callers` for that symbol and inspect any disambiguation candidates.
---
## File Map
- Create `scripts/check-engine-dynamic-import.sh` — repository-anchored Bash wrapper for default and explicit input routing.
- Create `scripts/check-engine-dynamic-import.ts` — TypeScript AST policy scanner for runtime `import()` expressions, parse/read failures, and exact-line comment-trivia opt-outs.
- Create `test/scripts/check-engine-dynamic-import.test.ts` — 22 hermetic adversarial, CRLF, fail-closed, real-tree, and wiring tests.
- Modify `src/core/pglite-engine.ts` — hoist three safe import statements and mark two deliberate gateway imports.
- Modify `src/core/postgres-engine.ts` — hoist eight safe import statements and mark two deliberate gateway imports.
- Modify `src/core/migrate.ts` — hoist two safe migration helper import statements.
- Modify `package.json` — expose `check:engine-dynamic-import` and append it to `check:all` through `bash`.
- Modify `scripts/run-verify-parallel.sh` — add the package check to the authoritative verify dispatcher.
- Modify `CLAUDE.md` — add the cross-cutting current-state invariant.
- Modify `docs/architecture/KEY_FILES.md` — update current-state entries for the three engine-path files.
- Regenerate `llms.txt` and `llms-full.txt` — required derived bundles after CLAUDE/reference documentation changes.
---
### Task 1: Establish and enforce the source invariant
**Files:**
- Create: `scripts/check-engine-dynamic-import.sh`
- Create: `scripts/check-engine-dynamic-import.ts`
- Create: `test/scripts/check-engine-dynamic-import.test.ts`
- Modify: `src/core/pglite-engine.ts`
- Modify: `src/core/postgres-engine.ts`
- Modify: `src/core/migrate.ts`
**Interfaces:**
- Consumes: shell positional arguments `FILE...`; without arguments, the guard scans the three repository files.
- Produces: `scripts/check-engine-dynamic-import.sh [FILE...]`, exit `0` when every runtime dynamic import is allowed and exit `1` after reporting every `file:line:text` violation plus every read/parse error on stderr.
- Produces: one line-level opt-out token, `engine-dynamic-import-ok`, accepted only in real comment trivia on the same physical line as the deliberately lazy import.
- Fails closed on missing/unreadable inputs, TypeScript parse diagnostics, and scanner/process failures; comments, strings, templates, regex literals, and type-position `import(...)` syntax are not runtime imports.
- [ ] **Step 1: Record call-graph blast radius before touching functions**
First call `sources_list` and select the source whose registered path is this gbrain checkout. Then run `code_blast` and `code_callers` for these qualified symbols with that exact `source_id`, following `did_you_mean`/`candidates` when a method name is ambiguous:
```text
src/core/pglite-engine.ts::PGLiteEngine.initSchema
src/core/pglite-engine.ts::PGLiteEngine.batchRetry
src/core/pglite-engine.ts::PGLiteEngine._upsertChunksOnce
src/core/pglite-engine.ts::PGLiteEngine.mergeOntologyFact
src/core/pglite-engine.ts::PGLiteEngine.getRecentSalience
src/core/postgres-engine.ts::PostgresEngine.disconnect
src/core/postgres-engine.ts::PostgresEngine.initSchema
src/core/postgres-engine.ts::PostgresEngine.batchRetry
src/core/postgres-engine.ts::PostgresEngine._upsertChunksOnce
src/core/postgres-engine.ts::PostgresEngine.mergeOntologyFact
src/core/postgres-engine.ts::PostgresEngine.reconnect
src/core/postgres-engine.ts::PostgresEngine.getRecentSalience
src/core/migrate.ts::runMigrationSQLWithRetry
src/core/migrate.ts::runMigrations
```
Use `depth: 5`, `max_nodes: 200`, and `limit: 100`. Expected: no caller requires a signature or behavior change; the patch only changes module binding time and retains all local fallback/error handling.
- [ ] **Step 2: Write the failing guard regression test**
Create `test/scripts/check-engine-dynamic-import.test.ts` as a hermetic subprocess suite. The completed 22-test surface covers:
- unmarked runtime `import()` rejection, including bare and trivia-separated forms;
- same-line markers in real line or multiline block-comment trivia;
- rejection of markers on prior lines or inside strings, templates, and module paths;
- comments and comment-like delimiters inside strings, templates, and regex literals;
- live code after same-line or multiline block comments close;
- CRLF input and complete multi-file violation aggregation;
- missing/readable mixed inputs and TypeScript parse diagnostics;
- default repository anchoring when invoked from a foreign Git repository;
- the reconciled three-file source scan plus package/parallel-verifier wiring.
Use the TypeScript parser rather than a partial lexical reimplementation. On Windows, set the test default to 30 seconds because each case launches Git Bash and Bun, whose startup can exceed Bun's 5-second per-test default.
- [ ] **Step 3: Run the test to prove the pre-implementation red state**
```bash
bun test test/scripts/check-engine-dynamic-import.test.ts > .context/engine-dynamic-import-red.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit 0
```
Expected: non-zero Bun result captured inside the log. At minimum, the `exists` assertion fails because `scripts/check-engine-dynamic-import.sh` does not exist. Read `.context/engine-dynamic-import-red.txt`; do not infer the result from a truncated pipeline.
- [ ] **Step 4: Add the CRLF-safe, fail-closed guard**
Create `scripts/check-engine-dynamic-import.sh` as a thin LF-terminated wrapper. Resolve its own directory first; when no explicit files are passed, anchor the repository with `git -C "$SCRIPT_DIR/.."` and scan the two engines plus `migrate.ts`. Delegate with `exec bun "$SCRIPT_DIR/check-engine-dynamic-import.ts" "${FILES[@]}"` so scanner failures propagate.
Create `scripts/check-engine-dynamic-import.ts` using the TypeScript compiler API:
- read every requested file and aggregate read failures;
- parse as TypeScript and aggregate parse diagnostics;
- walk the AST for `CallExpression`s whose expression is `ImportKeyword`;
- locate all marker occurrences in the full source and use `ts.getTokenAtPosition` to admit only occurrences outside AST tokens (real comment trivia), recording their physical source lines;
- require each runtime import's line to have an admitted marker or report its original `file:line:text`;
- print every read/parse error and every violation before exiting nonzero.
This preserves CRLF line accounting, ignores comment/literal/type-only false positives, catches every legal runtime `import()` shape the TypeScript parser recognizes, rejects marker spoofing, and fails closed.
- [ ] **Step 5: Run the guard test to prove the source-tree midpoint is still red**
```bash
bun test test/scripts/check-engine-dynamic-import.test.ts > .context/engine-dynamic-import-midpoint.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit 0
```
Expected: the synthetic violation, marker, comments, and CRLF cases pass. The default repository scan fails and reports all 17 current imports: 13 unmarked safe candidates plus the four not-yet-marked gateway calls.
- [ ] **Step 6: Hoist the three safe PGLite import statements**
Replace the existing `retry.ts` import and add the ontology/recency imports near the top of `src/core/pglite-engine.ts`:
```ts
// Engine-path imports stay static unless a call site carries an explicit
// engine-dynamic-import-ok justification. The gateway is the only current
// exception because its local try/catch preserves a soft fallback.
import {
withRetry,
BULK_RETRY_OPTS,
resolveBulkRetryOpts,
computeNextDelay,
isRetryableConnError,
type BatchAuditSite,
} from './retry.ts';
import {
valueHash,
normalizeDimension,
isNovelDimension,
} from './chronicle/ontology.ts';
import {
resolveRecencyDecayMap,
DEFAULT_FALLBACK,
} from './search/recency-decay.ts';
```
Delete only these three in-method destructuring imports, leaving their uses unchanged:
```ts
const { isRetryableConnError } = await import('./retry.ts');
const { valueHash, normalizeDimension, isNovelDimension } = await import('./chronicle/ontology.ts');
const { resolveRecencyDecayMap, DEFAULT_FALLBACK } = await import('./search/recency-decay.ts');
```
- [ ] **Step 7: Mark both PGLite gateway soft-failure boundaries**
In `PGLiteEngine.initSchema`, preserve the `try/catch` and accessors, changing only the rationale and import line:
```ts
try {
// Keep the gateway lazy: its static closure is large, and evaluation inside
// this try/catch preserves the unconfigured-gateway default fallback.
const gw = await import('./ai/gateway.ts'); // engine-dynamic-import-ok
// Both accessors THROW when the gateway is unconfigured (they never
// return falsy), so the catch below is the only fallback path (#3461).
dims = gw.getEmbeddingDimensions();
model = gw.getEmbeddingModel();
} catch { /* gateway not configured — use defaults */ }
```
In `PGLiteEngine._upsertChunksOnce`, preserve the config-row and compile-time fallback chain:
```ts
try {
// Keep the gateway lazy so module-load failure remains inside this soft
// fallback boundary; eager evaluation would bypass the config-row fallback.
const gw = await import('./ai/gateway.ts'); // engine-dynamic-import-ok
resolvedModel = gw.getEmbeddingModel();
} catch {
```
- [ ] **Step 8: Hoist the eight safe Postgres import statements**
Replace the existing `retry.ts` import and add these imports near the top of `src/core/postgres-engine.ts`:
```ts
// Engine-path imports stay static unless a call site carries an explicit
// engine-dynamic-import-ok justification. The gateway is the only current
// exception because its local try/catch preserves a soft fallback.
import {
withRetry,
BULK_RETRY_OPTS,
resolveBulkRetryOpts,
computeNextDelay,
isRetryableConnError,
type BatchAuditSite,
} from './retry.ts';
import { isConnectionEndedError } from './retry-matcher.ts';
import {
valueHash,
normalizeDimension,
isNovelDimension,
} from './chronicle/ontology.ts';
import {
resolveRecencyDecayMap,
DEFAULT_FALLBACK,
} from './search/recency-decay.ts';
import { logDbDisconnect } from './audit/db-disconnect-audit.ts';
import { logPoolRecovery } from './audit/pool-recovery-audit.ts';
```
Delete the eight safe dynamic-import statements while keeping their surrounding `try/catch` blocks and calls unchanged:
```ts
const { logDbDisconnect } = await import('./audit/db-disconnect-audit.ts');
const { isRetryableConnError } = await import('./retry.ts');
const { valueHash, normalizeDimension, isNovelDimension } = await import('./chronicle/ontology.ts');
const { isConnectionEndedError } = await import('./retry-matcher.ts');
const { logPoolRecovery } = await import('./audit/pool-recovery-audit.ts');
const { logPoolRecovery } = await import('./audit/pool-recovery-audit.ts');
const { logPoolRecovery } = await import('./audit/pool-recovery-audit.ts');
const { resolveRecencyDecayMap, DEFAULT_FALLBACK } = await import('./search/recency-decay.ts');
```
Update the stale `batchRetry` comment from “Lazy-import to avoid a circular dep concern” to current truth:
```ts
// retry.ts is already in this module's static graph through withRetry, so
// classifying the exhausted error does not need a second runtime import.
```
- [ ] **Step 9: Mark both Postgres gateway soft-failure boundaries**
In `PostgresEngine.initSchema`, mirror the PGLite rationale and preserve behavior:
```ts
try {
// Keep the gateway lazy: its static closure is large, and evaluation inside
// this try/catch preserves the unconfigured-gateway default fallback.
const gw = await import('./ai/gateway.ts'); // engine-dynamic-import-ok
// Both accessors THROW when the gateway is unconfigured (they never
// return falsy), so the catch below is the only fallback path (#3461).
dims = gw.getEmbeddingDimensions();
model = gw.getEmbeddingModel();
} catch { /* gateway not yet configured — use defaults */ }
```
In `PostgresEngine._upsertChunksOnce`, preserve the DB-config fallback:
```ts
try {
// Keep the gateway lazy so module-load failure remains inside this soft
// fallback boundary; eager evaluation would bypass the config-row fallback.
const gw = await import('./ai/gateway.ts'); // engine-dynamic-import-ok
resolvedModel = gw.getEmbeddingModel();
} catch {
```
- [ ] **Step 10: Hoist the two migration helper import statements**
Add these static imports at the top of `src/core/migrate.ts`:
```ts
// runMigrations executes while an initialized engine is live. Keep its helper
// modules in the static graph rather than importing them from async handlers.
import {
isStatementTimeoutError,
isRetryableConnError,
} from './retry-matcher.ts';
import { repairTimelineDedupIndex } from './timeline-dedup-repair.ts';
```
Delete only these two local destructuring imports:
```ts
const { isStatementTimeoutError, isRetryableConnError } = await import('./retry-matcher.ts');
const { repairTimelineDedupIndex } = await import('./timeline-dedup-repair.ts');
```
- [ ] **Step 11: Run the complete guard test and direct guard**
```bash
bun test test/scripts/check-engine-dynamic-import.test.ts > .context/engine-dynamic-import-green.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`; the full guard regression suite passes.
```bash
bash scripts/check-engine-dynamic-import.sh > .context/engine-dynamic-import-guard.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`; output contains `check-engine-dynamic-import: ok (3 file(s) scanned)`.
- [ ] **Step 12: Prove the guard leaves exactly four marked dynamic imports**
```bash
git grep -n -F "import('./ai/gateway.ts'); // engine-dynamic-import-ok" -- src/core/pglite-engine.ts src/core/postgres-engine.ts src/core/migrate.ts > .context/engine-dynamic-import-sites.txt; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exactly four lines, all importing `./ai/gateway.ts` and all carrying `engine-dynamic-import-ok`; no match in `src/core/migrate.ts`.
- [ ] **Step 13: Run focused behavior tests**
```bash
bun test test/chronicle-ontology.test.ts test/chronicle-ontology-ops.test.ts test/recency-decay.test.ts test/core/retry.test.ts test/retry-matcher.test.ts test/audit/pool-recovery-audit.test.ts test/migrate-retry.test.ts test/timeline-dedup-repair.test.ts > .context/engine-dynamic-import-focused.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`. If Windows resource pressure aborts the process, record the exact exit code and rerun the failing file alone; do not relabel an infrastructure abort as a source pass.
- [ ] **Step 14: Commit the source invariant locally**
```bash
git add scripts/check-engine-dynamic-import.sh scripts/check-engine-dynamic-import.ts test/scripts/check-engine-dynamic-import.test.ts src/core/pglite-engine.ts src/core/postgres-engine.ts src/core/migrate.ts
```
```bash
git commit -m "fix(engine): reconcile dynamic import hardening"
```
Expected: one local commit; no version or release files staged.
---
### Task 2: Wire the guard into repository checks
**Files:**
- Modify: `test/scripts/check-engine-dynamic-import.test.ts`
- Modify: `package.json`
- Modify: `scripts/run-verify-parallel.sh`
**Interfaces:**
- Consumes: `scripts/check-engine-dynamic-import.sh` from Task 1.
- Produces: package script `check:engine-dynamic-import` and verify dry-list entry of the same name.
- [ ] **Step 1: Add failing wiring assertions**
Add these imports/constants to `test/scripts/check-engine-dynamic-import.test.ts`:
```ts
const PACKAGE_JSON = resolve(REPO_ROOT, 'package.json');
```
Append this test block:
```ts
describe('engine dynamic-import guard wiring', () => {
it('is invoked through bash by check:all', () => {
const pkg = JSON.parse(readFileSync(PACKAGE_JSON, 'utf8')) as {
scripts: Record<string, string>;
};
expect(pkg.scripts['check:engine-dynamic-import']).toBe(
'bash scripts/check-engine-dynamic-import.sh',
);
expect(pkg.scripts['check:all']).toContain(
'bash scripts/check-engine-dynamic-import.sh',
);
});
it('is listed by the authoritative verify dispatcher', () => {
const result = spawnSync(BASH, [VERIFY_DISPATCHER, '--dry-list'], {
cwd: REPO_ROOT,
encoding: 'utf8',
timeout: 30_000,
});
expect(result.status).toBe(0);
expect(new Set((result.stdout ?? '').trim().split('\n'))).toContain(
'check:engine-dynamic-import',
);
});
});
```
- [ ] **Step 2: Run the test and verify both wiring assertions fail**
```bash
bun test test/scripts/check-engine-dynamic-import.test.ts > .context/engine-dynamic-import-wiring-red.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit 0
```
Expected: non-zero Bun result. The source guard tests remain green; package-script and verify-list assertions fail because the wiring is absent.
- [ ] **Step 3: Add the package scripts**
In `package.json`, add this script alongside the other `check:*` entries:
```json
"check:engine-dynamic-import": "bash scripts/check-engine-dynamic-import.sh"
```
Append the guard to the existing `check:all` chain, preserving every existing check:
```text
&& bash scripts/check-engine-dynamic-import.sh
```
Do not rewrite any existing shell entry without its `bash` prefix.
- [ ] **Step 4: Add the authoritative verify entry**
In `scripts/run-verify-parallel.sh`, add this stable `CHECKS` entry near the other source-shape guards:
```bash
"check:engine-dynamic-import"
```
- [ ] **Step 5: Run the regression test and package check**
```bash
bun test test/scripts/check-engine-dynamic-import.test.ts > .context/engine-dynamic-import-wiring-green.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`; the full guard regression suite passes.
```bash
bun run check:engine-dynamic-import > .context/engine-dynamic-import-package-check.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0` and three files scanned.
- [ ] **Step 6: Commit the wiring locally**
```bash
git add package.json scripts/run-verify-parallel.sh test/scripts/check-engine-dynamic-import.test.ts
```
```bash
git commit -m "test(engine): guard dynamic import policy"
```
Expected: one local commit with the guard wiring and its regression assertions.
---
### Task 3: Document the current-state invariant
**Files:**
- Modify: `CLAUDE.md`
- Modify: `docs/architecture/KEY_FILES.md`
- Regenerate: `llms.txt`
- Regenerate: `llms-full.txt`
**Interfaces:**
- Consumes: the four-marked-import source state and the `check:engine-dynamic-import` package surface.
- Produces: current-state contributor guidance and fresh generated documentation bundles.
- [ ] **Step 1: Add the cross-cutting invariant to `CLAUDE.md`**
Add this bullet under “Cross-cutting invariants” near the other language/filesystem guards:
```md
- **Engine-live paths use static imports by default.** In
`src/core/pglite-engine.ts`, `src/core/postgres-engine.ts`, and
`src/core/migrate.ts`, helper modules are top-level imports. The only current
exceptions are the four `ai/gateway.ts` lookups in both engines'
`initSchema()` and `_upsertChunksOnce()` methods; each remains lazy inside a
local `try/catch` because the gateway has a large provider/config closure and,
more importantly, eager evaluation would occur before the catch and could
turn a recoverable default/config-row fallback into a module-load failure.
Every exception carries `engine-dynamic-import-ok` on the import line.
`scripts/check-engine-dynamic-import.sh` enforces the rule. For history, use
`git log -G'await[[:space:]]+import\\('`, not `git log -S`: a dynamic-to-static
rewrite can preserve the searched token while changing its context.
```
Do not add release tags, Windows-crash certainty, or historical branch names.
- [ ] **Step 2: Update the PGLite current-state entry in `KEY_FILES.md`**
Append this current-state sentence to the existing `src/core/pglite-engine.ts` entry, preserving the entry as one bullet:
```md
Engine-path helper dependencies (`retry`, ontology, recency decay) bind statically; the only lazy imports are `ai/gateway.ts` in `initSchema` and `_upsertChunksOnce`, line-marked because their local catches preserve compiled-default and stored-config fallbacks that eager module evaluation would bypass.
```
- [ ] **Step 3: Update the Postgres current-state entry in `KEY_FILES.md`**
Append this sentence to the existing `src/core/postgres-engine.ts` entry:
```md
Retry classifiers, ontology/recency helpers, and disconnect/pool-recovery audit writers bind statically; only the two `ai/gateway.ts` fallback lookups stay lazy and line-marked, in parity with PGLite.
```
- [ ] **Step 4: Update the migration current-state entry in `KEY_FILES.md`**
Append this sentence to the canonical `src/core/migrate.ts` entry (the broad runner entry, not the older v95-specific index note):
```md
`retry-matcher.ts` and `timeline-dedup-repair.ts` are static dependencies because `runMigrations()` executes from live engine initialization; the engine dynamic-import guard scans this file with both engine implementations.
```
Keep all three entries current-state only: no `v0.42.x`, branch, commit, “previously,” or “was/now” narration.
- [ ] **Step 5: Regenerate the llms bundles**
```bash
bun run build:llms > .context/engine-dynamic-import-build-llms.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`; `llms.txt` and/or `llms-full.txt` update according to their configured linked/inlined status. Byte-identical output for a linked source is acceptable; the freshness test is authoritative.
- [ ] **Step 6: Run documentation freshness checks**
```bash
bun test test/build-llms.test.ts > .context/engine-dynamic-import-llms-test.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`.
```bash
bun run check:doc-history > .context/engine-dynamic-import-doc-history.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`; no release-history marker is introduced into current-state reference docs.
- [ ] **Step 7: Confirm prohibited release files remain untouched**
```bash
git diff --name-only d7f52d8c..HEAD -- VERSION CHANGELOG.md TODOS.md
```
Expected: no output.
- [ ] **Step 8: Commit documentation and generated bundles locally**
```bash
git add CLAUDE.md docs/architecture/KEY_FILES.md llms.txt llms-full.txt
```
```bash
git commit -m "docs(engine): record static import invariant"
```
Expected: one local documentation commit. If one generated bundle is byte-identical, Git simply omits it.
---
### Task 4: Verify and review the complete local reconciliation
**Files:**
- Verify all files changed since `d7f52d8c`.
- Do not create or modify release/publication metadata.
**Interfaces:**
- Consumes: Tasks 13.
- Produces: full local verification evidence and an implementation diff ready for user review, not publication.
- [ ] **Step 1: Run the regression test and direct guard again**
```bash
bun test test/scripts/check-engine-dynamic-import.test.ts > .context/engine-dynamic-import-final-test.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`; the full guard regression suite passes.
```bash
bash scripts/check-engine-dynamic-import.sh > .context/engine-dynamic-import-final-guard.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`; three files scanned.
- [ ] **Step 2: Run TypeScript checking**
```bash
bun run typecheck > .context/engine-dynamic-import-typecheck.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`. Report exact diagnostics if the branch or current Windows environment has a pre-existing failure.
- [ ] **Step 3: Run the authoritative verify dispatcher**
```bash
bun run verify > .context/engine-dynamic-import-verify.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`, including `check:engine-dynamic-import`. On Windows, classify any per-check timeout from the complete log instead of treating the aggregate result as a source regression without evidence.
- [ ] **Step 4: Re-run focused tests as an ownership check**
```bash
bun test test/chronicle-ontology.test.ts test/chronicle-ontology-ops.test.ts test/recency-decay.test.ts test/core/retry.test.ts test/retry-matcher.test.ts test/audit/pool-recovery-audit.test.ts test/migrate-retry.test.ts test/timeline-dedup-repair.test.ts > .context/engine-dynamic-import-final-focused.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`; record any infrastructure abort separately and rerun only the named file before classifying it.
- [ ] **Step 5: Run the llms freshness test after all documentation settles**
```bash
bun test test/build-llms.test.ts > .context/engine-dynamic-import-final-llms.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`.
- [ ] **Step 6: Run whitespace and scope checks**
```bash
git diff --check d7f52d8c..HEAD
```
Expected: exit `0`, no output.
```bash
git diff --name-only d7f52d8c..HEAD
```
Expected files only:
```text
CLAUDE.md
docs/architecture/KEY_FILES.md
docs/superpowers/plans/2026-07-28-engine-dynamic-import-reconciliation.md
llms-full.txt
llms.txt
package.json
scripts/check-engine-dynamic-import.sh
scripts/check-engine-dynamic-import.ts
scripts/run-verify-parallel.sh
src/core/migrate.ts
src/core/pglite-engine.ts
src/core/postgres-engine.ts
test/scripts/check-engine-dynamic-import.test.ts
```
Either generated llms file may be absent if regeneration proves it byte-identical. `VERSION`, `CHANGELOG.md`, and `TODOS.md` must be absent.
- [ ] **Step 7: Review the exact implementation diff**
```bash
git diff --stat d7f52d8c..HEAD && git diff d7f52d8c..HEAD -- src/core/pglite-engine.ts src/core/postgres-engine.ts src/core/migrate.ts scripts/check-engine-dynamic-import.sh test/scripts/check-engine-dynamic-import.test.ts package.json scripts/run-verify-parallel.sh CLAUDE.md docs/architecture/KEY_FILES.md
```
Expected review findings:
- Exactly 13 safe `await import(...)` statements are removed.
- Exactly four `ai/gateway.ts` imports remain, all marked on the same line.
- All four gateway imports remain inside their original local `try/catch` fallback boundaries.
- No accessor logic, fallback ordering, SQL, public signature, or engine parity behavior changes.
- The parser-backed guard reports all violations plus read/parse failures, preserves CRLF line accounting, ignores comments/literals/type-only syntax, detects every runtime `import()` call expression, and accepts opt-outs only from real comment trivia on the same physical line.
- The package script invokes the shell guard through Bash; `check:all` invokes that shell guard directly, and the parallel verify dispatcher invokes the package check.
- Documentation is current-state and makes no deterministic Windows-crash claim.
**Observed Windows verification classification:** The authoritative aggregate completed with 25 of 33 checks passing. Individual reruns showed `check:test-names` and `typecheck` green; privacy/isolation exceeded Windows timing budgets; WASM failed in unrelated temporary-symlink setup; eval-glossary was CRLF/LF drift; resolver/brain-first findings predated and did not intersect this branch. The focused aggregate produced 103 pass / 5 fail: three setup-hook timeouts reproduced at the untouched base, and the known `migrate-retry` polling failure reproduced there. Its additional race-status assertion did not reproduce at base, so it remains an unresolved timing-sensitive limitation in untouched code—not evidence of an in-scope defect and not claimed as conclusively pre-existing.
- [ ] **Step 8: Commit the approved plan document locally**
The plan is an approved, tracked execution artifact and must not be left as an uncommitted file after implementation:
```bash
git add docs/superpowers/plans/2026-07-28-engine-dynamic-import-reconciliation.md
```
```bash
git commit -m "docs: plan engine dynamic-import reconciliation"
```
Expected: one local plan commit; no release metadata staged.
- [ ] **Step 9: Inspect final status without publishing**
```bash
git status --short --branch
```
Expected: branch `claude/kind-meitner-330c90` with a clean working tree. No push, PR, upstream comment, or other external side effect.
- [ ] **Step 10: Capture the completed milestone to memory**
Before writing, search MemPalace wing `gbrain` for this exact reconciliation to avoid duplication. Add a verbatim drawer recording exact base/head commits, the 13 hoists, four gateway opt-outs and rationale, guard/test/docs files, every verification command with exit code, and any environment-owned failures. Add a GBrain project timeline entry only if there is an existing relevant gbrain project page; do not create duplicate release metadata.
- [ ] **Step 11: Report the local result and ask separately before publication**
Report:
- exact local commits;
- changed files;
- test/check exit codes;
- any blocked or pre-existing failures;
- confirmation that release files were untouched;
- confirmation that nothing was pushed or published.
Do not run any publication command. Wait for explicit user approval before any push, PR, or upstream interaction.
@@ -0,0 +1,175 @@
# Scalar-source Backlink Validation Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Make backlink validation compare exact `(source_id, slug)` endpoint identities while preserving existing scalar, unscoped, and federated link-read semantics.
**Architecture:** Enrich every engine link-read row with the source identity of its joined from, to, and visible origin pages. Pass the validated page's scalar or federated scope into validator context; the backlink validator scopes its initial read consistently, groups targets by exact identity, and accepts only an exact reverse endpoint pair. SQL predicates remain unchanged, so trusted scalar cross-source visibility and federated all-endpoint containment remain intact.
**Tech Stack:** TypeScript, Bun test, PGLite, PostgreSQL/postgres.js.
## Global Constraints
- Use strict red-before-green TDD with duplicate slugs across sources.
- Preserve unscoped historical reads, scalar near-endpoint scoping, scalar explicit cross-source visibility, federated all-endpoint containment, and `sourceIds` precedence.
- Keep PostgreSQL and PGLite projections in parity.
- Do not change schema or conditional-write conflict semantics.
- Keep deployment, restart, migration, and push actions outside the implementation tasks; a separately authorized release workflow may perform them after verification.
- Capture full test output to files before inspecting it.
---
### Task 1: Pin the backlink false-negative in PGLite
**Files:**
- Modify: `test/writer.test.ts`
**Interfaces:**
- Consumes: `backLinkValidator.validate(PageValidationContext)` and source-qualified `putPage`/`addLink`.
- Produces: regressions for wrong-source reverse rejection, exact reverse acceptance, cross-source pair acceptance, and exact target deduplication.
- [ ] **Step 1: Add the minimal failing duplicate-slug regression**
Create `default` and `team-x` copies of the origin and target, add `(team-x, origin) -> (team-x, target)` plus the wrong reverse `(team-x, target) -> (default, origin)`, validate with `sourceId: 'team-x'`, and require one warning.
- [ ] **Step 2: Run the focused test and verify RED**
```bash
bun test test/writer.test.ts -t "wrong-source reverse" > "$TEMP/backlink-red.txt" 2>&1
```
Expected: assertion failure because current slug-only validation returns zero findings.
- [ ] **Step 3: Add the remaining behavioral regressions after the first red is recorded**
Add tests proving that the exact reverse clears the warning, a legitimate cross-source forward/reverse pair passes, and two destinations sharing one slug but differing by source are validated independently.
### Task 2: Expose exact endpoint identity from both engines
**Files:**
- Modify: `src/core/types.ts:1204-1229`
- Modify: `src/core/postgres-engine.ts:3021-3124`
- Modify: `src/core/pglite-engine.ts:2941-3037`
- Modify: `test/get-page-federated-scope.test.ts:187-246,289-306`
- Modify: `test/e2e/multi-source-bug-class.test.ts:184-205`
- Modify: `test/e2e/engine-parity.test.ts:813-875`
**Interfaces:**
- Produces: `Link.from_source_id: string`, `Link.to_source_id: string`, and `Link.origin_source_id?: string | null`.
- Preserves: `getLinks(slug, { sourceId?, sourceIds? })` and `getBacklinks(...)` filtering semantics.
- [ ] **Step 1: Add engine-contract assertions before implementation**
Assert scalar cross-source rows expose `beta -> default`, federated rows expose only in-grant endpoint IDs, `sourceIds` still beats scalar `sourceId`, and an out-of-grant origin has both `origin_slug` and `origin_source_id` null.
- [ ] **Step 2: Run the focused contract tests and verify RED**
```bash
bun test test/get-page-federated-scope.test.ts test/e2e/multi-source-bug-class.test.ts > "$TEMP/link-identity-red.txt" 2>&1
```
Expected: source-ID assertions fail because fields are absent.
- [ ] **Step 3: Extend `Link` and project IDs without changing predicates**
Use this additive contract:
```ts
export interface Link {
from_slug: string;
from_source_id: string;
to_slug: string;
to_source_id: string;
link_type: string;
context: string;
link_source?: string | null;
origin_slug?: string | null;
origin_source_id?: string | null;
origin_field?: string | null;
}
```
In all six branches per engine, project:
```sql
f.source_id AS from_source_id,
t.source_id AS to_source_id,
o.source_id AS origin_source_id
```
Keep every `WHERE` and grant-aware origin `LEFT JOIN` unchanged.
- [ ] **Step 4: Re-run contract tests and verify GREEN**
Use the same command and require all focused tests to pass.
### Task 3: Validate exact reverse identities and propagate scope
**Files:**
- Modify: `src/core/output/writer.ts:89-96,240-318`
- Modify: `src/core/output/post-write.ts:36-41,73-118`
- Modify: `src/core/output/validators/back-link.ts:24-47`
- Modify: `src/core/operations.ts:1227-1246`
- Modify: `test/post-write-lint.test.ts:67-130`
**Interfaces:**
- Produces: optional `PageValidationContext.sourceId` and `sourceIds`, with `sourceIds` taking precedence.
- `runPostWriteLint(..., opts)` accepts the same optional scope and loads the validated page through it.
- [ ] **Step 1: Add a post-write nested-read regression and verify RED**
Validate a non-default page with a wrong-source reverse via `runPostWriteLint(..., { force: true, noLog: true, sourceId: 'team-x' })`; require a backlink warning.
- [ ] **Step 2: Implement minimal scope propagation**
Add `sourceId?`/`sourceIds?` to validation context and lint options. Load pages using `sourceIds` when non-empty, otherwise scalar `sourceId`. Pass the same scope into nested validators. In the put-page success hook, call lint with the already-resolved write source ID.
- [ ] **Step 3: Implement exact backlink matching**
Initial outbound reads use the validation scope. Deduplicate rows by all four endpoint identity fields so every distinct expected origin remains represented even when targets share a source-qualified identity. Read each target using the federated grant when present, otherwise the target's exact scalar source. Accept only a row matching all four endpoint fields of the expected reverse.
- [ ] **Step 4: Run writer and post-write tests and verify GREEN**
```bash
bun test test/writer.test.ts test/post-write-lint.test.ts > "$TEMP/backlink-green.txt" 2>&1
```
Expected: all tests pass, including the recorded false-negative.
### Task 4: Verify PostgreSQL/PGLite parity and final scope
**Files:**
- Modify: `test/e2e/engine-parity.test.ts:813-875`
- Verify: all files above
**Interfaces:**
- Consumes: exact endpoint fields and unchanged filtering semantics.
- Produces: parity evidence for scalar cross-source and federated reads.
- [ ] **Step 1: Compare complete endpoint tuples across engines**
Compare sorted tuples containing `from_source_id`, `from_slug`, `to_source_id`, `to_slug`, `origin_source_id`, and `origin_slug` for scalar and federated fixtures.
- [ ] **Step 2: Run focused PGLite/source-isolation tests**
```bash
bun test test/writer.test.ts test/post-write-lint.test.ts test/get-page-federated-scope.test.ts test/e2e/multi-source-bug-class.test.ts > "$TEMP/backlink-focused.txt" 2>&1
```
Expected: exit 0.
- [ ] **Step 3: Run PostgreSQL parity when the test database is available**
```bash
bun test test/e2e/engine-parity.test.ts -t "federated sourceIds" --timeout=300000 > "$TEMP/backlink-parity.txt" 2>&1
```
Expected: exit 0; if the configured test database is unavailable, report the exact environmental blocker rather than claiming parity execution.
- [ ] **Step 4: Typecheck and inspect the final diff**
```bash
bun run typecheck > "$TEMP/backlink-typecheck.txt" 2>&1
```
Expected: exit 0. Then run `git diff --check` and confirm no version, schema, migration, deployment, or conditional-write files changed.
@@ -0,0 +1,142 @@
# Engine dynamic-import reconciliation design
**Date:** 2026-07-28
## Goal
Reconcile the overlapping engine dynamic-import changes from:
- `claude/hungry-edison-8bb1cd` at release commits `48ada48f` and `248bfe55`
- `claude/elegant-gates-e5275e` at `ef4cf7a8`
onto a fresh branch from current `origin/master`, without merging or cherry-picking either lineage wholesale and without adding a release/version bump.
## Established state
At investigation time:
- `origin/master` was `6136e139972a5449630b4f47f5ed7b4cbe5b811b`, version `0.42.67.0`.
- Upstream PR #3511 was still open, so trunk did not contain its two `chronicle/ontology.ts` hoists.
- Neither source branch was an ancestor of trunk.
- Trunk contained 17 dynamic imports in the three engine-path files:
- 13 safe-hoist candidates: two ontology imports, nine engine helper/audit imports, and two migration imports.
- Four `ai/gateway.ts` imports, all inside `try/catch` fallback paths.
- `git log -G` showed the separate ontology, helper, migration, and gateway histories. `git log -S` is not suitable for this dynamic-to-static replacement because the relevant token can remain present while its context changes.
- The guard from `ef4cf7a8` passed against that commit but failed against trunk. It also knew about only two gateway opt-outs because two `_upsertChunksOnce` gateway lookups landed later in trunk.
## Selected approach
Reconstruct the intended current state directly on fresh `origin/master`.
Do not merge or cherry-pick either old lineage. Selectively reproduce the desired source changes, adapt the guard to the current four gateway call sites, and write current-state documentation. This avoids importing stale release metadata, stale TODO claims, and unrelated lineage changes.
## Source changes
### Safe static imports
Hoist all 13 safe candidates:
- `src/core/pglite-engine.ts`
- `valueHash`, `normalizeDimension`, `isNovelDimension` from `chronicle/ontology.ts`
- `isRetryableConnError` through the existing `retry.ts` import
- `resolveRecencyDecayMap`, `DEFAULT_FALLBACK` from `search/recency-decay.ts`
- `src/core/postgres-engine.ts`
- the same ontology, retry, and recency helpers
- `isConnectionEndedError` from `retry-matcher.ts`
- `logDbDisconnect` from `audit/db-disconnect-audit.ts`
- `logPoolRecovery` from `audit/pool-recovery-audit.ts`
- `src/core/migrate.ts`
- `isStatementTimeoutError`, `isRetryableConnError` from `retry-matcher.ts`
- `repairTimelineDedupIndex` from `timeline-dedup-repair.ts`
The implementation must keep the two engines in parity where the behavior is shared. Comments should describe current invariants, not repeat an unproven causal claim that these hoists fix the Windows test-runner crash.
### Deliberately lazy gateway imports
Keep all four `await import('./ai/gateway.ts')` call sites lazy:
- PGLite `initSchema`
- PGLite `_upsertChunksOnce`
- Postgres `initSchema`
- Postgres `_upsertChunksOnce`
Each line receives the explicit `engine-dynamic-import-ok` marker and a concise nearby rationale.
The rationale has two parts:
1. The gateway's static closure includes the AI SDK, provider packages, and validation/config machinery, so eager loading would tax engine startup paths that do not otherwise need it.
2. More importantly, each lookup is inside a `try/catch` that preserves a soft fallback (compiled defaults or the brain's stored embedding-model config). Hoisting the module would evaluate it before that catch can run and could convert a recoverable configuration/import failure into a module-load-time hard failure.
The guard must not allow unmarked gateway imports or a broad file-level exemption.
## Guard and wiring
Add `scripts/check-engine-dynamic-import.sh`, adapted from `ef4cf7a8`, with these properties:
- Default scan set:
- `src/core/pglite-engine.ts`
- `src/core/postgres-engine.ts`
- `src/core/migrate.ts`
- Normalize trailing CR before matching so CRLF checkouts cannot bypass the check.
- Ignore comment-only lines.
- Ignore only lines carrying `engine-dynamic-import-ok`.
- Report every unmarked `await import(` with file and line.
- Explain that contributors should prefer a static import and must justify a real opt-out.
- Avoid asserting that every dynamic import deterministically crashes Windows; the measured evidence supports treating the pattern as an engine-path hardening invariant, while box-level commit exhaustion remained a confound in prior runs.
Wire it into:
- `package.json` as `check:engine-dynamic-import`
- `package.json` `check:all`
- `scripts/run-verify-parallel.sh`
Follow trunk's current rule that package scripts invoke repository shell scripts through `bash`.
## Regression coverage
Add an automated test for the guard. It must cover:
- A real dynamic import produces exit 1 and is reported.
- A line carrying `engine-dynamic-import-ok` is allowed.
- Line comments and block-comment lines do not produce findings.
- The same violation is caught with CRLF input.
- The default repository scan passes after the source reconciliation.
Use a temporary fixture rather than mutating tracked source files. Keep assertions path-portable.
The pre-fix red demonstration is the exact guard from `ef4cf7a8` run against current trunk: it exits 1 and reports the existing unmarked imports. The post-fix guard and test must pass.
## Documentation policy
Preserve current behavior, not either old release narrative:
- Do not modify `VERSION` or add a release `CHANGELOG.md` entry.
- Do not copy old version headings or completed release TODO blocks.
- Do not retain the old TODO claiming that extracting gateway accessors is necessarily the fix; the lazy imports are deliberately protected by their local soft-failure boundaries.
- Add the cross-cutting no-unmarked-dynamic-import invariant to `CLAUDE.md`.
- Update the current-state entries for `src/core/pglite-engine.ts`, `src/core/postgres-engine.ts`, and `src/core/migrate.ts` in `docs/architecture/KEY_FILES.md` where needed.
- Regenerate `llms.txt` and `llms-full.txt` after the documentation edits.
- Add a TODO only if implementation uncovers a real unresolved action.
Public documentation must use generic language and must not overstate the historical Windows crash causality.
## Verification
Capture full output to files before inspecting summaries. Run, at minimum:
1. The guard regression test.
2. `bash scripts/check-engine-dynamic-import.sh`.
3. Focused tests that exercise the touched engine, migration, retry, audit, and recency modules.
4. `bun run typecheck`.
5. `bun run verify`.
6. `bun run build:llms` followed by `bun test test/build-llms.test.ts`.
7. `git diff --check` and a final clean-status/diff review.
If platform contention or existing Windows suite defects block a broad test, report the exact command, exit code, and ownership classification rather than declaring success from a partial run.
## Git and publication boundary
- Work on `claude/kind-meitner-330c90`, reset locally to the exact investigated `origin/master` base.
- Preserve the previous worktree tip under `claude/kind-meitner-330c90-pre-reconcile`.
- Keep implementation and verification commits local.
- Do not push, create a PR, comment upstream, or otherwise publish without explicit user approval after the local result is complete.
@@ -0,0 +1,184 @@
# Scalar-source backlink validation design
## Problem
A page identity in a multi-source brain is `(source_id, slug)`, but the back-link validator currently reasons only about `slug`.
For an outbound edge:
```text
(source-a, concepts/origin) -> (source-a, people/target)
```
the validator accepts any reverse row whose bare slugs are:
```text
people/target -> concepts/origin
```
That can incorrectly accept a row ending at `(default, concepts/origin)` instead of `(source-a, concepts/origin)`.
The bug is not that scalar `getLinks(slug, { sourceId })` permits cross-source destinations. That behavior is intentional: scalar scope qualifies the near/from endpoint while trusted local callers retain visibility into explicit cross-source edges. The gap is that a returned `Link` does not carry the source identity of either endpoint, so callers cannot distinguish same-slug pages.
## Reproduction and evidence
A deterministic PGLite reproduction creates duplicate `concepts/a` and `people/b` pages in `default` and `team-x`, then adds:
```text
(team-x, concepts/a) -> (team-x, people/b)
(team-x, people/b) -> (default, concepts/a)
```
The second edge is not a valid reverse of the first. Nevertheless:
```ts
await engine.getLinks('people/b', { sourceId: 'team-x' })
```
returns the second row, and the current validator accepts it because `to_slug === 'concepts/a'`.
Both engines implement the same scalar rule: filter `f.slug` and `f.source_id`, join the actual destination by `to_page_id`, and do not filter `t.source_id`. Federated `sourceIds` is a separate branch that constrains all visible endpoints and takes precedence over scalar scope.
## Goals
1. Validate back-links by exact source-qualified endpoint identity.
2. Preserve explicit cross-source links for trusted scalar reads.
3. Preserve federated all-endpoint containment and `sourceIds` precedence.
4. Keep PostgreSQL and PGLite behavior identical.
5. Add strict red-before-green regressions using duplicate slugs across sources.
6. Avoid schema migrations and production operational changes.
## Non-goals
- Changing scalar link reads to same-source-only reads.
- Weakening or widening federated reads.
- Changing link write identity or database schema.
- Refactoring the atomic conditional-write branch.
- Coupling deployment, restart, or migration mechanics to the backlink code change. Release operations are handled separately after verification.
## Chosen approach
Extend the engine `Link` result with endpoint source identities and use those fields in the validator.
```ts
interface Link {
from_slug: string;
from_source_id: string;
to_slug: string;
to_source_id: string;
// existing fields
origin_slug?: string | null;
origin_source_id?: string | null;
}
```
All `getLinks` and `getBacklinks` query branches in PostgreSQL and PGLite will project the source IDs from the pages already joined as `f`, `t`, and `o`. No filtering behavior changes.
This approach is preferred over a dedicated `hasExactLink` method because it keeps source identity attached to the link data everywhere, avoids duplicate engine SQL and per-edge existence queries, and matches existing source-qualified link-write and batch-row contracts.
Validator-only raw SQL is rejected because validators should consume the `BrainEngine` contract rather than bypass it with engine-specific schema knowledge.
## Engine semantics
The existing three read modes remain unchanged.
### Unscoped
`getLinks(slug)` returns rows from all same-slug from-pages across sources. Each row identifies the actual source of both endpoints.
### Scalar source
`getLinks(slug, { sourceId })` matches exactly `(sourceId, slug)` on the from side. A destination may belong to another source, and `to_source_id` reveals that exact identity.
The corresponding scalar `getBacklinks` rule continues to match the exact destination/to-page identity while allowing a cross-source referrer.
### Federated sources
`getLinks(slug, { sourceIds })` continues to constrain from and to endpoints to the grant. The origin join continues to redact an out-of-grant origin. `sourceIds` continues to take precedence over scalar `sourceId`.
Adding source IDs to returned in-grant endpoints does not disclose anything new: the existing result already discloses those pages' slugs and edges. An out-of-grant endpoint remains absent.
## Validator algorithm
The validator receives the source scope associated with the page being validated.
For every outbound edge:
```text
(from_source_id, from_slug) -> (to_source_id, to_slug)
```
it requires a reverse row:
```text
(to_source_id, to_slug) -> (from_source_id, from_slug)
```
Duplicate edge rows are deduplicated by the full endpoint pair `(from_source_id, from_slug, to_source_id, to_slug)`, not by bare target slug. This preserves separate reverse requirements when multiple same-slug origin pages point to one exact target.
For each target:
1. Read target outbound links using the target's exact scalar source when validation is scalar-scoped.
2. Under federated validation, retain the caller's `sourceIds` grant rather than converting it to scalar scope.
3. Accept only a returned row whose `from_source_id`, `from_slug`, `to_source_id`, and `to_slug` exactly match the expected reverse identity.
4. Emit the existing warning when no exact reverse exists.
This preserves legitimate cross-source pairs. For example:
```text
(source-a, concepts/origin) -> (source-b, people/target)
(source-b, people/target) -> (source-a, concepts/origin)
```
is valid.
## Validation context propagation
`PageValidationContext` must carry the relevant scalar or federated source scope. The writer and post-write lint paths must load the page with that scope and pass the same scope to nested validator reads.
This change is scoped to source routing needed by validation. It does not modify conditional-write revision or conflict semantics and must not be applied to the atomic conditional-write branch.
## Testing strategy
### PGLite strict-TDD regression
Add duplicate pages across `default` and a second source, then prove before the production fix that:
1. A forward edge in the second source plus a wrong-source reverse produces a warning.
2. Adding the exact reverse removes the warning.
3. A legitimate cross-source forward/reverse pair passes.
4. Two same-slug destination pages are not collapsed into one target identity.
The first assertion must fail against the pre-fix implementation.
### Engine contract tests
For PGLite and PostgreSQL:
1. Assert link rows expose exact from/to source IDs.
2. Assert scalar reads still return explicit cross-source destinations.
3. Assert federated reads still exclude out-of-grant endpoints.
4. Assert `sourceIds` still takes precedence over scalar `sourceId`.
5. Assert origin source identity is null when the origin is redacted by the federated branch.
### Parity and focused verification
Run:
- the focused backlink validator test;
- source-isolation and federated link tests;
- the Postgres/PGLite parity fixture with a test database;
- related writer/post-write tests;
- `bun run typecheck`.
Capture complete command output to files before inspecting summaries. Do not use production databases or restart the live service.
## Compatibility
The `Link` change is additive at runtime. Existing consumers that read only slug or provenance fields continue to work. TypeScript object literals typed as complete `Link` values may need source fields; if compatibility pressure is high, the source fields can initially be optional in the public type while engine implementations and validator tests require their presence. The preferred contract is required endpoint source IDs because every persisted link always has both pages and therefore both source IDs.
No schema migration is required because source IDs already live on the joined `pages` rows.
## Operational constraints
The implementation phase does not deploy, restart GBrain, run production migrations, or alter the atomic conditional-write branch. Release, migration, and restart operations are a separate verified workflow and do not change this design's engine or validator semantics.
+1 -1
View File
@@ -13,7 +13,7 @@ Step-by-step walkthroughs that take you from zero to a working outcome. Concrete
These are the next tutorials on the roadmap. Open an issue if one of them is the one you need most; that's how we'll prioritize.
- **Set up GBrain for VC dealflow** — the operator's recipe. People pages for founders, companies with typed Facts fence carrying ARR / team-size / runway across dates, meetings auto-ingested, deal pages linking everything. Shows `gbrain whoknows`, `gbrain find_trajectory`, and `gbrain founder scorecard` on real workflows.
- **Set up GBrain for VC dealflow** — the operator's recipe. People pages for founders, companies with typed Facts fence carrying ARR / team-size / runway across dates, meetings auto-ingested, deal pages linking everything. Shows `gbrain whoknows`, `gbrain find-trajectory`, and `gbrain founder scorecard` on real workflows.
- **Migrate your existing vault into GBrain** — for Notion / Obsidian / Roam users with a vault that doesn't match GBrain's default layout. Walks through `gbrain schema detect``suggest``review-candidates` so the brain learns your shape instead of forcing you to learn its.
+3 -3
View File
@@ -93,7 +93,7 @@ There are two ways to scope teammates' access. They suit different deployment sh
**Model A: separate sources with OAuth scoping (recommended for true multi-user with different AI clients).** What this tutorial walks you through. Each teammate gets their own OAuth client, which carries `--source` + `--federated-read` flags. The brain refuses cross-source reads at the SQL layer; isolation is database-enforced. Each teammate can run their own MCP-aware client (Claude Code, Cursor, their own OpenClaw, etc.) and the scoping holds.
**Model B: one source, directory-based per-person scoping (simpler for one-agent-serves-everyone setups).** The shape I actually run in production: a single source called `default`, with a `partners/<slug>/` convention inside it (e.g. `partners/alice-example/`, `partners/bob-example/`). Each partner gets their own subdirectory holding their personal pages: `partners/alice-example/USER.md`, `partners/alice-example/concepts/`, `partners/alice-example/sources/`, etc. There's no OAuth-enforced isolation; the agent itself enforces "Alice's writes go to her partners/ subdir." This is the right model when ONE agent (yours) serves everyone over Telegram or a single shared interface. It's simpler ops, no per-user OAuth, but the scoping is convention-only.
**Model B: one source, directory-based per-person scoping (simpler for one-agent-serves-everyone setups).** The shape I actually run in production: a single source called `default`, with a `partners/<slug>/` convention inside it (e.g. `partners/alice-example/`, `partners/bob-example/`). Each partner gets their own subdirectory holding their personal pages: `partners/alice-example/USER.md`, `partners/alice-example/concepts/`, `partners/alice-example/sources/`, etc. This is the right model when ONE agent (yours) serves everyone over Telegram or a single shared interface. It's simpler ops, no per-user OAuth. **Write scoping within the shared source can be server-enforced:** register each per-person client with `--bound-slug-prefixes partners/alice-example/` and every slug-mutating write outside that prefix is rejected with `permission_denied` (v0.42.72.0+). Without the binding, the scoping is convention-only (the agent polices itself). Read scoping stays source-granular in both models — within a shared source, everyone entitled to the source can read every folder.
For most company-brain installs (10+ teammates each with their own AI client), Model A is the right starting point. If you're running the fat-agent-serves-everyone pattern from the personal-brain tutorial, Model B is genuinely simpler. You can also mix: separate sources for the obviously-different ones (customer notes vs internal-only) AND a `partners/<slug>/` convention inside the shared source for per-person workspace.
@@ -210,7 +210,7 @@ Each `register-client` command prints a `client_id` and a `client_secret`. Save
A note on the flags:
- `--scopes read,write` lets the client query the brain and write new pages. You can omit `write` for read-only clients (executive summaries, dashboards). The `admin` scope is needed for operational commands like `gbrain remote doctor` and is usually reserved for your own admin client.
- `--source` controls write authority. A client can only write to one source. Within that source, your folder convention from Part 3 keeps each person's writes in their own subfolder.
- `--source` controls write authority. A client can only write to one source. Within that source, your folder convention from Part 3 keeps each person's writes in their own subfolder — and you can make that server-enforced with `--bound-slug-prefixes alice-example/` (v0.42.72.0+): every slug-mutating write op (put_page, delete_page, tags, links, timeline, revert, raw data) outside the bound prefixes is rejected with `permission_denied`. Update the binding later with `gbrain auth rescope-client <id> --bound-slug-prefixes <p1,p2|none>`. **Adding a binding to an existing client narrows it in ways you should expect:** ops that write by something other than a slug (`extract_entities`, `extract_facts`, `forget_fact`, `ontology_propose`, `sources_add`/`sources_remove`) and `POST /ingest` become unavailable to that client, and `put_page`'s automatic fact extraction is skipped — all because none of them can be confined to a prefix. Reads are unaffected. See [the qm-harness guide](../integrations/qm-harness.md) for the full model.
- `--federated-read` controls read scope. A client can read from one or more sources.
### Verify the scoping actually scopes
@@ -554,7 +554,7 @@ What to do next:
- **Wire ingestion** from external systems (Granola, Linear, Slack) using the [ingestion source contract](../skillpack-anatomy.md). Most companies want their meetings auto-ingested so the brain stays current without anyone typing notes.
- **Set up team-specific dashboards** through the admin UI. Each team lead can have their own view of brain health and activity.
- **Explore the rest of the brain layer.** `gbrain whoknows` (find the expert on a topic), `gbrain find_trajectory` (how a metric changed over time), `gbrain founder scorecard` (especially useful for VC and ops teams), the contradiction-detection cycle that surfaces conflicts between different people's notes.
- **Explore the rest of the brain layer.** `gbrain whoknows` (find the expert on a topic), `gbrain find-trajectory` (how a metric changed over time), `gbrain founder scorecard` (especially useful for VC and ops teams), the contradiction-detection cycle that surfaces conflicts between different people's notes.
If you're building in this space (which YC has flagged as the [company-brain category in its Request for Startups](https://www.ycombinator.com/rfs#company-brain)), you might as well build on this. Everything described above is open source, MIT licensed, and what I run in production behind my own AI agents.
+9 -9
View File
@@ -115,21 +115,21 @@ You can use the same keys across multiple agents.
## Step 6: Install GBrain
Once OpenClaw is running:
Once OpenClaw is running, installation is two commands — one in the brain repo, one in the agent workspace:
```bash
gbrain install
# In the BRAIN repo (the git repo that holds your markdown pages):
gbrain init --supabase
# In the AGENT WORKSPACE repo (where OpenClaw runs):
gbrain skillpack scaffold --all
```
This installs:
`gbrain init --supabase` walks a short wizard that asks for your Supabase connection string and creates the schema. You'll get that connection string in Step 7 — read 7a and 7b first so you paste the right one (the transaction pooler, not the direct connection). If you'd rather try things locally before paying for a database, `gbrain init --pglite` gives you a zero-config embedded engine instead; you can migrate to Supabase later with `gbrain migrate --to supabase`.
- About 60 skills
- About 9 skill packs
- Default brain structure
- MCP server configuration
- Supabase connection (for embeddings and search)
`gbrain skillpack scaffold --all` copies the ~43 bundled skills into your agent workspace as first-class files you can edit freely. (The old managed-install model was retired in v0.36.0.0; see `docs/INSTALL.md` if you're upgrading from an older release.)
GBrain populates the brain repo with its default directory structure, skill files, and configuration. From this point, the agent has working memory and access to every skill.
From this point, the agent has working memory and access to every skill.
---
+3 -3
View File
@@ -32,7 +32,7 @@ gbrain schema sync --apply
The sync backfills `page.type = 'meeting'` on all 4000 pages in 1000-row batches. Now:
- `gbrain whoknows "Q3 roadmap discussion"` routes through the meeting type, ranking by `expert_routing` signal (attendees, recency, salience) instead of raw text.
- `gbrain extract-facts` runs on every meeting page automatically (because `extractable: true`), pulling typed facts like `attended_by=alice-example`, `date=2026-05-23`.
- The `extract_facts` cycle runs on every meeting page automatically (because `extractable: true`), pulling typed facts like `attended_by=alice-example`, `date=2026-05-23`.
- The downstream `think` skill can now answer "what did we decide about pricing in the last three roadmap meetings" by querying the meeting graph instead of grep'ing 4000 files.
One command. 4000 pages went from invisible to queryable. The content didn't change. The structure did.
@@ -62,7 +62,7 @@ gbrain schema add-link-type led-by --page-type deal --target-type inves
gbrain schema sync --apply
```
Now `gbrain whoknows "Series A SaaS"` routes through `investor` and `portco` types specifically, not the noisy general type set. `gbrain graph-query alice-example --type intro-from --depth 2` walks two hops of intros to surface "Alice introduced you to Bob who introduced you to Charlie." `gbrain extract-facts` starts producing typed claims from the fence in your deal pages: `(deals/acme-seed, raise=2000000, valuation=15000000, lead=widget-vc, closed_at=2026-05-23)`.
Now `gbrain whoknows "Series A SaaS"` routes through `investor` and `portco` types specifically, not the noisy general type set. `gbrain graph-query alice-example --type intro-from --depth 2` walks two hops of intros to surface "Alice introduced you to Bob who introduced you to Charlie." The `extract_facts` cycle starts producing typed claims from the fence in your deal pages: `(deals/acme-seed, raise=2000000, valuation=15000000, lead=widget-vc, closed_at=2026-05-23)`.
The CRM you've been promising yourself you'll set up next quarter? You just shipped it in 4 commands. It's downstream of your notes, not parallel to them.
@@ -143,7 +143,7 @@ Re-run the same `whoknows` query. Top-3 should shift, because the new type is no
Three things gbrain does that generic note systems can't:
**1. The brain knows the difference between a person and an idea.** Page-type matters at query time. `gbrain whoknows` only considers `expert_routing: true` types. `gbrain extract-facts` only runs on `extractable: true` types. `gbrain graph-query` walks declared link verbs. None of that works on a flat tag system because tags don't have semantics — they're labels. Types are first-class citizens with rules attached.
**1. The brain knows the difference between a person and an idea.** Page-type matters at query time. `gbrain whoknows` only considers `expert_routing: true` types. The `extract_facts` cycle only runs on `extractable: true` types. `gbrain graph-query` walks declared link verbs. None of that works on a flat tag system because tags don't have semantics — they're labels. Types are first-class citizens with rules attached.
**2. Untyped content is invisible content.** If your meetings are typed as `note`, expert routing skips them, facts extraction ignores them, link inference doesn't fire. They exist on disk and they're indexed for text search, but the structural surfaces (whoknows, find_experts, recall, think) treat them as second-class. Adding a type isn't cosmetic; it's structural promotion.
+35 -5
View File
@@ -216,6 +216,19 @@ Per-file detail is in `docs/architecture/KEY_FILES.md`.
text, the cast parses it). Guarded by `scripts/check-jsonb-pattern.sh` (template grep) +
`scripts/check-jsonb-params.mjs` (positional AST scanner); the real backstop is the DATABASE_URL-gated
e2e parity tests, since PGLite can't surface the bug. Full rule in `docs/ENGINES.md`.
- **Engine-live paths avoid runtime dynamic `import()` for helper dependencies.** In
`src/core/pglite-engine.ts`, `src/core/postgres-engine.ts`, and
`src/core/migrate.ts`, dependencies previously reached through runtime dynamic
imports use static top-level imports. The only current dynamic-`import()` exceptions
are the four `ai/gateway.ts` lookups in both engines'
`initSchema()` and `_upsertChunksOnce()` methods; each remains lazy inside a
local `try/catch` because the gateway has a large provider/config closure and,
more importantly, eager evaluation would occur before the catch and could
turn a recoverable default/config-row fallback into a module-load failure.
Every exception carries `engine-dynamic-import-ok` on the import line.
`scripts/check-engine-dynamic-import.sh` enforces the rule. For history, use
`git log -G'await[[:space:]]+import\\('`, not `git log -S`: a dynamic-to-static
rewrite can preserve the searched token while changing its context.
- **Engine parity.** `src/core/postgres-engine.ts` and `src/core/pglite-engine.ts` move in
lockstep — a new method/SQL shape lands in BOTH, pinned by `test/e2e/engine-parity.test.ts`.
Forward-referenced columns/indexes go in the bootstrap probe set (guarded by
@@ -1006,6 +1019,13 @@ If you fetched this file by URL without cloning yet, the companion files live at
## Step 1: Install GBrain
> **NEVER install from the npm registry.** GBrain is not distributed on npm; the npm
> package named `gbrain` is an unrelated package. Do NOT run `npm install -g gbrain` or
> `bun add -g gbrain` (note the missing `github:` prefix — that's the trap). The only
> supported sources are `github:garrytan/gbrain` and a git clone, exactly as shown below.
> If an unrelated npm install is already present, remove it first
> (`npm uninstall -g gbrain` / `bun remove -g gbrain`); `gbrain doctor` also detects this.
Default path (Bun is required — gbrain is a Bun + TypeScript runtime):
```bash
@@ -1559,6 +1579,16 @@ This is the difference between a search engine and a brain. Search finds the pag
## Install
> [!WARNING]
> **GBrain is NOT distributed on npm.** The npm package named `gbrain` is an unrelated
> package with no connection to this project. Do not run `npm install -g gbrain` or
> `bun add -g gbrain` — you'll get something else, and it can shadow the real binary on
> your PATH. Install and upgrade ONLY via the documented paths below
> (`bun install -g github:garrytan/gbrain`, or `git clone` + `bun install && bun link`).
> If you already ran the npm install by mistake: `npm uninstall -g gbrain` /
> `bun remove -g gbrain`, then reinstall from GitHub. `gbrain doctor` detects a
> shadowing npm install and prints the fix.
GBrain is designed to be installed and operated by an AI agent. The fastest path is to have your agent do it for you. The CLI and MCP paths below are for people who want to wire it up themselves.
### Have your agent install it (recommended)
@@ -1700,7 +1730,7 @@ Most personal-knowledge tools force one fixed layout: their idea of "notes" + "p
**gbrain doesn't have a fixed layout.** It ships with bundled schema packs and lets you author your own when none fit:
- **`gbrain-base-v2`** (default as of v0.41.22) — 15-type DRY/MECE canonical taxonomy (14 canonical + `note` catch-all): `person`, `company`, `media`, `tweet`, `social-digest`, `analysis`, `atom`, `concept`, `source`, `deal`, `email`, `slack`, `writing`, `project`, `note`. Subtypes/format/origin pushed to frontmatter. The taxonomy that responds to issue #1479.
- **`gbrain-base`** (legacy, v0.41 and earlier brains) — the original 24-type layout. Stays bundled for back-compat; brains on it can upgrade via `gbrain onboard --check --explain` → `gbrain jobs submit unify-types --allow-protected --params '{"target_pack":"gbrain-base-v2"}'`.
- **`gbrain-base`** (legacy, v0.41 and earlier brains) — the original 24-type layout. Stays bundled for back-compat; brains on it can upgrade via `gbrain onboard --check --explain` → `gbrain jobs submit unify-types --allow-protected --params '{"target_pack":"gbrain-base-v2","apply":true}'` (omit `"apply":true` for a dry-run preview — that is the default).
- **`gbrain-recommended`** — extends `gbrain-base` with the 13 additional directories from `docs/GBRAIN_RECOMMENDED_SCHEMA.md` (source, place, trip, conversation, personal, civic, project, etc.). Activate with `gbrain schema use gbrain-recommended`.
- **Your own pack** — `gbrain schema detect` clusters your actual filesystem into proposed types, `gbrain schema suggest` runs an LLM pass over them, and `gbrain schema review-candidates --apply` promotes the ones you like. Three commands and the brain knows your shape. Authoring a successor pack (declares `migration_from:` so existing brains can opt in): see [`docs/architecture/pack-upgrade-mechanism.md`](docs/architecture/pack-upgrade-mechanism.md).
@@ -2316,7 +2346,7 @@ gbrain schema sync --apply
The sync backfills `page.type = 'meeting'` on all 4000 pages in 1000-row batches. Now:
- `gbrain whoknows "Q3 roadmap discussion"` routes through the meeting type, ranking by `expert_routing` signal (attendees, recency, salience) instead of raw text.
- `gbrain extract-facts` runs on every meeting page automatically (because `extractable: true`), pulling typed facts like `attended_by=alice-example`, `date=2026-05-23`.
- The `extract_facts` cycle runs on every meeting page automatically (because `extractable: true`), pulling typed facts like `attended_by=alice-example`, `date=2026-05-23`.
- The downstream `think` skill can now answer "what did we decide about pricing in the last three roadmap meetings" by querying the meeting graph instead of grep'ing 4000 files.
One command. 4000 pages went from invisible to queryable. The content didn't change. The structure did.
@@ -2346,7 +2376,7 @@ gbrain schema add-link-type led-by --page-type deal --target-type inves
gbrain schema sync --apply
```
Now `gbrain whoknows "Series A SaaS"` routes through `investor` and `portco` types specifically, not the noisy general type set. `gbrain graph-query alice-example --type intro-from --depth 2` walks two hops of intros to surface "Alice introduced you to Bob who introduced you to Charlie." `gbrain extract-facts` starts producing typed claims from the fence in your deal pages: `(deals/acme-seed, raise=2000000, valuation=15000000, lead=widget-vc, closed_at=2026-05-23)`.
Now `gbrain whoknows "Series A SaaS"` routes through `investor` and `portco` types specifically, not the noisy general type set. `gbrain graph-query alice-example --type intro-from --depth 2` walks two hops of intros to surface "Alice introduced you to Bob who introduced you to Charlie." The `extract_facts` cycle starts producing typed claims from the fence in your deal pages: `(deals/acme-seed, raise=2000000, valuation=15000000, lead=widget-vc, closed_at=2026-05-23)`.
The CRM you've been promising yourself you'll set up next quarter? You just shipped it in 4 commands. It's downstream of your notes, not parallel to them.
@@ -2427,7 +2457,7 @@ Re-run the same `whoknows` query. Top-3 should shift, because the new type is no
Three things gbrain does that generic note systems can't:
**1. The brain knows the difference between a person and an idea.** Page-type matters at query time. `gbrain whoknows` only considers `expert_routing: true` types. `gbrain extract-facts` only runs on `extractable: true` types. `gbrain graph-query` walks declared link verbs. None of that works on a flat tag system because tags don't have semantics — they're labels. Types are first-class citizens with rules attached.
**1. The brain knows the difference between a person and an idea.** Page-type matters at query time. `gbrain whoknows` only considers `expert_routing: true` types. The `extract_facts` cycle only runs on `extractable: true` types. `gbrain graph-query` walks declared link verbs. None of that works on a flat tag system because tags don't have semantics — they're labels. Types are first-class citizens with rules attached.
**2. Untyped content is invisible content.** If your meetings are typed as `note`, expert routing skips them, facts extraction ignores them, link inference doesn't fire. They exist on disk and they're indexed for text search, but the structural surfaces (whoknows, find_experts, recall, think) treat them as second-class. Adding a type isn't cosmetic; it's structural promotion.
@@ -3897,7 +3927,7 @@ All 30 GBrain operations are available remotely, including `sync_brain` and
directory where `gbrain serve` was launched. Symlinks, `..` traversal, and absolute
paths outside cwd are rejected. Page slugs and filenames are allowlist-validated
(alphanumeric + hyphens; no control chars, RTL overrides, or backslashes). Local
CLI callers (`gbrain file upload ...`) keep unrestricted filesystem access since
CLI callers (`gbrain files upload ...`) keep unrestricted filesystem access since
the user owns the machine.
## Deployment Options
+7 -5
View File
@@ -48,7 +48,8 @@
"check:system-of-record": "bash scripts/check-system-of-record.sh",
"check:admin-scope-drift": "bash scripts/check-admin-scope-drift.sh",
"check:cli-exec": "bash scripts/check-cli-executable.sh",
"check:all": "bash scripts/check-privacy.sh && bash scripts/check-proposal-pii.sh && bash scripts/check-test-real-names.sh && bash scripts/check-jsonb-pattern.sh && bash scripts/check-source-id-projection.sh && bash scripts/check-source-config-leak.sh && bash scripts/check-progress-to-stdout.sh && bash scripts/check-no-tracked-symlinks.sh && bash scripts/check-no-legacy-getconnection.sh && bash scripts/check-test-isolation.sh && bash scripts/check-trailing-newline.sh && bash scripts/check-wasm-embedded.sh && bash scripts/check-exports-count.sh && bash scripts/check-admin-build.sh && bash scripts/check-admin-scope-drift.sh && bash scripts/check-cli-executable.sh && bash scripts/check-skill-brain-first.sh && bash scripts/check-operations-filter-bypass.sh && bash scripts/check-gateway-routed-no-direct-anthropic.sh && bash scripts/check-worker-pool-atomicity.sh && bash scripts/check-key-files-current-state.sh && bash scripts/check-no-double-retry.sh && bash scripts/check-batch-audit-site.sh",
"check:engine-dynamic-import": "bash scripts/check-engine-dynamic-import.sh",
"check:all": "bash scripts/check-privacy.sh && bash scripts/check-proposal-pii.sh && bash scripts/check-test-real-names.sh && bash scripts/check-jsonb-pattern.sh && bash scripts/check-source-id-projection.sh && bash scripts/check-source-config-leak.sh && bash scripts/check-progress-to-stdout.sh && bash scripts/check-no-tracked-symlinks.sh && bash scripts/check-no-legacy-getconnection.sh && bash scripts/check-test-isolation.sh && bash scripts/check-trailing-newline.sh && bash scripts/check-wasm-embedded.sh && bash scripts/check-exports-count.sh && bash scripts/check-admin-build.sh && bash scripts/check-admin-scope-drift.sh && bash scripts/check-cli-executable.sh && bash scripts/check-skill-brain-first.sh && bash scripts/check-operations-filter-bypass.sh && bash scripts/check-gateway-routed-no-direct-anthropic.sh && bash scripts/check-worker-pool-atomicity.sh && bash scripts/check-key-files-current-state.sh && bash scripts/check-no-double-retry.sh && bash scripts/check-batch-audit-site.sh && bash scripts/check-engine-dynamic-import.sh",
"check:gateway-routed": "bash scripts/check-gateway-routed-no-direct-anthropic.sh",
"check:worker-pool-atomicity": "bash scripts/check-worker-pool-atomicity.sh",
"check:doc-history": "bash scripts/check-key-files-current-state.sh",
@@ -74,6 +75,7 @@
"check:privacy": "bash scripts/check-privacy.sh",
"check:proposal-pii": "bash scripts/check-proposal-pii.sh",
"check:eval-glossary": "bash scripts/check-eval-glossary-fresh.sh",
"check:skills-manifest": "bash scripts/check-skills-manifest-fresh.sh",
"check:test-names": "bash scripts/check-test-real-names.sh",
"check:progress": "bash scripts/check-progress-to-stdout.sh",
"check:no-tracked-symlinks": "bash scripts/check-no-tracked-symlinks.sh",
@@ -146,16 +148,16 @@
"bun": ">=1.3.10"
},
"license": "MIT",
"version": "0.42.67.0",
"version": "0.42.73.0",
"overrides": {
"@hono/node-server": "^2.0.5",
"fast-uri": "^3.1.4",
"fast-uri": "^3.1.5",
"body-parser": "^2.3.0",
"fast-xml-builder": "^1.1.7",
"fast-xml-parser": "^5.7.0",
"form-data": "^4.0.6",
"hono": "^4.12.25",
"ip-address": "^10.1.1",
"hono": "^4.12.34",
"ip-address": "^10.3.1",
"qs": "^6.15.2",
"js-yaml": "^3.15.0"
}
+6 -1
View File
@@ -1,7 +1,7 @@
---
id: calendar-to-brain
name: Calendar-to-Brain
version: 0.7.0
version: 0.8.0
description: Google Calendar events become searchable brain pages. Daily files with attendees, locations, and meeting prep context.
category: sense
requires: [credential-gateway]
@@ -28,6 +28,11 @@ health_checks:
- type: env_exists
name: GOOGLE_CLIENT_ID
label: "Google OAuth"
- type: heartbeat_max_age
max_age: 48h
label: "Calendar data freshness"
output_paths:
- daily/calendar/
setup_time: 20 min
cost_estimate: "$0 (both options are free)"
---
+15 -7
View File
@@ -1,12 +1,12 @@
---
id: x-to-brain
name: X-to-Brain
version: 0.8.2
version: 0.8.3
description: Twitter timeline, mentions, and keyword monitoring flow into brain pages. Tracks deletions, engagement velocity, OCR on images, and real-time alerts.
category: sense
requires: []
secrets:
- name: X_BEARER_TOKEN
- name: X_API_BEARER_TOKEN
description: X API v2 Bearer token (Basic tier minimum, $200/mo for full archive search)
where: https://developer.x.com/en/portal/dashboard — create a project + app, copy the Bearer Token from "Keys and tokens"
- name: X_HANDLE
@@ -16,7 +16,7 @@ health_checks:
- type: http
url: "https://api.x.com/2/users/by/username/$X_HANDLE"
auth: bearer
auth_token: "$X_BEARER_TOKEN"
auth_token: "$X_API_BEARER_TOKEN"
label: "X API"
setup_time: 15 min
cost_estimate: "$0-200/mo (Free tier: 1 app, read-only. Basic: $200/mo for search + higher limits)"
@@ -118,11 +118,11 @@ Tell the user:
Note: Free tier gives read-only access with low limits. Basic tier ($200/mo)
gives search/recent endpoint and higher limits. Pro tier gets full archive search."
Set both `X_BEARER_TOKEN` and `X_HANDLE` in the environment. Validate immediately
Set both `X_API_BEARER_TOKEN` and `X_HANDLE` in the environment. Validate immediately
(app-only bearer tokens cannot call `/users/me` — that endpoint requires
user-context OAuth — so validation uses the by-username lookup):
```bash
curl -sf -H "Authorization: Bearer $X_BEARER_TOKEN" \
curl -sf -H "Authorization: Bearer $X_API_BEARER_TOKEN" \
"https://api.x.com/2/users/by/username/$X_HANDLE" \
&& echo "PASS: X API connected" \
|| echo "FAIL: X API token invalid"
@@ -138,7 +138,7 @@ starting with 'AAA...', (3) if you just created the app, the token is valid imme
```bash
# Look up the user's X user ID from their handle
curl -sf -H "Authorization: Bearer $X_BEARER_TOKEN" \
curl -sf -H "Authorization: Bearer $X_API_BEARER_TOKEN" \
"https://api.x.com/2/users/by/username/$X_HANDLE" | grep -o '"id":"[^"]*"'
```
@@ -210,7 +210,7 @@ The agent should review collected data 2-3x daily and run enrichment.
```bash
mkdir -p ~/.gbrain/integrations/x-to-brain
echo '{"ts":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'","event":"setup_complete","source_version":"0.8.2","status":"ok","details":{"user_id":"X_USER_ID"}}' >> ~/.gbrain/integrations/x-to-brain/heartbeat.jsonl
echo '{"ts":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'","event":"setup_complete","source_version":"0.8.3","status":"ok","details":{"user_id":"X_USER_ID"}}' >> ~/.gbrain/integrations/x-to-brain/heartbeat.jsonl
```
## Production Patterns (v0.8.1)
@@ -438,6 +438,14 @@ Free tier works for personal monitoring. Basic tier needed for keyword search.
## Troubleshooting
**Upgrading from recipe v0.8.2 or earlier (token shows [missing] after upgrade):**
- Older versions of this recipe named the token `X_BEARER_TOKEN`. The canonical
name is `X_API_BEARER_TOKEN` — the name the built-in `x_handle_to_tweet`
resolver reads. Rename the variable wherever you set it (shell profile, cron
environment, `.env`) — same value, new name. A collector installed under the
old name keeps running either way; the rename is what makes the integrations
dashboard and the resolver see the token.
**API returns 403:**
- Check your app has the right access level (Read or Read+Write)
- Free tier apps can only use basic endpoints
+21
View File
@@ -0,0 +1,21 @@
#!/usr/bin/env bash
# Print the CHANGELOG.md body for one version (Keep-a-Changelog format),
# without its `## [X.Y.Z.W] - date` header line. Used by
# .github/workflows/release.yml as the GitHub release notes; tested by
# test/release-workflow.test.ts.
#
# Usage: changelog-entry.sh <version> [changelog-file]
# Exits 1 when the version has no entry (caller falls back to a link stub).
set -euo pipefail
ver="${1:?usage: changelog-entry.sh <version> [changelog-file]}"
file="${2:-CHANGELOG.md}"
# Exact-string prefix match on "## [<ver>]" — no regex, so dots in the
# version can't glob and a 3-segment version can't match a 4-segment header.
awk -v ver="$ver" '
index($0, "## [" ver "]") == 1 { found = 1; next }
found && /^## \[/ { exit }
found { print }
END { exit found ? 0 : 1 }
' "$file"
+41
View File
@@ -0,0 +1,41 @@
#!/usr/bin/env bash
# CI guard: every `bun test` invocation in workflows and runner scripts must
# pass an explicit --timeout.
#
# Why: bun ignores bunfig.toml's `timeout` key (verified on 1.3.14), so a bare
# `bun test` gets the 5000ms default for BOTH tests and beforeAll/beforeEach/
# afterAll/afterEach hooks. Hooks do NOT inherit a test's third-arg timeout —
# a file whose tests all declare `}, 30_000)` still has a 5s hook budget, and
# slow setup (Postgres connect + migrations, PGLite cold start) flakes on
# loaded CI runners with the signature `(unnamed) [5001ms] ... hook timed out`
# (the #3545 jsonb-parity failure). The CLI --timeout flag is the one measured
# mechanism that raises the hook budget uniformly; per-hook second-arg
# timeouts work too but don't scale to ~400 slow hooks.
#
# Usage: scripts/check-bun-test-timeout.sh
# Exit: 0 when clean, 1 when a bare `bun test` invocation is found.
set -euo pipefail
ROOT="$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
cd "$ROOT"
# Match executable `bun test` invocations. Exclude comment lines (#, //, *)
# and lines that already carry --timeout anywhere.
# Scope: workflows + runner scripts (the surfaces CI executes). package.json
# script bodies route through scripts/ already; editing it is out of scope here.
violations="$(grep -rnE '\bbun test\b' .github/workflows scripts 2>/dev/null \
| grep -v -- '--timeout' \
| grep -vE ':[[:space:]]*(#|//|\*)' \
| grep -v 'check-bun-test-timeout' \
|| true)"
if [ -n "$violations" ]; then
echo "FAIL: bare 'bun test' without --timeout (5s default kills slow setup hooks):" >&2
echo "$violations" >&2
echo "" >&2
echo "Add --timeout=60000 (see scripts/run-unit-shard.sh for the convention)." >&2
exit 1
fi
echo "OK: every bun test invocation passes an explicit --timeout."
+31
View File
@@ -0,0 +1,31 @@
#!/usr/bin/env bash
# Engine-live paths use static imports by default. A line-level
# `engine-dynamic-import-ok` marker is required for a justified lazy import.
#
# Historical Windows runs associated imports on these paths with abrupt Bun
# test-process exits, but system-wide commit exhaustion remained a confound.
# This guard therefore enforces a reviewed engine-path hardening invariant; it
# does not claim every dynamic import deterministically crashes Windows.
#
# Usage:
# bash scripts/check-engine-dynamic-import.sh
# bash scripts/check-engine-dynamic-import.sh FILE [FILE...]
set -uo pipefail
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" || exit 1
if [ "$#" -gt 0 ]; then
FILES=("$@")
else
ROOT="$(git -C "$SCRIPT_DIR/.." rev-parse --show-toplevel 2>/dev/null || true)"
[ -n "$ROOT" ] || ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
cd "$ROOT" || exit 1
FILES=(
src/core/pglite-engine.ts
src/core/postgres-engine.ts
src/core/migrate.ts
)
fi
exec bun "$SCRIPT_DIR/check-engine-dynamic-import.ts" "${FILES[@]}"
+80
View File
@@ -0,0 +1,80 @@
#!/usr/bin/env bun
import { readFile } from 'node:fs/promises';
import ts from 'typescript';
const MARKER = 'engine-dynamic-import-ok';
const MARKER_TOKEN_CHAR = /[\p{ID_Continue}$-]/u;
const files = process.argv.slice(2);
const violations: string[] = [];
const readErrors: string[] = [];
for (const file of files) {
let sourceText: string;
try {
sourceText = await readFile(file, 'utf8');
} catch (error) {
const detail = error instanceof Error ? error.message : String(error);
readErrors.push(`ERROR: cannot read input file ${file}: ${detail}`);
continue;
}
const sourceFile = ts.createSourceFile(
file,
sourceText,
ts.ScriptTarget.Latest,
true,
ts.ScriptKind.TS,
);
const lines = sourceText.split(/\r?\n/);
const markerLines = new Set<number>();
if (sourceFile.parseDiagnostics.length > 0) {
const diagnostics = sourceFile.parseDiagnostics
.map((diagnostic) => ts.flattenDiagnosticMessageText(diagnostic.messageText, ' '))
.join('; ');
readErrors.push(`ERROR: cannot parse input file ${file}: ${diagnostics}`);
}
for (let markerPos = sourceText.indexOf(MARKER); markerPos >= 0; markerPos = sourceText.indexOf(MARKER, markerPos + MARKER.length)) {
const before = Array.from(sourceText.slice(0, markerPos)).at(-1);
const after = Array.from(sourceText.slice(markerPos + MARKER.length))[0];
const standaloneMarker = (!before || !MARKER_TOKEN_CHAR.test(before))
&& (!after || !MARKER_TOKEN_CHAR.test(after));
const token = ts.getTokenAtPosition(sourceFile, markerPos);
const insideToken = token.getStart(sourceFile) <= markerPos && markerPos < token.end;
if (standaloneMarker && !insideToken) {
markerLines.add(sourceFile.getLineAndCharacterOfPosition(markerPos).line);
}
}
function visit(node: ts.Node): void {
if (ts.isCallExpression(node) && node.expression.kind === ts.SyntaxKind.ImportKeyword) {
const { line } = sourceFile.getLineAndCharacterOfPosition(node.expression.getStart(sourceFile));
const sourceLine = lines[line] ?? '';
if (!markerLines.has(line)) {
violations.push(` ${file}:${line + 1}:${sourceLine}`);
}
}
ts.forEachChild(node, visit);
}
visit(sourceFile);
}
for (const error of readErrors) console.error(error);
if (violations.length > 0) {
console.error('ERROR: unreviewed dynamic import on an engine-live path:');
console.error();
console.error(violations.join('\n'));
console.error();
console.error('Prefer a static top-level import. If lazy loading is load-bearing,');
console.error("append 'engine-dynamic-import-ok' to that exact line and document");
console.error('the startup or soft-failure boundary that requires it.');
process.exit(1);
}
if (readErrors.length > 0) process.exit(1);
console.log(`check-engine-dynamic-import: ok (${files.length} file(s) scanned)`);
+41
View File
@@ -0,0 +1,41 @@
#!/usr/bin/env bash
# CI guard for skills/skills.lock.json freshness (#159).
#
# Mirrors scripts/check-eval-glossary-fresh.sh: regenerate the manifest into
# a tmp file, diff against the committed version, fail the build if they
# drift. Tamper-evidence, not a signature system — the point is that any
# change under skills/ ships with an explicit manifest diff.
#
# Run: bash scripts/check-skills-manifest-fresh.sh
# Wired through `bun run verify` (scripts/run-verify-parallel.sh) so PRs that
# edit skills/ without regenerating the manifest are caught before review.
set -euo pipefail
REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
COMMITTED="$REPO_ROOT/skills/skills.lock.json"
TMP="$(mktemp)"
trap 'rm -f "$TMP"' EXIT
if [ ! -f "$COMMITTED" ]; then
echo "ERROR: $COMMITTED not found." >&2
echo "Run: bun run scripts/generate-skills-manifest.ts" >&2
exit 1
fi
cd "$REPO_ROOT"
# Render directly via bun + a one-liner that exposes the module function.
bun -e "import { renderSkillsManifest } from './src/core/skills-integrity.ts'; process.stdout.write(renderSkillsManifest('skills'));" > "$TMP"
if ! diff -q "$COMMITTED" "$TMP" >/dev/null 2>&1; then
echo "ERROR: skills/skills.lock.json is stale." >&2
echo "" >&2
echo "Diff between committed and freshly-generated:" >&2
echo "" >&2
diff -u "$COMMITTED" "$TMP" >&2 || true
echo "" >&2
echo "To regenerate: bun run scripts/generate-skills-manifest.ts" >&2
exit 1
fi
echo "✓ skills/skills.lock.json is fresh"
+121
View File
@@ -0,0 +1,121 @@
#!/usr/bin/env node
/**
* Import an envelope-v0 file (a JSON serialization of AI chat history; format
* spec: github.com/memvelope/memvelope) into a brain repo as one Markdown page
* per conversation, which `gbrain sync` ingests.
*
* Usage:
* node scripts/envelope-to-gbrain.mjs <envelope.mve.json> [outDir]
*
* Zero dependencies. Deterministic. No network. It does NOT call gbrain it
* only writes Markdown files.
*
* Output layout:
* - One page per conversation, filename = date + conversation id (shared
* titles cannot collide; the id is the natural key). A duplicate id
* overwrites its own filename and warns on stderr; stdout reports DISTINCT
* files written, not write calls.
* - Frontmatter: `type: conversation` (keeps pages eligible for
* conversation-facts extraction and chronicle behavior after sync), the
* source provider, the conversation id, and `origin: memvelope/envelope-v0`.
* - Page `date` is the first 10 chars of the conversation's ISO-8601
* `created_at`. Body keeps message-id citations beside each speaker turn.
*
* Memory: the whole envelope is held in memory (no streaming); envelopes are
* far smaller than the vendor exports they serialize.
*
* Verify:
* node scripts/envelope-to-gbrain.mjs test/fixtures/memvelope/sample.mve.json /tmp/out
* -> expect "wrote 1 markdown page(s)"
* bun test test/envelope-to-gbrain.test.ts
*
* STATUS: live-verified against gbrain v0.42.56.0 on 2026-07-03: the sample
* fixture -> 1 page; a real 662MB Claude export -> 353 conversations = 353
* distinct pages (no collisions), searchable after sync with provenance and
* message-id citations intact.
*/
import { readFileSync, writeFileSync, mkdirSync } from 'node:fs';
import { join } from 'node:path';
const [, , envelopePath, outDir = './brain/conversations'] = process.argv;
if (!envelopePath) {
console.error('usage: node envelope-to-gbrain.mjs <envelope.mve.json> [outDir]');
process.exit(1);
}
const env = JSON.parse(readFileSync(envelopePath, 'utf8'));
if (env.memvelope !== 'envelope-v0') {
console.error(`not an envelope-v0 file (memvelope field = ${JSON.stringify(env.memvelope)})`);
process.exit(1);
}
const slug = (s, fallback) =>
(String(s || '').toLowerCase().replace(/[^a-z0-9]+/g, '-').replace(/^-+|-+$/g, '') || fallback).slice(0, 60);
mkdirSync(outDir, { recursive: true });
const filesWritten = new Set();
let collisions = 0;
const conversations = env.conversations || [];
for (const [i, c] of conversations.entries()) {
const date = (c.created_at || '').slice(0, 10);
// Name the file by the conversation's own id — the natural unique key — so two
// conversations that share a date and title can never silently overwrite each
// other. The date only leads as a human/chronological sort prefix; the id
// carries uniqueness. Positional fallback keeps names unique and deterministic
// when an envelope omits an id.
// One predicate for "this conversation carries its own id", shared by the
// filename and the frontmatter below. Keeping it in a single place is what
// stops the two from disagreeing about whether an id exists.
const hasId = typeof c.id === 'string' && c.id.trim() !== '';
const convId = hasId ? c.id.trim() : `conv-${i + 1}`;
// `date` is third-party, exactly like `convId`, so it gets the same slug()
// treatment. Interpolating it raw let a `created_at` of `../…` resolve the
// join below outside outDir and write there.
const name = `${slug(date, '0000-00-00')}-${slug(convId, `conv-${i + 1}`)}.md`;
// gbrain reads YAML frontmatter + markdown body; keep provenance in frontmatter.
// Emit `type: conversation` so gbrain stores these as conversation pages rather
// than defaulting to the generic `concept`. gbrain is open-typed — it takes an
// explicit frontmatter `type` verbatim — and its conversation-aware features
// (conversation-facts extraction, the conversation_format_coverage check,
// chronicle eligibility) key off `type == 'conversation'`.
const front = [
'---',
'type: conversation',
// Every interpolated value below is quoted. An envelope is a third-party
// file, so any string carrying a newline would otherwise close its scalar
// and inject arbitrary frontmatter keys into the page gbrain ingests — or
// duplicate an existing key, which makes the parse throw and silently
// strips every provenance field from the page.
`title: ${JSON.stringify(c.title || 'Untitled conversation')}`,
// `date` is the first 10 chars of the envelope's `created_at`; 10 is plenty
// to smuggle a newline plus a short key. Absent stays an unquoted YAML null.
`date: ${date ? JSON.stringify(date) : 'null'}`,
`source: ${JSON.stringify(env.meta?.source_provider || 'unknown')}`,
// Omit the key entirely when the envelope carries no id, rather than
// emitting the literal `undefined` or a synthesized `conv-N` — the positional
// fallback names the file, but it is not a memvelope conversation id and
// must not be recorded as one.
...(hasId ? [`memvelope_conversation_id: ${JSON.stringify(convId)}`] : []),
'origin: memvelope/envelope-v0',
'---',
'',
].join('\n');
const body = (c.messages || [])
.map((m) => `**${m.role === 'user' ? 'Me' : 'Assistant'}** (${m.ts || 'no timestamp'} · ${m.id}):\n\n${m.text}`)
.join('\n\n---\n\n');
// Never lose a page silently: if two conversations still map to the same
// filename (e.g. an envelope carrying duplicate ids), warn loudly instead of
// overwriting in silence, and report the count of DISTINCT files written — not
// the number of write calls, which is what hid the old title-collision bug.
if (filesWritten.has(name)) {
collisions += 1;
console.warn(`warning: filename collision on "${name}" — conversation id ${JSON.stringify(c.id)} is not unique; overwriting the earlier page.`);
}
writeFileSync(join(outDir, name), front + `# ${c.title || 'Conversation'}\n\n` + body + '\n');
filesWritten.add(name);
}
console.log(`wrote ${filesWritten.size} markdown page(s) to ${outDir} — point gbrain's sync at this directory.`);
if (collisions) {
console.warn(`warning: ${collisions} filename collision(s) — ${collisions} page(s) overwritten. Deduplicate conversation ids in the envelope to avoid data loss.`);
}
+24
View File
@@ -0,0 +1,24 @@
#!/usr/bin/env bun
/**
* Regenerates skills/skills.lock.json the tamper-evidence manifest mapping
* every bundled file under skills/ to its sha256 (#159). Not a signature
* system: it turns silent skill edits into explicit diffs. `gbrain doctor`
* warns (never fails) on drift; scripts/check-skills-manifest-fresh.sh keeps
* the committed manifest in sync in CI.
*
* Run after any change under skills/:
* bun run scripts/generate-skills-manifest.ts
*/
import { writeFileSync } from 'node:fs';
import { dirname, join } from 'node:path';
import { fileURLToPath } from 'node:url';
import {
SKILLS_MANIFEST_FILENAME,
renderSkillsManifest,
} from '../src/core/skills-integrity.ts';
const repoRoot = join(dirname(fileURLToPath(import.meta.url)), '..');
const skillsDir = join(repoRoot, 'skills');
const outPath = join(skillsDir, SKILLS_MANIFEST_FILENAME);
writeFileSync(outPath, renderSkillsManifest(skillsDir));
console.log(`Wrote ${outPath}`);
+97
View File
@@ -0,0 +1,97 @@
/** Type surface of scripts/pr-gate.mjs for test/pr-gate-workflow.test.ts (tsc-only). */
export interface TitleCheck {
ok: boolean;
reason?: string;
}
export declare function checkTitle(title: string): TitleCheck;
export interface ChangedFile {
filename: string;
status: string;
patch?: string;
additions?: number;
deletions?: number;
}
export interface RedFlag {
id: string;
detail: string;
}
export declare function detectRedFlags(input: {
changedFiles: number;
files: ChangedFile[];
diff: string;
}): RedFlag[];
export declare const RUBRIC: string;
export declare const MAX_STRING: number;
export declare const MAX_ITEMS: number;
export declare const NET_SOURCE_LINE_LIMIT: number;
export declare const DOWNGRADE_FLAG_IDS: string[];
/** CONTRIBUTING.md #3745: human-written intent paragraph + screenshot of gbrain in use. */
export declare const CONTRIBUTING_URL: string;
export declare const INTENT_MIN_WORDS: number;
export declare const POLICY_FLAG_IDS: string[];
export declare const POLICY_SCAN_MAX: number;
export declare const POLICY_EXEMPT_ASSOCIATIONS: string[];
export declare const AI_INTENT_DOWNGRADE: string;
export declare function stripCodeFences(body: unknown): string;
export declare const MODEL_BODY_MAX: number;
export declare function modelBody(pr: { body?: string } | null | undefined): string;
export declare function hasScreenshot(body: unknown): boolean;
export declare function intentWordCount(body: unknown): number;
export declare function hasIntentParagraph(body: unknown): boolean;
export declare function detectPolicyMisses(body: unknown): RedFlag[];
export declare function sanitizeModelText(value: unknown, max?: number): string;
export declare function sanitizeList(value: unknown, maxItems?: number, maxString?: number): string[];
export declare function applyMechanicalDowngrades(
lane: string,
flags: RedFlag[],
intentAuthenticity?: string,
): { lane: string; downgrades: string[] };
/** The pr.json fields the #3745 exemption reads (all GitHub-computed). */
export interface PrIdentity {
author_association?: string;
draft?: boolean;
user?: { type?: string; login?: string };
}
export declare function policyExemption(pr: PrIdentity | null | undefined): string | null;
export interface GhComment {
id?: number;
body?: unknown;
user?: { type?: string; login?: string };
}
export declare function isOwnComment(comment: GhComment | null | undefined): boolean;
export declare function hashInputs(
pr: PrIdentity & { title?: string; body?: string; head?: { sha?: string } },
/** The assembled model payload (changed files + diff). runGate always passes it. */
payload?: string,
): string;
export declare function parseState(body: unknown): { hash: string; lane?: string } | null;
export declare function renderComment(input: {
lane?: string;
verdict?: { confidence?: number; reasons?: unknown; reviewer_checklist?: unknown };
titleCheck: TitleCheck;
flags: RedFlag[];
neutralReason?: string;
downgrades?: string[];
policyMisses?: RedFlag[];
policyExempt?: string | null;
labelsCleared?: boolean;
state?: { hash: string; lane: string };
}): string;
export declare function runGate(
dir: string,
env?: Record<string, string | undefined>,
fetchImpl?: typeof fetch,
): Promise<number>;
export declare function isPermissionFailure(err: unknown): boolean;
export declare const PERMISSION_HELP: (msg: string) => string;
+1216
View File
File diff suppressed because it is too large Load Diff
+3 -2
View File
@@ -162,8 +162,9 @@ for f in "${files[@]}"; do
if [ -n "${DATABASE_URL:-}" ]; then
psql "$DATABASE_URL" -At -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE pid != pg_backend_pid() AND datname = current_database()" >/dev/null 2>&1 || true
fi
# Hard outer timeout (180s per file). bun's --timeout is per-test; if a
# PGLite WASM call hangs in beforeAll/afterAll, --timeout never fires and
# Hard outer timeout (180s per file). bun's --timeout covers tests AND
# hooks (measured on 1.3.14), but it's timer-based: a PGLite WASM call
# that blocks the event loop synchronously never lets the timer fire and
# the file wedges indefinitely. gtimeout/timeout SIGKILLs the file so the
# suite advances. gtimeout (macOS via coreutils) preferred; timeout (Linux)
# fallback; bare bun (no outer cap) if neither is installed.
+10 -5
View File
@@ -13,7 +13,8 @@
#
# Env overrides:
# SHARDS=N same as --shards
# GBRAIN_TEST_SHARD_TIMEOUT per-shard wallclock cap, seconds (default 600)
# GBRAIN_TEST_SHARD_TIMEOUT per-shard wallclock cap, seconds (default 1500)
# GBRAIN_TEST_SHARD_KILL_AFTER grace after TERM before KILL (default 30)
# GBRAIN_TEST_MAX_CONCURRENCY passed through to bun test (default 4)
#
# Output files (workspace-local; falls back to /tmp if .context/ unwritable):
@@ -79,6 +80,10 @@ INTRA_CONC="${MAX_CONCURRENCY_OVERRIDE:-${GBRAIN_TEST_MAX_CONCURRENCY:-4}}"
# 4-shard wallclock; real hangs still hit it. Override via
# GBRAIN_TEST_SHARD_TIMEOUT=N.
SHARD_TIMEOUT="${GBRAIN_TEST_SHARD_TIMEOUT:-1500}"
SHARD_KILL_AFTER="${GBRAIN_TEST_SHARD_KILL_AFTER:-30}"
if ! printf '%s' "$SHARD_KILL_AFTER" | grep -qE '^[0-9]+$' || [ "$SHARD_KILL_AFTER" -lt 1 ]; then
echo "ERROR: invalid shard kill-after: $SHARD_KILL_AFTER" >&2; exit 2
fi
# ──────────────────────────────────────────────────────────────────────────
# Output directories. Prefer workspace-local .context/, fall back to /tmp.
@@ -109,7 +114,7 @@ elif command -v timeout >/dev/null 2>&1; then TIMEOUT_BIN="timeout"
fi
START_TS=$(date +%s)
echo "[unit-parallel] N=$N shards | --max-concurrency=$INTRA_CONC | timeout=${SHARD_TIMEOUT}s | logs=$LOG_DIR" >&2
echo "[unit-parallel] N=$N shards | --max-concurrency=$INTRA_CONC | timeout=${SHARD_TIMEOUT}s | kill-after=${SHARD_KILL_AFTER}s | logs=$LOG_DIR" >&2
if [ "$DRY_RUN" = "1" ]; then
echo "[unit-parallel] dry-run: would spawn $N shards with the above settings."
@@ -129,7 +134,7 @@ for i in $(seq 1 "$N"); do
(
SHARD_LOG="$LOG_DIR/shard-$i.log"
if [ -n "$TIMEOUT_BIN" ]; then
"$TIMEOUT_BIN" "${SHARD_TIMEOUT}s" \
"$TIMEOUT_BIN" --signal=TERM --kill-after="${SHARD_KILL_AFTER}s" "${SHARD_TIMEOUT}s" \
env SHARD="$i/$N" \
bash scripts/run-unit-shard.sh --max-concurrency="$INTRA_CONC" \
> "$SHARD_LOG" 2>&1
@@ -140,7 +145,7 @@ for i in $(seq 1 "$N"); do
> "$SHARD_LOG" 2>&1 &
pid=$!
( sleep "$SHARD_TIMEOUT" && kill -TERM "$pid" 2>/dev/null && \
sleep 5 && kill -KILL "$pid" 2>/dev/null ) &
sleep "$SHARD_KILL_AFTER" && kill -KILL "$pid" 2>/dev/null ) &
cap_pid=$!
wait "$pid" 2>/dev/null
# Capture the shard's exit code from ITS `wait`, before any watchdog
@@ -158,7 +163,7 @@ for i in $(seq 1 "$N"); do
wait "$cap_pid" 2>/dev/null
fi
echo "$rc" > "$LOG_DIR/shard-$i.exit"
[ "$rc" = "124" ] && echo "WEDGED" > "$LOG_DIR/shard-$i.wedged"
{ [ "$rc" = "124" ] || [ "$rc" = "137" ]; } && echo "WEDGED" > "$LOG_DIR/shard-$i.wedged"
) &
SHARD_PIDS+=($!)
done
+2
View File
@@ -50,6 +50,7 @@ CHECKS=(
"check:cli-exec"
"check:system-of-record"
"check:eval-glossary"
"check:skills-manifest"
"check:no-pii-agent-voice"
"check:synthetic-corpus-privacy"
"check:skill-brain-first"
@@ -64,6 +65,7 @@ CHECKS=(
"check:source-scope-onboard"
"check:no-double-retry"
"check:batch-audit-site"
"check:engine-dynamic-import"
"check:worker-lock-renewal-shape"
"typecheck"
)
+1 -1
View File
@@ -248,7 +248,7 @@ before submission.
After the brain page is written, render to PDF using `skills/brain-pdf`:
```bash
gbrain put_page # already done by the CLI; nothing to add here
gbrain put # already done by the CLI; nothing to add here
# Then invoke brain-pdf:
# (see skills/brain-pdf/SKILL.md for the make-pdf invocation)
```
+6 -6
View File
@@ -73,13 +73,13 @@ stock worker auto-loads on startup) registers handlers before `start()`.
Users who set `minion_mode: off` in `~/.gbrain/preferences.json` keep
using `agentTurn`. Respect that. No auto-rewrite.
## Forward note (v0.12.0)
## Forward note
GBrain v0.12.0 ships `gbrain cron`: a scheduler loop inside
`gbrain jobs work` that owns cron expressions natively — no more
handing off to host schedulers. Until v0.12.0 lands, the host
scheduler keeps firing on schedule; v0.11.1 only replaces the execution
layer (what the cron trigger *does*), not the scheduling layer.
A native scheduler loop inside `gbrain jobs work` (owning cron
expressions directly, with no host-scheduler hand-off) has been on the
roadmap since v0.11.1 but has not shipped. The host scheduler keeps
firing on schedule; this convention only replaces the execution layer
(what the cron trigger *does*), not the scheduling layer.
## Related
+2 -1
View File
@@ -125,7 +125,8 @@ v0.41.22 ships **gbrain-base-v2** as the declared successor to
gbrain-base@1.x — collapses 94 noisy types to 15 canonical via
declarative mapping_rules. Run via `gbrain onboard --check --explain`
(preview) → `gbrain jobs submit unify-types --allow-protected --params
'{"target_pack":"gbrain-base-v2"}'` (apply). See
'{"target_pack":"gbrain-base-v2","apply":true}'` (apply — `apply`
defaults to false, so a bare submit is a dry run). See
`skills/schema-unify/SKILL.md` for the full playbook.
Authoring a successor pack: declare
+2 -2
View File
@@ -54,8 +54,8 @@ Ask the user what they want to track. Either:
- Define a custom recipe with: source queries, classification rules, extraction schema,
tracker page path, tracker format
Recipes are YAML files at `~/.gbrain/recipes/{name}.yaml`. Use `gbrain research init`
to scaffold a new one.
Recipes are YAML files at `~/.gbrain/recipes/{name}.yaml`. Scaffold a new one by
copying a built-in recipe file and editing its fields.
### Phase 2: Search Sources
+1 -1
View File
@@ -201,7 +201,7 @@ Use the brain page template. MUST include:
### 4b. Entity pages (people, companies)
For each entity mentioned:
- Check if a brain page exists (`gbrain search "<name>"` or `gbrain get_page people/<slug>`).
- Check if a brain page exists (`gbrain search "<name>"` or `gbrain get people/<slug>`).
- If exists: update State, append Timeline entry citing this research.
- If not: create with enrichment.
+1 -1
View File
@@ -112,7 +112,7 @@ gbrain query "<topic keywords>"
# -d '{"model": "sonar-pro", "messages": [{"role":"user","content":"..."}]}'
# 4. Write the structured research page via put_page:
gbrain put_page research/<slug> # via the put_page operation
gbrain put research/<slug> # via the put_page operation
# 5. Cross-link entities mentioned (people, companies) per Iron Law.
```
+9 -5
View File
@@ -11,7 +11,7 @@ tools:
- gbrain schema active
- gbrain schema use
- gbrain schema stats
- gbrain pages restore
- gbrain restore
- mcp:run_onboard
triggers:
- "unify my types"
@@ -90,9 +90,13 @@ The handler is PROTECTED (manual_only per D17) — autopilot will never auto-fir
```bash
gbrain jobs submit unify-types \
--allow-protected \
--params '{"target_pack":"gbrain-base-v2"}'
--params '{"target_pack":"gbrain-base-v2","apply":true}'
```
`apply` defaults to **false** (dry-run) per the handler contract, so
`"apply":true` is required here or the job reports success having retyped
nothing and left the active pack unflipped. Omit it to preview.
Watch progress per phase:
```bash
@@ -143,7 +147,7 @@ WHERE source_id = 'default' AND frontmatter->>'legacy_type' IS NOT NULL;
Page-to-alias and page-to-link source pages soft-delete with 72h TTL. Restore within that window:
```bash
gbrain pages restore <slug>
gbrain restore <slug>
```
Revert the active pack flip:
@@ -197,7 +201,7 @@ Outputs:
- Active pack flipped to `gbrain-base-v2` atomically at end of successful run.
Side effects:
- Source pages soft-deleted with 72h restore TTL (`gbrain pages restore <slug>`).
- Source pages soft-deleted with 72h restore TTL (`gbrain restore <slug>`).
- One-time cache invalidation on KNOBS_HASH_VERSION bump (5→6); self-healing in `cache.ttl_seconds`.
- Query-time `--type X` alias-expands via `expandTypeFilter` (D14 back-compat).
@@ -212,7 +216,7 @@ DON'T:
- Submit `unify-types` directly via the MCP `submit_job` op without `--allow-protected`. PROTECTED handlers require trusted local callers; remote MCP rejection is the intentional trust boundary.
- Edit `mapping_rules` in `gbrain-base-v2.yaml` to skip clusters you don't trust. Fork the pack instead (`gbrain schema fork`) so the source-of-truth migration stays consistent across brains.
- Run `unify-types` from inside an autopilot tick. The check is `manual_only` per D17 — autopilot deliberately never auto-fires it because pack upgrades are one-time consenting taxonomy decisions.
- Hard-delete soft-deleted source pages before the 72h restore window. Use `gbrain pages restore <slug>` first if rollback is needed.
- Hard-delete soft-deleted source pages before the 72h restore window. Use `gbrain restore <slug>` first if rollback is needed.
- Assume `frontmatter.legacy_type` survives every roundtrip. The marker is canonical for the immediate post-migration window; downstream re-imports may overwrite it.
## Output Format
+2 -2
View File
@@ -139,9 +139,9 @@ edits writes a new receipt).
| Slot | Default | Provider |
|------|---------|----------|
| A | `openai:gpt-4o` | OpenAI |
| A | `openai:gpt-5.2` | OpenAI |
| B | `anthropic:claude-opus-4-7` | Anthropic |
| C | `google:gemini-1.5-pro` | Google |
| C | `deepseek:deepseek-v4-pro` | DeepSeek |
**These MUST be frontier models from DIFFERENT providers.** Using a single
provider's family or budget models defeats the purpose — different families
+130
View File
@@ -0,0 +1,130 @@
{
"RESOLVER.md": "ad9f85d0f953ad0ec80c70091cb31c7227a89e4a08c924454be1d61d4df5b65c",
"_AGENT_README.md": "3dd82df125ceb87bdbc1ad16be4306fb3ca6e3a491c70122f6c7724a862f0b21",
"_brain-filing-rules.json": "cf850df6a7425464c6d63b3ace71991cc93497fa0cc8cd21acd31883e17939c6",
"_brain-filing-rules.md": "2d2d75b7c76081c56f41b2c0a5a978c355ce957300f9b0a5575dc4079ef1f877",
"_friction-protocol.md": "1b6e7cfa58725a6a5dc2dc787242141bc33f5fde524540d85b14ec22266140f7",
"_output-rules.md": "239022bd9003b4d45870fea2ddb177a132f0684d898ed5075dc5dcb90cc56ab7",
"academic-verify/SKILL.md": "1c19e27e75249d869da428ce8d060075feef8fbbfe146af58b305d11a260ebbc",
"academic-verify/routing-eval.jsonl": "90d894a9829d9936e6ac7a6507e4de67ad26e46a1fe13b7a34e7dec1c0d887dd",
"archive-crawler/SKILL.md": "10c529797e1cba75da5bdb13bdd3609ae5f962f7dcc429c3b85d51c0032d9a26",
"archive-crawler/routing-eval.jsonl": "a90c607d69737adf58771d1b986c5a7d9d11f5303df7de4c7def74aa2c86a2f5",
"article-enrichment/SKILL.md": "fcdbce0f250aa2c38b86299dae510b8dc7f1e1cd854961cb8520c61642309549",
"article-enrichment/routing-eval.jsonl": "408fa8cc80caf1a2208afeb049e5ada69beb378f22d6b7114828201a4657e6e3",
"ask-user/SKILL.md": "a40f484721e548a3a14d4b33a4636111d92f99619ecc4e4ea54c3da3a15f8331",
"book-mirror/SKILL.md": "e8b8cc7a6eba4ecd302a840446b48c0e1aecc245e8f240e233c738daa8dff78a",
"book-mirror/routing-eval.jsonl": "79fd23642cfa37b1255a799907e71bb2904585cf79dd6a596e3a2f019787e54c",
"brain-ops/SKILL.md": "40553ad3bf0f27fc8363b69bec3ef89ae0ea0d9290de8d1efbaa0c532c2a9590",
"brain-pdf/SKILL.md": "13c3e3162763a4503685db0a10663475d3687c4874b5f04d539af83a990f643e",
"brain-pdf/routing-eval.jsonl": "119e4fa113ea45783cee4499e63a729fdeecb4d9a45d47497754b4f5b21d0734",
"brain-taxonomist/SKILL.md": "dea4557b540868ec2c56bf43ee7f63c5d03a22d4047cd0dfbeaf19adef334f60",
"brain-taxonomist/routing-eval.jsonl": "8b485b3d735aace60be703854f0f2e9d97c52d52564efdaf7334a0c39e8d20ae",
"briefing/SKILL.md": "a661804c3eb2ce5bd4eb6f3e7106283046913945d9dba0b967bdc8483a58dd66",
"capture/SKILL.md": "98568ac96331f57397ea072749641d9748b1ce31e8b09d512b8db25c8fcda65f",
"citation-fixer/SKILL.md": "abdadbf0740a529b9c4f86f05bba416417624503fdcbc6054402d5546afd08b4",
"citation-fixer/routing-eval.jsonl": "52b23b71e66fdc18aee67d0576099b0c83997d648cf4ecf8fe7753b91b6c9c53",
"cold-start/SKILL.md": "a2c42dd7c4eceb7d3ce6449a414b417d55195aa445e723e6c798d90906cbf4e6",
"concept-synthesis/SKILL.md": "2bc060ae6d706c4e8e7d784cbe3e577b85e211a21c68cba094a1754d3f34436b",
"concept-synthesis/routing-eval.jsonl": "51d1da894158503ce18b892a34edd203f40732e79ac1c0e85141fd37e0b9922f",
"conventions/brain-first.md": "29d020470d0168f8f0b29dde0350a485a9b0472f7ac9962e34948f4897455590",
"conventions/brain-routing.md": "a8035f7dbadff0ea68b8babb8314b3d044cafbed8242dce5b931fa08b028fc45",
"conventions/calibration.md": "eda7ca76f80c8a17ae546110484389f805c5b21fc0a57f951bbe8b6abba26e03",
"conventions/cron-via-minions.md": "badb1cd6cd825d6f1ac0b6b28cc47e5d80facc783a3e59a14146ae901ee0f933",
"conventions/cross-modal.yaml": "c012c3d72614a87b1ee698173dce2a0fb0d057a54df7aab87993c4b07fff6280",
"conventions/model-routing.md": "fb7ae8746a578500d6789b68ff40049037aa4d337b65b42f7c1745ae7080c2db",
"conventions/quality.md": "8aa681001114689d34268ccadaf0e2ff07b8f68aa5987c093a8c4a7a744f12a6",
"conventions/salience-and-recency.md": "62b0b303bf48bef10adcf08d3b51f23d3b1f1187b3850476a4b5532c2ee88926",
"conventions/schema-evolution.md": "b5cdd5a17e43b4f2d0546ebf8c421cc0378cbcf74cb2fe97199e115ccb05f9c2",
"conventions/search-modes.md": "2a920225d1c95ea978fb1c77c5170f6a598377ab86fc0024b70962d5a84d54d0",
"conventions/subagent-routing.md": "59afd362ff0cbaf3a63586e97c53f68feb837a258a2bd43f9177af4e57d2b20e",
"conventions/test-before-bulk.md": "5073e5b93d570445f72f3c10ed3e6ec10c1fee7574ad73c2e2695993c850eca5",
"cron-scheduler/SKILL.md": "e3f9745c4f8e2dacba5b055f408b1ffe541b450aee6bbfe4773f7b424d90e04c",
"cross-modal-review/SKILL.md": "685233b1afd477e96697562c502233eea22eda5db8df116b81dd2fa78f01f80c",
"daily-task-manager/SKILL.md": "e616f74a6befffc7bb64c0e29b2d2caa37c51bc470bca96df70101c772429d83",
"daily-task-prep/SKILL.md": "9fe89f85fae139adac25c3bdc6f23bbf64239f3738a9e679c447e686175516f0",
"data-research/SKILL.md": "990ccec01a23d3e46c7b5abbd2650ac5507480b50f7ec4de8b466863b3b61cb6",
"eiirp/SKILL.md": "9d42d6a5f61bba30cb47400db58927c501cd89e5cc3241b5218d7962d1a68804",
"eiirp/routing-eval.jsonl": "416459ff68da2e5f5eb216a0c24368c8eebdf4edbbc540a28fe730ceeb4c9700",
"enrich/SKILL.md": "9988168348f6c3391d3aeec6621f9c99c8d75d1e1bfdd6c44d48e4c1067ab775",
"frontmatter-guard/SKILL.md": "5142ab53f5428ebc084ded78f1fb7eb4bfa386d276ee034bf4d57273256570c6",
"frontmatter-guard/routing-eval.jsonl": "243c28b04b557bac5360318c0f47bb6f1dd56d2f720aad3763b93fc0b2ea081e",
"functional-area-resolver/SKILL.md": "52df04bc4f8e678f931c3b2078b2126524e6d2d72676ad46b6b710d13271b46c",
"functional-area-resolver/routing-eval.jsonl": "f80674d915acdfe229046737a5b171da834be15ac6524b5a3fd18048e9b37028",
"gbrain-advisor/SKILL.md": "c15c7a88bee2c96733d718a168dd9afcb123b7b6b2c0014c5260d37c72e8736a",
"gbrain-upgrade/SKILL.md": "7cd05f43027fa20d56ed651f696288f9e4814b7bdf7b396f132fde7c96dac620",
"idea-ingest/SKILL.md": "e118bc32d5044a4a6fba4fed1911b8be50210ade14957e6cfe8e7ed26ff95298",
"idea-lineage/SKILL.md": "bbf37781d93b71ddc7909ecc5ab635872c874fb8591995dbf88b45ffeac6b1de",
"idea-lineage/routing-eval.jsonl": "ee2e00704b9accb7dd58bb8f126a3bc04a2c40be499180fa505dbf6d5061cd41",
"ingest/SKILL.md": "dc40ecc0072806fb8c7bb6ab9cf1f103842e05653eb55d67632d7e3ffc4dd7d2",
"install/SKILL.md": "881bd0a422f34c6df4642aae66c51e2a4cc18ad5ca6d0b52d44b4de93512a3c4",
"maintain/SKILL.md": "e80cf5bb170c979b773a67a18e0d880c086e0b983db9a576a207074b0112c4f8",
"manifest.json": "52b970cfc3ed340ee4f25323b6fadeb5d125741b02d1d2dc2f85c39f0c701258",
"media-ingest/SKILL.md": "33db12830ed31a4ff4a6a58c4f126bf2596ee28d54cee0027680c83fee648a20",
"meeting-ingestion/SKILL.md": "7767334c63ff3bd8e60cd4d7cd1d1b44f0d6b0a7e0ac411529a1d59cc0a7781b",
"migrate/SKILL.md": "442c117cfe50026a142ff4f934e489c56ee226d454b639d20bba3b111a86d8fd",
"migrations/.gitkeep": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"migrations/v0.10.3.md": "1e12db6fdb88d21df479659e236bad111a041a205d47aa0894afde2f04903124",
"migrations/v0.11.0.md": "ac45c6144eeee5033f2b24308a5063ceced2be7831b04e48154aa0b6f8ff55d8",
"migrations/v0.12.0.md": "f30b452caed5ce2679cbd60dc2d0b9e60449bab20c3a3e0c895e6cd167644f44",
"migrations/v0.12.1.md": "a461d8fcd6b19dabe3a0ce849a54e3c577667ae2af41870c6f456c6195e76064",
"migrations/v0.13.0.md": "3bf61eb1da933e6c3c35d67d0ea6d9dada5d5f878bbd14020616d4e1fe48feb2",
"migrations/v0.14.0.md": "1025395f033b91480451aad6a98dc10f2c18bfacd1b1a4e2a9fb4dc99f8fb571",
"migrations/v0.15.2.md": "468a58e8a9f0f5c5867e0ed8eb9dd1bd60dbc8d66111647997ebd8d9d2119915",
"migrations/v0.17.0.md": "761dda3b6090bfe20c9b4354aff5248d5a9b6a79a24702ecc2b29873d4f5b413",
"migrations/v0.18.0.md": "94efc1d571070600a78aca13c2735866a987e6a8f188210b7062f81b54839986",
"migrations/v0.19.0.md": "c47584e8306680f21ddc7f21a4a89f2184da522acb177f0dd73a09f1cd77ae57",
"migrations/v0.21.0.md": "83569ce6fb2843f81964833affe5eae72c94a2983eba265c836aa05be7876957",
"migrations/v0.22.14.md": "8898c109716fbc48c058a44c606f50fb7c02be599541d367380aa2e63bffea9e",
"migrations/v0.22.4.md": "5f9b2ed86cb64d92416448e744ca6c4984b89728895cd4e363f9c76a860074d9",
"migrations/v0.23.0.md": "f0363274e5071fa35d0c60e3981c4caee0494c43adb254d65acf6b743d4762d5",
"migrations/v0.25.1.md": "981ef49abadf19a891cb68b655b9f35ee67d9f4ac232ceb29d83dfc3b1c9982e",
"migrations/v0.27.1.md": "719dd4ebcc7ac81413ee79af20f80de5141577e3860e5049a49fc8f6d09ddf89",
"migrations/v0.28.0.md": "21929a392af2e1f454272e80aade6c58c5f6ccd7253d24344d08c6b53bcdf4f9",
"migrations/v0.29.1.md": "7b85373a62a8500ee8c4238ffc9878318b44ee62543adb8b2462049c6c0cd1f8",
"migrations/v0.32.2.md": "101623fa48f946242d7edc8f126967822563c3fb30a6ad50d029894f5fe3499d",
"migrations/v0.32.6.md": "e9438a910afe8a1f7861c5222fd7db04a0f8d8b8283778476b07b7c3d539d3f0",
"migrations/v0.33.0.md": "11710cb11d6eb7dc3ea54b764e3c4a25f8679cf76590acd330f97bfa1c684945",
"migrations/v0.33.3.0.md": "188a03ca86a97a9aa697cbbc83cc8ca37843fab24db2bd82f1383c400173d5bd",
"migrations/v0.34.0.0.md": "d421c5ecff0765ac1de3592d3175734db7df52e8658ec101567779c7c56c2db2",
"migrations/v0.35.0.0.md": "0fc21dc0b098f87fff1ac79a669b00a3d69ab1510ebfc5eac4a66f5c6d783809",
"migrations/v0.35.7.0.md": "c6d4454bd39e2aa243b3b3d9bc72fe5a4fd25d097be7be2bb14b25604b5c2cc5",
"migrations/v0.36.2.0.md": "1b59328240ae19c5e7e8d3eafda245809cca1fea27146607334dbee53cbeb270",
"migrations/v0.36.5.0.md": "a01a722202dfc3c799693596750c8bee611fe4dafe3cb662f6b4cd0b635cb429",
"migrations/v0.40.3.0.md": "5f500f8c543c2b6f41778b0bd3beedada68f7284f7933ad8b769322b433a8fe9",
"migrations/v0.40.5.md": "b9837d52a030517698dfb31c439f562cde60a1015ae488dab09be2c16ff182e5",
"migrations/v0.41.11.0.md": "5c6873ab969d14def4a450d792f070f1259d08b3aca43bc7825d0a9114b2b36b",
"migrations/v0.5.0.md": "5e0dabc451595295c4d971e19bcb33c258a127223d25859d8321cb7e1ce60711",
"migrations/v0.7.0.md": "97c2740445a10b1c5c7123c17dbd625fa27a94095b85d27c2b278da756c4c59a",
"migrations/v0.8.0.md": "1919ff8b8f3680612ff888e7cfcc0d86ece5d5304ae19af4497bdf40b050561a",
"migrations/v0.8.1.md": "fad7341cfb5e02545fb8a23221d12ab395fc3d8db15d1d8ee8a18844aea6563a",
"migrations/v0.9.0.md": "773fab0a8d7f330576265a3f510c1f318f47789b6136c46d43e08121acbc20eb",
"migrations/v0.9.1.md": "75761bad6c0ad37b69ec8197c6a678bb6a1484f9a76e4b70f2d1e86dc80102b3",
"minion-orchestrator/SKILL.md": "669e23f485561cf6fef16d445dfb8a6d05f76127d0ebf9feb17534af8843df5f",
"perplexity-research/SKILL.md": "c25f5c471cbe3c6e0f975d8397e8382b00a85f8aa75302231d53c52855369e97",
"perplexity-research/routing-eval.jsonl": "f1a40d87e710d5d2acd602a372d83f46c95da022b6e635228fffeaacb3bb2b27",
"publish/SKILL.md": "e06b609db780a3cc93a1755a87b30ff08ffdc0fdbc834c1422b2ad2489b57497",
"query/SKILL.md": "e155a08049984c524b838988ba456d16ccedf162442160f6ba66bfd97cd5208e",
"query/routing-eval.jsonl": "74f5a91e52fabc54e0e9403fa17db87ee26bb7ebb8ae8005148c51142abc62fe",
"repo-architecture/SKILL.md": "4ec2b8f45d168aaa55f17ecd1ed404ab04217a75c2317f0710c71705846f5394",
"reports/SKILL.md": "5dc190a0c3a2ee518254e8b596418dbe19ff389ea5ec8c8d30fcb0dfef4d0ed5",
"schema-author/SKILL.md": "4da9a472c966f8e4fb43d97a3a3608c67ec3d43da6b8c626ee0c30a4e70da26f",
"schema-unify/SKILL.md": "e1d50a54a6ff29434d38a841a572e815d236a8167d459609ae697e548770f500",
"setup/SKILL.md": "4a47a5f6ee99ac8a2649a304abb261fd34810cf6185fd7d6a53e1b6c4cbc0764",
"signal-detector/SKILL.md": "64e4547f5a8624c53d875001b423d240ec73ee9fd026a96c7b799d287c5fb6e4",
"skill-creator/SKILL.md": "4a11f8935d4214b21b4664a5c0c03149733731020ec0d5d09dc9fd8c40bd92f6",
"skill-optimizer/SKILL.md": "ba3028c7351dec3e644a7114e08e59cae7c60dc367dc4fd10560265a162baa90",
"skill-optimizer/routing-eval.jsonl": "48f7fc04414b194ee8674577c3e58e03ebd7f74d836766bd16cb5abfd4effb76",
"skill-optimizer/skillopt-benchmark.jsonl": "5552457d6eaa32486b79796d12fcbe2c078b0c53d7fbdaf582f0fd1a17e9a381",
"skillify/SKILL.md": "a154e43409458136e1e1cea084aa973e4cf46cd8450c6be8e9891a4a3ddf6146",
"skillpack-check/SKILL.md": "3f347ec8b498530a662be212d05f4cd06b205bce5795c2231b6a4cecef149ea0",
"skillpack-harvest/SKILL.md": "3c4c591b33f03a5ccf11ca0ddde56b54fba541efef0d590b6d435687733182d7",
"skillpack-harvest/routing-eval.jsonl": "cb4783288e95af3132b32ecb40a54cffc095f57b36240f96a25c2d2adf5e68c6",
"smoke-test/SKILL.md": "f2f2172d41e63e288095451132a0c56848ccc34101d265b330b6cf00e5769f5d",
"soul-audit/SKILL.md": "7f162dddcc511e97a24db3a46136295fcbda76023019ad8994546744d7b8eb0a",
"strategic-reading/SKILL.md": "5be656c39c830153ec7c2f328dc8bdeac05c1412b01b3415de6b5b008926e7a2",
"strategic-reading/routing-eval.jsonl": "eb0fc239c93aac53cf7d856190967bb65eaf970b4fe8bd0aad1c87495fbb8ccd",
"testing/SKILL.md": "f1846ba7c35076d910744a6b867c9ee850c1895f75a104a319c9b7d18b1d5e90",
"voice-note-ingest/SKILL.md": "145c02e636430abba5648e026aef77606d6ba2594285fbfee4f4dd4253cf833a",
"voice-note-ingest/routing-eval.jsonl": "374aaec16fbde336d1e376edce89e51adc4ecaa93c13fb4a69b967ba299b8742",
"webhook-transforms/SKILL.md": "b774293297af4d513c7efa92a79cf65b8438a714adafc16802b492613e679d17"
}
+6 -4
View File
@@ -43,8 +43,9 @@ The Analysis section can interpret; the transcript section is sacred.
The user sends an audio or voice message via any channel (Telegram, voice
memo upload, openclaw audio attachment). The host agent typically provides
the transcript text. If not, transcribe via `gbrain transcription` (Groq
Whisper by default; OpenAI fallback for audio > 25MB segmented via ffmpeg).
the transcript text. If not, transcribe it with your host's transcription
tool (Groq Whisper is fast and cheap; OpenAI Whisper works too — segment
audio > 25MB via ffmpeg first).
## The pipeline
@@ -52,8 +53,9 @@ Whisper by default; OpenAI fallback for audio > 25MB segmented via ffmpeg).
1. STORE → Upload original audio to gbrain storage backend
(S3 / Supabase Storage / local — pluggable per
src/core/storage.ts).
2. TRANSCRIBE → Use the agent-provided transcript verbatim, OR call
gbrain transcription if no transcript was supplied.
2. TRANSCRIBE → Use the agent-provided transcript verbatim, OR
transcribe the audio yourself (see "When to invoke")
if no transcript was supplied.
3. ROUTE → Apply the decision tree (below) to find the right
destination directory.
4. WRITE → Create / update the destination brain page; preserve the
+132 -12
View File
@@ -55,12 +55,17 @@ export function bigintToStringReplacer(_key: string, value: unknown): unknown {
}
// CLI-only commands that bypass the operation layer
export const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'maintain', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'reconcile-links', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'calibration', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'retrieval-upgrade', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'advisor', 'watch', 'reindex-search-vector', 'backfill']);
export const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'maintain', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'reconcile-links', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'calibration', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'retrieval-upgrade', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'advisor', 'watch', 'reindex-search-vector', 'pages', 'bench', 'backfill']);
// CLI-only commands whose handlers print their own --help text. These are
// excluded from the generic short-circuit so detailed per-command and
// per-subcommand usage stays reachable.
const CLI_ONLY_SELF_HELP = new Set([
'upgrade', 'post-upgrade', 'check-update',
// #3502 sweep: pages + bench print their own usage (pages.ts printHelp,
// bench-publish.ts printHelp). Both were documented but undispatchable —
// `pages` had a live handleCliOnly case but was missing from CLI_ONLY
// (the #2035 calibration bug class); `bench` was never wired at all.
'pages', 'bench',
'embed', 'config',
'skillpack', 'skillpack-check',
'integrations', 'friction',
@@ -107,6 +112,11 @@ const CLI_ONLY_SELF_HELP = new Set([
// `gbrain connect --help` prints its own usage (flags + examples) from
// runConnect; route around the generic one-line short-circuit.
'connect',
// `gbrain init --help` prints its own usage from runInit; route around the
// generic one-line short-circuit (matches `connect`). Without this, `init`
// is in CLI_ONLY but not CLI_ONLY_SELF_HELP, so the dispatcher's generic
// short-circuit fires and the printInitHelp() guard in init.ts is dead code.
'init',
// #3390 — `gbrain migrate embeddings --help` / `gbrain retrieval-upgrade
// --help` print the migration flags from runMigrateEmbeddings. `migrate`
// (engine transfer) keeps its own dispatch too.
@@ -394,6 +404,18 @@ async function main() {
if (op.localOnly) {
refuseThinClient(command, cfgPre!.remote_mcp!.mcp_url);
}
// A thin client has no local mounts — an explicit --brain cannot be
// honored and must not be silently dropped (same loud-beats-silent rule
// as applyThinClientSourceScope's --source refusal). Ambient tiers
// (GBRAIN_BRAIN_ID / .gbrain-mount) are ignored here, matching the
// source axis's ambient-with-nowhere-to-send behavior.
if (cliOpts.brain) {
console.error(
'--brain is not supported on a thin-client install: the remote server is a single brain. ' +
'Remove the flag, or run from a machine with local mounts (gbrain mounts list).',
);
process.exit(1);
}
// #2098: the local path resolves --source / GBRAIN_SOURCE / .gbrain-source
// inside makeContext (ctx.sourceId), which this route never reaches — so
// scope must be mapped onto the op's source_id wire param before the call.
@@ -1004,6 +1026,10 @@ export async function makeContext(engine: BrainEngine, params: Record<string, un
// table). Matches dispatch.ts's auto-fill so the contract holds across
// every transport.
sourceId: sourceId ?? 'default',
// Brain axis: the id connectEngine resolved for this process. Module
// state, NEVER params — caller-supplied params.brain must not select a
// brain (that would be an untrusted-caller cross-brain hole over MCP).
brainId: activeBrainId,
...(localFederated ? { localFederatedSourceIds: localFederated } : {}),
};
}
@@ -1265,6 +1291,20 @@ async function handleCliOnly(command: string, args: string[]) {
await runInit(args);
return;
}
if (command === 'bench') {
// #3502 sweep: `gbrain bench publish` was documented (docs/eval-bench.md,
// KEY_FILES.md, and eval-gate's own --help text) but never dispatched —
// the promised-but-unwired class retrieval-upgrade (#3390) fixed before.
// Pure file-in/file-out (NDJSON → baseline); no DB, no engine.
if (args[0] === 'publish') {
const { runBenchPublish } = await import('./commands/bench-publish.ts');
await runBenchPublish(args.slice(1));
return;
}
console.error('Usage: gbrain bench publish --from <captured.ndjson> --to <X.baseline.ndjson> [flags]');
console.error('Run `gbrain bench publish --help` for the full flag list.');
process.exit(args[0] === '--help' || args[0] === '-h' ? 0 : 2);
}
// v0.37 fix wave (deferred TODO, shipped): one-command wipe-and-reinit.
// Spawns its own engine internally so no pre-bound engine needed.
if (command === 'reinit-pglite') {
@@ -1707,14 +1747,12 @@ async function handleCliOnly(command: string, args: string[]) {
// Per-command default: search 30s, sources list 10s. User --timeout=Ns wins.
// Other commands (import, embed, doctor, etc.) keep their existing
// unbounded connect — destructive / long-running commands shouldn't get
// a default kill switch.
const readOnlyDefaultTimeoutMs =
command === 'search' ? 30_000 :
command === 'sources' && (args[0] === 'list' || args[0] === undefined) ? 10_000 :
null;
// a default kill switch. The gate below is per-command (#3013): only the
// commands dispatchReadOnlyCommand handles may enter this path — a
// user-supplied --timeout on a write command must never reroute it here.
const cliOptsResolved = getCliOptions();
const userTimeoutMs = cliOptsResolved.timeoutMs;
const readOnlyTimeoutMs = userTimeoutMs ?? readOnlyDefaultTimeoutMs;
const readOnlyTimeoutMs = resolveReadOnlyDispatchTimeoutMs(command, args, userTimeoutMs);
if (readOnlyTimeoutMs !== null) {
const { withTimeout, OperationTimeoutError } = await import('./core/timeout.ts');
@@ -1829,7 +1867,14 @@ async function handleCliOnly(command: string, args: string[]) {
}
case 'embed': {
const { runEmbed } = await import('./commands/embed.ts');
await runEmbed(engine, args);
// #3037: mirror the `import` case above — the CLI was discarding the
// result, so a run where every chunk failed to embed still exited 0
// and cron/CI/health gates read total silence as success. Surface
// non-zero on failures > 0. (undefined = backgrounded via --background.)
const embedResult = await runEmbed(engine, args);
if (embedResult && embedResult.failures > 0) {
setCliExitVerdict(1);
}
break;
}
case 'serve': {
@@ -2253,16 +2298,24 @@ async function handleCliOnly(command: string, args: string[]) {
//
// v0.30.1: still works; canonical entrypoint is now `gbrain backfill
// effective_date`. This command stays as a thin alias for back-compat.
//
// #1963: pass the already-connected engine. The command used to build
// + connect its OWN engine here, which self-deadlocked on the PGLite
// data-dir lock (this process already holds it via connectEngine
// above) — 30s spin, then exit 1, on every PGLite invocation.
const { reindexFrontmatterCli } = await import('./commands/reindex-frontmatter.ts');
await reindexFrontmatterCli(args);
return; // reindexFrontmatterCli handles its own engine lifecycle
await reindexFrontmatterCli(engine, args);
break;
}
case 'backfill': {
// v0.30.1: first-class generic backfill command. Subcommand dispatch
// is inside runBackfillCommand (kind | list | --help).
// #1963: same double-connect class as reindex-frontmatter — reuse the
// connected engine instead of building a second one on the same
// PGLite data dir.
const { runBackfillCommand } = await import('./commands/backfill.ts');
await runBackfillCommand(args);
return;
await runBackfillCommand(engine, args);
break;
}
case 'code-callers': {
// v0.20.0 Cathedral II Layer 10 (C4): "who calls <symbol>?"
@@ -2305,6 +2358,28 @@ async function handleCliOnly(command: string, args: string[]) {
}
}
/**
* #3013: decide whether an invocation enters the read-only connect+dispatch
* timeout path, and with what wallclock. Returns null for every command
* dispatchReadOnlyCommand can't handle. The gate used to be "a timeout is
* present" so a user-supplied --timeout on a write command (`sync`,
* `embed`, `import`, ...) hijacked dispatch into the read-only path, which
* threw and exited 1 before any work ran. Pure; exported for the
* regression test.
*/
export function resolveReadOnlyDispatchTimeoutMs(
command: string,
subArgs: string[],
userTimeoutMs: number | null,
): number | null {
if (command !== 'search' && command !== 'sources') return null;
const defaultMs =
command === 'search' ? 30_000 :
(subArgs[0] === 'list' || subArgs[0] === undefined) ? 10_000 :
null;
return userTimeoutMs ?? defaultMs;
}
/**
* v0.41.6.0 D3: dispatch helper for the read-only commands that take a
* default wallclock timeout (`gbrain search`, `gbrain sources list`).
@@ -2340,7 +2415,52 @@ async function dispatchReadOnlyCommand(engine: BrainEngine, command: string, arg
import { buildGatewayConfig } from './core/ai/build-gateway-config.ts';
export { buildGatewayConfig };
/**
* Which brain this process's engine targets. Set by connectEngine after brain
* resolution; read by makeContext so ctx.brainId carries the audit id. Never
* derived from op params an untrusted caller must not be able to name a
* brain (same fail-closed shape as the #3524 remote source sentinel).
*/
let activeBrainId: string = 'host';
/**
* Connect to a mounted brain (brain axis, non-host). Routes through
* BrainRegistry so:
* - an unknown/disabled mount id throws UnknownBrainError. Fail-closed:
* the pre-fix CLI silently fell back to the host brain, returning
* confident wrong answers (mirror of #3524's explicit --source decision);
* - postgres mounts get a per-instance pool, never the db.ts singleton;
* - NO migrations run against the mount schema is the publisher's job
* (same decision as BrainRegistry.initMountBrain). Write access control
* is the mount's own DB credential grants: a read-only role rejects
* writes at the database; gbrain does not re-implement that client-side.
* The AI gateway still configures from the HOST config (the caller's API
* keys + model tiers) embedding/expansion spend stays the caller's, and
* a mount's DB-plane model config is never merged into the caller's gateway.
*/
async function connectMountEngine(brainId: string): Promise<BrainEngine> {
const config = loadConfig();
if (config) {
const { configureGateway } = await import('./core/ai/gateway.ts');
configureGateway(buildGatewayConfig(config));
}
const { loadRegistry } = await import('./core/brain-registry.ts');
const handle = await loadRegistry().getBrain(brainId);
activeBrainId = brainId;
return handle.engine;
}
async function connectEngine(opts?: { probeOnly?: boolean }): Promise<BrainEngine> {
// Brain axis: resolve WHICH DATABASE this invocation targets before touching
// the host engine. --brain (global flag) / GBRAIN_BRAIN_ID / .gbrain-mount /
// mount-path-prefix resolve via the canonical 6-tier chain — the mirror of
// the source axis in makeContext. connectEngine is the single choke point
// every local CLI command routes through (shared ops, CLI-only commands,
// and the search-dashboard path), so routing lands here once.
const { resolveBrainId } = await import('./core/brain-resolver.ts');
const brainId = resolveBrainId(getCliOptions().brain);
if (brainId !== 'host') return connectMountEngine(brainId);
const config = loadConfig();
if (!config) {
console.error('No brain configured. Run: gbrain init');
+60 -12
View File
@@ -108,7 +108,7 @@ Flags:
Exit codes:
0 Success (including "nothing to do").
1 An orchestrator failed.
1 An orchestrator failed, or schema migrations are pending (re-run with --yes).
2 Invalid arguments.
`);
}
@@ -260,6 +260,41 @@ function printDryRun(plan: Plan, installed: string): void {
}
}
/**
* #1530: schema-drift pre-flight resolution. When the schema version is
* behind, `--yes`/`--non-interactive` runs the schema migrations right there
* (the engine is already connected); interactive runs warn and return true so
* the caller exits non-zero instead of claiming "All migrations up to date".
* All output goes to stderr (migrations never print to stdout).
*
* Returns true when the schema is STILL behind after this call.
*/
async function resolveSchemaBehind(opts: {
schemaVer: number;
latest: number;
autoApply: boolean;
run: () => Promise<{ applied: number; current: number }>;
}): Promise<boolean> {
const { schemaVer, latest, autoApply, run } = opts;
if (schemaVer >= latest) return false;
if (autoApply) {
console.error(`Schema version ${schemaVer} is behind latest ${latest}; running schema migrations...`);
try {
const result = await run();
console.error(`Applied ${result.applied} schema migration(s); now at v${result.current}.`);
return false;
} catch (err) {
console.error(`Schema migration failed: ${err instanceof Error ? err.message : String(err)}`);
return true;
}
}
console.warn(
`\n⚠️ Schema version ${schemaVer} is behind latest ${latest}.\n` +
` Run \`gbrain apply-migrations --yes\` to apply now, or \`gbrain init --migrate-only\`.\n`,
);
return true;
}
function orchestratorOptsFrom(cli: ApplyMigrationsArgs): OrchestratorOpts {
return {
yes: cli.yes || cli.nonInteractive,
@@ -355,10 +390,13 @@ export async function runApplyMigrations(args: string[]): Promise<void> {
if (cli.forceAll) return; // both surfaces flushed
}
// Pre-flight: warn if schema migrations (migrate.ts) are behind.
// apply-migrations runs orchestrator migrations only; schema migrations
// run via connectEngine() / initSchema(). Users often expect this CLI
// to handle everything (Issue 1 from v0.18.0 field report).
// Pre-flight: detect schema migrations (migrate.ts) being behind.
// apply-migrations historically ran orchestrator migrations only; schema
// migrations run via connectEngine() / initSchema(). Users expect this CLI
// to handle everything (Issue 1 from v0.18.0 field report; #1530). With
// --yes/--non-interactive we apply them here; otherwise we warn and make
// sure the run does NOT report "All migrations up to date" with exit 0.
let schemaBehind = false;
try {
const { LATEST_VERSION } = await import('../core/migrate.ts');
const { loadConfig: lc, toEngineConfig } = await import('../core/config.ts');
@@ -378,14 +416,16 @@ export async function runApplyMigrations(args: string[]): Promise<void> {
await eng.connect(toEngineConfig(cfg));
const verStr = await eng.getConfig('version');
const schemaVer = parseInt(verStr || '1', 10);
const { runMigrations } = await import('../core/migrate.ts');
schemaBehind = await resolveSchemaBehind({
schemaVer,
latest: LATEST_VERSION,
// --list and --dry-run are read-only surfaces: never mutate schema
// even when combined with --yes/--non-interactive.
autoApply: (cli.yes || cli.nonInteractive) && !cli.dryRun && !cli.list,
run: () => runMigrations(eng),
});
await eng.disconnect();
if (schemaVer < LATEST_VERSION) {
console.warn(
`\n⚠️ Schema version ${schemaVer} is behind latest ${LATEST_VERSION}.\n` +
` Schema migrations run automatically on next connectEngine() / initSchema().\n` +
` To run them now: gbrain init --migrate-only\n`,
);
}
}
}
} catch {
@@ -420,6 +460,13 @@ export async function runApplyMigrations(args: string[]): Promise<void> {
const toRun: Migration[] = [...plan.partial, ...plan.pending];
if (toRun.length === 0) {
if (schemaBehind) {
console.error(
'Orchestrator migrations are up to date, but schema migrations are behind. ' +
'Run `gbrain apply-migrations --yes` (or `--force-schema`) to apply them.',
);
process.exit(1);
}
console.log('All migrations up to date.');
process.exit(0);
}
@@ -511,4 +558,5 @@ export const __testing = {
buildPlan,
indexCompleted,
statusForVersion,
resolveSchemaBehind,
};
+25 -6
View File
@@ -524,13 +524,17 @@ async function registerClient(name: string, args: string[]) {
* /admin/api/rescope-client endpoint.
*/
async function rescopeClient(clientId: string, args: string[]) {
const usage = 'Usage: auth rescope-client <client_id> [--source SOURCE] [--federated-read SRC1,SRC2,...]';
const usage = 'Usage: auth rescope-client <client_id> [--source SOURCE] [--federated-read SRC1,SRC2,...] [--bound-slug-prefixes P1,P2|none]';
if (!clientId) {
console.error(usage);
process.exit(1);
}
let sourceId: string | undefined;
let federatedRead: string[] | undefined;
// v0.42.72.0: tri-state — undefined = untouched, null = clear ('none'),
// array = replace. Lets roster churn (channel joins/leaves) update the
// write fence in place instead of register+rotate.
let boundSlugPrefixes: string[] | null | undefined;
for (let i = 0; i < args.length; i += 2) {
const flag = args[i];
const value = args[i + 1];
@@ -542,14 +546,18 @@ async function rescopeClient(clientId: string, args: string[]) {
if (flag === '--source') sourceId = value;
else if (flag === '--federated-read') {
federatedRead = value.split(',').map(s => s.trim()).filter(Boolean);
} else if (flag === '--bound-slug-prefixes') {
boundSlugPrefixes = value === 'none'
? null
: value.split(',').map(s => s.trim()).filter(Boolean);
} else {
console.error(`Error: Unknown flag: ${flag}`);
console.error(usage);
process.exit(1);
}
}
if (sourceId === undefined && federatedRead === undefined) {
console.error('Error: pass --source and/or --federated-read');
if (sourceId === undefined && federatedRead === undefined && boundSlugPrefixes === undefined) {
console.error('Error: pass --source, --federated-read, and/or --bound-slug-prefixes');
console.error(usage);
process.exit(1);
}
@@ -557,10 +565,13 @@ async function rescopeClient(clientId: string, args: string[]) {
await withConfiguredSql(async (sql) => {
const { GBrainOAuthProvider } = await import('../core/oauth-provider.ts');
const provider = new GBrainOAuthProvider({ sql });
const result = await provider.rescopeClient(clientId, { sourceId, federatedRead });
const result = await provider.rescopeClient(clientId, { sourceId, federatedRead, boundSlugPrefixes });
console.log(`OAuth client rescoped: "${result.clientName}" (${result.clientId})\n`);
console.log(` Write source: ${result.sourceId}`);
console.log(` Federated reads: ${result.federatedRead.join(', ') || '<none>'}`);
if (result.boundSlugPrefixes !== undefined) {
console.log(` Bound slug prefixes: ${result.boundSlugPrefixes?.join(', ') ?? '<none — full-source write authority>'}`);
}
console.log('\nTakes effect on the client\'s next request (existing tokens included).');
});
} catch (e: any) {
@@ -645,14 +656,22 @@ Usage:
--bound-tools <tool1,tool2> Bind submit_agent to an allow-list of tools
--bound-source <id> Bind submit_agent jobs to a source id
--bound-brain <id> Bind submit_agent jobs to a brain id
--bound-slug-prefixes <prefix1,prefix2> Bind submit_agent writes to slug prefixes
--bound-slug-prefixes <prefix1,prefix2> Fence ALL direct slug writes (put_page, delete_page,
tags, links, timeline, revert, raw data) AND
submit_agent to these prefixes. Each MUST end with
'/' or '/*' a boundary-less 'emp-alice' would also
name 'emp-alice-2/...'. Ops that write by something
other than a slug (extract_*, forget_fact,
ontology_propose, sources_*) and POST /ingest become
unavailable to a bound client. Omit = full-source writes.
--bound-max-concurrent <n> Bound submit_agent concurrency (default: 1)
--budget-usd-per-day <usd> Bound submit_agent daily spend cap
gbrain auth rescope-client <client_id> [options] Change an existing client's source scope (e.g. a DCR
client stuck on the 'default' source). Only the flags
you pass change; the other axis is left as-is.
you pass change; the other axes are left as-is.
--source <id> New write source
--federated-read <id1,id2,...> New read-scope source list
--bound-slug-prefixes <p1,p2|none> Replace the slug-prefix write fence ('none' clears it)
gbrain auth revoke-client <client_id> Hard-delete an OAuth 2.1 client (cascades to tokens + codes)
gbrain auth test <url> --token <token> Smoke-test a remote MCP server
`);
+34 -1
View File
@@ -1,9 +1,42 @@
import { defaultTimeoutMsFor } from '../core/minions/handler-timeouts.ts';
// #2781: the full-cycle floor used to be a literal `1_800_000` that merely
// HAPPENED to match the 'autopilot-cycle' / 'autopilot-global-maintenance'
// handler anchors (`HANDLER_DEFAULT_TIMEOUT_MS`, #1737) instead of being
// derived from them. A duplicated literal can silently drift from the
// handler default it's supposed to track — which is exactly the bug class
// #2781 reported (an explicit `timeout_ms` stamp permanently overrides the
// handler default per `queue.ts`'s `opts?.timeout_ms ?? defaultTimeoutMsFor`,
// so a stale/lower literal here would starve a phase the handler default
// was sized for). Deriving the floor from `defaultTimeoutMsFor` for both
// full-cycle job names keeps the stamp coupled to its anchor by construction.
// Fail fast (not `?? 0`) if either handler ever loses its entry in
// HANDLER_DEFAULT_TIMEOUT_MS — silently falling back to "no floor" would
// reintroduce #2781 rather than surface the drift.
function requireHandlerAnchorMs(jobName: string): number {
const ms = defaultTimeoutMsFor(jobName);
if (ms === null) {
throw new Error(
`resolveAutopilotDispatchTimeoutMs: '${jobName}' has no entry in HANDLER_DEFAULT_TIMEOUT_MS ` +
'(handler-timeouts.ts) — the full-cycle timeout floor can no longer be derived from it. ' +
'See #2781: a missing/removed anchor here silently reintroduces the interval-derived stamp ' +
'permanently overriding the handler default.',
);
}
return ms;
}
const FULL_CYCLE_TIMEOUT_FLOOR_MS = Math.max(
requireHandlerAnchorMs('autopilot-cycle'),
requireHandlerAnchorMs('autopilot-global-maintenance'),
);
export function resolveAutopilotDispatchTimeoutMs(
baseIntervalSeconds: number,
fullCycle: boolean,
): number {
const intervalDerivedTimeoutMs = Math.max(baseIntervalSeconds * 2 * 1000, 300_000);
return fullCycle
? Math.max(intervalDerivedTimeoutMs, 1_800_000)
? Math.max(intervalDerivedTimeoutMs, FULL_CYCLE_TIMEOUT_FLOOR_MS)
: intervalDerivedTimeoutMs;
}
+13 -4
View File
@@ -981,12 +981,21 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
// can't shrink throughput (codex #9/D5). autopilot-cycle jobs run on
// the 'default' queue, so that's the concurrency we compare against.
const fanoutMax = await resolveEffectiveFanoutMax(engine, 'default');
// #2781: both 'autopilot-cycle' (per-source) and 'autopilot-global-
// maintenance' carry a 30-min handler anchor (handler-timeouts.ts)
// because a full cycle can outlive short daemon intervals — unlike
// the lighter interval-derived `timeoutMs` above (sync/freshness,
// extract-atoms-drain, targeted small-plan steps), which have no
// such anchor and are meant to stay interval-derived. Naming this
// separately (rather than reusing the outer `timeoutMs`) avoids
// the #2781 bug class: dispatchGlobalMaintenance previously reused
// the outer non-full-cycle `timeoutMs` by shorthand, silently
// dropping its own handler anchor.
const fullCycleTimeoutMs = resolveAutopilotDispatchTimeoutMs(baseInterval, true);
const result = await dispatchPerSource(engine, queue, {
repoPath,
slot,
// Full cycles can outlive short daemon intervals. Keep lighter dispatches
// interval-derived while giving per-source consolidation enough time.
timeoutMs: resolveAutopilotDispatchTimeoutMs(baseInterval, true),
timeoutMs: fullCycleTimeoutMs,
fanoutMax,
jsonMode,
});
@@ -997,7 +1006,7 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
// the per-source path (legacy single-source still runs everything).
if (!result.legacy_fallback) {
try {
await dispatchGlobalMaintenance(engine, queue, { repoPath, slot, timeoutMs, jsonMode });
await dispatchGlobalMaintenance(engine, queue, { repoPath, slot, timeoutMs: fullCycleTimeoutMs, jsonMode });
} catch (e) {
if (jsonMode) process.stderr.write(JSON.stringify({ event: 'global_maintenance_dispatch_failed', error: e instanceof Error ? e.message : String(e) }) + '\n');
}
+9 -13
View File
@@ -16,10 +16,10 @@
* always reserving 1 connection for HNSW + heartbeat + doctor probes.
*/
import type { BrainEngine } from '../core/engine.ts';
import { resolveDirectPoolSize } from '../core/connection-manager.ts';
import { listBackfills, getBackfill } from '../core/backfill-registry.ts';
import { runBackfill, clearBackfillCheckpoint } from '../core/backfill-base.ts';
import { loadConfig, toEngineConfig } from '../core/config.ts';
interface BackfillArgs {
kind?: string;
@@ -114,7 +114,14 @@ function clampConcurrency(requested: number | undefined): { effective: number; w
return { effective: requested };
}
export async function runBackfillCommand(args: string[]): Promise<void> {
/**
* #1963 (same class as reindex-frontmatter): takes the ALREADY-CONNECTED
* engine from cli.ts's dispatch. Building a second engine here deadlocked on
* the PGLite data-dir lock (cli.ts's `connectEngine()` already holds it in
* this same process) every `gbrain backfill <kind>` on PGLite timed out
* after 30s. Engine lifecycle belongs to cli.ts's connect + teardown.
*/
export async function runBackfillCommand(engine: BrainEngine, args: string[]): Promise<void> {
const cli = parseArgs(args);
if (cli.help) { printHelp(); return; }
@@ -144,20 +151,10 @@ export async function runBackfillCommand(args: string[]): Promise<void> {
process.exit(2);
}
const config = loadConfig();
if (!config) {
console.error('No brain configured. Run: gbrain init');
process.exit(2);
}
// X5 admission control — clamp concurrency to direct-pool capacity.
const { effective: concurrency, warning } = clampConcurrency(cli.concurrency);
if (warning) console.warn(warning);
const { createEngine } = await import('../core/engine-factory.ts');
const engine = await createEngine(toEngineConfig(config));
await engine.connect(toEngineConfig(config));
if (cli.fresh) {
await clearBackfillCheckpoint(engine, reg.spec.name);
console.log(`Cleared checkpoint for backfill.${reg.spec.name}`);
@@ -192,7 +189,6 @@ export async function runBackfillCommand(args: string[]): Promise<void> {
if (result.cappedByMaxRows) console.log(` ⚠️ Capped by --max-rows; more remain.`);
if (result.cappedByErrors) console.log(` ⚠️ Capped by --max-errors at ${result.errors}.`);
await engine.disconnect();
if (result.cappedByErrors) process.exit(1);
}
+9 -6
View File
@@ -45,12 +45,15 @@ function upgradeCommandForMethod(method: string): string {
}
}
/** Where the latest version is resolved from. gbrain publishes NO GitHub
* releases (the `releases/latest` API is a permanent 404), so the release
* train's source of truth is the `VERSION` file on master same trusted host
* `fetchChangelog` already uses. An npm fallback was rejected: the `gbrain`
* package on npm is an unrelated GPU library (#505), so it would produce false
* upgrade prompts pointing at a stranger's package. */
/** Where the latest version is resolved from. The release train's source of
* truth is the `VERSION` file on master same trusted host `fetchChangelog`
* already uses. GitHub releases are published from it per VERSION bump
* (`.github/workflows/release.yml`, #3521) and carry the binary assets, but
* this check deliberately does NOT read `releases/latest`: it was a permanent
* 404 before releases existed (#3520) and can still lag master. An npm
* fallback was rejected: the `gbrain` package on npm is an unrelated GPU
* library (#505), so it would produce false upgrade prompts pointing at a
* stranger's package. */
const VERSION_SOURCE_URL = 'https://raw.githubusercontent.com/garrytan/gbrain/master/VERSION';
const RELEASE_NOTES_URL = 'https://github.com/garrytan/gbrain/blob/master/CHANGELOG.md';
+8 -1
View File
@@ -46,7 +46,14 @@ export async function runConfig(engine: BrainEngine, args: string[]) {
}
console.log('GBrain config:');
for (const [k, v] of Object.entries(config)) {
const display = typeof v === 'string' ? redactConfigValue(k, v) : v;
// #575: objects interpolated into the template literal printed
// `[object Object]` — render them as JSON instead. Sensitive keys
// stay redacted whether the value is a string or an object.
const display = typeof v === 'string'
? redactConfigValue(k, v)
: v !== null && typeof v === 'object'
? (isSensitiveConfigKey(k) ? '***' : JSON.stringify(v))
: v;
console.log(` ${k}: ${display}`);
}
return;
+108
View File
@@ -0,0 +1,108 @@
/**
* #1835 storage_path resolution for the doctor `image_assets` check.
*
* `files.storage_path` rows written by a Windows gbrain install carry Windows
* drive paths (`D:/foo/img.jpg`, `D:\foo\img.jpg`). On POSIX,
* `path.isAbsolute()` is false for those, so the old code joined them onto the
* repo root and produced a path that can never exist a false-positive
* "missing from disk, restore from git" WARN under WSL and macOS.
*
* Policy:
* - win32: drive paths are absolute; stat them as-is.
* - WSL (linux + "microsoft" in /proc/version): translate `D:/x` to
* `<automount root>/d/x` (automount root read from /etc/wsl.conf
* `[automount] root`, default `/mnt`) and stat that.
* - any other POSIX host (macOS, plain Linux): the path is unresolvable on
* this platform report it as foreign so the caller SKIPS the stat
* instead of inventing a path that will never exist.
*
* Kept in its own module (not doctor.ts) so the pure tests don't pull the
* 7k-line doctor dep graph, and so open PRs rewriting the image_assets block
* (e.g. a `resolveImageAssetPath` helper) can adopt it with a one-line call.
*/
import { readFileSync } from 'node:fs';
import { join, posix, win32 } from 'node:path';
const WINDOWS_DRIVE_RE = /^([A-Za-z]):[\\/](.*)$/;
export interface AssetPathResolution {
/** Absolute path to stat, or null when the path is unresolvable here. */
abs: string | null;
/** True when storage_path is a Windows drive path this host cannot stat. */
foreign: boolean;
}
/**
* Resolve a files.storage_path to a stat-able absolute path.
* `opts.platform` / `opts.wslMountRoot` exist for tests; production callers
* pass neither (process.platform + detected WSL automount root).
* `wslMountRoot: null` means "not under WSL".
*/
export function resolveAssetPath(
storagePath: string,
repoRoot: string,
opts: { platform?: NodeJS.Platform; wslMountRoot?: string | null } = {},
): AssetPathResolution {
const platform = opts.platform ?? process.platform;
if (platform !== 'win32') {
const m = WINDOWS_DRIVE_RE.exec(storagePath);
if (m) {
const root = opts.wslMountRoot !== undefined ? opts.wslMountRoot : detectWslMountRoot();
if (root === null) return { abs: null, foreign: true };
const abs = `${root.replace(/\/+$/, '')}/${m[1].toLowerCase()}/${m[2].replace(/\\/g, '/')}`;
return { abs, foreign: false };
}
}
// Platform-appropriate absoluteness (not the host's) so injected-platform
// tests behave identically everywhere; in production platform === host.
const isAbs = platform === 'win32' ? win32.isAbsolute(storagePath) : posix.isAbsolute(storagePath);
return {
abs: isAbs ? storagePath : join(repoRoot, storagePath),
foreign: false,
};
}
/**
* Extract the `[automount] root` value from /etc/wsl.conf content.
* Defaults to `/mnt` (WSL's own default) when absent/unparseable.
*/
export function parseWslAutomountRoot(conf: string): string {
let inAutomount = false;
for (const raw of conf.split(/\r?\n/)) {
const line = raw.replace(/[#;].*$/, '').trim();
if (line.startsWith('[')) {
inAutomount = /^\[automount\]$/i.test(line);
continue;
}
if (!inAutomount) continue;
const m = /^root\s*=\s*"?([^"]+?)"?\s*$/.exec(line);
if (m) return m[1];
}
return '/mnt';
}
let cachedWslMountRoot: string | null | undefined;
/**
* Detect the WSL Windows-drive automount root. Returns null when not running
* under WSL (including macOS and plain Linux). Memoized per process.
*/
export function detectWslMountRoot(): string | null {
if (cachedWslMountRoot === undefined) cachedWslMountRoot = computeWslMountRoot();
return cachedWslMountRoot;
}
function computeWslMountRoot(): string | null {
if (process.platform !== 'linux') return null;
try {
// The standard WSL tell: kernel version string names Microsoft.
if (!/microsoft/i.test(readFileSync('/proc/version', 'utf8'))) return null;
} catch {
return null;
}
try {
return parseWslAutomountRoot(readFileSync('/etc/wsl.conf', 'utf8'));
} catch {
return '/mnt'; // WSL default when wsl.conf is absent.
}
}
+336 -10
View File
@@ -6,6 +6,11 @@ import { LATEST_VERSION, getIdleBlockers } from '../core/migrate.ts';
import { checkResolvable } from '../core/check-resolvable.ts';
import { autoFixDryViolations, type AutoFixReport, type FixOutcome } from '../core/dry-fix.ts';
import { autoDetectSkillsDirReadOnly } from '../core/repo-root.ts';
import {
SKILLS_MANIFEST_FILENAME,
verifySkillsManifest,
type SkillsManifest,
} from '../core/skills-integrity.ts';
import { loadOrDeriveManifest } from '../core/skill-manifest.ts';
import { parseSkillFrontmatter } from '../core/skill-frontmatter.ts';
import {
@@ -47,6 +52,13 @@ import { lagFromContentMs } from '../core/source-health.ts';
import { CHUNKER_VERSION } from '../core/chunkers/code.ts';
import { LINK_EXTRACTOR_VERSION_TS } from '../core/link-extraction.ts';
import { isUndefinedColumnError } from '../core/utils.ts';
import {
loadStorageConfig,
effectiveDbOnlyDirs,
DERIVE_PHASE_DB_ONLY_DEFAULTS,
findDbOnlyCollisions,
} from '../core/storage-config.ts';
import { slugifyPath } from '../core/sync.ts';
// issue #1777: hidden_by_search_policy — count chunked pages withheld from
// default search by the hard-exclude prefix policy. Reuses the canonical
// exclude resolver + LIKE escaper + visibility clause so the doctor count can't
@@ -3667,6 +3679,198 @@ export async function checkUnverifiedExtractions(
}
}
/**
* issue #2250 (reported by @615Works) content_hash_duplicates.
*
* `gbrain import` run from the wrong root (one level too deep) drops the
* path prefix from every slug, leaving `people/x` and `x` coexisting with
* identical content. `dream --phase purge` never removes them (they aren't
* file-backed orphans) and nothing surfaced the condition. One GROUP BY
* never an N² hash comparison flags hash groups that contain BOTH a bare
* slug (no '/') and a path-prefixed slug.
*/
export async function checkContentHashDuplicates(engine: BrainEngine): Promise<Check> {
const name = 'content_hash_duplicates';
const fix = 'Fix: gbrain pages delete <bare-slug> for each pair, then gbrain pages purge-deleted --older-than 0';
try {
const rows = await engine.executeRaw<{ source_id: string; content_hash: string; slugs: string }>(
`SELECT source_id, content_hash,
string_agg(slug, '|' ORDER BY length(slug), slug) AS slugs
FROM pages
WHERE deleted_at IS NULL AND content_hash IS NOT NULL AND content_hash <> ''
GROUP BY source_id, content_hash
HAVING count(*) > 1
AND count(*) FILTER (WHERE strpos(slug, '/') = 0) > 0
AND count(*) FILTER (WHERE strpos(slug, '/') > 0) > 0
LIMIT 50`,
);
if (rows.length === 0) {
return { name, status: 'ok', message: 'No content-hash duplicate pairs (bare vs path-prefixed slugs)' };
}
let pairCount = 0;
const samples: string[] = [];
for (const r of rows) {
const slugs = String(r.slugs).split('|');
const prefixed = slugs.filter(s => s.includes('/'));
for (const bare of slugs.filter(s => !s.includes('/'))) {
const twin = prefixed.find(p => p.endsWith('/' + bare)) ?? prefixed[0];
pairCount++;
if (samples.length < 5) samples.push(`${bare} <-> ${twin}`);
}
}
return {
name,
status: 'warn',
message: `${pairCount} content-hash duplicate pair(s) detected (same content, differing slug forms — usually an import run from the wrong root, which drops the path prefix). Sample: ${samples.join('; ')}. ${fix}`,
details: { pair_count: pairCount, hash_groups: rows.length, sample_pairs: samples },
};
} catch (e) {
return { name, status: 'warn', message: `Could not check content-hash duplicates: ${(e as Error).message}` };
}
}
/** Walk a repo for markdown files and return their slugified (lowercased) slugs. */
function collectMarkdownSlugs(root: string): Set<string> {
const out = new Set<string>();
const stack = [''];
while (stack.length > 0) {
const rel = stack.pop()!;
let entries;
try {
entries = readdirSync(rel ? join(root, rel) : root, { withFileTypes: true });
} catch {
continue;
}
for (const e of entries) {
if (e.name.startsWith('.') || e.name === 'node_modules') continue;
const childRel = rel ? `${rel}/${e.name}` : e.name;
if (e.isDirectory()) stack.push(childRel);
else if (/\.mdx?$/i.test(e.name)) out.add(slugifyPath(childRel).toLowerCase());
}
}
return out;
}
/**
* issue #2784 (reported by @alexputici) undeclared_db_only_pages.
*
* A markdown page with no backing file that sits outside every declared
* db_only path is invisible to any file-lane backup/recovery reasoning: an
* operator auditing "what would survive a DB loss" gets a silently wrong
* answer. The engine's own derive-phase output prefixes
* (DERIVE_PHASE_DB_ONLY_DEFAULTS) count as implicitly declared so the check
* stays quiet on healthy brains. Deliberately allowed to stat the source
* repo (the one thing the SQL-only check registry could never see).
*/
export async function checkUndeclaredDbOnlyPages(engine: BrainEngine): Promise<Check> {
const name = 'undeclared_db_only_pages';
try {
const sources = await engine.executeRaw<{ id: string; local_path: string | null }>(
`SELECT id, local_path FROM sources WHERE local_path IS NOT NULL`,
);
const checkable = sources.filter(s => s.local_path && existsSync(s.local_path));
if (checkable.length === 0) {
return { name, status: 'ok', message: 'Not applicable (no sources with a local repo path on this host)' };
}
let total = 0;
const samples: string[] = [];
const perSource: Record<string, number> = {};
for (const src of checkable) {
let declared: string[] = [];
try {
declared = loadStorageConfig(src.local_path)?.db_only ?? [];
} catch {
// invalid gbrain.yml — treated as no declarations; the sync path
// already surfaces the config error itself.
}
const dbOnlyDirs = effectiveDbOnlyDirs(declared);
const rows = await engine.executeRaw<{ slug: string }>(
`SELECT slug FROM pages WHERE deleted_at IS NULL AND source_id = $1 AND page_kind = 'markdown'`,
[src.id],
);
if (rows.length === 0) continue;
const backed = collectMarkdownSlugs(src.local_path!);
for (const { slug } of rows) {
if (dbOnlyDirs.some(dir => slug.startsWith(dir))) continue;
if (backed.has(slug)) continue;
total++;
perSource[src.id] = (perSource[src.id] ?? 0) + 1;
if (samples.length < 5) samples.push(`${slug} (src=${src.id})`);
}
}
if (total === 0) {
return {
name,
status: 'ok',
message: `Every DB page is file-backed or under a declared/default db_only path (derive-phase defaults: ${DERIVE_PHASE_DB_ONLY_DEFAULTS.join(' ')})`,
};
}
return {
name,
status: 'warn',
message: `${total} DB page(s) have no backing file and sit outside every declared/default db_only path — invisible to file-lane backup/recovery. Sample: ${samples.join('; ')}. Fix: restore or export the files, or declare their prefixes under storage.db_only in gbrain.yml (derive-phase defaults already cover: ${DERIVE_PHASE_DB_ONLY_DEFAULTS.join(' ')})`,
details: { total, per_source: perSource, sample_slugs: samples },
};
} catch (e) {
return { name, status: 'warn', message: `Could not check undeclared db-only pages: ${(e as Error).message}` };
}
}
/**
* issue #2788 (reported by @alexputici) db_only_collector_collision.
*
* Declaring a collector's output dir in storage.db_only silently kills its
* ingestion: manageGitignore auto-gitignores the dir, the git-walking sync
* never sees the files, and import honors .gitignore too everything stays
* green while nothing reaches the DB (a 7-week outage in the field). The
* recipe's `output_paths` frontmatter is the ground truth; the same warning
* also fires at .gitignore-write time inside sync's manageGitignore.
*/
export async function checkDbOnlyCollectorCollision(
engine: BrainEngine,
opts?: { collectors?: Array<{ id: string; output_path: string }> },
): Promise<Check> {
const name = 'db_only_collector_collision';
try {
let collectors = opts?.collectors;
if (!collectors) {
const { getConfiguredCollectorOutputs } = await import('./integrations.ts');
collectors = getConfiguredCollectorOutputs();
}
if (collectors.length === 0) {
return { name, status: 'ok', message: 'No configured collectors declare output paths' };
}
const sources = await engine.executeRaw<{ id: string; local_path: string | null }>(
`SELECT id, local_path FROM sources WHERE local_path IS NOT NULL`,
);
const hits: string[] = [];
for (const src of sources) {
if (!src.local_path || !existsSync(src.local_path)) continue;
let dbOnly: string[] = [];
try {
dbOnly = loadStorageConfig(src.local_path)?.db_only ?? [];
} catch {
continue;
}
if (dbOnly.length === 0) continue;
for (const hit of findDbOnlyCollisions(collectors, dbOnly)) {
hits.push(`collector '${hit.id}' writes to '${hit.output_path}' which is inside db_only path '${hit.db_only_dir}' (source ${src.id})`);
}
}
if (hits.length === 0) {
return { name, status: 'ok', message: 'No collector output dir falls inside a db_only path' };
}
return {
name,
status: 'warn',
message: `${hits.length} collector/db_only collision(s): ${hits.join('; ')}. db_only dirs are auto-gitignored, so sync AND import silently skip files there — the collector runs green while nothing reaches the DB. Fix: remove the prefix from storage.db_only in gbrain.yml, or move the collector output.`,
details: { collisions: hits },
};
} catch (e) {
return { name, status: 'warn', message: `Could not check collector/db_only collisions: ${(e as Error).message}` };
}
}
/**
* issue #1678 extract_atoms_backlog doctor check.
*
@@ -4349,8 +4553,18 @@ export async function checkCycleFreshness(
: `'${source.id}'`;
const raw = source.config?.last_full_cycle_at;
if (typeof raw !== 'string') {
// #2540: WARN, not FAIL. This check iterates EVERY local_path source,
// so on a multi-source install where only some vaults are cycled
// (e.g. one nightly `gbrain dream --dir <vault>`), a never-cycled
// sibling source turned doctor permanently red — which erodes the
// check's signal until real staleness hides inside the noise (the
// reporter's install masked genuinely stale sources for weeks this
// way). "Never cycled" also fires on a source added minutes ago.
// A source that HAS cycled and then went stale still escalates
// through the warn/fail age thresholds below — that is the
// regression signal this check exists for.
issues.push(`Source ${display} has never completed a full cycle`);
hasFailures = true;
hasWarnings = true;
continue;
}
const last = new Date(raw).getTime();
@@ -4386,7 +4600,7 @@ export async function checkCycleFreshness(
return {
name: 'cycle_freshness',
status: 'warn',
message: `${issues.join('; ')}.`,
message: `${issues.join('; ')}. Run \`gbrain dream --source <id>\` to cycle a source, or start \`gbrain autopilot\`.`,
};
}
return {
@@ -4835,6 +5049,15 @@ export async function buildChecks(
checks.push(skillBrainFirstCheck(skillsDir));
}
// 2c. Skills manifest integrity (#159): tamper-evidence, not signatures.
// Compares the skills tree against its committed skills.lock.json and
// WARNS on drift — never fails, never blocks. No manifest (e.g. a user
// workspace skills dir, or a compiled binary far from the repo) → ok/skip.
// SKILL group — gated.
if (scope === 'all' && skillsDir) {
checks.push(skillsManifestIntegrityCheck(skillsDir));
}
// 3. Half-migrated Minions detection (filesystem-only).
// If completed.jsonl has any status:"partial" entry with no later
// status:"complete" for the same version, the install is mid-migration.
@@ -5592,6 +5815,42 @@ export async function buildChecks(
// Best-effort filesystem-hygiene check; never block doctor.
}
// 3f. npm_squat (#505). The npm registry name `gbrain` belongs to an
// unrelated third-party package — this project is NOT distributed on npm.
// A reflexive `npm i -g gbrain` / `bun add -g gbrain` installs something
// unrelated that can shadow the real binary on PATH. Classify every
// `gbrain` that `which -a` finds (pure helpers in
// src/core/npm-squat-check.ts) and warn when an unrelated install wins on
// PATH or the entry is broken. Skips silently when gbrain isn't on PATH
// at all (e.g. running via `bun src/cli.ts`).
try {
const { execSync } = await import('node:child_process');
let candidates: string[] = [];
try {
candidates = execSync('which -a gbrain', {
encoding: 'utf8',
stdio: ['ignore', 'pipe', 'ignore'],
})
.split('\n')
.map((s) => s.trim())
.filter(Boolean);
} catch {
// `which` exits non-zero when gbrain isn't on PATH (or is missing
// entirely on this platform) — nothing to check.
}
const { assessGbrainBinaries } = await import('../core/npm-squat-check.ts');
const assessment = assessGbrainBinaries(candidates);
if (assessment.status !== 'skip') {
checks.push({
name: 'npm_squat',
status: assessment.status,
message: assessment.message,
});
}
} catch {
// Best-effort environment check; never block doctor.
}
// 3b-multi-source. Multi-source drift (v0.31.8 — D8 + D17 + OV12 + OV13).
// Pre-v0.30.3 putPage misrouted multi-source writes to (default, slug).
// For each non-default source with local_path set, walk the FS and surface
@@ -6056,7 +6315,10 @@ export async function buildChecks(
} else {
// Live embed test
const start = Date.now();
const vec = await embedOne('gbrain doctor embedding smoke test');
// Doctor is itself the provider-health circuit breaker. A permanent
// billing/auth failure must be sampled once, not multiplied by the AI
// SDK's default retries (which can add ~90s to every health check).
const vec = await embedOne('gbrain doctor embedding smoke test', { maxRetries: 0 });
const ms = Date.now() - start;
const actualDims = vec.length;
@@ -7536,33 +7798,44 @@ export async function buildChecks(
`SELECT storage_path FROM files WHERE mime_type LIKE 'image/%' LIMIT 1000`
);
let vanished = 0;
let foreign = 0;
const vanishedPaths: string[] = [];
const fs = await import('node:fs');
const nodePath = await import('node:path');
const { resolveAssetPath } = await import('./doctor-asset-paths.ts');
// storage_path is repo-relative for sync-ingested assets. Resolving
// against cwd made this check a false-positive WARN whenever doctor
// ran outside the brain repo.
const repoRoot = (await engine.getConfig('sync.repo_path')) ?? process.cwd();
for (const r of rows) {
const abs = nodePath.isAbsolute(r.storage_path)
? r.storage_path
: nodePath.join(repoRoot, r.storage_path);
// #1835: Windows drive paths (D:/…) translate to the WSL automount
// (/mnt/d/…) under WSL, and are SKIPPED (not "missing") on hosts
// where they cannot exist (macOS / plain Linux) — never joined onto
// repoRoot, which produced a false "restore from git" WARN.
const resolved = resolveAssetPath(r.storage_path, repoRoot);
if (resolved.abs === null) {
foreign++;
continue;
}
try {
fs.statSync(abs);
fs.statSync(resolved.abs);
} catch {
vanished++;
if (vanishedPaths.length < 5) vanishedPaths.push(r.storage_path);
}
}
const checked = rows.length - foreign;
const foreignNote = foreign > 0
? ` (${foreign} Windows-drive path(s) skipped — not resolvable on this platform)`
: '';
if (rows.length === 0) {
checks.push({ name: 'image_assets', status: 'ok', message: 'No image assets indexed yet' });
} else if (vanished === 0) {
checks.push({ name: 'image_assets', status: 'ok', message: `${rows.length} image(s) all present on disk` });
checks.push({ name: 'image_assets', status: 'ok', message: `${checked} image(s) all present on disk${foreignNote}` });
} else {
checks.push({
name: 'image_assets',
status: 'warn',
message: `${vanished} of ${rows.length} image(s) missing from disk (e.g. ${vanishedPaths.join(', ')}). ` +
message: `${vanished} of ${checked} image(s) missing from disk (e.g. ${vanishedPaths.join(', ')})${foreignNote}. ` +
`Fix: restore from git, or \`gbrain sync --skip-failed\` to acknowledge.`,
});
}
@@ -7622,6 +7895,14 @@ export async function buildChecks(
// per-source dispatch gate sees.
progress.heartbeat('cycle_freshness');
checks.push(await checkCycleFreshness(engine));
// Silent-failure batch (#2250 / #2784 / #2788): wrong-root import
// duplicates, undeclared DB-only pages, collector-output-in-db_only.
progress.heartbeat('content_hash_duplicates');
checks.push(await checkContentHashDuplicates(engine));
progress.heartbeat('undeclared_db_only_pages');
checks.push(await checkUndeclaredDbOnlyPages(engine));
progress.heartbeat('db_only_collector_collision');
checks.push(await checkDbOnlyCollectorCollision(engine));
}
// v0.32.3 search-lite — mode + eval_drift surfaces. Status stays 'ok' per
@@ -7856,6 +8137,51 @@ export function skillConformanceCheck(skillsDir: string): Check {
* Test seam: pure function, no `process.exit`. Direct call from tests
* with a synthetic skills dir under tempdir.
*/
/**
* Skills-manifest integrity check (#159). Verifies the skills tree against
* the committed skills.lock.json tamper-evidence manifest. Advisory only:
* drift is a WARN (local edits are legitimate), and a missing/unreadable
* manifest is an ok/skip a user's workspace skills dir or a compiled
* binary far from the repo has no manifest, and that is not a problem.
*/
export function skillsManifestIntegrityCheck(skillsDir: string): Check {
const name = 'skills_manifest_integrity';
const manifestPath = join(skillsDir, SKILLS_MANIFEST_FILENAME);
if (!existsSync(manifestPath)) {
return { name, status: 'ok', message: `No ${SKILLS_MANIFEST_FILENAME} in ${skillsDir} — integrity check not applicable` };
}
let drift: ReturnType<typeof verifySkillsManifest>;
let tracked: number;
try {
const manifest = JSON.parse(readFileSync(manifestPath, 'utf-8')) as SkillsManifest;
tracked = Object.keys(manifest).length;
drift = verifySkillsManifest(skillsDir, manifest);
} catch (err) {
// Fail-safe: an unreadable/unparseable manifest or a filesystem error
// skips the check rather than warning — this check must never block.
const msg = err instanceof Error ? err.message : String(err);
return { name, status: 'ok', message: `Could not verify ${SKILLS_MANIFEST_FILENAME} (${msg}) — integrity check skipped` };
}
const total = drift.modified.length + drift.missing.length + drift.extra.length;
if (total === 0) {
return { name, status: 'ok', message: `${tracked} bundled skill files match ${SKILLS_MANIFEST_FILENAME}` };
}
const sample = (files: string[]): string =>
files.slice(0, 5).join(', ') + (files.length > 5 ? `, … +${files.length - 5} more` : '');
const parts: string[] = [];
if (drift.modified.length > 0) parts.push(`${drift.modified.length} modified (${sample(drift.modified)})`);
if (drift.missing.length > 0) parts.push(`${drift.missing.length} missing (${sample(drift.missing)})`);
if (drift.extra.length > 0) parts.push(`${drift.extra.length} extra (${sample(drift.extra)})`);
return {
name,
status: 'warn',
message:
`skills/ drifted from ${SKILLS_MANIFEST_FILENAME} (advisory — local edits are fine): ${parts.join('; ')}. ` +
`If intentional, regenerate: bun run scripts/generate-skills-manifest.ts`,
details: { modified: drift.modified, missing: drift.missing, extra: drift.extra },
};
}
export function skillBrainFirstCheck(skillsDir: string): Check {
let manifest: ReturnType<typeof loadOrDeriveManifest>;
try {
+217 -36
View File
@@ -19,10 +19,26 @@ import {
} from '../core/pace-mode.ts';
import { tryAcquireDbLock, type DbLockHandle } from '../core/db-lock.ts';
import { embedBackfillLockId } from '../core/embed-backfill-lock.ts';
import { AITransientError } from '../core/ai/errors.ts';
import { wrapChunkTextsForStoredMode } from '../core/embedding-context.ts';
import { titleTierCorpusGeneration } from '../core/contextual-retrieval-service.ts';
import type { Page } from '../core/types.ts';
/** #3037: cap failure samples so a corpus-wide outage doesn't bloat --json. */
const FAILURE_SAMPLE_CAP = 10;
/**
* #3037: record embed failures on the run result. `chunkCount` is the number
* of chunks left un-embedded by this failure (1 for page-level errors where
* the chunk count isn't known at the catch site).
*/
function recordFailure(result: EmbedResult, chunkCount: number, slug: string, e: unknown): void {
result.failures += chunkCount;
if (result.failure_samples.length < FAILURE_SAMPLE_CAP) {
result.failure_samples.push(`${slug}: ${e instanceof Error ? e.message : String(e)}`);
}
}
/**
* #3507 after a plain re-embed fully re-embedded a `per_chunk_synopsis`
* page at the title-only tier (see wrapChunkTextsForStoredMode), restamp the
@@ -166,6 +182,20 @@ export interface EmbedResult {
total_chunks: number;
/** Number of pages processed (whether or not they had stale chunks). */
pages_processed: number;
/**
* #3037: chunks that FAILED to embed this run (batch failures + per-chunk
* isolation failures). Callers must not read total silence as success:
* `src/cli.ts` turns `failures > 0` into a non-zero exit verdict (mirrors
* the `import` errors>0 guard), and structured consumers (--json, minion
* handlers) can surface it. 0 on a clean run. Additive field.
*/
failures: number;
/**
* #3037: up to 10 `slug: error-message` samples of what failed, so the
* operator gets a diagnosis without scrolling stderr. Capped so a
* corpus-wide outage doesn't bloat structured output. Additive field.
*/
failure_samples: string[];
/** True if this run was a dry-run. */
dryRun: boolean;
/**
@@ -284,6 +314,8 @@ export async function runEmbedCore(engine: BrainEngine, opts: EmbedOpts): Promis
would_embed: 0,
total_chunks: 0,
pages_processed: 0,
failures: 0,
failure_samples: [],
dryRun: !!opts.dryRun,
};
@@ -293,6 +325,11 @@ export async function runEmbedCore(engine: BrainEngine, opts: EmbedOpts): Promis
try {
await embedPage(engine, s, !!opts.dryRun, result, opts.sourceId, opts.signal, opts.quiet);
} catch (e: unknown) {
if (isAborted(opts.signal)) break; // shutdown, not a failure
// #3037: a page-level error (not found, DB write) must not exit 0.
// Chunk-level embed failures are counted inside embedPage; this
// counts the page itself (chunk count unknown at this site).
recordFailure(result, 1, s, e);
serr(` Error embedding ${s}: ${e instanceof Error ? e.message : e}`);
}
}
@@ -535,6 +572,12 @@ export async function runEmbed(engine: BrainEngine, args: string[]): Promise<Emb
try {
const result = await runEmbedCore(engine, opts);
if (progressStarted) progress.finish();
// #3037: loud end-of-run summary so failures are visible even when the
// per-page stderr lines scrolled away. cli.ts turns failures>0 into a
// non-zero exit verdict.
if (result.failures > 0) {
serr(`[embed] ${result.failures} chunk(s) failed to embed. First error: ${result.failure_samples[0] ?? 'unknown'}`);
}
return result;
} catch (e) {
if (progressStarted) progress.finish();
@@ -623,10 +666,32 @@ async function embedPage(
// contextual prefix when the page was embedded wrapped), not raw
// chunk_text — otherwise a re-embed silently strips the contextual
// prefixes the sync path applied. fenced_code chunks stay unwrapped.
const embeddings = await embedBatch(wrapChunkTextsForStoredMode(page, toEmbed), { abortSignal: signal });
// #3037: per-chunk failure isolation — one bad chunk must not leave the
// page's sibling chunks NULL. The wrapped texts (computed once) feed the
// fan-out too, so an isolation retry never strips the prefixes. Total
// embed failure is recorded here (where the chunk count is known) and
// swallowed: the page stays NULL exactly as before, but the run now
// reports it (result.failures → non-zero exit) instead of pretending
// success. Abort (shutdown) still propagates.
let embeddings: (Float32Array | null)[];
let failed = 0;
let firstError: unknown;
try {
({ embeddings, failed, firstError } = await embedPageTexts(
wrapChunkTextsForStoredMode(page, toEmbed),
signal ? { abortSignal: signal } : {},
));
} catch (e: unknown) {
if (isAborted(signal)) throw e;
recordFailure(result, toEmbed.length, slug, e);
result.pages_processed++;
serr(` Error embedding ${slug}: ${e instanceof Error ? e.message : e}`);
return;
}
const embeddingMap = new Map<number, Float32Array>();
for (let j = 0; j < toEmbed.length; j++) {
embeddingMap.set(toEmbed[j].chunk_index, embeddings[j]);
const emb = embeddings[j];
if (emb) embeddingMap.set(toEmbed[j].chunk_index, emb);
}
const updated: ChunkInput[] = chunks.map(c => preserveCodeMetadata(c, {
chunk_index: c.chunk_index,
@@ -643,16 +708,21 @@ async function embedPage(
// Guard: only stamp when EVERY chunk was (re)embedded this pass. If some
// chunks were preserved from a prior embed (unknown/old provenance), the
// page is mixed — don't claim it's current. `embed --all` fully re-embeds
// such a page and then stamps it.
if (toEmbed.length === chunks.length) {
// such a page and then stamps it. #3037: a partial failure leaves failed
// chunks NULL, so don't stamp then either.
if (failed === 0 && toEmbed.length === chunks.length) {
await engine.setPageEmbeddingSignature(slug, { sourceId, signature: currentEmbeddingSignature() });
// #3507: a fully re-embedded per_chunk_synopsis page landed at the
// title tier — keep the stamped mode honest.
await restampIfDemotedToTitleTier(engine, page, slug, page.source_id);
}
result.embedded += toEmbed.length;
result.embedded += toEmbed.length - failed;
if (failed > 0) {
recordFailure(result, failed, slug, firstError);
serr(` ${slug}: ${failed} chunk(s) failed to embed; embedded the other ${toEmbed.length - failed}`);
}
result.pages_processed++;
if (!quiet) slog(`${slug}: embedded ${toEmbed.length} chunks`);
if (!quiet) slog(`${slug}: embedded ${toEmbed.length - failed} chunks`);
}
/**
@@ -791,11 +861,18 @@ async function embedAll(
try {
// #3507: reproduce the page's stored wrapping convention (see embedPage).
const embeddings = await embedBatch(wrapChunkTextsForStoredMode(page, toEmbed));
// #3037: per-chunk failure isolation — one bad chunk costs one chunk,
// not the whole page's siblings. The wrapped texts feed the fan-out
// too, so an isolation retry never strips the contextual prefixes.
const { embeddings, failed, firstError } = await embedPageTexts(
wrapChunkTextsForStoredMode(page, toEmbed),
signal ? { abortSignal: signal } : {},
);
// Build a map of new embeddings by chunk_index
const embeddingMap = new Map<number, Float32Array>();
for (let j = 0; j < toEmbed.length; j++) {
embeddingMap.set(toEmbed[j].chunk_index, embeddings[j]);
const emb = embeddings[j];
if (emb) embeddingMap.set(toEmbed[j].chunk_index, emb);
}
// Preserve ALL chunks, only update embeddings for stale ones.
// preserveCodeMetadata threads code-chunk metadata (#769) so re-embed
@@ -809,17 +886,30 @@ async function embedAll(
}));
await observed(pacer, () => engine.upsertChunks(page.slug, updated, pageOpts));
// v0.41.31: stamp embedding provenance so a later model swap is
// detectable as stale.
await observed(pacer, () =>
engine.setPageEmbeddingSignature(page.slug, { sourceId: pageSourceId, signature }),
);
// #3507: --all fully re-embeds; a per_chunk_synopsis page landed at
// the title tier — keep the stamped mode honest.
await observed(pacer, () =>
restampIfDemotedToTitleTier(engine, page, page.slug, pageSourceId),
);
result.embedded += toEmbed.length;
// detectable as stale. #3037: not on partial failure — failed chunks
// stay NULL under unknown provenance.
if (failed === 0) {
await observed(pacer, () =>
engine.setPageEmbeddingSignature(page.slug, { sourceId: pageSourceId, signature }),
);
// #3507: --all fully re-embeds; a per_chunk_synopsis page landed at
// the title tier — keep the stamped mode honest. #3037: gated on
// failed === 0 — a partially-failed page was NOT fully re-embedded,
// so restamping would make contextual_retrieval_mode lie again
// (the exact #3461 bug).
await observed(pacer, () =>
restampIfDemotedToTitleTier(engine, page, page.slug, pageSourceId),
);
}
result.embedded += toEmbed.length - failed;
if (failed > 0) {
recordFailure(result, failed, page.slug, firstError);
serr(`\n ${page.slug}: ${failed} chunk(s) failed to embed; embedded the other ${toEmbed.length - failed}`);
}
} catch (e: unknown) {
// #3037: count the darkened page so the run can't exit 0 (abort is a
// shutdown, not a failure).
if (!isAborted(signal)) recordFailure(result, toEmbed.length, page.slug, e);
serr(`\n Error embedding ${page.slug}: ${e instanceof Error ? e.message : e}`);
}
@@ -1037,10 +1127,8 @@ async function embedAllStale(
let afterUpdatedAt: string | null = null;
let totalChunksLoaded = 0;
let budgetExitNotified = false;
// #1946 (OV2a): track chunks that errored out so a catch-up pass that finishes
// with stale chunks still remaining (un-embeddable for a non-transient reason)
// surfaces that loudly instead of looking like a clean run.
let embedFailures = 0;
// #1946 (OV2a) + #3037: embed failures are tracked on result.failures so
// the catch-up warning below AND the CLI exit verdict both see them.
// E-3 (paced-backfill): bounded end-of-run re-entry. A longer paced run gives
// a live writer (sync / put_page) more time to insert NEW stale rows BEHIND
@@ -1137,12 +1225,19 @@ async function embedAllStale(
// NORMAL post-model-migration path, so raw-text embedding here
// quietly converted whole corpora to the unwrapped convention.
const pageRow = await observed(pacer, () => engine.getPage(slug, { sourceId: keySourceId }));
const embeddings = await embedBatchWithBackoff(wrapChunkTextsForStoredMode(pageRow, stale), { abortSignal: effectiveSignal });
// #3037: per-chunk failure isolation — one bad chunk costs one
// chunk, not the whole page's siblings. The wrapped texts feed the
// fan-out too, so an isolation retry never strips the prefixes.
const { embeddings, failed, firstError } = await embedPageTexts(
wrapChunkTextsForStoredMode(pageRow, stale),
{ abortSignal: effectiveSignal },
);
// Re-fetch existing chunks and merge to avoid deleting non-stale chunks.
const existing = await observed(pacer, () => engine.getChunks(slug, { sourceId: keySourceId }));
const staleIdxToEmbedding = new Map<number, Float32Array>();
for (let j = 0; j < stale.length; j++) {
staleIdxToEmbedding.set(stale[j].chunk_index, embeddings[j]);
const emb = embeddings[j];
if (emb) staleIdxToEmbedding.set(stale[j].chunk_index, emb);
}
// preserveCodeMetadata threads code-chunk metadata (#769) so the
// autopilot --stale path doesn't clobber language/symbol_name/etc
@@ -1160,7 +1255,8 @@ async function embedAllStale(
// A partially-stale page keeps preserved chunks of unknown/old
// provenance, so don't claim it's current. (After invalidate, a
// signature-drifted page IS fully stale → this stamps it.)
if (signature && stale.length === existing.length) {
// #3037: not on partial failure — failed chunks stay NULL.
if (signature && failed === 0 && stale.length === existing.length) {
await observed(pacer, () =>
engine.setPageEmbeddingSignature(slug, { sourceId: keySourceId, signature }),
);
@@ -1168,17 +1264,24 @@ async function embedAllStale(
// #3507: a FULLY re-embedded per_chunk_synopsis page landed at the
// title tier — keep the stamped mode honest. Partially-stale pages
// stay stamped as-is (mixed provenance; reindex sweeps fix them).
if (stale.length === existing.length) {
// #3037: `failed === 0` is part of "fully re-embedded" — if the
// per-chunk isolation left some chunks NULL, restamping would make
// contextual_retrieval_mode lie again (the exact #3461 bug).
if (failed === 0 && stale.length === existing.length) {
await observed(pacer, () =>
restampIfDemotedToTitleTier(engine, pageRow, slug, keySourceId),
);
}
result.embedded += stale.length;
result.embedded += stale.length - failed;
if (failed > 0) {
recordFailure(result, failed, slug, firstError);
serr(`\n ${slug}: ${failed} chunk(s) failed to embed; embedded the other ${stale.length - failed}`);
}
} catch (e: unknown) {
// Budget/abort-fired cancellations are expected on the way out; don't
// spam per-page "Error embedding" lines when we're shutting down.
if (effectiveSignal.aborted) return;
embedFailures++;
recordFailure(result, stale.length, slug, e);
serr(`\n Error embedding ${slug}: ${e instanceof Error ? e.message : e}`);
}
totalProcessedPages++;
@@ -1231,14 +1334,14 @@ async function embedAllStale(
// chunks unembedded means those chunks are stuck (a non-transient embed
// failure), not that we ran out of time. Surface it loudly so it doesn't read
// as a clean run — re-running won't help until the underlying failure is fixed.
if (staleOpts?.catchUp && !effectiveSignal.aborted && embedFailures > 0) {
if (staleOpts?.catchUp && !effectiveSignal.aborted && result.failures > 0) {
const remaining = await engine.countStaleChunks(
signature
? { signature, ...(sourceId ? { sourceId } : {}), ...(includeNullSig && { includeNullSignature: true }) }
: (sourceId ? { sourceId } : undefined),
);
if (remaining > 0) {
serr(`\n [embed] catch-up finished but ${remaining} chunk(s) remain stale after ${embedFailures} embed failure(s). These are not embeddable as-is; re-running won't clear them until the underlying error is resolved.`);
serr(`\n [embed] catch-up finished but ${remaining} chunk(s) remain stale after ${result.failures} embed failure(s). These are not embeddable as-is; re-running won't clear them until the underlying error is resolved.`);
}
}
}
@@ -1358,12 +1461,7 @@ export async function embedBatchWithBackoff(
// If the budget fired we may have been aborted mid-fetch; bubble out.
if (signal?.aborted) throw e;
const msg = e instanceof Error ? e.message : String(e);
// D4: structured detection first (handles gateway-wrapped errors via
// cause chain); message-match as fallback for providers whose wrappers
// strip `cause.status`.
const isRateLimit = detect429FromCause(e)
|| /rate.?limit|429/i.test(msg);
if (!isRateLimit || attempt === MAX_RATE_LIMIT_RETRIES) throw e;
if (!isRateLimitError(e) || attempt === MAX_RATE_LIMIT_RETRIES) throw e;
const delayMs = parseRetryDelayMs(msg);
serr(` [rate-limit] attempt ${attempt + 1}/${MAX_RATE_LIMIT_RETRIES}, waiting ${delayMs}ms...`);
@@ -1373,3 +1471,86 @@ export async function embedBatchWithBackoff(
// Unreachable, but TypeScript needs it.
return embedBatch(texts);
}
/**
* 429 judgment shared by embedBatchWithBackoff (retry decision) and
* embedPageTexts (fan-out decision). D4: structured detection first
* (gateway-wrapped errors via cause chain); message-match as fallback for
* providers whose wrappers strip `cause.status`.
*/
function isRateLimitError(e: unknown): boolean {
const msg = e instanceof Error ? e.message : String(e);
return detect429FromCause(e) || /rate.?limit|429/i.test(msg);
}
/** Walk the cause chain (like detect429FromCause) for the first HTTP status. */
function statusFromCause(e: unknown): number | undefined {
let cur: unknown = e;
for (let depth = 0; depth < 5 && cur !== undefined && cur !== null; depth++) {
const obj = cur as { status?: unknown; statusCode?: unknown; cause?: unknown };
if (typeof obj.status === 'number') return obj.status;
if (typeof obj.statusCode === 'number') return obj.statusCode;
cur = obj.cause;
}
return undefined;
}
/**
* #3037: embed one page's chunk texts with per-chunk failure isolation.
*
* All three embed paths used to send a page's chunks in ONE
* embedBatch call, so one bad chunk (e.g. an oversized chunk the provider
* 400s) left EVERY sibling chunk NULL an ~8.6x blast radius. This wrapper
* tries the batch first (the cheap, common path), and only on a
* PERMANENT-looking batch failure retries once per chunk so one bad chunk
* costs one chunk.
*
* Cost bounding when we do NOT fan out (rethrow instead):
* - 429 / rate limit: embedBatchWithBackoff already retried with backoff;
* fanning out N single-chunk calls would hammer the same limiter N-fold.
* - AITransientError (5xx / network / unknown, per normalizeAIError): the
* batch CONTENT isn't the problem, so isolation can't help during an
* outage it would just multiply failing calls per page.
* - 401/403 (auth): nothing chunk-specific; every call would fail.
* When we DO fan out (permanent request-shaped 4xx like 400/413/422), the
* per-chunk pass happens at most ONCE per page per run and re-spends roughly
* the same tokens the failed batch would have bounded, no recursion. A
* fresh 429 arising DURING the fan-out still gets the normal backoff (each
* single-chunk call goes through embedBatchWithBackoff).
*
* Throws when nothing could be embedded (total failure same contract as
* the pre-#3037 single batch call). Returns `null` at the index of each
* failed chunk otherwise.
*/
async function embedPageTexts(
texts: string[],
opts: EmbedBatchWithBackoffOpts = {},
): Promise<{ embeddings: (Float32Array | null)[]; failed: number; firstError?: unknown }> {
try {
return { embeddings: await embedBatchWithBackoff(texts, opts), failed: 0 };
} catch (e: unknown) {
if (opts.abortSignal?.aborted) throw e; // shutdown, not a chunk problem
if (texts.length <= 1) throw e; // nothing to isolate
if (isRateLimitError(e) || e instanceof AITransientError) throw e;
const status = statusFromCause(e);
if (status === 401 || status === 403) throw e;
const embeddings: (Float32Array | null)[] = [];
let failed = 0;
let firstError: unknown;
for (const t of texts) {
try {
const single = await embedBatchWithBackoff([t], opts);
embeddings.push(single[0] ?? null);
if (single[0] === undefined) { failed++; firstError ??= e; }
} catch (chunkErr: unknown) {
if (opts.abortSignal?.aborted) throw chunkErr;
embeddings.push(null);
failed++;
firstError ??= chunkErr;
}
}
if (failed === texts.length) throw firstError ?? e; // total failure: pre-#3037 contract
return { embeddings, failed, firstError };
}
}
+1 -1
View File
@@ -78,7 +78,7 @@ FLAGS:
cycle is 3 model calls; verdict aggregates over them.
--slot-a-model <id> Override default 'openai:gpt-5.2'.
--slot-b-model <id> Override default 'anthropic:claude-opus-4-7'.
--slot-c-model <id> Override default 'google:gemini-1.5-pro'.
--slot-c-model <id> Override default 'deepseek:deepseek-v4-pro'.
--receipt-dir <path> Default: gbrainPath('eval-receipts').
--max-tokens N Output token budget per call. Default: 4000.
--json Emit final aggregate as JSON to stdout (progress to stderr).
+39 -10
View File
@@ -43,7 +43,7 @@ import {
} from '../core/link-extraction.ts';
import { createProgress } from '../core/progress.ts';
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
import { pathToSlug, pruneDir, isSyncable } from '../core/sync.ts';
import { pathToSlug, slugifyPath, pruneDir, isSyncable } from '../core/sync.ts';
// v0.41.18.0: withRetry + isRetryableConnError + WithRetryOpts moved to
// src/core/retry.ts as the canonical primitive. Engine methods
// (addLinksBatch/addTimelineEntriesBatch/upsertChunks) now self-retry via
@@ -269,14 +269,24 @@ export function extractMarkdownLinks(content: string): { name: string; relTarget
export function resolveSlug(fileDir: string, relTarget: string, allSlugs: Set<string>): string | null {
const targetNoExt = relTarget.endsWith('.md') ? relTarget.slice(0, -3) : relTarget;
const s1 = join(fileDir, targetNoExt);
if (allSlugs.has(s1)) return s1;
// Issue #1964: wikilinks carry raw Obsidian paths (`[[llm-wiki/entities/AI 3.0]]`)
// but allSlugs holds sync-slugified slugs (`llm-wiki/entities/ai-3.0`). Try the
// raw candidate first (back-compat), then the sync-consistent slugified form.
const hit = (candidate: string): string | null => {
if (allSlugs.has(candidate)) return candidate;
const slugified = slugifyPath(candidate);
if (slugified !== candidate && allSlugs.has(slugified)) return slugified;
return null;
};
const s1 = hit(join(fileDir, targetNoExt));
if (s1) return s1;
const parts = fileDir.split('/').filter(Boolean);
for (let strip = 1; strip <= parts.length; strip++) {
const ancestor = parts.slice(0, parts.length - strip).join('/');
const candidate = ancestor ? join(ancestor, targetNoExt) : targetNoExt;
if (allSlugs.has(candidate)) return candidate;
const candidate = hit(ancestor ? join(ancestor, targetNoExt) : targetNoExt);
if (candidate) return candidate;
}
return null;
@@ -349,7 +359,13 @@ function inferTypeByDir(fromDir: string, toDir: string, frontmatter?: Record<str
const to = toDir.split('/')[0];
if (from === 'people' && to === 'companies') {
if (Array.isArray(frontmatter?.founded)) return 'founded';
return 'works_at';
// #3466: bare people/ -> companies/ adjacency is not evidence of
// employment, so it gets the neutral 'mentions' verb instead of
// 'works_at'. Real works_at edges still come from the two paths that
// read actual evidence: the company:/companies: frontmatter fields
// (FRONTMATTER_LINK_MAP) and employment phrasing in prose
// (inferLinkType in link-extraction.ts).
return 'mentions';
}
if (from === 'people' && to === 'deals') return 'involved_in';
if (from === 'deals' && to === 'companies') return 'deal_for';
@@ -1452,6 +1468,8 @@ async function extractLinksFromDB(
slugToSources.set(ref.slug, list);
}
let processed = 0, created = 0;
// #2576: skipped-candidate counter — see extractStaleFromDB's twin.
let skippedMissingTarget = 0;
// v0.42.7 (#1696): pages whose links we extracted this run — stamped after
// the loop so a manual `gbrain extract links|all --source db` clears the
// links_extraction_lag doctor signal. Non-dry-run only.
@@ -1508,7 +1526,7 @@ async function extractLinksFromDB(
// endpoint-validation + from/to source-id picking (null = skip: missing
// endpoint OR target only in a non-origin/non-default source).
const resolved = resolveCandidateSources(c, slug, source_id, allSlugs, slugToSources);
if (!resolved) continue;
if (!resolved) { skippedMissingTarget++; continue; }
const { fromSlug, fromSourceId, toSourceId } = resolved;
if (dryRunSeen) {
@@ -1565,6 +1583,9 @@ async function extractLinksFromDB(
if (!jsonMode) {
const label = dryRun ? '(dry run) would create' : 'created';
console.log(`Links: ${label} ${created} from ${processed} pages (db source)`);
if (skippedMissingTarget > 0) {
console.log(`Skipped ${skippedMissingTarget} candidate(s) whose target page doesn't exist (references to non-pages are never persisted).`);
}
if (includeFrontmatter && unresolved.length > 0) {
// Top-20 preview of unresolvable frontmatter names so the user can
// see where the graph has holes (codex tension 6.4).
@@ -1710,7 +1731,7 @@ export async function extractStaleFromDB(
sourceIdFilter?: string;
catchUp: boolean;
},
): Promise<{ linksCreated: number; timelineCreated: number; pagesProcessed: number; staleRemaining: number }> {
): Promise<{ linksCreated: number; timelineCreated: number; pagesProcessed: number; staleRemaining: number; skippedMissingTarget?: number }> {
const { dryRun, jsonMode, includeFrontmatter, sourceIdFilter, catchUp } = opts;
const versionTs = LINK_EXTRACTOR_VERSION_TS;
@@ -1762,6 +1783,10 @@ export async function extractStaleFromDB(
let afterPageId = 0;
let linksCreated = 0, timelineCreated = 0, pagesProcessed = 0;
let budgetHit = false;
// #2576: candidates whose endpoint pages don't exist are skipped, not
// persisted. Counted so a dropped reference is observable in the summary
// instead of vanishing silently (the failure mode that hid bug 2).
let skippedMissingTarget = 0;
for (;;) {
const rows = await engine.listStalePagesForExtraction({
@@ -1781,7 +1806,7 @@ export async function extractStaleFromDB(
);
for (const c of extracted.candidates) {
const r = resolveCandidateSources(c, page.slug, page.source_id, allSlugs, slugToSources);
if (!r) continue;
if (!r) { skippedMissingTarget++; continue; }
linkRows.push({
from_slug: r.fromSlug, to_slug: c.targetSlug, link_type: c.linkType,
context: c.context, link_source: c.linkSource, origin_slug: c.originSlug,
@@ -1842,6 +1867,9 @@ export async function extractStaleFromDB(
if (!jsonMode) {
console.log(`Extract --stale: ${linksCreated} link(s) + ${timelineCreated} timeline entr(ies) from ${pagesProcessed} page(s).`);
if (skippedMissingTarget > 0) {
console.log(`Skipped ${skippedMissingTarget} candidate(s) whose target page doesn't exist (references to non-pages are never persisted).`);
}
if (budgetHit && staleRemaining > 0) {
console.log(`Time budget reached — ${staleRemaining} page(s) still stale. Re-run 'gbrain extract --stale' (or pass --catch-up) to continue.`);
}
@@ -1849,9 +1877,10 @@ export async function extractStaleFromDB(
process.stdout.write(JSON.stringify({
action: 'extract_stale_done', links_created: linksCreated, timeline_created: timelineCreated,
pages_processed: pagesProcessed, stale_remaining: staleRemaining, budget_hit: budgetHit,
skipped_missing_target: skippedMissingTarget,
}) + '\n');
}
return { linksCreated, timelineCreated, pagesProcessed, staleRemaining };
return { linksCreated, timelineCreated, pagesProcessed, staleRemaining, skippedMissingTarget };
}
/**
+1 -1
View File
@@ -42,7 +42,7 @@ interface FeatureScanResult {
const RECIPE_META = [
{ id: 'email-to-brain', name: 'Email to Brain', secrets: ['GMAIL_APP_PASSWORD'] },
{ id: 'calendar-to-brain', name: 'Calendar Sync', secrets: ['GOOGLE_CALENDAR_API_KEY'] },
{ id: 'x-to-brain', name: 'X/Twitter to Brain', secrets: ['X_BEARER_TOKEN'] },
{ id: 'x-to-brain', name: 'X/Twitter to Brain', secrets: ['X_API_BEARER_TOKEN'] },
{ id: 'twilio-voice-brain', name: 'Voice to Brain', secrets: ['TWILIO_AUTH_TOKEN'] },
{ id: 'meeting-sync', name: 'Meeting Sync', secrets: ['CIRCLEBACK_API_KEY'] },
{ id: 'credential-gateway', name: 'Credential Gateway', secrets: ['OAUTH_CLIENT_SECRET'] },
+10 -2
View File
@@ -16,7 +16,7 @@ interface FileRecord {
filename: string;
storage_path: string;
mime_type: string | null;
size_bytes: number;
size_bytes: number | bigint | string | null;
content_hash: string;
metadata: Record<string, unknown>;
created_at: string;
@@ -42,6 +42,14 @@ function fileHash(filePath: string): string {
return createHash('sha256').update(content).digest('hex');
}
export function formatFileSizeKb(rawSizeBytes: number | bigint | string | null): string {
if (rawSizeBytes == null) return '?';
const sizeBytes = Number(rawSizeBytes);
return Number.isFinite(sizeBytes) && sizeBytes >= 0
? `${Math.round(sizeBytes / 1024)}KB`
: '?';
}
export async function runFiles(engine: BrainEngine, args: string[]) {
const subcommand = args[0];
@@ -116,7 +124,7 @@ async function listFiles(engine: BrainEngine, slug?: string) {
console.log(`${rows.length} file(s):`);
for (const row of rows) {
const size = row.size_bytes ? `${Math.round(Number(row.size_bytes) / 1024)}KB` : '?';
const size = formatFileSizeKb(row.size_bytes as FileRecord['size_bytes']);
console.log(` ${row.page_slug || '(unlinked)'} / ${row.filename} [${size}, ${row.mime_type || '?'}]`);
}
}
+57 -6
View File
@@ -78,6 +78,17 @@ export async function runImport(
const jsonOutput = args.includes('--json');
const includeGitignored = args.includes('--include-gitignored') || opts.includeGitignored === true;
// #3637: under --json, stdout belongs to the JSON document alone. The
// informational lines below are useful — they just belong on the other
// channel, the same rule progress already follows (CLAUDE.md: "Progress
// always writes to stderr. Stdout stays clean for data output (--json
// payloads)"). Pre-fix, `import --json` prefixed the payload with
// "Found N markdown files", so JSON.parse of stdout failed outright.
const info = (msg: string): void => {
if (jsonOutput) console.error(msg);
else console.log(msg);
};
// T7 (D9): refuse cleanly when init persisted the deferred-setup sentinel,
// unless the user is explicitly skipping embedding via `--no-embed` (in
// which case the chunks land without vectors and the user can backfill
@@ -225,7 +236,7 @@ export async function runImport(
if (opts.exclude && opts.exclude.length > 0) {
const beforeExclude = allFiles.length;
allFiles = allFiles.filter(abs => !matchesAnyGlob(relative(dir, abs), opts.exclude));
console.log(
info(
`Found ${allFiles.length} ${fileTypeLabel} files ` +
`(${beforeExclude - allFiles.length} excluded by --exclude patterns)`,
);
@@ -237,7 +248,7 @@ export async function runImport(
);
}
} else {
console.log(`Found ${allFiles.length} ${fileTypeLabel} files`);
info(`Found ${allFiles.length} ${fileTypeLabel} files`);
}
// Sort newest-first so date-prefixed brain paths get embedded before older ones.
@@ -253,7 +264,7 @@ export async function runImport(
const cp = loadCheckpoint(checkpointPath, dir);
if (cp) {
for (const p of cp.completedPaths) completed.add(p);
console.log(`Resuming from checkpoint: skipping ${completed.size} already-processed files`);
info(`Resuming from checkpoint: skipping ${completed.size} already-processed files`);
}
}
const files = resumeFilter(allFiles, dir, completed);
@@ -261,13 +272,19 @@ export async function runImport(
// Determine actual worker count
const actualWorkers = workerCount > 1 ? workerCount : 1;
if (actualWorkers > 1) {
console.log(`Using ${actualWorkers} parallel workers`);
info(`Using ${actualWorkers} parallel workers`);
}
let imported = 0;
let skipped = 0;
let errors = 0;
let processed = 0;
// Time-based checkpoint floor (see the save site below). Chunking cost scales
// with paragraph count, not bytes, so a single reference-style file can take
// many minutes; a count-only trigger leaves that work undurable.
const CHECKPOINT_MAX_INTERVAL_MS = 120_000;
let lastCheckpointMs = Date.now();
let lastCheckpointSize = completed.size;
let chunksCreated = 0;
const importedSlugs: string[] = [];
const errorCounts: Record<string, number> = {};
@@ -343,7 +360,17 @@ export async function runImport(
// Save checkpoint every 100 SUCCESSFUL adds (not every 100 processed).
// Failed files never enter `completed`, so a flaky file can't push the
// checkpoint past it — the next run will retry it.
if (completed.size > 0 && completed.size % 100 === 0) {
// ...and ALSO save on a time interval. On a corpus with an expensive tail
// `completed` can advance ~1 file per several minutes, so the next
// 100-boundary may be hours away; any kill before it discards every file
// since the last boundary and the run can never converge.
const nowMs = Date.now();
const dueByCount = completed.size > 0 && completed.size % 100 === 0;
const dueByTime = completed.size > lastCheckpointSize
&& nowMs - lastCheckpointMs >= CHECKPOINT_MAX_INTERVAL_MS;
if (dueByCount || dueByTime) {
lastCheckpointMs = nowMs;
lastCheckpointSize = completed.size;
const cpDir = gbrainPath();
if (!existsSync(cpDir)) {
try { const { mkdirSync } = await import('fs'); mkdirSync(cpDir, { recursive: true }); }
@@ -429,13 +456,37 @@ export async function runImport(
}
}
// Final checkpoint save BEFORE the clear/preserve decision below. The
// periodic triggers above are gated on a 100-file boundary or an interval,
// so a run that ends between them would otherwise leave its tail unsaved.
// This must run before clearCheckpoint() so a clean run still ends with no
// checkpoint file — it only makes the ERROR path's preserved checkpoint
// complete.
if (errors > 0 && completed.size > lastCheckpointSize) {
try {
const cpDir = gbrainPath();
if (!existsSync(cpDir)) {
const { mkdirSync } = await import('fs');
mkdirSync(cpDir, { recursive: true });
}
saveCheckpoint(checkpointPath, {
schema_version: 1,
owner: 'gbrain',
kind: 'import',
dir,
completedPaths: Array.from(completed),
timestamp: new Date().toISOString(),
});
} catch { /* non-fatal: the next run simply redoes the tail */ }
}
// Clear checkpoint on clean completion. On error, the path-based checkpoint
// preserves only the successfully-completed paths, so the next run retries
// failed files automatically (they never entered `completed`).
if (errors === 0) {
clearCheckpoint(checkpointPath);
} else if (existsSync(checkpointPath)) {
console.log(` Checkpoint preserved (${errors} errors). Run again to retry failed files.`);
info(` Checkpoint preserved (${errors} errors). Run again to retry failed files.`);
}
const totalTime = ((Date.now() - startTime) / 1000).toFixed(1);
+113 -8
View File
@@ -23,7 +23,8 @@ import matter from 'gray-matter';
import { readFileSync, existsSync, writeFileSync, mkdirSync, readdirSync } from 'fs';
import { join, basename } from 'path';
import { homedir } from 'os';
import { gbrainPath } from '../core/config.ts';
import { gbrainPath, loadConfig } from '../core/config.ts';
import { buildGatewayConfig } from '../core/ai/build-gateway-config.ts';
import { execSync } from 'child_process';
// --- Types ---
@@ -54,6 +55,13 @@ interface RecipeFrontmatter {
health_checks: HealthCheck[];
setup_time: string;
cost_estimate?: string;
/**
* Repo-relative dirs (slug prefixes, trailing '/') this recipe's collector
* writes files to. Ground truth for the `db_only_collector_collision`
* doctor check (issue #2788): output inside a db_only path is silently
* skipped by sync and import (auto-gitignored).
*/
output_paths: string[];
}
interface ParsedRecipe {
@@ -105,7 +113,20 @@ interface AnyOfCheck {
checks: HealthCheck[];
}
type HealthCheck = string | HttpCheck | EnvExistsCheck | CommandCheck | AnyOfCheck;
/**
* Staleness-aware check type (issue #2787, reported by @alexputici). All
* other types are point-in-time a sense whose gateway is up and env vars
* are set passes forever even when zero data flows. This one reads the
* integration's heartbeat file and FAILS when the newest event is older
* than the declared cadence (`max_age`, e.g. "48h", "2d", "90m").
*/
interface HeartbeatMaxAgeCheck {
type: 'heartbeat_max_age';
max_age: string;
label?: string;
}
type HealthCheck = string | HttpCheck | EnvExistsCheck | CommandCheck | AnyOfCheck | HeartbeatMaxAgeCheck;
interface CheckResult {
integration: string;
@@ -122,9 +143,48 @@ export function isUnsafeHealthCheck(check: string): boolean {
return /[;&|`$(){}\\<>\n]/.test(check);
}
/** Expand $VAR references with process.env values */
/**
* Env view for secret resolution (#2789): apply the same config.jsonenv
* folding the runtime applies via buildGatewayConfig, so a credential stored
* only in ~/.gbrain/config.json which powers a perfectly healthy
* integration is not reported [missing] by show/status. process.env still
* wins for non-empty values (buildGatewayConfig spreads it last, dropping
* only ''/undefined entries). Falls back to bare process.env before
* `gbrain init` (no config file yet). Mirrors the #2728 fix on the
* providers command.
*/
export function secretEnv(): Record<string, string | undefined> {
try {
const cfg = loadConfig();
if (cfg) return buildGatewayConfig(cfg).env;
} catch { /* integrations must keep working pre-init — fall through */ }
return process.env;
}
/**
* Parse a heartbeat_max_age duration string ("30s", "90m", "48h", "2d")
* into milliseconds. Returns null on anything unparseable.
*/
export function parseMaxAge(s: string): number | null {
const m = /^(\d+(?:\.\d+)?)\s*(s|m|h|d)$/i.exec(String(s).trim());
if (!m) return null;
const n = Number(m[1]);
if (!Number.isFinite(n) || n <= 0) return null;
const unit = { s: 1_000, m: 60_000, h: 3_600_000, d: 86_400_000 }[m[2].toLowerCase() as 's' | 'm' | 'h' | 'd'];
return n * unit;
}
/** Human-readable age for heartbeat_max_age output ("3d", "17h", "42m"). */
function formatAge(ms: number): string {
if (ms >= 86_400_000) return `${Math.floor(ms / 86_400_000)}d`;
if (ms >= 3_600_000) return `${Math.floor(ms / 3_600_000)}h`;
return `${Math.max(0, Math.floor(ms / 60_000))}m`;
}
/** Expand $VAR references with gateway-env (config-folded) values */
export function expandVars(s: string): string {
return s.replace(/\$([A-Z_][A-Z0-9_]*)/g, (_, name) => process.env[name] || '');
const env = secretEnv();
return s.replace(/\$([A-Z_][A-Z0-9_]*)/g, (_, name) => env[name] || '');
}
// --- SSRF Protection ---
@@ -249,7 +309,7 @@ export async function executeHealthCheck(
}
case 'env_exists': {
const val = process.env[check.name];
const val = secretEnv()[check.name];
return {
...base,
status: val ? 'ok' : 'fail',
@@ -279,6 +339,29 @@ export async function executeHealthCheck(
}
}
case 'heartbeat_max_age': {
// No embedded gate: reads only the local heartbeat file — no exec, no
// network. Safe for user-provided recipes.
const maxMs = parseMaxAge(check.max_age);
if (maxMs === null) {
return { ...base, status: 'fail', output: `${check.label || 'heartbeat_max_age'}: invalid max_age '${check.max_age}' (use e.g. 90m, 48h, 2d)` };
}
const entries = readHeartbeat(integrationId);
if (entries.length === 0) {
return { ...base, status: 'fail', output: `${check.label || 'heartbeat'}: no heartbeat events in the last 30 days (expected activity within ${check.max_age}) — the sense has stopped producing data` };
}
let newest = 0;
for (const e of entries) {
const t = new Date(e.ts).getTime();
if (Number.isFinite(t) && t > newest) newest = t;
}
const ageMs = Date.now() - newest;
if (ageMs > maxMs) {
return { ...base, status: 'fail', output: `${check.label || 'heartbeat'}: last event ${formatAge(ageMs)} ago exceeds max_age ${check.max_age} — the sense has stopped producing data` };
}
return { ...base, status: 'ok', output: `${check.label || 'heartbeat'}: last event ${formatAge(ageMs)} ago (within ${check.max_age})` };
}
case 'any_of': {
for (const sub of check.checks) {
const result = await executeHealthCheck(sub, integrationId, isEmbedded);
@@ -320,6 +403,7 @@ export function parseRecipe(content: string, filename: string): ParsedRecipe | n
health_checks: (data.health_checks || []) as HealthCheck[],
setup_time: data.setup_time || 'unknown',
cost_estimate: data.cost_estimate,
output_paths: Array.isArray(data.output_paths) ? data.output_paths.map(String) : [],
},
body: body.trim(),
filename,
@@ -383,6 +467,25 @@ function loadAllRecipes(): ParsedRecipe[] {
return recipes;
}
/**
* Output paths of every CONFIGURED recipe (secrets present the collector
* can actually be running). Ground truth for the
* `db_only_collector_collision` doctor check and the sync-time warning
* (issue #2788). Unconfigured recipes are skipped: a collector that can't
* run can't silently die.
*/
export function getConfiguredCollectorOutputs(): Array<{ id: string; output_path: string }> {
const out: Array<{ id: string; output_path: string }> = [];
for (const r of loadAllRecipes()) {
if (r.frontmatter.output_paths.length === 0) continue;
if (getStatus(r) === 'available') continue;
for (const p of r.frontmatter.output_paths) {
out.push({ id: r.frontmatter.id, output_path: p });
}
}
return out;
}
function findRecipe(id: string): ParsedRecipe | null {
const recipes = loadAllRecipes();
const exact = recipes.find(r => r.frontmatter.id === id);
@@ -457,11 +560,12 @@ function readHeartbeat(id: string): HeartbeatEntry[] {
// --- Secret Checking ---
function checkSecrets(secrets: RecipeSecret[]): { set: string[]; missing: RecipeSecret[] } {
export function checkSecrets(secrets: RecipeSecret[]): { set: string[]; missing: RecipeSecret[] } {
const set: string[] = [];
const missing: RecipeSecret[] = [];
const env = secretEnv();
for (const s of secrets) {
if (process.env[s.name]) {
if (env[s.name]) {
set.push(s.name);
} else {
missing.push(s);
@@ -607,8 +711,9 @@ function cmdShow(args: string[]): void {
if (f.requires.length > 0) console.log(`Requires: ${f.requires.join(', ')}`);
console.log('\nSecrets needed:');
const env = secretEnv();
for (const s of f.secrets) {
const isSet = process.env[s.name] ? ' [set]' : ' [missing]';
const isSet = env[s.name] ? ' [set]' : ' [missing]';
console.log(` ${s.name}${isSet}`);
console.log(` ${s.description}`);
console.log(` Get it: ${s.where}`);
+10 -2
View File
@@ -12,7 +12,7 @@
* Subcommands:
* gbrain integrity check Read-only report to stdout
* gbrain integrity auto Three-bucket repair with confidence
* gbrain integrity --dry-run Same as auto, no writes
* gbrain integrity auto --dry-run Same as auto, no writes
*
* Three-bucket confidence (contract with x_handle_to_tweet resolver):
* >= 0.8 auto-repair through BrainWriter transaction
@@ -449,6 +449,7 @@ async function cmdAuto(args: string[]): Promise<void> {
let bucketReview = 0;
let bucketSkip = 0;
let bucketErr = 0;
let bucketDeadLink = 0;
let pagesProcessed = 0;
const { createProgress } = await import('../core/progress.ts');
@@ -549,7 +550,13 @@ async function cmdAuto(args: string[]): Promise<void> {
hit: { slug, line: hit.line, rawLine: hit.url, phrase: 'dead-link' },
reason: `dead link: ${result.value.reason ?? 'unknown'}`,
});
bucketReview++;
// Dead links have no confidence score and are never written to
// the review file (appendReview() is only called from the
// bare-tweet path above) — they land in the skip log via
// logSkip() a few lines up. Count them separately so the
// printed "Review queue" total only ever reflects what's
// actually in ~/.gbrain/integrity-review.md.
bucketDeadLink++;
}
} catch {
/* transient; don't fail the run */
@@ -568,6 +575,7 @@ async function cmdAuto(args: string[]): Promise<void> {
console.log(`Review queue (≥${reviewLower} <${confidenceThreshold}): ${bucketReview}`);
console.log(`Skipped (<${reviewLower}): ${bucketSkip}`);
if (bucketErr > 0) console.log(`Resolver errors: ${bucketErr}`);
if (bucketDeadLink > 0) console.log(`Dead links surfaced (see skipped log): ${bucketDeadLink}`);
console.log(`\nReview queue: ${getReviewFile()}`);
console.log(`Skipped log: ${getLogFile()}`);
console.log(`Progress: ${getProgressFile()}`);
+8 -3
View File
@@ -2194,8 +2194,9 @@ export async function registerBuiltinHandlers(
// migration that retypes 25K+ pages, creates alias rows, converts edge-
// shaped pages to link rows, AND flips the active pack at end of run.
// manual_only via src/core/onboard/render.ts:MANUAL_ONLY_PROTECTED_JOBS.
// Operator path: `gbrain jobs submit unify-types --allow-protected --params
// '{"target_pack":"gbrain-base-v2"}'`.
// Dry-run preview: `gbrain jobs submit unify-types --allow-protected
// --params '{"target_pack":"gbrain-base-v2"}'`; apply with
// '{"target_pack":"gbrain-base-v2","apply":true}'.
worker.register('unify-types', async (job) => {
const { runUnifyTypes } = await import('../core/schema-pack/unify-types-handler.ts');
const data = (job.data ?? {}) as {
@@ -2213,7 +2214,11 @@ export async function registerBuiltinHandlers(
} as unknown as import('../core/operations.ts').OperationContext;
return await runUnifyTypes(ctx, {
target_pack: data.target_pack,
apply: data.apply ?? true,
// #1575: default matches the handler interface's "Default false
// (dry-run)" — a destructive one-shot migration must be opted into
// with apply:true (the onboard remediation + the printed migration
// command both carry it explicitly).
apply: data.apply ?? false,
sourceId: data.sourceId,
onProgress: (msg: string) => {
job.updateProgress({ phase: 'unify-types', message: msg }).catch(() => {});

Some files were not shown because too many files have changed in this diff Show More