Compare commits

...
Author SHA1 Message Date
Garry TanandClaude Opus 5 dc0504785a fix(check-update): resolve the latest version from VERSION, not the empty releases API (#486)
The repo publishes zero GitHub releases, so releases/latest is a permanent
404 and fetchLatestRelease() could never succeed — the entire upgrade
notification subsystem was a silent no-op, and refreshUpdateCache() cached
a fabricated up_to_date marker on every failure.

- Resolve the latest version from raw.githubusercontent.com/.../master/VERSION
  (same trusted host fetchChangelog already uses). Bounded, shape-gated parse;
  handles legacy 3-segment and -suffix channel forms.
- Discriminate network_error from no_releases in the --json error field and
  human output.
- Never write up_to_date on a failed check: preserve the last-known-good
  marker (mtime bump keeps the TTL throttle) or write nothing.
- Rejected the issue's proposed npm fallback: the gbrain npm package is an
  unrelated GPU library (#505) and would produce false upgrade prompts.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-28 13:02:33 -07:00
cybernaut6404 6920744dd8 fix(ai): enable OpenRouter query expansion (#3499) 2026-07-28 11:56:06 -07:00
daragao3andClaude Opus 5 3df20f9f18 v0.42.67.0 fix(build): force LF for shell scripts and route package.json checks through bash (#3506)
* v0.42.67.0 fix(build): force LF for shell scripts and route package.json checks through bash

Two independent defects left `bun run test`, `verify`, `ci:local` and
`test:e2e` dead on Windows. All four dispatch through bash.

First, every tracked *.sh is checked out with CRLF. The committed blobs are
clean LF; system-level core.autocrlf=true rewrites them on checkout, and a
strict bash then dies at run-unit-parallel.sh line 23 with
"$'\r': command not found". A root .gitattributes pinning `*.sh text eol=lf`
overrides autocrlf regardless of the contributor's git config.

Second, 33 package.json scripts invoked `scripts/foo.sh` directly, which bun
cannot exec via shebang on Windows. They now go through `bash`, matching the
11 that already did; all 59 tracked *.sh files are bash-shebanged (52
`#!/usr/bin/env bash`, 7 `#!/bin/bash`), so the change is uniform. The five
`scripts/*.ts` entries still run under bun.

Measured on this base, `bun run verify` goes from pass=1 fail=31 to pass=25
fail=7. Every one of the baseline's 29 `command not found: scripts/...`
errors is gone; those were the shebang defect, and they account for the
measured delta.

The line-ending defect is verified structurally rather than by that number,
because the bash on PATH for this measurement tolerates CR and so cannot
exhibit it: under the new attribute all 59 tracked *.sh check out LF-only
(0/59 carry a CR byte, against 59/59 before), and `git add --renormalize .`
is a no-op, confirming the index was always correct and only the working
tree was wrong. Zero content churn.

All 7 residual failures also fail on the pristine baseline: four exceed the
harness's 120s cap (standalone `bun run typecheck` exits 0), and check:wasm,
check:skill-brain-first and check:resolver are pre-existing content or
environment issues. check:resolver is not even a shell script.

No behavior change on Linux or macOS.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs: sync docs to v0.42.67.0

CONTRIBUTING.md gains a Windows section: the `.gitattributes` LF pin makes a
fresh clone correct with no extra steps, working copies cloned earlier need a
one-time `git rm --cached -r . -q && git reset --hard`, and new shell-script
checks must be registered as `bash scripts/<name>.sh`.

docs/TESTING.md records the shell-dispatch convention alongside the command-tier
table, and notes that the table's wallclock figures are Mac numbers: on Windows
`check:privacy`, `check:test-names`, `check:test-isolation` and `typecheck` can
exceed run-verify-parallel.sh's 120s per-check cap while passing on Linux and
macOS. It also flags that the Cygwin bash shipped with Git for Windows tolerates
CRLF where a strict bash does not, so a green local run is not evidence that a
script is CRLF-clean.

CHANGELOG.md's itemized list covers both doc updates.

`bun run build:llms` regenerates byte-identical bundles: docs/TESTING.md is
linked rather than inlined, so llms.txt / llms-full.txt do not move.
`bun test test/build-llms.test.ts` passes 12/12.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-07-28 11:45:09 -07:00
cybernaut6404 e58abd652c fix(extract): preserve facts on parse warnings (#3494) 2026-07-28 11:44:26 -07:00
MasaandTime Attakc a104f98dca fix(cycle): extract_facts guard counts only active legacy rows; reconcile preserves forget records (#2646) (#3474)
master's empty-fence guard counts soft-expired legacy rows
(row_num IS NULL, expired_at set), so forget_fact — the sanctioned
removal path, which soft-expires rather than deletes — can never drain
the backlog: apply-migrations no-ops (already marked applied) and the
guard stays triggered forever, jamming extract_facts.

Narrowed re-send of #3252-sibling #3234, scoped to exactly what the
maintainer named reviewable: "one predicate plus the
reconcile-preservation guard."

- Guard predicate: the legacy COUNT adds `AND f.expired_at IS NULL`,
  so each forget_fact visibly drains the pending counter.
- Reconcile preservation: listExistingFactsForPage excludes soft-expired
  legacy rows (they are never fence-owned, so they must neither read as
  perpetually stale nor mask a fence row), and the two wipe call sites
  pass `preserveExpiredLegacy: true` so deleteFactsForPage keeps the
  forget record. The option is the minimal seam for that guard —
  implemented identically in both engines (~6 lines each).

Explicitly NOT included from #3234 (per the close review): the
drift-repair lane, the re-runnable migration orchestrator path, and the
race-accounting layer.

Fence-is-canonical semantics are preserved and pinned by test: if the
fence still carries an expired legacy row's claim, the reconcile
reinserts it as a fresh active fence-owned row (legacy DB-only forgets
are documented non-durable; the expired row survives as the audit
record of the forget).

Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-07-28 11:44:17 -07:00
cybernaut6404 2ac6959b46 test(schema): exercise v121 bootstrap coverage (#3489) 2026-07-28 11:44:12 -07:00
paul-0320andTime Attakc 18ec732e1b fix(chunker): CJK-aware oversize measurement + close capOversizedChunks' fallback-path gaps (#3477)
Shape requested in #3475's closing review: make capOversizedChunks use a
CJK-aware estimate instead of adding a parallel opt-in cap.

Measurement (vs the Qwen3-Embedding tokenizer, the strict-backend class
from #2826): cl100k matches embedding-family tokenizers on pure-ASCII
source (identical counts on English prose and JSON) but undercounts
MIXED CJK+ASCII chunks — −31% on URL-dense Korean text. The heuristic
fallback (~3.5 chars/token) undercounts CJK ~2.5×.

estimateEmbedTokens(): for chunks containing CJK, max(cl100k, per-char-
class overestimate — CJK 1.0 / other non-ws 0.75 / ws 0.1). ASCII-only
chunks short-circuit to estimateTokens verbatim (bit-identical, pinned);
CJK-DOMINANT text is unchanged too (cl100k already exceeds the weighted
form, so max() returns today's value — pinned). Only mixed-script
chunks, the measured divergence class, estimate higher. Reuses cjk.ts's
existing exports — no new module, no config.

Also: only the empty-AST branch routed its fallback through
capOversizedChunks. The no-language, parse-timeout, no-semantic-nodes
(every JSON/YAML fence — their node types aren't in TOP_LEVEL_TYPES) and
parse-throw branches shipped word-counted chunks unchecked, letting a
14K-char JSON fence emit ~2,700-token chunks past the 2,000 default cap.
Hoist the cap into fallbackChunks so all five emission paths share the
net. Hard-split slice budget becomes 1 char/token for CJK-bearing pieces
(the weighted estimate can reach 1 token/char).

Refs #2826

Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-07-28 11:41:08 -07:00
24 changed files with 954 additions and 128 deletions
+16
View File
@@ -0,0 +1,16 @@
# Line-ending policy.
#
# Shell scripts MUST be checked out with LF endings on every platform.
# Git for Windows installs with `core.autocrlf=true` by default, which
# rewrites LF -> CRLF on checkout. A strict bash (WSL, Linux CI, macOS)
# then chokes on the trailing CR:
#
# scripts/run-unit-parallel.sh: line 23: $'\r': command not found
# scripts/run-unit-parallel.sh: line 24: set: pipefail : invalid option name
# scripts/run-unit-parallel.sh: line 32: syntax error near unexpected token `$'{\r''
#
# That silently disabled `bun run test`, `bun run verify`, `bun run ci:local`
# and `bun run test:e2e` for Windows contributors, since all four dispatch
# through bash. `eol=lf` pins the checkout regardless of the user's
# core.autocrlf setting.
*.sh text eol=lf
+39
View File
@@ -2,6 +2,45 @@
All notable changes to GBrain will be documented in this file.
## [0.42.67.0] - 2026-07-28
**If you develop GBrain on Windows, the test and check commands now actually run. Until this release they were quietly doing almost nothing.**
`bun run test`, `bun run verify`, `bun run ci:local` and `bun run test:e2e` all hand off to shell scripts, and on Windows that hand-off was broken in two separate places. The commands did not stop with an obvious error. They reported a result, so a run could look finished when barely any of the checks had actually inspected anything. On a clean Windows clone, `bun run verify` got 1 check to pass and 31 to fail. It now gets 25 to pass and 7 to fail, and none of the 7 are caused by this change.
The first problem was line endings. Git for Windows installs with `core.autocrlf=true`, which rewrites shell scripts to Windows line endings when you clone or check out. Bash refuses to run those, so a script died on its second line before doing any work. The scripts stored in the repository were always correct; only the copy on your disk was wrong. A new `.gitattributes` pins every `.sh` file to Unix line endings at checkout, no matter how your Git is configured.
The second problem was how the checks were started. Thirty three of them pointed straight at a `.sh` file. On macOS and Linux the shell reads the `#!/usr/bin/env bash` line at the top of the script and runs it correctly. Bun on Windows does not do that, so those commands failed the moment they were called. They now go through `bash` explicitly, the same way the other eleven were already written.
Nothing changes for macOS and Linux. No stored file content moves, and no check behaves differently on those platforms.
## To take advantage of v0.42.67.0
Only Windows contributors need to do anything, and only once. `.gitattributes` applies at checkout time, so shell scripts already sitting on your disk keep their old line endings until you refresh them.
1. **Refresh the working copy** from the repository root:
```bash
git rm --cached -r . -q
git reset --hard
```
2. **Confirm bash can read the scripts:**
```bash
bash -n scripts/run-unit-parallel.sh
```
Silence means it worked. `$'\r': command not found` means step 1 did not take effect.
3. **Run the gate:**
```bash
bun run verify
```
### Itemized changes
- New root `.gitattributes` pins `*.sh text eol=lf`, so shell scripts check out with Unix line endings regardless of the contributor's `core.autocrlf` setting. All 59 tracked `.sh` files were already stored with Unix endings, so `git add --renormalize .` reports nothing to do and no stored content changes.
- `package.json` now routes the remaining 33 `.sh` check commands through `bash`, matching the 11 that already did. Every tracked `.sh` file carries a bash shebang (52 `#!/usr/bin/env bash` and 7 `#!/bin/bash`), so the treatment is uniform across all of them.
- The five `scripts/*.ts` entries still run under bun and are untouched.
- `CONTRIBUTING.md` gains a Windows section covering the one-time working-copy refresh and the `bash scripts/<name>.sh` convention for new checks.
- `docs/TESTING.md` records how the test commands dispatch through bash, and notes that three tree-walking checks plus `typecheck` can exceed the 120s per-check cap on Windows while passing on Linux and macOS.
## [0.42.66.1] - 2026-07-27
### Fixed
+22
View File
@@ -11,6 +11,28 @@ bun test
Requires Bun 1.0+.
### Windows
`bun run test`, `verify`, `ci:local` and `test:e2e` all dispatch through bash, so
the shell scripts under `scripts/` must be checked out with Unix line endings.
The root `.gitattributes` pins `*.sh text eol=lf`, which overrides the
`core.autocrlf=true` that Git for Windows installs by default. A fresh clone is
correct with no extra steps.
If you cloned before that pin existed, your working copy still has the old
Windows line endings and bash will fail with `$'\r': command not found`. Refresh
it once, from the repository root:
```bash
git rm --cached -r . -q
git reset --hard
bash -n scripts/run-unit-parallel.sh # silence means bash can read the scripts
```
Every `check:*` entry in `package.json` invokes its script as `bash scripts/<name>.sh`
rather than relying on the shebang, because bun on Windows cannot exec a `.sh`
directly. Keep that prefix when you add a new shell-script check.
## Project structure
```
+27 -4
View File
@@ -1,5 +1,25 @@
# TODOS
## v0.42.67.0 follow-ups (Windows build tooling)
Filed as follow-ups from v0.42.67.0 (`.gitattributes` LF pin for `*.sh` +
`bash` prefix on the 33 `package.json` check commands). Both items are newly
observable: before that release these checks never executed on Windows at all,
so nothing about their runtime was measurable.
- [ ] **P2 — three guard scripts exceed the 120s `run-verify-parallel.sh` cap on Windows.**
With the dispatch fixed, `bun run verify` on Windows gets 25 passes and 7 failures, and
`check:privacy`, `check:test-names` and `check:test-isolation` are timeouts rather than
real failures (they pass on Linux and macOS well inside the cap). They walk the tree with
per-file shell loops, which is far slower under Windows process creation. Either raise the
cap for these three, or replace the per-file loop with a single `grep -r` pass. Same cap
swallows `typecheck`, though standalone `bun run typecheck` exits 0.
- [ ] **P3 — `check:wasm` cannot create its `node_modules` symlink on Windows.**
`scripts/check-wasm-embedded.sh` fails with `ln: failed to create symbolic link
'/tmp/gbrain-wasm-check.XXXX/node_modules': No such file or directory`. Unprivileged
Windows accounts cannot create symlinks without developer mode. Consider a junction, a
copy, or skipping the check with a clear message when symlink creation is unavailable.
## community fix-wave follow-ups (filed v0.42.60.0)
- [x] **P2 — cherry-pick #2112's uncovered doctor.ts hunk.** Fix-wave A (#2820) superseded
@@ -62,17 +82,20 @@ Deferred from the provider-agnostic plumbing wave (#1249/#1250/#1292/#2271/#2209
Plan + review trail at `~/.claude/plans/system-instruction-you-are-working-keen-newell.md`.
The eng-review + Codex outside-voice narrowed the wave to these deferrals:
- [ ] **P2 — Capability-aware query expansion on OpenAI-compat providers (#2372).**
- [x] **P2 — Capability-aware query expansion on OpenAI-compat providers (#2372).**
Expansion only runs for recipes that declare an `expansion` touchpoint, and only the
native providers (anthropic/openai/google) do. To make expansion work on
litellm/openrouter/groq/together/deepseek you must ADD expansion touchpoints to those
chat-capable recipes AND add a `generateObject``generateText` capability fallback for
backends without strict structured outputs. Feature-shaped; overlaps the general
OpenAI-compat proxy story (`docs/designs/COMMUNITY_IDEAS.md`). Community PR #2373 is a
starting point. Where: `src/core/ai/gateway.ts:expand`, recipe files, `types.ts` (ExpansionTouchpoint).
- [ ] **P2 — LiteLLM as a chat/expansion backend.** `litellm-proxy` declares ONLY an
starting point. Implemented by #2373 plus the DeepSeek/Groq/Together recipe wave,
LiteLLM chat/expansion support, and the OpenRouter expansion touchpoint. Where:
`src/core/ai/gateway.ts:expand`, recipe files, `types.ts` (ExpansionTouchpoint).
- [x] **P2 — LiteLLM as a chat/expansion backend.** `litellm-proxy` declares ONLY an
embedding touchpoint, so `think`/chat on LiteLLM is dead. Add chat (and expansion) so a
LiteLLM proxy is a full LLM backend, not embedding-only. The general OpenAI-compat proxy story.
LiteLLM proxy is a full LLM backend, not embedding-only. Implemented by #2208.
The general OpenAI-compat proxy story.
- [ ] **P3 — Per-model embedding dims metadata on `EmbeddingTouchpoint`.** `default_dims`
is recipe-wide, so a recipe (ollama) can't carry different native dims per model. This
wave added the modern ollama model NAMES + a `trust_custom_dims` passthrough (user supplies
+1 -1
View File
@@ -1 +1 @@
0.42.66.1
0.42.67.0
+23
View File
@@ -19,6 +19,29 @@ Seven test command tiers, each with a clear scope:
| `bun run test:e2e` | Real Postgres E2E. Requires Docker + `DATABASE_URL`. Sequential. | ~5-10min | Pre-ship; nightly. |
| `bun run check:all` | The historical pre-check scripts (22, chained sequentially in package.json). Overlaps `verify` heavily but is NOT a superset — `verify`'s `CHECKS` array in `scripts/run-verify-parallel.sh` (~30 entries incl. typecheck) is the authoritative gate; `check:all` keeps a few local-only extras (trailing-newline, exports-count, no-legacy-getconnection). | ~10s | Local-only sweep for the extras. |
### Shell dispatch and Windows
All four of `test`, `verify`, `ci:local` and `test:e2e` hand off to shell scripts
under `scripts/`, so every `check:*` entry in `package.json` invokes its script as
`bash scripts/<name>.sh` instead of relying on the shebang — bun on Windows cannot
exec a `.sh` directly. Add a new shell-script check with that same prefix. The
`scripts/*.ts` entries run under bun and take no prefix.
The scripts must also be on disk with Unix line endings. A strict bash (WSL, Linux
CI, macOS) rejects CRLF and dies on the script's first meaningful line; the Cygwin
bash that ships with Git for Windows tolerates it, so a green local run is not by
itself evidence that a script is CRLF-clean.
The root `.gitattributes` pins `*.sh text eol=lf`, which overrides the
`core.autocrlf=true` default that Git for Windows installs. Working copies cloned
before that pin need a one-time `git rm --cached -r . -q && git reset --hard` to
pick it up; see the Windows section of `CONTRIBUTING.md`.
Wallclock figures in the table above are from a Mac dev box. Windows is
substantially slower because each check pays full process-creation cost, and three
tree-walking checks (`check:privacy`, `check:test-names`, `check:test-isolation`)
plus `typecheck` can exceed the 120s per-check cap in `run-verify-parallel.sh`
there even though they pass on Linux and macOS.
### CI vs local: intentionally divergent file sets
- **CI matrix** (`.github/workflows/test.yml`) runs `scripts/test-shard.sh` across 10 matrix shards partitioned by weight-aware LPT bin-packing (`scripts/sharding.ts`) and INCLUDES `*.slow.test.ts` (the two outlier slow files run as dedicated jobs alongside the matrix). CI EXCLUDES `*.serial.test.ts` from the shards and runs them in a dedicated job via `bun run test:serial`, one bun process per file — keeping serial files out of the shard processes is what preserves the `mock.module` quarantine (a top-level mock in one file leaks into every other file sharing its process). `bun run verify` gets its own job too. CI is the ground truth for "did everything pass."
+34 -34
View File
@@ -42,20 +42,20 @@
"eval:autocut": "bun test test/search/autocut-eval.test.ts",
"test:full": "bun run verify && bash scripts/run-unit-parallel.sh && bun run test:slow && ([ -n \"$DATABASE_URL\" ] && bash scripts/run-e2e.sh || echo '[test:full] skipped E2E (no DATABASE_URL); run docker-compose -f docker-compose.ci.yml up + bun run test:e2e to include' 1>&2)",
"verify": "bash scripts/run-verify-parallel.sh",
"check:source-config-leak": "scripts/check-source-config-leak.sh",
"check:no-pii-agent-voice": "scripts/check-no-pii-in-agent-voice.sh",
"check:synthetic-corpus-privacy": "scripts/check-synthetic-corpus-privacy.sh",
"check:system-of-record": "scripts/check-system-of-record.sh",
"check:admin-scope-drift": "scripts/check-admin-scope-drift.sh",
"check:cli-exec": "scripts/check-cli-executable.sh",
"check:all": "scripts/check-privacy.sh && scripts/check-proposal-pii.sh && scripts/check-test-real-names.sh && scripts/check-jsonb-pattern.sh && scripts/check-source-id-projection.sh && scripts/check-source-config-leak.sh && scripts/check-progress-to-stdout.sh && scripts/check-no-tracked-symlinks.sh && scripts/check-no-legacy-getconnection.sh && scripts/check-test-isolation.sh && scripts/check-trailing-newline.sh && scripts/check-wasm-embedded.sh && scripts/check-exports-count.sh && scripts/check-admin-build.sh && scripts/check-admin-scope-drift.sh && scripts/check-cli-executable.sh && scripts/check-skill-brain-first.sh && scripts/check-operations-filter-bypass.sh && scripts/check-gateway-routed-no-direct-anthropic.sh && scripts/check-worker-pool-atomicity.sh && scripts/check-key-files-current-state.sh && scripts/check-no-double-retry.sh && scripts/check-batch-audit-site.sh",
"check:gateway-routed": "scripts/check-gateway-routed-no-direct-anthropic.sh",
"check:worker-pool-atomicity": "scripts/check-worker-pool-atomicity.sh",
"check:doc-history": "scripts/check-key-files-current-state.sh",
"check:source-config-leak": "bash scripts/check-source-config-leak.sh",
"check:no-pii-agent-voice": "bash scripts/check-no-pii-in-agent-voice.sh",
"check:synthetic-corpus-privacy": "bash scripts/check-synthetic-corpus-privacy.sh",
"check:system-of-record": "bash scripts/check-system-of-record.sh",
"check:admin-scope-drift": "bash scripts/check-admin-scope-drift.sh",
"check:cli-exec": "bash scripts/check-cli-executable.sh",
"check:all": "bash scripts/check-privacy.sh && bash scripts/check-proposal-pii.sh && bash scripts/check-test-real-names.sh && bash scripts/check-jsonb-pattern.sh && bash scripts/check-source-id-projection.sh && bash scripts/check-source-config-leak.sh && bash scripts/check-progress-to-stdout.sh && bash scripts/check-no-tracked-symlinks.sh && bash scripts/check-no-legacy-getconnection.sh && bash scripts/check-test-isolation.sh && bash scripts/check-trailing-newline.sh && bash scripts/check-wasm-embedded.sh && bash scripts/check-exports-count.sh && bash scripts/check-admin-build.sh && bash scripts/check-admin-scope-drift.sh && bash scripts/check-cli-executable.sh && bash scripts/check-skill-brain-first.sh && bash scripts/check-operations-filter-bypass.sh && bash scripts/check-gateway-routed-no-direct-anthropic.sh && bash scripts/check-worker-pool-atomicity.sh && bash scripts/check-key-files-current-state.sh && bash scripts/check-no-double-retry.sh && bash scripts/check-batch-audit-site.sh",
"check:gateway-routed": "bash scripts/check-gateway-routed-no-direct-anthropic.sh",
"check:worker-pool-atomicity": "bash scripts/check-worker-pool-atomicity.sh",
"check:doc-history": "bash scripts/check-key-files-current-state.sh",
"check:resolver": "bun src/cli.ts check-resolvable --strict --skills-dir skills/",
"check:skill-brain-first": "scripts/check-skill-brain-first.sh",
"check:wasm": "scripts/check-wasm-embedded.sh",
"check:newlines": "scripts/check-trailing-newline.sh",
"check:skill-brain-first": "bash scripts/check-skill-brain-first.sh",
"check:wasm": "bash scripts/check-wasm-embedded.sh",
"check:newlines": "bash scripts/check-trailing-newline.sh",
"test:e2e": "bash scripts/run-e2e.sh",
"test:slow": "bash scripts/run-slow-tests.sh",
"test:heavy": "bash scripts/run-heavy.sh",
@@ -65,27 +65,27 @@
"ci:local:diff": "bash scripts/ci-local.sh --diff",
"ci:select-e2e": "bun run scripts/select-e2e.ts",
"typecheck": "tsc --noEmit",
"check:jsonb": "scripts/check-jsonb-pattern.sh",
"check:search-path": "scripts/check-search-path.sh",
"check:no-double-retry": "scripts/check-no-double-retry.sh",
"check:batch-audit-site": "scripts/check-batch-audit-site.sh",
"check:worker-lock-renewal-shape": "scripts/check-worker-lock-renewal-shape.sh",
"check:source-id-projection": "scripts/check-source-id-projection.sh",
"check:privacy": "scripts/check-privacy.sh",
"check:proposal-pii": "scripts/check-proposal-pii.sh",
"check:eval-glossary": "scripts/check-eval-glossary-fresh.sh",
"check:test-names": "scripts/check-test-real-names.sh",
"check:progress": "scripts/check-progress-to-stdout.sh",
"check:no-tracked-symlinks": "scripts/check-no-tracked-symlinks.sh",
"check:exports-count": "scripts/check-exports-count.sh",
"check:admin-build": "scripts/check-admin-build.sh",
"check:admin-embedded": "scripts/check-admin-embedded.sh",
"check:test-isolation": "scripts/check-test-isolation.sh",
"check:fuzz-purity": "scripts/check-fuzz-purity.sh",
"check:operations-filter-bypass": "scripts/check-operations-filter-bypass.sh",
"check:fixture-privacy": "scripts/check-fixture-privacy.sh",
"check:jsonb": "bash scripts/check-jsonb-pattern.sh",
"check:search-path": "bash scripts/check-search-path.sh",
"check:no-double-retry": "bash scripts/check-no-double-retry.sh",
"check:batch-audit-site": "bash scripts/check-batch-audit-site.sh",
"check:worker-lock-renewal-shape": "bash scripts/check-worker-lock-renewal-shape.sh",
"check:source-id-projection": "bash scripts/check-source-id-projection.sh",
"check:privacy": "bash scripts/check-privacy.sh",
"check:proposal-pii": "bash scripts/check-proposal-pii.sh",
"check:eval-glossary": "bash scripts/check-eval-glossary-fresh.sh",
"check:test-names": "bash scripts/check-test-real-names.sh",
"check:progress": "bash scripts/check-progress-to-stdout.sh",
"check:no-tracked-symlinks": "bash scripts/check-no-tracked-symlinks.sh",
"check:exports-count": "bash scripts/check-exports-count.sh",
"check:admin-build": "bash scripts/check-admin-build.sh",
"check:admin-embedded": "bash scripts/check-admin-embedded.sh",
"check:test-isolation": "bash scripts/check-test-isolation.sh",
"check:fuzz-purity": "bash scripts/check-fuzz-purity.sh",
"check:operations-filter-bypass": "bash scripts/check-operations-filter-bypass.sh",
"check:fixture-privacy": "bash scripts/check-fixture-privacy.sh",
"check:conversation-parser": "bun src/cli.ts eval conversation-parser test/fixtures/conversation-formats/all.jsonl --no-llm",
"check:source-scope-onboard": "scripts/check-source-scope-onboard.sh",
"check:source-scope-onboard": "bash scripts/check-source-scope-onboard.sh",
"postinstall": "bun run scripts/postinstall.ts",
"prepublish:clawhub": "bun run build:all",
"publish:clawhub": "clawhub package publish . --family bundle-plugin"
@@ -146,7 +146,7 @@
"bun": ">=1.3.10"
},
"license": "MIT",
"version": "0.42.66.1",
"version": "0.42.67.0",
"overrides": {
"@hono/node-server": "^2.0.5",
"fast-uri": "^3.1.4",
+71 -27
View File
@@ -8,7 +8,7 @@ import {
semverGt,
semverLte,
} from '../core/semver.ts';
import { writeUpdateCache, type UpdateMarker } from '../core/self-upgrade.ts';
import { readUpdateCache, writeUpdateCache, type UpdateMarker } from '../core/self-upgrade.ts';
/** Best-effort cache write — a read-only ~/.gbrain must never make the check throw. */
function safeWriteCache(marker: UpdateMarker): void {
@@ -45,26 +45,53 @@ function upgradeCommandForMethod(method: string): string {
}
}
/** Where the latest version is resolved from. gbrain publishes NO GitHub
* releases (the `releases/latest` API is a permanent 404), so the release
* train's source of truth is the `VERSION` file on master — same trusted host
* `fetchChangelog` already uses. An npm fallback was rejected: the `gbrain`
* package on npm is an unrelated GPU library (#505), so it would produce false
* upgrade prompts pointing at a stranger's package. */
const VERSION_SOURCE_URL = 'https://raw.githubusercontent.com/garrytan/gbrain/master/VERSION';
const RELEASE_NOTES_URL = 'https://github.com/garrytan/gbrain/blob/master/CHANGELOG.md';
/** Extract a version from the raw VERSION file body: first line, optional `v`
* prefix, optional `-suffix` channel tag (`0.31.1.1-fixwave` compares as its
* numeric base — fail-safe: a suffix-only bump never prompts). Body is bounded
* before parsing so a malformed/huge response can't blow up the check. */
export function parseVersionFileBody(body: string): string | null {
const firstLine = body.slice(0, 256).trim().split('\n')[0].trim();
const m = firstLine.match(/^v?(\d+\.\d+\.\d+(?:\.\d+)?)(?:[-+][0-9A-Za-z.-]+)?$/);
return m && isValidVersionString(m[1]) ? m[1] : null;
}
export type LatestReleaseResult =
| { ok: true; tag: string; published_at: string; url: string }
| { ok: false; reason: 'network_error' | 'no_releases' };
/**
* Fetch the latest GitHub release. Exported (v0.42) so the self-upgrade refresh
* path and tests can reuse it. 5s timeout (was 10s) — this runs on the detached
* refresh, never the hot path, but a tight bound keeps the refresh cheap.
* Resolve the latest published gbrain version (from VERSION on master — see
* VERSION_SOURCE_URL). Exported (v0.42) so the self-upgrade refresh path and
* tests can reuse it. 5s timeout this runs on the detached refresh, never the
* hot path. Failures are discriminated: `network_error` (offline/timeout) vs
* `no_releases` (endpoint answered but no usable version).
*/
export async function fetchLatestRelease(): Promise<{ tag: string; published_at: string; url: string } | null> {
export async function fetchLatestRelease(): Promise<LatestReleaseResult> {
let res: Response;
try {
const res = await fetch('https://api.github.com/repos/garrytan/gbrain/releases/latest', {
res = await fetch(VERSION_SOURCE_URL, {
headers: { 'User-Agent': `gbrain/${VERSION}` },
signal: AbortSignal.timeout(5_000),
});
if (!res.ok) return null;
const data = await res.json() as any;
return {
tag: data.tag_name || '',
published_at: data.published_at || '',
url: data.html_url || '',
};
} catch {
return null;
return { ok: false, reason: 'network_error' };
}
try {
if (!res.ok) return { ok: false, reason: 'no_releases' };
const tag = parseVersionFileBody(await res.text());
if (!tag) return { ok: false, reason: 'no_releases' };
return { ok: true, tag, published_at: '', url: RELEASE_NOTES_URL };
} catch {
return { ok: false, reason: 'network_error' };
}
}
@@ -118,17 +145,33 @@ export function extractChangelogBetween(changelog: string, from: string, to: str
}
/**
* Fetch the latest release and write the self-upgrade cache (the marker line
* read by the CLI startup hook). Fail-open: on any network failure we cache
* `UP_TO_DATE <current>` so the TTL prevents hammering GitHub on every
* invocation. Returns the resolved marker for callers that want it. This is the
* function the detached single-flight refresh (`gbrain check-update
* --refresh-cache`) invokes.
* A failed check must NEVER write `up_to_date` — that was #486: the fetch
* failed permanently (dead releases API) and every user was told "you're
* current" forever. Instead, re-write the last-known-good marker (bumping its
* mtime so the cache TTL still throttles retries and a network blip can't
* erase a pending upgrade_available notice). No prior marker → write nothing;
* the next invocation retries.
*/
function preserveCacheOnFailedCheck(): void {
try {
const prior = readUpdateCache();
if (prior) safeWriteCache(prior.marker);
} catch {
/* best-effort */
}
}
/**
* Fetch the latest version and write the self-upgrade cache (the marker line
* read by the CLI startup hook). On fetch failure the last-known-good marker is
* preserved (see preserveCacheOnFailedCheck) — never a fabricated `up_to_date`.
* This is the function the detached single-flight refresh (`gbrain
* check-update --refresh-cache`) invokes.
*/
export async function refreshUpdateCache(): Promise<void> {
const release = await fetchLatestRelease();
if (!release) {
safeWriteCache({ kind: 'up_to_date', current: VERSION });
if (!release.ok) {
preserveCacheOnFailedCheck();
return;
}
const latestVersion = release.tag.replace(/^v/, '');
@@ -166,9 +209,8 @@ export async function runCheckUpdate(args: string[]) {
const release = await fetchLatestRelease();
if (!release) {
// Warm the cache fail-open so the startup hook doesn't re-fetch every call.
safeWriteCache({ kind: 'up_to_date', current: VERSION });
if (!release.ok) {
preserveCacheOnFailedCheck();
if (json) {
console.log(JSON.stringify({
current_version: VERSION,
@@ -179,10 +221,12 @@ export async function runCheckUpdate(args: string[]) {
release_url: '',
changelog_diff: '',
published_at: '',
error: 'no_releases',
error: release.reason,
}, null, 2));
} else if (release.reason === 'network_error') {
console.log(`GBrain ${VERSION} — could not check for updates (network unavailable).`);
} else {
console.log(`GBrain ${VERSION} — could not check for updates (no releases found or network unavailable).`);
console.log(`GBrain ${VERSION} — could not determine the latest published version.`);
}
return;
}
+2 -1
View File
@@ -35,7 +35,8 @@ export async function runSelfUpgrade(args: string[]): Promise<void> {
const force = args.includes('--force');
const json = args.includes('--json');
const release = await fetchLatestRelease();
const result = await fetchLatestRelease();
const release = result.ok ? result : null;
const latest = release ? release.tag.replace(/^v/, '') : null;
const behind = !!latest && isValidVersionString(latest) && isNewerVersion(VERSION, latest);
+11
View File
@@ -180,6 +180,17 @@ export const openrouter: Recipe = {
// to pre-split batches, NOT per-input. Per-input is enforced upstream.
max_batch_tokens: 300_000,
},
// Expansion uses the same routed OpenAI-compatible language-model endpoint
// as chat. Keep a small cheap/fast advisory set; the openai-compat tier
// still accepts any user-configured OpenRouter provider/model ID.
expansion: {
models: [
'anthropic/claude-haiku-4.5',
'google/gemini-3-flash-preview',
'deepseek/deepseek-chat',
],
price_last_verified: '2026-05-20',
},
chat: {
// Curated entry points (verified against OR's catalog 2026-05-20). The
// openai-compat tier does NOT enforce this list at runtime — users can
+42 -7
View File
@@ -20,6 +20,7 @@
import { chunkText as recursiveChunk } from './recursive.ts';
import { buildQualifiedName } from './qualified-names.ts';
import { CJK_SLUG_CHARS, CJK_RANGES_REGEX } from '../cjk.ts';
// Embed the tree-sitter runtime + per-language grammars as files.
// `with { type: 'file' }` returns a path (string) at runtime. Bun bundles
@@ -716,7 +717,7 @@ export async function chunkCodeTextFull(
}
if (chunks.length === 0) {
return { chunks: capOversizedChunks(fallbackChunks(source, filePath, language, opts), filePath, language, opts), edges: rawEdges };
return { chunks: fallbackChunks(source, filePath, language, opts), edges: rawEdges };
}
return { chunks: capOversizedChunks(mergeSmallSiblings(chunks, chunkTarget), filePath, language, opts), edges: rawEdges };
} catch {
@@ -842,10 +843,10 @@ function capOversizedChunks(
opts: CodeChunkOptions,
): CodeChunk[] {
const cap = opts.maxChunkTokens ?? DEFAULT_MAX_CHUNK_TOKENS;
if (!chunks.some((c) => estimateTokens(c.text) > cap)) return chunks;
if (!chunks.some((c) => estimateEmbedTokens(c.text) > cap)) return chunks;
const out: CodeChunk[] = [];
for (const c of chunks) {
if (estimateTokens(c.text) <= cap) {
if (estimateEmbedTokens(c.text) <= cap) {
out.push({ ...c, index: out.length });
continue;
}
@@ -880,17 +881,43 @@ function splitToTokenBudget(text: string, cap: number, opts: CodeChunkOptions):
chunkOverlap: opts.fallbackOverlapWords ?? 50,
}).map((p) => p.text);
for (const piece of pieces) {
if (estimateTokens(piece) <= cap) {
if (estimateEmbedTokens(piece) <= cap) {
out.push(piece);
continue;
}
// ~3.5 chars/token is a conservative cl100k estimate for source text.
const charBudget = Math.max(1, Math.floor(cap * 3.5));
// Hard-split slice size. Pure-ASCII pieces: ~3.5 chars/token is a
// conservative cl100k estimate for source text. CJK-containing pieces:
// the weighted estimate can reach 1 token/char, so budget 1 char/token
// to keep every slice under cap by construction.
const charBudget = Math.max(1, Math.floor(cap * (CJK_RANGES_REGEX.test(piece) ? 1 : 3.5)));
for (let i = 0; i < piece.length; i += charBudget) out.push(piece.slice(i, i + charBudget));
}
return out;
}
const CJK_CHARS_G = new RegExp(`[${CJK_SLUG_CHARS}]`, 'g');
/**
* Embedding-safe token estimate for the oversize cap. estimateTokens
* (cl100k) matches embedding-family tokenizers closely on pure-ASCII source
* (measured identical on English prose and JSON vs Qwen3-Embedding), but
* UNDERCOUNTS mixed CJK+ASCII chunks — measured 31% on URL-dense Korean
* text vs the Qwen3 embedding tokenizer, which is exactly the shape that
* overflows strict embedding backends (#2826). For chunks containing CJK,
* take the max of cl100k and a per-char-class overestimate (CJK 1.0/char,
* other non-whitespace 0.75/char, whitespace 0.1/char). CJK-DOMINANT text
* is unaffected too: cl100k already counts it above the weighted form, so
* max() returns the same value as today. Only mixed-script chunks — the
* measured divergence class — estimate higher.
*/
export function estimateEmbedTokens(text: string): number {
const cjk = (text.match(CJK_CHARS_G) || []).length;
if (cjk === 0) return estimateTokens(text);
const ws = (text.match(/\s/g) || []).length;
const weighted = Math.ceil(cjk + (text.length - cjk - ws) * 0.75 + ws * 0.1);
return Math.max(estimateTokens(text), weighted);
}
// ---------- Internals ----------
function fallbackChunks(
@@ -901,7 +928,7 @@ function fallbackChunks(
): CodeChunk[] {
const size = opts.fallbackChunkSizeWords ?? 300;
const overlap = opts.fallbackOverlapWords ?? 50;
return recursiveChunk(source, { chunkSize: size, chunkOverlap: overlap }).map((chunk, index) =>
const chunks = recursiveChunk(source, { chunkSize: size, chunkOverlap: overlap }).map((chunk, index) =>
buildChunk({
body: chunk.text, filePath, language,
symbolName: null, symbolType: 'module',
@@ -909,6 +936,14 @@ function fallbackChunks(
index,
}),
);
// Route every fallback emission through the oversize net. Previously only
// the empty-AST branch wrapped its fallback in capOversizedChunks — the
// no-language, parse-timeout, no-semantic-nodes (every JSON/YAML fence:
// their node types aren't in TOP_LEVEL_TYPES) and parse-throw branches
// shipped word-counted chunks unchecked, and the word pipeline undercounts
// exactly the dense content (JSON, minified, CJK-mixed) that overflows
// embedders. Hoisting the cap here covers all five paths at once.
return capOversizedChunks(chunks, filePath, language, opts);
}
function buildChunk(input: {
+59 -19
View File
@@ -17,17 +17,19 @@
* DB rows need cleanup (#1781 — the unconditional wipe-and-reinsert
* made every cycle non-idempotent, re-appending duplicate rows).
*
* After the phase, the DB index for every affected page matches the
* fence's canonical (claim, source) row set (modulo embeddings +
* runtime-derived fields). Pages with no fence wipe DB rows for that
* page coordinate only; legacy NULL-source_markdown_slug rows survive
* because deleteFactsForPage targets source_markdown_slug = slug only.
* After the phase, the DB index for every cleanly parsed affected page
* matches the fence's canonical (claim, source) row set (modulo embeddings
* + runtime-derived fields). Warning-bearing parses are non-authoritative
* and preserve that page's existing index. Pages with no fence wipe DB rows
* for that page coordinate only; legacy NULL-source_markdown_slug rows
* survive because deleteFactsForPage targets source_markdown_slug = slug only.
*
* Empty-fence guard (Codex R2-#7; #2484): the phase refuses to do its
* destructive reconciliation pass when genuinely-backfillable legacy
* Empty-fence guard (Codex R2-#7; #2484; #2646): the phase refuses to do
* its destructive reconciliation pass when genuinely-backfillable legacy
* rows still exist — `row_num IS NULL` (never fenced) AND `entity_slug`
* resolves to a live page in this source (so the v0_32_2 migration's
* Phase B could fence them). Status returns `warn` with a hint to run
* Phase B could fence them) AND the row is not soft-expired
* (`expired_at IS NULL`). Status returns `warn` with a hint to run
* `gbrain apply-migrations --yes`. Without the guard, an interrupted
* upgrade where v0_32_2 hasn't run could leave the cycle silently
* misreporting "0 facts on people/alice" while legacy rows linger.
@@ -41,6 +43,10 @@
* the phase jams forever (~16/day observed). Requiring a backing page
* keeps genuine pre-v0.32.2 rows (whose entity page exists) gating
* while excluding the inline-writer's permanent-unfenceable rows.
*
* Soft-expired rows don't count either (#2646): they're what
* `forget_fact` produces, so excluding them lets operators drain the
* backlog through the sanctioned removal path instead of raw SQL.
*/
import type { BrainEngine } from '../engine.ts';
@@ -88,6 +94,24 @@ function dedupeFactsByContentKey(facts: FenceExtractedFact[]): FenceExtractedFac
* neither count as "stale" (which would force a wipe every cycle) nor
* be compared against the fence's row set. Mirrors the
* excludeSourcePrefixes filter deleteFactsForPage applies on the wipe.
*
* Also excludes soft-expired legacy rows (#2646: `row_num IS NULL AND
* expired_at IS NOT NULL`) — rows that `forget_fact` expired via its
* legacy DB-only path. They are not fence-owned (fence rows always
* carry a row_num), so they must neither count as "stale" (forcing a
* wipe every cycle) nor mask a fence row from insertion. Mirrors the
* preserveExpiredLegacy filter deleteFactsForPage applies on the wipe.
*
* Deliberate consequence: if the fence still carries the same
* (claim, source) as an expired legacy row, the reconcile inserts it
* as a fresh ACTIVE fence-owned row. That is the fence-is-canonical
* contract working as documented — legacy DB-only forgets "DO NOT
* survive rebuild" (see forget.ts header); suppressing the insert
* would instead create silent fence↔DB divergence, the exact failure
* mode the empty-fence guard exists to prevent. To durably forget
* such a claim, forget the fence-owned row (forget_fact now takes the
* fence path, which strikes the row through in markdown). The expired
* legacy row survives alongside as the record of the earlier forget.
*/
async function listExistingFactsForPage(
engine: BrainEngine,
@@ -100,6 +124,7 @@ async function listExistingFactsForPage(
WHERE source_id = $1
AND source_markdown_slug = $2
AND COALESCE(source, '') NOT LIKE 'cli:%'
AND NOT (row_num IS NULL AND expired_at IS NOT NULL)
ORDER BY row_num ASC, id ASC`,
[sourceId, slug],
);
@@ -173,7 +198,7 @@ export async function runExtractFacts(
phantomsMorePending: false,
};
// ── Empty-fence guard (Codex R2-#7; #2484) ─────────────────────
// ── Empty-fence guard (Codex R2-#7; #2484; #2646) ──────────────
// Pre-check: if any genuinely-backfillable legacy fact rows exist,
// refuse to run the destructive reconciliation pass — the v0_32_2
// orchestrator must fence them first.
@@ -181,12 +206,13 @@ export async function runExtractFacts(
// A row is a real backfill candidate only when `row_num IS NULL`
// (never fenced) AND its `entity_slug` resolves to a LIVE page in
// this source (the migration's Phase B only fences rows whose
// entity_slug maps to a writable page). #2484: the original
// predicate was just `row_num IS NULL AND entity_slug IS NOT NULL`,
// which ALSO matched structurally-unfenceable hot-memory rows the
// inline writer keeps producing post-migration: the legacy DB-only
// fallback (backstop.ts) writes `entity_slug` (a resolved slug, e.g.
// a slugify-floor or stub-guard-blocked unprefixed slug like
// entity_slug maps to a writable page) AND it is not soft-expired.
// #2484: the original predicate was just `row_num IS NULL AND
// entity_slug IS NOT NULL`, which ALSO matched
// structurally-unfenceable hot-memory rows the inline writer keeps
// producing post-migration: the legacy DB-only fallback
// (backstop.ts) writes `entity_slug` (a resolved slug, e.g. a
// slugify-floor or stub-guard-blocked unprefixed slug like
// `people-jane-doe`) with `row_num` NULL whenever the slug has no
// fenceable page. Those rows can never satisfy the migration's exit
// condition (no page to fence onto, and `apply-migrations` is a
@@ -194,12 +220,17 @@ export async function runExtractFacts(
// — ~16/day, mislabeled "v0.31 pending backfill." We now require a
// live backing page, which both genuine pre-v0.32.2 rows (their
// entity page exists) satisfy and inline-writer unfenceable rows do
// not.
// not. #2646: soft-expired rows (`expired_at IS NOT NULL`) are also
// excluded — `forget_fact`, the officially sanctioned removal path,
// soft-expires legacy rows rather than deleting them, so counting
// expired rows would leave the guard permanently stuck with no
// supported way to drain the backlog.
const legacy = await engine.executeRaw<{ n: string }>(
`SELECT COUNT(*) AS n
FROM facts f
WHERE f.row_num IS NULL
AND f.entity_slug IS NOT NULL
AND f.expired_at IS NULL
AND EXISTS (
SELECT 1 FROM pages p
WHERE p.source_id = f.source_id
@@ -303,6 +334,11 @@ export async function runExtractFacts(
result.warnings.push(
...parsed.warnings.map(w => `${slug}: ${w}`),
);
// The parser deliberately skips malformed rows and returns any rows it
// could still recover. That partial result is not authoritative: using
// it for reconciliation would interpret skipped rows as deletions.
// Preserve this page's existing index and continue with other pages.
continue;
}
if (parsed.facts.length > 0) result.pagesWithFacts += 1;
@@ -334,9 +370,12 @@ export async function runExtractFacts(
// partial-UNIQUE-index keyspace). #1928: `cli:`-origin facts
// (conversation facts from extract-conversation-facts) are NOT
// fence-owned — the page carries no `## Facts` fence to recreate
// them — so they MUST survive this reconcile.
// them — so they MUST survive this reconcile. #2646: soft-expired
// legacy rows (forget_fact's record of the forget) likewise
// survive via preserveExpiredLegacy.
const deleted = await engine.deleteFactsForPage(slug, sourceId, {
excludeSourcePrefixes: ['cli:'],
preserveExpiredLegacy: true,
});
result.factsDeleted += deleted.deleted;
}
@@ -363,10 +402,11 @@ export async function runExtractFacts(
if (hasStaleExisting || hasDuplicateExisting || hasRowNumDrift) {
// Fall back to the legacy page-level reconcile when old DB rows must
// be removed. Same delete scoping as above: legacy
// NULL-source_markdown_slug rows and `cli:`-origin conversation
// facts (#1928) survive.
// NULL-source_markdown_slug rows, `cli:`-origin conversation
// facts (#1928), and soft-expired legacy rows (#2646) survive.
const deleted = await engine.deleteFactsForPage(slug, sourceId, {
excludeSourcePrefixes: ['cli:'],
preserveExpiredLegacy: true,
});
result.factsDeleted += deleted.deleted;
toInsert = extracted;
+12 -1
View File
@@ -1815,11 +1815,22 @@ export interface BrainEngine {
* never recreate them (the page has no `## Facts` fence). Omitted ⇒ legacy
* behavior (delete every fact on the page coordinate). NULL/empty `source`
* rows are always deletable (fence default).
*
* #2646: `preserveExpiredLegacy` protects soft-expired legacy rows
* (`row_num IS NULL AND expired_at IS NOT NULL`) — the record left by
* `forget_fact`'s legacy DB-only path. Fence rows always carry a
* `row_num`, so these rows are never fence-owned and a wipe would
* destroy the forget record (the audit trail of the forget). Note what
* this does NOT promise: it protects the record, not the forget itself —
* if the fence still carries the same claim, fence canonicality
* independently reinserts it as a fresh active row (legacy DB-only
* forgets are documented as non-durable; see extract-facts.ts). Omitted
* ⇒ legacy behavior.
*/
deleteFactsForPage(
slug: string,
source_id: string,
opts?: { excludeSourcePrefixes?: string[] },
opts?: { excludeSourcePrefixes?: string[]; preserveExpiredLegacy?: boolean },
): Promise<{ deleted: number }>;
/**
+9 -3
View File
@@ -4281,9 +4281,15 @@ export class PGLiteEngine implements BrainEngine {
async deleteFactsForPage(
slug: string,
source_id: string,
opts?: { excludeSourcePrefixes?: string[] },
opts?: { excludeSourcePrefixes?: string[]; preserveExpiredLegacy?: boolean },
): Promise<{ deleted: number }> {
const prefixes = opts?.excludeSourcePrefixes;
// #2646: keep soft-expired legacy rows (row_num NULL — never
// fence-owned) so a fence reconcile can't destroy forget_fact's
// legacy DB-only forget record.
const expiredLegacyFilter = opts?.preserveExpiredLegacy
? ` AND NOT (row_num IS NULL AND expired_at IS NOT NULL)`
: '';
if (prefixes && prefixes.length > 0) {
// #1928: keep rows whose `source` matches an excluded prefix (e.g.
// `cli:` conversation facts). COALESCE so NULL/empty-source fence rows
@@ -4292,13 +4298,13 @@ export class PGLiteEngine implements BrainEngine {
const result = await this.db.query(
`DELETE FROM facts
WHERE source_id = $1 AND source_markdown_slug = $2
AND NOT (COALESCE(source, '') LIKE ANY($3::text[]))`,
AND NOT (COALESCE(source, '') LIKE ANY($3::text[]))${expiredLegacyFilter}`,
[source_id, slug, patterns],
);
return { deleted: result.affectedRows ?? 0 };
}
const result = await this.db.query(
`DELETE FROM facts WHERE source_id = $1 AND source_markdown_slug = $2`,
`DELETE FROM facts WHERE source_id = $1 AND source_markdown_slug = $2${expiredLegacyFilter}`,
[source_id, slug],
);
return { deleted: result.affectedRows ?? 0 };
+9 -2
View File
@@ -4438,10 +4438,16 @@ export class PostgresEngine implements BrainEngine {
async deleteFactsForPage(
slug: string,
source_id: string,
opts?: { excludeSourcePrefixes?: string[] },
opts?: { excludeSourcePrefixes?: string[]; preserveExpiredLegacy?: boolean },
): Promise<{ deleted: number }> {
const sql = this.sql;
const prefixes = opts?.excludeSourcePrefixes;
// #2646: keep soft-expired legacy rows (row_num NULL — never
// fence-owned) so a fence reconcile can't destroy forget_fact's
// legacy DB-only forget record.
const expiredLegacyFilter = opts?.preserveExpiredLegacy
? sql`AND NOT (row_num IS NULL AND expired_at IS NOT NULL)`
: sql``;
if (prefixes && prefixes.length > 0) {
// #1928: keep rows whose `source` matches an excluded prefix (e.g.
// `cli:` conversation facts). COALESCE so NULL/empty-source fence rows
@@ -4452,11 +4458,12 @@ export class PostgresEngine implements BrainEngine {
WHERE source_id = ${source_id}
AND source_markdown_slug = ${slug}
AND NOT (COALESCE(source, '') LIKE ANY(${patterns}))
${expiredLegacyFilter}
`;
return { deleted: result.count ?? 0 };
}
const result = await sql`
DELETE FROM facts WHERE source_id = ${source_id} AND source_markdown_slug = ${slug}
DELETE FROM facts WHERE source_id = ${source_id} AND source_markdown_slug = ${slug} ${expiredLegacyFilter}
`;
return { deleted: result.count ?? 0 };
}
+2 -1
View File
@@ -114,11 +114,12 @@ describe('gateway.isAvailable (silent-drop regression surface)', () => {
// #1135 — an explicit expansion_model pointed at a chat-capable
// OpenAI-compatible provider used to silently yield no expansion because
// the recipe declared no expansion touchpoint.
test('expansion available for chat-capable openai-compat providers (deepseek/groq/together)', () => {
test('expansion available for chat-capable openai-compat providers (deepseek/groq/together/openrouter)', () => {
const cases: Array<[string, Record<string, string>]> = [
['deepseek:deepseek-chat', { DEEPSEEK_API_KEY: 'fake' }],
['groq:llama-3.1-8b-instant', { GROQ_API_KEY: 'fake' }],
['together:meta-llama/Llama-3.3-70B-Instruct-Turbo', { TOGETHER_API_KEY: 'fake' }],
['openrouter:google/gemini-3-flash-preview', { OPENROUTER_API_KEY: 'fake' }],
];
for (const [model, env] of cases) {
resetGateway();
+12
View File
@@ -65,6 +65,18 @@ describe('recipe: openrouter', () => {
).not.toThrow();
});
test('3b. expansion reuses routed chat models and accepts arbitrary provider/model IDs', () => {
const r = getRecipe('openrouter')!;
expect(r.touchpoints.expansion).toBeDefined();
expect(r.touchpoints.expansion!.models.length).toBeGreaterThanOrEqual(3);
expect(() =>
assertTouchpoint(r, 'expansion', 'some/provider-model'),
).not.toThrow();
expect(() =>
assertTouchpoint(r, 'expansion', 'meta-llama/llama-future-2030'),
).not.toThrow();
});
test('4. chat models list — every entry matches provider/model shape (D5 regression)', () => {
// Codex correction: pinning specific slugs creates false confidence (the
// list is advisory; OR's catalog churns). The shape test catches the
+133 -25
View File
@@ -1,8 +1,10 @@
/**
* Serial (stubs globalThis.fetch): exercises the self-upgrade cache REFRESH
* orchestration end-to-end — `refreshUpdateCache()` fetches the latest release
* and writes the correct marker to the shared cache file that the CLI startup
* hook reads. Network is stubbed; the cache write + marker logic are real.
* orchestration end-to-end — `refreshUpdateCache()` resolves the latest version
* (from the VERSION file on master, #486 — the repo has zero GitHub releases,
* so the old `releases/latest` API path could never succeed) and writes the
* correct marker to the shared cache file that the CLI startup hook reads.
* Network is stubbed; the cache write + marker logic are real.
*
* Quarantined as *.serial.test.ts because it reassigns the process-global
* `fetch` (cross-file-unsafe under the parallel runner).
@@ -13,10 +15,11 @@ import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { VERSION } from '../src/version.ts';
import { parseSemver } from '../src/core/semver.ts';
import { readUpdateCache } from '../src/core/self-upgrade.ts';
import { refreshUpdateCache } from '../src/commands/check-update.ts';
import { readUpdateCache, writeUpdateCache } from '../src/core/self-upgrade.ts';
import { fetchLatestRelease, parseVersionFileBody, refreshUpdateCache, runCheckUpdate } from '../src/commands/check-update.ts';
const realFetch = globalThis.fetch;
const realLog = console.log;
let homeDir: string;
let priorHome: string | undefined;
@@ -27,14 +30,13 @@ function bump(kind: 'minor' | 'patch' | 'micro'): string {
return `${v[0]}.${v[1]}.${v[2]}.${v[3] + 1}`;
}
function stubReleaseFetch(tag: string | null, ok = true): void {
/** Stub the VERSION-file fetch. body === null → network throw. */
function stubVersionFetch(body: string | null, status = 200): void {
globalThis.fetch = (async (url: any) => {
const u = String(url);
if (u.includes('/releases/latest')) {
if (tag === null) throw new Error('network down');
return new Response(JSON.stringify({ tag_name: tag, published_at: '2026-01-01T00:00:00Z', html_url: 'https://x' }), {
status: ok ? 200 : 500,
});
if (u.includes('/gbrain/master/VERSION')) {
if (body === null) throw new Error('network down');
return new Response(body, { status });
}
// Changelog fetch (only happens when update available) — return empty.
return new Response('', { status: 200 });
@@ -49,49 +51,155 @@ beforeEach(() => {
afterEach(() => {
globalThis.fetch = realFetch;
console.log = realLog;
if (priorHome === undefined) delete process.env.GBRAIN_HOME;
else process.env.GBRAIN_HOME = priorHome;
rmSync(homeDir, { recursive: true, force: true });
});
describe('fetchLatestRelease — resolves from the VERSION file, discriminates failures', () => {
test('bare version body → ok with that tag', async () => {
stubVersionFetch('0.99.1.0\n');
expect(await fetchLatestRelease()).toMatchObject({ ok: true, tag: '0.99.1.0' });
});
test('network throw → network_error (NOT no_releases — offline users are not told "no releases exist")', async () => {
stubVersionFetch(null);
expect(await fetchLatestRelease()).toEqual({ ok: false, reason: 'network_error' });
});
test('HTTP 404 → no_releases', async () => {
stubVersionFetch('Not Found', 404);
expect(await fetchLatestRelease()).toEqual({ ok: false, reason: 'no_releases' });
});
test('garbage body → no_releases', async () => {
stubVersionFetch('<html>rate limited</html>');
expect(await fetchLatestRelease()).toEqual({ ok: false, reason: 'no_releases' });
});
});
describe('parseVersionFileBody — shape gate over the raw fetch body', () => {
test('trailing newline, v prefix, 3-segment legacy, suffix channel', () => {
expect(parseVersionFileBody('0.42.67.0\n')).toBe('0.42.67.0');
expect(parseVersionFileBody('v0.42.67.0')).toBe('0.42.67.0');
expect(parseVersionFileBody('0.31.3\n')).toBe('0.31.3'); // legacy 3-segment
expect(parseVersionFileBody('0.31.1.1-fixwave\n')).toBe('0.31.1.1'); // suffix compares as base
});
test('malformed / huge / injected bodies → null', () => {
expect(parseVersionFileBody('')).toBeNull();
expect(parseVersionFileBody('not a version')).toBeNull();
expect(parseVersionFileBody('$(rm -rf /)')).toBeNull();
expect(parseVersionFileBody('1.2')).toBeNull(); // 2-segment: not a gbrain version
expect(parseVersionFileBody('9'.repeat(10_000_000))).toBeNull(); // bounded, no blowup
});
});
describe('refreshUpdateCache — full refresh orchestration (network stubbed)', () => {
test('minor-bump release → writes upgrade_available marker', async () => {
test('minor-bump VERSION on master → writes upgrade_available marker', async () => {
const latest = bump('minor');
stubReleaseFetch(`v${latest}`);
stubVersionFetch(`${latest}\n`);
await refreshUpdateCache();
const entry = readUpdateCache();
expect(entry?.marker).toEqual({ kind: 'upgrade_available', current: VERSION, latest });
});
test('patch release → writes upgrade_available marker', async () => {
test('patch bump → writes upgrade_available marker', async () => {
const latest = bump('patch');
stubReleaseFetch(`v${latest}`);
stubVersionFetch(`${latest}\n`);
await refreshUpdateCache();
expect(readUpdateCache()?.marker).toEqual({ kind: 'upgrade_available', current: VERSION, latest });
});
test('micro release → writes upgrade_available marker', async () => {
test('micro bump → writes upgrade_available marker', async () => {
const latest = bump('micro');
stubReleaseFetch(`v${latest}`);
stubVersionFetch(`${latest}\n`);
await refreshUpdateCache();
expect(readUpdateCache()?.marker).toEqual({ kind: 'upgrade_available', current: VERSION, latest });
});
test('network failure → writes up_to_date marker (fail-open, TTL prevents hammering)', async () => {
stubReleaseFetch(null);
test('minor bump published as legacy 3-segment → still detected', async () => {
const v = parseSemver(VERSION)!;
const latest = `${v[0]}.${v[1] + 1}.0`; // 3-segment, no micro
stubVersionFetch(`${latest}\n`);
await refreshUpdateCache();
expect(readUpdateCache()?.marker).toEqual({ kind: 'upgrade_available', current: VERSION, latest });
});
test('suffix channel release (X.Y.Z.W-fixwave) → compares as numeric base', async () => {
const latest = bump('micro');
stubVersionFetch(`${latest}-fixwave\n`);
await refreshUpdateCache();
expect(readUpdateCache()?.marker).toEqual({ kind: 'upgrade_available', current: VERSION, latest });
});
test('same version on master → up_to_date marker', async () => {
stubVersionFetch(`${VERSION}\n`);
await refreshUpdateCache();
expect(readUpdateCache()?.marker).toEqual({ kind: 'up_to_date', current: VERSION });
});
test('non-OK HTTP → fail-open up_to_date', async () => {
stubReleaseFetch(`v${bump('minor')}`, false);
// The #486 bug class: a failed check must never fabricate "you're current".
test('network failure with NO prior cache → writes NOTHING (never a fabricated up_to_date)', async () => {
stubVersionFetch(null);
await refreshUpdateCache();
expect(readUpdateCache()?.marker).toEqual({ kind: 'up_to_date', current: VERSION });
expect(readUpdateCache()).toBeNull();
});
test('garbage tag → fail-open up_to_date (forged/invalid version never cached as upgrade)', async () => {
stubReleaseFetch('v$(rm -rf /)');
test('network failure with prior upgrade_available → pending notice PRESERVED, not erased', async () => {
const latest = bump('minor');
writeUpdateCache({ kind: 'upgrade_available', current: VERSION, latest });
stubVersionFetch(null);
await refreshUpdateCache();
expect(readUpdateCache()?.marker).toEqual({ kind: 'up_to_date', current: VERSION });
expect(readUpdateCache()?.marker).toEqual({ kind: 'upgrade_available', current: VERSION, latest });
});
test('non-OK HTTP → no fabricated up_to_date', async () => {
stubVersionFetch('nope', 500);
await refreshUpdateCache();
expect(readUpdateCache()).toBeNull();
});
test('garbage body → no fabricated marker (forged/invalid version never cached as upgrade)', async () => {
stubVersionFetch('$(rm -rf /)');
await refreshUpdateCache();
expect(readUpdateCache()).toBeNull();
});
});
describe('runCheckUpdate --json — failure discrimination (#486)', () => {
function capture(): string[] {
const lines: string[] = [];
console.log = (...a: unknown[]) => { lines.push(a.join(' ')); };
return lines;
}
test('offline → error: network_error (not "no releases exist")', async () => {
stubVersionFetch(null);
const lines = capture();
await runCheckUpdate(['--json']);
const out = JSON.parse(lines.join('\n'));
expect(out.error).toBe('network_error');
expect(out.update_available).toBe(false);
expect(readUpdateCache()).toBeNull(); // and no fabricated up_to_date cache
});
test('endpoint answers but no usable version → error: no_releases', async () => {
stubVersionFetch('garbage');
const lines = capture();
await runCheckUpdate(['--json']);
expect(JSON.parse(lines.join('\n')).error).toBe('no_releases');
});
test('newer VERSION on master → update_available true with latest_version set', async () => {
const latest = bump('minor');
stubVersionFetch(`${latest}\n`);
const lines = capture();
await runCheckUpdate(['--json']);
const out = JSON.parse(lines.join('\n'));
expect(out.update_available).toBe(true);
expect(out.latest_version).toBe(latest);
expect(readUpdateCache()?.marker).toEqual({ kind: 'upgrade_available', current: VERSION, latest });
});
});
+7
View File
@@ -35,6 +35,13 @@ describe('isNewerVersion', () => {
expect(isNewerVersion('0.42.66.0', '0.42.66.1')).toBe(true);
});
test('orders legacy 3-segment against 4-segment: 0.42.67.0 > 0.42.66.1 > 0.42.66', () => {
expect(isNewerVersion('0.42.66.1', '0.42.67.0')).toBe(true);
expect(isNewerVersion('0.42.66', '0.42.66.1')).toBe(true);
expect(isNewerVersion('0.42.66', '0.42.66.0')).toBe(false); // 3-segment == its .0 micro
expect(isNewerVersion('0.42.67.0', '0.42.66.1')).toBe(false);
});
test('rejects equal, older, and malformed versions', () => {
expect(isNewerVersion('0.42.66.0', '0.42.66.0')).toBe(false);
expect(isNewerVersion('0.42.66.1', '0.42.66.0')).toBe(false);
+97
View File
@@ -0,0 +1,97 @@
/**
* capOversizedChunks — CJK-aware oversize measurement (follow-up to #1675,
* shape requested in #3475's closing review).
*
* cl100k (estimateTokens) matches embedding-family tokenizers on pure-ASCII
* source (measured identical on English prose and JSON vs Qwen3-Embedding),
* but undercounts MIXED CJK+ASCII chunks — measured 31% on URL-dense Korean
* text (#2826's failure shape). estimateEmbedTokens lifts only that class:
* ASCII-only input short-circuits to estimateTokens verbatim, and max()
* keeps CJK-dominant text at the cl100k count it gets today.
*/
import { describe, test, expect } from 'bun:test';
import { chunkCodeText, estimateTokens, estimateEmbedTokens } from '../../src/core/chunkers/code.ts';
/** URL-dense Korean rollup lines — the measured 31% divergence shape. */
function urlDenseKoreanMix(lines: number): string {
return Array.from({ length: lines }, (_, i) =>
`- 항목 ${i}: 검증용 한국어 문장 · 링크: https://docs.example.com/pages/${String(i).padStart(32, '0')}?v=abcdef0123456789&ref=sample`,
).join('\n');
}
function bigJsonWithKoreanValues(targetChars: number): string {
const entries: string[] = [];
let i = 0;
let len = 0;
while (len < targetChars) {
const row =
` "item_${i}": { "name": "예시-${i}", "url": "https://example.com/api/v2/items/${i}?token=abc${i}def", "qty": ${i % 100}, "memo": "한국어 값이 섞인 예시 데이터" }`;
entries.push(row);
len += row.length;
i++;
}
return `{\n${entries.join(',\n')}\n}`;
}
describe('estimateEmbedTokens — measurement gate', () => {
test('ASCII-only input is bit-identical to estimateTokens (no CJK → short-circuit)', () => {
const en = 'function ordinary() { return compute(42) + helper(); } '.repeat(80);
const json = '{"item": {"name": "sample", "url": "https://example.com/a?b=c", "qty": 42}}, '.repeat(60);
expect(estimateEmbedTokens(en)).toBe(estimateTokens(en));
expect(estimateEmbedTokens(json)).toBe(estimateTokens(json));
});
test('never estimates below estimateTokens (max composition)', () => {
for (const s of [urlDenseKoreanMix(20), '이 문장은 순수 한국어 산문 예시입니다. '.repeat(40), 'plain ascii ', '']) {
expect(estimateEmbedTokens(s)).toBeGreaterThanOrEqual(estimateTokens(s));
}
});
test('mixed CJK+ASCII (the measured divergence class) estimates strictly higher', () => {
const mix = urlDenseKoreanMix(20);
// Real Qwen3-Embedding count for this shape measures ~45% ABOVE cl100k;
// the weighted form stays above the real count (+15% measured margin).
expect(estimateEmbedTokens(mix)).toBeGreaterThan(estimateTokens(mix));
});
});
describe('capOversizedChunks with the CJK-aware estimate', () => {
test('oversized json fence with Korean values re-splits under the default cap', async () => {
const src = bigJsonWithKoreanValues(14_000);
const chunks = await chunkCodeText(src, 'fence.json');
expect(chunks.length).toBeGreaterThan(1);
for (const c of chunks) {
// Small slack for the "[JSON] fence.json:…" header buildChunk re-adds
// after the body-level split.
expect(estimateEmbedTokens(c.text)).toBeLessThanOrEqual(2000 + 60);
}
// Content preserved — spot-check first / last entries survive.
const joined = chunks.map((c) => c.text).join('\n');
expect(joined).toContain('"item_0"');
expect(joined).toContain('한국어 값이 섞인 예시 데이터');
});
test('hard-split fallback makes progress on whitespace-less CJK-mixed input and stays under cap', async () => {
const blob = '한a민b국c'.repeat(3_000); // 18K chars, no whitespace
const chunks = await chunkCodeText(`{"blob": "${blob}"}`, 'fence.json');
expect(chunks.length).toBeGreaterThan(1);
for (const c of chunks) {
expect(estimateEmbedTokens(c.text)).toBeLessThanOrEqual(2000 + 60);
}
});
test('pure-ASCII chunks are measured by the identical estimator (cap decisions unchanged)', async () => {
const entries = Array.from({ length: 120 }, (_, i) =>
` "item_${i}": { "name": "sample-${i}", "url": "https://example.com/api/v2/items/${i}?token=abc${i}def", "qty": ${i % 100} }`,
);
const src = `{\n${entries.join(',\n')}\n}`;
const chunks = await chunkCodeText(src, 'fence.json');
expect(chunks.length).toBeGreaterThan(0);
for (const c of chunks) {
// For ASCII-only chunks the two estimators are identical (pinned
// above), so cap decisions — and therefore boundaries — are unchanged.
expect(estimateEmbedTokens(c.text)).toBe(estimateTokens(c.text));
}
});
});
@@ -73,3 +73,90 @@ describe.skipIf(skip)('facts-fence escaped-pipe reconciliation on Postgres', ()
]);
}, 30_000);
});
describe.skipIf(skip)('deleteFactsForPage preserveExpiredLegacy on Postgres (#2646)', () => {
// The PGLite side of this contract is pinned by
// test/extract-facts-phase.test.ts; this pins the postgres.js
// tagged-fragment SQL (the two branches interpolate `expiredLegacyFilter`
// differently) AND the returned delete count on a real Postgres.
const slug = 'people/expired-legacy-preserve-example';
let engine: PostgresEngine;
beforeAll(async () => {
engine = new PostgresEngine();
await engine.connect({ database_url: databaseUrl! });
await engine.initSchema();
});
afterAll(async () => {
if (engine) {
await engine.executeRaw('DELETE FROM facts WHERE source_markdown_slug = $1', [slug]);
await engine.executeRaw('DELETE FROM pages WHERE slug = $1', [slug]);
await engine.disconnect();
}
});
async function seedRows(): Promise<void> {
await engine.executeRaw('DELETE FROM facts WHERE source_markdown_slug = $1', [slug]);
// One fence-owned active row (deletable) + one soft-expired legacy row
// (row_num NULL, expired_at set — forget_fact's record, must survive).
await engine.executeRaw(
`INSERT INTO facts (source_id, entity_slug, fact, kind, visibility, notability,
valid_from, source, confidence, row_num, expired_at, source_markdown_slug)
VALUES
('default', $1, 'fence-owned active fact', 'fact', 'world', 'high',
now(), 'fence:reconcile', 1.0, 1, NULL, $1),
('default', $1, 'forgotten legacy claim', 'fact', 'private', 'medium',
now(), 'mcp:put_page', 1.0, NULL, now(), $1)`,
[slug],
);
}
test('no-prefix branch: expired legacy row survives, count reflects only real deletions', async () => {
await seedRows();
const { deleted } = await engine.deleteFactsForPage(slug, 'default', {
preserveExpiredLegacy: true,
});
expect(deleted).toBe(1); // only the fence-owned row
const rows = await engine.executeRaw<{ fact: string }>(
'SELECT fact FROM facts WHERE source_markdown_slug = $1', [slug],
);
expect(Array.from(rows).map(r => r.fact)).toEqual(['forgotten legacy claim']);
}, 30_000);
test('prefix branch: excludeSourcePrefixes and preserveExpiredLegacy compose', async () => {
await seedRows();
// Add a cli:-origin row that the prefix exclusion must protect.
await engine.executeRaw(
`INSERT INTO facts (source_id, entity_slug, fact, kind, visibility, notability,
valid_from, source, confidence, row_num, expired_at, source_markdown_slug)
VALUES ('default', $1, 'conversation fact', 'fact', 'private', 'medium',
now(), 'cli:extract-conversation-facts', 1.0, NULL, NULL, $1)`,
[slug],
);
const { deleted } = await engine.deleteFactsForPage(slug, 'default', {
excludeSourcePrefixes: ['cli:'],
preserveExpiredLegacy: true,
});
expect(deleted).toBe(1); // only the fence-owned row
const rows = await engine.executeRaw<{ fact: string }>(
'SELECT fact FROM facts WHERE source_markdown_slug = $1 ORDER BY id', [slug],
);
expect(Array.from(rows).map(r => r.fact)).toEqual([
'forgotten legacy claim',
'conversation fact',
]);
}, 30_000);
test('omitted option keeps legacy wipe behavior (expired row IS deleted)', async () => {
await seedRows();
const { deleted } = await engine.deleteFactsForPage(slug, 'default');
expect(deleted).toBe(2);
const rows = await engine.executeRaw<{ fact: string }>(
'SELECT fact FROM facts WHERE source_markdown_slug = $1', [slug],
);
expect(Array.from(rows)).toHaveLength(0);
}, 30_000);
});
+218
View File
@@ -231,6 +231,44 @@ describe('runExtractFacts — happy path', () => {
expect(rows.rows[0].fact).toBe('A');
});
test('malformed fence rows make the page non-authoritative and preserve its indexed facts', async () => {
await putPage('people/alice', FACT_FENCE(
`| 1 | A | fact | 1.0 | world | medium | 2026-01-01 | | s | |
| 2 | B | fact | 1.0 | world | medium | 2026-01-01 | | s | |`,
));
await runExtractFacts(engine, { slugs: ['people/alice'] });
// A hand edit corrupts row 2. The parser can still recover row 1, but
// that partial result is not an authoritative replacement for the page.
await putPage('people/alice', FACT_FENCE(
`| 1 | A | fact | 1.0 | world | medium | 2026-01-01 | | s | |
| 2 | B | bogus | 1.0 | world | medium | 2026-01-01 | | s | |`,
));
await putPage('people/bob', FACT_FENCE(
`| 1 | Clean | fact | 1.0 | world | medium | 2026-01-01 | | s | |`,
));
const r = await runExtractFacts(engine, { slugs: ['people/alice', 'people/bob'] });
expect(r.warnings.some(w => w.includes('FACTS_TABLE_MALFORMED'))).toBe(true);
expect(r.pagesScanned).toBe(2);
expect(r.factsInserted).toBe(1);
expect(r.factsDeleted).toBe(0);
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const rows = await (engine as any).db.query(
`SELECT fact FROM facts WHERE source_markdown_slug = 'people/alice' ORDER BY row_num`,
);
expect(rows.rows.map((row: { fact: string }) => row.fact)).toEqual(['A', 'B']);
// A warning is page-local: clean pages in the same cycle still reconcile.
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const cleanRows = await (engine as any).db.query(
`SELECT fact FROM facts WHERE source_markdown_slug = 'people/bob'`,
);
expect(cleanRows.rows.map((row: { fact: string }) => row.fact)).toEqual(['Clean']);
});
test('page with no facts fence → DB facts for that page wiped (empty fence reconciles to empty index)', async () => {
await putPage('people/alice', FACT_FENCE(
`| 1 | seeded | fact | 1.0 | world | medium | 2026-01-01 | | s | |`,
@@ -334,6 +372,186 @@ describe('runExtractFacts — empty-fence guard (Codex R2-#7)', () => {
expect(r.factsInserted).toBe(1);
});
test('soft-expired legacy rows do NOT trigger the guard (#2646 — forget_fact drains the backlog)', async () => {
// A legacy row that forget_fact already soft-expired. Before #2646
// the guard counted it forever: apply-migrations no-ops (migration
// marked applied) and forget_fact only sets expired_at, so the
// phase was permanently blocked with no sanctioned way out.
// eslint-disable-next-line @typescript-eslint/no-explicit-any
await (engine as any).db.query(
`INSERT INTO facts (source_id, entity_slug, fact, kind, visibility, notability,
valid_from, source, confidence, expired_at)
VALUES ('default', 'people/alice', 'forgotten legacy claim', 'fact', 'private', 'medium',
now(), 'mcp:put_page', 1.0, now())`,
);
await putPage('people/alice', FACT_FENCE(
`| 1 | new fact | fact | 1.0 | world | high | 2026-01-01 | | s | |`,
));
const r = await runExtractFacts(engine, { slugs: ['people/alice'] });
expect(r.guardTriggered).toBe(false);
expect(r.legacyRowsPending).toBe(0);
expect(r.factsInserted).toBe(1);
// The expired legacy row itself is untouched (soft-expire is the
// record of the forget; the phase must not hard-delete it).
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const rows = await (engine as any).db.query(
`SELECT fact FROM facts WHERE row_num IS NULL AND expired_at IS NOT NULL`,
);
expect(rows.rows).toHaveLength(1);
expect(rows.rows[0].fact).toBe('forgotten legacy claim');
});
test('expired legacy row WITH source_markdown_slug set survives reconcile untouched (#2646 codex P2)', async () => {
// Hybrid shape: row_num NULL (legacy — never fence-owned) but
// source_markdown_slug matching a live page. Without the
// preserveExpiredLegacy filter, the reconcile pass would count it
// as "stale", trigger a wipe, hard-delete the forget record, and
// reinsert the fence's rows fresh — reviving a forgotten claim as
// an active fact.
// eslint-disable-next-line @typescript-eslint/no-explicit-any
await (engine as any).db.query(
`INSERT INTO facts (source_id, entity_slug, fact, kind, visibility, notability,
valid_from, source, confidence, expired_at, source_markdown_slug)
VALUES ('default', 'people/alice', 'forgotten hybrid claim', 'fact', 'private', 'medium',
now(), 'mcp:put_page', 1.0, now(), 'people/alice')`,
);
await putPage('people/alice', FACT_FENCE(
`| 1 | fence fact | fact | 1.0 | world | high | 2026-01-01 | | s | |`,
));
const r1 = await runExtractFacts(engine, { slugs: ['people/alice'] });
const r2 = await runExtractFacts(engine, { slugs: ['people/alice'] });
expect(r1.guardTriggered).toBe(false);
// The expired hybrid row is invisible to the reconcile: the fence
// fact inserts normally, nothing is wiped, and re-running stays
// idempotent (the hybrid row must not read as perpetually stale).
expect(r1.factsInserted).toBe(1);
expect(r1.factsDeleted).toBe(0);
expect(r2.factsInserted).toBe(0);
expect(r2.factsDeleted).toBe(0);
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const rows = await (engine as any).db.query(
`SELECT fact, expired_at FROM facts WHERE source_markdown_slug = 'people/alice' ORDER BY id`,
);
expect(rows.rows).toHaveLength(2);
expect(rows.rows[0].fact).toBe('forgotten hybrid claim');
expect(rows.rows[0].expired_at).not.toBeNull();
expect(rows.rows[1].fact).toBe('fence fact');
expect(rows.rows[1].expired_at).toBeNull();
});
test('expired legacy hybrid row survives even a stale-row wipe on the same page (#2646 codex P2)', async () => {
// Force the wipe path: seed a fence, reconcile, then change the
// fence so the old DB row goes stale. The wipe must delete the
// stale fence-owned row but preserve the expired legacy hybrid.
// eslint-disable-next-line @typescript-eslint/no-explicit-any
await (engine as any).db.query(
`INSERT INTO facts (source_id, entity_slug, fact, kind, visibility, notability,
valid_from, source, confidence, expired_at, source_markdown_slug)
VALUES ('default', 'people/alice', 'forgotten hybrid claim', 'fact', 'private', 'medium',
now(), 'mcp:put_page', 1.0, now(), 'people/alice')`,
);
await putPage('people/alice', FACT_FENCE(
`| 1 | old fact | fact | 1.0 | world | high | 2026-01-01 | | s | |`,
));
await runExtractFacts(engine, { slugs: ['people/alice'] });
// Replace the fence content — 'old fact' is now stale in the DB.
await putPage('people/alice', FACT_FENCE(
`| 1 | replacement fact | fact | 1.0 | world | high | 2026-01-01 | | s | |`,
));
const r = await runExtractFacts(engine, { slugs: ['people/alice'] });
expect(r.factsDeleted).toBe(1); // only the stale fence-owned row
expect(r.factsInserted).toBe(1);
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const rows = await (engine as any).db.query(
`SELECT fact FROM facts WHERE source_markdown_slug = 'people/alice' ORDER BY id`,
);
expect(rows.rows.map((row: { fact: string }) => row.fact))
.toEqual(['forgotten hybrid claim', 'replacement fact']);
});
test('fence claim matching an expired legacy row is inserted active — fence is canonical (#2646)', async () => {
// Deliberate semantics, pinned: legacy DB-only forgets are
// documented NOT to survive rebuild (forget.ts header — the
// explicit DB-only exception). When the fence still carries the
// same (claim, source), the reconcile inserts a fresh active
// fence-owned row; the expired legacy row survives alongside as
// the record of the earlier forget. Suppressing the insert would
// create silent fence↔DB divergence ("0 facts" while the fence
// says otherwise) — the exact failure mode the guard prevents.
// To durably forget, forget the fence-owned row (fence path).
// eslint-disable-next-line @typescript-eslint/no-explicit-any
await (engine as any).db.query(
`INSERT INTO facts (source_id, entity_slug, fact, kind, visibility, notability,
valid_from, source, confidence, expired_at, source_markdown_slug)
VALUES ('default', 'people/alice', 'shared claim', 'fact', 'private', 'medium',
now(), 's', 1.0, now(), 'people/alice')`,
);
await putPage('people/alice', FACT_FENCE(
`| 1 | shared claim | fact | 1.0 | world | high | 2026-01-01 | | s | |`,
));
const r1 = await runExtractFacts(engine, { slugs: ['people/alice'] });
const r2 = await runExtractFacts(engine, { slugs: ['people/alice'] });
expect(r1.factsInserted).toBe(1);
expect(r1.factsDeleted).toBe(0);
// Idempotent thereafter — the coexisting pair is stable state.
expect(r2.factsInserted).toBe(0);
expect(r2.factsDeleted).toBe(0);
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const rows = await (engine as any).db.query(
`SELECT fact, row_num, expired_at FROM facts
WHERE source_markdown_slug = 'people/alice' ORDER BY id`,
);
expect(rows.rows).toHaveLength(2);
expect(rows.rows[0]).toMatchObject({ fact: 'shared claim', row_num: null });
expect(rows.rows[0].expired_at).not.toBeNull(); // forget record preserved
expect(rows.rows[1]).toMatchObject({ fact: 'shared claim', row_num: 1 });
expect(rows.rows[1].expired_at).toBeNull(); // fence-canonical active row
});
test('mixed active + expired legacy rows: guard counts only the active ones (#2646)', async () => {
// One active legacy row + one soft-expired legacy row. The guard
// must still trigger (an active row is pending backfill) but the
// pending count must exclude the expired row — so each forget_fact
// visibly drains the counter toward release.
// eslint-disable-next-line @typescript-eslint/no-explicit-any
await (engine as any).db.query(
`INSERT INTO facts (source_id, entity_slug, fact, kind, visibility, notability,
valid_from, source, confidence, expired_at)
VALUES
('default', 'people/alice', 'active legacy claim', 'fact', 'private', 'medium',
now(), 'mcp:put_page', 1.0, NULL),
('default', 'people/alice', 'expired legacy claim', 'fact', 'private', 'medium',
now(), 'mcp:put_page', 1.0, now())`,
);
await putPage('people/alice', FACT_FENCE(
`| 1 | new fact | fact | 1.0 | world | high | 2026-01-01 | | s | |`,
));
const r = await runExtractFacts(engine, { slugs: ['people/alice'] });
expect(r.guardTriggered).toBe(true);
expect(r.legacyRowsPending).toBe(1);
expect(r.factsInserted).toBe(0);
expect(r.factsDeleted).toBe(0);
});
test('NULL entity_slug legacy rows do NOT trigger the guard (they are structurally unfenceable)', async () => {
// eslint-disable-next-line @typescript-eslint/no-explicit-any
await (engine as any).db.query(
+18
View File
@@ -256,6 +256,11 @@ test('applyForwardReferenceBootstrap covers every forward reference declared in
ALTER TABLE pages DROP COLUMN IF EXISTS generation;
ALTER TABLE pages DROP COLUMN IF EXISTS contextual_retrieval_mode;
ALTER TABLE pages DROP COLUMN IF EXISTS corpus_generation;
DROP INDEX IF EXISTS idx_timeline_event_dedup;
DROP INDEX IF EXISTS idx_timeline_event_page;
ALTER TABLE timeline_entries DROP CONSTRAINT IF EXISTS timeline_entries_event_page_id_fkey;
ALTER TABLE timeline_entries DROP COLUMN IF EXISTS event_page_id;
`);
// Note: we don't strip sources.archived* here because they're inline in the
@@ -264,6 +269,14 @@ test('applyForwardReferenceBootstrap covers every forward reference declared in
// The bootstrap's needsPagesBootstrap branch recreates sources without the
// archive columns; the new needsSourcesArchive probe adds them.
const { rows: preBootstrapTimelineEventPageId } = await db.query(`
SELECT 1 FROM information_schema.columns
WHERE table_schema = 'public'
AND table_name = 'timeline_entries'
AND column_name = 'event_page_id'
`);
expect(preBootstrapTimelineEventPageId).toHaveLength(0);
// Run bootstrap in isolation (NOT initSchema). This is what we're testing.
await (engine as any).applyForwardReferenceBootstrap();
@@ -328,6 +341,11 @@ test('after bootstrap, PGLITE_SCHEMA_SQL replays without crashing on missing for
ALTER TABLE pages DROP COLUMN IF EXISTS import_filename;
ALTER TABLE pages DROP COLUMN IF EXISTS salience_touched_at;
ALTER TABLE pages DROP COLUMN IF EXISTS emotional_weight;
DROP INDEX IF EXISTS idx_timeline_event_dedup;
DROP INDEX IF EXISTS idx_timeline_event_page;
ALTER TABLE timeline_entries DROP CONSTRAINT IF EXISTS timeline_entries_event_page_id_fkey;
ALTER TABLE timeline_entries DROP COLUMN IF EXISTS event_page_id;
`);
// Bootstrap, then schema replay. Either step crashing fails the test.
+3 -3
View File
@@ -31,9 +31,9 @@ function microBump(): string {
function stub(tag: string | null, changelog: string): void {
globalThis.fetch = (async (url: any) => {
const u = String(url);
if (u.includes('/releases/latest')) {
if (u.includes('/gbrain/master/VERSION')) {
if (tag === null) throw new Error('network down');
return new Response(JSON.stringify({ tag_name: tag, published_at: '2026-01-01', html_url: 'https://x/rel' }), { status: 200 });
return new Response(tag + '\n', { status: 200 });
}
if (u.includes('CHANGELOG.md')) return new Response(changelog, { status: 200 });
return new Response('', { status: 200 });
@@ -65,7 +65,7 @@ describe('self-upgrade --check-only surfaces what you get', () => {
const out = JSON.parse(captured.join('\n'));
expect(out.update_available).toBe(true);
expect(out.latest_version).toBe(latest);
expect(out.release_url).toBe('https://x/rel');
expect(out.release_url).toBe('https://github.com/garrytan/gbrain/blob/master/CHANGELOG.md');
expect(out.changelog_diff).toContain('Shiny new thing');
});