mirror of
https://github.com/garrytan/gbrain.git
synced 2026-08-14 08:53:22 +00:00
Compare commits
7
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
dc0504785a | ||
|
|
6920744dd8 | ||
|
|
3df20f9f18 | ||
|
|
e58abd652c | ||
|
|
a104f98dca | ||
|
|
2ac6959b46 | ||
|
|
18ec732e1b |
@@ -0,0 +1,16 @@
|
||||
# Line-ending policy.
|
||||
#
|
||||
# Shell scripts MUST be checked out with LF endings on every platform.
|
||||
# Git for Windows installs with `core.autocrlf=true` by default, which
|
||||
# rewrites LF -> CRLF on checkout. A strict bash (WSL, Linux CI, macOS)
|
||||
# then chokes on the trailing CR:
|
||||
#
|
||||
# scripts/run-unit-parallel.sh: line 23: $'\r': command not found
|
||||
# scripts/run-unit-parallel.sh: line 24: set: pipefail : invalid option name
|
||||
# scripts/run-unit-parallel.sh: line 32: syntax error near unexpected token `$'{\r''
|
||||
#
|
||||
# That silently disabled `bun run test`, `bun run verify`, `bun run ci:local`
|
||||
# and `bun run test:e2e` for Windows contributors, since all four dispatch
|
||||
# through bash. `eol=lf` pins the checkout regardless of the user's
|
||||
# core.autocrlf setting.
|
||||
*.sh text eol=lf
|
||||
@@ -2,6 +2,45 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.42.67.0] - 2026-07-28
|
||||
|
||||
**If you develop GBrain on Windows, the test and check commands now actually run. Until this release they were quietly doing almost nothing.**
|
||||
|
||||
`bun run test`, `bun run verify`, `bun run ci:local` and `bun run test:e2e` all hand off to shell scripts, and on Windows that hand-off was broken in two separate places. The commands did not stop with an obvious error. They reported a result, so a run could look finished when barely any of the checks had actually inspected anything. On a clean Windows clone, `bun run verify` got 1 check to pass and 31 to fail. It now gets 25 to pass and 7 to fail, and none of the 7 are caused by this change.
|
||||
|
||||
The first problem was line endings. Git for Windows installs with `core.autocrlf=true`, which rewrites shell scripts to Windows line endings when you clone or check out. Bash refuses to run those, so a script died on its second line before doing any work. The scripts stored in the repository were always correct; only the copy on your disk was wrong. A new `.gitattributes` pins every `.sh` file to Unix line endings at checkout, no matter how your Git is configured.
|
||||
|
||||
The second problem was how the checks were started. Thirty three of them pointed straight at a `.sh` file. On macOS and Linux the shell reads the `#!/usr/bin/env bash` line at the top of the script and runs it correctly. Bun on Windows does not do that, so those commands failed the moment they were called. They now go through `bash` explicitly, the same way the other eleven were already written.
|
||||
|
||||
Nothing changes for macOS and Linux. No stored file content moves, and no check behaves differently on those platforms.
|
||||
|
||||
## To take advantage of v0.42.67.0
|
||||
|
||||
Only Windows contributors need to do anything, and only once. `.gitattributes` applies at checkout time, so shell scripts already sitting on your disk keep their old line endings until you refresh them.
|
||||
|
||||
1. **Refresh the working copy** from the repository root:
|
||||
```bash
|
||||
git rm --cached -r . -q
|
||||
git reset --hard
|
||||
```
|
||||
2. **Confirm bash can read the scripts:**
|
||||
```bash
|
||||
bash -n scripts/run-unit-parallel.sh
|
||||
```
|
||||
Silence means it worked. `$'\r': command not found` means step 1 did not take effect.
|
||||
3. **Run the gate:**
|
||||
```bash
|
||||
bun run verify
|
||||
```
|
||||
|
||||
### Itemized changes
|
||||
|
||||
- New root `.gitattributes` pins `*.sh text eol=lf`, so shell scripts check out with Unix line endings regardless of the contributor's `core.autocrlf` setting. All 59 tracked `.sh` files were already stored with Unix endings, so `git add --renormalize .` reports nothing to do and no stored content changes.
|
||||
- `package.json` now routes the remaining 33 `.sh` check commands through `bash`, matching the 11 that already did. Every tracked `.sh` file carries a bash shebang (52 `#!/usr/bin/env bash` and 7 `#!/bin/bash`), so the treatment is uniform across all of them.
|
||||
- The five `scripts/*.ts` entries still run under bun and are untouched.
|
||||
- `CONTRIBUTING.md` gains a Windows section covering the one-time working-copy refresh and the `bash scripts/<name>.sh` convention for new checks.
|
||||
- `docs/TESTING.md` records how the test commands dispatch through bash, and notes that three tree-walking checks plus `typecheck` can exceed the 120s per-check cap on Windows while passing on Linux and macOS.
|
||||
|
||||
## [0.42.66.1] - 2026-07-27
|
||||
|
||||
### Fixed
|
||||
|
||||
@@ -11,6 +11,28 @@ bun test
|
||||
|
||||
Requires Bun 1.0+.
|
||||
|
||||
### Windows
|
||||
|
||||
`bun run test`, `verify`, `ci:local` and `test:e2e` all dispatch through bash, so
|
||||
the shell scripts under `scripts/` must be checked out with Unix line endings.
|
||||
The root `.gitattributes` pins `*.sh text eol=lf`, which overrides the
|
||||
`core.autocrlf=true` that Git for Windows installs by default. A fresh clone is
|
||||
correct with no extra steps.
|
||||
|
||||
If you cloned before that pin existed, your working copy still has the old
|
||||
Windows line endings and bash will fail with `$'\r': command not found`. Refresh
|
||||
it once, from the repository root:
|
||||
|
||||
```bash
|
||||
git rm --cached -r . -q
|
||||
git reset --hard
|
||||
bash -n scripts/run-unit-parallel.sh # silence means bash can read the scripts
|
||||
```
|
||||
|
||||
Every `check:*` entry in `package.json` invokes its script as `bash scripts/<name>.sh`
|
||||
rather than relying on the shebang, because bun on Windows cannot exec a `.sh`
|
||||
directly. Keep that prefix when you add a new shell-script check.
|
||||
|
||||
## Project structure
|
||||
|
||||
```
|
||||
|
||||
@@ -1,5 +1,25 @@
|
||||
# TODOS
|
||||
|
||||
## v0.42.67.0 follow-ups (Windows build tooling)
|
||||
|
||||
Filed as follow-ups from v0.42.67.0 (`.gitattributes` LF pin for `*.sh` +
|
||||
`bash` prefix on the 33 `package.json` check commands). Both items are newly
|
||||
observable: before that release these checks never executed on Windows at all,
|
||||
so nothing about their runtime was measurable.
|
||||
|
||||
- [ ] **P2 — three guard scripts exceed the 120s `run-verify-parallel.sh` cap on Windows.**
|
||||
With the dispatch fixed, `bun run verify` on Windows gets 25 passes and 7 failures, and
|
||||
`check:privacy`, `check:test-names` and `check:test-isolation` are timeouts rather than
|
||||
real failures (they pass on Linux and macOS well inside the cap). They walk the tree with
|
||||
per-file shell loops, which is far slower under Windows process creation. Either raise the
|
||||
cap for these three, or replace the per-file loop with a single `grep -r` pass. Same cap
|
||||
swallows `typecheck`, though standalone `bun run typecheck` exits 0.
|
||||
- [ ] **P3 — `check:wasm` cannot create its `node_modules` symlink on Windows.**
|
||||
`scripts/check-wasm-embedded.sh` fails with `ln: failed to create symbolic link
|
||||
'/tmp/gbrain-wasm-check.XXXX/node_modules': No such file or directory`. Unprivileged
|
||||
Windows accounts cannot create symlinks without developer mode. Consider a junction, a
|
||||
copy, or skipping the check with a clear message when symlink creation is unavailable.
|
||||
|
||||
## community fix-wave follow-ups (filed v0.42.60.0)
|
||||
|
||||
- [x] **P2 — cherry-pick #2112's uncovered doctor.ts hunk.** Fix-wave A (#2820) superseded
|
||||
@@ -62,17 +82,20 @@ Deferred from the provider-agnostic plumbing wave (#1249/#1250/#1292/#2271/#2209
|
||||
Plan + review trail at `~/.claude/plans/system-instruction-you-are-working-keen-newell.md`.
|
||||
The eng-review + Codex outside-voice narrowed the wave to these deferrals:
|
||||
|
||||
- [ ] **P2 — Capability-aware query expansion on OpenAI-compat providers (#2372).**
|
||||
- [x] **P2 — Capability-aware query expansion on OpenAI-compat providers (#2372).**
|
||||
Expansion only runs for recipes that declare an `expansion` touchpoint, and only the
|
||||
native providers (anthropic/openai/google) do. To make expansion work on
|
||||
litellm/openrouter/groq/together/deepseek you must ADD expansion touchpoints to those
|
||||
chat-capable recipes AND add a `generateObject`→`generateText` capability fallback for
|
||||
backends without strict structured outputs. Feature-shaped; overlaps the general
|
||||
OpenAI-compat proxy story (`docs/designs/COMMUNITY_IDEAS.md`). Community PR #2373 is a
|
||||
starting point. Where: `src/core/ai/gateway.ts:expand`, recipe files, `types.ts` (ExpansionTouchpoint).
|
||||
- [ ] **P2 — LiteLLM as a chat/expansion backend.** `litellm-proxy` declares ONLY an
|
||||
starting point. Implemented by #2373 plus the DeepSeek/Groq/Together recipe wave,
|
||||
LiteLLM chat/expansion support, and the OpenRouter expansion touchpoint. Where:
|
||||
`src/core/ai/gateway.ts:expand`, recipe files, `types.ts` (ExpansionTouchpoint).
|
||||
- [x] **P2 — LiteLLM as a chat/expansion backend.** `litellm-proxy` declares ONLY an
|
||||
embedding touchpoint, so `think`/chat on LiteLLM is dead. Add chat (and expansion) so a
|
||||
LiteLLM proxy is a full LLM backend, not embedding-only. The general OpenAI-compat proxy story.
|
||||
LiteLLM proxy is a full LLM backend, not embedding-only. Implemented by #2208.
|
||||
The general OpenAI-compat proxy story.
|
||||
- [ ] **P3 — Per-model embedding dims metadata on `EmbeddingTouchpoint`.** `default_dims`
|
||||
is recipe-wide, so a recipe (ollama) can't carry different native dims per model. This
|
||||
wave added the modern ollama model NAMES + a `trust_custom_dims` passthrough (user supplies
|
||||
|
||||
@@ -19,6 +19,29 @@ Seven test command tiers, each with a clear scope:
|
||||
| `bun run test:e2e` | Real Postgres E2E. Requires Docker + `DATABASE_URL`. Sequential. | ~5-10min | Pre-ship; nightly. |
|
||||
| `bun run check:all` | The historical pre-check scripts (22, chained sequentially in package.json). Overlaps `verify` heavily but is NOT a superset — `verify`'s `CHECKS` array in `scripts/run-verify-parallel.sh` (~30 entries incl. typecheck) is the authoritative gate; `check:all` keeps a few local-only extras (trailing-newline, exports-count, no-legacy-getconnection). | ~10s | Local-only sweep for the extras. |
|
||||
|
||||
### Shell dispatch and Windows
|
||||
|
||||
All four of `test`, `verify`, `ci:local` and `test:e2e` hand off to shell scripts
|
||||
under `scripts/`, so every `check:*` entry in `package.json` invokes its script as
|
||||
`bash scripts/<name>.sh` instead of relying on the shebang — bun on Windows cannot
|
||||
exec a `.sh` directly. Add a new shell-script check with that same prefix. The
|
||||
`scripts/*.ts` entries run under bun and take no prefix.
|
||||
|
||||
The scripts must also be on disk with Unix line endings. A strict bash (WSL, Linux
|
||||
CI, macOS) rejects CRLF and dies on the script's first meaningful line; the Cygwin
|
||||
bash that ships with Git for Windows tolerates it, so a green local run is not by
|
||||
itself evidence that a script is CRLF-clean.
|
||||
The root `.gitattributes` pins `*.sh text eol=lf`, which overrides the
|
||||
`core.autocrlf=true` default that Git for Windows installs. Working copies cloned
|
||||
before that pin need a one-time `git rm --cached -r . -q && git reset --hard` to
|
||||
pick it up; see the Windows section of `CONTRIBUTING.md`.
|
||||
|
||||
Wallclock figures in the table above are from a Mac dev box. Windows is
|
||||
substantially slower because each check pays full process-creation cost, and three
|
||||
tree-walking checks (`check:privacy`, `check:test-names`, `check:test-isolation`)
|
||||
plus `typecheck` can exceed the 120s per-check cap in `run-verify-parallel.sh`
|
||||
there even though they pass on Linux and macOS.
|
||||
|
||||
### CI vs local: intentionally divergent file sets
|
||||
|
||||
- **CI matrix** (`.github/workflows/test.yml`) runs `scripts/test-shard.sh` across 10 matrix shards partitioned by weight-aware LPT bin-packing (`scripts/sharding.ts`) and INCLUDES `*.slow.test.ts` (the two outlier slow files run as dedicated jobs alongside the matrix). CI EXCLUDES `*.serial.test.ts` from the shards and runs them in a dedicated job via `bun run test:serial`, one bun process per file — keeping serial files out of the shard processes is what preserves the `mock.module` quarantine (a top-level mock in one file leaks into every other file sharing its process). `bun run verify` gets its own job too. CI is the ground truth for "did everything pass."
|
||||
|
||||
+34
-34
@@ -42,20 +42,20 @@
|
||||
"eval:autocut": "bun test test/search/autocut-eval.test.ts",
|
||||
"test:full": "bun run verify && bash scripts/run-unit-parallel.sh && bun run test:slow && ([ -n \"$DATABASE_URL\" ] && bash scripts/run-e2e.sh || echo '[test:full] skipped E2E (no DATABASE_URL); run docker-compose -f docker-compose.ci.yml up + bun run test:e2e to include' 1>&2)",
|
||||
"verify": "bash scripts/run-verify-parallel.sh",
|
||||
"check:source-config-leak": "scripts/check-source-config-leak.sh",
|
||||
"check:no-pii-agent-voice": "scripts/check-no-pii-in-agent-voice.sh",
|
||||
"check:synthetic-corpus-privacy": "scripts/check-synthetic-corpus-privacy.sh",
|
||||
"check:system-of-record": "scripts/check-system-of-record.sh",
|
||||
"check:admin-scope-drift": "scripts/check-admin-scope-drift.sh",
|
||||
"check:cli-exec": "scripts/check-cli-executable.sh",
|
||||
"check:all": "scripts/check-privacy.sh && scripts/check-proposal-pii.sh && scripts/check-test-real-names.sh && scripts/check-jsonb-pattern.sh && scripts/check-source-id-projection.sh && scripts/check-source-config-leak.sh && scripts/check-progress-to-stdout.sh && scripts/check-no-tracked-symlinks.sh && scripts/check-no-legacy-getconnection.sh && scripts/check-test-isolation.sh && scripts/check-trailing-newline.sh && scripts/check-wasm-embedded.sh && scripts/check-exports-count.sh && scripts/check-admin-build.sh && scripts/check-admin-scope-drift.sh && scripts/check-cli-executable.sh && scripts/check-skill-brain-first.sh && scripts/check-operations-filter-bypass.sh && scripts/check-gateway-routed-no-direct-anthropic.sh && scripts/check-worker-pool-atomicity.sh && scripts/check-key-files-current-state.sh && scripts/check-no-double-retry.sh && scripts/check-batch-audit-site.sh",
|
||||
"check:gateway-routed": "scripts/check-gateway-routed-no-direct-anthropic.sh",
|
||||
"check:worker-pool-atomicity": "scripts/check-worker-pool-atomicity.sh",
|
||||
"check:doc-history": "scripts/check-key-files-current-state.sh",
|
||||
"check:source-config-leak": "bash scripts/check-source-config-leak.sh",
|
||||
"check:no-pii-agent-voice": "bash scripts/check-no-pii-in-agent-voice.sh",
|
||||
"check:synthetic-corpus-privacy": "bash scripts/check-synthetic-corpus-privacy.sh",
|
||||
"check:system-of-record": "bash scripts/check-system-of-record.sh",
|
||||
"check:admin-scope-drift": "bash scripts/check-admin-scope-drift.sh",
|
||||
"check:cli-exec": "bash scripts/check-cli-executable.sh",
|
||||
"check:all": "bash scripts/check-privacy.sh && bash scripts/check-proposal-pii.sh && bash scripts/check-test-real-names.sh && bash scripts/check-jsonb-pattern.sh && bash scripts/check-source-id-projection.sh && bash scripts/check-source-config-leak.sh && bash scripts/check-progress-to-stdout.sh && bash scripts/check-no-tracked-symlinks.sh && bash scripts/check-no-legacy-getconnection.sh && bash scripts/check-test-isolation.sh && bash scripts/check-trailing-newline.sh && bash scripts/check-wasm-embedded.sh && bash scripts/check-exports-count.sh && bash scripts/check-admin-build.sh && bash scripts/check-admin-scope-drift.sh && bash scripts/check-cli-executable.sh && bash scripts/check-skill-brain-first.sh && bash scripts/check-operations-filter-bypass.sh && bash scripts/check-gateway-routed-no-direct-anthropic.sh && bash scripts/check-worker-pool-atomicity.sh && bash scripts/check-key-files-current-state.sh && bash scripts/check-no-double-retry.sh && bash scripts/check-batch-audit-site.sh",
|
||||
"check:gateway-routed": "bash scripts/check-gateway-routed-no-direct-anthropic.sh",
|
||||
"check:worker-pool-atomicity": "bash scripts/check-worker-pool-atomicity.sh",
|
||||
"check:doc-history": "bash scripts/check-key-files-current-state.sh",
|
||||
"check:resolver": "bun src/cli.ts check-resolvable --strict --skills-dir skills/",
|
||||
"check:skill-brain-first": "scripts/check-skill-brain-first.sh",
|
||||
"check:wasm": "scripts/check-wasm-embedded.sh",
|
||||
"check:newlines": "scripts/check-trailing-newline.sh",
|
||||
"check:skill-brain-first": "bash scripts/check-skill-brain-first.sh",
|
||||
"check:wasm": "bash scripts/check-wasm-embedded.sh",
|
||||
"check:newlines": "bash scripts/check-trailing-newline.sh",
|
||||
"test:e2e": "bash scripts/run-e2e.sh",
|
||||
"test:slow": "bash scripts/run-slow-tests.sh",
|
||||
"test:heavy": "bash scripts/run-heavy.sh",
|
||||
@@ -65,27 +65,27 @@
|
||||
"ci:local:diff": "bash scripts/ci-local.sh --diff",
|
||||
"ci:select-e2e": "bun run scripts/select-e2e.ts",
|
||||
"typecheck": "tsc --noEmit",
|
||||
"check:jsonb": "scripts/check-jsonb-pattern.sh",
|
||||
"check:search-path": "scripts/check-search-path.sh",
|
||||
"check:no-double-retry": "scripts/check-no-double-retry.sh",
|
||||
"check:batch-audit-site": "scripts/check-batch-audit-site.sh",
|
||||
"check:worker-lock-renewal-shape": "scripts/check-worker-lock-renewal-shape.sh",
|
||||
"check:source-id-projection": "scripts/check-source-id-projection.sh",
|
||||
"check:privacy": "scripts/check-privacy.sh",
|
||||
"check:proposal-pii": "scripts/check-proposal-pii.sh",
|
||||
"check:eval-glossary": "scripts/check-eval-glossary-fresh.sh",
|
||||
"check:test-names": "scripts/check-test-real-names.sh",
|
||||
"check:progress": "scripts/check-progress-to-stdout.sh",
|
||||
"check:no-tracked-symlinks": "scripts/check-no-tracked-symlinks.sh",
|
||||
"check:exports-count": "scripts/check-exports-count.sh",
|
||||
"check:admin-build": "scripts/check-admin-build.sh",
|
||||
"check:admin-embedded": "scripts/check-admin-embedded.sh",
|
||||
"check:test-isolation": "scripts/check-test-isolation.sh",
|
||||
"check:fuzz-purity": "scripts/check-fuzz-purity.sh",
|
||||
"check:operations-filter-bypass": "scripts/check-operations-filter-bypass.sh",
|
||||
"check:fixture-privacy": "scripts/check-fixture-privacy.sh",
|
||||
"check:jsonb": "bash scripts/check-jsonb-pattern.sh",
|
||||
"check:search-path": "bash scripts/check-search-path.sh",
|
||||
"check:no-double-retry": "bash scripts/check-no-double-retry.sh",
|
||||
"check:batch-audit-site": "bash scripts/check-batch-audit-site.sh",
|
||||
"check:worker-lock-renewal-shape": "bash scripts/check-worker-lock-renewal-shape.sh",
|
||||
"check:source-id-projection": "bash scripts/check-source-id-projection.sh",
|
||||
"check:privacy": "bash scripts/check-privacy.sh",
|
||||
"check:proposal-pii": "bash scripts/check-proposal-pii.sh",
|
||||
"check:eval-glossary": "bash scripts/check-eval-glossary-fresh.sh",
|
||||
"check:test-names": "bash scripts/check-test-real-names.sh",
|
||||
"check:progress": "bash scripts/check-progress-to-stdout.sh",
|
||||
"check:no-tracked-symlinks": "bash scripts/check-no-tracked-symlinks.sh",
|
||||
"check:exports-count": "bash scripts/check-exports-count.sh",
|
||||
"check:admin-build": "bash scripts/check-admin-build.sh",
|
||||
"check:admin-embedded": "bash scripts/check-admin-embedded.sh",
|
||||
"check:test-isolation": "bash scripts/check-test-isolation.sh",
|
||||
"check:fuzz-purity": "bash scripts/check-fuzz-purity.sh",
|
||||
"check:operations-filter-bypass": "bash scripts/check-operations-filter-bypass.sh",
|
||||
"check:fixture-privacy": "bash scripts/check-fixture-privacy.sh",
|
||||
"check:conversation-parser": "bun src/cli.ts eval conversation-parser test/fixtures/conversation-formats/all.jsonl --no-llm",
|
||||
"check:source-scope-onboard": "scripts/check-source-scope-onboard.sh",
|
||||
"check:source-scope-onboard": "bash scripts/check-source-scope-onboard.sh",
|
||||
"postinstall": "bun run scripts/postinstall.ts",
|
||||
"prepublish:clawhub": "bun run build:all",
|
||||
"publish:clawhub": "clawhub package publish . --family bundle-plugin"
|
||||
@@ -146,7 +146,7 @@
|
||||
"bun": ">=1.3.10"
|
||||
},
|
||||
"license": "MIT",
|
||||
"version": "0.42.66.1",
|
||||
"version": "0.42.67.0",
|
||||
"overrides": {
|
||||
"@hono/node-server": "^2.0.5",
|
||||
"fast-uri": "^3.1.4",
|
||||
|
||||
@@ -8,7 +8,7 @@ import {
|
||||
semverGt,
|
||||
semverLte,
|
||||
} from '../core/semver.ts';
|
||||
import { writeUpdateCache, type UpdateMarker } from '../core/self-upgrade.ts';
|
||||
import { readUpdateCache, writeUpdateCache, type UpdateMarker } from '../core/self-upgrade.ts';
|
||||
|
||||
/** Best-effort cache write — a read-only ~/.gbrain must never make the check throw. */
|
||||
function safeWriteCache(marker: UpdateMarker): void {
|
||||
@@ -45,26 +45,53 @@ function upgradeCommandForMethod(method: string): string {
|
||||
}
|
||||
}
|
||||
|
||||
/** Where the latest version is resolved from. gbrain publishes NO GitHub
|
||||
* releases (the `releases/latest` API is a permanent 404), so the release
|
||||
* train's source of truth is the `VERSION` file on master — same trusted host
|
||||
* `fetchChangelog` already uses. An npm fallback was rejected: the `gbrain`
|
||||
* package on npm is an unrelated GPU library (#505), so it would produce false
|
||||
* upgrade prompts pointing at a stranger's package. */
|
||||
const VERSION_SOURCE_URL = 'https://raw.githubusercontent.com/garrytan/gbrain/master/VERSION';
|
||||
const RELEASE_NOTES_URL = 'https://github.com/garrytan/gbrain/blob/master/CHANGELOG.md';
|
||||
|
||||
/** Extract a version from the raw VERSION file body: first line, optional `v`
|
||||
* prefix, optional `-suffix` channel tag (`0.31.1.1-fixwave` compares as its
|
||||
* numeric base — fail-safe: a suffix-only bump never prompts). Body is bounded
|
||||
* before parsing so a malformed/huge response can't blow up the check. */
|
||||
export function parseVersionFileBody(body: string): string | null {
|
||||
const firstLine = body.slice(0, 256).trim().split('\n')[0].trim();
|
||||
const m = firstLine.match(/^v?(\d+\.\d+\.\d+(?:\.\d+)?)(?:[-+][0-9A-Za-z.-]+)?$/);
|
||||
return m && isValidVersionString(m[1]) ? m[1] : null;
|
||||
}
|
||||
|
||||
export type LatestReleaseResult =
|
||||
| { ok: true; tag: string; published_at: string; url: string }
|
||||
| { ok: false; reason: 'network_error' | 'no_releases' };
|
||||
|
||||
/**
|
||||
* Fetch the latest GitHub release. Exported (v0.42) so the self-upgrade refresh
|
||||
* path and tests can reuse it. 5s timeout (was 10s) — this runs on the detached
|
||||
* refresh, never the hot path, but a tight bound keeps the refresh cheap.
|
||||
* Resolve the latest published gbrain version (from VERSION on master — see
|
||||
* VERSION_SOURCE_URL). Exported (v0.42) so the self-upgrade refresh path and
|
||||
* tests can reuse it. 5s timeout — this runs on the detached refresh, never the
|
||||
* hot path. Failures are discriminated: `network_error` (offline/timeout) vs
|
||||
* `no_releases` (endpoint answered but no usable version).
|
||||
*/
|
||||
export async function fetchLatestRelease(): Promise<{ tag: string; published_at: string; url: string } | null> {
|
||||
export async function fetchLatestRelease(): Promise<LatestReleaseResult> {
|
||||
let res: Response;
|
||||
try {
|
||||
const res = await fetch('https://api.github.com/repos/garrytan/gbrain/releases/latest', {
|
||||
res = await fetch(VERSION_SOURCE_URL, {
|
||||
headers: { 'User-Agent': `gbrain/${VERSION}` },
|
||||
signal: AbortSignal.timeout(5_000),
|
||||
});
|
||||
if (!res.ok) return null;
|
||||
const data = await res.json() as any;
|
||||
return {
|
||||
tag: data.tag_name || '',
|
||||
published_at: data.published_at || '',
|
||||
url: data.html_url || '',
|
||||
};
|
||||
} catch {
|
||||
return null;
|
||||
return { ok: false, reason: 'network_error' };
|
||||
}
|
||||
try {
|
||||
if (!res.ok) return { ok: false, reason: 'no_releases' };
|
||||
const tag = parseVersionFileBody(await res.text());
|
||||
if (!tag) return { ok: false, reason: 'no_releases' };
|
||||
return { ok: true, tag, published_at: '', url: RELEASE_NOTES_URL };
|
||||
} catch {
|
||||
return { ok: false, reason: 'network_error' };
|
||||
}
|
||||
}
|
||||
|
||||
@@ -118,17 +145,33 @@ export function extractChangelogBetween(changelog: string, from: string, to: str
|
||||
}
|
||||
|
||||
/**
|
||||
* Fetch the latest release and write the self-upgrade cache (the marker line
|
||||
* read by the CLI startup hook). Fail-open: on any network failure we cache
|
||||
* `UP_TO_DATE <current>` so the TTL prevents hammering GitHub on every
|
||||
* invocation. Returns the resolved marker for callers that want it. This is the
|
||||
* function the detached single-flight refresh (`gbrain check-update
|
||||
* --refresh-cache`) invokes.
|
||||
* A failed check must NEVER write `up_to_date` — that was #486: the fetch
|
||||
* failed permanently (dead releases API) and every user was told "you're
|
||||
* current" forever. Instead, re-write the last-known-good marker (bumping its
|
||||
* mtime so the cache TTL still throttles retries and a network blip can't
|
||||
* erase a pending upgrade_available notice). No prior marker → write nothing;
|
||||
* the next invocation retries.
|
||||
*/
|
||||
function preserveCacheOnFailedCheck(): void {
|
||||
try {
|
||||
const prior = readUpdateCache();
|
||||
if (prior) safeWriteCache(prior.marker);
|
||||
} catch {
|
||||
/* best-effort */
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Fetch the latest version and write the self-upgrade cache (the marker line
|
||||
* read by the CLI startup hook). On fetch failure the last-known-good marker is
|
||||
* preserved (see preserveCacheOnFailedCheck) — never a fabricated `up_to_date`.
|
||||
* This is the function the detached single-flight refresh (`gbrain
|
||||
* check-update --refresh-cache`) invokes.
|
||||
*/
|
||||
export async function refreshUpdateCache(): Promise<void> {
|
||||
const release = await fetchLatestRelease();
|
||||
if (!release) {
|
||||
safeWriteCache({ kind: 'up_to_date', current: VERSION });
|
||||
if (!release.ok) {
|
||||
preserveCacheOnFailedCheck();
|
||||
return;
|
||||
}
|
||||
const latestVersion = release.tag.replace(/^v/, '');
|
||||
@@ -166,9 +209,8 @@ export async function runCheckUpdate(args: string[]) {
|
||||
|
||||
const release = await fetchLatestRelease();
|
||||
|
||||
if (!release) {
|
||||
// Warm the cache fail-open so the startup hook doesn't re-fetch every call.
|
||||
safeWriteCache({ kind: 'up_to_date', current: VERSION });
|
||||
if (!release.ok) {
|
||||
preserveCacheOnFailedCheck();
|
||||
if (json) {
|
||||
console.log(JSON.stringify({
|
||||
current_version: VERSION,
|
||||
@@ -179,10 +221,12 @@ export async function runCheckUpdate(args: string[]) {
|
||||
release_url: '',
|
||||
changelog_diff: '',
|
||||
published_at: '',
|
||||
error: 'no_releases',
|
||||
error: release.reason,
|
||||
}, null, 2));
|
||||
} else if (release.reason === 'network_error') {
|
||||
console.log(`GBrain ${VERSION} — could not check for updates (network unavailable).`);
|
||||
} else {
|
||||
console.log(`GBrain ${VERSION} — could not check for updates (no releases found or network unavailable).`);
|
||||
console.log(`GBrain ${VERSION} — could not determine the latest published version.`);
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -35,7 +35,8 @@ export async function runSelfUpgrade(args: string[]): Promise<void> {
|
||||
const force = args.includes('--force');
|
||||
const json = args.includes('--json');
|
||||
|
||||
const release = await fetchLatestRelease();
|
||||
const result = await fetchLatestRelease();
|
||||
const release = result.ok ? result : null;
|
||||
const latest = release ? release.tag.replace(/^v/, '') : null;
|
||||
const behind = !!latest && isValidVersionString(latest) && isNewerVersion(VERSION, latest);
|
||||
|
||||
|
||||
@@ -180,6 +180,17 @@ export const openrouter: Recipe = {
|
||||
// to pre-split batches, NOT per-input. Per-input is enforced upstream.
|
||||
max_batch_tokens: 300_000,
|
||||
},
|
||||
// Expansion uses the same routed OpenAI-compatible language-model endpoint
|
||||
// as chat. Keep a small cheap/fast advisory set; the openai-compat tier
|
||||
// still accepts any user-configured OpenRouter provider/model ID.
|
||||
expansion: {
|
||||
models: [
|
||||
'anthropic/claude-haiku-4.5',
|
||||
'google/gemini-3-flash-preview',
|
||||
'deepseek/deepseek-chat',
|
||||
],
|
||||
price_last_verified: '2026-05-20',
|
||||
},
|
||||
chat: {
|
||||
// Curated entry points (verified against OR's catalog 2026-05-20). The
|
||||
// openai-compat tier does NOT enforce this list at runtime — users can
|
||||
|
||||
@@ -20,6 +20,7 @@
|
||||
|
||||
import { chunkText as recursiveChunk } from './recursive.ts';
|
||||
import { buildQualifiedName } from './qualified-names.ts';
|
||||
import { CJK_SLUG_CHARS, CJK_RANGES_REGEX } from '../cjk.ts';
|
||||
|
||||
// Embed the tree-sitter runtime + per-language grammars as files.
|
||||
// `with { type: 'file' }` returns a path (string) at runtime. Bun bundles
|
||||
@@ -716,7 +717,7 @@ export async function chunkCodeTextFull(
|
||||
}
|
||||
|
||||
if (chunks.length === 0) {
|
||||
return { chunks: capOversizedChunks(fallbackChunks(source, filePath, language, opts), filePath, language, opts), edges: rawEdges };
|
||||
return { chunks: fallbackChunks(source, filePath, language, opts), edges: rawEdges };
|
||||
}
|
||||
return { chunks: capOversizedChunks(mergeSmallSiblings(chunks, chunkTarget), filePath, language, opts), edges: rawEdges };
|
||||
} catch {
|
||||
@@ -842,10 +843,10 @@ function capOversizedChunks(
|
||||
opts: CodeChunkOptions,
|
||||
): CodeChunk[] {
|
||||
const cap = opts.maxChunkTokens ?? DEFAULT_MAX_CHUNK_TOKENS;
|
||||
if (!chunks.some((c) => estimateTokens(c.text) > cap)) return chunks;
|
||||
if (!chunks.some((c) => estimateEmbedTokens(c.text) > cap)) return chunks;
|
||||
const out: CodeChunk[] = [];
|
||||
for (const c of chunks) {
|
||||
if (estimateTokens(c.text) <= cap) {
|
||||
if (estimateEmbedTokens(c.text) <= cap) {
|
||||
out.push({ ...c, index: out.length });
|
||||
continue;
|
||||
}
|
||||
@@ -880,17 +881,43 @@ function splitToTokenBudget(text: string, cap: number, opts: CodeChunkOptions):
|
||||
chunkOverlap: opts.fallbackOverlapWords ?? 50,
|
||||
}).map((p) => p.text);
|
||||
for (const piece of pieces) {
|
||||
if (estimateTokens(piece) <= cap) {
|
||||
if (estimateEmbedTokens(piece) <= cap) {
|
||||
out.push(piece);
|
||||
continue;
|
||||
}
|
||||
// ~3.5 chars/token is a conservative cl100k estimate for source text.
|
||||
const charBudget = Math.max(1, Math.floor(cap * 3.5));
|
||||
// Hard-split slice size. Pure-ASCII pieces: ~3.5 chars/token is a
|
||||
// conservative cl100k estimate for source text. CJK-containing pieces:
|
||||
// the weighted estimate can reach 1 token/char, so budget 1 char/token
|
||||
// to keep every slice under cap by construction.
|
||||
const charBudget = Math.max(1, Math.floor(cap * (CJK_RANGES_REGEX.test(piece) ? 1 : 3.5)));
|
||||
for (let i = 0; i < piece.length; i += charBudget) out.push(piece.slice(i, i + charBudget));
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
const CJK_CHARS_G = new RegExp(`[${CJK_SLUG_CHARS}]`, 'g');
|
||||
|
||||
/**
|
||||
* Embedding-safe token estimate for the oversize cap. estimateTokens
|
||||
* (cl100k) matches embedding-family tokenizers closely on pure-ASCII source
|
||||
* (measured identical on English prose and JSON vs Qwen3-Embedding), but
|
||||
* UNDERCOUNTS mixed CJK+ASCII chunks — measured −31% on URL-dense Korean
|
||||
* text vs the Qwen3 embedding tokenizer, which is exactly the shape that
|
||||
* overflows strict embedding backends (#2826). For chunks containing CJK,
|
||||
* take the max of cl100k and a per-char-class overestimate (CJK 1.0/char,
|
||||
* other non-whitespace 0.75/char, whitespace 0.1/char). CJK-DOMINANT text
|
||||
* is unaffected too: cl100k already counts it above the weighted form, so
|
||||
* max() returns the same value as today. Only mixed-script chunks — the
|
||||
* measured divergence class — estimate higher.
|
||||
*/
|
||||
export function estimateEmbedTokens(text: string): number {
|
||||
const cjk = (text.match(CJK_CHARS_G) || []).length;
|
||||
if (cjk === 0) return estimateTokens(text);
|
||||
const ws = (text.match(/\s/g) || []).length;
|
||||
const weighted = Math.ceil(cjk + (text.length - cjk - ws) * 0.75 + ws * 0.1);
|
||||
return Math.max(estimateTokens(text), weighted);
|
||||
}
|
||||
|
||||
// ---------- Internals ----------
|
||||
|
||||
function fallbackChunks(
|
||||
@@ -901,7 +928,7 @@ function fallbackChunks(
|
||||
): CodeChunk[] {
|
||||
const size = opts.fallbackChunkSizeWords ?? 300;
|
||||
const overlap = opts.fallbackOverlapWords ?? 50;
|
||||
return recursiveChunk(source, { chunkSize: size, chunkOverlap: overlap }).map((chunk, index) =>
|
||||
const chunks = recursiveChunk(source, { chunkSize: size, chunkOverlap: overlap }).map((chunk, index) =>
|
||||
buildChunk({
|
||||
body: chunk.text, filePath, language,
|
||||
symbolName: null, symbolType: 'module',
|
||||
@@ -909,6 +936,14 @@ function fallbackChunks(
|
||||
index,
|
||||
}),
|
||||
);
|
||||
// Route every fallback emission through the oversize net. Previously only
|
||||
// the empty-AST branch wrapped its fallback in capOversizedChunks — the
|
||||
// no-language, parse-timeout, no-semantic-nodes (every JSON/YAML fence:
|
||||
// their node types aren't in TOP_LEVEL_TYPES) and parse-throw branches
|
||||
// shipped word-counted chunks unchecked, and the word pipeline undercounts
|
||||
// exactly the dense content (JSON, minified, CJK-mixed) that overflows
|
||||
// embedders. Hoisting the cap here covers all five paths at once.
|
||||
return capOversizedChunks(chunks, filePath, language, opts);
|
||||
}
|
||||
|
||||
function buildChunk(input: {
|
||||
|
||||
@@ -17,17 +17,19 @@
|
||||
* DB rows need cleanup (#1781 — the unconditional wipe-and-reinsert
|
||||
* made every cycle non-idempotent, re-appending duplicate rows).
|
||||
*
|
||||
* After the phase, the DB index for every affected page matches the
|
||||
* fence's canonical (claim, source) row set (modulo embeddings +
|
||||
* runtime-derived fields). Pages with no fence wipe DB rows for that
|
||||
* page coordinate only; legacy NULL-source_markdown_slug rows survive
|
||||
* because deleteFactsForPage targets source_markdown_slug = slug only.
|
||||
* After the phase, the DB index for every cleanly parsed affected page
|
||||
* matches the fence's canonical (claim, source) row set (modulo embeddings
|
||||
* + runtime-derived fields). Warning-bearing parses are non-authoritative
|
||||
* and preserve that page's existing index. Pages with no fence wipe DB rows
|
||||
* for that page coordinate only; legacy NULL-source_markdown_slug rows
|
||||
* survive because deleteFactsForPage targets source_markdown_slug = slug only.
|
||||
*
|
||||
* Empty-fence guard (Codex R2-#7; #2484): the phase refuses to do its
|
||||
* destructive reconciliation pass when genuinely-backfillable legacy
|
||||
* Empty-fence guard (Codex R2-#7; #2484; #2646): the phase refuses to do
|
||||
* its destructive reconciliation pass when genuinely-backfillable legacy
|
||||
* rows still exist — `row_num IS NULL` (never fenced) AND `entity_slug`
|
||||
* resolves to a live page in this source (so the v0_32_2 migration's
|
||||
* Phase B could fence them). Status returns `warn` with a hint to run
|
||||
* Phase B could fence them) AND the row is not soft-expired
|
||||
* (`expired_at IS NULL`). Status returns `warn` with a hint to run
|
||||
* `gbrain apply-migrations --yes`. Without the guard, an interrupted
|
||||
* upgrade where v0_32_2 hasn't run could leave the cycle silently
|
||||
* misreporting "0 facts on people/alice" while legacy rows linger.
|
||||
@@ -41,6 +43,10 @@
|
||||
* the phase jams forever (~16/day observed). Requiring a backing page
|
||||
* keeps genuine pre-v0.32.2 rows (whose entity page exists) gating
|
||||
* while excluding the inline-writer's permanent-unfenceable rows.
|
||||
*
|
||||
* Soft-expired rows don't count either (#2646): they're what
|
||||
* `forget_fact` produces, so excluding them lets operators drain the
|
||||
* backlog through the sanctioned removal path instead of raw SQL.
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from '../engine.ts';
|
||||
@@ -88,6 +94,24 @@ function dedupeFactsByContentKey(facts: FenceExtractedFact[]): FenceExtractedFac
|
||||
* neither count as "stale" (which would force a wipe every cycle) nor
|
||||
* be compared against the fence's row set. Mirrors the
|
||||
* excludeSourcePrefixes filter deleteFactsForPage applies on the wipe.
|
||||
*
|
||||
* Also excludes soft-expired legacy rows (#2646: `row_num IS NULL AND
|
||||
* expired_at IS NOT NULL`) — rows that `forget_fact` expired via its
|
||||
* legacy DB-only path. They are not fence-owned (fence rows always
|
||||
* carry a row_num), so they must neither count as "stale" (forcing a
|
||||
* wipe every cycle) nor mask a fence row from insertion. Mirrors the
|
||||
* preserveExpiredLegacy filter deleteFactsForPage applies on the wipe.
|
||||
*
|
||||
* Deliberate consequence: if the fence still carries the same
|
||||
* (claim, source) as an expired legacy row, the reconcile inserts it
|
||||
* as a fresh ACTIVE fence-owned row. That is the fence-is-canonical
|
||||
* contract working as documented — legacy DB-only forgets "DO NOT
|
||||
* survive rebuild" (see forget.ts header); suppressing the insert
|
||||
* would instead create silent fence↔DB divergence, the exact failure
|
||||
* mode the empty-fence guard exists to prevent. To durably forget
|
||||
* such a claim, forget the fence-owned row (forget_fact now takes the
|
||||
* fence path, which strikes the row through in markdown). The expired
|
||||
* legacy row survives alongside as the record of the earlier forget.
|
||||
*/
|
||||
async function listExistingFactsForPage(
|
||||
engine: BrainEngine,
|
||||
@@ -100,6 +124,7 @@ async function listExistingFactsForPage(
|
||||
WHERE source_id = $1
|
||||
AND source_markdown_slug = $2
|
||||
AND COALESCE(source, '') NOT LIKE 'cli:%'
|
||||
AND NOT (row_num IS NULL AND expired_at IS NOT NULL)
|
||||
ORDER BY row_num ASC, id ASC`,
|
||||
[sourceId, slug],
|
||||
);
|
||||
@@ -173,7 +198,7 @@ export async function runExtractFacts(
|
||||
phantomsMorePending: false,
|
||||
};
|
||||
|
||||
// ── Empty-fence guard (Codex R2-#7; #2484) ─────────────────────
|
||||
// ── Empty-fence guard (Codex R2-#7; #2484; #2646) ──────────────
|
||||
// Pre-check: if any genuinely-backfillable legacy fact rows exist,
|
||||
// refuse to run the destructive reconciliation pass — the v0_32_2
|
||||
// orchestrator must fence them first.
|
||||
@@ -181,12 +206,13 @@ export async function runExtractFacts(
|
||||
// A row is a real backfill candidate only when `row_num IS NULL`
|
||||
// (never fenced) AND its `entity_slug` resolves to a LIVE page in
|
||||
// this source (the migration's Phase B only fences rows whose
|
||||
// entity_slug maps to a writable page). #2484: the original
|
||||
// predicate was just `row_num IS NULL AND entity_slug IS NOT NULL`,
|
||||
// which ALSO matched structurally-unfenceable hot-memory rows the
|
||||
// inline writer keeps producing post-migration: the legacy DB-only
|
||||
// fallback (backstop.ts) writes `entity_slug` (a resolved slug, e.g.
|
||||
// a slugify-floor or stub-guard-blocked unprefixed slug like
|
||||
// entity_slug maps to a writable page) AND it is not soft-expired.
|
||||
// #2484: the original predicate was just `row_num IS NULL AND
|
||||
// entity_slug IS NOT NULL`, which ALSO matched
|
||||
// structurally-unfenceable hot-memory rows the inline writer keeps
|
||||
// producing post-migration: the legacy DB-only fallback
|
||||
// (backstop.ts) writes `entity_slug` (a resolved slug, e.g. a
|
||||
// slugify-floor or stub-guard-blocked unprefixed slug like
|
||||
// `people-jane-doe`) with `row_num` NULL whenever the slug has no
|
||||
// fenceable page. Those rows can never satisfy the migration's exit
|
||||
// condition (no page to fence onto, and `apply-migrations` is a
|
||||
@@ -194,12 +220,17 @@ export async function runExtractFacts(
|
||||
// — ~16/day, mislabeled "v0.31 pending backfill." We now require a
|
||||
// live backing page, which both genuine pre-v0.32.2 rows (their
|
||||
// entity page exists) satisfy and inline-writer unfenceable rows do
|
||||
// not.
|
||||
// not. #2646: soft-expired rows (`expired_at IS NOT NULL`) are also
|
||||
// excluded — `forget_fact`, the officially sanctioned removal path,
|
||||
// soft-expires legacy rows rather than deleting them, so counting
|
||||
// expired rows would leave the guard permanently stuck with no
|
||||
// supported way to drain the backlog.
|
||||
const legacy = await engine.executeRaw<{ n: string }>(
|
||||
`SELECT COUNT(*) AS n
|
||||
FROM facts f
|
||||
WHERE f.row_num IS NULL
|
||||
AND f.entity_slug IS NOT NULL
|
||||
AND f.expired_at IS NULL
|
||||
AND EXISTS (
|
||||
SELECT 1 FROM pages p
|
||||
WHERE p.source_id = f.source_id
|
||||
@@ -303,6 +334,11 @@ export async function runExtractFacts(
|
||||
result.warnings.push(
|
||||
...parsed.warnings.map(w => `${slug}: ${w}`),
|
||||
);
|
||||
// The parser deliberately skips malformed rows and returns any rows it
|
||||
// could still recover. That partial result is not authoritative: using
|
||||
// it for reconciliation would interpret skipped rows as deletions.
|
||||
// Preserve this page's existing index and continue with other pages.
|
||||
continue;
|
||||
}
|
||||
|
||||
if (parsed.facts.length > 0) result.pagesWithFacts += 1;
|
||||
@@ -334,9 +370,12 @@ export async function runExtractFacts(
|
||||
// partial-UNIQUE-index keyspace). #1928: `cli:`-origin facts
|
||||
// (conversation facts from extract-conversation-facts) are NOT
|
||||
// fence-owned — the page carries no `## Facts` fence to recreate
|
||||
// them — so they MUST survive this reconcile.
|
||||
// them — so they MUST survive this reconcile. #2646: soft-expired
|
||||
// legacy rows (forget_fact's record of the forget) likewise
|
||||
// survive via preserveExpiredLegacy.
|
||||
const deleted = await engine.deleteFactsForPage(slug, sourceId, {
|
||||
excludeSourcePrefixes: ['cli:'],
|
||||
preserveExpiredLegacy: true,
|
||||
});
|
||||
result.factsDeleted += deleted.deleted;
|
||||
}
|
||||
@@ -363,10 +402,11 @@ export async function runExtractFacts(
|
||||
if (hasStaleExisting || hasDuplicateExisting || hasRowNumDrift) {
|
||||
// Fall back to the legacy page-level reconcile when old DB rows must
|
||||
// be removed. Same delete scoping as above: legacy
|
||||
// NULL-source_markdown_slug rows and `cli:`-origin conversation
|
||||
// facts (#1928) survive.
|
||||
// NULL-source_markdown_slug rows, `cli:`-origin conversation
|
||||
// facts (#1928), and soft-expired legacy rows (#2646) survive.
|
||||
const deleted = await engine.deleteFactsForPage(slug, sourceId, {
|
||||
excludeSourcePrefixes: ['cli:'],
|
||||
preserveExpiredLegacy: true,
|
||||
});
|
||||
result.factsDeleted += deleted.deleted;
|
||||
toInsert = extracted;
|
||||
|
||||
+12
-1
@@ -1815,11 +1815,22 @@ export interface BrainEngine {
|
||||
* never recreate them (the page has no `## Facts` fence). Omitted ⇒ legacy
|
||||
* behavior (delete every fact on the page coordinate). NULL/empty `source`
|
||||
* rows are always deletable (fence default).
|
||||
*
|
||||
* #2646: `preserveExpiredLegacy` protects soft-expired legacy rows
|
||||
* (`row_num IS NULL AND expired_at IS NOT NULL`) — the record left by
|
||||
* `forget_fact`'s legacy DB-only path. Fence rows always carry a
|
||||
* `row_num`, so these rows are never fence-owned and a wipe would
|
||||
* destroy the forget record (the audit trail of the forget). Note what
|
||||
* this does NOT promise: it protects the record, not the forget itself —
|
||||
* if the fence still carries the same claim, fence canonicality
|
||||
* independently reinserts it as a fresh active row (legacy DB-only
|
||||
* forgets are documented as non-durable; see extract-facts.ts). Omitted
|
||||
* ⇒ legacy behavior.
|
||||
*/
|
||||
deleteFactsForPage(
|
||||
slug: string,
|
||||
source_id: string,
|
||||
opts?: { excludeSourcePrefixes?: string[] },
|
||||
opts?: { excludeSourcePrefixes?: string[]; preserveExpiredLegacy?: boolean },
|
||||
): Promise<{ deleted: number }>;
|
||||
|
||||
/**
|
||||
|
||||
@@ -4281,9 +4281,15 @@ export class PGLiteEngine implements BrainEngine {
|
||||
async deleteFactsForPage(
|
||||
slug: string,
|
||||
source_id: string,
|
||||
opts?: { excludeSourcePrefixes?: string[] },
|
||||
opts?: { excludeSourcePrefixes?: string[]; preserveExpiredLegacy?: boolean },
|
||||
): Promise<{ deleted: number }> {
|
||||
const prefixes = opts?.excludeSourcePrefixes;
|
||||
// #2646: keep soft-expired legacy rows (row_num NULL — never
|
||||
// fence-owned) so a fence reconcile can't destroy forget_fact's
|
||||
// legacy DB-only forget record.
|
||||
const expiredLegacyFilter = opts?.preserveExpiredLegacy
|
||||
? ` AND NOT (row_num IS NULL AND expired_at IS NOT NULL)`
|
||||
: '';
|
||||
if (prefixes && prefixes.length > 0) {
|
||||
// #1928: keep rows whose `source` matches an excluded prefix (e.g.
|
||||
// `cli:` conversation facts). COALESCE so NULL/empty-source fence rows
|
||||
@@ -4292,13 +4298,13 @@ export class PGLiteEngine implements BrainEngine {
|
||||
const result = await this.db.query(
|
||||
`DELETE FROM facts
|
||||
WHERE source_id = $1 AND source_markdown_slug = $2
|
||||
AND NOT (COALESCE(source, '') LIKE ANY($3::text[]))`,
|
||||
AND NOT (COALESCE(source, '') LIKE ANY($3::text[]))${expiredLegacyFilter}`,
|
||||
[source_id, slug, patterns],
|
||||
);
|
||||
return { deleted: result.affectedRows ?? 0 };
|
||||
}
|
||||
const result = await this.db.query(
|
||||
`DELETE FROM facts WHERE source_id = $1 AND source_markdown_slug = $2`,
|
||||
`DELETE FROM facts WHERE source_id = $1 AND source_markdown_slug = $2${expiredLegacyFilter}`,
|
||||
[source_id, slug],
|
||||
);
|
||||
return { deleted: result.affectedRows ?? 0 };
|
||||
|
||||
@@ -4438,10 +4438,16 @@ export class PostgresEngine implements BrainEngine {
|
||||
async deleteFactsForPage(
|
||||
slug: string,
|
||||
source_id: string,
|
||||
opts?: { excludeSourcePrefixes?: string[] },
|
||||
opts?: { excludeSourcePrefixes?: string[]; preserveExpiredLegacy?: boolean },
|
||||
): Promise<{ deleted: number }> {
|
||||
const sql = this.sql;
|
||||
const prefixes = opts?.excludeSourcePrefixes;
|
||||
// #2646: keep soft-expired legacy rows (row_num NULL — never
|
||||
// fence-owned) so a fence reconcile can't destroy forget_fact's
|
||||
// legacy DB-only forget record.
|
||||
const expiredLegacyFilter = opts?.preserveExpiredLegacy
|
||||
? sql`AND NOT (row_num IS NULL AND expired_at IS NOT NULL)`
|
||||
: sql``;
|
||||
if (prefixes && prefixes.length > 0) {
|
||||
// #1928: keep rows whose `source` matches an excluded prefix (e.g.
|
||||
// `cli:` conversation facts). COALESCE so NULL/empty-source fence rows
|
||||
@@ -4452,11 +4458,12 @@ export class PostgresEngine implements BrainEngine {
|
||||
WHERE source_id = ${source_id}
|
||||
AND source_markdown_slug = ${slug}
|
||||
AND NOT (COALESCE(source, '') LIKE ANY(${patterns}))
|
||||
${expiredLegacyFilter}
|
||||
`;
|
||||
return { deleted: result.count ?? 0 };
|
||||
}
|
||||
const result = await sql`
|
||||
DELETE FROM facts WHERE source_id = ${source_id} AND source_markdown_slug = ${slug}
|
||||
DELETE FROM facts WHERE source_id = ${source_id} AND source_markdown_slug = ${slug} ${expiredLegacyFilter}
|
||||
`;
|
||||
return { deleted: result.count ?? 0 };
|
||||
}
|
||||
|
||||
@@ -114,11 +114,12 @@ describe('gateway.isAvailable (silent-drop regression surface)', () => {
|
||||
// #1135 — an explicit expansion_model pointed at a chat-capable
|
||||
// OpenAI-compatible provider used to silently yield no expansion because
|
||||
// the recipe declared no expansion touchpoint.
|
||||
test('expansion available for chat-capable openai-compat providers (deepseek/groq/together)', () => {
|
||||
test('expansion available for chat-capable openai-compat providers (deepseek/groq/together/openrouter)', () => {
|
||||
const cases: Array<[string, Record<string, string>]> = [
|
||||
['deepseek:deepseek-chat', { DEEPSEEK_API_KEY: 'fake' }],
|
||||
['groq:llama-3.1-8b-instant', { GROQ_API_KEY: 'fake' }],
|
||||
['together:meta-llama/Llama-3.3-70B-Instruct-Turbo', { TOGETHER_API_KEY: 'fake' }],
|
||||
['openrouter:google/gemini-3-flash-preview', { OPENROUTER_API_KEY: 'fake' }],
|
||||
];
|
||||
for (const [model, env] of cases) {
|
||||
resetGateway();
|
||||
|
||||
@@ -65,6 +65,18 @@ describe('recipe: openrouter', () => {
|
||||
).not.toThrow();
|
||||
});
|
||||
|
||||
test('3b. expansion reuses routed chat models and accepts arbitrary provider/model IDs', () => {
|
||||
const r = getRecipe('openrouter')!;
|
||||
expect(r.touchpoints.expansion).toBeDefined();
|
||||
expect(r.touchpoints.expansion!.models.length).toBeGreaterThanOrEqual(3);
|
||||
expect(() =>
|
||||
assertTouchpoint(r, 'expansion', 'some/provider-model'),
|
||||
).not.toThrow();
|
||||
expect(() =>
|
||||
assertTouchpoint(r, 'expansion', 'meta-llama/llama-future-2030'),
|
||||
).not.toThrow();
|
||||
});
|
||||
|
||||
test('4. chat models list — every entry matches provider/model shape (D5 regression)', () => {
|
||||
// Codex correction: pinning specific slugs creates false confidence (the
|
||||
// list is advisory; OR's catalog churns). The shape test catches the
|
||||
|
||||
@@ -1,8 +1,10 @@
|
||||
/**
|
||||
* Serial (stubs globalThis.fetch): exercises the self-upgrade cache REFRESH
|
||||
* orchestration end-to-end — `refreshUpdateCache()` fetches the latest release
|
||||
* and writes the correct marker to the shared cache file that the CLI startup
|
||||
* hook reads. Network is stubbed; the cache write + marker logic are real.
|
||||
* orchestration end-to-end — `refreshUpdateCache()` resolves the latest version
|
||||
* (from the VERSION file on master, #486 — the repo has zero GitHub releases,
|
||||
* so the old `releases/latest` API path could never succeed) and writes the
|
||||
* correct marker to the shared cache file that the CLI startup hook reads.
|
||||
* Network is stubbed; the cache write + marker logic are real.
|
||||
*
|
||||
* Quarantined as *.serial.test.ts because it reassigns the process-global
|
||||
* `fetch` (cross-file-unsafe under the parallel runner).
|
||||
@@ -13,10 +15,11 @@ import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { VERSION } from '../src/version.ts';
|
||||
import { parseSemver } from '../src/core/semver.ts';
|
||||
import { readUpdateCache } from '../src/core/self-upgrade.ts';
|
||||
import { refreshUpdateCache } from '../src/commands/check-update.ts';
|
||||
import { readUpdateCache, writeUpdateCache } from '../src/core/self-upgrade.ts';
|
||||
import { fetchLatestRelease, parseVersionFileBody, refreshUpdateCache, runCheckUpdate } from '../src/commands/check-update.ts';
|
||||
|
||||
const realFetch = globalThis.fetch;
|
||||
const realLog = console.log;
|
||||
let homeDir: string;
|
||||
let priorHome: string | undefined;
|
||||
|
||||
@@ -27,14 +30,13 @@ function bump(kind: 'minor' | 'patch' | 'micro'): string {
|
||||
return `${v[0]}.${v[1]}.${v[2]}.${v[3] + 1}`;
|
||||
}
|
||||
|
||||
function stubReleaseFetch(tag: string | null, ok = true): void {
|
||||
/** Stub the VERSION-file fetch. body === null → network throw. */
|
||||
function stubVersionFetch(body: string | null, status = 200): void {
|
||||
globalThis.fetch = (async (url: any) => {
|
||||
const u = String(url);
|
||||
if (u.includes('/releases/latest')) {
|
||||
if (tag === null) throw new Error('network down');
|
||||
return new Response(JSON.stringify({ tag_name: tag, published_at: '2026-01-01T00:00:00Z', html_url: 'https://x' }), {
|
||||
status: ok ? 200 : 500,
|
||||
});
|
||||
if (u.includes('/gbrain/master/VERSION')) {
|
||||
if (body === null) throw new Error('network down');
|
||||
return new Response(body, { status });
|
||||
}
|
||||
// Changelog fetch (only happens when update available) — return empty.
|
||||
return new Response('', { status: 200 });
|
||||
@@ -49,49 +51,155 @@ beforeEach(() => {
|
||||
|
||||
afterEach(() => {
|
||||
globalThis.fetch = realFetch;
|
||||
console.log = realLog;
|
||||
if (priorHome === undefined) delete process.env.GBRAIN_HOME;
|
||||
else process.env.GBRAIN_HOME = priorHome;
|
||||
rmSync(homeDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
describe('fetchLatestRelease — resolves from the VERSION file, discriminates failures', () => {
|
||||
test('bare version body → ok with that tag', async () => {
|
||||
stubVersionFetch('0.99.1.0\n');
|
||||
expect(await fetchLatestRelease()).toMatchObject({ ok: true, tag: '0.99.1.0' });
|
||||
});
|
||||
|
||||
test('network throw → network_error (NOT no_releases — offline users are not told "no releases exist")', async () => {
|
||||
stubVersionFetch(null);
|
||||
expect(await fetchLatestRelease()).toEqual({ ok: false, reason: 'network_error' });
|
||||
});
|
||||
|
||||
test('HTTP 404 → no_releases', async () => {
|
||||
stubVersionFetch('Not Found', 404);
|
||||
expect(await fetchLatestRelease()).toEqual({ ok: false, reason: 'no_releases' });
|
||||
});
|
||||
|
||||
test('garbage body → no_releases', async () => {
|
||||
stubVersionFetch('<html>rate limited</html>');
|
||||
expect(await fetchLatestRelease()).toEqual({ ok: false, reason: 'no_releases' });
|
||||
});
|
||||
});
|
||||
|
||||
describe('parseVersionFileBody — shape gate over the raw fetch body', () => {
|
||||
test('trailing newline, v prefix, 3-segment legacy, suffix channel', () => {
|
||||
expect(parseVersionFileBody('0.42.67.0\n')).toBe('0.42.67.0');
|
||||
expect(parseVersionFileBody('v0.42.67.0')).toBe('0.42.67.0');
|
||||
expect(parseVersionFileBody('0.31.3\n')).toBe('0.31.3'); // legacy 3-segment
|
||||
expect(parseVersionFileBody('0.31.1.1-fixwave\n')).toBe('0.31.1.1'); // suffix compares as base
|
||||
});
|
||||
|
||||
test('malformed / huge / injected bodies → null', () => {
|
||||
expect(parseVersionFileBody('')).toBeNull();
|
||||
expect(parseVersionFileBody('not a version')).toBeNull();
|
||||
expect(parseVersionFileBody('$(rm -rf /)')).toBeNull();
|
||||
expect(parseVersionFileBody('1.2')).toBeNull(); // 2-segment: not a gbrain version
|
||||
expect(parseVersionFileBody('9'.repeat(10_000_000))).toBeNull(); // bounded, no blowup
|
||||
});
|
||||
});
|
||||
|
||||
describe('refreshUpdateCache — full refresh orchestration (network stubbed)', () => {
|
||||
test('minor-bump release → writes upgrade_available marker', async () => {
|
||||
test('minor-bump VERSION on master → writes upgrade_available marker', async () => {
|
||||
const latest = bump('minor');
|
||||
stubReleaseFetch(`v${latest}`);
|
||||
stubVersionFetch(`${latest}\n`);
|
||||
await refreshUpdateCache();
|
||||
const entry = readUpdateCache();
|
||||
expect(entry?.marker).toEqual({ kind: 'upgrade_available', current: VERSION, latest });
|
||||
});
|
||||
|
||||
test('patch release → writes upgrade_available marker', async () => {
|
||||
test('patch bump → writes upgrade_available marker', async () => {
|
||||
const latest = bump('patch');
|
||||
stubReleaseFetch(`v${latest}`);
|
||||
stubVersionFetch(`${latest}\n`);
|
||||
await refreshUpdateCache();
|
||||
expect(readUpdateCache()?.marker).toEqual({ kind: 'upgrade_available', current: VERSION, latest });
|
||||
});
|
||||
|
||||
test('micro release → writes upgrade_available marker', async () => {
|
||||
test('micro bump → writes upgrade_available marker', async () => {
|
||||
const latest = bump('micro');
|
||||
stubReleaseFetch(`v${latest}`);
|
||||
stubVersionFetch(`${latest}\n`);
|
||||
await refreshUpdateCache();
|
||||
expect(readUpdateCache()?.marker).toEqual({ kind: 'upgrade_available', current: VERSION, latest });
|
||||
});
|
||||
|
||||
test('network failure → writes up_to_date marker (fail-open, TTL prevents hammering)', async () => {
|
||||
stubReleaseFetch(null);
|
||||
test('minor bump published as legacy 3-segment → still detected', async () => {
|
||||
const v = parseSemver(VERSION)!;
|
||||
const latest = `${v[0]}.${v[1] + 1}.0`; // 3-segment, no micro
|
||||
stubVersionFetch(`${latest}\n`);
|
||||
await refreshUpdateCache();
|
||||
expect(readUpdateCache()?.marker).toEqual({ kind: 'upgrade_available', current: VERSION, latest });
|
||||
});
|
||||
|
||||
test('suffix channel release (X.Y.Z.W-fixwave) → compares as numeric base', async () => {
|
||||
const latest = bump('micro');
|
||||
stubVersionFetch(`${latest}-fixwave\n`);
|
||||
await refreshUpdateCache();
|
||||
expect(readUpdateCache()?.marker).toEqual({ kind: 'upgrade_available', current: VERSION, latest });
|
||||
});
|
||||
|
||||
test('same version on master → up_to_date marker', async () => {
|
||||
stubVersionFetch(`${VERSION}\n`);
|
||||
await refreshUpdateCache();
|
||||
expect(readUpdateCache()?.marker).toEqual({ kind: 'up_to_date', current: VERSION });
|
||||
});
|
||||
|
||||
test('non-OK HTTP → fail-open up_to_date', async () => {
|
||||
stubReleaseFetch(`v${bump('minor')}`, false);
|
||||
// The #486 bug class: a failed check must never fabricate "you're current".
|
||||
test('network failure with NO prior cache → writes NOTHING (never a fabricated up_to_date)', async () => {
|
||||
stubVersionFetch(null);
|
||||
await refreshUpdateCache();
|
||||
expect(readUpdateCache()?.marker).toEqual({ kind: 'up_to_date', current: VERSION });
|
||||
expect(readUpdateCache()).toBeNull();
|
||||
});
|
||||
|
||||
test('garbage tag → fail-open up_to_date (forged/invalid version never cached as upgrade)', async () => {
|
||||
stubReleaseFetch('v$(rm -rf /)');
|
||||
test('network failure with prior upgrade_available → pending notice PRESERVED, not erased', async () => {
|
||||
const latest = bump('minor');
|
||||
writeUpdateCache({ kind: 'upgrade_available', current: VERSION, latest });
|
||||
stubVersionFetch(null);
|
||||
await refreshUpdateCache();
|
||||
expect(readUpdateCache()?.marker).toEqual({ kind: 'up_to_date', current: VERSION });
|
||||
expect(readUpdateCache()?.marker).toEqual({ kind: 'upgrade_available', current: VERSION, latest });
|
||||
});
|
||||
|
||||
test('non-OK HTTP → no fabricated up_to_date', async () => {
|
||||
stubVersionFetch('nope', 500);
|
||||
await refreshUpdateCache();
|
||||
expect(readUpdateCache()).toBeNull();
|
||||
});
|
||||
|
||||
test('garbage body → no fabricated marker (forged/invalid version never cached as upgrade)', async () => {
|
||||
stubVersionFetch('$(rm -rf /)');
|
||||
await refreshUpdateCache();
|
||||
expect(readUpdateCache()).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('runCheckUpdate --json — failure discrimination (#486)', () => {
|
||||
function capture(): string[] {
|
||||
const lines: string[] = [];
|
||||
console.log = (...a: unknown[]) => { lines.push(a.join(' ')); };
|
||||
return lines;
|
||||
}
|
||||
|
||||
test('offline → error: network_error (not "no releases exist")', async () => {
|
||||
stubVersionFetch(null);
|
||||
const lines = capture();
|
||||
await runCheckUpdate(['--json']);
|
||||
const out = JSON.parse(lines.join('\n'));
|
||||
expect(out.error).toBe('network_error');
|
||||
expect(out.update_available).toBe(false);
|
||||
expect(readUpdateCache()).toBeNull(); // and no fabricated up_to_date cache
|
||||
});
|
||||
|
||||
test('endpoint answers but no usable version → error: no_releases', async () => {
|
||||
stubVersionFetch('garbage');
|
||||
const lines = capture();
|
||||
await runCheckUpdate(['--json']);
|
||||
expect(JSON.parse(lines.join('\n')).error).toBe('no_releases');
|
||||
});
|
||||
|
||||
test('newer VERSION on master → update_available true with latest_version set', async () => {
|
||||
const latest = bump('minor');
|
||||
stubVersionFetch(`${latest}\n`);
|
||||
const lines = capture();
|
||||
await runCheckUpdate(['--json']);
|
||||
const out = JSON.parse(lines.join('\n'));
|
||||
expect(out.update_available).toBe(true);
|
||||
expect(out.latest_version).toBe(latest);
|
||||
expect(readUpdateCache()?.marker).toEqual({ kind: 'upgrade_available', current: VERSION, latest });
|
||||
});
|
||||
});
|
||||
|
||||
@@ -35,6 +35,13 @@ describe('isNewerVersion', () => {
|
||||
expect(isNewerVersion('0.42.66.0', '0.42.66.1')).toBe(true);
|
||||
});
|
||||
|
||||
test('orders legacy 3-segment against 4-segment: 0.42.67.0 > 0.42.66.1 > 0.42.66', () => {
|
||||
expect(isNewerVersion('0.42.66.1', '0.42.67.0')).toBe(true);
|
||||
expect(isNewerVersion('0.42.66', '0.42.66.1')).toBe(true);
|
||||
expect(isNewerVersion('0.42.66', '0.42.66.0')).toBe(false); // 3-segment == its .0 micro
|
||||
expect(isNewerVersion('0.42.67.0', '0.42.66.1')).toBe(false);
|
||||
});
|
||||
|
||||
test('rejects equal, older, and malformed versions', () => {
|
||||
expect(isNewerVersion('0.42.66.0', '0.42.66.0')).toBe(false);
|
||||
expect(isNewerVersion('0.42.66.1', '0.42.66.0')).toBe(false);
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
/**
|
||||
* capOversizedChunks — CJK-aware oversize measurement (follow-up to #1675,
|
||||
* shape requested in #3475's closing review).
|
||||
*
|
||||
* cl100k (estimateTokens) matches embedding-family tokenizers on pure-ASCII
|
||||
* source (measured identical on English prose and JSON vs Qwen3-Embedding),
|
||||
* but undercounts MIXED CJK+ASCII chunks — measured −31% on URL-dense Korean
|
||||
* text (#2826's failure shape). estimateEmbedTokens lifts only that class:
|
||||
* ASCII-only input short-circuits to estimateTokens verbatim, and max()
|
||||
* keeps CJK-dominant text at the cl100k count it gets today.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { chunkCodeText, estimateTokens, estimateEmbedTokens } from '../../src/core/chunkers/code.ts';
|
||||
|
||||
/** URL-dense Korean rollup lines — the measured −31% divergence shape. */
|
||||
function urlDenseKoreanMix(lines: number): string {
|
||||
return Array.from({ length: lines }, (_, i) =>
|
||||
`- 항목 ${i}: 검증용 한국어 문장 · 링크: https://docs.example.com/pages/${String(i).padStart(32, '0')}?v=abcdef0123456789&ref=sample`,
|
||||
).join('\n');
|
||||
}
|
||||
|
||||
function bigJsonWithKoreanValues(targetChars: number): string {
|
||||
const entries: string[] = [];
|
||||
let i = 0;
|
||||
let len = 0;
|
||||
while (len < targetChars) {
|
||||
const row =
|
||||
` "item_${i}": { "name": "예시-${i}", "url": "https://example.com/api/v2/items/${i}?token=abc${i}def", "qty": ${i % 100}, "memo": "한국어 값이 섞인 예시 데이터" }`;
|
||||
entries.push(row);
|
||||
len += row.length;
|
||||
i++;
|
||||
}
|
||||
return `{\n${entries.join(',\n')}\n}`;
|
||||
}
|
||||
|
||||
describe('estimateEmbedTokens — measurement gate', () => {
|
||||
test('ASCII-only input is bit-identical to estimateTokens (no CJK → short-circuit)', () => {
|
||||
const en = 'function ordinary() { return compute(42) + helper(); } '.repeat(80);
|
||||
const json = '{"item": {"name": "sample", "url": "https://example.com/a?b=c", "qty": 42}}, '.repeat(60);
|
||||
expect(estimateEmbedTokens(en)).toBe(estimateTokens(en));
|
||||
expect(estimateEmbedTokens(json)).toBe(estimateTokens(json));
|
||||
});
|
||||
|
||||
test('never estimates below estimateTokens (max composition)', () => {
|
||||
for (const s of [urlDenseKoreanMix(20), '이 문장은 순수 한국어 산문 예시입니다. '.repeat(40), 'plain ascii ', '']) {
|
||||
expect(estimateEmbedTokens(s)).toBeGreaterThanOrEqual(estimateTokens(s));
|
||||
}
|
||||
});
|
||||
|
||||
test('mixed CJK+ASCII (the measured divergence class) estimates strictly higher', () => {
|
||||
const mix = urlDenseKoreanMix(20);
|
||||
// Real Qwen3-Embedding count for this shape measures ~45% ABOVE cl100k;
|
||||
// the weighted form stays above the real count (+15% measured margin).
|
||||
expect(estimateEmbedTokens(mix)).toBeGreaterThan(estimateTokens(mix));
|
||||
});
|
||||
});
|
||||
|
||||
describe('capOversizedChunks with the CJK-aware estimate', () => {
|
||||
test('oversized json fence with Korean values re-splits under the default cap', async () => {
|
||||
const src = bigJsonWithKoreanValues(14_000);
|
||||
const chunks = await chunkCodeText(src, 'fence.json');
|
||||
expect(chunks.length).toBeGreaterThan(1);
|
||||
for (const c of chunks) {
|
||||
// Small slack for the "[JSON] fence.json:…" header buildChunk re-adds
|
||||
// after the body-level split.
|
||||
expect(estimateEmbedTokens(c.text)).toBeLessThanOrEqual(2000 + 60);
|
||||
}
|
||||
// Content preserved — spot-check first / last entries survive.
|
||||
const joined = chunks.map((c) => c.text).join('\n');
|
||||
expect(joined).toContain('"item_0"');
|
||||
expect(joined).toContain('한국어 값이 섞인 예시 데이터');
|
||||
});
|
||||
|
||||
test('hard-split fallback makes progress on whitespace-less CJK-mixed input and stays under cap', async () => {
|
||||
const blob = '한a민b국c'.repeat(3_000); // 18K chars, no whitespace
|
||||
const chunks = await chunkCodeText(`{"blob": "${blob}"}`, 'fence.json');
|
||||
expect(chunks.length).toBeGreaterThan(1);
|
||||
for (const c of chunks) {
|
||||
expect(estimateEmbedTokens(c.text)).toBeLessThanOrEqual(2000 + 60);
|
||||
}
|
||||
});
|
||||
|
||||
test('pure-ASCII chunks are measured by the identical estimator (cap decisions unchanged)', async () => {
|
||||
const entries = Array.from({ length: 120 }, (_, i) =>
|
||||
` "item_${i}": { "name": "sample-${i}", "url": "https://example.com/api/v2/items/${i}?token=abc${i}def", "qty": ${i % 100} }`,
|
||||
);
|
||||
const src = `{\n${entries.join(',\n')}\n}`;
|
||||
const chunks = await chunkCodeText(src, 'fence.json');
|
||||
expect(chunks.length).toBeGreaterThan(0);
|
||||
for (const c of chunks) {
|
||||
// For ASCII-only chunks the two estimators are identical (pinned
|
||||
// above), so cap decisions — and therefore boundaries — are unchanged.
|
||||
expect(estimateEmbedTokens(c.text)).toBe(estimateTokens(c.text));
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -73,3 +73,90 @@ describe.skipIf(skip)('facts-fence escaped-pipe reconciliation on Postgres', ()
|
||||
]);
|
||||
}, 30_000);
|
||||
});
|
||||
|
||||
describe.skipIf(skip)('deleteFactsForPage preserveExpiredLegacy on Postgres (#2646)', () => {
|
||||
// The PGLite side of this contract is pinned by
|
||||
// test/extract-facts-phase.test.ts; this pins the postgres.js
|
||||
// tagged-fragment SQL (the two branches interpolate `expiredLegacyFilter`
|
||||
// differently) AND the returned delete count on a real Postgres.
|
||||
const slug = 'people/expired-legacy-preserve-example';
|
||||
let engine: PostgresEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PostgresEngine();
|
||||
await engine.connect({ database_url: databaseUrl! });
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
if (engine) {
|
||||
await engine.executeRaw('DELETE FROM facts WHERE source_markdown_slug = $1', [slug]);
|
||||
await engine.executeRaw('DELETE FROM pages WHERE slug = $1', [slug]);
|
||||
await engine.disconnect();
|
||||
}
|
||||
});
|
||||
|
||||
async function seedRows(): Promise<void> {
|
||||
await engine.executeRaw('DELETE FROM facts WHERE source_markdown_slug = $1', [slug]);
|
||||
// One fence-owned active row (deletable) + one soft-expired legacy row
|
||||
// (row_num NULL, expired_at set — forget_fact's record, must survive).
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO facts (source_id, entity_slug, fact, kind, visibility, notability,
|
||||
valid_from, source, confidence, row_num, expired_at, source_markdown_slug)
|
||||
VALUES
|
||||
('default', $1, 'fence-owned active fact', 'fact', 'world', 'high',
|
||||
now(), 'fence:reconcile', 1.0, 1, NULL, $1),
|
||||
('default', $1, 'forgotten legacy claim', 'fact', 'private', 'medium',
|
||||
now(), 'mcp:put_page', 1.0, NULL, now(), $1)`,
|
||||
[slug],
|
||||
);
|
||||
}
|
||||
|
||||
test('no-prefix branch: expired legacy row survives, count reflects only real deletions', async () => {
|
||||
await seedRows();
|
||||
const { deleted } = await engine.deleteFactsForPage(slug, 'default', {
|
||||
preserveExpiredLegacy: true,
|
||||
});
|
||||
expect(deleted).toBe(1); // only the fence-owned row
|
||||
|
||||
const rows = await engine.executeRaw<{ fact: string }>(
|
||||
'SELECT fact FROM facts WHERE source_markdown_slug = $1', [slug],
|
||||
);
|
||||
expect(Array.from(rows).map(r => r.fact)).toEqual(['forgotten legacy claim']);
|
||||
}, 30_000);
|
||||
|
||||
test('prefix branch: excludeSourcePrefixes and preserveExpiredLegacy compose', async () => {
|
||||
await seedRows();
|
||||
// Add a cli:-origin row that the prefix exclusion must protect.
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO facts (source_id, entity_slug, fact, kind, visibility, notability,
|
||||
valid_from, source, confidence, row_num, expired_at, source_markdown_slug)
|
||||
VALUES ('default', $1, 'conversation fact', 'fact', 'private', 'medium',
|
||||
now(), 'cli:extract-conversation-facts', 1.0, NULL, NULL, $1)`,
|
||||
[slug],
|
||||
);
|
||||
const { deleted } = await engine.deleteFactsForPage(slug, 'default', {
|
||||
excludeSourcePrefixes: ['cli:'],
|
||||
preserveExpiredLegacy: true,
|
||||
});
|
||||
expect(deleted).toBe(1); // only the fence-owned row
|
||||
|
||||
const rows = await engine.executeRaw<{ fact: string }>(
|
||||
'SELECT fact FROM facts WHERE source_markdown_slug = $1 ORDER BY id', [slug],
|
||||
);
|
||||
expect(Array.from(rows).map(r => r.fact)).toEqual([
|
||||
'forgotten legacy claim',
|
||||
'conversation fact',
|
||||
]);
|
||||
}, 30_000);
|
||||
|
||||
test('omitted option keeps legacy wipe behavior (expired row IS deleted)', async () => {
|
||||
await seedRows();
|
||||
const { deleted } = await engine.deleteFactsForPage(slug, 'default');
|
||||
expect(deleted).toBe(2);
|
||||
const rows = await engine.executeRaw<{ fact: string }>(
|
||||
'SELECT fact FROM facts WHERE source_markdown_slug = $1', [slug],
|
||||
);
|
||||
expect(Array.from(rows)).toHaveLength(0);
|
||||
}, 30_000);
|
||||
});
|
||||
|
||||
@@ -231,6 +231,44 @@ describe('runExtractFacts — happy path', () => {
|
||||
expect(rows.rows[0].fact).toBe('A');
|
||||
});
|
||||
|
||||
test('malformed fence rows make the page non-authoritative and preserve its indexed facts', async () => {
|
||||
await putPage('people/alice', FACT_FENCE(
|
||||
`| 1 | A | fact | 1.0 | world | medium | 2026-01-01 | | s | |
|
||||
| 2 | B | fact | 1.0 | world | medium | 2026-01-01 | | s | |`,
|
||||
));
|
||||
await runExtractFacts(engine, { slugs: ['people/alice'] });
|
||||
|
||||
// A hand edit corrupts row 2. The parser can still recover row 1, but
|
||||
// that partial result is not an authoritative replacement for the page.
|
||||
await putPage('people/alice', FACT_FENCE(
|
||||
`| 1 | A | fact | 1.0 | world | medium | 2026-01-01 | | s | |
|
||||
| 2 | B | bogus | 1.0 | world | medium | 2026-01-01 | | s | |`,
|
||||
));
|
||||
await putPage('people/bob', FACT_FENCE(
|
||||
`| 1 | Clean | fact | 1.0 | world | medium | 2026-01-01 | | s | |`,
|
||||
));
|
||||
|
||||
const r = await runExtractFacts(engine, { slugs: ['people/alice', 'people/bob'] });
|
||||
|
||||
expect(r.warnings.some(w => w.includes('FACTS_TABLE_MALFORMED'))).toBe(true);
|
||||
expect(r.pagesScanned).toBe(2);
|
||||
expect(r.factsInserted).toBe(1);
|
||||
expect(r.factsDeleted).toBe(0);
|
||||
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
const rows = await (engine as any).db.query(
|
||||
`SELECT fact FROM facts WHERE source_markdown_slug = 'people/alice' ORDER BY row_num`,
|
||||
);
|
||||
expect(rows.rows.map((row: { fact: string }) => row.fact)).toEqual(['A', 'B']);
|
||||
|
||||
// A warning is page-local: clean pages in the same cycle still reconcile.
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
const cleanRows = await (engine as any).db.query(
|
||||
`SELECT fact FROM facts WHERE source_markdown_slug = 'people/bob'`,
|
||||
);
|
||||
expect(cleanRows.rows.map((row: { fact: string }) => row.fact)).toEqual(['Clean']);
|
||||
});
|
||||
|
||||
test('page with no facts fence → DB facts for that page wiped (empty fence reconciles to empty index)', async () => {
|
||||
await putPage('people/alice', FACT_FENCE(
|
||||
`| 1 | seeded | fact | 1.0 | world | medium | 2026-01-01 | | s | |`,
|
||||
@@ -334,6 +372,186 @@ describe('runExtractFacts — empty-fence guard (Codex R2-#7)', () => {
|
||||
expect(r.factsInserted).toBe(1);
|
||||
});
|
||||
|
||||
test('soft-expired legacy rows do NOT trigger the guard (#2646 — forget_fact drains the backlog)', async () => {
|
||||
// A legacy row that forget_fact already soft-expired. Before #2646
|
||||
// the guard counted it forever: apply-migrations no-ops (migration
|
||||
// marked applied) and forget_fact only sets expired_at, so the
|
||||
// phase was permanently blocked with no sanctioned way out.
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
await (engine as any).db.query(
|
||||
`INSERT INTO facts (source_id, entity_slug, fact, kind, visibility, notability,
|
||||
valid_from, source, confidence, expired_at)
|
||||
VALUES ('default', 'people/alice', 'forgotten legacy claim', 'fact', 'private', 'medium',
|
||||
now(), 'mcp:put_page', 1.0, now())`,
|
||||
);
|
||||
|
||||
await putPage('people/alice', FACT_FENCE(
|
||||
`| 1 | new fact | fact | 1.0 | world | high | 2026-01-01 | | s | |`,
|
||||
));
|
||||
|
||||
const r = await runExtractFacts(engine, { slugs: ['people/alice'] });
|
||||
|
||||
expect(r.guardTriggered).toBe(false);
|
||||
expect(r.legacyRowsPending).toBe(0);
|
||||
expect(r.factsInserted).toBe(1);
|
||||
|
||||
// The expired legacy row itself is untouched (soft-expire is the
|
||||
// record of the forget; the phase must not hard-delete it).
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
const rows = await (engine as any).db.query(
|
||||
`SELECT fact FROM facts WHERE row_num IS NULL AND expired_at IS NOT NULL`,
|
||||
);
|
||||
expect(rows.rows).toHaveLength(1);
|
||||
expect(rows.rows[0].fact).toBe('forgotten legacy claim');
|
||||
});
|
||||
|
||||
test('expired legacy row WITH source_markdown_slug set survives reconcile untouched (#2646 codex P2)', async () => {
|
||||
// Hybrid shape: row_num NULL (legacy — never fence-owned) but
|
||||
// source_markdown_slug matching a live page. Without the
|
||||
// preserveExpiredLegacy filter, the reconcile pass would count it
|
||||
// as "stale", trigger a wipe, hard-delete the forget record, and
|
||||
// reinsert the fence's rows fresh — reviving a forgotten claim as
|
||||
// an active fact.
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
await (engine as any).db.query(
|
||||
`INSERT INTO facts (source_id, entity_slug, fact, kind, visibility, notability,
|
||||
valid_from, source, confidence, expired_at, source_markdown_slug)
|
||||
VALUES ('default', 'people/alice', 'forgotten hybrid claim', 'fact', 'private', 'medium',
|
||||
now(), 'mcp:put_page', 1.0, now(), 'people/alice')`,
|
||||
);
|
||||
|
||||
await putPage('people/alice', FACT_FENCE(
|
||||
`| 1 | fence fact | fact | 1.0 | world | high | 2026-01-01 | | s | |`,
|
||||
));
|
||||
|
||||
const r1 = await runExtractFacts(engine, { slugs: ['people/alice'] });
|
||||
const r2 = await runExtractFacts(engine, { slugs: ['people/alice'] });
|
||||
|
||||
expect(r1.guardTriggered).toBe(false);
|
||||
// The expired hybrid row is invisible to the reconcile: the fence
|
||||
// fact inserts normally, nothing is wiped, and re-running stays
|
||||
// idempotent (the hybrid row must not read as perpetually stale).
|
||||
expect(r1.factsInserted).toBe(1);
|
||||
expect(r1.factsDeleted).toBe(0);
|
||||
expect(r2.factsInserted).toBe(0);
|
||||
expect(r2.factsDeleted).toBe(0);
|
||||
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
const rows = await (engine as any).db.query(
|
||||
`SELECT fact, expired_at FROM facts WHERE source_markdown_slug = 'people/alice' ORDER BY id`,
|
||||
);
|
||||
expect(rows.rows).toHaveLength(2);
|
||||
expect(rows.rows[0].fact).toBe('forgotten hybrid claim');
|
||||
expect(rows.rows[0].expired_at).not.toBeNull();
|
||||
expect(rows.rows[1].fact).toBe('fence fact');
|
||||
expect(rows.rows[1].expired_at).toBeNull();
|
||||
});
|
||||
|
||||
test('expired legacy hybrid row survives even a stale-row wipe on the same page (#2646 codex P2)', async () => {
|
||||
// Force the wipe path: seed a fence, reconcile, then change the
|
||||
// fence so the old DB row goes stale. The wipe must delete the
|
||||
// stale fence-owned row but preserve the expired legacy hybrid.
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
await (engine as any).db.query(
|
||||
`INSERT INTO facts (source_id, entity_slug, fact, kind, visibility, notability,
|
||||
valid_from, source, confidence, expired_at, source_markdown_slug)
|
||||
VALUES ('default', 'people/alice', 'forgotten hybrid claim', 'fact', 'private', 'medium',
|
||||
now(), 'mcp:put_page', 1.0, now(), 'people/alice')`,
|
||||
);
|
||||
|
||||
await putPage('people/alice', FACT_FENCE(
|
||||
`| 1 | old fact | fact | 1.0 | world | high | 2026-01-01 | | s | |`,
|
||||
));
|
||||
await runExtractFacts(engine, { slugs: ['people/alice'] });
|
||||
|
||||
// Replace the fence content — 'old fact' is now stale in the DB.
|
||||
await putPage('people/alice', FACT_FENCE(
|
||||
`| 1 | replacement fact | fact | 1.0 | world | high | 2026-01-01 | | s | |`,
|
||||
));
|
||||
const r = await runExtractFacts(engine, { slugs: ['people/alice'] });
|
||||
|
||||
expect(r.factsDeleted).toBe(1); // only the stale fence-owned row
|
||||
expect(r.factsInserted).toBe(1);
|
||||
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
const rows = await (engine as any).db.query(
|
||||
`SELECT fact FROM facts WHERE source_markdown_slug = 'people/alice' ORDER BY id`,
|
||||
);
|
||||
expect(rows.rows.map((row: { fact: string }) => row.fact))
|
||||
.toEqual(['forgotten hybrid claim', 'replacement fact']);
|
||||
});
|
||||
|
||||
test('fence claim matching an expired legacy row is inserted active — fence is canonical (#2646)', async () => {
|
||||
// Deliberate semantics, pinned: legacy DB-only forgets are
|
||||
// documented NOT to survive rebuild (forget.ts header — the
|
||||
// explicit DB-only exception). When the fence still carries the
|
||||
// same (claim, source), the reconcile inserts a fresh active
|
||||
// fence-owned row; the expired legacy row survives alongside as
|
||||
// the record of the earlier forget. Suppressing the insert would
|
||||
// create silent fence↔DB divergence ("0 facts" while the fence
|
||||
// says otherwise) — the exact failure mode the guard prevents.
|
||||
// To durably forget, forget the fence-owned row (fence path).
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
await (engine as any).db.query(
|
||||
`INSERT INTO facts (source_id, entity_slug, fact, kind, visibility, notability,
|
||||
valid_from, source, confidence, expired_at, source_markdown_slug)
|
||||
VALUES ('default', 'people/alice', 'shared claim', 'fact', 'private', 'medium',
|
||||
now(), 's', 1.0, now(), 'people/alice')`,
|
||||
);
|
||||
|
||||
await putPage('people/alice', FACT_FENCE(
|
||||
`| 1 | shared claim | fact | 1.0 | world | high | 2026-01-01 | | s | |`,
|
||||
));
|
||||
|
||||
const r1 = await runExtractFacts(engine, { slugs: ['people/alice'] });
|
||||
const r2 = await runExtractFacts(engine, { slugs: ['people/alice'] });
|
||||
|
||||
expect(r1.factsInserted).toBe(1);
|
||||
expect(r1.factsDeleted).toBe(0);
|
||||
// Idempotent thereafter — the coexisting pair is stable state.
|
||||
expect(r2.factsInserted).toBe(0);
|
||||
expect(r2.factsDeleted).toBe(0);
|
||||
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
const rows = await (engine as any).db.query(
|
||||
`SELECT fact, row_num, expired_at FROM facts
|
||||
WHERE source_markdown_slug = 'people/alice' ORDER BY id`,
|
||||
);
|
||||
expect(rows.rows).toHaveLength(2);
|
||||
expect(rows.rows[0]).toMatchObject({ fact: 'shared claim', row_num: null });
|
||||
expect(rows.rows[0].expired_at).not.toBeNull(); // forget record preserved
|
||||
expect(rows.rows[1]).toMatchObject({ fact: 'shared claim', row_num: 1 });
|
||||
expect(rows.rows[1].expired_at).toBeNull(); // fence-canonical active row
|
||||
});
|
||||
|
||||
test('mixed active + expired legacy rows: guard counts only the active ones (#2646)', async () => {
|
||||
// One active legacy row + one soft-expired legacy row. The guard
|
||||
// must still trigger (an active row is pending backfill) but the
|
||||
// pending count must exclude the expired row — so each forget_fact
|
||||
// visibly drains the counter toward release.
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
await (engine as any).db.query(
|
||||
`INSERT INTO facts (source_id, entity_slug, fact, kind, visibility, notability,
|
||||
valid_from, source, confidence, expired_at)
|
||||
VALUES
|
||||
('default', 'people/alice', 'active legacy claim', 'fact', 'private', 'medium',
|
||||
now(), 'mcp:put_page', 1.0, NULL),
|
||||
('default', 'people/alice', 'expired legacy claim', 'fact', 'private', 'medium',
|
||||
now(), 'mcp:put_page', 1.0, now())`,
|
||||
);
|
||||
|
||||
await putPage('people/alice', FACT_FENCE(
|
||||
`| 1 | new fact | fact | 1.0 | world | high | 2026-01-01 | | s | |`,
|
||||
));
|
||||
|
||||
const r = await runExtractFacts(engine, { slugs: ['people/alice'] });
|
||||
|
||||
expect(r.guardTriggered).toBe(true);
|
||||
expect(r.legacyRowsPending).toBe(1);
|
||||
expect(r.factsInserted).toBe(0);
|
||||
expect(r.factsDeleted).toBe(0);
|
||||
});
|
||||
|
||||
test('NULL entity_slug legacy rows do NOT trigger the guard (they are structurally unfenceable)', async () => {
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
await (engine as any).db.query(
|
||||
|
||||
@@ -256,6 +256,11 @@ test('applyForwardReferenceBootstrap covers every forward reference declared in
|
||||
ALTER TABLE pages DROP COLUMN IF EXISTS generation;
|
||||
ALTER TABLE pages DROP COLUMN IF EXISTS contextual_retrieval_mode;
|
||||
ALTER TABLE pages DROP COLUMN IF EXISTS corpus_generation;
|
||||
|
||||
DROP INDEX IF EXISTS idx_timeline_event_dedup;
|
||||
DROP INDEX IF EXISTS idx_timeline_event_page;
|
||||
ALTER TABLE timeline_entries DROP CONSTRAINT IF EXISTS timeline_entries_event_page_id_fkey;
|
||||
ALTER TABLE timeline_entries DROP COLUMN IF EXISTS event_page_id;
|
||||
`);
|
||||
|
||||
// Note: we don't strip sources.archived* here because they're inline in the
|
||||
@@ -264,6 +269,14 @@ test('applyForwardReferenceBootstrap covers every forward reference declared in
|
||||
// The bootstrap's needsPagesBootstrap branch recreates sources without the
|
||||
// archive columns; the new needsSourcesArchive probe adds them.
|
||||
|
||||
const { rows: preBootstrapTimelineEventPageId } = await db.query(`
|
||||
SELECT 1 FROM information_schema.columns
|
||||
WHERE table_schema = 'public'
|
||||
AND table_name = 'timeline_entries'
|
||||
AND column_name = 'event_page_id'
|
||||
`);
|
||||
expect(preBootstrapTimelineEventPageId).toHaveLength(0);
|
||||
|
||||
// Run bootstrap in isolation (NOT initSchema). This is what we're testing.
|
||||
await (engine as any).applyForwardReferenceBootstrap();
|
||||
|
||||
@@ -328,6 +341,11 @@ test('after bootstrap, PGLITE_SCHEMA_SQL replays without crashing on missing for
|
||||
ALTER TABLE pages DROP COLUMN IF EXISTS import_filename;
|
||||
ALTER TABLE pages DROP COLUMN IF EXISTS salience_touched_at;
|
||||
ALTER TABLE pages DROP COLUMN IF EXISTS emotional_weight;
|
||||
|
||||
DROP INDEX IF EXISTS idx_timeline_event_dedup;
|
||||
DROP INDEX IF EXISTS idx_timeline_event_page;
|
||||
ALTER TABLE timeline_entries DROP CONSTRAINT IF EXISTS timeline_entries_event_page_id_fkey;
|
||||
ALTER TABLE timeline_entries DROP COLUMN IF EXISTS event_page_id;
|
||||
`);
|
||||
|
||||
// Bootstrap, then schema replay. Either step crashing fails the test.
|
||||
|
||||
@@ -31,9 +31,9 @@ function microBump(): string {
|
||||
function stub(tag: string | null, changelog: string): void {
|
||||
globalThis.fetch = (async (url: any) => {
|
||||
const u = String(url);
|
||||
if (u.includes('/releases/latest')) {
|
||||
if (u.includes('/gbrain/master/VERSION')) {
|
||||
if (tag === null) throw new Error('network down');
|
||||
return new Response(JSON.stringify({ tag_name: tag, published_at: '2026-01-01', html_url: 'https://x/rel' }), { status: 200 });
|
||||
return new Response(tag + '\n', { status: 200 });
|
||||
}
|
||||
if (u.includes('CHANGELOG.md')) return new Response(changelog, { status: 200 });
|
||||
return new Response('', { status: 200 });
|
||||
@@ -65,7 +65,7 @@ describe('self-upgrade --check-only surfaces what you get', () => {
|
||||
const out = JSON.parse(captured.join('\n'));
|
||||
expect(out.update_available).toBe(true);
|
||||
expect(out.latest_version).toBe(latest);
|
||||
expect(out.release_url).toBe('https://x/rel');
|
||||
expect(out.release_url).toBe('https://github.com/garrytan/gbrain/blob/master/CHANGELOG.md');
|
||||
expect(out.changelog_diff).toContain('Shiny new thing');
|
||||
});
|
||||
|
||||
|
||||
Reference in New Issue
Block a user