mirror of
https://github.com/garrytan/gbrain.git
synced 2026-08-15 17:32:37 +00:00
Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
7bee2e48f3 | ||
|
|
3aa064bcc6 | ||
|
|
53bb974eaf | ||
|
|
6136e13997 | ||
|
|
b3b43d0f91 | ||
|
|
5b9a87f1a3 | ||
|
|
661f1f05cc | ||
|
|
3fec2123d2 | ||
|
|
176836f84d | ||
|
|
539d015cc5 | ||
|
|
91464564cd | ||
|
|
bd049d2969 | ||
|
|
784358f5fd | ||
|
|
d58bb2b0bb | ||
|
|
b252acfce3 | ||
|
|
bdd23cdede | ||
|
|
45689dd1bd | ||
|
|
6920744dd8 | ||
|
|
3df20f9f18 | ||
|
|
e58abd652c | ||
|
|
a104f98dca | ||
|
|
2ac6959b46 | ||
|
|
18ec732e1b | ||
|
|
fd8be831c5 | ||
|
|
9664cad329 | ||
|
|
fcc6e670f2 | ||
|
|
2a17a4dab5 | ||
|
|
ddd66e1d25 | ||
|
|
faf5cdba54 | ||
|
|
e9fa962929 | ||
|
|
0413c93e72 | ||
|
|
56aac51a08 | ||
|
|
0bbaed2e48 |
@@ -0,0 +1,16 @@
|
||||
# Line-ending policy.
|
||||
#
|
||||
# Shell scripts MUST be checked out with LF endings on every platform.
|
||||
# Git for Windows installs with `core.autocrlf=true` by default, which
|
||||
# rewrites LF -> CRLF on checkout. A strict bash (WSL, Linux CI, macOS)
|
||||
# then chokes on the trailing CR:
|
||||
#
|
||||
# scripts/run-unit-parallel.sh: line 23: $'\r': command not found
|
||||
# scripts/run-unit-parallel.sh: line 24: set: pipefail : invalid option name
|
||||
# scripts/run-unit-parallel.sh: line 32: syntax error near unexpected token `$'{\r''
|
||||
#
|
||||
# That silently disabled `bun run test`, `bun run verify`, `bun run ci:local`
|
||||
# and `bun run test:e2e` for Windows contributors, since all four dispatch
|
||||
# through bash. `eol=lf` pins the checkout regardless of the user's
|
||||
# core.autocrlf setting.
|
||||
*.sh text eol=lf
|
||||
+5
-2
@@ -1,4 +1,7 @@
|
||||
node_modules/
|
||||
# No trailing slash: a bare `node_modules/` pattern matches directories only,
|
||||
# so a *symlink* named node_modules slips past it and can be committed
|
||||
# (that's how the /tmp-pointing symlink in faf5cdba got in). Match any type.
|
||||
node_modules
|
||||
bin/
|
||||
.DS_Store
|
||||
*.log
|
||||
@@ -15,7 +18,7 @@ supabase/.temp/
|
||||
# self-contained binaries (the bun --compile path embeds it via
|
||||
# `import path from 'admin/dist/index.html' with { type: 'file' }`).
|
||||
# Build via: cd admin && bun install && bun run build.
|
||||
admin/node_modules/
|
||||
admin/node_modules
|
||||
.idea
|
||||
eval/reports/
|
||||
eval/data/world-v1/world.html
|
||||
|
||||
@@ -2,6 +2,45 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.42.67.0] - 2026-07-28
|
||||
|
||||
**If you develop GBrain on Windows, the test and check commands now actually run. Until this release they were quietly doing almost nothing.**
|
||||
|
||||
`bun run test`, `bun run verify`, `bun run ci:local` and `bun run test:e2e` all hand off to shell scripts, and on Windows that hand-off was broken in two separate places. The commands did not stop with an obvious error. They reported a result, so a run could look finished when barely any of the checks had actually inspected anything. On a clean Windows clone, `bun run verify` got 1 check to pass and 31 to fail. It now gets 25 to pass and 7 to fail, and none of the 7 are caused by this change.
|
||||
|
||||
The first problem was line endings. Git for Windows installs with `core.autocrlf=true`, which rewrites shell scripts to Windows line endings when you clone or check out. Bash refuses to run those, so a script died on its second line before doing any work. The scripts stored in the repository were always correct; only the copy on your disk was wrong. A new `.gitattributes` pins every `.sh` file to Unix line endings at checkout, no matter how your Git is configured.
|
||||
|
||||
The second problem was how the checks were started. Thirty three of them pointed straight at a `.sh` file. On macOS and Linux the shell reads the `#!/usr/bin/env bash` line at the top of the script and runs it correctly. Bun on Windows does not do that, so those commands failed the moment they were called. They now go through `bash` explicitly, the same way the other eleven were already written.
|
||||
|
||||
Nothing changes for macOS and Linux. No stored file content moves, and no check behaves differently on those platforms.
|
||||
|
||||
## To take advantage of v0.42.67.0
|
||||
|
||||
Only Windows contributors need to do anything, and only once. `.gitattributes` applies at checkout time, so shell scripts already sitting on your disk keep their old line endings until you refresh them.
|
||||
|
||||
1. **Refresh the working copy** from the repository root:
|
||||
```bash
|
||||
git rm --cached -r . -q
|
||||
git reset --hard
|
||||
```
|
||||
2. **Confirm bash can read the scripts:**
|
||||
```bash
|
||||
bash -n scripts/run-unit-parallel.sh
|
||||
```
|
||||
Silence means it worked. `$'\r': command not found` means step 1 did not take effect.
|
||||
3. **Run the gate:**
|
||||
```bash
|
||||
bun run verify
|
||||
```
|
||||
|
||||
### Itemized changes
|
||||
|
||||
- New root `.gitattributes` pins `*.sh text eol=lf`, so shell scripts check out with Unix line endings regardless of the contributor's `core.autocrlf` setting. All 59 tracked `.sh` files were already stored with Unix endings, so `git add --renormalize .` reports nothing to do and no stored content changes.
|
||||
- `package.json` now routes the remaining 33 `.sh` check commands through `bash`, matching the 11 that already did. Every tracked `.sh` file carries a bash shebang (52 `#!/usr/bin/env bash` and 7 `#!/bin/bash`), so the treatment is uniform across all of them.
|
||||
- The five `scripts/*.ts` entries still run under bun and are untouched.
|
||||
- `CONTRIBUTING.md` gains a Windows section covering the one-time working-copy refresh and the `bash scripts/<name>.sh` convention for new checks.
|
||||
- `docs/TESTING.md` records how the test commands dispatch through bash, and notes that three tree-walking checks plus `typecheck` can exceed the 120s per-check cap on Windows while passing on Linux and macOS.
|
||||
|
||||
## [0.42.66.1] - 2026-07-27
|
||||
|
||||
### Fixed
|
||||
|
||||
@@ -11,6 +11,28 @@ bun test
|
||||
|
||||
Requires Bun 1.0+.
|
||||
|
||||
### Windows
|
||||
|
||||
`bun run test`, `verify`, `ci:local` and `test:e2e` all dispatch through bash, so
|
||||
the shell scripts under `scripts/` must be checked out with Unix line endings.
|
||||
The root `.gitattributes` pins `*.sh text eol=lf`, which overrides the
|
||||
`core.autocrlf=true` that Git for Windows installs by default. A fresh clone is
|
||||
correct with no extra steps.
|
||||
|
||||
If you cloned before that pin existed, your working copy still has the old
|
||||
Windows line endings and bash will fail with `$'\r': command not found`. Refresh
|
||||
it once, from the repository root:
|
||||
|
||||
```bash
|
||||
git rm --cached -r . -q
|
||||
git reset --hard
|
||||
bash -n scripts/run-unit-parallel.sh # silence means bash can read the scripts
|
||||
```
|
||||
|
||||
Every `check:*` entry in `package.json` invokes its script as `bash scripts/<name>.sh`
|
||||
rather than relying on the shebang, because bun on Windows cannot exec a `.sh`
|
||||
directly. Keep that prefix when you add a new shell-script check.
|
||||
|
||||
## Project structure
|
||||
|
||||
```
|
||||
|
||||
@@ -1,5 +1,25 @@
|
||||
# TODOS
|
||||
|
||||
## v0.42.67.0 follow-ups (Windows build tooling)
|
||||
|
||||
Filed as follow-ups from v0.42.67.0 (`.gitattributes` LF pin for `*.sh` +
|
||||
`bash` prefix on the 33 `package.json` check commands). Both items are newly
|
||||
observable: before that release these checks never executed on Windows at all,
|
||||
so nothing about their runtime was measurable.
|
||||
|
||||
- [ ] **P2 — three guard scripts exceed the 120s `run-verify-parallel.sh` cap on Windows.**
|
||||
With the dispatch fixed, `bun run verify` on Windows gets 25 passes and 7 failures, and
|
||||
`check:privacy`, `check:test-names` and `check:test-isolation` are timeouts rather than
|
||||
real failures (they pass on Linux and macOS well inside the cap). They walk the tree with
|
||||
per-file shell loops, which is far slower under Windows process creation. Either raise the
|
||||
cap for these three, or replace the per-file loop with a single `grep -r` pass. Same cap
|
||||
swallows `typecheck`, though standalone `bun run typecheck` exits 0.
|
||||
- [ ] **P3 — `check:wasm` cannot create its `node_modules` symlink on Windows.**
|
||||
`scripts/check-wasm-embedded.sh` fails with `ln: failed to create symbolic link
|
||||
'/tmp/gbrain-wasm-check.XXXX/node_modules': No such file or directory`. Unprivileged
|
||||
Windows accounts cannot create symlinks without developer mode. Consider a junction, a
|
||||
copy, or skipping the check with a clear message when symlink creation is unavailable.
|
||||
|
||||
## community fix-wave follow-ups (filed v0.42.60.0)
|
||||
|
||||
- [x] **P2 — cherry-pick #2112's uncovered doctor.ts hunk.** Fix-wave A (#2820) superseded
|
||||
@@ -62,17 +82,20 @@ Deferred from the provider-agnostic plumbing wave (#1249/#1250/#1292/#2271/#2209
|
||||
Plan + review trail at `~/.claude/plans/system-instruction-you-are-working-keen-newell.md`.
|
||||
The eng-review + Codex outside-voice narrowed the wave to these deferrals:
|
||||
|
||||
- [ ] **P2 — Capability-aware query expansion on OpenAI-compat providers (#2372).**
|
||||
- [x] **P2 — Capability-aware query expansion on OpenAI-compat providers (#2372).**
|
||||
Expansion only runs for recipes that declare an `expansion` touchpoint, and only the
|
||||
native providers (anthropic/openai/google) do. To make expansion work on
|
||||
litellm/openrouter/groq/together/deepseek you must ADD expansion touchpoints to those
|
||||
chat-capable recipes AND add a `generateObject`→`generateText` capability fallback for
|
||||
backends without strict structured outputs. Feature-shaped; overlaps the general
|
||||
OpenAI-compat proxy story (`docs/designs/COMMUNITY_IDEAS.md`). Community PR #2373 is a
|
||||
starting point. Where: `src/core/ai/gateway.ts:expand`, recipe files, `types.ts` (ExpansionTouchpoint).
|
||||
- [ ] **P2 — LiteLLM as a chat/expansion backend.** `litellm-proxy` declares ONLY an
|
||||
starting point. Implemented by #2373 plus the DeepSeek/Groq/Together recipe wave,
|
||||
LiteLLM chat/expansion support, and the OpenRouter expansion touchpoint. Where:
|
||||
`src/core/ai/gateway.ts:expand`, recipe files, `types.ts` (ExpansionTouchpoint).
|
||||
- [x] **P2 — LiteLLM as a chat/expansion backend.** `litellm-proxy` declares ONLY an
|
||||
embedding touchpoint, so `think`/chat on LiteLLM is dead. Add chat (and expansion) so a
|
||||
LiteLLM proxy is a full LLM backend, not embedding-only. The general OpenAI-compat proxy story.
|
||||
LiteLLM proxy is a full LLM backend, not embedding-only. Implemented by #2208.
|
||||
The general OpenAI-compat proxy story.
|
||||
- [ ] **P3 — Per-model embedding dims metadata on `EmbeddingTouchpoint`.** `default_dims`
|
||||
is recipe-wide, so a recipe (ollama) can't carry different native dims per model. This
|
||||
wave added the modern ollama model NAMES + a `trust_custom_dims` passthrough (user supplies
|
||||
|
||||
@@ -19,6 +19,29 @@ Seven test command tiers, each with a clear scope:
|
||||
| `bun run test:e2e` | Real Postgres E2E. Requires Docker + `DATABASE_URL`. Sequential. | ~5-10min | Pre-ship; nightly. |
|
||||
| `bun run check:all` | The historical pre-check scripts (22, chained sequentially in package.json). Overlaps `verify` heavily but is NOT a superset — `verify`'s `CHECKS` array in `scripts/run-verify-parallel.sh` (~30 entries incl. typecheck) is the authoritative gate; `check:all` keeps a few local-only extras (trailing-newline, exports-count, no-legacy-getconnection). | ~10s | Local-only sweep for the extras. |
|
||||
|
||||
### Shell dispatch and Windows
|
||||
|
||||
All four of `test`, `verify`, `ci:local` and `test:e2e` hand off to shell scripts
|
||||
under `scripts/`, so every `check:*` entry in `package.json` invokes its script as
|
||||
`bash scripts/<name>.sh` instead of relying on the shebang — bun on Windows cannot
|
||||
exec a `.sh` directly. Add a new shell-script check with that same prefix. The
|
||||
`scripts/*.ts` entries run under bun and take no prefix.
|
||||
|
||||
The scripts must also be on disk with Unix line endings. A strict bash (WSL, Linux
|
||||
CI, macOS) rejects CRLF and dies on the script's first meaningful line; the Cygwin
|
||||
bash that ships with Git for Windows tolerates it, so a green local run is not by
|
||||
itself evidence that a script is CRLF-clean.
|
||||
The root `.gitattributes` pins `*.sh text eol=lf`, which overrides the
|
||||
`core.autocrlf=true` default that Git for Windows installs. Working copies cloned
|
||||
before that pin need a one-time `git rm --cached -r . -q && git reset --hard` to
|
||||
pick it up; see the Windows section of `CONTRIBUTING.md`.
|
||||
|
||||
Wallclock figures in the table above are from a Mac dev box. Windows is
|
||||
substantially slower because each check pays full process-creation cost, and three
|
||||
tree-walking checks (`check:privacy`, `check:test-names`, `check:test-isolation`)
|
||||
plus `typecheck` can exceed the 120s per-check cap in `run-verify-parallel.sh`
|
||||
there even though they pass on Linux and macOS.
|
||||
|
||||
### CI vs local: intentionally divergent file sets
|
||||
|
||||
- **CI matrix** (`.github/workflows/test.yml`) runs `scripts/test-shard.sh` across 10 matrix shards partitioned by weight-aware LPT bin-packing (`scripts/sharding.ts`) and INCLUDES `*.slow.test.ts` (the two outlier slow files run as dedicated jobs alongside the matrix). CI EXCLUDES `*.serial.test.ts` from the shards and runs them in a dedicated job via `bun run test:serial`, one bun process per file — keeping serial files out of the shard processes is what preserves the `mock.module` quarantine (a top-level mock in one file leaks into every other file sharing its process). `bun run verify` gets its own job too. CI is the ground truth for "did everything pass."
|
||||
|
||||
@@ -188,7 +188,8 @@ per-release `**vX.Y.Z:**` narration — CI enforces this
|
||||
- `scripts/check-no-double-retry.sh` + `scripts/check-batch-audit-site.sh` — CI lint guards wired into `bun run verify`. The former greps src/ for `withRetry(...engine.{addLinksBatch|addTimelineEntriesBatch|upsertChunks})` patterns and fails the build on hit (prevents 3×3=9 retry amplification on incomplete reverts). The latter extracts every string-literal `auditSite: '...'` from src/ and validates each appears in the `BATCH_AUDIT_SITES` const in `src/core/retry.ts` (typo guard — prevents fragmented doctor output).
|
||||
- `src/core/fail-improve.ts` — Deterministic-first, LLM-fallback loop with JSONL failure logging and auto-test generation.
|
||||
- `src/core/transcription.ts` — Audio transcription: Groq Whisper (default), OpenAI fallback, ffmpeg segmentation for >25MB.
|
||||
- `src/core/enrichment-service.ts` — Global enrichment service: entity slug generation, tier auto-escalation, batch throttling.
|
||||
- `src/core/enrichment-service.ts` — Global enrichment service: entity slug generation, tier auto-escalation, batch throttling. Write path is trust-gated (issue #160): `enrichEntity` / `enrichEntities` / `extractAndEnrich` take `EnrichmentTrustOptions { trusted?, sourceId? }`; only an explicit `trusted: true` writes authoritative `people/` / `companies/` stubs. Anything else (undefined/false — fail-closed, mirroring `OperationContext.remote`) creates the stub with the extraction quarantine markers from `src/core/extraction-review.ts` and reports `quarantined: true` in `EnrichmentResult`. The ONLY sanctioned op surface is `extract_entities` (operations.ts), which grants `trusted` solely for `ctx.remote === false` callers passing `--trusted-extraction`.
|
||||
- `src/core/extraction-review.ts` — Extraction quarantine lane markers (issue #160), sibling of `src/core/quarantine.ts` / `embed-skip.ts` (frontmatter-key pattern, no schema migration). Auto-extracted stubs from untrusted input carry the PAIR `provenance: 'auto-extracted'` + `status: 'unverified'` (both required — user pages with their own `status`/`provenance` never match). Exports `quarantineMarkers()`, `isUnverifiedExtraction()` (JS predicate) and `unverifiedExtractionFragment(alias)` — the single SQL source of truth consumed by `buildSourceFactorCase` (namespace source-boost guard), both engines' `getUnverifiedExtractionPageIds`, the `extraction_pending` op, and the `unverified_extractions` doctor check, so filter and marker keys can never drift. Consequences: unverified stubs are excluded from the compiled-truth fusion boost + the `people/`/`companies/` source-boost (rank as ordinary content), stamped `unverified: true` in search results (`stampUnverifiedExtractions`, hybrid.ts), listed by `extraction_pending`, promoted (status → `verified`, provenance kept for audit) or rejected (soft-delete) by the owner-only `extraction_review` op. Pinned by `test/extraction-review.test.ts` (PGLite) + `test/e2e/extraction-review-postgres.test.ts` (live Postgres parity).
|
||||
- `src/commands/enrich.ts` + `src/core/enrich/thin.ts` + `src/core/cycle/enrich-thin.ts` — `gbrain enrich --thin`: batch-develops stub (thin) pages via **brain-internal grounded synthesis**. gbrain's model tooling sees only brain-internal context (search / get_page / facts / backlinks), not the web, so enrich consolidates what the brain ALREADY knows about an entity (scattered across meetings, other pages, deals, facts) into one cited page via ONE `gateway.chat` call per page; web research stays the agent-driven `enrich` SKILL's job. `runEnrichCore(engine, opts, signal)` (strict per-source; multi-source iteration is the caller's job) drives `enrichOne` per candidate: `withRefreshingLock('enrich:<src>:<slug>')` → `getPage` → deterministic retrieve (hybridSearch + getBacklinks + facts + raw_data, source-scoped, sanitized via `INJECTION_PATTERNS`) → `assessGrounding` gate (skip < `MIN_CONTEXT_CHARS`, no LLM) → `buildEnrichPrompt` (grounded dossier, `[Source: slug]` citations, SKIP sentinel) → synth → `put_page` handler (`remote:false`, auto-link + write-through) stamping `enriched_at` + `enriched_by:'cli:enrich'`. Candidate selection is the SQL-native `engine.listEnrichCandidates(opts)` (`src/core/engine.ts` interface + `EnrichCandidate`/`EnrichCandidatesOpts`/`ENRICH_ORDER_SQL` in `src/core/types.ts` + pg/pglite impls): thin-filter + per-page source-correct inbound count (`to_page_id = p.id`, `mentions` excluded) + `enriched_at` recency guard + whitelisted ORDER BY + LIMIT, lightweight projection (NO bodies). Resume via `src/core/op-checkpoint.ts` (local `enrichFingerprint`); budget via `BudgetTracker` + `withBudgetTracker` (best-effort under `--workers > 1` — `runSlidingPool` aborts new claims on `BUDGET_EXHAUSTED` but does NOT cancel in-flight `gateway.chat`; pin `--workers 1` for a hard ceiling). `sanitizeContext` (thin.ts) neutralizes the `<context>…</context>` data-envelope delimiters (injection escape, mirrors the `</trajectory>` convention); the `--background` multi-source fan-out idempotency key carries the run fingerprint via exported `backgroundIdempotencyKey(sid, args)` (a bare `enrich:${sid}` would return stale completed jobs); `runEnrichCore` flags `budget_exhausted` post-hoc when `tracker.totalSpent > tracker.cap` even when the gateway swallowed the final-call throw (via read-only `BudgetTracker.cap` getter); `body()` flushes the checkpoint on `BudgetExhausted` before it propagates so resume doesn't re-charge. The opt-in `enrich_thin` cycle phase (default OFF via `cycle.enrich_thin.enabled`) trickles `max_pages_per_tick` (default 3) per source with per-source cost cap enforced as `min(per_source_cap, brain_wide_remaining)` + brain-wide total + walltime caps. Wired into `cycle.ts` (`CyclePhase`/`ALL_PHASES` between `conversation_facts_backfill` and `skillopt`/`embed`; `PHASE_SCOPE='source'`; `NEEDS_LOCK`; dispatch), `cli.ts` (`CLI_ONLY` + `CLI_ONLY_SELF_HELP` + `THIN_CLIENT_REFUSED_COMMANDS` + dispatch), `jobs.ts` (Minion `enrich` handler, strict per-source, NOT in `PROTECTED_JOB_NAMES`). DI seam `opts.synthesizeFn` keeps tests hermetic (no API key, no mock.module). Pinned by `test/enrich/thin.test.ts`, `test/enrich/idempotency.test.ts`, `test/enrich-cycle-phase.test.ts`, `test/e2e/enrich-pglite.test.ts` (grew-cited, skip, ordering, multi-source, recency, resume, budget abort + checkpoint flush, final-call overage, lock-skip, provenance), `test/e2e/engine-parity.test.ts` (`listEnrichCandidates` pg↔pglite parity).
|
||||
- `src/core/data-research.ts` — Recipe validation, field extraction (MRR/ARR regex), dedup, tracker parsing, HTML stripping.
|
||||
- `src/commands/embed.ts` — `gbrain embed [--stale|--all] [--slugs ...]`. `--stale` starts with `engine.countStaleChunks()` (single SELECT count(*) WHERE embedding IS NULL, ~50 bytes wire) so a fully-embedded brain short-circuits with no further reads. When stale chunks exist, `engine.listStaleChunks()` returns just the chunks needing embeddings (slug + chunk_index + chunk_text + metadata, no `vector(1536)` payload); caller groups by slug, embeds, re-upserts via `upsertChunks`. All `console.log`/`console.error` call sites use `slog`/`serr` from `src/core/console-prefix.ts` so when `runEmbedCore` runs inside a per-source `withSourcePrefix` scope (installed by the `gbrain sync --all` worker pool) every line carries the `[<source-id>] ` prefix; standalone callers see identical output because slog/serr fall through to bare console fns outside the wrap. Every embed-write path stamps `pages.embedding_signature` via `engine.setPageEmbeddingSignature(slug, {sourceId, signature: currentEmbeddingSignature()})` so a later model/dims swap is detectable as stale. The per-slug path (`embedPage`, used by `gbrain embed <slug>` AND sync's post-import embed step) and the full-re-embed path (`embedAll`) stamp unconditionally per page. The stale path (`embedAllStale`) first calls `invalidateStaleSignatureEmbeddings` on a live run so signature-drifted pages flow through the NULL cursor, then stamps each page — but ONLY when EVERY chunk was stale this pass (a partially-stale page keeps preserved chunks of unknown provenance, so it stays unstamped rather than falsely marked current; `embed --all` fully re-embeds + stamps those). dry-run never mutates: it counts signature-drift via the widened `countStaleChunks({signature})` predicate without NULLing anything. `--include-null-signature` (#3391) lifts the NULL-signature grandfather clause: threads `includeNullSignature: true` into the invalidation + counts so pages that predate the v108 stamp re-embed too after a model swap (both engines' `countStaleChunks`/`sumStaleChunkChars`/`invalidateStaleSignatureEmbeddings` accept the flag; predicate becomes `sig IS NULL OR sig <> current`). Without the flag, a live stale run that just invalidated drifted rows probes for left-behind NULL-signature chunks and emits a loud stderr warning naming the count + the fix — mixed embedding spaces in one index are never silent. Pinned by `test/embedding-migration.test.ts` + `test/e2e/migrate-embeddings-postgres.test.ts`.
|
||||
@@ -395,7 +396,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this
|
||||
- `src/commands/lint.ts` — Page quality linter (catches LLM artifacts, placeholder dates)
|
||||
- `src/commands/report.ts` — Structured report saver (audit trail for maintenance/enrichment)
|
||||
- `src/core/destructive-guard.ts` — three-layer protection against accidental data loss. `assessDestructiveImpact(engine, sourceId)` counts pages/chunks/embeddings/files for a source. `checkDestructiveConfirmation(impact, opts)` is the fail-closed gate (`--confirm-destructive` required when data is present; `--yes` alone is rejected). `softDeleteSource` / `restoreSource` / `listArchivedSources` / `purgeExpiredSources` drive the source-level archive lifecycle via `sources.archived BOOLEAN`, `archived_at TIMESTAMPTZ`, `archive_expires_at TIMESTAMPTZ`. Page-level analog: `BrainEngine.softDeletePage` / `restorePage` / `purgeDeletedPages` plus `pages.deleted_at TIMESTAMPTZ` and a partial purge index. The MCP `delete_page` op rewires to `softDeletePage`; ops `restore_page` (`scope: write`) and `purge_deleted_pages` (`scope: admin`, `localOnly: true`) round out the surface. Search visibility (`buildVisibilityClause` in `src/core/search/sql-ranking.ts`) hides soft-deleted pages and archived sources from `searchKeyword` / `searchKeywordChunks` / `searchVector` in both engines. The autopilot cycle's `purge` phase calls `purgeExpiredSources` + `engine.purgeDeletedPages(72)` so the 72h TTL is real.
|
||||
- `src/commands/pages.ts` — `gbrain pages purge-deleted [--older-than HOURS|Nd] [--dry-run] [--json]` operator escape hatch. Mirror of `gbrain sources purge` for the page-level lifecycle. Hard-deletes pages whose `deleted_at` is older than the cutoff; cascades to content_chunks/page_links/chunk_relations.
|
||||
- `src/commands/pages.ts` — `gbrain purge-deleted [--older-than HOURS|Nd] [--dry-run] [--json]` operator escape hatch. Mirror of `gbrain sources purge` for the page-level lifecycle. Hard-deletes pages whose `deleted_at` is older than the cutoff; cascades to content_chunks/page_links/chunk_relations.
|
||||
- `src/core/op-checkpoint.ts` — DB-backed checkpoint primitive for long-running ops. Migration v67 introduces `op_checkpoints (op TEXT, fingerprint TEXT, completed_keys JSONB, updated_at TIMESTAMPTZ, PK(op, fingerprint))`. Per-op fingerprint helpers (`embedFingerprint`, `extractFingerprint`, `reindexFingerprint`, `integrityFingerprint`, `purgeFingerprint`) compute `sha8(canonical-JSON(relevant-params))` so re-running with the same params resumes from `completed_keys` and re-running with different params (e.g. `--limit 100` vs `--limit 200`) starts fresh. Cross-worker safe on Postgres (DB row, no file-lock race); PGLite degrades gracefully. Replaces per-op file-backed JSON checkpoints scattered across `import.ts`, `embed.ts`, `reindex.ts`. The 7-day TTL GC runs in the cycle's `purge` phase. All writes (`recordCompleted`, `clearOpCheckpoint`) route through `engine.executeRawDirect` + `withRetry(BULK_RETRY_OPTS)` so they survive Supavisor pool exhaustion, and `recordCompleted` returns `boolean` (banked vs failed-after-retries) — the 9 non-sync consumers keep its REPLACE-into-`completed_keys` semantics. Resumable sync uses the additive `appendCompleted(key, deltaKeys)` / `appendCompletedOnce` (the latter no-retry for the SIGTERM path) which INSERT a delta into the `op_checkpoint_paths` child table (migration v115: `(op, fingerprint, path)` PK, FK to `op_checkpoints` ON DELETE CASCADE) via a single writable-CTE `unnest($3::text[])` write — O(delta), killing the old O(N²) full-set rewrite. `loadOpCheckpoint` returns the `UNION ALL` of legacy `completed_keys` + child-table paths (deduped in JS), so an in-flight upgrade loses nothing. The legacy arm is gated on `jsonb_typeof(completed_keys) = 'array'` so a non-array (scalar) parent row can't make `jsonb_array_elements_text` throw "cannot extract elements from a scalar" and take down the whole union (which would discard the valid child rows and lose all banked progress for the key); a third union arm flags the corruption so the loader logs it once and keeps the child rows. Migration v119 adds the `op_checkpoints_completed_keys_array` CHECK (`jsonb_typeof(completed_keys) = 'array'`) — a DB-enforced, always-on guard that makes the scalar-corruption class structurally impossible going forward; the migration repairs any pre-existing scalar to `'[]'` under `LOCK TABLE ... IN SHARE ROW EXCLUSIVE MODE` and `src/core/schema-embedded.ts` + `src/core/pglite-schema.ts` ship the same CHECK on fresh installs (a loader hit now implies schema drift, a disabled constraint, or an out-of-band writer). `recordCompleted` binds its array through `$3::text::jsonb` (NOT a bare `$3::jsonb`) so postgres.js `.unsafe()` doesn't double-encode `JSON.stringify(sorted)` into the scalar string that CHECK rejects — the #2339 bug that aborted every multi-source sync at the first pin write (PGLite parsed it silently, so it shipped). A DATABASE_URL-gated `test/e2e/op-checkpoint-jsonb-parity.test.ts` (its own CI job) asserts the array shape on real Postgres. `syncFingerprint({sourceId, lastCommit})` keys the sync rows. Pinned by `test/op-checkpoint.test.ts` (incl. delta-append, union read, cascade clear, durable-write boolean, and the scalar-parent guard). `import-checkpoint.ts` was NOT migrated to this primitive — both checkpoint systems coexist without conflict; migrating requires async-propagating 4 sync call sites in `src/commands/import.ts` and rewriting 18 tests, deferred.
|
||||
- `src/core/brain-score-recommendations.ts` — pure data layer consumed by both `gbrain doctor --remediation-plan` / `--remediate` and `gbrain features`. `computeRecommendations(checks, opts)` returns `Remediation[]` with stable `id`, content-hash `idempotency_key`, `severity`, `est_seconds`, `est_usd_cost`, `depends_on` (references stable ids, not check names — so plan order is reproducible). `classifyChecks(report)` triages every doctor check three-state into `remediable | human_only | blocked` (`human_only` covers RLS warnings and other human-judgment gates; `blocked` covers dependency chains where a parent check failed). `maxReachableScore(checks)` computes the ceiling for empty/under-configured brains (no entity pages → graph_coverage caps at 70; no embedding key → embedding_coverage caps at 60). Cost estimates pull from `anthropic-pricing.ts` (synthesize/patterns/consolidate) and `embedding-pricing.ts` (embed jobs). Pinned by `test/brain-score-recommendations.test.ts` (~27 cases incl. determinism, content-hash idempotency, DB-backed checkpoint provenance, three-state triage).
|
||||
- `src/commands/doctor.ts` extension — `--remediation-plan [--json] [--target-score N]` prints what would run (stable `id`, `idempotency_key`, `severity`, `est_seconds`, `est_usd_cost`, `depends_on`); `--remediate [--yes] [--target-score N] [--max-usd N]` submits each plan step as a Minion job in dependency order, re-checking score between steps. `--target-score N` defaults to 90; refuses to start when target exceeds `maxReachableScore()` and lists what's missing. `--max-usd N` is the cron-safety guard — submission refuses when the plan's `est_total_usd_cost` exceeds the cap. JSON envelope adds a `Check.remediation` field (additive, schema_version unchanged). Pinned by tests in `test/doctor.test.ts`.
|
||||
|
||||
@@ -87,6 +87,15 @@ embedding proximity. Four layers, added after the incident in
|
||||
deciding "is this page already here, safe to NOT write a duplicate?" keys off
|
||||
`create_safety`, not a raw blended score.
|
||||
|
||||
**Extraction quarantine lane (issue #160):** pages carrying the unverified
|
||||
auto-extracted markers (frontmatter `provenance: auto-extracted` +
|
||||
`status: unverified`, see `src/core/extraction-review.ts`) rank as ordinary
|
||||
content — they are skipped by the compiled-truth fusion boost and by the
|
||||
`people/`/`companies/` namespace source-boost, and every search result from
|
||||
such a page carries `unverified: true` so agents can label the provenance.
|
||||
Promote or reject them via `gbrain extraction-pending` / `gbrain
|
||||
extraction-review`.
|
||||
|
||||
The `search` MCP/CLI op is **cheap-hybrid** (vector + keyword + RRF + pool +
|
||||
title + alias, expansion off); `query` is the full-control variant. NamedThingBench
|
||||
(`gbrain eval retrieval-quality`) gates these families on every PR. Diagnose a
|
||||
|
||||
@@ -229,7 +229,7 @@ add `GBRAIN_AUDIT_FULL=1` (v0.43+ TODO; not yet wired).
|
||||
- Per-source pack-upgrade (the handler accepts `sourceId` but
|
||||
`findPackSuccessors` doesn't yet pass it through)
|
||||
- Cross-brain federated mounts that disagree on canonical packs
|
||||
- Automatic rollback (today: manual SQL or `gbrain pages restore`)
|
||||
- Automatic rollback (today: manual SQL or `gbrain restore`)
|
||||
- LLM-assisted mapping_rules codegen from production data (`gbrain
|
||||
schema detect-mappings`; deferred to v0.43+)
|
||||
|
||||
|
||||
@@ -214,7 +214,7 @@ gbrain schema downgrade
|
||||
|
||||
1. `git revert <merge-commit>` — restores the code.
|
||||
2. `gbrain schema downgrade --to gbrain-base` — restores config.
|
||||
3. (Optional) `gbrain pages purge-deleted --older-than 0h` — drops
|
||||
3. (Optional) `gbrain purge-deleted --older-than 0h` — drops
|
||||
v0.39-typed pages that no longer have a matching type in the active
|
||||
pack.
|
||||
|
||||
|
||||
@@ -19,11 +19,13 @@ entire DB from scratch.
|
||||
|
||||
This means:
|
||||
|
||||
- **Disaster recovery is one command.** If your DB volume corrupts, if
|
||||
Postgres eats itself, if PGLite's WASM lock wedges — you don't need
|
||||
a backup. You wipe the DB, re-import from your brain repo, and the
|
||||
derived state regenerates. v0.32.3 ships `gbrain rebuild
|
||||
--confirm-destructive` as the documented one-liner.
|
||||
- **Disaster recovery is a short, boring sequence.** If your DB volume
|
||||
corrupts, if Postgres eats itself, if PGLite's WASM lock wedges — you
|
||||
don't need a backup. You wipe the derived tables (on PGLite,
|
||||
`gbrain reinit-pglite` wipes the whole embedded DB), re-import from
|
||||
your brain repo with `gbrain sync`, and `gbrain extract all`
|
||||
regenerates the derived state. See "Disaster recovery" below for the
|
||||
exact commands.
|
||||
- **Multi-machine sync is git.** Your brain is a repo. Push from one
|
||||
machine, pull from another, and the second machine's DB rebuilds on
|
||||
its next sync. No "back up the database" step.
|
||||
@@ -146,11 +148,9 @@ The promise the rule makes:
|
||||
# Snapshot what's there
|
||||
gbrain stats > /tmp/before.txt
|
||||
|
||||
# Wipe and rebuild
|
||||
gbrain rebuild --confirm-destructive # v0.32.3 — deletes derived tables
|
||||
# (pages + content_chunks survive
|
||||
# the CASCADE-safe design)
|
||||
# OR manually for v0.32.2:
|
||||
# Wipe and rebuild — delete the derived tables (pages + content_chunks
|
||||
# survive the CASCADE-safe design), then re-derive from the repo.
|
||||
# On PGLite, `gbrain reinit-pglite` wipes the whole embedded DB instead.
|
||||
psql -c 'DELETE FROM facts; DELETE FROM takes; DELETE FROM links; DELETE FROM timeline_entries;'
|
||||
gbrain sync
|
||||
gbrain extract all
|
||||
|
||||
@@ -108,8 +108,8 @@ Every primitive ships with a documented rollback:
|
||||
| Operation | Rollback |
|
||||
|-----------|----------|
|
||||
| Retype | `frontmatter.legacy_type = <original>` preserved on every page (D8). One SQL UPDATE restores types: `UPDATE pages SET type = frontmatter->>'legacy_type' WHERE frontmatter ? 'legacy_type'`. |
|
||||
| Page-to-link | Source page soft-deleted with 72h TTL. `gbrain pages restore <slug>` within 72h. Link row stays harmless if source restored. |
|
||||
| Page-to-alias | Source page soft-deleted with 72h TTL. `gbrain pages restore <slug>` within 72h. Alias row stays harmless (or `DELETE FROM slug_aliases WHERE alias_slug = <slug>` to clean up). |
|
||||
| Page-to-link | Source page soft-deleted with 72h TTL. `gbrain restore <slug>` within 72h. Link row stays harmless if source restored. |
|
||||
| Page-to-alias | Source page soft-deleted with 72h TTL. `gbrain restore <slug>` within 72h. Alias row stays harmless (or `DELETE FROM slug_aliases WHERE alias_slug = <slug>` to clean up). |
|
||||
| Active-pack flip | `gbrain schema use gbrain-base` reverses the flip. |
|
||||
|
||||
## What if my brain doesn't fit?
|
||||
|
||||
@@ -183,6 +183,6 @@ This also means the best AI agent setups will be open source by default. Closed,
|
||||
|
||||
Software distribution reimagined: the package is a markdown file, the runtime is a sufficiently smart model, the package manager is your AI agent, and the app store is a git repo.
|
||||
|
||||
`gbrain install voice-agent`
|
||||
`gbrain skillpack scaffold voice-agent`
|
||||
|
||||
That's it.
|
||||
|
||||
@@ -69,7 +69,7 @@ update_brain_page(slug, new_info, source):
|
||||
page = gbrain get {slug}
|
||||
|
||||
// TIMELINE: always APPEND (never edit existing entries)
|
||||
gbrain add_timeline_entry {slug} {
|
||||
gbrain timeline-add {slug} {
|
||||
date: today,
|
||||
summary: new_info.summary,
|
||||
detail: new_info.detail,
|
||||
|
||||
@@ -46,10 +46,10 @@ on user_shares_media(url_or_file):
|
||||
|
||||
# Step 4: Extract and cross-reference entities
|
||||
for person in transcript.mentioned_people:
|
||||
gbrain add_link <slug> <person_slug>
|
||||
gbrain add_link <person_slug> <slug>
|
||||
gbrain add_timeline_entry <person_slug> \
|
||||
--entry "Discussed in {video_title}: {what_was_said}" \
|
||||
gbrain link <slug> <person_slug>
|
||||
gbrain link <person_slug> <slug>
|
||||
gbrain timeline-add <person_slug> {date} \
|
||||
"Discussed in {video_title}: {what_was_said}" \
|
||||
--source "YouTube: {url}"
|
||||
|
||||
# PATTERN 2: Social Media Bundles
|
||||
@@ -80,8 +80,8 @@ on user_shares_media(url_or_file):
|
||||
|
||||
# Extract entities and cross-reference
|
||||
for entity in bundle.mentioned_entities:
|
||||
gbrain add_link <slug> <entity_slug>
|
||||
gbrain add_link <entity_slug> <slug>
|
||||
gbrain link <slug> <entity_slug>
|
||||
gbrain link <entity_slug> <slug>
|
||||
|
||||
# PATTERN 3: PDFs and Documents
|
||||
elif media.type == "pdf" or media.type == "document":
|
||||
@@ -109,8 +109,8 @@ on user_shares_media(url_or_file):
|
||||
"""
|
||||
|
||||
for entity in document.mentioned_entities:
|
||||
gbrain add_link <slug> <entity_slug>
|
||||
gbrain add_link <entity_slug> <slug>
|
||||
gbrain link <slug> <entity_slug>
|
||||
gbrain link <entity_slug> <slug>
|
||||
|
||||
# Always sync after ingestion
|
||||
gbrain sync
|
||||
@@ -127,7 +127,7 @@ on user_shares_media(url_or_file):
|
||||
## How to Verify
|
||||
|
||||
1. Ingest a YouTube video. Run `gbrain get media/youtube/{slug}`. Confirm the page has: the agent's analysis (not just a summary), key quotes with speaker attribution, and the full diarized transcript.
|
||||
2. Run `gbrain get_links media/youtube/{slug}`. Confirm back-links exist to brain pages for every person and company mentioned in the video.
|
||||
2. Run `gbrain call get_links '{"slug": "media/youtube/{slug}"}'`. Confirm back-links exist to brain pages for every person and company mentioned in the video.
|
||||
3. Pick a person mentioned in the video. Run `gbrain get <person_slug>`. Confirm their timeline has a new entry referencing the video with specific context.
|
||||
4. Ingest a tweet. Confirm the brain page includes the thread context, linked article summaries, and entity cross-references -- not just the tweet text.
|
||||
5. Run `gbrain search "{topic_from_video}"`. Confirm the media page appears in search results (verifies the content is indexed and searchable).
|
||||
|
||||
@@ -49,23 +49,23 @@ on enrich(entity, trigger):
|
||||
data["contacts"] = google_contacts(entity.email) # Contact data
|
||||
|
||||
# Step 5: Store raw data (auditable, re-processable)
|
||||
gbrain put_raw_data <entity_slug> \
|
||||
--data '{"sources": {"crustdata": {"fetched_at": "...", "data": {...}}, ...}}'
|
||||
gbrain call put_raw_data \
|
||||
'{"slug": "<entity_slug>", "data": {"sources": {"crustdata": {"fetched_at": "...", "data": {...}}, ...}}}'
|
||||
# Overwrite on re-enrichment, don't append
|
||||
|
||||
# Step 6: Write to brain page
|
||||
if path == "CREATE":
|
||||
gbrain put <entity_slug> --content "<compiled_truth_from_all_sources>"
|
||||
gbrain add_timeline_entry <entity_slug> --entry "Page created via enrichment"
|
||||
gbrain timeline-add <entity_slug> {date} "Page created via enrichment"
|
||||
elif path == "UPDATE":
|
||||
# Append timeline, update compiled truth ONLY if materially new
|
||||
gbrain add_timeline_entry <entity_slug> --entry "Enriched: {new_signal}"
|
||||
gbrain timeline-add <entity_slug> {date} "Enriched: {new_signal}"
|
||||
# Flag contradictions -- don't silently resolve them
|
||||
|
||||
# Step 7: Cross-reference the graph
|
||||
gbrain add_link <person_slug> <company_slug> # person -> company
|
||||
gbrain add_link <company_slug> <person_slug> # company -> person
|
||||
gbrain add_link <person_slug> <deal_slug> # person -> deal
|
||||
gbrain link <person_slug> <company_slug> # person -> company
|
||||
gbrain link <company_slug> <person_slug> # company -> person
|
||||
gbrain link <person_slug> <deal_slug> # person -> deal
|
||||
# Every entity page links to every other entity page that references it
|
||||
|
||||
# People page sections (not a LinkedIn profile -- a living portrait):
|
||||
@@ -94,8 +94,8 @@ on enrich(entity, trigger):
|
||||
## How to Verify
|
||||
|
||||
1. Enrich a Tier 1 person. Run `gbrain get <slug>` and confirm the page has Executive Summary, State, What They Believe, Contact, and Timeline sections populated from multiple sources.
|
||||
2. Run `gbrain get_raw_data <slug>`. Confirm raw API responses are stored with `sources.{provider}.fetched_at` timestamps.
|
||||
3. Run `gbrain get_links <slug>`. Confirm cross-reference links exist to the person's company page, deal pages, and related entities.
|
||||
2. Run `gbrain call get_raw_data '{"slug": "<slug>"}'`. Confirm raw API responses are stored with `sources.{provider}.fetched_at` timestamps.
|
||||
3. Run `gbrain call get_links '{"slug": "<slug>"}'`. Confirm cross-reference links exist to the person's company page, deal pages, and related entities.
|
||||
4. Check a page that was enriched AND has a user-written Assessment. Confirm the Assessment section was preserved, not overwritten by API data.
|
||||
5. Try to re-enrich the same person. Confirm the system checks the `fetched_at` timestamp and skips if less than a week old.
|
||||
|
||||
|
||||
@@ -53,7 +53,7 @@ on upcoming_meeting(meeting):
|
||||
"last_interaction": page.timeline[0], # most recent
|
||||
"open_threads": page.open_threads,
|
||||
"relationship_temperature": page.relationship,
|
||||
"relevant_deals": gbrain get_links <attendee_slug>,
|
||||
"relevant_deals": gbrain call get_links '{"slug": "<attendee_slug>"}',
|
||||
}
|
||||
else:
|
||||
briefing[attendee] = "No brain page -- consider enriching"
|
||||
@@ -67,14 +67,14 @@ on inbox_cleared():
|
||||
for email in processed_emails:
|
||||
if email.contained_new_information:
|
||||
# Update the sender's brain page with new signal
|
||||
gbrain add_timeline_entry <sender_slug> \
|
||||
--entry "Email re: {subject}. Key info: {extracted_signal}" \
|
||||
gbrain timeline-add <sender_slug> {date} \
|
||||
"Email re: {subject}. Key info: {extracted_signal}" \
|
||||
--source "email from {sender} re {subject}, {date}"
|
||||
|
||||
# Update any mentioned entity pages too
|
||||
for entity in email.mentioned_entities:
|
||||
gbrain add_timeline_entry <entity_slug> \
|
||||
--entry "{what_was_said_about_them}" \
|
||||
gbrain timeline-add <entity_slug> {date} \
|
||||
"{what_was_said_about_them}" \
|
||||
--source "email from {sender}, {date}"
|
||||
|
||||
# WORKFLOW 4: Scheduling Nudges
|
||||
|
||||
@@ -32,15 +32,15 @@ on new_meeting_transcript(meeting):
|
||||
|
||||
# Step 3: Propagate to ALL entity pages (MANDATORY -- most agents skip this)
|
||||
for person in meeting.attendees + meeting.mentioned_people:
|
||||
gbrain add_timeline_entry <person_slug> \
|
||||
--entry "Met in '{meeting.title}' on {date}. Key points: ..." \
|
||||
gbrain timeline-add <person_slug> {date} \
|
||||
"Met in '{meeting.title}' on {date}. Key points: ..." \
|
||||
--source "Meeting notes '{meeting.title}', {date}"
|
||||
# Update their State section if new information surfaced
|
||||
# Update company pages for each person's company if relevant
|
||||
|
||||
for company in meeting.mentioned_companies:
|
||||
gbrain add_timeline_entry <company_slug> \
|
||||
--entry "Discussed in '{meeting.title}': {what_was_said}" \
|
||||
gbrain timeline-add <company_slug> {date} \
|
||||
"Discussed in '{meeting.title}': {what_was_said}" \
|
||||
--source "Meeting notes '{meeting.title}', {date}"
|
||||
|
||||
# Step 4: Extract action items
|
||||
@@ -49,8 +49,8 @@ on new_meeting_transcript(meeting):
|
||||
|
||||
# Step 5: Back-link everything (bidirectional graph)
|
||||
for entity in all_entities_mentioned:
|
||||
gbrain add_link <slug> <entity_slug> # meeting -> entity
|
||||
gbrain add_link <entity_slug> <slug> # entity -> meeting
|
||||
gbrain link <slug> <entity_slug> # meeting -> entity
|
||||
gbrain link <entity_slug> <slug> # entity -> meeting
|
||||
|
||||
# Step 6: Sync so new pages are immediately searchable
|
||||
gbrain sync
|
||||
@@ -73,7 +73,7 @@ on new_meeting_transcript(meeting):
|
||||
1. After ingesting a meeting, run `gbrain get meetings/{date}-{slug}`. Confirm the page has the agent's analysis above the bar and the full diarized transcript below it.
|
||||
2. For each attendee, run `gbrain get <attendee_slug>`. Check that their timeline has a new entry referencing the meeting with specific insights (not just "attended meeting").
|
||||
3. Pick a company mentioned in the meeting. Run `gbrain get <company_slug>`. Confirm a timeline entry exists referencing what was discussed about the company.
|
||||
4. Run `gbrain get_links meetings/{date}-{slug}`. Verify back-links exist to all attendee and entity pages.
|
||||
4. Run `gbrain call get_links '{"slug": "meetings/{date}-{slug}"}'`. Verify back-links exist to all attendee and entity pages.
|
||||
5. Run `gbrain search "{meeting_topic}"`. Confirm the meeting page appears in search results (verifies sync ran).
|
||||
|
||||
---
|
||||
|
||||
@@ -91,7 +91,7 @@ first):
|
||||
6. The seeded `default` source.
|
||||
|
||||
So inside `~/.gstack/plans/` on a brain that pinned `gstack` to
|
||||
`~/.gstack` via `.gbrain-source`, `gbrain put-page` implicitly writes to
|
||||
`~/.gstack` via `.gbrain-source`, `gbrain put` implicitly writes to
|
||||
the `gstack` source. Outside any registered directory with no env/dotfile
|
||||
set, it writes to the default.
|
||||
|
||||
@@ -188,10 +188,10 @@ citations keep working.
|
||||
|
||||
```bash
|
||||
# Pass --source explicitly
|
||||
gbrain put-page topics/ai ... --source wiki
|
||||
gbrain put topics/ai ... --source wiki
|
||||
|
||||
# Or rely on the dotfile / env / CWD match
|
||||
cd ~/.gstack && gbrain put-page plans/multi-repo ...
|
||||
cd ~/.gstack && gbrain put plans/multi-repo ...
|
||||
# → source auto-resolves to gstack
|
||||
```
|
||||
|
||||
|
||||
@@ -20,8 +20,8 @@ on every_inbound_message(message):
|
||||
for entity in entities:
|
||||
existing = gbrain search "{entity.name}"
|
||||
if existing:
|
||||
gbrain add_timeline_entry <entity_slug> \
|
||||
--entry "{what_was_said}" \
|
||||
gbrain timeline-add <entity_slug> {date} \
|
||||
"{what_was_said}" \
|
||||
--source "User, direct message, {timestamp}"
|
||||
# else: flag for enrichment if important enough
|
||||
|
||||
@@ -64,13 +64,13 @@ on nightly_schedule("02:00"):
|
||||
# The brain COMPOUNDS overnight.
|
||||
|
||||
# 5a: Entity sweep -- find unlinked mentions
|
||||
pages = gbrain list_pages
|
||||
pages = gbrain list
|
||||
for page in pages:
|
||||
mentions = extract_entity_mentions(page.content)
|
||||
existing_links = gbrain get_links <page.slug>
|
||||
existing_links = gbrain call get_links '{"slug": "<page.slug>"}'
|
||||
for mention in mentions:
|
||||
if mention not in existing_links:
|
||||
gbrain add_link <page.slug> <mention_slug> # fix broken graph
|
||||
gbrain link <page.slug> <mention_slug> # fix broken graph
|
||||
|
||||
# 5b: Citation audit -- find facts without sources
|
||||
for page in pages:
|
||||
@@ -80,7 +80,7 @@ on nightly_schedule("02:00"):
|
||||
|
||||
# 5c: Memory consolidation -- update compiled truth from timeline
|
||||
for page in stale_pages(older_than="7d"):
|
||||
timeline = gbrain get_timeline <page.slug>
|
||||
timeline = gbrain timeline <page.slug>
|
||||
if timeline.has_new_entries_since_last_consolidation:
|
||||
# Re-synthesize compiled truth from accumulated timeline
|
||||
updated_truth = consolidate(page.compiled_truth, timeline.new_entries)
|
||||
@@ -110,11 +110,11 @@ on nightly_schedule("02:00"):
|
||||
|
||||
## How to Verify
|
||||
|
||||
1. Send a message mentioning a person with a brain page. Confirm the agent detects the entity and adds a timeline entry to their page (`gbrain get_timeline <slug>`).
|
||||
1. Send a message mentioning a person with a brain page. Confirm the agent detects the entity and adds a timeline entry to their page (`gbrain timeline <slug>`).
|
||||
2. Ask the agent about someone in the brain. Confirm it runs `gbrain search` or `gbrain get` BEFORE reaching for external APIs (check the tool call order).
|
||||
3. Write a new page with `gbrain put`, then immediately run `gbrain search` for it. Confirm it appears in results (verifies sync ran).
|
||||
4. Run `gbrain doctor`. Confirm it returns a health report with database status, page count, and any flagged issues.
|
||||
5. After a dream cycle runs, check a page that had unlinked entity mentions. Confirm new links were added (`gbrain get_links <slug>`).
|
||||
5. After a dream cycle runs, check a page that had unlinked entity mentions. Confirm new links were added (`gbrain call get_links '{"slug": "<slug>"}'`).
|
||||
|
||||
---
|
||||
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*
|
||||
|
||||
@@ -47,8 +47,8 @@ on user_message(message):
|
||||
|
||||
# Step 3: Cross-link to everything that shaped the thinking
|
||||
for entity in idea.influences:
|
||||
gbrain add_link originals/{slug} <entity_slug>
|
||||
gbrain add_link <entity_slug> originals/{slug}
|
||||
gbrain link originals/{slug} <entity_slug>
|
||||
gbrain link <entity_slug> originals/{slug}
|
||||
|
||||
# Step 4: Sync
|
||||
gbrain sync
|
||||
@@ -79,7 +79,7 @@ on user_message(message):
|
||||
|
||||
1. Generate an original idea in conversation (e.g., "I call this the 'ambition debt' problem -- every year you delay going big, the compound interest works against you"). Confirm a new page appears at `brain/originals/ambition-debt` with `gbrain get originals/ambition-debt`.
|
||||
2. Check that the page uses the user's exact phrasing for the title and slug -- not a sanitized version.
|
||||
3. Run `gbrain get_links originals/ambition-debt`. Confirm cross-links exist to related people, meetings, or other originals.
|
||||
3. Run `gbrain call get_links '{"slug": "originals/ambition-debt"}'`. Confirm cross-links exist to related people, meetings, or other originals.
|
||||
4. Express a take on someone else's idea (e.g., "I think Thiel's contrarian question is wrong because..."). Confirm it goes to `originals/` (synthesis is original), not `concepts/`.
|
||||
5. Run `gbrain search "ambition debt"`. Confirm the originals page appears in search results and is discoverable.
|
||||
|
||||
|
||||
@@ -87,7 +87,7 @@ expect it.
|
||||
| `version` | string | yes | Your plugin's semver. Informational. |
|
||||
| `plugin_version` | string | yes | Contract lock. Must equal `"gbrain-plugin-v1"` for v0.15. |
|
||||
| `subagents` | string | no | Subdir name (default `subagents`). Escape-attempts are rejected. |
|
||||
| `description` | string | no | Shown in future `gbrain plugin list`. |
|
||||
| `description` | string | no | Shown in a future plugin-listing command. |
|
||||
|
||||
## Subagent definition files
|
||||
|
||||
|
||||
+1
-1
@@ -250,7 +250,7 @@ All 30 GBrain operations are available remotely, including `sync_brain` and
|
||||
directory where `gbrain serve` was launched. Symlinks, `..` traversal, and absolute
|
||||
paths outside cwd are rejected. Page slugs and filenames are allowlist-validated
|
||||
(alphanumeric + hyphens; no control chars, RTL overrides, or backslashes). Local
|
||||
CLI callers (`gbrain file upload ...`) keep unrestricted filesystem access since
|
||||
CLI callers (`gbrain files upload ...`) keep unrestricted filesystem access since
|
||||
the user owns the machine.
|
||||
|
||||
## Deployment Options
|
||||
|
||||
@@ -13,7 +13,7 @@ Step-by-step walkthroughs that take you from zero to a working outcome. Concrete
|
||||
|
||||
These are the next tutorials on the roadmap. Open an issue if one of them is the one you need most; that's how we'll prioritize.
|
||||
|
||||
- **Set up GBrain for VC dealflow** — the operator's recipe. People pages for founders, companies with typed Facts fence carrying ARR / team-size / runway across dates, meetings auto-ingested, deal pages linking everything. Shows `gbrain whoknows`, `gbrain find_trajectory`, and `gbrain founder scorecard` on real workflows.
|
||||
- **Set up GBrain for VC dealflow** — the operator's recipe. People pages for founders, companies with typed Facts fence carrying ARR / team-size / runway across dates, meetings auto-ingested, deal pages linking everything. Shows `gbrain whoknows`, `gbrain find-trajectory`, and `gbrain founder scorecard` on real workflows.
|
||||
|
||||
- **Migrate your existing vault into GBrain** — for Notion / Obsidian / Roam users with a vault that doesn't match GBrain's default layout. Walks through `gbrain schema detect` → `suggest` → `review-candidates` so the brain learns your shape instead of forcing you to learn its.
|
||||
|
||||
|
||||
@@ -554,7 +554,7 @@ What to do next:
|
||||
|
||||
- **Wire ingestion** from external systems (Granola, Linear, Slack) using the [ingestion source contract](../skillpack-anatomy.md). Most companies want their meetings auto-ingested so the brain stays current without anyone typing notes.
|
||||
- **Set up team-specific dashboards** through the admin UI. Each team lead can have their own view of brain health and activity.
|
||||
- **Explore the rest of the brain layer.** `gbrain whoknows` (find the expert on a topic), `gbrain find_trajectory` (how a metric changed over time), `gbrain founder scorecard` (especially useful for VC and ops teams), the contradiction-detection cycle that surfaces conflicts between different people's notes.
|
||||
- **Explore the rest of the brain layer.** `gbrain whoknows` (find the expert on a topic), `gbrain find-trajectory` (how a metric changed over time), `gbrain founder scorecard` (especially useful for VC and ops teams), the contradiction-detection cycle that surfaces conflicts between different people's notes.
|
||||
|
||||
If you're building in this space (which YC has flagged as the [company-brain category in its Request for Startups](https://www.ycombinator.com/rfs#company-brain)), you might as well build on this. Everything described above is open source, MIT licensed, and what I run in production behind my own AI agents.
|
||||
|
||||
|
||||
@@ -115,21 +115,21 @@ You can use the same keys across multiple agents.
|
||||
|
||||
## Step 6: Install GBrain
|
||||
|
||||
Once OpenClaw is running:
|
||||
Once OpenClaw is running, installation is two commands — one in the brain repo, one in the agent workspace:
|
||||
|
||||
```bash
|
||||
gbrain install
|
||||
# In the BRAIN repo (the git repo that holds your markdown pages):
|
||||
gbrain init --supabase
|
||||
|
||||
# In the AGENT WORKSPACE repo (where OpenClaw runs):
|
||||
gbrain skillpack scaffold --all
|
||||
```
|
||||
|
||||
This installs:
|
||||
`gbrain init --supabase` walks a short wizard that asks for your Supabase connection string and creates the schema. You'll get that connection string in Step 7 — read 7a and 7b first so you paste the right one (the transaction pooler, not the direct connection). If you'd rather try things locally before paying for a database, `gbrain init --pglite` gives you a zero-config embedded engine instead; you can migrate to Supabase later with `gbrain migrate --to supabase`.
|
||||
|
||||
- About 60 skills
|
||||
- About 9 skill packs
|
||||
- Default brain structure
|
||||
- MCP server configuration
|
||||
- Supabase connection (for embeddings and search)
|
||||
`gbrain skillpack scaffold --all` copies the ~43 bundled skills into your agent workspace as first-class files you can edit freely. (The old managed-install model was retired in v0.36.0.0; see `docs/INSTALL.md` if you're upgrading from an older release.)
|
||||
|
||||
GBrain populates the brain repo with its default directory structure, skill files, and configuration. From this point, the agent has working memory and access to every skill.
|
||||
From this point, the agent has working memory and access to every skill.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -32,7 +32,7 @@ gbrain schema sync --apply
|
||||
The sync backfills `page.type = 'meeting'` on all 4000 pages in 1000-row batches. Now:
|
||||
|
||||
- `gbrain whoknows "Q3 roadmap discussion"` routes through the meeting type, ranking by `expert_routing` signal (attendees, recency, salience) instead of raw text.
|
||||
- `gbrain extract-facts` runs on every meeting page automatically (because `extractable: true`), pulling typed facts like `attended_by=alice-example`, `date=2026-05-23`.
|
||||
- The `extract_facts` cycle runs on every meeting page automatically (because `extractable: true`), pulling typed facts like `attended_by=alice-example`, `date=2026-05-23`.
|
||||
- The downstream `think` skill can now answer "what did we decide about pricing in the last three roadmap meetings" by querying the meeting graph instead of grep'ing 4000 files.
|
||||
|
||||
One command. 4000 pages went from invisible to queryable. The content didn't change. The structure did.
|
||||
@@ -62,7 +62,7 @@ gbrain schema add-link-type led-by --page-type deal --target-type inves
|
||||
gbrain schema sync --apply
|
||||
```
|
||||
|
||||
Now `gbrain whoknows "Series A SaaS"` routes through `investor` and `portco` types specifically, not the noisy general type set. `gbrain graph-query alice-example --type intro-from --depth 2` walks two hops of intros to surface "Alice introduced you to Bob who introduced you to Charlie." `gbrain extract-facts` starts producing typed claims from the fence in your deal pages: `(deals/acme-seed, raise=2000000, valuation=15000000, lead=widget-vc, closed_at=2026-05-23)`.
|
||||
Now `gbrain whoknows "Series A SaaS"` routes through `investor` and `portco` types specifically, not the noisy general type set. `gbrain graph-query alice-example --type intro-from --depth 2` walks two hops of intros to surface "Alice introduced you to Bob who introduced you to Charlie." The `extract_facts` cycle starts producing typed claims from the fence in your deal pages: `(deals/acme-seed, raise=2000000, valuation=15000000, lead=widget-vc, closed_at=2026-05-23)`.
|
||||
|
||||
The CRM you've been promising yourself you'll set up next quarter? You just shipped it in 4 commands. It's downstream of your notes, not parallel to them.
|
||||
|
||||
@@ -143,7 +143,7 @@ Re-run the same `whoknows` query. Top-3 should shift, because the new type is no
|
||||
|
||||
Three things gbrain does that generic note systems can't:
|
||||
|
||||
**1. The brain knows the difference between a person and an idea.** Page-type matters at query time. `gbrain whoknows` only considers `expert_routing: true` types. `gbrain extract-facts` only runs on `extractable: true` types. `gbrain graph-query` walks declared link verbs. None of that works on a flat tag system because tags don't have semantics — they're labels. Types are first-class citizens with rules attached.
|
||||
**1. The brain knows the difference between a person and an idea.** Page-type matters at query time. `gbrain whoknows` only considers `expert_routing: true` types. The `extract_facts` cycle only runs on `extractable: true` types. `gbrain graph-query` walks declared link verbs. None of that works on a flat tag system because tags don't have semantics — they're labels. Types are first-class citizens with rules attached.
|
||||
|
||||
**2. Untyped content is invisible content.** If your meetings are typed as `note`, expert routing skips them, facts extraction ignores them, link inference doesn't fire. They exist on disk and they're indexed for text search, but the structural surfaces (whoknows, find_experts, recall, think) treat them as second-class. Adding a type isn't cosmetic; it's structural promotion.
|
||||
|
||||
|
||||
+4
-4
@@ -2316,7 +2316,7 @@ gbrain schema sync --apply
|
||||
The sync backfills `page.type = 'meeting'` on all 4000 pages in 1000-row batches. Now:
|
||||
|
||||
- `gbrain whoknows "Q3 roadmap discussion"` routes through the meeting type, ranking by `expert_routing` signal (attendees, recency, salience) instead of raw text.
|
||||
- `gbrain extract-facts` runs on every meeting page automatically (because `extractable: true`), pulling typed facts like `attended_by=alice-example`, `date=2026-05-23`.
|
||||
- The `extract_facts` cycle runs on every meeting page automatically (because `extractable: true`), pulling typed facts like `attended_by=alice-example`, `date=2026-05-23`.
|
||||
- The downstream `think` skill can now answer "what did we decide about pricing in the last three roadmap meetings" by querying the meeting graph instead of grep'ing 4000 files.
|
||||
|
||||
One command. 4000 pages went from invisible to queryable. The content didn't change. The structure did.
|
||||
@@ -2346,7 +2346,7 @@ gbrain schema add-link-type led-by --page-type deal --target-type inves
|
||||
gbrain schema sync --apply
|
||||
```
|
||||
|
||||
Now `gbrain whoknows "Series A SaaS"` routes through `investor` and `portco` types specifically, not the noisy general type set. `gbrain graph-query alice-example --type intro-from --depth 2` walks two hops of intros to surface "Alice introduced you to Bob who introduced you to Charlie." `gbrain extract-facts` starts producing typed claims from the fence in your deal pages: `(deals/acme-seed, raise=2000000, valuation=15000000, lead=widget-vc, closed_at=2026-05-23)`.
|
||||
Now `gbrain whoknows "Series A SaaS"` routes through `investor` and `portco` types specifically, not the noisy general type set. `gbrain graph-query alice-example --type intro-from --depth 2` walks two hops of intros to surface "Alice introduced you to Bob who introduced you to Charlie." The `extract_facts` cycle starts producing typed claims from the fence in your deal pages: `(deals/acme-seed, raise=2000000, valuation=15000000, lead=widget-vc, closed_at=2026-05-23)`.
|
||||
|
||||
The CRM you've been promising yourself you'll set up next quarter? You just shipped it in 4 commands. It's downstream of your notes, not parallel to them.
|
||||
|
||||
@@ -2427,7 +2427,7 @@ Re-run the same `whoknows` query. Top-3 should shift, because the new type is no
|
||||
|
||||
Three things gbrain does that generic note systems can't:
|
||||
|
||||
**1. The brain knows the difference between a person and an idea.** Page-type matters at query time. `gbrain whoknows` only considers `expert_routing: true` types. `gbrain extract-facts` only runs on `extractable: true` types. `gbrain graph-query` walks declared link verbs. None of that works on a flat tag system because tags don't have semantics — they're labels. Types are first-class citizens with rules attached.
|
||||
**1. The brain knows the difference between a person and an idea.** Page-type matters at query time. `gbrain whoknows` only considers `expert_routing: true` types. The `extract_facts` cycle only runs on `extractable: true` types. `gbrain graph-query` walks declared link verbs. None of that works on a flat tag system because tags don't have semantics — they're labels. Types are first-class citizens with rules attached.
|
||||
|
||||
**2. Untyped content is invisible content.** If your meetings are typed as `note`, expert routing skips them, facts extraction ignores them, link inference doesn't fire. They exist on disk and they're indexed for text search, but the structural surfaces (whoknows, find_experts, recall, think) treat them as second-class. Adding a type isn't cosmetic; it's structural promotion.
|
||||
|
||||
@@ -3897,7 +3897,7 @@ All 30 GBrain operations are available remotely, including `sync_brain` and
|
||||
directory where `gbrain serve` was launched. Symlinks, `..` traversal, and absolute
|
||||
paths outside cwd are rejected. Page slugs and filenames are allowlist-validated
|
||||
(alphanumeric + hyphens; no control chars, RTL overrides, or backslashes). Local
|
||||
CLI callers (`gbrain file upload ...`) keep unrestricted filesystem access since
|
||||
CLI callers (`gbrain files upload ...`) keep unrestricted filesystem access since
|
||||
the user owns the machine.
|
||||
|
||||
## Deployment Options
|
||||
|
||||
+34
-33
@@ -42,20 +42,20 @@
|
||||
"eval:autocut": "bun test test/search/autocut-eval.test.ts",
|
||||
"test:full": "bun run verify && bash scripts/run-unit-parallel.sh && bun run test:slow && ([ -n \"$DATABASE_URL\" ] && bash scripts/run-e2e.sh || echo '[test:full] skipped E2E (no DATABASE_URL); run docker-compose -f docker-compose.ci.yml up + bun run test:e2e to include' 1>&2)",
|
||||
"verify": "bash scripts/run-verify-parallel.sh",
|
||||
"check:source-config-leak": "scripts/check-source-config-leak.sh",
|
||||
"check:no-pii-agent-voice": "scripts/check-no-pii-in-agent-voice.sh",
|
||||
"check:synthetic-corpus-privacy": "scripts/check-synthetic-corpus-privacy.sh",
|
||||
"check:system-of-record": "scripts/check-system-of-record.sh",
|
||||
"check:admin-scope-drift": "scripts/check-admin-scope-drift.sh",
|
||||
"check:cli-exec": "scripts/check-cli-executable.sh",
|
||||
"check:all": "scripts/check-privacy.sh && scripts/check-proposal-pii.sh && scripts/check-test-real-names.sh && scripts/check-jsonb-pattern.sh && scripts/check-source-id-projection.sh && scripts/check-source-config-leak.sh && scripts/check-progress-to-stdout.sh && scripts/check-no-legacy-getconnection.sh && scripts/check-test-isolation.sh && scripts/check-trailing-newline.sh && scripts/check-wasm-embedded.sh && scripts/check-exports-count.sh && scripts/check-admin-build.sh && scripts/check-admin-scope-drift.sh && scripts/check-cli-executable.sh && scripts/check-skill-brain-first.sh && scripts/check-operations-filter-bypass.sh && scripts/check-gateway-routed-no-direct-anthropic.sh && scripts/check-worker-pool-atomicity.sh && scripts/check-key-files-current-state.sh && scripts/check-no-double-retry.sh && scripts/check-batch-audit-site.sh",
|
||||
"check:gateway-routed": "scripts/check-gateway-routed-no-direct-anthropic.sh",
|
||||
"check:worker-pool-atomicity": "scripts/check-worker-pool-atomicity.sh",
|
||||
"check:doc-history": "scripts/check-key-files-current-state.sh",
|
||||
"check:source-config-leak": "bash scripts/check-source-config-leak.sh",
|
||||
"check:no-pii-agent-voice": "bash scripts/check-no-pii-in-agent-voice.sh",
|
||||
"check:synthetic-corpus-privacy": "bash scripts/check-synthetic-corpus-privacy.sh",
|
||||
"check:system-of-record": "bash scripts/check-system-of-record.sh",
|
||||
"check:admin-scope-drift": "bash scripts/check-admin-scope-drift.sh",
|
||||
"check:cli-exec": "bash scripts/check-cli-executable.sh",
|
||||
"check:all": "bash scripts/check-privacy.sh && bash scripts/check-proposal-pii.sh && bash scripts/check-test-real-names.sh && bash scripts/check-jsonb-pattern.sh && bash scripts/check-source-id-projection.sh && bash scripts/check-source-config-leak.sh && bash scripts/check-progress-to-stdout.sh && bash scripts/check-no-tracked-symlinks.sh && bash scripts/check-no-legacy-getconnection.sh && bash scripts/check-test-isolation.sh && bash scripts/check-trailing-newline.sh && bash scripts/check-wasm-embedded.sh && bash scripts/check-exports-count.sh && bash scripts/check-admin-build.sh && bash scripts/check-admin-scope-drift.sh && bash scripts/check-cli-executable.sh && bash scripts/check-skill-brain-first.sh && bash scripts/check-operations-filter-bypass.sh && bash scripts/check-gateway-routed-no-direct-anthropic.sh && bash scripts/check-worker-pool-atomicity.sh && bash scripts/check-key-files-current-state.sh && bash scripts/check-no-double-retry.sh && bash scripts/check-batch-audit-site.sh",
|
||||
"check:gateway-routed": "bash scripts/check-gateway-routed-no-direct-anthropic.sh",
|
||||
"check:worker-pool-atomicity": "bash scripts/check-worker-pool-atomicity.sh",
|
||||
"check:doc-history": "bash scripts/check-key-files-current-state.sh",
|
||||
"check:resolver": "bun src/cli.ts check-resolvable --strict --skills-dir skills/",
|
||||
"check:skill-brain-first": "scripts/check-skill-brain-first.sh",
|
||||
"check:wasm": "scripts/check-wasm-embedded.sh",
|
||||
"check:newlines": "scripts/check-trailing-newline.sh",
|
||||
"check:skill-brain-first": "bash scripts/check-skill-brain-first.sh",
|
||||
"check:wasm": "bash scripts/check-wasm-embedded.sh",
|
||||
"check:newlines": "bash scripts/check-trailing-newline.sh",
|
||||
"test:e2e": "bash scripts/run-e2e.sh",
|
||||
"test:slow": "bash scripts/run-slow-tests.sh",
|
||||
"test:heavy": "bash scripts/run-heavy.sh",
|
||||
@@ -65,26 +65,27 @@
|
||||
"ci:local:diff": "bash scripts/ci-local.sh --diff",
|
||||
"ci:select-e2e": "bun run scripts/select-e2e.ts",
|
||||
"typecheck": "tsc --noEmit",
|
||||
"check:jsonb": "scripts/check-jsonb-pattern.sh",
|
||||
"check:search-path": "scripts/check-search-path.sh",
|
||||
"check:no-double-retry": "scripts/check-no-double-retry.sh",
|
||||
"check:batch-audit-site": "scripts/check-batch-audit-site.sh",
|
||||
"check:worker-lock-renewal-shape": "scripts/check-worker-lock-renewal-shape.sh",
|
||||
"check:source-id-projection": "scripts/check-source-id-projection.sh",
|
||||
"check:privacy": "scripts/check-privacy.sh",
|
||||
"check:proposal-pii": "scripts/check-proposal-pii.sh",
|
||||
"check:eval-glossary": "scripts/check-eval-glossary-fresh.sh",
|
||||
"check:test-names": "scripts/check-test-real-names.sh",
|
||||
"check:progress": "scripts/check-progress-to-stdout.sh",
|
||||
"check:exports-count": "scripts/check-exports-count.sh",
|
||||
"check:admin-build": "scripts/check-admin-build.sh",
|
||||
"check:admin-embedded": "scripts/check-admin-embedded.sh",
|
||||
"check:test-isolation": "scripts/check-test-isolation.sh",
|
||||
"check:fuzz-purity": "scripts/check-fuzz-purity.sh",
|
||||
"check:operations-filter-bypass": "scripts/check-operations-filter-bypass.sh",
|
||||
"check:fixture-privacy": "scripts/check-fixture-privacy.sh",
|
||||
"check:jsonb": "bash scripts/check-jsonb-pattern.sh",
|
||||
"check:search-path": "bash scripts/check-search-path.sh",
|
||||
"check:no-double-retry": "bash scripts/check-no-double-retry.sh",
|
||||
"check:batch-audit-site": "bash scripts/check-batch-audit-site.sh",
|
||||
"check:worker-lock-renewal-shape": "bash scripts/check-worker-lock-renewal-shape.sh",
|
||||
"check:source-id-projection": "bash scripts/check-source-id-projection.sh",
|
||||
"check:privacy": "bash scripts/check-privacy.sh",
|
||||
"check:proposal-pii": "bash scripts/check-proposal-pii.sh",
|
||||
"check:eval-glossary": "bash scripts/check-eval-glossary-fresh.sh",
|
||||
"check:test-names": "bash scripts/check-test-real-names.sh",
|
||||
"check:progress": "bash scripts/check-progress-to-stdout.sh",
|
||||
"check:no-tracked-symlinks": "bash scripts/check-no-tracked-symlinks.sh",
|
||||
"check:exports-count": "bash scripts/check-exports-count.sh",
|
||||
"check:admin-build": "bash scripts/check-admin-build.sh",
|
||||
"check:admin-embedded": "bash scripts/check-admin-embedded.sh",
|
||||
"check:test-isolation": "bash scripts/check-test-isolation.sh",
|
||||
"check:fuzz-purity": "bash scripts/check-fuzz-purity.sh",
|
||||
"check:operations-filter-bypass": "bash scripts/check-operations-filter-bypass.sh",
|
||||
"check:fixture-privacy": "bash scripts/check-fixture-privacy.sh",
|
||||
"check:conversation-parser": "bun src/cli.ts eval conversation-parser test/fixtures/conversation-formats/all.jsonl --no-llm",
|
||||
"check:source-scope-onboard": "scripts/check-source-scope-onboard.sh",
|
||||
"check:source-scope-onboard": "bash scripts/check-source-scope-onboard.sh",
|
||||
"postinstall": "bun run scripts/postinstall.ts",
|
||||
"prepublish:clawhub": "bun run build:all",
|
||||
"publish:clawhub": "clawhub package publish . --family bundle-plugin"
|
||||
@@ -145,7 +146,7 @@
|
||||
"bun": ">=1.3.10"
|
||||
},
|
||||
"license": "MIT",
|
||||
"version": "0.42.66.1",
|
||||
"version": "0.42.67.0",
|
||||
"overrides": {
|
||||
"@hono/node-server": "^2.0.5",
|
||||
"fast-uri": "^3.1.4",
|
||||
|
||||
Executable
+67
@@ -0,0 +1,67 @@
|
||||
#!/usr/bin/env bash
|
||||
# CI guard: fail if any symlink is tracked in git.
|
||||
#
|
||||
# A symlink committed from a build sandbox points at a path that exists on
|
||||
# exactly one machine. Everywhere else the checkout produces a dangling
|
||||
# link, and anything that opens it fails. That is not hypothetical: commit
|
||||
# faf5cdba landed `node_modules -> /tmp/fleet/repo/node_modules`, which made
|
||||
# `bun install` abort with `ENOENT: could not open the "node_modules"
|
||||
# directory` on every fresh clone, and took `gbrain upgrade`'s bun-link path
|
||||
# down with it (the auto-upgrade runs `bun install`, so the printed manual
|
||||
# fallback failed the same way).
|
||||
#
|
||||
# .gitignore alone does not prevent this. A `node_modules/` pattern with a
|
||||
# trailing slash matches directories ONLY, so a symlink of the same name is
|
||||
# never ignored. Dropping the slash closes that hole, but `git add -f` still
|
||||
# walks straight past it. This guard is the backstop.
|
||||
#
|
||||
# The repo has no legitimate tracked symlinks, so the allowlist starts
|
||||
# empty. If you ever need one, add its exact repo-relative path to ALLOWLIST
|
||||
# below and explain why — a relative link that resolves inside the repo is
|
||||
# defensible; an absolute one almost never is.
|
||||
#
|
||||
# Usage: scripts/check-no-tracked-symlinks.sh
|
||||
# Exit: 0 when clean, 1 when a tracked symlink is found.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
ROOT="$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
|
||||
cd "$ROOT"
|
||||
|
||||
# Paths permitted to be tracked symlinks. Empty by design.
|
||||
ALLOWLIST=()
|
||||
|
||||
# Git records symlinks with mode 120000. Field 4 of `ls-files -s` is the path
|
||||
# (tab-separated from the stage number), so cut on the tab to keep paths with
|
||||
# spaces intact.
|
||||
found="$(git ls-files -s | awk '$1 == "120000"' | cut -f2- || true)"
|
||||
|
||||
if [ -n "$found" ]; then
|
||||
filtered="$found"
|
||||
for f in "${ALLOWLIST[@]:-}"; do
|
||||
[ -z "$f" ] && continue
|
||||
filtered="$(echo "$filtered" | grep -vxF "$f" || true)"
|
||||
done
|
||||
|
||||
if [ -n "$filtered" ]; then
|
||||
echo "ERROR: symlink(s) tracked in git:"
|
||||
echo
|
||||
while IFS= read -r path; do
|
||||
[ -z "$path" ] && continue
|
||||
target="$(git cat-file blob ":$path" 2>/dev/null || echo '<unreadable>')"
|
||||
echo " $path -> $target"
|
||||
done <<< "$filtered"
|
||||
echo
|
||||
echo "A committed symlink resolves on the machine that created it and"
|
||||
echo "nowhere else. Untrack it:"
|
||||
echo
|
||||
echo " git rm --cached <path>"
|
||||
echo
|
||||
echo "If the path is build output (node_modules, dist, bin), also confirm"
|
||||
echo "it is covered by .gitignore WITHOUT a trailing slash — a trailing"
|
||||
echo "slash matches directories only and lets the symlink through."
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
echo "check-no-tracked-symlinks: OK (no tracked symlinks)"
|
||||
@@ -42,6 +42,7 @@ CHECKS=(
|
||||
"check:source-id-projection"
|
||||
"check:source-config-leak"
|
||||
"check:progress"
|
||||
"check:no-tracked-symlinks"
|
||||
"check:test-isolation"
|
||||
"check:wasm"
|
||||
"check:admin-build"
|
||||
|
||||
@@ -248,7 +248,7 @@ before submission.
|
||||
After the brain page is written, render to PDF using `skills/brain-pdf`:
|
||||
|
||||
```bash
|
||||
gbrain put_page # already done by the CLI; nothing to add here
|
||||
gbrain put # already done by the CLI; nothing to add here
|
||||
# Then invoke brain-pdf:
|
||||
# (see skills/brain-pdf/SKILL.md for the make-pdf invocation)
|
||||
```
|
||||
|
||||
@@ -73,13 +73,13 @@ stock worker auto-loads on startup) registers handlers before `start()`.
|
||||
Users who set `minion_mode: off` in `~/.gbrain/preferences.json` keep
|
||||
using `agentTurn`. Respect that. No auto-rewrite.
|
||||
|
||||
## Forward note (v0.12.0)
|
||||
## Forward note
|
||||
|
||||
GBrain v0.12.0 ships `gbrain cron`: a scheduler loop inside
|
||||
`gbrain jobs work` that owns cron expressions natively — no more
|
||||
handing off to host schedulers. Until v0.12.0 lands, the host
|
||||
scheduler keeps firing on schedule; v0.11.1 only replaces the execution
|
||||
layer (what the cron trigger *does*), not the scheduling layer.
|
||||
A native scheduler loop inside `gbrain jobs work` (owning cron
|
||||
expressions directly, with no host-scheduler hand-off) has been on the
|
||||
roadmap since v0.11.1 but has not shipped. The host scheduler keeps
|
||||
firing on schedule; this convention only replaces the execution layer
|
||||
(what the cron trigger *does*), not the scheduling layer.
|
||||
|
||||
## Related
|
||||
|
||||
|
||||
@@ -54,8 +54,8 @@ Ask the user what they want to track. Either:
|
||||
- Define a custom recipe with: source queries, classification rules, extraction schema,
|
||||
tracker page path, tracker format
|
||||
|
||||
Recipes are YAML files at `~/.gbrain/recipes/{name}.yaml`. Use `gbrain research init`
|
||||
to scaffold a new one.
|
||||
Recipes are YAML files at `~/.gbrain/recipes/{name}.yaml`. Scaffold a new one by
|
||||
copying a built-in recipe file and editing its fields.
|
||||
|
||||
### Phase 2: Search Sources
|
||||
|
||||
|
||||
@@ -201,7 +201,7 @@ Use the brain page template. MUST include:
|
||||
|
||||
### 4b. Entity pages (people, companies)
|
||||
For each entity mentioned:
|
||||
- Check if a brain page exists (`gbrain search "<name>"` or `gbrain get_page people/<slug>`).
|
||||
- Check if a brain page exists (`gbrain search "<name>"` or `gbrain get people/<slug>`).
|
||||
- If exists: update State, append Timeline entry citing this research.
|
||||
- If not: create with enrichment.
|
||||
|
||||
|
||||
@@ -112,7 +112,7 @@ gbrain query "<topic keywords>"
|
||||
# -d '{"model": "sonar-pro", "messages": [{"role":"user","content":"..."}]}'
|
||||
|
||||
# 4. Write the structured research page via put_page:
|
||||
gbrain put_page research/<slug> # via the put_page operation
|
||||
gbrain put research/<slug> # via the put_page operation
|
||||
|
||||
# 5. Cross-link entities mentioned (people, companies) per Iron Law.
|
||||
```
|
||||
|
||||
@@ -11,7 +11,7 @@ tools:
|
||||
- gbrain schema active
|
||||
- gbrain schema use
|
||||
- gbrain schema stats
|
||||
- gbrain pages restore
|
||||
- gbrain restore
|
||||
- mcp:run_onboard
|
||||
triggers:
|
||||
- "unify my types"
|
||||
@@ -143,7 +143,7 @@ WHERE source_id = 'default' AND frontmatter->>'legacy_type' IS NOT NULL;
|
||||
Page-to-alias and page-to-link source pages soft-delete with 72h TTL. Restore within that window:
|
||||
|
||||
```bash
|
||||
gbrain pages restore <slug>
|
||||
gbrain restore <slug>
|
||||
```
|
||||
|
||||
Revert the active pack flip:
|
||||
@@ -197,7 +197,7 @@ Outputs:
|
||||
- Active pack flipped to `gbrain-base-v2` atomically at end of successful run.
|
||||
|
||||
Side effects:
|
||||
- Source pages soft-deleted with 72h restore TTL (`gbrain pages restore <slug>`).
|
||||
- Source pages soft-deleted with 72h restore TTL (`gbrain restore <slug>`).
|
||||
- One-time cache invalidation on KNOBS_HASH_VERSION bump (5→6); self-healing in `cache.ttl_seconds`.
|
||||
- Query-time `--type X` alias-expands via `expandTypeFilter` (D14 back-compat).
|
||||
|
||||
@@ -212,7 +212,7 @@ DON'T:
|
||||
- Submit `unify-types` directly via the MCP `submit_job` op without `--allow-protected`. PROTECTED handlers require trusted local callers; remote MCP rejection is the intentional trust boundary.
|
||||
- Edit `mapping_rules` in `gbrain-base-v2.yaml` to skip clusters you don't trust. Fork the pack instead (`gbrain schema fork`) so the source-of-truth migration stays consistent across brains.
|
||||
- Run `unify-types` from inside an autopilot tick. The check is `manual_only` per D17 — autopilot deliberately never auto-fires it because pack upgrades are one-time consenting taxonomy decisions.
|
||||
- Hard-delete soft-deleted source pages before the 72h restore window. Use `gbrain pages restore <slug>` first if rollback is needed.
|
||||
- Hard-delete soft-deleted source pages before the 72h restore window. Use `gbrain restore <slug>` first if rollback is needed.
|
||||
- Assume `frontmatter.legacy_type` survives every roundtrip. The marker is canonical for the immediate post-migration window; downstream re-imports may overwrite it.
|
||||
|
||||
## Output Format
|
||||
|
||||
@@ -43,8 +43,9 @@ The Analysis section can interpret; the transcript section is sacred.
|
||||
|
||||
The user sends an audio or voice message via any channel (Telegram, voice
|
||||
memo upload, openclaw audio attachment). The host agent typically provides
|
||||
the transcript text. If not, transcribe via `gbrain transcription` (Groq
|
||||
Whisper by default; OpenAI fallback for audio > 25MB segmented via ffmpeg).
|
||||
the transcript text. If not, transcribe it with your host's transcription
|
||||
tool (Groq Whisper is fast and cheap; OpenAI Whisper works too — segment
|
||||
audio > 25MB via ffmpeg first).
|
||||
|
||||
## The pipeline
|
||||
|
||||
@@ -52,8 +53,9 @@ Whisper by default; OpenAI fallback for audio > 25MB segmented via ffmpeg).
|
||||
1. STORE → Upload original audio to gbrain storage backend
|
||||
(S3 / Supabase Storage / local — pluggable per
|
||||
src/core/storage.ts).
|
||||
2. TRANSCRIBE → Use the agent-provided transcript verbatim, OR call
|
||||
gbrain transcription if no transcript was supplied.
|
||||
2. TRANSCRIBE → Use the agent-provided transcript verbatim, OR
|
||||
transcribe the audio yourself (see "When to invoke")
|
||||
if no transcript was supplied.
|
||||
3. ROUTE → Apply the decision tree (below) to find the right
|
||||
destination directory.
|
||||
4. WRITE → Create / update the destination brain page; preserve the
|
||||
|
||||
+125
-13
@@ -9,7 +9,7 @@ installSigchldHandler();
|
||||
import { installSignalHandlers as installCleanupSignalHandlers } from './core/process-cleanup.ts';
|
||||
installCleanupSignalHandlers();
|
||||
|
||||
import { readFileSync, existsSync, unlinkSync } from 'fs';
|
||||
import { readFileSync, existsSync, unlinkSync, fstatSync } from 'fs';
|
||||
import { spawn } from 'child_process';
|
||||
import {
|
||||
readUpdateCache,
|
||||
@@ -55,12 +55,17 @@ export function bigintToStringReplacer(_key: string, value: unknown): unknown {
|
||||
}
|
||||
|
||||
// CLI-only commands that bypass the operation layer
|
||||
export const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'maintain', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'reconcile-links', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'calibration', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'retrieval-upgrade', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'advisor', 'watch', 'reindex-search-vector']);
|
||||
export const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'maintain', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'reconcile-links', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'calibration', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'retrieval-upgrade', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'advisor', 'watch', 'reindex-search-vector', 'pages', 'bench', 'backfill']);
|
||||
// CLI-only commands whose handlers print their own --help text. These are
|
||||
// excluded from the generic short-circuit so detailed per-command and
|
||||
// per-subcommand usage stays reachable.
|
||||
const CLI_ONLY_SELF_HELP = new Set([
|
||||
'upgrade', 'post-upgrade', 'check-update',
|
||||
// #3502 sweep: pages + bench print their own usage (pages.ts printHelp,
|
||||
// bench-publish.ts printHelp). Both were documented but undispatchable —
|
||||
// `pages` had a live handleCliOnly case but was missing from CLI_ONLY
|
||||
// (the #2035 calibration bug class); `bench` was never wired at all.
|
||||
'pages', 'bench',
|
||||
'embed', 'config',
|
||||
'skillpack', 'skillpack-check',
|
||||
'integrations', 'friction',
|
||||
@@ -344,6 +349,11 @@ async function main() {
|
||||
// them out of the engine try/catch is safe and unlocks routing.
|
||||
const params = parseOpArgs(op, subArgs);
|
||||
|
||||
// #3513: stdin fill moved out of parseOpArgs so a non-TTY stdin with no
|
||||
// piped input can't block the parse forever — the bounded read leaves the
|
||||
// param unset on timeout and the required-param check below fails fast.
|
||||
await applyStdinParam(op, params);
|
||||
|
||||
// v0.27.1 (`gbrain query --image <path>`): swap the `image` param from
|
||||
// a filesystem path into base64 bytes + mime. The op accepts base64; the
|
||||
// CLI accepts a path. Helper is exported so tests can exercise the
|
||||
@@ -804,18 +814,99 @@ export function parseOpArgs(op: Operation, args: string[]): Record<string, unkno
|
||||
}
|
||||
}
|
||||
|
||||
// Read stdin for content params
|
||||
return params;
|
||||
}
|
||||
|
||||
/**
|
||||
* #3513: read stdin into an op's stdin-capable param without ever blocking
|
||||
* forever. The old inline `readFileSync(0)` in parseOpArgs assumed non-TTY
|
||||
* implies piped content; a non-TTY stdin with NO input (CI step, cron job,
|
||||
* agent harness holding an unwritten pipe open) blocked the read until kill.
|
||||
*
|
||||
* Strategy by fd kind (fstat):
|
||||
* - TTY: skip, as before (interactive input is not an op-param source).
|
||||
* - regular file / /dev/null / anything not a pipe or socket: readFileSync
|
||||
* returns without blocking (`gbrain put x < file`, `< /dev/null` → '').
|
||||
* - FIFO/socket: stream-read with a deadline on the FIRST byte only. A real
|
||||
* pipe (`echo foo | gbrain put x`, heredocs) delivers its first byte
|
||||
* within milliseconds; once any data arrives the deadline is lifted and
|
||||
* we read to EOF like readFileSync did (slow producers stay supported).
|
||||
* An empty-but-closed pipe (`: | gbrain put x`) EOFs immediately → ''.
|
||||
* A pipe that never delivers a byte times out → param stays unset, so
|
||||
* the existing required-param usage error fires (fail fast, exit 1).
|
||||
*
|
||||
* GBRAIN_STDIN_TIMEOUT_MS overrides the first-byte deadline (default 5000).
|
||||
* Exported for tests; called by the op dispatch right after parseOpArgs.
|
||||
*/
|
||||
export async function applyStdinParam(
|
||||
op: Operation,
|
||||
params: Record<string, unknown>,
|
||||
): Promise<void> {
|
||||
// Branch shape (stdin hint + missing param + `!process.stdin.isTTY` gate +
|
||||
// 5MB cap) is pinned by the R4 regression test for PR #1325's Windows fix
|
||||
// (test/cycle/regression-pr-wave-r1-r2-r4.test.ts) — keep the spelling.
|
||||
if (op.cliHints?.stdin && !params[op.cliHints.stdin] && !process.stdin.isTTY) {
|
||||
const stdinContent = readFileSync(0, 'utf-8');
|
||||
const content = await readStdinBounded();
|
||||
if (content === null) return; // no input arrived — let the required-param check fail fast
|
||||
const MAX_STDIN = 5_000_000; // 5MB
|
||||
if (Buffer.byteLength(stdinContent, 'utf-8') > MAX_STDIN) {
|
||||
if (Buffer.byteLength(content, 'utf-8') > MAX_STDIN) {
|
||||
console.error(`Error: stdin content exceeds ${MAX_STDIN} bytes. Split into smaller inputs.`);
|
||||
process.exit(1);
|
||||
}
|
||||
params[op.cliHints.stdin] = stdinContent;
|
||||
params[op.cliHints.stdin] = content;
|
||||
}
|
||||
}
|
||||
|
||||
return params;
|
||||
/** First-byte deadline for pipe/socket stdin (#3513). Env-overridable escape hatch. */
|
||||
function stdinFirstByteTimeoutMs(): number {
|
||||
const n = Number(process.env.GBRAIN_STDIN_TIMEOUT_MS);
|
||||
return Number.isFinite(n) && n > 0 ? n : 5000;
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns the full stdin content, '' for a readable-but-empty stdin, or
|
||||
* null when stdin is a pipe/socket that never delivered a byte within the
|
||||
* first-byte deadline (or the fd is closed/unreadable).
|
||||
*/
|
||||
export async function readStdinBounded(): Promise<string | null> {
|
||||
let isPipeOrSocket: boolean;
|
||||
try {
|
||||
const st = fstatSync(0);
|
||||
isPipeOrSocket = st.isFIFO() || st.isSocket();
|
||||
} catch {
|
||||
return null; // closed/invalid fd — treat as no input
|
||||
}
|
||||
if (!isPipeOrSocket) {
|
||||
// Regular file redirect, /dev/null, etc. — read returns without blocking.
|
||||
try {
|
||||
return readFileSync(0, 'utf-8');
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
return await new Promise<string | null>((resolve) => {
|
||||
const chunks: Buffer[] = [];
|
||||
let gotData = false;
|
||||
const timer = setTimeout(() => {
|
||||
if (!gotData) {
|
||||
process.stdin.destroy();
|
||||
resolve(null);
|
||||
}
|
||||
}, stdinFirstByteTimeoutMs());
|
||||
const finish = () => {
|
||||
clearTimeout(timer);
|
||||
resolve(Buffer.concat(chunks).toString('utf-8'));
|
||||
};
|
||||
process.stdin.on('data', (c: Buffer) => {
|
||||
if (!gotData) {
|
||||
gotData = true;
|
||||
clearTimeout(timer); // deadline applies to the FIRST byte only
|
||||
}
|
||||
chunks.push(c);
|
||||
});
|
||||
process.stdin.once('end', finish);
|
||||
process.stdin.once('error', finish);
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -872,7 +963,8 @@ export function applyThinClientSourceScope(
|
||||
params.source_id = resolved;
|
||||
}
|
||||
|
||||
async function makeContext(engine: BrainEngine, params: Record<string, unknown>): Promise<OperationContext> {
|
||||
// Exported for tests (same import-safety contract as applyThinClientSourceScope).
|
||||
export async function makeContext(engine: BrainEngine, params: Record<string, unknown>): Promise<OperationContext> {
|
||||
// v0.31.8 (D11): resolve sourceId via the canonical 6-tier chain. Honors
|
||||
// --source / GBRAIN_SOURCE / .gbrain-source / path-match / brain default /
|
||||
// 'default'. Wrapped in try/catch so a doctor / single-source brain that
|
||||
@@ -884,16 +976,21 @@ async function makeContext(engine: BrainEngine, params: Record<string, unknown>)
|
||||
// trusted local boundary) and consumed by federatedSearchScope in
|
||||
// operations.ts, which additionally gates on ctx.remote === false.
|
||||
let localFederated: string[] | undefined;
|
||||
// params.source is set when a CLI flag was parsed for the op (rare; most
|
||||
// CLI ops don't take --source). Falls through to env/dotfile/path-match.
|
||||
const explicit = (params.source as string | undefined) ?? null;
|
||||
try {
|
||||
const { resolveSourceWithTier, localFederatedSourceIds } = await import('./core/source-resolver.ts');
|
||||
// params.source is set when a CLI flag was parsed for the op (rare; most
|
||||
// CLI ops don't take --source). Falls through to env/dotfile/path-match.
|
||||
const explicit = (params.source as string | undefined) ?? null;
|
||||
const resolved = await resolveSourceWithTier(engine, explicit);
|
||||
sourceId = resolved.source_id;
|
||||
localFederated = await localFederatedSourceIds(engine, resolved.source_id, resolved.tier);
|
||||
} catch {
|
||||
// Source resolution failed (e.g. sources table doesn't exist on a fresh
|
||||
} catch (err) {
|
||||
// #1712: an EXPLICIT --source that fails to resolve (invalid id, or a
|
||||
// source that doesn't exist) must error loudly — the blanket swallow
|
||||
// turned `--source __all__` and typos into a silent `default` scope,
|
||||
// which is how three bug reports became debugging sessions.
|
||||
if (explicit) throw err;
|
||||
// Ambient resolution failed (e.g. sources table doesn't exist on a fresh
|
||||
// pre-init brain). Leave sourceId unset; engine read methods fall through
|
||||
// to the cross-source view (D16 back-compat path).
|
||||
sourceId = undefined;
|
||||
@@ -1173,6 +1270,20 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
await runInit(args);
|
||||
return;
|
||||
}
|
||||
if (command === 'bench') {
|
||||
// #3502 sweep: `gbrain bench publish` was documented (docs/eval-bench.md,
|
||||
// KEY_FILES.md, and eval-gate's own --help text) but never dispatched —
|
||||
// the promised-but-unwired class retrieval-upgrade (#3390) fixed before.
|
||||
// Pure file-in/file-out (NDJSON → baseline); no DB, no engine.
|
||||
if (args[0] === 'publish') {
|
||||
const { runBenchPublish } = await import('./commands/bench-publish.ts');
|
||||
await runBenchPublish(args.slice(1));
|
||||
return;
|
||||
}
|
||||
console.error('Usage: gbrain bench publish --from <captured.ndjson> --to <X.baseline.ndjson> [flags]');
|
||||
console.error('Run `gbrain bench publish --help` for the full flag list.');
|
||||
process.exit(args[0] === '--help' || args[0] === '-h' ? 0 : 2);
|
||||
}
|
||||
// v0.37 fix wave (deferred TODO, shipped): one-command wipe-and-reinit.
|
||||
// Spawns its own engine internally so no pre-bound engine needed.
|
||||
if (command === 'reinit-pglite') {
|
||||
@@ -2448,6 +2559,7 @@ TOOLS
|
||||
publish <page.md> [--password] Shareable HTML (strips private data, optional AES-256)
|
||||
check-backlinks <check|fix> [dir] Find/fix missing back-links across brain
|
||||
lint <dir|file> [--fix] Catch LLM artifacts, placeholder dates, bad frontmatter
|
||||
backfill <kind|list> v0.30.1: run a registered backfill (effective-date, ...)
|
||||
orphans [--json] [--count] Find pages with no inbound wikilinks
|
||||
salience [--days N] [--kind P] v0.29: pages ranked by emotional + activity salience
|
||||
anomalies [--since D] [--sigma N] v0.29: cohort-based statistical anomalies (tag, type)
|
||||
|
||||
@@ -8,7 +8,7 @@ import {
|
||||
semverGt,
|
||||
semverLte,
|
||||
} from '../core/semver.ts';
|
||||
import { writeUpdateCache, type UpdateMarker } from '../core/self-upgrade.ts';
|
||||
import { readUpdateCache, writeUpdateCache, type UpdateMarker } from '../core/self-upgrade.ts';
|
||||
|
||||
/** Best-effort cache write — a read-only ~/.gbrain must never make the check throw. */
|
||||
function safeWriteCache(marker: UpdateMarker): void {
|
||||
@@ -45,26 +45,53 @@ function upgradeCommandForMethod(method: string): string {
|
||||
}
|
||||
}
|
||||
|
||||
/** Where the latest version is resolved from. gbrain publishes NO GitHub
|
||||
* releases (the `releases/latest` API is a permanent 404), so the release
|
||||
* train's source of truth is the `VERSION` file on master — same trusted host
|
||||
* `fetchChangelog` already uses. An npm fallback was rejected: the `gbrain`
|
||||
* package on npm is an unrelated GPU library (#505), so it would produce false
|
||||
* upgrade prompts pointing at a stranger's package. */
|
||||
const VERSION_SOURCE_URL = 'https://raw.githubusercontent.com/garrytan/gbrain/master/VERSION';
|
||||
const RELEASE_NOTES_URL = 'https://github.com/garrytan/gbrain/blob/master/CHANGELOG.md';
|
||||
|
||||
/** Extract a version from the raw VERSION file body: first line, optional `v`
|
||||
* prefix, optional `-suffix` channel tag (`0.31.1.1-fixwave` compares as its
|
||||
* numeric base — fail-safe: a suffix-only bump never prompts). Body is bounded
|
||||
* before parsing so a malformed/huge response can't blow up the check. */
|
||||
export function parseVersionFileBody(body: string): string | null {
|
||||
const firstLine = body.slice(0, 256).trim().split('\n')[0].trim();
|
||||
const m = firstLine.match(/^v?(\d+\.\d+\.\d+(?:\.\d+)?)(?:[-+][0-9A-Za-z.-]+)?$/);
|
||||
return m && isValidVersionString(m[1]) ? m[1] : null;
|
||||
}
|
||||
|
||||
export type LatestReleaseResult =
|
||||
| { ok: true; tag: string; published_at: string; url: string }
|
||||
| { ok: false; reason: 'network_error' | 'no_releases' };
|
||||
|
||||
/**
|
||||
* Fetch the latest GitHub release. Exported (v0.42) so the self-upgrade refresh
|
||||
* path and tests can reuse it. 5s timeout (was 10s) — this runs on the detached
|
||||
* refresh, never the hot path, but a tight bound keeps the refresh cheap.
|
||||
* Resolve the latest published gbrain version (from VERSION on master — see
|
||||
* VERSION_SOURCE_URL). Exported (v0.42) so the self-upgrade refresh path and
|
||||
* tests can reuse it. 5s timeout — this runs on the detached refresh, never the
|
||||
* hot path. Failures are discriminated: `network_error` (offline/timeout) vs
|
||||
* `no_releases` (endpoint answered but no usable version).
|
||||
*/
|
||||
export async function fetchLatestRelease(): Promise<{ tag: string; published_at: string; url: string } | null> {
|
||||
export async function fetchLatestRelease(): Promise<LatestReleaseResult> {
|
||||
let res: Response;
|
||||
try {
|
||||
const res = await fetch('https://api.github.com/repos/garrytan/gbrain/releases/latest', {
|
||||
res = await fetch(VERSION_SOURCE_URL, {
|
||||
headers: { 'User-Agent': `gbrain/${VERSION}` },
|
||||
signal: AbortSignal.timeout(5_000),
|
||||
});
|
||||
if (!res.ok) return null;
|
||||
const data = await res.json() as any;
|
||||
return {
|
||||
tag: data.tag_name || '',
|
||||
published_at: data.published_at || '',
|
||||
url: data.html_url || '',
|
||||
};
|
||||
} catch {
|
||||
return null;
|
||||
return { ok: false, reason: 'network_error' };
|
||||
}
|
||||
try {
|
||||
if (!res.ok) return { ok: false, reason: 'no_releases' };
|
||||
const tag = parseVersionFileBody(await res.text());
|
||||
if (!tag) return { ok: false, reason: 'no_releases' };
|
||||
return { ok: true, tag, published_at: '', url: RELEASE_NOTES_URL };
|
||||
} catch {
|
||||
return { ok: false, reason: 'network_error' };
|
||||
}
|
||||
}
|
||||
|
||||
@@ -118,17 +145,33 @@ export function extractChangelogBetween(changelog: string, from: string, to: str
|
||||
}
|
||||
|
||||
/**
|
||||
* Fetch the latest release and write the self-upgrade cache (the marker line
|
||||
* read by the CLI startup hook). Fail-open: on any network failure we cache
|
||||
* `UP_TO_DATE <current>` so the TTL prevents hammering GitHub on every
|
||||
* invocation. Returns the resolved marker for callers that want it. This is the
|
||||
* function the detached single-flight refresh (`gbrain check-update
|
||||
* --refresh-cache`) invokes.
|
||||
* A failed check must NEVER write `up_to_date` — that was #486: the fetch
|
||||
* failed permanently (dead releases API) and every user was told "you're
|
||||
* current" forever. Instead, re-write the last-known-good marker (bumping its
|
||||
* mtime so the cache TTL still throttles retries and a network blip can't
|
||||
* erase a pending upgrade_available notice). No prior marker → write nothing;
|
||||
* the next invocation retries.
|
||||
*/
|
||||
function preserveCacheOnFailedCheck(): void {
|
||||
try {
|
||||
const prior = readUpdateCache();
|
||||
if (prior) safeWriteCache(prior.marker);
|
||||
} catch {
|
||||
/* best-effort */
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Fetch the latest version and write the self-upgrade cache (the marker line
|
||||
* read by the CLI startup hook). On fetch failure the last-known-good marker is
|
||||
* preserved (see preserveCacheOnFailedCheck) — never a fabricated `up_to_date`.
|
||||
* This is the function the detached single-flight refresh (`gbrain
|
||||
* check-update --refresh-cache`) invokes.
|
||||
*/
|
||||
export async function refreshUpdateCache(): Promise<void> {
|
||||
const release = await fetchLatestRelease();
|
||||
if (!release) {
|
||||
safeWriteCache({ kind: 'up_to_date', current: VERSION });
|
||||
if (!release.ok) {
|
||||
preserveCacheOnFailedCheck();
|
||||
return;
|
||||
}
|
||||
const latestVersion = release.tag.replace(/^v/, '');
|
||||
@@ -166,9 +209,8 @@ export async function runCheckUpdate(args: string[]) {
|
||||
|
||||
const release = await fetchLatestRelease();
|
||||
|
||||
if (!release) {
|
||||
// Warm the cache fail-open so the startup hook doesn't re-fetch every call.
|
||||
safeWriteCache({ kind: 'up_to_date', current: VERSION });
|
||||
if (!release.ok) {
|
||||
preserveCacheOnFailedCheck();
|
||||
if (json) {
|
||||
console.log(JSON.stringify({
|
||||
current_version: VERSION,
|
||||
@@ -179,10 +221,12 @@ export async function runCheckUpdate(args: string[]) {
|
||||
release_url: '',
|
||||
changelog_diff: '',
|
||||
published_at: '',
|
||||
error: 'no_releases',
|
||||
error: release.reason,
|
||||
}, null, 2));
|
||||
} else if (release.reason === 'network_error') {
|
||||
console.log(`GBrain ${VERSION} — could not check for updates (network unavailable).`);
|
||||
} else {
|
||||
console.log(`GBrain ${VERSION} — could not check for updates (no releases found or network unavailable).`);
|
||||
console.log(`GBrain ${VERSION} — could not determine the latest published version.`);
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
+81
-8
@@ -40,7 +40,7 @@ import {
|
||||
buildBasenameIndex,
|
||||
queryBasenameIndex,
|
||||
} from '../core/link-extraction.ts';
|
||||
import { isSourceUnchangedSinceSync } from '../core/git-head.ts';
|
||||
import { probeSourceGitState } from '../core/git-head.ts';
|
||||
// v0.41.32.0: remote staleness reads the stored newest_content_at column via
|
||||
// this pure comparator (no git subprocess on the HTTP MCP doctor path).
|
||||
import { lagFromContentMs } from '../core/source-health.ts';
|
||||
@@ -53,6 +53,7 @@ import { isUndefinedColumnError } from '../core/utils.ts';
|
||||
// drift from what search actually filters.
|
||||
import { resolveHardExcludes, DEFAULT_HARD_EXCLUDES } from '../core/search/source-boost.ts';
|
||||
import { escapeLikePattern, buildVisibilityClause } from '../core/search/sql-ranking.ts';
|
||||
import { unverifiedExtractionFragment } from '../core/extraction-review.ts';
|
||||
import { hnswIndexExpected, hnswMaxDimsForType } from '../core/vector-index.ts';
|
||||
|
||||
export interface Check {
|
||||
@@ -3620,6 +3621,52 @@ export async function checkLinksExtractionLag(
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* issue #160 — unverified_extractions doctor check.
|
||||
*
|
||||
* The extraction quarantine lane parks auto-extracted entity stubs
|
||||
* (frontmatter `provenance: 'auto-extracted'` + `status: 'unverified'`)
|
||||
* until the owner promotes or rejects them. A queue nobody reviews decays
|
||||
* into invisible clutter, so this check counts stubs older than N days
|
||||
* (default 7) and nudges toward the review surface. Exported for direct
|
||||
* testing (mirrors checkLinksExtractionLag).
|
||||
*/
|
||||
export async function checkUnverifiedExtractions(
|
||||
engine: BrainEngine,
|
||||
opts?: { sourceId?: string; days?: number },
|
||||
): Promise<Check> {
|
||||
const name = 'unverified_extractions';
|
||||
const days = opts?.days ?? 7;
|
||||
const sourceId = opts?.sourceId;
|
||||
try {
|
||||
const params: unknown[] = [String(days)];
|
||||
let srcClause = '';
|
||||
if (sourceId) {
|
||||
params.push(sourceId);
|
||||
srcClause = 'AND p.source_id = $2';
|
||||
}
|
||||
const rows = await engine.executeRaw<{ n: string | number }>(
|
||||
`SELECT COUNT(*)::int AS n FROM pages p
|
||||
WHERE p.deleted_at IS NULL
|
||||
AND ${unverifiedExtractionFragment('p')}
|
||||
AND p.created_at < now() - ($1 || ' days')::interval
|
||||
${srcClause}`,
|
||||
params,
|
||||
);
|
||||
const n = Number(rows[0]?.n ?? 0);
|
||||
return {
|
||||
name,
|
||||
status: n > 0 ? 'warn' : 'ok',
|
||||
message: n > 0
|
||||
? `${n} unverified auto-extracted entity stub(s) older than ${days} days awaiting review. List with 'gbrain extraction-pending'; promote/reject with 'gbrain extraction-review <promote|reject> --slugs <slug,...>'.`
|
||||
: 'No stale unverified extraction stubs',
|
||||
details: { count: n, days, source_id: sourceId ?? null },
|
||||
};
|
||||
} catch (e) {
|
||||
return { name, status: 'warn', message: `Could not check unverified_extractions: ${(e as Error).message}` };
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* issue #1678 — extract_atoms_backlog doctor check.
|
||||
*
|
||||
@@ -4015,29 +4062,51 @@ export async function checkSyncFreshness(
|
||||
// All four must hold; otherwise fall through to the time-based check.
|
||||
// The chunker version match is computed here (not in the helper)
|
||||
// because it depends on engine state, not git state.
|
||||
//
|
||||
// Clone-unavailable fallback: on stateless deploys (Docker on EB /
|
||||
// K8s / Fly — the platforms the cloud recipes produce), a container
|
||||
// restart wipes `local_path` and each clone is only re-materialized
|
||||
// when that source's next sync job runs. Until then the HEAD probe
|
||||
// cannot run at all ('unavailable'), which previously fell through to
|
||||
// raw wall-clock age — and since a no-op sync doesn't advance
|
||||
// `last_sync_at`, every QUIET source read as stale/FAIL after a
|
||||
// restart (score-sinking alert storm; observed live: 16-source brain,
|
||||
// 12 clones gone after a config-update restart, doctor 70→30).
|
||||
// 'unavailable' + chunker match now reuses the v0.41.32.0 REMOTE lag
|
||||
// signal (newest_content_at) below — DB-only, no subprocess, and it
|
||||
// still reports staleness whenever content really is newer than the
|
||||
// last sync. 'changed' (readable clone with real work) keeps
|
||||
// wall-clock exactly as before, and a chunker mismatch is never
|
||||
// masked (D7): it disables the fallback too.
|
||||
let cloneUnavailable = false;
|
||||
if (localOnly) {
|
||||
const gitUnchanged = isSourceUnchangedSinceSync(
|
||||
const gitState = probeSourceGitState(
|
||||
source.local_path,
|
||||
source.last_commit,
|
||||
{ requireCleanWorkingTree: 'ignore-untracked' },
|
||||
);
|
||||
const chunkerMatch = source.chunker_version === currentChunkerVersion;
|
||||
if (gitUnchanged && chunkerMatch) {
|
||||
if (gitState === 'unchanged' && chunkerMatch) {
|
||||
unchanged_count++;
|
||||
continue;
|
||||
}
|
||||
cloneUnavailable = gitState === 'unavailable' && chunkerMatch;
|
||||
}
|
||||
|
||||
// v0.41.32.0: REMOTE path (doctorReportRemote, !localOnly) computes lag
|
||||
// from the stored newest_content_at column — NO git subprocess on a
|
||||
// DB-supplied local_path (preserves the v0.41.27.0 trust boundary). A
|
||||
// quiet repo whose newest commit predates its last sync reports 0; NULL
|
||||
// column → wall-clock fallback. LOCAL fall-through keeps wall-clock: the
|
||||
// short-circuit already failed, so the source genuinely has work and
|
||||
// "hours since last sync" is the right staleness measure. The `ageMs < 0`
|
||||
// skew check above still runs on raw wall-clock for both paths (A1).
|
||||
// column → wall-clock fallback. LOCAL fall-through keeps wall-clock when
|
||||
// the clone is READABLE: the short-circuit failed on real evidence
|
||||
// (HEAD moved / dirty tree), so the source genuinely has work and
|
||||
// "hours since last sync" is the right staleness measure. A local clone
|
||||
// that is UNAVAILABLE (not yet re-materialized, see above) carries no
|
||||
// evidence either way, so it borrows this same DB-only lag. The
|
||||
// `ageMs < 0` skew check above still runs on raw wall-clock for both
|
||||
// paths (A1).
|
||||
let thresholdAgeMs = ageMs;
|
||||
if (!localOnly) {
|
||||
if (!localOnly || cloneUnavailable) {
|
||||
const contentMs = source.newest_content_at
|
||||
? new Date(source.newest_content_at).getTime()
|
||||
: null;
|
||||
@@ -6891,6 +6960,10 @@ export async function buildChecks(
|
||||
checks.push({ name: 'flagged_pages', status: 'ok', message: `Skipped (${msg})` });
|
||||
}
|
||||
|
||||
// issue #160: extraction quarantine lane review nudge.
|
||||
progress.heartbeat('unverified_extractions');
|
||||
checks.push(await checkUnverifiedExtractions(engine, { sourceId: orphanRatioSourceId }));
|
||||
|
||||
// 11a. Frontmatter integrity (v0.22.4, hardened in v0.38.2.0).
|
||||
// scanBrainSources walks every registered source's local_path on disk
|
||||
// (not from the DB), invoking parseMarkdown(..., {validate:true}) per
|
||||
|
||||
+50
-3
@@ -19,6 +19,26 @@ import {
|
||||
} from '../core/pace-mode.ts';
|
||||
import { tryAcquireDbLock, type DbLockHandle } from '../core/db-lock.ts';
|
||||
import { embedBackfillLockId } from '../core/embed-backfill-lock.ts';
|
||||
import { wrapChunkTextsForStoredMode } from '../core/embedding-context.ts';
|
||||
import { titleTierCorpusGeneration } from '../core/contextual-retrieval-service.ts';
|
||||
import type { Page } from '../core/types.ts';
|
||||
|
||||
/**
|
||||
* #3507 — after a plain re-embed fully re-embedded a `per_chunk_synopsis`
|
||||
* page at the title-only tier (see wrapChunkTextsForStoredMode), restamp the
|
||||
* page's CR state to 'title' so `contextual_retrieval_mode` keeps describing
|
||||
* the vectors actually in the column. The reindex sweep restores the synopsis
|
||||
* tier later. No-op for every other mode.
|
||||
*/
|
||||
export async function restampIfDemotedToTitleTier(
|
||||
engine: BrainEngine,
|
||||
page: Pick<Page, 'contextual_retrieval_mode'> | null | undefined,
|
||||
slug: string,
|
||||
sourceId: string,
|
||||
): Promise<void> {
|
||||
if (page?.contextual_retrieval_mode !== 'per_chunk_synopsis') return;
|
||||
await engine.updatePageContextualRetrievalState(slug, sourceId, 'title', titleTierCorpusGeneration());
|
||||
}
|
||||
|
||||
export interface EmbedOpts {
|
||||
/** Embed ALL pages (every chunk). */
|
||||
@@ -599,7 +619,11 @@ async function embedPage(
|
||||
return;
|
||||
}
|
||||
|
||||
const embeddings = await embedBatch(toEmbed.map(c => c.chunk_text), { abortSignal: signal });
|
||||
// #3507: embed with the page's STORED wrapping convention (title-tier
|
||||
// contextual prefix when the page was embedded wrapped), not raw
|
||||
// chunk_text — otherwise a re-embed silently strips the contextual
|
||||
// prefixes the sync path applied. fenced_code chunks stay unwrapped.
|
||||
const embeddings = await embedBatch(wrapChunkTextsForStoredMode(page, toEmbed), { abortSignal: signal });
|
||||
const embeddingMap = new Map<number, Float32Array>();
|
||||
for (let j = 0; j < toEmbed.length; j++) {
|
||||
embeddingMap.set(toEmbed[j].chunk_index, embeddings[j]);
|
||||
@@ -622,6 +646,9 @@ async function embedPage(
|
||||
// such a page and then stamps it.
|
||||
if (toEmbed.length === chunks.length) {
|
||||
await engine.setPageEmbeddingSignature(slug, { sourceId, signature: currentEmbeddingSignature() });
|
||||
// #3507: a fully re-embedded per_chunk_synopsis page landed at the
|
||||
// title tier — keep the stamped mode honest.
|
||||
await restampIfDemotedToTitleTier(engine, page, slug, page.source_id);
|
||||
}
|
||||
result.embedded += toEmbed.length;
|
||||
result.pages_processed++;
|
||||
@@ -763,7 +790,8 @@ async function embedAll(
|
||||
}
|
||||
|
||||
try {
|
||||
const embeddings = await embedBatch(toEmbed.map(c => c.chunk_text));
|
||||
// #3507: reproduce the page's stored wrapping convention (see embedPage).
|
||||
const embeddings = await embedBatch(wrapChunkTextsForStoredMode(page, toEmbed));
|
||||
// Build a map of new embeddings by chunk_index
|
||||
const embeddingMap = new Map<number, Float32Array>();
|
||||
for (let j = 0; j < toEmbed.length; j++) {
|
||||
@@ -785,6 +813,11 @@ async function embedAll(
|
||||
await observed(pacer, () =>
|
||||
engine.setPageEmbeddingSignature(page.slug, { sourceId: pageSourceId, signature }),
|
||||
);
|
||||
// #3507: --all fully re-embeds; a per_chunk_synopsis page landed at
|
||||
// the title tier — keep the stamped mode honest.
|
||||
await observed(pacer, () =>
|
||||
restampIfDemotedToTitleTier(engine, page, page.slug, pageSourceId),
|
||||
);
|
||||
result.embedded += toEmbed.length;
|
||||
} catch (e: unknown) {
|
||||
serr(`\n Error embedding ${page.slug}: ${e instanceof Error ? e.message : e}`);
|
||||
@@ -1098,7 +1131,13 @@ async function embedAllStale(
|
||||
const keySourceId = stale[0]?.source_id ?? 'default';
|
||||
const slug = stale[0].slug;
|
||||
try {
|
||||
const embeddings = await embedBatchWithBackoff(stale.map(c => c.chunk_text), { abortSignal: effectiveSignal });
|
||||
// #3507: fetch the page row for its title + stored CR mode so the
|
||||
// re-embed reproduces the page's wrapping convention instead of
|
||||
// silently stripping contextual prefixes — `embed --stale` is the
|
||||
// NORMAL post-model-migration path, so raw-text embedding here
|
||||
// quietly converted whole corpora to the unwrapped convention.
|
||||
const pageRow = await observed(pacer, () => engine.getPage(slug, { sourceId: keySourceId }));
|
||||
const embeddings = await embedBatchWithBackoff(wrapChunkTextsForStoredMode(pageRow, stale), { abortSignal: effectiveSignal });
|
||||
// Re-fetch existing chunks and merge to avoid deleting non-stale chunks.
|
||||
const existing = await observed(pacer, () => engine.getChunks(slug, { sourceId: keySourceId }));
|
||||
const staleIdxToEmbedding = new Map<number, Float32Array>();
|
||||
@@ -1126,6 +1165,14 @@ async function embedAllStale(
|
||||
engine.setPageEmbeddingSignature(slug, { sourceId: keySourceId, signature }),
|
||||
);
|
||||
}
|
||||
// #3507: a FULLY re-embedded per_chunk_synopsis page landed at the
|
||||
// title tier — keep the stamped mode honest. Partially-stale pages
|
||||
// stay stamped as-is (mixed provenance; reindex sweeps fix them).
|
||||
if (stale.length === existing.length) {
|
||||
await observed(pacer, () =>
|
||||
restampIfDemotedToTitleTier(engine, pageRow, slug, keySourceId),
|
||||
);
|
||||
}
|
||||
result.embedded += stale.length;
|
||||
} catch (e: unknown) {
|
||||
// Budget/abort-fired cancellations are expected on the way out; don't
|
||||
|
||||
@@ -149,6 +149,40 @@ export const ALLOWED_TYPES = [
|
||||
] as const;
|
||||
export type AllowedType = (typeof ALLOWED_TYPES)[number];
|
||||
|
||||
/**
|
||||
* Granular collector page-types that alias into each canonical conversation
|
||||
* bucket. The v2 type-consolidation pack retypes these to the canonical names
|
||||
* (`slack-dm-day`/`slack-thread` → `slack`, `email-digest` → `email`), but a
|
||||
* brain that hasn't run that pack still carries the collector's granular types
|
||||
* in `pages.type`. Without this expansion, `listPages({ type: 'slack' })`
|
||||
* matches zero rows on such brains and the whole comms corpus is silently
|
||||
* skipped (facts stay empty → `find_trajectory` returns nothing). The canonical
|
||||
* name is always included first so consolidated brains keep working unchanged.
|
||||
*/
|
||||
export const ALLOWED_TYPE_ALIASES: Record<AllowedType, readonly string[]> = {
|
||||
conversation: ['conversation'],
|
||||
meeting: ['meeting'],
|
||||
slack: ['slack', 'slack-dm-day', 'slack-thread'],
|
||||
email: ['email', 'email-digest'],
|
||||
imessage: ['imessage'],
|
||||
'imessage-daily': ['imessage-daily'],
|
||||
};
|
||||
|
||||
/**
|
||||
* Expand the requested logical types to the concrete `pages.type` values to
|
||||
* enumerate, canonical-first and de-duplicated. Unknown types pass through
|
||||
* unchanged so an explicit override is never dropped.
|
||||
*/
|
||||
export function pageTypesForAllowed(types: readonly AllowedType[]): string[] {
|
||||
const out: string[] = [];
|
||||
for (const t of types) {
|
||||
for (const concrete of ALLOWED_TYPE_ALIASES[t] ?? [t]) {
|
||||
if (!out.includes(concrete)) out.push(concrete);
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* Pagination batch size for listPages enumeration. Per-batch memory
|
||||
* worst case = BATCH × MAX_PAGE_BODY_BYTES = 250MB at default 10
|
||||
@@ -1264,13 +1298,18 @@ export async function runExtractConversationFactsCore(
|
||||
}
|
||||
};
|
||||
|
||||
// Expand logical types (conversation/meeting/slack/email) to the concrete
|
||||
// `pages.type` values to enumerate, so brains on the granular collector
|
||||
// types are not silently skipped (see ALLOWED_TYPE_ALIASES).
|
||||
const concreteTypes = pageTypesForAllowed(types);
|
||||
|
||||
if (opts.slug) {
|
||||
const page = await engine.getPage(opts.slug, { sourceId });
|
||||
if (!page) {
|
||||
result.pages_skipped_disappeared++;
|
||||
return;
|
||||
}
|
||||
if (!types.includes(page.type as AllowedType)) {
|
||||
if (!concreteTypes.includes(page.type)) {
|
||||
result.pages_skipped++;
|
||||
return;
|
||||
}
|
||||
@@ -1284,7 +1323,7 @@ export async function runExtractConversationFactsCore(
|
||||
// honors AbortSignal at each claim boundary and threads
|
||||
// BudgetExhausted abort (D13) automatically.
|
||||
let processedPagesCount = 0;
|
||||
pageLoop: for (const type of types) {
|
||||
pageLoop: for (const type of concreteTypes) {
|
||||
let offset = 0;
|
||||
// eslint-disable-next-line no-constant-condition
|
||||
while (true) {
|
||||
|
||||
+23
-4
@@ -35,7 +35,7 @@ import type { BrainEngine, LinkBatchInput, TimelineBatchInput } from '../core/en
|
||||
import type { PageType } from '../core/types.ts';
|
||||
import { parseMarkdown } from '../core/markdown.ts';
|
||||
import {
|
||||
extractPageLinks, parseTimelineEntries, inferLinkType, makeResolver,
|
||||
extractPageLinks, parseTimelineEntries, deriveTimelineAnchor, inferLinkType, makeResolver,
|
||||
extractFrontmatterLinks, isGlobalBasenameEnabled, LINK_EXTRACTOR_VERSION_TS,
|
||||
WIKILINK_BASENAME_LINK_TYPE,
|
||||
buildBasenameIndex, queryBasenameIndex, stripCodeBlocks,
|
||||
@@ -749,6 +749,12 @@ export async function runExtract(engine: BrainEngine, args: string[]) {
|
||||
// v0.41.18.0 (A11, T8): --from-meetings extracts timeline entries from
|
||||
// meeting pages onto each discussed entity. Timeline subcommand only.
|
||||
const fromMeetings = args.includes('--from-meetings');
|
||||
// --infer-dates: for pages whose body has NO parseable timeline line, anchor
|
||||
// one entry at the page's computed effective_date (frontmatter / filename date,
|
||||
// never the updated_at fallback). Default OFF for back-compat — comms/calendar
|
||||
// brains opt in to populate timeline from slug/frontmatter dates. DB-source only
|
||||
// (needs the full Page.effective_date, which getPage projects).
|
||||
const inferDates = args.includes('--infer-dates');
|
||||
// v0.41.17.0 (T7, D9): --workers N parsed via the shared validator.
|
||||
// Honored on the fs-walk inner loops only; DB-source paths stay
|
||||
// serial in v0.41.17.0 (see ExtractOpts.workers doc).
|
||||
@@ -963,7 +969,7 @@ Status (v0.42):
|
||||
result.pages_processed = r.pages;
|
||||
}
|
||||
if (subcommand === 'timeline' || subcommand === 'all') {
|
||||
const r = await extractTimelineFromDB(engine, dryRun, jsonMode, typeFilter, since, { sourceIdFilter });
|
||||
const r = await extractTimelineFromDB(engine, dryRun, jsonMode, typeFilter, since, { sourceIdFilter, inferDates });
|
||||
result.timeline_entries_created = r.created;
|
||||
result.pages_processed = Math.max(result.pages_processed, r.pages);
|
||||
}
|
||||
@@ -1583,7 +1589,7 @@ async function extractTimelineFromDB(
|
||||
jsonMode: boolean,
|
||||
typeFilter: PageType | undefined,
|
||||
since: string | undefined,
|
||||
opts?: { sourceIdFilter?: string },
|
||||
opts?: { sourceIdFilter?: string; inferDates?: boolean },
|
||||
): Promise<{ created: number; pages: number }> {
|
||||
// v0.32.8: listAllPageRefs enumerates (slug, source_id) pairs so we can
|
||||
// thread sourceId to getPage and addTimelineEntriesBatch. Pre-fix used
|
||||
@@ -1592,6 +1598,7 @@ async function extractTimelineFromDB(
|
||||
// v0.37.7.0 #1204: when sourceIdFilter is set, scope the walk to one
|
||||
// source so federated brain users can extract per-source.
|
||||
const sourceIdFilter = opts?.sourceIdFilter;
|
||||
const inferDates = opts?.inferDates ?? false;
|
||||
const allRefs = sourceIdFilter
|
||||
? (await engine.listAllPageRefs()).filter(r => r.source_id === sourceIdFilter)
|
||||
: await engine.listAllPageRefs();
|
||||
@@ -1631,7 +1638,19 @@ async function extractTimelineFromDB(
|
||||
}
|
||||
|
||||
const fullContent = page.compiled_truth + '\n' + page.timeline;
|
||||
const entries = parseTimelineEntries(fullContent);
|
||||
let entries = parseTimelineEntries(fullContent);
|
||||
// --infer-dates: pages with no in-body timeline line but a trustworthy
|
||||
// content date (frontmatter / filename) get one anchor entry at that date.
|
||||
// Applied ONLY on the zero-entry path so it never shadows a real timeline.
|
||||
if (entries.length === 0 && inferDates) {
|
||||
const anchor = deriveTimelineAnchor({
|
||||
slug,
|
||||
title: page.title,
|
||||
effectiveDate: page.effective_date,
|
||||
effectiveDateSource: page.effective_date_source,
|
||||
});
|
||||
if (anchor) entries = [anchor];
|
||||
}
|
||||
|
||||
for (const entry of entries) {
|
||||
if (dryRunSeen) {
|
||||
|
||||
+37
-5
@@ -143,6 +143,31 @@ export function resolveWorkerConcurrency(args: string[], env: NodeJS.ProcessEnv
|
||||
return parsed;
|
||||
}
|
||||
|
||||
/**
|
||||
* #3026: the thin-client `list`/`get` branches receive jobs as parsed JSON
|
||||
* off the MCP wire, where every timestamp is an ISO string — but formatJob /
|
||||
* formatJobDetail (and the stalled-detection comparison) hold a Date
|
||||
* contract, hydrated locally by MinionQueue.rowToJob. Rehydrate once at the
|
||||
* unpack boundary so both paths hand the formatters real Dates. Exported for
|
||||
* unit tests.
|
||||
*/
|
||||
const JOB_DATE_FIELDS = [
|
||||
'created_at', 'updated_at', 'started_at', 'finished_at', 'lock_until', 'delay_until',
|
||||
] as const;
|
||||
|
||||
export function rehydrateJobDates<T>(job: T): T {
|
||||
if (!job || typeof job !== 'object') return job;
|
||||
const rec = job as { [k: string]: unknown };
|
||||
for (const field of JOB_DATE_FIELDS) {
|
||||
const v = rec[field];
|
||||
if (typeof v === 'string') {
|
||||
const d = new Date(v);
|
||||
if (!Number.isNaN(d.getTime())) rec[field] = d;
|
||||
}
|
||||
}
|
||||
return job;
|
||||
}
|
||||
|
||||
function formatJob(job: MinionJob): string {
|
||||
const dur = job.finished_at && job.started_at
|
||||
? `${((job.finished_at.getTime() - job.started_at.getTime()) / 1000).toFixed(1)}s`
|
||||
@@ -208,7 +233,7 @@ USAGE
|
||||
gbrain jobs get <id>
|
||||
gbrain jobs cancel <id>
|
||||
gbrain jobs retry <id>
|
||||
gbrain jobs prune [--older-than 30d]
|
||||
gbrain jobs prune [--older-than 30d] [--dry-run]
|
||||
gbrain jobs delete <id>
|
||||
gbrain jobs stats
|
||||
gbrain jobs smoke
|
||||
@@ -496,7 +521,7 @@ HANDLER TYPES (built in)
|
||||
const raw = await callRemoteTool(cfg!, 'list_jobs', {
|
||||
status, queue: queueName, limit,
|
||||
}, { timeoutMs: 30_000 });
|
||||
jobs = unpackToolResult<MinionJob[]>(raw);
|
||||
jobs = unpackToolResult<MinionJob[]>(raw).map((j) => rehydrateJobDates(j));
|
||||
} else {
|
||||
try { await queue.ensureSchema(); }
|
||||
catch (e) { console.error(e instanceof Error ? e.message : String(e)); process.exit(1); }
|
||||
@@ -525,7 +550,7 @@ HANDLER TYPES (built in)
|
||||
if (isThinClient(cfg)) {
|
||||
try {
|
||||
const raw = await callRemoteTool(cfg!, 'get_job', { id }, { timeoutMs: 30_000 });
|
||||
job = unpackToolResult<MinionJob | null>(raw);
|
||||
job = rehydrateJobDates(unpackToolResult<MinionJob | null>(raw));
|
||||
} catch (e) {
|
||||
// The remote op throws `invalid_params` on not-found; surface as
|
||||
// the same "Job not found" exit-1 the local path produces.
|
||||
@@ -608,8 +633,15 @@ HANDLER TYPES (built in)
|
||||
try { await queue.ensureSchema(); }
|
||||
catch (e) { console.error(e instanceof Error ? e.message : String(e)); process.exit(1); }
|
||||
|
||||
const count = await queue.prune({ olderThan: new Date(Date.now() - days * 86400000) });
|
||||
console.log(`Pruned ${count} jobs older than ${days} days.`);
|
||||
// #2712: --dry-run previews the count without deleting. It used to be
|
||||
// silently ignored (the destructive default ran anyway).
|
||||
const dryRun = hasFlag(args, '--dry-run');
|
||||
const count = await queue.prune({ olderThan: new Date(Date.now() - days * 86400000), dryRun });
|
||||
if (dryRun) {
|
||||
console.log(`[dry-run] Would prune ${count} jobs older than ${days} days. Nothing deleted.`);
|
||||
} else {
|
||||
console.log(`Pruned ${count} jobs older than ${days} days.`);
|
||||
}
|
||||
break;
|
||||
}
|
||||
|
||||
|
||||
@@ -35,7 +35,8 @@ export async function runSelfUpgrade(args: string[]): Promise<void> {
|
||||
const force = args.includes('--force');
|
||||
const json = args.includes('--json');
|
||||
|
||||
const release = await fetchLatestRelease();
|
||||
const result = await fetchLatestRelease();
|
||||
const release = result.ok ? result : null;
|
||||
const latest = release ? release.tag.replace(/^v/, '') : null;
|
||||
const behind = !!latest && isValidVersionString(latest) && isNewerVersion(VERSION, latest);
|
||||
|
||||
|
||||
+15
-8
@@ -3,6 +3,7 @@
|
||||
*
|
||||
* Subcommands:
|
||||
* takes <slug> — list takes for a page
|
||||
* takes list — list all active takes (#2079)
|
||||
* takes search "<query>" [--who h] — keyword search across all takes
|
||||
* takes add <slug> ...flags — append a take (markdown + DB)
|
||||
* takes update <slug> --row N ...flags — update mutable fields
|
||||
@@ -129,11 +130,10 @@ function writeBody(path: string, body: string): void {
|
||||
// --- Subcommands ---
|
||||
|
||||
async function cmdList(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const slug = args[0];
|
||||
if (!slug) {
|
||||
console.error('Usage: gbrain takes <slug> [--json]');
|
||||
process.exit(1);
|
||||
}
|
||||
// #2079: slug is optional. `gbrain takes list` (no slug) lists ALL active
|
||||
// takes — CLI parity with the takes_list operation. A leading flag is not
|
||||
// a slug.
|
||||
const slug = args[0] && !args[0].startsWith('-') ? args[0] : undefined;
|
||||
const json = flagPresent(args, '--json');
|
||||
const holder = flagValue(args, '--who');
|
||||
const kind = flagValue(args, '--kind') as string | undefined;
|
||||
@@ -153,17 +153,19 @@ async function cmdList(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
return;
|
||||
}
|
||||
|
||||
const scope = slug ?? 'this brain';
|
||||
if (takes.length === 0) {
|
||||
console.log(`No takes on ${slug}.`);
|
||||
console.log(`No takes on ${scope}.`);
|
||||
return;
|
||||
}
|
||||
console.log(`# Takes on ${slug}\n`);
|
||||
console.log(`# Takes on ${scope}\n`);
|
||||
for (const t of takes) {
|
||||
const tag = t.active ? '' : ' [superseded]';
|
||||
const w = Number(t.weight).toFixed(2);
|
||||
const since = t.since_date ?? '';
|
||||
const src = t.source ? ` — ${t.source}` : '';
|
||||
console.log(`#${t.row_num} [${t.kind} • ${t.holder} • w=${w}${since ? ` • ${since}` : ''}]${tag}\n ${t.claim}${src}\n`);
|
||||
const where = slug ? '' : `${t.page_slug} `;
|
||||
console.log(`${where}#${t.row_num} [${t.kind} • ${t.holder} • w=${w}${since ? ` • ${since}` : ''}]${tag}\n ${t.claim}${src}\n`);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -555,6 +557,8 @@ export async function runTakes(engine: BrainEngine, args: string[]): Promise<voi
|
||||
Subcommands:
|
||||
takes <slug> [--json] [--who h] [--kind k] [--sort weight|since_date|created_at] [--expired]
|
||||
List takes for a page
|
||||
takes list [--json] [--who h] [--kind k] [--sort ...] [--expired]
|
||||
List all active takes across the brain (#2079)
|
||||
takes search "<query>" [--limit N] [--json]
|
||||
Keyword search across all takes
|
||||
takes add <slug> --claim "..." --kind <fact|take|bet|hunch> --who <holder>
|
||||
@@ -584,6 +588,9 @@ Common flags:
|
||||
const rest = args.slice(1);
|
||||
|
||||
switch (sub) {
|
||||
// #2079: `takes list` used to be parsed as page slug "list" and printed
|
||||
// "No takes on list." — reading exactly like an empty takes table.
|
||||
case 'list': return cmdList(engine, rest);
|
||||
case 'search': return cmdSearch(engine, rest);
|
||||
case 'add': return cmdAdd(engine, rest, await resolveTakesSourceId(engine));
|
||||
case 'update': return cmdUpdate(engine, rest, await resolveTakesSourceId(engine));
|
||||
|
||||
+63
-2
@@ -642,8 +642,42 @@ function warnRecipesMissingBatchTokens(): void {
|
||||
}
|
||||
}
|
||||
|
||||
/** Reset (for tests). */
|
||||
export function resetGateway(): void {
|
||||
/**
|
||||
* Test-only reset baseline (#3554). The bunfig preload
|
||||
* (`test/helpers/legacy-embedding-preload.ts`) pins the gateway to the legacy
|
||||
* OpenAI/1536 config at process start, but `resetGateway()` used to wipe that
|
||||
* pin to `_config = null`. The next test file's engine connect then
|
||||
* reconfigured from the SHIPPED default (zembed-1 @ 1280) and every 1536-d
|
||||
* fixture in that file exploded with `expected 1280 dimensions, not 1536` —
|
||||
* a cross-file mine whose placement depended on shard bin-packing.
|
||||
*
|
||||
* When a baseline factory is registered, `resetGateway()` means "back to the
|
||||
* test baseline" instead of "unconfigured": it clears everything as before,
|
||||
* then re-applies the factory's config via `configureGateway()`. A factory
|
||||
* (not a frozen config) so each re-application captures fresh
|
||||
* `process.env`, matching the preload's original `applyLegacy()` semantics.
|
||||
*
|
||||
* Production is untouched: nothing in `src/` calls `resetGateway()` or this
|
||||
* setter, so in production the baseline is never registered and
|
||||
* `resetGateway()` still fully unconfigures. Same `__*ForTests` seam
|
||||
* convention as `__setEmbedTransportForTests` above.
|
||||
*/
|
||||
let _resetBaseline: (() => AIGatewayConfig) | null = null;
|
||||
|
||||
/**
|
||||
* Register (or clear, with `null`) the config factory that `resetGateway()`
|
||||
* re-applies. Called once by the bunfig test preload.
|
||||
*
|
||||
* @internal exported for tests; not part of the public gateway API.
|
||||
*/
|
||||
export function __setGatewayResetBaselineForTests(
|
||||
factory: (() => AIGatewayConfig) | null,
|
||||
): void {
|
||||
_resetBaseline = factory;
|
||||
}
|
||||
|
||||
/** Clear every piece of module state. Shared by both reset flavors. */
|
||||
function clearGatewayState(): void {
|
||||
_config = null;
|
||||
_modelCache.clear();
|
||||
_shrinkState.clear();
|
||||
@@ -655,6 +689,33 @@ export function resetGateway(): void {
|
||||
_extendedModels.clear();
|
||||
}
|
||||
|
||||
/**
|
||||
* Reset (for tests). Clears all module state (config, model cache, shrink
|
||||
* state, transports, warned recipes, extended models), then — if a test
|
||||
* baseline is registered — re-applies it so the gateway returns to the
|
||||
* process-wide test default instead of an unconfigured limbo (#3554).
|
||||
*/
|
||||
export function resetGateway(): void {
|
||||
clearGatewayState();
|
||||
// configureGateway re-clears _modelCache/_shrinkState/_extendedModels and
|
||||
// registers the baseline's models; transports are NOT touched by it, so a
|
||||
// stale test transport can never leak back in through this path.
|
||||
if (_resetBaseline) configureGateway(_resetBaseline());
|
||||
}
|
||||
|
||||
/**
|
||||
* Reset AND stay unconfigured, ignoring any registered baseline. For the
|
||||
* handful of tests that assert genuine no-gateway behavior
|
||||
* (`no_gateway_config` diagnosis, `isAvailable() === false`, graceful
|
||||
* degradation paths). The preload's per-test beforeEach restores the
|
||||
* baseline before the next test, so this cannot leak across tests.
|
||||
*
|
||||
* @internal exported for tests; not part of the public gateway API.
|
||||
*/
|
||||
export function __unconfigureGatewayForTests(): void {
|
||||
clearGatewayState();
|
||||
}
|
||||
|
||||
/**
|
||||
* Test-only seam. Replaces the function the gateway calls to embed a
|
||||
* sub-batch. Pass `null` to restore the real `embedMany` from the AI SDK.
|
||||
|
||||
@@ -31,6 +31,11 @@ export const dashscope: Recipe = {
|
||||
// path. Conservative declaration so the gateway pre-splits before
|
||||
// hitting whatever undocumented server-side limit exists.
|
||||
max_batch_tokens: 8192,
|
||||
// DashScope's OpenAI-compat /embeddings endpoint rejects requests with
|
||||
// more than 10 input items (documented Model Studio cap). The gateway's
|
||||
// capBatchItems pre-split enforces this; max_batch_tokens above keeps
|
||||
// guarding aggregate token size. Concept from community PRs #2643/#2405.
|
||||
max_batch_items: 10,
|
||||
// text-embedding-v3 mixes English + CJK heavily; the tokenizer is
|
||||
// closer to Voyage density than OpenAI tiktoken for CJK-dominant
|
||||
// content. Conservative chars_per_token=2 leaves headroom.
|
||||
|
||||
@@ -180,6 +180,17 @@ export const openrouter: Recipe = {
|
||||
// to pre-split batches, NOT per-input. Per-input is enforced upstream.
|
||||
max_batch_tokens: 300_000,
|
||||
},
|
||||
// Expansion uses the same routed OpenAI-compatible language-model endpoint
|
||||
// as chat. Keep a small cheap/fast advisory set; the openai-compat tier
|
||||
// still accepts any user-configured OpenRouter provider/model ID.
|
||||
expansion: {
|
||||
models: [
|
||||
'anthropic/claude-haiku-4.5',
|
||||
'google/gemini-3-flash-preview',
|
||||
'deepseek/deepseek-chat',
|
||||
],
|
||||
price_last_verified: '2026-05-20',
|
||||
},
|
||||
chat: {
|
||||
// Curated entry points (verified against OR's catalog 2026-05-20). The
|
||||
// openai-compat tier does NOT enforce this list at runtime — users can
|
||||
|
||||
+125
-11
@@ -40,6 +40,7 @@ export const LINKABLE_ENTITY_TYPES = ['person', 'company', 'organization', 'enti
|
||||
* types in.
|
||||
*/
|
||||
const MIN_NAME_LENGTH = 4;
|
||||
const MIN_CJK_NAME_LENGTH = 2;
|
||||
|
||||
/**
|
||||
* Built-in ignore list — common ambiguous tokens whose body-text mentions
|
||||
@@ -104,12 +105,12 @@ export interface FindMentionsOpts {
|
||||
// ============================================================
|
||||
|
||||
/**
|
||||
* Token-only tokenizer. Returns `[token, offset]` pairs for every
|
||||
* `[a-zA-Z0-9]+` run, lowercased. Non-ASCII (CJK, accented) is
|
||||
* deliberately not tokenized in v1 — entity gazetteer is English-dominant
|
||||
* in production today. Widening to `\p{L}+` is a future option once a
|
||||
* real CJK entity catalog appears (filed under TODO-1 + a TODO for
|
||||
* Unicode-aware tokenization).
|
||||
* Token-only tokenizer. Returns `[token, offset]` pairs.
|
||||
*
|
||||
* ASCII: each `[a-zA-Z0-9]+` run is a single token, lowercased.
|
||||
* CJK: each CJK character (Chinese/Japanese/Korean) is an individual
|
||||
* token, lowercased. This allows the normal maximal-munch scan path
|
||||
* to reach CJK gazetteer entries without a separate substring pass.
|
||||
*
|
||||
* Possessive "Acme's" tokenizes as ['acme', 's'] (single-quote breaks the
|
||||
* run) — single-word "Acme" lookup succeeds at offset 0; the trailing 's'
|
||||
@@ -127,18 +128,129 @@ function tokenizeForScan(text: string): ScannedToken[] {
|
||||
const out: ScannedToken[] = [];
|
||||
TOKEN_RE.lastIndex = 0;
|
||||
let m: RegExpExecArray | null;
|
||||
|
||||
// Collect ASCII token spans first.
|
||||
const asciiSpans: Array<{ start: number; end: number }> = [];
|
||||
while ((m = TOKEN_RE.exec(text)) !== null) {
|
||||
out.push({ text: m[0].toLowerCase(), offset: m.index, length: m[0].length });
|
||||
asciiSpans.push({ start: m.index, end: m.index + m[0].length });
|
||||
}
|
||||
|
||||
// Walk character-by-character: emit ASCII tokens at their start positions,
|
||||
// then emit individual CJK characters for non-ASCII positions that fall
|
||||
// outside ASCII token spans.
|
||||
let asciiIdx = 0;
|
||||
for (let i = 0; i < text.length;) {
|
||||
const cp = text.codePointAt(i) ?? 0;
|
||||
const isCJK = (cp >= 0x4e00 && cp <= 0x9fff) || (cp >= 0x3400 && cp <= 0x4dbf) ||
|
||||
(cp >= 0x3040 && cp <= 0x309f) || (cp >= 0x30a0 && cp <= 0x30ff) ||
|
||||
(cp >= 0xac00 && cp <= 0xd7af);
|
||||
|
||||
// Advance asciiIdx past any spans that end before or at i.
|
||||
while (asciiIdx < asciiSpans.length && asciiSpans[asciiIdx]!.end <= i) {
|
||||
asciiIdx++;
|
||||
}
|
||||
|
||||
// If position i is inside an ASCII token span, emit the full ASCII token
|
||||
// and jump past it.
|
||||
if (asciiIdx < asciiSpans.length && i >= asciiSpans[asciiIdx]!.start && i < asciiSpans[asciiIdx]!.end) {
|
||||
const span = asciiSpans[asciiIdx]!;
|
||||
const token = text.slice(span.start, span.end);
|
||||
out.push({ text: token.toLowerCase(), offset: span.start, length: token.length });
|
||||
i = span.end;
|
||||
asciiIdx++;
|
||||
continue;
|
||||
}
|
||||
|
||||
// CJK: emit as individual character token.
|
||||
if (isCJK) {
|
||||
const charLen = cp > 0xffff ? 2 : 1; // surrogate pair
|
||||
const charStr = text.slice(i, i + charLen);
|
||||
out.push({ text: charStr.toLowerCase(), offset: i, length: charLen });
|
||||
i += charLen;
|
||||
} else {
|
||||
i++;
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
function hasCJK(s: string): boolean {
|
||||
for (const ch of s) {
|
||||
const cp = ch.codePointAt(0) ?? 0;
|
||||
if ((cp >= 0x4e00 && cp <= 0x9fff) || (cp >= 0x3400 && cp <= 0x4dbf) ||
|
||||
(cp >= 0x3040 && cp <= 0x309f) || (cp >= 0x30a0 && cp <= 0x30ff) ||
|
||||
(cp >= 0xac00 && cp <= 0xd7af)) return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
function cjkCharCount(s: string): number {
|
||||
let count = 0;
|
||||
for (const ch of s) {
|
||||
const cp = ch.codePointAt(0) ?? 0;
|
||||
if ((cp >= 0x4e00 && cp <= 0x9fff) || (cp >= 0x3400 && cp <= 0x4dbf) ||
|
||||
(cp >= 0x3040 && cp <= 0x309f) || (cp >= 0x30a0 && cp <= 0x30ff) ||
|
||||
(cp >= 0xac00 && cp <= 0xd7af)) count++;
|
||||
}
|
||||
return count;
|
||||
}
|
||||
|
||||
/**
|
||||
* Tokenize a page title for gazetteer insertion.
|
||||
*
|
||||
* ASCII titles: standard `[a-zA-Z0-9]+` tokenization, lowercased.
|
||||
* CJK titles (no ASCII content): split into individual characters —
|
||||
* e.g. "纳瓦尔" → ["纳","瓦","尔"]. This allows normal multi-token
|
||||
* maximal-munch matching to work with character-level CJK tokens
|
||||
* produced by `tokenizeForScan`.
|
||||
* Mixed CJK+ASCII titles: ASCII parts tokenized normally, CJK parts
|
||||
* split into individual characters.
|
||||
*/
|
||||
function tokenizeTitle(title: string): string[] {
|
||||
const tokens: string[] = [];
|
||||
TOKEN_RE.lastIndex = 0;
|
||||
let m: RegExpExecArray | null;
|
||||
while ((m = TOKEN_RE.exec(title)) !== null) tokens.push(m[0].toLowerCase());
|
||||
return tokens;
|
||||
const hasAscii = TOKEN_RE.test(title);
|
||||
if (hasAscii) {
|
||||
// Mixed ASCII+CJK or pure ASCII: tokenize ASCII normally, then
|
||||
// append individual CJK characters in order.
|
||||
TOKEN_RE.lastIndex = 0;
|
||||
let m: RegExpExecArray | null;
|
||||
const asciiSpans: Array<{ start: number; end: number; text: string }> = [];
|
||||
while ((m = TOKEN_RE.exec(title)) !== null) {
|
||||
asciiSpans.push({ start: m.index, end: m.index + m[0].length, text: m[0].toLowerCase() });
|
||||
}
|
||||
let asciiIdx = 0;
|
||||
for (let i = 0; i < title.length;) {
|
||||
while (asciiIdx < asciiSpans.length && asciiSpans[asciiIdx]!.end <= i) asciiIdx++;
|
||||
if (asciiIdx < asciiSpans.length && i >= asciiSpans[asciiIdx]!.start && i < asciiSpans[asciiIdx]!.end) {
|
||||
tokens.push(asciiSpans[asciiIdx]!.text);
|
||||
i = asciiSpans[asciiIdx]!.end;
|
||||
asciiIdx++;
|
||||
continue;
|
||||
}
|
||||
const cp = title.codePointAt(i) ?? 0;
|
||||
if (hasCJK(title[i]!)) {
|
||||
const charLen = cp > 0xffff ? 2 : 1;
|
||||
tokens.push(title.slice(i, i + charLen).toLowerCase());
|
||||
i += charLen;
|
||||
} else {
|
||||
i++;
|
||||
}
|
||||
}
|
||||
return tokens;
|
||||
}
|
||||
// Pure CJK (no ASCII content): split into individual characters.
|
||||
if (hasCJK(title)) {
|
||||
for (let i = 0; i < title.length;) {
|
||||
const cp = title.codePointAt(i) ?? 0;
|
||||
const charLen = cp > 0xffff ? 2 : 1;
|
||||
tokens.push(title.slice(i, i + charLen).toLowerCase());
|
||||
i += charLen;
|
||||
}
|
||||
return tokens;
|
||||
}
|
||||
// Non-ASCII, non-CJK title (emoji, symbols, etc.) — empty set.
|
||||
return [];
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -175,7 +287,9 @@ export async function buildGazetteer(
|
||||
|
||||
const gazetteer: Gazetteer = new Map();
|
||||
for (const row of rows) {
|
||||
if (!row.title || row.title.length < MIN_NAME_LENGTH) continue;
|
||||
if (!row.title) continue;
|
||||
if (!hasCJK(row.title) && row.title.length < MIN_NAME_LENGTH) continue;
|
||||
if (hasCJK(row.title) && cjkCharCount(row.title) < MIN_CJK_NAME_LENGTH) continue;
|
||||
if (ignoreSet.has(row.title) && !existingTitles.has(row.title)) continue;
|
||||
|
||||
const tokens = tokenizeTitle(row.title);
|
||||
|
||||
@@ -20,6 +20,7 @@
|
||||
|
||||
import { chunkText as recursiveChunk } from './recursive.ts';
|
||||
import { buildQualifiedName } from './qualified-names.ts';
|
||||
import { CJK_SLUG_CHARS, CJK_RANGES_REGEX } from '../cjk.ts';
|
||||
|
||||
// Embed the tree-sitter runtime + per-language grammars as files.
|
||||
// `with { type: 'file' }` returns a path (string) at runtime. Bun bundles
|
||||
@@ -716,7 +717,7 @@ export async function chunkCodeTextFull(
|
||||
}
|
||||
|
||||
if (chunks.length === 0) {
|
||||
return { chunks: capOversizedChunks(fallbackChunks(source, filePath, language, opts), filePath, language, opts), edges: rawEdges };
|
||||
return { chunks: fallbackChunks(source, filePath, language, opts), edges: rawEdges };
|
||||
}
|
||||
return { chunks: capOversizedChunks(mergeSmallSiblings(chunks, chunkTarget), filePath, language, opts), edges: rawEdges };
|
||||
} catch {
|
||||
@@ -842,10 +843,10 @@ function capOversizedChunks(
|
||||
opts: CodeChunkOptions,
|
||||
): CodeChunk[] {
|
||||
const cap = opts.maxChunkTokens ?? DEFAULT_MAX_CHUNK_TOKENS;
|
||||
if (!chunks.some((c) => estimateTokens(c.text) > cap)) return chunks;
|
||||
if (!chunks.some((c) => estimateEmbedTokens(c.text) > cap)) return chunks;
|
||||
const out: CodeChunk[] = [];
|
||||
for (const c of chunks) {
|
||||
if (estimateTokens(c.text) <= cap) {
|
||||
if (estimateEmbedTokens(c.text) <= cap) {
|
||||
out.push({ ...c, index: out.length });
|
||||
continue;
|
||||
}
|
||||
@@ -880,17 +881,43 @@ function splitToTokenBudget(text: string, cap: number, opts: CodeChunkOptions):
|
||||
chunkOverlap: opts.fallbackOverlapWords ?? 50,
|
||||
}).map((p) => p.text);
|
||||
for (const piece of pieces) {
|
||||
if (estimateTokens(piece) <= cap) {
|
||||
if (estimateEmbedTokens(piece) <= cap) {
|
||||
out.push(piece);
|
||||
continue;
|
||||
}
|
||||
// ~3.5 chars/token is a conservative cl100k estimate for source text.
|
||||
const charBudget = Math.max(1, Math.floor(cap * 3.5));
|
||||
// Hard-split slice size. Pure-ASCII pieces: ~3.5 chars/token is a
|
||||
// conservative cl100k estimate for source text. CJK-containing pieces:
|
||||
// the weighted estimate can reach 1 token/char, so budget 1 char/token
|
||||
// to keep every slice under cap by construction.
|
||||
const charBudget = Math.max(1, Math.floor(cap * (CJK_RANGES_REGEX.test(piece) ? 1 : 3.5)));
|
||||
for (let i = 0; i < piece.length; i += charBudget) out.push(piece.slice(i, i + charBudget));
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
const CJK_CHARS_G = new RegExp(`[${CJK_SLUG_CHARS}]`, 'g');
|
||||
|
||||
/**
|
||||
* Embedding-safe token estimate for the oversize cap. estimateTokens
|
||||
* (cl100k) matches embedding-family tokenizers closely on pure-ASCII source
|
||||
* (measured identical on English prose and JSON vs Qwen3-Embedding), but
|
||||
* UNDERCOUNTS mixed CJK+ASCII chunks — measured −31% on URL-dense Korean
|
||||
* text vs the Qwen3 embedding tokenizer, which is exactly the shape that
|
||||
* overflows strict embedding backends (#2826). For chunks containing CJK,
|
||||
* take the max of cl100k and a per-char-class overestimate (CJK 1.0/char,
|
||||
* other non-whitespace 0.75/char, whitespace 0.1/char). CJK-DOMINANT text
|
||||
* is unaffected too: cl100k already counts it above the weighted form, so
|
||||
* max() returns the same value as today. Only mixed-script chunks — the
|
||||
* measured divergence class — estimate higher.
|
||||
*/
|
||||
export function estimateEmbedTokens(text: string): number {
|
||||
const cjk = (text.match(CJK_CHARS_G) || []).length;
|
||||
if (cjk === 0) return estimateTokens(text);
|
||||
const ws = (text.match(/\s/g) || []).length;
|
||||
const weighted = Math.ceil(cjk + (text.length - cjk - ws) * 0.75 + ws * 0.1);
|
||||
return Math.max(estimateTokens(text), weighted);
|
||||
}
|
||||
|
||||
// ---------- Internals ----------
|
||||
|
||||
function fallbackChunks(
|
||||
@@ -901,7 +928,7 @@ function fallbackChunks(
|
||||
): CodeChunk[] {
|
||||
const size = opts.fallbackChunkSizeWords ?? 300;
|
||||
const overlap = opts.fallbackOverlapWords ?? 50;
|
||||
return recursiveChunk(source, { chunkSize: size, chunkOverlap: overlap }).map((chunk, index) =>
|
||||
const chunks = recursiveChunk(source, { chunkSize: size, chunkOverlap: overlap }).map((chunk, index) =>
|
||||
buildChunk({
|
||||
body: chunk.text, filePath, language,
|
||||
symbolName: null, symbolType: 'module',
|
||||
@@ -909,6 +936,14 @@ function fallbackChunks(
|
||||
index,
|
||||
}),
|
||||
);
|
||||
// Route every fallback emission through the oversize net. Previously only
|
||||
// the empty-AST branch wrapped its fallback in capOversizedChunks — the
|
||||
// no-language, parse-timeout, no-semantic-nodes (every JSON/YAML fence:
|
||||
// their node types aren't in TOP_LEVEL_TYPES) and parse-throw branches
|
||||
// shipped word-counted chunks unchecked, and the word pipeline undercounts
|
||||
// exactly the dense content (JSON, minified, CJK-mixed) that overflows
|
||||
// embedders. Hoisting the cap here covers all five paths at once.
|
||||
return capOversizedChunks(chunks, filePath, language, opts);
|
||||
}
|
||||
|
||||
function buildChunk(input: {
|
||||
|
||||
+28
-5
@@ -21,12 +21,35 @@ export const CJK_SLUG_CHARS = '一-鿿-ゟ゠-ヿ가-';
|
||||
export const CJK_RANGES_REGEX = new RegExp(`[${CJK_SLUG_CHARS}]`);
|
||||
|
||||
/**
|
||||
* Page-slug segment grammar (no anchors): alnum-or-CJK lead char, then
|
||||
* alnum/CJK/hyphen continuation. Single source for validatePageSlug
|
||||
* (operations.ts), SlugRegistry's SLUG_RE, and the dream-cycle
|
||||
* SUMMARY_SLUG_RE so every slug validator shares one grammar (#738).
|
||||
* Slug "word" character class (#3417): every script's letters, not just
|
||||
* Latin + CJK. Unicode property escapes — REQUIRES the `u` flag on any
|
||||
* regex composed from this string (without `u`, `\p{Ll}` silently matches
|
||||
* the literal chars `p`, `L`, `l`, `{`, `}`).
|
||||
*
|
||||
* \p{Ll} lowercase letters (a-z, Cyrillic/Greek lowercase, đ, …)
|
||||
* \p{Lm} modifier letters
|
||||
* \p{Lo} caseless-script letters (Hebrew, Arabic, Thai, CJK, Devanagari, …)
|
||||
* \p{M} combining marks that survive the Latin accent-strip pass
|
||||
* (Hebrew niqqud, Arabic harakat, Thai/Devanagari vowel signs)
|
||||
* \p{N} numbers (0-9, Arabic-Indic digits, …)
|
||||
*
|
||||
* Uppercase (\p{Lu}/\p{Lt}) is deliberately excluded: slugifySegment()
|
||||
* lowercases before filtering, so validators stay lowercase-canonical.
|
||||
*
|
||||
* Distinct from CJK_SLUG_CHARS above, which also drives the
|
||||
* countCJKAwareWords density heuristic — do NOT merge the two, or slug
|
||||
* grammar changes silently change chunking behavior.
|
||||
*/
|
||||
export const PAGE_SLUG_SEG = `[a-z0-9${CJK_SLUG_CHARS}][a-z0-9${CJK_SLUG_CHARS}\\-]*`;
|
||||
export const SLUG_WORD_CHARS = '\\p{Ll}\\p{Lm}\\p{Lo}\\p{M}\\p{N}';
|
||||
|
||||
/**
|
||||
* Page-slug segment grammar (no anchors): word-char lead, then word-char or
|
||||
* hyphen continuation. Single source for validatePageSlug (operations.ts),
|
||||
* SlugRegistry's SLUG_RE, and the dream-cycle SUMMARY_SLUG_RE so every slug
|
||||
* validator shares one grammar (#738). Compose with the `u` flag — see
|
||||
* SLUG_WORD_CHARS.
|
||||
*/
|
||||
export const PAGE_SLUG_SEG = `[${SLUG_WORD_CHARS}][${SLUG_WORD_CHARS}\\-]*`;
|
||||
|
||||
export const CJK_SENTENCE_DELIMITERS = ['。', '!', '?']; // 。!?
|
||||
export const CJK_CLAUSE_DELIMITERS = [';', ':', ',', '、']; // ;:,、
|
||||
|
||||
@@ -145,6 +145,19 @@ export function computeCorpusGeneration(args: {
|
||||
return h.digest('hex').slice(0, 16);
|
||||
}
|
||||
|
||||
/**
|
||||
* #3507 — the corpus_generation a page lands on when a plain re-embed path
|
||||
* (`embed --stale` and friends) re-embeds a `per_chunk_synopsis` page at the
|
||||
* title-only tier (the D14 fallback tier; synopsis re-generation is a paid
|
||||
* backfill concern). Callers restamp
|
||||
* `updatePageContextualRetrievalState(slug, sourceId, 'title', titleTierCorpusGeneration())`
|
||||
* so the stamped mode keeps describing the vectors actually in the column.
|
||||
* Matches what the inline import path writes for its title-tier pages.
|
||||
*/
|
||||
export function titleTierCorpusGeneration(): string {
|
||||
return computeCorpusGeneration({ crMode: 'title', haikuModel: DEFAULT_HAIKU_MODEL });
|
||||
}
|
||||
|
||||
/**
|
||||
* Compute source_text_hash for D27 P1-4 cache key composition. The
|
||||
* synopsis cache invalidates correctly when adjacent text changes (page
|
||||
|
||||
@@ -0,0 +1,94 @@
|
||||
/**
|
||||
* Block-format conversation normalizer.
|
||||
*
|
||||
* Some chat exports — notably the Slack collector gbrain's own ingestion
|
||||
* uses — emit a HEADER + indented-body BLOCK per message instead of the
|
||||
* single-line `**Name** (time): body` shape the built-in patterns
|
||||
* (`builtins.ts`) recognize:
|
||||
*
|
||||
* - **Theo** (Mon 11:18)
|
||||
* Hey everyone — quick update on the renewal.
|
||||
*
|
||||
* Second paragraph of the same message.
|
||||
* - **Juan** (Mon 11:20)
|
||||
* Reply body...
|
||||
*
|
||||
* None of the 14 line-oriented built-ins match this: a leading `- ` list
|
||||
* marker, a day-of-week + time with no trailing colon, and the message body on
|
||||
* the following indented lines. Result: `phase: 'no_match'`, zero messages,
|
||||
* and the whole comms corpus is silently un-extractable (facts stay empty →
|
||||
* `find_trajectory` returns nothing).
|
||||
*
|
||||
* This collapses each block into the canonical `**Name** (HH:MM): <body joined
|
||||
* to one line>` shape so the existing `bold-paren-time` pattern matches; the
|
||||
* per-message date fills in downstream via `fallbackDate` (the page date).
|
||||
*
|
||||
* STRICT no-op unless the block signature is present: the header regex requires
|
||||
* the paren-group to END the line (no inline `: body`), which is exactly what
|
||||
* the single-line patterns always produce — so feeding already-canonical
|
||||
* content through this function returns it unchanged.
|
||||
*/
|
||||
|
||||
// `- **Name** (Mon 11:18)` / `- **Name** (11:18 AM)` / `- **Name** (16:36)`.
|
||||
// Day-of-week optional; 12h/24h time; optional am/pm; the line ENDS at the
|
||||
// close paren (no inline `: body` — that is what distinguishes a block header
|
||||
// from the single-line `**Name** (time): body` patterns).
|
||||
const BLOCK_HEADER =
|
||||
/^\s*-\s+\*\*(.+?)\*\*\s+\((?:[A-Za-z]{2,9}\.?\s+)?(\d{1,2}):(\d{2})(?::\d{2})?\s*([AaPp][Mm])?\)\s*$/;
|
||||
|
||||
/** True when at least one line is a block-format message header. */
|
||||
export function looksLikeBlockConversation(body: string): boolean {
|
||||
for (const line of body.split('\n')) {
|
||||
if (BLOCK_HEADER.test(line)) return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
function to24h(hour: number, ampm?: string): number {
|
||||
if (!ampm) return hour;
|
||||
const lower = ampm.toLowerCase();
|
||||
if (lower === 'pm' && hour < 12) return hour + 12;
|
||||
if (lower === 'am' && hour === 12) return 0;
|
||||
return hour;
|
||||
}
|
||||
|
||||
/**
|
||||
* Collapse block-format messages into canonical single-line `**Name** (HH:MM):
|
||||
* body` lines. Returns `body` unchanged when no block header is present.
|
||||
*/
|
||||
export function normalizeBlockConversation(body: string): string {
|
||||
if (!looksLikeBlockConversation(body)) return body;
|
||||
|
||||
const lines = body.split('\n');
|
||||
const out: string[] = [];
|
||||
let current: { name: string; time: string } | null = null;
|
||||
let bodyParts: string[] = [];
|
||||
|
||||
const flush = () => {
|
||||
if (current) {
|
||||
const text = bodyParts.join(' ').replace(/\s+/g, ' ').trim();
|
||||
out.push(`**${current.name}** (${current.time}): ${text}`);
|
||||
}
|
||||
current = null;
|
||||
bodyParts = [];
|
||||
};
|
||||
|
||||
for (const line of lines) {
|
||||
const m = BLOCK_HEADER.exec(line);
|
||||
if (m) {
|
||||
flush();
|
||||
const hour = to24h(parseInt(m[2], 10), m[4]);
|
||||
const time = `${String(hour).padStart(2, '0')}:${m[3]}`;
|
||||
current = { name: m[1].trim(), time };
|
||||
} else if (current) {
|
||||
// Body line of the current message. Drop blank lines; keep the rest.
|
||||
const trimmed = line.trim();
|
||||
if (trimmed) bodyParts.push(trimmed);
|
||||
}
|
||||
// Lines before the first header (page title, leading blanks) are dropped —
|
||||
// they never matched a pattern anyway.
|
||||
}
|
||||
flush();
|
||||
|
||||
return out.length > 0 ? out.join('\n') : body;
|
||||
}
|
||||
@@ -29,6 +29,7 @@ import {
|
||||
BUILTIN_PATTERNS,
|
||||
cleanSpeaker,
|
||||
} from './builtins.ts';
|
||||
import { normalizeBlockConversation } from './normalize-block.ts';
|
||||
import type {
|
||||
DateContext,
|
||||
MatchedMessage,
|
||||
@@ -473,6 +474,12 @@ export function parseConversation(
|
||||
return { messages: [], phase: 'no_match' };
|
||||
}
|
||||
|
||||
// Pre-pass: collapse block-format chat exports (header + indented body, e.g.
|
||||
// the Slack collector's `- **Name** (Mon 11:18)\n body…`) into the canonical
|
||||
// single-line shape the built-in patterns recognize. Strict no-op when no
|
||||
// block header is present, so already-canonical content is untouched.
|
||||
body = normalizeBlockConversation(body);
|
||||
|
||||
const dateCtx = deriveDateContext(opts);
|
||||
|
||||
// Assemble candidate pool: built-ins (minus disabled) + user patterns.
|
||||
|
||||
+3
-1
@@ -1179,7 +1179,9 @@ async function runPhaseExtractFacts(
|
||||
summary: `extract_facts skipped: ${result.legacyRowsPending} legacy v0.31 facts pending fence backfill`,
|
||||
details: {
|
||||
legacyRowsPending: result.legacyRowsPending,
|
||||
hint: 'gbrain apply-migrations --yes',
|
||||
// A bare `apply-migrations --yes` no-ops once the v0.32.2 ledger
|
||||
// entry is complete; the retry marker is what re-runs Phase B.
|
||||
hint: 'gbrain apply-migrations --force-retry 0.32.2 && gbrain apply-migrations --yes',
|
||||
warnings: result.warnings,
|
||||
},
|
||||
};
|
||||
|
||||
@@ -17,20 +17,26 @@
|
||||
* DB rows need cleanup (#1781 — the unconditional wipe-and-reinsert
|
||||
* made every cycle non-idempotent, re-appending duplicate rows).
|
||||
*
|
||||
* After the phase, the DB index for every affected page matches the
|
||||
* fence's canonical (claim, source) row set (modulo embeddings +
|
||||
* runtime-derived fields). Pages with no fence wipe DB rows for that
|
||||
* page coordinate only; legacy NULL-source_markdown_slug rows survive
|
||||
* because deleteFactsForPage targets source_markdown_slug = slug only.
|
||||
* After the phase, the DB index for every cleanly parsed affected page
|
||||
* matches the fence's canonical (claim, source) row set (modulo embeddings
|
||||
* + runtime-derived fields). Warning-bearing parses are non-authoritative
|
||||
* and preserve that page's existing index. Pages with no fence wipe DB rows
|
||||
* for that page coordinate only; legacy NULL-source_markdown_slug rows
|
||||
* survive because deleteFactsForPage targets source_markdown_slug = slug only.
|
||||
*
|
||||
* Empty-fence guard (Codex R2-#7; #2484): the phase refuses to do its
|
||||
* destructive reconciliation pass when genuinely-backfillable legacy
|
||||
* rows still exist — `row_num IS NULL` (never fenced) AND `entity_slug`
|
||||
* resolves to a live page in this source (so the v0_32_2 migration's
|
||||
* Phase B could fence them). Status returns `warn` with a hint to run
|
||||
* `gbrain apply-migrations --yes`. Without the guard, an interrupted
|
||||
* upgrade where v0_32_2 hasn't run could leave the cycle silently
|
||||
* misreporting "0 facts on people/alice" while legacy rows linger.
|
||||
* Empty-fence guard (Codex R2-#7; #2484; #2646): the phase refuses to do
|
||||
* its destructive reconciliation pass when genuinely-backfillable legacy
|
||||
* rows still exist — in THIS run's source only (`source_id = sourceId`;
|
||||
* a pending row in source A must not jam extraction for source B — the
|
||||
* source-isolation invariant) — `row_num IS NULL` (never fenced) AND
|
||||
* `entity_slug` resolves to a live page in this source (so the v0_32_2
|
||||
* migration's Phase B could fence them) AND the row is not soft-expired
|
||||
* (`expired_at IS NULL`). Status returns `warn` with a hint to re-run
|
||||
* the v0.32.2 fence backfill (`apply-migrations --force-retry 0.32.2`
|
||||
* then `--yes` — a bare `--yes` is a no-op once the ledger says
|
||||
* complete). Without the guard, an interrupted upgrade where v0_32_2
|
||||
* hasn't run could leave the cycle silently misreporting "0 facts on
|
||||
* people/alice" while legacy rows linger.
|
||||
*
|
||||
* The live-page requirement (#2484) is load-bearing: the inline facts
|
||||
* writer keeps producing `row_num IS NULL, entity_slug IS NOT NULL`
|
||||
@@ -41,6 +47,10 @@
|
||||
* the phase jams forever (~16/day observed). Requiring a backing page
|
||||
* keeps genuine pre-v0.32.2 rows (whose entity page exists) gating
|
||||
* while excluding the inline-writer's permanent-unfenceable rows.
|
||||
*
|
||||
* Soft-expired rows don't count either (#2646): they're what
|
||||
* `forget_fact` produces, so excluding them lets operators drain the
|
||||
* backlog through the sanctioned removal path instead of raw SQL.
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from '../engine.ts';
|
||||
@@ -88,6 +98,24 @@ function dedupeFactsByContentKey(facts: FenceExtractedFact[]): FenceExtractedFac
|
||||
* neither count as "stale" (which would force a wipe every cycle) nor
|
||||
* be compared against the fence's row set. Mirrors the
|
||||
* excludeSourcePrefixes filter deleteFactsForPage applies on the wipe.
|
||||
*
|
||||
* Also excludes soft-expired legacy rows (#2646: `row_num IS NULL AND
|
||||
* expired_at IS NOT NULL`) — rows that `forget_fact` expired via its
|
||||
* legacy DB-only path. They are not fence-owned (fence rows always
|
||||
* carry a row_num), so they must neither count as "stale" (forcing a
|
||||
* wipe every cycle) nor mask a fence row from insertion. Mirrors the
|
||||
* preserveExpiredLegacy filter deleteFactsForPage applies on the wipe.
|
||||
*
|
||||
* Deliberate consequence: if the fence still carries the same
|
||||
* (claim, source) as an expired legacy row, the reconcile inserts it
|
||||
* as a fresh ACTIVE fence-owned row. That is the fence-is-canonical
|
||||
* contract working as documented — legacy DB-only forgets "DO NOT
|
||||
* survive rebuild" (see forget.ts header); suppressing the insert
|
||||
* would instead create silent fence↔DB divergence, the exact failure
|
||||
* mode the empty-fence guard exists to prevent. To durably forget
|
||||
* such a claim, forget the fence-owned row (forget_fact now takes the
|
||||
* fence path, which strikes the row through in markdown). The expired
|
||||
* legacy row survives alongside as the record of the earlier forget.
|
||||
*/
|
||||
async function listExistingFactsForPage(
|
||||
engine: BrainEngine,
|
||||
@@ -100,6 +128,7 @@ async function listExistingFactsForPage(
|
||||
WHERE source_id = $1
|
||||
AND source_markdown_slug = $2
|
||||
AND COALESCE(source, '') NOT LIKE 'cli:%'
|
||||
AND NOT (row_num IS NULL AND expired_at IS NOT NULL)
|
||||
ORDER BY row_num ASC, id ASC`,
|
||||
[sourceId, slug],
|
||||
);
|
||||
@@ -173,7 +202,7 @@ export async function runExtractFacts(
|
||||
phantomsMorePending: false,
|
||||
};
|
||||
|
||||
// ── Empty-fence guard (Codex R2-#7; #2484) ─────────────────────
|
||||
// ── Empty-fence guard (Codex R2-#7; #2484; #2646) ──────────────
|
||||
// Pre-check: if any genuinely-backfillable legacy fact rows exist,
|
||||
// refuse to run the destructive reconciliation pass — the v0_32_2
|
||||
// orchestrator must fence them first.
|
||||
@@ -181,12 +210,13 @@ export async function runExtractFacts(
|
||||
// A row is a real backfill candidate only when `row_num IS NULL`
|
||||
// (never fenced) AND its `entity_slug` resolves to a LIVE page in
|
||||
// this source (the migration's Phase B only fences rows whose
|
||||
// entity_slug maps to a writable page). #2484: the original
|
||||
// predicate was just `row_num IS NULL AND entity_slug IS NOT NULL`,
|
||||
// which ALSO matched structurally-unfenceable hot-memory rows the
|
||||
// inline writer keeps producing post-migration: the legacy DB-only
|
||||
// fallback (backstop.ts) writes `entity_slug` (a resolved slug, e.g.
|
||||
// a slugify-floor or stub-guard-blocked unprefixed slug like
|
||||
// entity_slug maps to a writable page) AND it is not soft-expired.
|
||||
// #2484: the original predicate was just `row_num IS NULL AND
|
||||
// entity_slug IS NOT NULL`, which ALSO matched
|
||||
// structurally-unfenceable hot-memory rows the inline writer keeps
|
||||
// producing post-migration: the legacy DB-only fallback
|
||||
// (backstop.ts) writes `entity_slug` (a resolved slug, e.g. a
|
||||
// slugify-floor or stub-guard-blocked unprefixed slug like
|
||||
// `people-jane-doe`) with `row_num` NULL whenever the slug has no
|
||||
// fenceable page. Those rows can never satisfy the migration's exit
|
||||
// condition (no page to fence onto, and `apply-migrations` is a
|
||||
@@ -194,27 +224,49 @@ export async function runExtractFacts(
|
||||
// — ~16/day, mislabeled "v0.31 pending backfill." We now require a
|
||||
// live backing page, which both genuine pre-v0.32.2 rows (their
|
||||
// entity page exists) satisfy and inline-writer unfenceable rows do
|
||||
// not.
|
||||
// not. #2646: soft-expired rows (`expired_at IS NOT NULL`) are also
|
||||
// excluded — `forget_fact`, the officially sanctioned removal path,
|
||||
// soft-expires legacy rows rather than deleting them, so counting
|
||||
// expired rows would leave the guard permanently stuck with no
|
||||
// supported way to drain the backlog.
|
||||
//
|
||||
// Source isolation (#3526): the count is scoped to THIS run's
|
||||
// sourceId. The pre-fix query counted brain-wide, so a single pending
|
||||
// legacy row in any mounted source jammed extract_facts for every
|
||||
// source — a cross-source leak of one source's migration state into
|
||||
// another's cycle (CLAUDE.md source-isolation invariant).
|
||||
const legacy = await engine.executeRaw<{ n: string }>(
|
||||
`SELECT COUNT(*) AS n
|
||||
FROM facts f
|
||||
WHERE f.row_num IS NULL
|
||||
WHERE f.source_id = $1
|
||||
AND f.row_num IS NULL
|
||||
AND f.entity_slug IS NOT NULL
|
||||
AND f.expired_at IS NULL
|
||||
AND EXISTS (
|
||||
SELECT 1 FROM pages p
|
||||
WHERE p.source_id = f.source_id
|
||||
AND p.slug = f.entity_slug
|
||||
AND p.deleted_at IS NULL
|
||||
)`,
|
||||
[sourceId],
|
||||
);
|
||||
const legacyCount = parseInt(legacy[0]?.n ?? '0', 10);
|
||||
result.legacyRowsPending = legacyCount;
|
||||
if (legacyCount > 0) {
|
||||
result.guardTriggered = true;
|
||||
// Drain advice must actually work: a bare `apply-migrations --yes`
|
||||
// is a no-op once the v0.32.2 ledger entry says complete (the
|
||||
// runner classifies it as already-applied), so the sanctioned
|
||||
// re-run path is the explicit retry marker first. Phase B is
|
||||
// idempotent — it only touches `row_num IS NULL` rows and de-dupes
|
||||
// against the existing fence — so the re-run is safe. Individual
|
||||
// rows can instead be drained through `forget_fact` (soft-expired
|
||||
// rows stop counting).
|
||||
result.warnings.push(
|
||||
`extract_facts: ${legacyCount} legacy v0.31 fact rows (entity page present, not yet ` +
|
||||
`fenced) pending fence backfill. Run \`gbrain apply-migrations --yes\` to complete ` +
|
||||
`v0_32_2 before this phase can safely reconcile fence → DB.`,
|
||||
`extract_facts: ${legacyCount} legacy v0.31 fact rows in source "${sourceId}" ` +
|
||||
`(entity page present, not yet fenced) pending fence backfill. Re-run the v0.32.2 ` +
|
||||
`fence backfill: \`gbrain apply-migrations --force-retry 0.32.2\` then ` +
|
||||
`\`gbrain apply-migrations --yes\`. Or drain individual rows via \`forget_fact\`.`,
|
||||
);
|
||||
return result;
|
||||
}
|
||||
@@ -303,6 +355,11 @@ export async function runExtractFacts(
|
||||
result.warnings.push(
|
||||
...parsed.warnings.map(w => `${slug}: ${w}`),
|
||||
);
|
||||
// The parser deliberately skips malformed rows and returns any rows it
|
||||
// could still recover. That partial result is not authoritative: using
|
||||
// it for reconciliation would interpret skipped rows as deletions.
|
||||
// Preserve this page's existing index and continue with other pages.
|
||||
continue;
|
||||
}
|
||||
|
||||
if (parsed.facts.length > 0) result.pagesWithFacts += 1;
|
||||
@@ -334,9 +391,12 @@ export async function runExtractFacts(
|
||||
// partial-UNIQUE-index keyspace). #1928: `cli:`-origin facts
|
||||
// (conversation facts from extract-conversation-facts) are NOT
|
||||
// fence-owned — the page carries no `## Facts` fence to recreate
|
||||
// them — so they MUST survive this reconcile.
|
||||
// them — so they MUST survive this reconcile. #2646: soft-expired
|
||||
// legacy rows (forget_fact's record of the forget) likewise
|
||||
// survive via preserveExpiredLegacy.
|
||||
const deleted = await engine.deleteFactsForPage(slug, sourceId, {
|
||||
excludeSourcePrefixes: ['cli:'],
|
||||
preserveExpiredLegacy: true,
|
||||
});
|
||||
result.factsDeleted += deleted.deleted;
|
||||
}
|
||||
@@ -363,10 +423,11 @@ export async function runExtractFacts(
|
||||
if (hasStaleExisting || hasDuplicateExisting || hasRowNumDrift) {
|
||||
// Fall back to the legacy page-level reconcile when old DB rows must
|
||||
// be removed. Same delete scoping as above: legacy
|
||||
// NULL-source_markdown_slug rows and `cli:`-origin conversation
|
||||
// facts (#1928) survive.
|
||||
// NULL-source_markdown_slug rows, `cli:`-origin conversation
|
||||
// facts (#1928), and soft-expired legacy rows (#2646) survive.
|
||||
const deleted = await engine.deleteFactsForPage(slug, sourceId, {
|
||||
excludeSourcePrefixes: ['cli:'],
|
||||
preserveExpiredLegacy: true,
|
||||
});
|
||||
result.factsDeleted += deleted.deleted;
|
||||
toInsert = extracted;
|
||||
|
||||
@@ -48,8 +48,9 @@ import { safeSplitIndex } from '../text-safe.ts';
|
||||
import { PAGE_SLUG_SEG } from '../cjk.ts';
|
||||
|
||||
// Slug grammar from validatePageSlug — shared via PAGE_SLUG_SEG (#738).
|
||||
// Used for the orchestrator-written summary index slug.
|
||||
const SUMMARY_SLUG_RE = new RegExp(`^${PAGE_SLUG_SEG}(\\/${PAGE_SLUG_SEG})*$`);
|
||||
// Used for the orchestrator-written summary index slug. `u` flag required
|
||||
// by PAGE_SLUG_SEG's \p{...} classes (#3417).
|
||||
const SUMMARY_SLUG_RE = new RegExp(`^${PAGE_SLUG_SEG}(\\/${PAGE_SLUG_SEG})*$`, 'u');
|
||||
|
||||
// ── Model context budget (D1, D5, D7, D9) ─────────────────────────────
|
||||
|
||||
|
||||
@@ -112,6 +112,7 @@ export const BRAIN_CHECK_NAMES: ReadonlySet<string> = new Set([
|
||||
'takes_weight_grid',
|
||||
'timeline_coverage',
|
||||
'unified_multimodal_coverage',
|
||||
'unverified_extractions',
|
||||
'voice_gate_health',
|
||||
]);
|
||||
|
||||
|
||||
+17
-2
@@ -19,7 +19,8 @@
|
||||
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
import type { ChunkInput } from './types.ts';
|
||||
import { embedBatchWithBackoff } from '../commands/embed.ts';
|
||||
import { embedBatchWithBackoff, restampIfDemotedToTitleTier } from '../commands/embed.ts';
|
||||
import { wrapChunkTextsForStoredMode } from './embedding-context.ts';
|
||||
import { type DbPacer, createNoopPacer, observed } from './db-pacer.ts';
|
||||
import { AbortError } from './abort-check.ts';
|
||||
|
||||
@@ -189,8 +190,15 @@ export async function embedStaleForSource(
|
||||
const keySourceId = stale[0]?.source_id ?? sourceId;
|
||||
const slug = stale[0].slug;
|
||||
try {
|
||||
// #3507: fetch the page row for its title + stored CR mode so the
|
||||
// re-embed reproduces the page's wrapping convention instead of
|
||||
// silently stripping contextual prefixes (mirrors
|
||||
// src/commands/embed.ts:embedAllStale).
|
||||
const pageRow = await observed(pacer, () =>
|
||||
engine.getPage(slug, { sourceId: keySourceId }),
|
||||
);
|
||||
const embeddings = await embedFn(
|
||||
stale.map((c) => c.chunk_text),
|
||||
wrapChunkTextsForStoredMode(pageRow, stale),
|
||||
{ abortSignal: signal },
|
||||
);
|
||||
const existing = await observed(pacer, () =>
|
||||
@@ -233,6 +241,13 @@ export async function embedStaleForSource(
|
||||
engine.setPageEmbeddingSignature(slug, { sourceId: keySourceId, signature }),
|
||||
);
|
||||
}
|
||||
// #3507: a FULLY re-embedded per_chunk_synopsis page landed at the
|
||||
// title tier — keep the stamped mode honest (mixed pages stay as-is).
|
||||
if (stale.length === existing.length) {
|
||||
await observed(pacer, () =>
|
||||
restampIfDemotedToTitleTier(engine, pageRow, slug, keySourceId),
|
||||
);
|
||||
}
|
||||
result.embedded += stale.length;
|
||||
result.pagesProcessed += 1;
|
||||
} catch (e: unknown) {
|
||||
|
||||
@@ -186,3 +186,41 @@ export function modeRequiresHaiku(mode: CRMode): boolean {
|
||||
export function modeRequiresWrapper(mode: CRMode): boolean {
|
||||
return mode !== 'none';
|
||||
}
|
||||
|
||||
/**
|
||||
* #3507 — build the embedding inputs for a re-embed of EXISTING chunk rows,
|
||||
* reproducing the wrapping convention the page's vectors were originally
|
||||
* built under (recorded in `pages.contextual_retrieval_mode`).
|
||||
*
|
||||
* Used by every plain re-embed path (`embed <slug>`, `embed --all`,
|
||||
* `embed --stale`, the embed-backfill Minion loop). Before this helper those
|
||||
* paths embedded raw `chunk_text`, so any re-embed — including the NORMAL
|
||||
* post-model-migration `embed --stale` — silently replaced context-wrapped
|
||||
* vectors with unwrapped ones, degrading retrieval with no signature change
|
||||
* to show for it.
|
||||
*
|
||||
* Convention rules (embed PRESERVES conventions; changing them is
|
||||
* sync/reindex's job):
|
||||
* - mode NULL/undefined/'none' → raw chunk_text (status quo).
|
||||
* - mode 'title' → title-only prefix (pure string concat).
|
||||
* - mode 'per_chunk_synopsis' → title-only prefix. Re-generating Haiku
|
||||
* synopses is a paid backfill concern; title-only is the service's own
|
||||
* documented fallback tier (D14). Callers that fully re-embed a page
|
||||
* this way should restamp the page to 'title' so the column stays
|
||||
* honest (see contextual-retrieval-service.ts:titleTierCorpusGeneration).
|
||||
* - `fenced_code` chunks are NEVER wrapped (D20-T4), same as sync.
|
||||
*/
|
||||
export function wrapChunkTextsForStoredMode(
|
||||
page:
|
||||
| { title?: string | null; contextual_retrieval_mode?: CRMode | null }
|
||||
| null
|
||||
| undefined,
|
||||
chunks: ReadonlyArray<{ chunk_text: string; chunk_source?: string | null }>,
|
||||
): string[] {
|
||||
const mode = page?.contextual_retrieval_mode;
|
||||
if (mode == null || !modeRequiresWrapper(mode)) {
|
||||
return chunks.map((c) => c.chunk_text);
|
||||
}
|
||||
const prefix = buildContextualPrefix(page?.title ?? '', null);
|
||||
return chunks.map((c) => wrapChunkForEmbedding(c.chunk_text, prefix, c.chunk_source));
|
||||
}
|
||||
|
||||
+22
-1
@@ -1341,6 +1341,16 @@ export interface BrainEngine {
|
||||
getContentFlagsByPageIds(
|
||||
pageIds: number[],
|
||||
): Promise<Map<number, { reason: string; detail: string }>>;
|
||||
/**
|
||||
* Extraction quarantine lane (issue #160): for a list of page_ids, return
|
||||
* the subset that are unverified auto-extracted entity stubs (frontmatter
|
||||
* `provenance: 'auto-extracted'` + `status: 'unverified'`). Used by hybrid
|
||||
* search to stamp `SearchResult.unverified` pre-fusion so the fusion-level
|
||||
* compiled-truth boost skips them. Single SQL query, not N+1. Empty input
|
||||
* → empty set (no query). SQL predicate is the shared
|
||||
* `unverifiedExtractionFragment` (src/core/extraction-review.ts).
|
||||
*/
|
||||
getUnverifiedExtractionPageIds(pageIds: number[]): Promise<Set<number>>;
|
||||
/**
|
||||
* v0.27.0: for a list of slugs, return their updated_at timestamps (or created_at fallback).
|
||||
* Used by hybrid search recency boost. Single SQL query, not N+1.
|
||||
@@ -1805,11 +1815,22 @@ export interface BrainEngine {
|
||||
* never recreate them (the page has no `## Facts` fence). Omitted ⇒ legacy
|
||||
* behavior (delete every fact on the page coordinate). NULL/empty `source`
|
||||
* rows are always deletable (fence default).
|
||||
*
|
||||
* #2646: `preserveExpiredLegacy` protects soft-expired legacy rows
|
||||
* (`row_num IS NULL AND expired_at IS NOT NULL`) — the record left by
|
||||
* `forget_fact`'s legacy DB-only path. Fence rows always carry a
|
||||
* `row_num`, so these rows are never fence-owned and a wipe would
|
||||
* destroy the forget record (the audit trail of the forget). Note what
|
||||
* this does NOT promise: it protects the record, not the forget itself —
|
||||
* if the fence still carries the same claim, fence canonicality
|
||||
* independently reinserts it as a fresh active row (legacy DB-only
|
||||
* forgets are documented as non-durable; see extract-facts.ts). Omitted
|
||||
* ⇒ legacy behavior.
|
||||
*/
|
||||
deleteFactsForPage(
|
||||
slug: string,
|
||||
source_id: string,
|
||||
opts?: { excludeSourcePrefixes?: string[] },
|
||||
opts?: { excludeSourcePrefixes?: string[]; preserveExpiredLegacy?: boolean },
|
||||
): Promise<{ deleted: number }>;
|
||||
|
||||
/**
|
||||
|
||||
@@ -15,6 +15,7 @@
|
||||
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
import { waitForCapacity } from './backoff.ts';
|
||||
import { quarantineMarkers } from './extraction-review.ts';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Types
|
||||
@@ -28,9 +29,32 @@ export interface EnrichmentRequest {
|
||||
tier?: 1 | 2 | 3;
|
||||
}
|
||||
|
||||
/**
|
||||
* Trust options for the enrichment write path (issue #160).
|
||||
*
|
||||
* `trusted: true` — the input text comes from the machine owner via the
|
||||
* trusted local CLI (ctx.remote === false) AND the caller passed an explicit
|
||||
* opt-in flag. Stubs write direct as authoritative entity pages.
|
||||
*
|
||||
* Anything else (undefined, false, absent) is UNTRUSTED — fail-closed,
|
||||
* mirroring the OperationContext.remote invariant ("anything not strictly
|
||||
* false is remote"). Created stubs land in the quarantine lane: frontmatter
|
||||
* `provenance: 'auto-extracted'` + `status: 'unverified'`. They are excluded
|
||||
* from authoritative retrieval boosts and wait in the review queue
|
||||
* (`extraction_pending` / `extraction_review` ops) until the owner promotes
|
||||
* or rejects them.
|
||||
*/
|
||||
export interface EnrichmentTrustOptions {
|
||||
trusted?: boolean;
|
||||
/** Source to read/write in (multi-source brains). Omitted → engine default. */
|
||||
sourceId?: string;
|
||||
}
|
||||
|
||||
export interface EnrichmentResult {
|
||||
slug: string;
|
||||
action: 'created' | 'updated' | 'skipped';
|
||||
/** True when the created stub landed in the quarantine lane (issue #160). */
|
||||
quarantined?: boolean;
|
||||
tier: 1 | 2 | 3;
|
||||
backlinkCreated: boolean;
|
||||
timelineAdded: boolean;
|
||||
@@ -72,11 +96,15 @@ export function entityPagePath(name: string, type: 'person' | 'company'): string
|
||||
export async function enrichEntity(
|
||||
engine: BrainEngine,
|
||||
request: EnrichmentRequest,
|
||||
opts?: EnrichmentTrustOptions,
|
||||
): Promise<EnrichmentResult> {
|
||||
const slug = slugifyEntity(request.entityName, request.entityType);
|
||||
// Fail-closed: only an explicit `trusted: true` writes authoritative pages.
|
||||
const trusted = opts?.trusted === true;
|
||||
const scope = opts?.sourceId ? { sourceId: opts.sourceId } : undefined;
|
||||
|
||||
// 1. Count existing mentions for tier auto-escalation
|
||||
const { mentionCount, mentionSources } = await countMentions(engine, request.entityName);
|
||||
const { mentionCount, mentionSources } = await countMentions(engine, request.entityName, opts?.sourceId);
|
||||
|
||||
// 2. Determine tier (auto-escalate based on mentions)
|
||||
const suggestedTier = suggestTier(mentionCount, mentionSources, request.context);
|
||||
@@ -84,7 +112,7 @@ export async function enrichEntity(
|
||||
const tierEscalated = suggestedTier < (request.tier || 3); // lower tier number = higher importance
|
||||
|
||||
// 3. Check if entity page exists
|
||||
const existingPage = await engine.getPage(slug);
|
||||
const existingPage = await engine.getPage(slug, scope);
|
||||
let action: 'created' | 'updated' | 'skipped';
|
||||
|
||||
if (existingPage) {
|
||||
@@ -104,8 +132,11 @@ export async function enrichEntity(
|
||||
created: new Date().toISOString().split('T')[0],
|
||||
source: request.sourceSlug,
|
||||
tier,
|
||||
// issue #160 quarantine lane: stubs extracted from untrusted input
|
||||
// carry provenance + unverified markers until the owner reviews them.
|
||||
...(trusted ? {} : quarantineMarkers()),
|
||||
},
|
||||
});
|
||||
}, scope);
|
||||
action = 'created';
|
||||
}
|
||||
|
||||
@@ -116,7 +147,7 @@ export async function enrichEntity(
|
||||
date: new Date().toISOString().split('T')[0] ?? '',
|
||||
summary: `Referenced in [${request.sourceSlug}](${request.sourceSlug}) — ${request.context}`,
|
||||
source: request.sourceSlug,
|
||||
});
|
||||
}, scope);
|
||||
timelineAdded = true;
|
||||
} catch {
|
||||
// Timeline add failed (page might not support it)
|
||||
@@ -125,7 +156,7 @@ export async function enrichEntity(
|
||||
// 5. Add backlink from entity to source
|
||||
let backlinkCreated = false;
|
||||
try {
|
||||
await engine.addLink(slug, request.sourceSlug, `Entity mention from ${request.sourceSlug}`); // gbrain-allow-direct-insert: auto-link reconciliation triggered by entity reference in source markdown
|
||||
await engine.addLink(slug, request.sourceSlug, `Entity mention from ${request.sourceSlug}`, undefined, undefined, undefined, undefined, opts?.sourceId ? { fromSourceId: opts.sourceId, toSourceId: opts.sourceId } : undefined); // gbrain-allow-direct-insert: auto-link reconciliation triggered by entity reference in source markdown
|
||||
backlinkCreated = true;
|
||||
} catch {
|
||||
// Link might already exist
|
||||
@@ -134,6 +165,7 @@ export async function enrichEntity(
|
||||
return {
|
||||
slug,
|
||||
action,
|
||||
...(action === 'created' && !trusted ? { quarantined: true } : {}),
|
||||
tier,
|
||||
backlinkCreated,
|
||||
timelineAdded,
|
||||
@@ -152,14 +184,14 @@ export async function enrichEntity(
|
||||
export async function enrichEntities(
|
||||
engine: BrainEngine,
|
||||
requests: EnrichmentRequest[],
|
||||
config?: { throttle?: boolean; onProgress?: (done: number, total: number, name: string) => void },
|
||||
config?: { throttle?: boolean; onProgress?: (done: number, total: number, name: string) => void } & EnrichmentTrustOptions,
|
||||
): Promise<EnrichmentResult[]> {
|
||||
const results: EnrichmentResult[] = [];
|
||||
for (const req of requests) {
|
||||
if (config?.throttle !== false) {
|
||||
await waitForCapacity({ maxAttempts: 5 }); // shorter timeout for batch items
|
||||
}
|
||||
const result = await enrichEntity(engine, req);
|
||||
const result = await enrichEntity(engine, req, { trusted: config?.trusted, sourceId: config?.sourceId });
|
||||
results.push(result);
|
||||
config?.onProgress?.(results.length, requests.length, req.entityName);
|
||||
}
|
||||
@@ -175,8 +207,11 @@ export async function extractAndEnrich(
|
||||
engine: BrainEngine,
|
||||
text: string,
|
||||
sourceSlug: string,
|
||||
opts?: EnrichmentTrustOptions & { throttle?: boolean; maxEntities?: number },
|
||||
): Promise<EnrichmentResult[]> {
|
||||
const entities = extractEntities(text);
|
||||
// Bounded by default (#160 hardening): the greedy regex on a large paste
|
||||
// can produce thousands of hits; each enrichment is several DB round-trips.
|
||||
const entities = extractEntities(text).slice(0, opts?.maxEntities ?? 200);
|
||||
if (entities.length === 0) return [];
|
||||
|
||||
const requests: EnrichmentRequest[] = entities.map(e => ({
|
||||
@@ -186,7 +221,7 @@ export async function extractAndEnrich(
|
||||
sourceSlug,
|
||||
}));
|
||||
|
||||
return enrichEntities(engine, requests);
|
||||
return enrichEntities(engine, requests, { trusted: opts?.trusted, sourceId: opts?.sourceId, throttle: opts?.throttle });
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -197,9 +232,10 @@ export async function extractAndEnrich(
|
||||
async function countMentions(
|
||||
engine: BrainEngine,
|
||||
entityName: string,
|
||||
sourceId?: string,
|
||||
): Promise<{ mentionCount: number; mentionSources: string[] }> {
|
||||
try {
|
||||
const results = await engine.searchKeyword(entityName, { limit: 100 });
|
||||
const results = await engine.searchKeyword(entityName, { limit: 100, ...(sourceId ? { sourceId } : {}) });
|
||||
// Derive sources from slug prefixes since SearchResult has no metadata.skill
|
||||
const sources = new Set<string>();
|
||||
for (const r of results) {
|
||||
|
||||
@@ -85,11 +85,21 @@ const RUN_ID_SHORT_LEN = 8;
|
||||
/**
|
||||
* Truncate a run id to the standard 8-char short form used in slug
|
||||
* paths. Idempotent — passing an already-short id returns it unchanged.
|
||||
* Non-hex / non-alphanumeric chars survive (op-checkpoint ids may
|
||||
* include dashes or other separators).
|
||||
* Non-hex / non-alphanumeric chars survive INSIDE the short form
|
||||
* (op-checkpoint ids may include dashes or other separators), but
|
||||
* boundary hyphens are trimmed (#3443): `slugifySegment()` strips
|
||||
* leading/trailing hyphens during repo sync, so a short form like
|
||||
* 'propose-' (from propose-<timestamp> run ids) made the DB receipt
|
||||
* slug and its Git-backed slug disagree — writing the receipt through
|
||||
* to the repo created a normalized sibling instead of materializing
|
||||
* the existing page. Invariant: slugifySegment(shortRunId(x)) ===
|
||||
* shortRunId(x) for slug-safe run ids.
|
||||
*/
|
||||
export function shortRunId(runId: string): string {
|
||||
return runId.slice(0, RUN_ID_SHORT_LEN);
|
||||
// ponytail: truncation-based discrimination is only as good as the run id's
|
||||
// first 8 chars; families that need per-run uniqueness must front-load it.
|
||||
const short = runId.slice(0, RUN_ID_SHORT_LEN).replace(/^-+|-+$/g, '');
|
||||
return short || (runId ? 'run' : '');
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -0,0 +1,88 @@
|
||||
/**
|
||||
* Extraction quarantine lane (issue #160).
|
||||
*
|
||||
* `extractAndEnrich` regex-extracts entity names from arbitrary ingested text
|
||||
* and creates `people/{slug}` / `companies/{slug}` stub pages. When the input
|
||||
* text comes from an untrusted channel (anything that is not the trusted local
|
||||
* CLI with an explicit opt-in), those stubs must NOT enter the brain as
|
||||
* authoritative entity pages. Instead they land in the quarantine lane:
|
||||
* ordinary pages carrying two frontmatter markers —
|
||||
*
|
||||
* provenance: 'auto-extracted' — HOW the page came to exist
|
||||
* status: 'unverified' — the owner has not reviewed it yet
|
||||
*
|
||||
* Consequences of the markers (each enforced at its own site):
|
||||
* - Search: unverified stubs are excluded from the compiled-truth authority
|
||||
* boost (they rank as ordinary content) and results carry
|
||||
* `unverified: true` so agents can label the provenance.
|
||||
* - Review: `extraction_pending` lists them; `extraction_review` promotes
|
||||
* (status → 'verified', provenance kept for audit) or rejects
|
||||
* (soft-delete) in batch. Promotion is local-owner-only.
|
||||
* - Doctor: counts unverified stubs older than N days as a review nudge.
|
||||
*
|
||||
* Fail-closed trust rule (mirrors OperationContext.remote): only an explicit
|
||||
* `trusted: true` writes direct; undefined/false/anything-else quarantines.
|
||||
*
|
||||
* Known scope (deliberate, documented — not gaps discovered later):
|
||||
* - CREATE-path only. The enrichment UPDATE path (timeline append + edge
|
||||
* onto an EXISTING page when a slug collides) is the separately-tracked
|
||||
* slug-collision finding referenced in issue #160; this lane does not
|
||||
* gate it.
|
||||
* - The markers are ordinary frontmatter keys, not put_page-strip-listed
|
||||
* (#1699). A caller holding generic remote put_page write scope can
|
||||
* rewrite a stub without them — but that caller can author an unmarked
|
||||
* people/ page directly anyway, so stripping here adds no privilege.
|
||||
* The promotion OP surface (extraction_review) is what stays owner-only.
|
||||
*
|
||||
* Sibling of `src/core/quarantine.ts` / `src/core/embed-skip.ts` — same
|
||||
* marker-as-frontmatter-JSONB pattern, same "SQL fragment lives next to the
|
||||
* marker key so they can never drift" rule. No schema migration needed.
|
||||
*/
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Marker keys + values (stable contract)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export const EXTRACTION_PROVENANCE_KEY = 'provenance';
|
||||
export const EXTRACTION_STATUS_KEY = 'status';
|
||||
|
||||
export const PROVENANCE_AUTO_EXTRACTED = 'auto-extracted';
|
||||
export const STATUS_UNVERIFIED = 'unverified';
|
||||
export const STATUS_VERIFIED = 'verified';
|
||||
|
||||
/** Frontmatter markers to spread onto a quarantined stub at create time. */
|
||||
export function quarantineMarkers(): Record<string, string> {
|
||||
return {
|
||||
[EXTRACTION_PROVENANCE_KEY]: PROVENANCE_AUTO_EXTRACTED,
|
||||
[EXTRACTION_STATUS_KEY]: STATUS_UNVERIFIED,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* JS-side predicate: true only when BOTH markers match. Requiring the pair
|
||||
* means user pages that happen to carry their own `status` or `provenance`
|
||||
* frontmatter are never captured by the review lane.
|
||||
*/
|
||||
export function isUnverifiedExtraction(
|
||||
frontmatter: Record<string, unknown> | null | undefined,
|
||||
): boolean {
|
||||
if (!frontmatter) return false;
|
||||
return (
|
||||
frontmatter[EXTRACTION_PROVENANCE_KEY] === PROVENANCE_AUTO_EXTRACTED &&
|
||||
frontmatter[EXTRACTION_STATUS_KEY] === STATUS_UNVERIFIED
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* SQL fragment matching unverified auto-extracted stubs, parameterized on the
|
||||
* page-table alias. Single source of truth for every SQL-side consumer
|
||||
* (extraction_pending list, doctor count) so the filter and the marker keys
|
||||
* can never drift. `pageAlias` is engine-supplied (never user input).
|
||||
* JSONB `->>` works identically on Postgres and PGLite (PostgreSQL-in-WASM).
|
||||
*/
|
||||
export function unverifiedExtractionFragment(pageAlias: string): string {
|
||||
return (
|
||||
`(COALESCE(${pageAlias}.frontmatter, '{}'::jsonb) ->> '${EXTRACTION_PROVENANCE_KEY}') = '${PROVENANCE_AUTO_EXTRACTED}'` +
|
||||
` AND (COALESCE(${pageAlias}.frontmatter, '{}'::jsonb) ->> '${EXTRACTION_STATUS_KEY}') = '${STATUS_UNVERIFIED}'`
|
||||
);
|
||||
}
|
||||
+55
-13
@@ -96,29 +96,71 @@ export interface GitFreshnessOpts {
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns true iff `localPath` is a git repo whose current HEAD matches
|
||||
* `lastCommit`, AND (when `requireCleanWorkingTree`) the working tree
|
||||
* is clean.
|
||||
* Three-state git probe verdict for a federated source clone.
|
||||
*
|
||||
* - `'unchanged'`: HEAD matches `last_commit` (and, when requested, the
|
||||
* working tree is clean). Sync has nothing to do.
|
||||
* - `'changed'`: the clone is readable but HEAD moved, the tree is
|
||||
* dirty, or the DB never recorded a `last_commit` —
|
||||
* sync genuinely has (or may have) work.
|
||||
* - `'unavailable'`: the HEAD probe itself could not run — the clone
|
||||
* directory is missing, not a git repo, or git errored.
|
||||
* On stateless deploys (containers on EB / K8s / Fly,
|
||||
* where `local_path` dies with the filesystem and is
|
||||
* lazily re-materialized by the next per-source sync)
|
||||
* this is a NORMAL steady state for quiet sources, not
|
||||
* evidence of pending work. Callers can fall back to a
|
||||
* DB-only freshness signal instead of wall-clock age.
|
||||
*/
|
||||
export type SourceGitState = 'unchanged' | 'changed' | 'unavailable';
|
||||
|
||||
/**
|
||||
* Probe a source clone and classify it (see `SourceGitState`).
|
||||
*
|
||||
* This is NOT a full mirror of `gbrain sync`'s "do work?" predicate.
|
||||
* Chunker-version match is computed by the caller because it depends on
|
||||
* engine state (`sources.chunker_version` vs `CURRENT_CHUNKER_VERSION`).
|
||||
* See `src/commands/doctor.ts:checkSyncFreshness` for the AND
|
||||
* combination at the call site.
|
||||
*
|
||||
* NULL-input guard stays first: a NULL `last_commit` (legacy row) returns
|
||||
* `'changed'` WITHOUT running the head probe — same short-circuit contract
|
||||
* `isSourceUnchangedSinceSync` always had (pinned by doctor.test.ts case 4).
|
||||
*/
|
||||
export function probeSourceGitState(
|
||||
localPath: string | null | undefined,
|
||||
lastCommit: string | null | undefined,
|
||||
opts?: GitFreshnessOpts,
|
||||
): SourceGitState {
|
||||
if (!localPath || !lastCommit) return 'changed';
|
||||
const head = _headProbe(localPath);
|
||||
if (head === null) return 'unavailable';
|
||||
if (head !== lastCommit) return 'changed';
|
||||
if (opts?.requireCleanWorkingTree) {
|
||||
const ignoreUntracked = opts.requireCleanWorkingTree === 'ignore-untracked';
|
||||
const isClean = _cleanProbe(localPath, ignoreUntracked);
|
||||
// null (probe error) AND false (known dirty) both fail the gate. A clean
|
||||
// probe error with a READABLE head is not classified 'unavailable' —
|
||||
// fail toward "may have work" so the gate can only relax, never mask.
|
||||
if (isClean !== true) return 'changed';
|
||||
}
|
||||
return 'unchanged';
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns true iff `localPath` is a git repo whose current HEAD matches
|
||||
* `lastCommit`, AND (when `requireCleanWorkingTree`) the working tree
|
||||
* is clean.
|
||||
*
|
||||
* Boolean façade over `probeSourceGitState` — `'unavailable'` and
|
||||
* `'changed'` both collapse to `false`, preserving the v0.41.27.0
|
||||
* fail-open contract for callers that only care about the short-circuit
|
||||
* (`src/core/source-health.ts`).
|
||||
*/
|
||||
export function isSourceUnchangedSinceSync(
|
||||
localPath: string | null | undefined,
|
||||
lastCommit: string | null | undefined,
|
||||
opts?: GitFreshnessOpts,
|
||||
): boolean {
|
||||
if (!localPath || !lastCommit) return false;
|
||||
const head = _headProbe(localPath);
|
||||
if (head === null || head !== lastCommit) return false;
|
||||
if (opts?.requireCleanWorkingTree) {
|
||||
const ignoreUntracked = opts.requireCleanWorkingTree === 'ignore-untracked';
|
||||
const isClean = _cleanProbe(localPath, ignoreUntracked);
|
||||
// null (probe error) AND false (known dirty) both fail the gate.
|
||||
if (isClean !== true) return false;
|
||||
}
|
||||
return true;
|
||||
return probeSourceGitState(localPath, lastCommit, opts) === 'unchanged';
|
||||
}
|
||||
|
||||
@@ -1092,8 +1092,8 @@ export async function importFromFile(
|
||||
chunks: 0,
|
||||
error:
|
||||
`Filename "${relativePath}" produces no usable slug. ` +
|
||||
`Add a "slug:" to the frontmatter, or rename the file to use ` +
|
||||
`ASCII / Chinese / Japanese / Korean characters.`,
|
||||
`Add a "slug:" to the frontmatter, or rename the file to include ` +
|
||||
`at least one letter or number (any script).`,
|
||||
};
|
||||
}
|
||||
} else if (parsed.slug !== expectedSlug) {
|
||||
|
||||
+75
-11
@@ -12,7 +12,7 @@
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
import type { PageType } from './types.ts';
|
||||
import type { PageType, EffectiveDateSource } from './types.ts';
|
||||
import { ensureWellFormed } from './text-safe.ts';
|
||||
|
||||
/**
|
||||
@@ -671,6 +671,17 @@ const FOUNDED_RE = /\b(?:founded|co-?founded|started the company|incorporated|fo
|
||||
// "security advisor to|at", "product advisor to|at", "industry advisor".
|
||||
const ADVISES_RE = /\b(?:advises|advised|advisor (?:to|at|for|of)|advisory (?:board|role|position|capacity|engagement|partnership|contract|relationship|work)|board advisor|on .{0,20} advisory board|joined .{0,20} advisory board|in an? advisory (?:capacity|role|position)|as an? (?:advisor|security advisor|technical advisor|strategic advisor|industry advisor|product advisor|board advisor|senior advisor)|(?:strategic|technical|security|product|industry|senior|board) advisor (?:to|at|for|of)|consults for|consulting role (?:at|with))\b/i;
|
||||
|
||||
// Chinese link type patterns for CJK entity mentions.
|
||||
// NOTE: These patterns are Chinese-only (zh). Japanese and Korean link
|
||||
// type extraction is not yet implemented. Entity NAME extraction in
|
||||
// by-mention.ts covers all three scripts (CJK = Chinese/Japanese/Korean)
|
||||
// via Unicode-aware tokenization.
|
||||
const ZH_FOUNDED_RE = /(?:创立|创办|成立|创建|建立|开创|发起)(?:了|的)/;
|
||||
const ZH_INVESTED_RE = /(?:投资|入股|融资|注资|参股)(?:了|的|了?于)/;
|
||||
const ZH_ADVISES_RE = /(?:顾问|咨询|指导)(?:了|的)?/;
|
||||
const ZH_WORKS_AT_RE = /(?:任职|就职|担任|供职|在.{0,10}(?:工作|上班|负责))(?:于|在|的)?/;
|
||||
const ZH_CITED_RE = /(?:引用|援引|提到|提及|转述|摘录)(?:了|的|自)?/;
|
||||
|
||||
// Page-role detection: if the source page describes a partner/investor at
|
||||
// page level, that's a strong prior for outbound company refs being
|
||||
// invested_in even when per-edge context lacks explicit investment verbs.
|
||||
@@ -724,6 +735,12 @@ export function inferLinkType(pageType: PageType, context: string, globalContext
|
||||
if (INVESTED_RE.test(context)) return 'invested_in';
|
||||
if (ADVISES_RE.test(context)) return 'advises';
|
||||
if (WORKS_AT_RE.test(context)) return 'works_at';
|
||||
// Chinese link type patterns
|
||||
if (ZH_FOUNDED_RE.test(context)) return 'founded';
|
||||
if (ZH_INVESTED_RE.test(context)) return 'invested_in';
|
||||
if (ZH_ADVISES_RE.test(context)) return 'advises';
|
||||
if (ZH_WORKS_AT_RE.test(context)) return 'works_at';
|
||||
if (ZH_CITED_RE.test(context)) return 'cited';
|
||||
// Page-role prior: only fires for person -> company links. Concept pages
|
||||
// about VC topics naturally contain "venture capital" in their text, but
|
||||
// their company refs are mentions, not investments. Partner pages mentioning
|
||||
@@ -1174,6 +1191,10 @@ export interface TimelineCandidate {
|
||||
// Match: `- **YYYY-MM-DD** | summary` or `- **YYYY-MM-DD** -- summary`
|
||||
// or `- **YYYY-MM-DD** - summary` or just `**YYYY-MM-DD** | summary`.
|
||||
const TIMELINE_LINE_RE = /^\s*-?\s*\*\*(\d{4}-\d{2}-\d{2})\*\*\s*[|\-–—]+\s*(.+?)\s*$/;
|
||||
// Chinese date lines: `- 2020年1月2日 | summary` (bold optional). Requires the
|
||||
// 年/月 markers so plain ASCII `- 2020-01-02 - text` does NOT match — non-bold
|
||||
// ASCII dates were never timeline entries and must stay that way.
|
||||
const TIMELINE_LINE_RE_CN = /^\s*-?\s*(?:\*\*)?(\d{4})年(\d{1,2})月(\d{1,2})日?(?:\*\*)?\s*[|\-–—]+\s*(.+?)\s*$/;
|
||||
|
||||
/**
|
||||
* Parse timeline entries from content. Looks at:
|
||||
@@ -1190,18 +1211,21 @@ export function parseTimelineEntries(content: string): TimelineCandidate[] {
|
||||
|
||||
let i = 0;
|
||||
while (i < lines.length) {
|
||||
// Try English format first, then Chinese
|
||||
const m = TIMELINE_LINE_RE.exec(lines[i]);
|
||||
if (!m) {
|
||||
i++;
|
||||
continue;
|
||||
let date: string;
|
||||
let summary: string;
|
||||
if (m) {
|
||||
date = m[1];
|
||||
summary = m[2].trim();
|
||||
} else {
|
||||
const cm = TIMELINE_LINE_RE_CN.exec(lines[i]);
|
||||
if (!cm) { i++; continue; }
|
||||
// Normalize Chinese date to YYYY-MM-DD
|
||||
date = `${cm[1]}-${cm[2].padStart(2, '0')}-${cm[3].padStart(2, '0')}`;
|
||||
summary = cm[4].trim();
|
||||
}
|
||||
const date = m[1];
|
||||
const summary = m[2].trim();
|
||||
if (!isValidDate(date) || summary.length === 0) {
|
||||
i++;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!isValidDate(date) || summary.length === 0) { i++; continue; }
|
||||
// Collect optional detail lines (indented, until next date or heading).
|
||||
const detailLines: string[] = [];
|
||||
let j = i + 1;
|
||||
@@ -1266,6 +1290,46 @@ function isValidDate(s: string): boolean {
|
||||
return dt.getUTCFullYear() === y && dt.getUTCMonth() === mo - 1 && dt.getUTCDate() === d;
|
||||
}
|
||||
|
||||
/** Input for {@link deriveTimelineAnchor}: a page's identity + its computed content date. */
|
||||
export interface TimelineAnchorInput {
|
||||
slug: string;
|
||||
title?: string | null;
|
||||
effectiveDate?: Date | string | null;
|
||||
effectiveDateSource?: EffectiveDateSource | null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Anchor a single timeline entry from a page's computed content date, for pages
|
||||
* whose body carries no parseable timeline line.
|
||||
*
|
||||
* Comms- and calendar-dominated brains keep the date in frontmatter or the
|
||||
* filename (slug `2026-04-24-...`), not in the prose, so `parseTimelineEntries`
|
||||
* returns nothing and the page-level `timeline` table stays empty even though
|
||||
* the page is firmly dated — leaving `get_timeline` and the brain-score
|
||||
* `timeline_coverage` component blind to it. This recovers that signal from the
|
||||
* already-computed `effective_date` (no re-parsing). (It does NOT feed the
|
||||
* facts-based `find_trajectory`, which reads the `facts` table by entity_slug.)
|
||||
*
|
||||
* Fires ONLY for a trustworthy content date — frontmatter (`event_date` / `date`
|
||||
* / `published`) or the `filename` date — never the `fallback` source, which is
|
||||
* `updated_at` (link-churn noise, not when the thing happened). Returns null
|
||||
* when no trustworthy date is available. Callers MUST apply this only when body
|
||||
* parsing yields zero entries, so it can never shadow a real in-body timeline.
|
||||
*/
|
||||
export function deriveTimelineAnchor(input: TimelineAnchorInput): TimelineCandidate | null {
|
||||
const { slug, title, effectiveDate, effectiveDateSource } = input;
|
||||
if (!effectiveDate) return null;
|
||||
// 'fallback' === updated_at; the rest ('event_date'|'date'|'published'|'filename')
|
||||
// are real content dates. null/undefined source is not trustworthy either.
|
||||
if (effectiveDateSource == null || effectiveDateSource === 'fallback') return null;
|
||||
const dt = typeof effectiveDate === 'string' ? new Date(effectiveDate) : effectiveDate;
|
||||
if (!(dt instanceof Date) || Number.isNaN(dt.getTime())) return null;
|
||||
const iso = dt.toISOString().slice(0, 10);
|
||||
if (!isValidDate(iso)) return null;
|
||||
const summary = (title ?? '').trim() || slug.split('/').pop() || slug;
|
||||
return { date: iso, summary, detail: '' };
|
||||
}
|
||||
|
||||
// ─── Auto-link config ───────────────────────────────────────────
|
||||
|
||||
/**
|
||||
|
||||
@@ -43,6 +43,11 @@ export const HANDLER_DEFAULT_TIMEOUT_MS: Readonly<Record<string, number>> = {
|
||||
// few writes. Generous 10-min budget (vs the tight null-default) covers a
|
||||
// slow gateway without the 30-min loop budget.
|
||||
chronicle_extract: TEN_MIN_MS,
|
||||
// #3207 — same shape as chronicle_extract: one page = one LLM extraction
|
||||
// call + a few writes. Was missing from this map, so it inherited the tight
|
||||
// null-default and got dead-lettered mid-generation on slow chat providers
|
||||
// (facts silently lost) — exactly the failure this file exists to prevent.
|
||||
'facts-absorb': TEN_MIN_MS,
|
||||
// Per-page contextual reindex jobs process chunks sequentially with one
|
||||
// rate-leased LLM synopsis call per chunk; large transcript pages need more
|
||||
// than the standard 30-min long-job budget.
|
||||
|
||||
@@ -534,10 +534,21 @@ export class MinionQueue {
|
||||
}
|
||||
|
||||
/** Prune old jobs in terminal statuses. Returns count of deleted rows. */
|
||||
async prune(opts?: { olderThan?: Date; status?: MinionJobStatus[] }): Promise<number> {
|
||||
async prune(opts?: { olderThan?: Date; status?: MinionJobStatus[]; dryRun?: boolean }): Promise<number> {
|
||||
const statuses = opts?.status ?? ['completed', 'dead', 'cancelled'];
|
||||
const olderThan = opts?.olderThan ?? new Date(Date.now() - 30 * 86400000);
|
||||
|
||||
// #2712: dryRun counts the would-be-pruned rows without deleting.
|
||||
// Silent-ignoring a safety flag on a delete path is data loss.
|
||||
if (opts?.dryRun) {
|
||||
const rows = await this.engine.executeRaw<{ count: string }>(
|
||||
`SELECT count(*)::text as count FROM minion_jobs
|
||||
WHERE status = ANY($1) AND updated_at < $2`,
|
||||
[statuses, olderThan.toISOString()]
|
||||
);
|
||||
return parseInt(rows[0]?.count ?? '0', 10);
|
||||
}
|
||||
|
||||
const rows = await this.engine.executeRaw<{ count: string }>(
|
||||
`WITH pruned AS (
|
||||
DELETE FROM minion_jobs
|
||||
|
||||
+226
-6
@@ -11,7 +11,7 @@ import type { GBrainConfig } from './config.ts';
|
||||
import type { PageType } from './types.ts';
|
||||
import { importFromContent } from './import-file.ts';
|
||||
import { writePageThrough } from './write-through.ts';
|
||||
import { hybridSearch, hybridSearchCached, stampContentFlags } from './search/hybrid.ts';
|
||||
import { hybridSearch, hybridSearchCached, stampContentFlags, stampUnverifiedExtractions } from './search/hybrid.ts';
|
||||
import { expandQuery } from './search/expansion.ts';
|
||||
import { dedupResults } from './search/dedup.ts';
|
||||
import { captureEvalCandidate, isEvalCaptureEnabled, isEvalScrubEnabled } from './eval-capture.ts';
|
||||
@@ -21,11 +21,14 @@ import { isFactsBackstopEligible } from './facts/eligibility.ts';
|
||||
import { stripTakesFence } from './takes-fence.ts';
|
||||
import { stripFactsFence } from './facts-fence.ts';
|
||||
import { getContentFlag } from './quarantine.ts';
|
||||
import { unverifiedExtractionFragment, isUnverifiedExtraction, EXTRACTION_STATUS_KEY, STATUS_VERIFIED } from './extraction-review.ts';
|
||||
import { buildVisibilityClause } from './search/sql-ranking.ts';
|
||||
import { bumpLastRetrievedAt } from './last-retrieved.ts';
|
||||
import { isSearchMode } from './search/mode.ts';
|
||||
import { stampEvidence } from './search/evidence.ts';
|
||||
import type { SearchResult } from './types.ts';
|
||||
import { CJK_SLUG_CHARS, PAGE_SLUG_SEG } from './cjk.ts';
|
||||
import { ALL_SOURCES } from './source-id.ts';
|
||||
import * as db from './db.ts';
|
||||
import { VERSION } from '../version.ts';
|
||||
import {
|
||||
@@ -160,10 +163,11 @@ export function validatePageSlug(slug: string): void {
|
||||
if (slug.length > 255) {
|
||||
throw new OperationError('invalid_params', 'page_slug exceeds 255 characters');
|
||||
}
|
||||
// v0.32.7: CJK ranges (Han / Hiragana / Katakana / Hangul Syllables) allowed
|
||||
// in segments. ASCII shape rules (lead char, hyphen continuation) preserved.
|
||||
if (!new RegExp(`^${PAGE_SLUG_SEG}(\\/${PAGE_SLUG_SEG})*$`, 'i').test(slug)) {
|
||||
throw new OperationError('invalid_params', `Invalid page_slug: ${slug} (allowed: alphanumeric, CJK, hyphens, forward-slash separated segments)`);
|
||||
// #3417: letters/numbers from any script allowed in segments (u flag required
|
||||
// for the \p{...} classes in PAGE_SLUG_SEG). Shape rules (lead char, hyphen
|
||||
// continuation) preserved.
|
||||
if (!new RegExp(`^${PAGE_SLUG_SEG}(\\/${PAGE_SLUG_SEG})*$`, 'iu').test(slug)) {
|
||||
throw new OperationError('invalid_params', `Invalid page_slug: ${slug} (allowed: letters/numbers in any script, hyphens, forward-slash separated segments)`);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -484,6 +488,14 @@ export function sourceScopeOpts(ctx: OperationContext): { sourceId?: string; sou
|
||||
// value of `[]` MUST NOT widen scope to "all sources" by being interpreted
|
||||
// as "no filter."
|
||||
if (allowed && allowed.length > 0) return { sourceIds: allowed };
|
||||
// #1712: the __all__ sentinel spans the brain — but ONLY for trusted local
|
||||
// callers (strictly `remote === false`). For remote/untrusted callers the
|
||||
// literal stays as-is: it can never match a real source id (underscores are
|
||||
// rejected at creation), so the read fail-closes to empty rather than
|
||||
// widening past the caller's grant. Do NOT "simplify" this to `{}`.
|
||||
if (ctx.sourceId === ALL_SOURCES) {
|
||||
return ctx.remote === false ? {} : { sourceId: ctx.sourceId };
|
||||
}
|
||||
if (ctx.sourceId) return { sourceId: ctx.sourceId };
|
||||
return {};
|
||||
}
|
||||
@@ -551,7 +563,7 @@ export function resolveRequestedScope(
|
||||
sourceIdParam: string | undefined,
|
||||
allSourcesParam = false,
|
||||
): { sourceId?: string; sourceIds?: string[] } {
|
||||
const wantsAll = allSourcesParam || sourceIdParam === '__all__';
|
||||
const wantsAll = allSourcesParam || sourceIdParam === ALL_SOURCES;
|
||||
if (wantsAll) {
|
||||
return ctx.remote === false ? {} : sourceScopeOpts(ctx);
|
||||
}
|
||||
@@ -1625,6 +1637,10 @@ const search: Operation = {
|
||||
// agent-warning channel (hybridSearch stamps it; this branch bypasses
|
||||
// hybridSearch, so stamp explicitly). Fail-open inside the helper.
|
||||
await stampContentFlags(ctx.engine, results);
|
||||
// #160: same for the unverified auto-extracted stub marker (no boost
|
||||
// to cancel on this path — keyword-only never applies the compiled-
|
||||
// truth boost — but the provenance marker must still surface).
|
||||
await stampUnverifiedExtractions(ctx.engine, results);
|
||||
bumpLastRetrievedAt(ctx.engine, results.map((r) => r.page_id));
|
||||
maybeCaptureSearch(ctx, queryText, results, Date.now() - startedAt, false);
|
||||
return results;
|
||||
@@ -5671,6 +5687,208 @@ const chronicle_backfill: Operation = {
|
||||
cliHints: { name: 'chronicle-backfill' },
|
||||
};
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Extraction quarantine lane (issue #160)
|
||||
//
|
||||
// `extractAndEnrich` regex-extracts entity names from arbitrary text and
|
||||
// creates people/ + companies/ stub pages. These three ops are its ONLY
|
||||
// sanctioned surface:
|
||||
// - extract_entities — run extraction. Direct authoritative writes need
|
||||
// BOTH the trusted local CLI (ctx.remote === false)
|
||||
// AND the explicit --trusted-extraction flag;
|
||||
// everything else lands in the quarantine lane
|
||||
// (frontmatter provenance/status markers).
|
||||
// - extraction_pending — list unverified stubs awaiting review.
|
||||
// - extraction_review — promote (status → verified) or reject
|
||||
// (soft-delete) in batch. Owner-only (fail-closed
|
||||
// on ctx.remote): THIS surface never lets a remote
|
||||
// caller flip the status markers. Scope note: the
|
||||
// markers are ordinary frontmatter, so a caller who
|
||||
// already holds generic remote put_page write scope
|
||||
// can rewrite the page (markers included) — that
|
||||
// caller could equally author an unmarked people/
|
||||
// page directly, so the lane adds no privilege
|
||||
// there; put_page authz is its own boundary.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
// Resource guards for extract_entities (#160 hardening): bound the work a
|
||||
// single remote write-scope call can trigger. ponytail: flat caps; make them
|
||||
// config knobs only if a real workload hits them.
|
||||
const MAX_EXTRACT_TEXT_CHARS = 200_000;
|
||||
const MAX_EXTRACT_ENTITIES = 200;
|
||||
|
||||
const extract_entities: Operation = {
|
||||
name: 'extract_entities',
|
||||
description: 'Extract entity names (people, companies) from text and create/update their brain stub pages. Stubs from untrusted input land in the quarantine lane (frontmatter `provenance: auto-extracted` + `status: unverified`) — excluded from authoritative retrieval boosts until reviewed. Direct authoritative writes require the trusted local CLI AND --trusted-extraction.',
|
||||
params: {
|
||||
text: { type: 'string', required: true, description: 'The text to extract entities from (email, transcript, pasted content, …). Max 200k characters — split larger inputs.' },
|
||||
source_slug: { type: 'string', required: true, description: 'Slug of the source page the text came from (used for backlinks + timeline attribution).' },
|
||||
trusted_extraction: { type: 'boolean', required: false, description: 'Local CLI only: write stubs directly as authoritative pages, skipping the quarantine lane. Ignored (always quarantined) for remote callers.' },
|
||||
},
|
||||
mutating: true,
|
||||
scope: 'write',
|
||||
handler: async (ctx, p) => {
|
||||
// Trust rule (#160, fail-closed like the CV6 provenance gate above):
|
||||
// `ctx.remote === false` is the ONLY truthy condition that can admit a
|
||||
// direct authoritative write, and even then the caller must opt in
|
||||
// explicitly. Remote/unset trust → quarantine lane, flag ignored.
|
||||
const trusted = ctx.remote === false && p.trusted_extraction === true;
|
||||
const text = p.text as string;
|
||||
// Resource guards: the greedy name regex on a huge paste can yield tens
|
||||
// of thousands of "entities", each costing several DB round-trips. Cap
|
||||
// input size loudly and entity count softly (surfaced as `truncated`).
|
||||
if (text.length > MAX_EXTRACT_TEXT_CHARS) {
|
||||
throw new OperationError(
|
||||
'invalid_params',
|
||||
`extract_entities: text is ${text.length} chars (max ${MAX_EXTRACT_TEXT_CHARS}).`,
|
||||
'Split the input and call extract_entities per section.',
|
||||
);
|
||||
}
|
||||
if (ctx.dryRun) return { dry_run: true, action: 'extract_entities', trusted };
|
||||
const { extractEntities, enrichEntities } = await import('./enrichment-service.ts');
|
||||
const found = extractEntities(text);
|
||||
const capped = found.slice(0, MAX_EXTRACT_ENTITIES);
|
||||
const results = await enrichEntities(
|
||||
ctx.engine,
|
||||
capped.map((e) => ({ entityName: e.name, entityType: e.type, context: e.context, sourceSlug: p.source_slug as string })),
|
||||
{
|
||||
trusted,
|
||||
...(ctx.sourceId ? { sourceId: ctx.sourceId } : {}),
|
||||
// Pure local DB writes — no external API call to pace, so the
|
||||
// system-load capacity gate would only stall the caller.
|
||||
throttle: false,
|
||||
},
|
||||
);
|
||||
return {
|
||||
status: 'ok',
|
||||
trusted,
|
||||
quarantined: results.filter((r) => r.quarantined === true).length,
|
||||
count: results.length,
|
||||
entities_found: found.length,
|
||||
truncated: found.length > capped.length,
|
||||
entities: results,
|
||||
};
|
||||
},
|
||||
cliHints: { name: 'extract-entities' },
|
||||
};
|
||||
|
||||
const extraction_pending: Operation = {
|
||||
name: 'extraction_pending',
|
||||
description: 'List unverified auto-extracted entity stubs awaiting owner review (the quarantine lane from extract_entities). Promote or reject them with extraction_review.',
|
||||
params: {
|
||||
limit: { type: 'number', required: false, description: 'Max rows (default 100, cap 500).' },
|
||||
offset: { type: 'number', required: false, description: 'Pagination offset.' },
|
||||
},
|
||||
scope: 'read',
|
||||
handler: async (ctx, p) => {
|
||||
const limit = Math.min(Math.max(Number(p.limit ?? 100) || 100, 1), 500);
|
||||
const offset = Math.max(Number(p.offset ?? 0) || 0, 0);
|
||||
// Read-side source isolation: route through sourceScopeOpts (federated
|
||||
// array > scalar > nothing), applied in SQL below.
|
||||
const scope = sourceScopeOpts(ctx);
|
||||
const params: unknown[] = [];
|
||||
let srcClause = '';
|
||||
if (scope.sourceIds && scope.sourceIds.length > 0) {
|
||||
params.push(scope.sourceIds);
|
||||
srcClause = `AND p.source_id = ANY($${params.length}::text[])`;
|
||||
} else if (scope.sourceId) {
|
||||
params.push(scope.sourceId);
|
||||
srcClause = `AND p.source_id = $${params.length}`;
|
||||
}
|
||||
params.push(limit, offset);
|
||||
const rows = await ctx.engine.executeRaw<{
|
||||
slug: string; title: string; type: string; source_id: string;
|
||||
extracted_from: string | null; created_at: string;
|
||||
}>(
|
||||
`SELECT p.slug, p.title, p.type, p.source_id,
|
||||
p.frontmatter ->> 'source' AS extracted_from,
|
||||
p.created_at::text AS created_at
|
||||
FROM pages p
|
||||
JOIN sources s ON s.id = p.source_id
|
||||
WHERE ${unverifiedExtractionFragment('p')}
|
||||
${buildVisibilityClause('p', 's')}
|
||||
${srcClause}
|
||||
ORDER BY p.created_at DESC
|
||||
LIMIT $${params.length - 1} OFFSET $${params.length}`,
|
||||
params,
|
||||
);
|
||||
return { count: rows.length, pending: rows };
|
||||
},
|
||||
cliHints: { name: 'extraction-pending' },
|
||||
};
|
||||
|
||||
const extraction_review: Operation = {
|
||||
name: 'extraction_review',
|
||||
description: 'Promote or reject unverified auto-extracted entity stubs (batch). Promote flips `status` to verified (provenance kept for audit); reject soft-deletes the stub. Owner-only: this op is refused for any non-local caller. (The markers are ordinary frontmatter — the boundary against rewriting them wholesale is put_page write authz, same as for any page.)',
|
||||
params: {
|
||||
action: { type: 'string', required: true, description: "'promote' or 'reject'." },
|
||||
slugs: { type: 'array', required: true, items: { type: 'string' }, description: 'Stub slugs to act on (batch).' },
|
||||
},
|
||||
mutating: true,
|
||||
scope: 'write',
|
||||
localOnly: true,
|
||||
handler: async (ctx, p) => {
|
||||
// The review decision IS the trust gate — if a remote caller could
|
||||
// promote, injected content could self-promote and the quarantine lane
|
||||
// would be decorative. Fail-closed: only strictly-local callers pass.
|
||||
if (ctx.remote !== false) {
|
||||
throw new OperationError(
|
||||
'permission_denied',
|
||||
'extraction_review is owner-only: promote/reject decisions must come from the trusted local CLI.',
|
||||
'Run `gbrain extraction-review <promote|reject> --slugs ...` on the host machine.',
|
||||
);
|
||||
}
|
||||
const action = p.action as string;
|
||||
if (action !== 'promote' && action !== 'reject') {
|
||||
throw new OperationError('invalid_params', `extraction_review: action must be 'promote' or 'reject'; got '${action}'.`);
|
||||
}
|
||||
// CLI passes `--slugs a,b,c` as one string; MCP passes a real array.
|
||||
const slugs = Array.isArray(p.slugs)
|
||||
? (p.slugs as string[])
|
||||
: typeof p.slugs === 'string'
|
||||
? p.slugs.split(',').map((s) => s.trim()).filter(Boolean)
|
||||
: [];
|
||||
if (slugs.length === 0) {
|
||||
throw new OperationError('invalid_params', 'extraction_review: slugs must be a non-empty array (CLI: --slugs slug1,slug2).');
|
||||
}
|
||||
if (ctx.dryRun) return { dry_run: true, action: `extraction_review:${action}`, slugs };
|
||||
const results: Array<{ slug: string; status: string }> = [];
|
||||
for (const slug of slugs) {
|
||||
const page = await ctx.engine.getPage(slug, ctx.sourceId ? { sourceId: ctx.sourceId } : undefined);
|
||||
if (!page) {
|
||||
results.push({ slug, status: 'not_found' });
|
||||
continue;
|
||||
}
|
||||
if (!isUnverifiedExtraction(page.frontmatter)) {
|
||||
results.push({ slug, status: 'not_unverified' });
|
||||
continue;
|
||||
}
|
||||
if (action === 'promote') {
|
||||
// Frontmatter-only flip via a targeted JSONB merge — NOT putPage,
|
||||
// whose upsert would reset non-carried columns (page_kind →
|
||||
// 'markdown', content_hash, …) for a change that only touches one
|
||||
// frontmatter key. provenance stays 'auto-extracted' as the audit
|
||||
// trail of HOW the page came to exist; status → 'verified' records
|
||||
// the owner's call. jsonb_build_object binds as text (no
|
||||
// JSON.stringify-into-::jsonb hazard); identical on both engines.
|
||||
await ctx.engine.executeRaw(
|
||||
`UPDATE pages
|
||||
SET frontmatter = COALESCE(frontmatter, '{}'::jsonb) || jsonb_build_object($1::text, $2::text),
|
||||
updated_at = now()
|
||||
WHERE slug = $3 AND source_id = $4`,
|
||||
[EXTRACTION_STATUS_KEY, STATUS_VERIFIED, slug, page.source_id],
|
||||
);
|
||||
results.push({ slug, status: 'promoted' });
|
||||
} else {
|
||||
await ctx.engine.softDeletePage(slug, { sourceId: page.source_id });
|
||||
results.push({ slug, status: 'rejected' });
|
||||
}
|
||||
}
|
||||
return { status: 'ok', action, results };
|
||||
},
|
||||
cliHints: { name: 'extraction-review', positional: ['action'] },
|
||||
};
|
||||
|
||||
export const operations: Operation[] = [
|
||||
// Page CRUD
|
||||
get_page, put_page, delete_page, list_pages,
|
||||
@@ -5727,6 +5945,8 @@ export const operations: Operation[] = [
|
||||
volunteer_chronicle, chronicle_backfill,
|
||||
// v0.43 (#2095): push-based context
|
||||
volunteer_context,
|
||||
// Extraction quarantine lane (#160): gated entity extraction + review queue
|
||||
extract_entities, extraction_pending, extraction_review,
|
||||
// v0.31: hot memory (facts table)
|
||||
extract_facts, recall, forget_fact,
|
||||
// v0.32.6: contradiction probe MCP surface (M3)
|
||||
|
||||
@@ -72,9 +72,10 @@ export class SlugRegistryError extends Error {
|
||||
// SlugRegistry
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
// Shares the page-slug segment grammar (incl. CJK ranges, #738) with
|
||||
// Shares the page-slug segment grammar (all scripts, #738/#3417) with
|
||||
// validatePageSlug; keeps this site's dir/name shape (>= 2 segments).
|
||||
const SLUG_RE = new RegExp(`^${PAGE_SLUG_SEG}(\\/${PAGE_SLUG_SEG})+$`);
|
||||
// `u` flag required by PAGE_SLUG_SEG's \p{...} classes.
|
||||
const SLUG_RE = new RegExp(`^${PAGE_SLUG_SEG}(\\/${PAGE_SLUG_SEG})+$`, 'u');
|
||||
|
||||
export class SlugRegistry {
|
||||
constructor(private engine: BrainEngine) {}
|
||||
|
||||
+61
-12
@@ -58,6 +58,7 @@ import { finalizeLastSeen } from './chronicle/last-seen.ts';
|
||||
import { computeAnomaliesFromBuckets } from './cycle/anomaly.ts';
|
||||
import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts';
|
||||
import { buildSourceFactorCase, buildHardExcludeClause, buildVisibilityClause, buildRecencyComponentSql, buildBestPerPagePoolCte, buildOrFallbackWebsearchQuery } from './search/sql-ranking.ts';
|
||||
import { unverifiedExtractionFragment } from './extraction-review.ts';
|
||||
import { shouldExcludeFromOrphanReporting, loadOrphanPolicyOverrides } from './orphan-policy.ts';
|
||||
import { LINK_EXTRACTOR_VERSION_TS } from './link-extraction.ts';
|
||||
import {
|
||||
@@ -419,8 +420,10 @@ export class PGLiteEngine implements BrainEngine {
|
||||
let model: string = DEFAULT_EMBEDDING_MODEL;
|
||||
try {
|
||||
const gw = await import('./ai/gateway.ts');
|
||||
// Both accessors THROW when the gateway is unconfigured (they never
|
||||
// return falsy), so the catch below is the only fallback path (#3461).
|
||||
dims = gw.getEmbeddingDimensions();
|
||||
model = gw.getEmbeddingModel() || model;
|
||||
model = gw.getEmbeddingModel();
|
||||
} catch { /* gateway not configured — use defaults */ }
|
||||
|
||||
await this.db.exec(getPGLiteSchema(dims, model));
|
||||
@@ -978,7 +981,8 @@ export class PGLiteEngine implements BrainEngine {
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT id, source_id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at, deleted_at,
|
||||
effective_date, effective_date_source,
|
||||
source_kind, source_uri, ingested_via, ingested_at
|
||||
source_kind, source_uri, ingested_via, ingested_at,
|
||||
contextual_retrieval_mode
|
||||
FROM pages WHERE ${where.join(' AND ')} LIMIT 1`,
|
||||
params
|
||||
);
|
||||
@@ -2072,7 +2076,10 @@ export class PGLiteEngine implements BrainEngine {
|
||||
// Built on the bare `slug` output column: applied inside the `scored` CTE
|
||||
// whose FROM is the single relation `hnsw_candidates`, so unqualified
|
||||
// `slug` resolves cleanly (T1 per-page pool restructure).
|
||||
const sourceFactorCaseOnSlug = buildSourceFactorCase('slug', boostMap, opts?.detail);
|
||||
// issue #160: guard predicate projected as `unverified_stub` in
|
||||
// hnsw_candidates (parity with postgres-engine) so unverified stubs get
|
||||
// factor 1.0, not the people/ 1.2x, inside the pre-LIMIT re-rank.
|
||||
const sourceFactorCaseOnSlug = buildSourceFactorCase('slug', boostMap, opts?.detail, 'unverified_stub');
|
||||
const hardExcludePrefixes = resolveHardExcludes(opts?.exclude_slug_prefixes, opts?.include_slug_prefixes);
|
||||
const hardExcludeClause = buildHardExcludeClause('p.slug', hardExcludePrefixes);
|
||||
const innerLimit = offset + Math.max(limit * 5, 100);
|
||||
@@ -2148,6 +2155,7 @@ export class PGLiteEngine implements BrainEngine {
|
||||
CASE WHEN NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL
|
||||
THEN NULLIF(p.frontmatter->>'subject', '') END AS source_subject,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
(${unverifiedExtractionFragment('p')}) AS unverified_stub,
|
||||
1 - (cc.${col} <=> ${castSql}) AS raw_score
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
@@ -2315,15 +2323,26 @@ export class PGLiteEngine implements BrainEngine {
|
||||
|
||||
// Provenance fallback for chunks without an explicit `model`: resolve the
|
||||
// gateway's runtime model, not the compile-time DEFAULT_EMBEDDING_MODEL.
|
||||
// See postgres-engine.ts _upsertChunksOnce for the full rationale — pglite
|
||||
// mirrors it for parity.
|
||||
let resolvedModel: string = DEFAULT_EMBEDDING_MODEL;
|
||||
// #3461: getEmbeddingModel() THROWS when unconfigured (never returns
|
||||
// falsy) — on the throw path fall back to the brain's own
|
||||
// `config.embedding_model` row, then the compile-time default as the
|
||||
// last resort. See postgres-engine.ts _upsertChunksOnce for the full
|
||||
// rationale — pglite mirrors it for parity.
|
||||
let resolvedModel: string | null = null;
|
||||
try {
|
||||
const gw = await import('./ai/gateway.ts');
|
||||
resolvedModel = gw.getEmbeddingModel() || resolvedModel;
|
||||
resolvedModel = gw.getEmbeddingModel();
|
||||
} catch {
|
||||
// Gateway unconfigured (unit tests / pre-connect): keep the default.
|
||||
try {
|
||||
const cfg = await this.db.query(
|
||||
`SELECT value FROM config WHERE key = 'embedding_model'`,
|
||||
);
|
||||
resolvedModel = ((cfg.rows[0] as { value?: string } | undefined)?.value) ?? null;
|
||||
} catch {
|
||||
// config table unreadable — fall through to the compile-time default.
|
||||
}
|
||||
}
|
||||
if (!resolvedModel) resolvedModel = DEFAULT_EMBEDDING_MODEL;
|
||||
|
||||
for (const chunk of chunks) {
|
||||
const embeddingStr = chunk.embedding
|
||||
@@ -2376,6 +2395,9 @@ export class PGLiteEngine implements BrainEngine {
|
||||
// Code-chunk metadata columns follow the same chunk_text-gated CASE pattern as `embedding`
|
||||
// (#769). Re-chunk trusts EXCLUDED outright; pure re-embed COALESCEs so a caller carrying
|
||||
// only embedding-shaped fields doesn't clobber metadata to NULL.
|
||||
//
|
||||
// #3461: `model` mirrors the `embedding` CASE branch-for-branch so the label always
|
||||
// describes whichever vector wins the upsert. See postgres-engine.ts for rationale.
|
||||
await this.db.query(
|
||||
`INSERT INTO content_chunks ${cols} VALUES ${rowParts.join(', ')}
|
||||
ON CONFLICT (page_id, chunk_index) DO UPDATE SET
|
||||
@@ -2389,7 +2411,14 @@ export class PGLiteEngine implements BrainEngine {
|
||||
THEN EXCLUDED.embedding
|
||||
ELSE content_chunks.embedding
|
||||
END,
|
||||
model = COALESCE(EXCLUDED.model, content_chunks.model),
|
||||
model = CASE
|
||||
WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.model
|
||||
WHEN content_chunks.embedding IS NULL THEN EXCLUDED.model
|
||||
WHEN EXCLUDED.embedded_at IS NOT NULL
|
||||
AND (content_chunks.embedded_at IS NULL OR EXCLUDED.embedded_at > content_chunks.embedded_at)
|
||||
THEN EXCLUDED.model
|
||||
ELSE content_chunks.model
|
||||
END,
|
||||
token_count = EXCLUDED.token_count,
|
||||
embedded_at = CASE
|
||||
WHEN EXCLUDED.chunk_text != content_chunks.chunk_text AND EXCLUDED.embedding IS NULL THEN NULL
|
||||
@@ -3418,6 +3447,20 @@ export class PGLiteEngine implements BrainEngine {
|
||||
return result;
|
||||
}
|
||||
|
||||
async getUnverifiedExtractionPageIds(pageIds: number[]): Promise<Set<number>> {
|
||||
if (pageIds.length === 0) return new Set();
|
||||
// Parity with PostgresEngine.getUnverifiedExtractionPageIds (issue #160).
|
||||
// Predicate is the shared unverifiedExtractionFragment so this query and
|
||||
// the SQL-side source-boost guard can never drift.
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT id FROM pages
|
||||
WHERE id = ANY($1::int[])
|
||||
AND ${unverifiedExtractionFragment('pages')}`,
|
||||
[pageIds]
|
||||
);
|
||||
return new Set((rows as { id: number }[]).map((r) => Number(r.id)));
|
||||
}
|
||||
|
||||
async getPageTimestamps(slugs: string[]): Promise<Map<string, Date>> {
|
||||
if (slugs.length === 0) return new Map();
|
||||
const { rows } = await this.db.query(
|
||||
@@ -4262,9 +4305,15 @@ export class PGLiteEngine implements BrainEngine {
|
||||
async deleteFactsForPage(
|
||||
slug: string,
|
||||
source_id: string,
|
||||
opts?: { excludeSourcePrefixes?: string[] },
|
||||
opts?: { excludeSourcePrefixes?: string[]; preserveExpiredLegacy?: boolean },
|
||||
): Promise<{ deleted: number }> {
|
||||
const prefixes = opts?.excludeSourcePrefixes;
|
||||
// #2646: keep soft-expired legacy rows (row_num NULL — never
|
||||
// fence-owned) so a fence reconcile can't destroy forget_fact's
|
||||
// legacy DB-only forget record.
|
||||
const expiredLegacyFilter = opts?.preserveExpiredLegacy
|
||||
? ` AND NOT (row_num IS NULL AND expired_at IS NOT NULL)`
|
||||
: '';
|
||||
if (prefixes && prefixes.length > 0) {
|
||||
// #1928: keep rows whose `source` matches an excluded prefix (e.g.
|
||||
// `cli:` conversation facts). COALESCE so NULL/empty-source fence rows
|
||||
@@ -4273,13 +4322,13 @@ export class PGLiteEngine implements BrainEngine {
|
||||
const result = await this.db.query(
|
||||
`DELETE FROM facts
|
||||
WHERE source_id = $1 AND source_markdown_slug = $2
|
||||
AND NOT (COALESCE(source, '') LIKE ANY($3::text[]))`,
|
||||
AND NOT (COALESCE(source, '') LIKE ANY($3::text[]))${expiredLegacyFilter}`,
|
||||
[source_id, slug, patterns],
|
||||
);
|
||||
return { deleted: result.affectedRows ?? 0 };
|
||||
}
|
||||
const result = await this.db.query(
|
||||
`DELETE FROM facts WHERE source_id = $1 AND source_markdown_slug = $2`,
|
||||
`DELETE FROM facts WHERE source_id = $1 AND source_markdown_slug = $2${expiredLegacyFilter}`,
|
||||
[source_id, slug],
|
||||
);
|
||||
return { deleted: result.affectedRows ?? 0 };
|
||||
|
||||
+65
-10
@@ -65,6 +65,7 @@ import { logConnectionEvent } from './connection-audit.ts';
|
||||
import { validateSlug, contentHash, rowToPage, rowToStalePage, rowToChunk, rowToSearchResult, parseEmbedding, tryParseEmbedding, takeRowToTake, takeHitRowToHit, isUndefinedTableError, warnOncePerProcess } from './utils.ts';
|
||||
import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts';
|
||||
import { buildSourceFactorCase, buildHardExcludeClause, buildVisibilityClause, buildRecencyComponentSql, buildBestPerPagePoolCte, buildOrFallbackWebsearchQuery } from './search/sql-ranking.ts';
|
||||
import { unverifiedExtractionFragment } from './extraction-review.ts';
|
||||
import { DEFAULT_EMBEDDING_MODEL, DEFAULT_EMBEDDING_DIMENSIONS } from './ai/defaults.ts';
|
||||
import { DELETE_BATCH_SIZE } from './engine-constants.ts';
|
||||
import { SOURCE_CONFIG_OBJECT_SQL } from './source-config-sql.ts';
|
||||
@@ -381,8 +382,10 @@ export class PostgresEngine implements BrainEngine {
|
||||
let model: string = DEFAULT_EMBEDDING_MODEL;
|
||||
try {
|
||||
const gw = await import('./ai/gateway.ts');
|
||||
// Both accessors THROW when the gateway is unconfigured (they never
|
||||
// return falsy), so the catch below is the only fallback path (#3461).
|
||||
dims = gw.getEmbeddingDimensions();
|
||||
model = gw.getEmbeddingModel() || model;
|
||||
model = gw.getEmbeddingModel();
|
||||
} catch { /* gateway not yet configured — use defaults */ }
|
||||
|
||||
const sqlText = getPostgresSchema(dims, model);
|
||||
@@ -1030,7 +1033,8 @@ export class PostgresEngine implements BrainEngine {
|
||||
const rows = await tx`
|
||||
SELECT id, source_id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at, deleted_at,
|
||||
effective_date, effective_date_source,
|
||||
source_kind, source_uri, ingested_via, ingested_at
|
||||
source_kind, source_uri, ingested_via, ingested_at,
|
||||
contextual_retrieval_mode
|
||||
FROM pages
|
||||
WHERE slug = ${slug} ${sourceCondition} ${deletedCondition}
|
||||
LIMIT 1
|
||||
@@ -2120,7 +2124,10 @@ export class PostgresEngine implements BrainEngine {
|
||||
// innerLimit scales with offset to preserve the pagination contract:
|
||||
// a fixed cap of 100 would silently empty offset > 100.
|
||||
const boostMap = resolveBoostMap();
|
||||
const sourceFactorCaseOnSlug = buildSourceFactorCase('slug', boostMap, opts?.detail);
|
||||
// issue #160: the guard predicate is projected as `unverified_stub` in
|
||||
// hnsw_candidates (frontmatter isn't otherwise available at re-rank), so
|
||||
// unverified auto-extracted stubs get factor 1.0, not the people/ 1.2x.
|
||||
const sourceFactorCaseOnSlug = buildSourceFactorCase('slug', boostMap, opts?.detail, 'unverified_stub');
|
||||
const hardExcludePrefixes = resolveHardExcludes(opts?.exclude_slug_prefixes, opts?.include_slug_prefixes);
|
||||
const hardExcludeClause = buildHardExcludeClause('p.slug', hardExcludePrefixes);
|
||||
const innerLimit = offset + Math.max(limit * 5, 100);
|
||||
@@ -2220,6 +2227,7 @@ export class PostgresEngine implements BrainEngine {
|
||||
CASE WHEN NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL
|
||||
THEN NULLIF(p.frontmatter->>'subject', '') END AS source_subject,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
(${unverifiedExtractionFragment('p')}) AS unverified_stub,
|
||||
1 - (cc.${col} <=> ${castSql}) AS raw_score
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
@@ -2432,14 +2440,28 @@ export class PostgresEngine implements BrainEngine {
|
||||
// hardcoded default (e.g. zeroentropyai:zembed-1) onto rows whose vectors
|
||||
// were produced by a different, config-resolved model — corrupting the
|
||||
// provenance that signature-drift staleness + dim-migration logic trust.
|
||||
// Mirrors the resolve-then-fallback pattern used for schema sizing above.
|
||||
let resolvedModel: string = DEFAULT_EMBEDDING_MODEL;
|
||||
//
|
||||
// #3461: getEmbeddingModel() THROWS when the gateway is unconfigured —
|
||||
// it never returns falsy — so an `||` guard here is dead code and the
|
||||
// catch path used to stamp the compile-time default onto rows whose
|
||||
// vectors came from the config-resolved provider. On the throw path we
|
||||
// now fall back to the brain's own `config.embedding_model` row (kept
|
||||
// current by init / migrate / retrieval-upgrade), which names the model
|
||||
// that actually produced this brain's vectors. The compile-time default
|
||||
// is the LAST resort (fresh brain whose config row doesn't exist yet).
|
||||
let resolvedModel: string | null = null;
|
||||
try {
|
||||
const gw = await import('./ai/gateway.ts');
|
||||
resolvedModel = gw.getEmbeddingModel() || resolvedModel;
|
||||
resolvedModel = gw.getEmbeddingModel();
|
||||
} catch {
|
||||
// Gateway unconfigured (unit tests / pre-connect): keep the default.
|
||||
try {
|
||||
const cfg = await sql`SELECT value FROM config WHERE key = 'embedding_model'`;
|
||||
resolvedModel = (cfg[0]?.value as string | undefined) ?? null;
|
||||
} catch {
|
||||
// config table unreadable — fall through to the compile-time default.
|
||||
}
|
||||
}
|
||||
if (!resolvedModel) resolvedModel = DEFAULT_EMBEDDING_MODEL;
|
||||
|
||||
for (const chunk of chunks) {
|
||||
const embeddingStr = chunk.embedding
|
||||
@@ -2503,6 +2525,11 @@ export class PostgresEngine implements BrainEngine {
|
||||
// pure re-embed (chunk_text unchanged) COALESCEs so a caller that only carries embedding
|
||||
// doesn't clobber metadata to NULL. Without this, every embed --stale pass nuked code-def's
|
||||
// primary index for thousands of chunks at once.
|
||||
//
|
||||
// #3461: `model` mirrors the `embedding` CASE branch-for-branch — the label must
|
||||
// describe whichever vector WINS the upsert. The old COALESCE(EXCLUDED.model, …)
|
||||
// relabeled preserved (older-model) vectors with the current gateway model on every
|
||||
// partial re-embed, corrupting provenance without changing the vector.
|
||||
await sql.unsafe(
|
||||
`INSERT INTO content_chunks ${cols} VALUES ${rows.join(', ')}
|
||||
ON CONFLICT (page_id, chunk_index) DO UPDATE SET
|
||||
@@ -2516,7 +2543,14 @@ export class PostgresEngine implements BrainEngine {
|
||||
THEN EXCLUDED.embedding
|
||||
ELSE content_chunks.embedding
|
||||
END,
|
||||
model = COALESCE(EXCLUDED.model, content_chunks.model),
|
||||
model = CASE
|
||||
WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.model
|
||||
WHEN content_chunks.embedding IS NULL THEN EXCLUDED.model
|
||||
WHEN EXCLUDED.embedded_at IS NOT NULL
|
||||
AND (content_chunks.embedded_at IS NULL OR EXCLUDED.embedded_at > content_chunks.embedded_at)
|
||||
THEN EXCLUDED.model
|
||||
ELSE content_chunks.model
|
||||
END,
|
||||
token_count = EXCLUDED.token_count,
|
||||
embedded_at = CASE
|
||||
WHEN EXCLUDED.chunk_text != content_chunks.chunk_text AND EXCLUDED.embedding IS NULL THEN NULL
|
||||
@@ -3571,6 +3605,20 @@ export class PostgresEngine implements BrainEngine {
|
||||
return result;
|
||||
}
|
||||
|
||||
async getUnverifiedExtractionPageIds(pageIds: number[]): Promise<Set<number>> {
|
||||
if (pageIds.length === 0) return new Set();
|
||||
const sql = this.sql;
|
||||
// Predicate is the shared unverifiedExtractionFragment (issue #160) so
|
||||
// this query and the SQL-side source-boost guard can never drift.
|
||||
const rows = await sql.unsafe(
|
||||
`SELECT id FROM pages
|
||||
WHERE id = ANY($1::int[])
|
||||
AND ${unverifiedExtractionFragment('pages')}`,
|
||||
[pageIds] as never,
|
||||
);
|
||||
return new Set((rows as unknown as { id: number }[]).map((r) => Number(r.id)));
|
||||
}
|
||||
|
||||
async getPageTimestamps(slugs: string[]): Promise<Map<string, Date>> {
|
||||
if (slugs.length === 0) return new Map();
|
||||
const sql = this.sql;
|
||||
@@ -4419,10 +4467,16 @@ export class PostgresEngine implements BrainEngine {
|
||||
async deleteFactsForPage(
|
||||
slug: string,
|
||||
source_id: string,
|
||||
opts?: { excludeSourcePrefixes?: string[] },
|
||||
opts?: { excludeSourcePrefixes?: string[]; preserveExpiredLegacy?: boolean },
|
||||
): Promise<{ deleted: number }> {
|
||||
const sql = this.sql;
|
||||
const prefixes = opts?.excludeSourcePrefixes;
|
||||
// #2646: keep soft-expired legacy rows (row_num NULL — never
|
||||
// fence-owned) so a fence reconcile can't destroy forget_fact's
|
||||
// legacy DB-only forget record.
|
||||
const expiredLegacyFilter = opts?.preserveExpiredLegacy
|
||||
? sql`AND NOT (row_num IS NULL AND expired_at IS NOT NULL)`
|
||||
: sql``;
|
||||
if (prefixes && prefixes.length > 0) {
|
||||
// #1928: keep rows whose `source` matches an excluded prefix (e.g.
|
||||
// `cli:` conversation facts). COALESCE so NULL/empty-source fence rows
|
||||
@@ -4433,11 +4487,12 @@ export class PostgresEngine implements BrainEngine {
|
||||
WHERE source_id = ${source_id}
|
||||
AND source_markdown_slug = ${slug}
|
||||
AND NOT (COALESCE(source, '') LIKE ANY(${patterns}))
|
||||
${expiredLegacyFilter}
|
||||
`;
|
||||
return { deleted: result.count ?? 0 };
|
||||
}
|
||||
const result = await sql`
|
||||
DELETE FROM facts WHERE source_id = ${source_id} AND source_markdown_slug = ${slug}
|
||||
DELETE FROM facts WHERE source_id = ${source_id} AND source_markdown_slug = ${slug} ${expiredLegacyFilter}
|
||||
`;
|
||||
return { deleted: result.count ?? 0 };
|
||||
}
|
||||
|
||||
@@ -76,6 +76,37 @@ export async function stampContentFlags(engine: BrainEngine, results: SearchResu
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Extraction quarantine lane (issue #160). Stamps `SearchResult.unverified`
|
||||
* for any result whose page is an unverified auto-extracted entity stub
|
||||
* (frontmatter `provenance: 'auto-extracted'` + `status: 'unverified'`).
|
||||
* MUST run PRE-fusion: rrfFusion/rrfFusionWeighted read the flag to skip the
|
||||
* COMPILED_TRUTH_BOOST for these pages, so a stub fabricated by hostile
|
||||
* ingested text ranks as ordinary content, never with entity authority.
|
||||
* One batched query over the candidate arms' page_ids. Fail-open on the
|
||||
* fetch (a marker-fetch failure must not break retrieval) — the boost then
|
||||
* applies, but the SQL-side source-boost guard still holds.
|
||||
*/
|
||||
export async function stampUnverifiedExtractions(
|
||||
engine: BrainEngine,
|
||||
results: SearchResult[],
|
||||
): Promise<void> {
|
||||
if (results.length === 0) return;
|
||||
try {
|
||||
const ids = [...new Set(
|
||||
results.map((r) => r.page_id).filter((n): n is number => typeof n === 'number' && Number.isFinite(n)),
|
||||
)];
|
||||
if (ids.length === 0) return;
|
||||
const unverified = await engine.getUnverifiedExtractionPageIds(ids);
|
||||
if (unverified.size === 0) return;
|
||||
for (const r of results) {
|
||||
if (unverified.has(r.page_id)) r.unverified = true;
|
||||
}
|
||||
} catch {
|
||||
// best-effort: never break retrieval.
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.42.20.0 — bounded drain (was an unbounded `Promise.allSettled`, codex
|
||||
* confirmed; TODOS retrofit). Mirrors `awaitPendingLastRetrievedWrites`: races
|
||||
@@ -1101,12 +1132,37 @@ export async function hybridSearch(
|
||||
// provider (Voyage, ZE) works fine.
|
||||
const { isAvailable } = await import('../ai/gateway.ts');
|
||||
const providerProbe = resolvedCol.embeddingModel || undefined;
|
||||
if (!isAvailable('embedding', providerProbe)) {
|
||||
// Image/both/unified routing embeds via the MULTIMODAL provider, not the
|
||||
// text provider — so a multimodal-only install (text provider absent) must
|
||||
// still reach the multimodal branch below. Probe the multimodal provider
|
||||
// explicitly and only short-circuit when neither the text provider nor (for
|
||||
// multimodal-routed queries) the multimodal provider is reachable. Without
|
||||
// this guard a multimodal-only install would fall to keyword-only here and
|
||||
// never run the image/unified vector path.
|
||||
const multimodalProviderProbe =
|
||||
cfgForColumn?.embedding_multimodal_model ?? 'voyage:voyage-multimodal-3';
|
||||
// The LLM intent tie-break (below) can escalate a regex-'text' query to
|
||||
// 'image'/'both'; account for that possibility so an ambiguous query on a
|
||||
// multimodal-only install still reaches the multimodal branch.
|
||||
const mayEscalateToMultimodal =
|
||||
earlyModality === 'text' &&
|
||||
resolvedMode.cross_modal_llm_intent &&
|
||||
isAmbiguousModalityQuery(query);
|
||||
const willTryMultimodal =
|
||||
(resolvedMode.unified_multimodal === true ||
|
||||
earlyModality === 'image' ||
|
||||
earlyModality === 'both' ||
|
||||
mayEscalateToMultimodal) &&
|
||||
isAvailable('embedding', multimodalProviderProbe);
|
||||
if (!isAvailable('embedding', providerProbe) && !willTryMultimodal) {
|
||||
// v0.43 — fuse the relational arm with keyword so typed-edge answers
|
||||
// survive on the no-embedding-provider path (the relational win is most
|
||||
// valuable exactly when vector is unavailable). The title arm fuses here
|
||||
// too — an exact-title lookup on a keyless install is precisely where
|
||||
// chunk-grain keyword FTS alone fails (D1).
|
||||
// issue #160: stamp unverified stubs BEFORE fusion so the compiled-truth
|
||||
// boost skips them (flag survives fusion's result spread).
|
||||
await stampUnverifiedExtractions(engine, [...keywordResults, ...titleResults, ...relationalList]);
|
||||
let noEmbedResults = keywordResults;
|
||||
if (relationalList.length > 0 || titleResults.length > 0) {
|
||||
const fk = opts?.rrfK ?? RRF_K;
|
||||
@@ -1233,7 +1289,10 @@ export async function hybridSearch(
|
||||
if (unifiedRouting) {
|
||||
try {
|
||||
const { isAvailable: aiIsAvailable, embedQueryMultimodal } = await import('../ai/gateway.ts');
|
||||
if (!aiIsAvailable('embedding')) {
|
||||
// Probe the MULTIMODAL provider, not the global default — on a
|
||||
// multimodal-only install the global default (text) is absent but the
|
||||
// multimodal provider is configured, and unified routing embeds via it.
|
||||
if (!aiIsAvailable('embedding', multimodalProviderProbe)) {
|
||||
throw new Error('gateway not configured for embedding — unified multimodal would also fail');
|
||||
}
|
||||
const unifiedEmbedding = await embedQueryMultimodal(query);
|
||||
@@ -1268,7 +1327,10 @@ export async function hybridSearch(
|
||||
// OR the embed throws, log a structured warning and fall through to text.
|
||||
try {
|
||||
const { isAvailable: aiIsAvailable, embedQueryMultimodal } = await import('../ai/gateway.ts');
|
||||
if (!aiIsAvailable('embedding')) {
|
||||
// Probe the MULTIMODAL provider, not the global default — the image side
|
||||
// embeds via the multimodal model, which may be configured even when the
|
||||
// text/global-default embedding provider is absent (multimodal-only).
|
||||
if (!aiIsAvailable('embedding', multimodalProviderProbe)) {
|
||||
throw new Error('gateway not configured for embedding — multimodal would also fail');
|
||||
}
|
||||
const imageEmbedding = await embedQueryMultimodal(query);
|
||||
@@ -1342,6 +1404,9 @@ export async function hybridSearch(
|
||||
// v0.43: fuse the relational arm with keyword via RRF so typed-edge
|
||||
// answers survive even when vector is unavailable. The title arm fuses
|
||||
// here too (same rationale as the no-embedding-provider path — D1).
|
||||
// issue #160: stamp unverified stubs BEFORE fusion (see the
|
||||
// no-embedding-provider path for rationale).
|
||||
await stampUnverifiedExtractions(engine, [...keywordResults, ...titleResults, ...relationalList]);
|
||||
let fallbackResults = keywordResults;
|
||||
if (relationalList.length > 0 || titleResults.length > 0) {
|
||||
const fk = opts?.rrfK ?? RRF_K;
|
||||
@@ -1431,6 +1496,10 @@ export async function hybridSearch(
|
||||
allLists.push({ list: relationalList, k: baseRrfK });
|
||||
}
|
||||
|
||||
// issue #160: stamp unverified auto-extracted stubs across ALL candidate
|
||||
// arms BEFORE fusion so the compiled-truth authority boost skips them.
|
||||
await stampUnverifiedExtractions(engine, allLists.flatMap((l) => l.list));
|
||||
|
||||
let fused = rrfFusionWeighted(allLists, detail !== 'high');
|
||||
|
||||
// Cosine re-scoring before dedup so semantically better chunks survive.
|
||||
@@ -1997,7 +2066,9 @@ export function rrfFusionWeighted(
|
||||
if (maxScore > 0) {
|
||||
for (const e of entries) {
|
||||
e.score = e.score / maxScore;
|
||||
const boost = applyBoost && e.result.chunk_source === 'compiled_truth' ? COMPILED_TRUTH_BOOST : 1.0;
|
||||
// issue #160: unverified auto-extracted stubs (stamped pre-fusion by
|
||||
// stampUnverifiedExtractions) never get the compiled-truth authority boost.
|
||||
const boost = applyBoost && e.result.chunk_source === 'compiled_truth' && e.result.unverified !== true ? COMPILED_TRUTH_BOOST : 1.0;
|
||||
e.score *= boost;
|
||||
}
|
||||
}
|
||||
@@ -2040,8 +2111,9 @@ export function rrfFusion(lists: SearchResult[][], k: number, applyBoost = true)
|
||||
const rawScore = e.score;
|
||||
e.score = e.score / maxScore;
|
||||
|
||||
// Apply compiled truth boost after normalization (skip for detail=high)
|
||||
const boost = applyBoost && e.result.chunk_source === 'compiled_truth' ? COMPILED_TRUTH_BOOST : 1.0;
|
||||
// Apply compiled truth boost after normalization (skip for detail=high;
|
||||
// skip for unverified auto-extracted stubs — issue #160)
|
||||
const boost = applyBoost && e.result.chunk_source === 'compiled_truth' && e.result.unverified !== true ? COMPILED_TRUTH_BOOST : 1.0;
|
||||
e.score *= boost;
|
||||
|
||||
if (DEBUG) {
|
||||
|
||||
@@ -18,6 +18,7 @@
|
||||
*/
|
||||
|
||||
import { quarantineFilterFragment } from '../quarantine.ts';
|
||||
import { unverifiedExtractionFragment } from '../extraction-review.ts';
|
||||
|
||||
/**
|
||||
* Escape `%`, `_`, and `\` so a string can be used as a LIKE prefix literal.
|
||||
@@ -63,6 +64,7 @@ export function buildSourceFactorCase(
|
||||
slugColumn: string,
|
||||
boostMap: Record<string, number>,
|
||||
detail: 'low' | 'medium' | 'high' | undefined,
|
||||
unverifiedGuardColumn?: string,
|
||||
): string {
|
||||
// Loose-string guard: agents passing `"HIGH"` or `"high "` over MCP/JSON
|
||||
// should still hit the temporal-bypass path. TypeScript narrows `detail`
|
||||
@@ -80,7 +82,26 @@ export function buildSourceFactorCase(
|
||||
`WHEN ${slugColumn} LIKE ${buildLikePrefixLiteral(prefix)} THEN ${factor}`
|
||||
).join(' ');
|
||||
|
||||
return `(CASE ${whens} ELSE 1.0 END)`;
|
||||
// Extraction quarantine lane (issue #160): unverified auto-extracted stubs
|
||||
// never receive the namespace-authority factor (people/ / companies/ 1.2x)
|
||||
// — they rank as ordinary content until promoted. Two forms:
|
||||
// - table-qualified slug column ('p.slug'): reference the sibling
|
||||
// `frontmatter` column inline via unverifiedExtractionFragment.
|
||||
// - bare column + `unverifiedGuardColumn`: the vector arm's re-rank CTE
|
||||
// has no frontmatter column, so its inner hnsw_candidates CTE projects
|
||||
// the predicate as a boolean (`... AS unverified_stub`) and passes the
|
||||
// column name here. Without this the 1.2x would apply INSIDE the
|
||||
// scored/best_per_page pipeline pre-LIMIT — an unverified stub could
|
||||
// outrank AND evict a legitimate page from the candidate pool, which
|
||||
// nothing downstream can restore.
|
||||
const alias = slugColumn.includes('.') ? slugColumn.split('.')[0] : null;
|
||||
const unverifiedGuard = unverifiedGuardColumn
|
||||
? `WHEN ${unverifiedGuardColumn} THEN 1.0 `
|
||||
: alias
|
||||
? `WHEN ${unverifiedExtractionFragment(alias)} THEN 1.0 `
|
||||
: '';
|
||||
|
||||
return `(CASE ${unverifiedGuard}${whens} ELSE 1.0 END)`;
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -33,6 +33,17 @@
|
||||
|
||||
export const SOURCE_ID_RE = /^[a-z0-9](?:[a-z0-9-]{0,30}[a-z0-9])?$/;
|
||||
|
||||
/**
|
||||
* Sentinel meaning "span every source" (#1712). Deliberately NOT a valid
|
||||
* source id (underscores are rejected by SOURCE_ID_RE), so it can never
|
||||
* collide with a real source, be created via `sources add`, or leak into
|
||||
* lock ids / path joins. The resolver's explicit/env tiers pass it through
|
||||
* verbatim; `sourceScopeOpts` translates it to an unscoped read for trusted
|
||||
* local callers and keeps it as an unsatisfiable literal for remote callers
|
||||
* (fail-closed).
|
||||
*/
|
||||
export const ALL_SOURCES = '__all__';
|
||||
|
||||
/** Returns true if the string matches the canonical source_id regex. */
|
||||
export function isValidSourceId(s: unknown): s is string {
|
||||
return typeof s === 'string' && SOURCE_ID_RE.test(s);
|
||||
|
||||
@@ -17,9 +17,13 @@ import { readFileSync, lstatSync, type Stats } from 'fs';
|
||||
import { join, dirname, resolve } from 'path';
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
import { isSourceFederated } from './sources-load.ts';
|
||||
import { SOURCE_ID_RE, isValidSourceId } from './source-id.ts';
|
||||
import { SOURCE_ID_RE, isValidSourceId, ALL_SOURCES } from './source-id.ts';
|
||||
import { isTrustedDotfile, realpathOrResolve } from './path-confine.ts';
|
||||
|
||||
// Re-export so scope-resolution call sites can import the sentinel from
|
||||
// either module (#1712).
|
||||
export { ALL_SOURCES };
|
||||
|
||||
const DOTFILE = '.gbrain-source';
|
||||
// Canonical SOURCE_ID_RE imported from `source-id.ts` (single source of truth).
|
||||
// Re-exported below as `__testing.SOURCE_ID_RE` for legacy test imports.
|
||||
@@ -83,8 +87,11 @@ export async function resolveSourceId(
|
||||
explicit: string | null | undefined,
|
||||
cwd: string = process.cwd(),
|
||||
): Promise<string> {
|
||||
// 1. Explicit flag wins.
|
||||
// 1. Explicit flag wins. The __all__ sentinel passes through verbatim
|
||||
// (#1712) — it is not a source id, so it skips both the regex and
|
||||
// assertSourceExists; sourceScopeOpts gives it span-everything semantics.
|
||||
if (explicit) {
|
||||
if (explicit === ALL_SOURCES) return ALL_SOURCES;
|
||||
if (!SOURCE_ID_RE.test(explicit)) {
|
||||
throw new Error(`Invalid --source value "${explicit}". Must match [a-z0-9-]{1,32}.`);
|
||||
}
|
||||
@@ -92,9 +99,10 @@ export async function resolveSourceId(
|
||||
return explicit;
|
||||
}
|
||||
|
||||
// 2. Env var.
|
||||
// 2. Env var. Same __all__ pass-through (#2140).
|
||||
const env = process.env.GBRAIN_SOURCE;
|
||||
if (env && env.length > 0) {
|
||||
if (env === ALL_SOURCES) return ALL_SOURCES;
|
||||
if (!SOURCE_ID_RE.test(env)) {
|
||||
throw new Error(`Invalid GBRAIN_SOURCE value "${env}". Must match [a-z0-9-]{1,32}.`);
|
||||
}
|
||||
@@ -173,6 +181,7 @@ export function resolveSourceIdEngineFree(
|
||||
cwd: string = process.cwd(),
|
||||
): string | null {
|
||||
if (explicit) {
|
||||
if (explicit === ALL_SOURCES) return ALL_SOURCES; // #1712 sentinel pass-through
|
||||
if (!SOURCE_ID_RE.test(explicit)) {
|
||||
throw new Error(`Invalid --source value "${explicit}". Must match [a-z0-9-]{1,32}.`);
|
||||
}
|
||||
@@ -180,6 +189,7 @@ export function resolveSourceIdEngineFree(
|
||||
}
|
||||
const env = process.env.GBRAIN_SOURCE;
|
||||
if (env && env.length > 0) {
|
||||
if (env === ALL_SOURCES) return ALL_SOURCES; // #2140 sentinel pass-through
|
||||
if (!SOURCE_ID_RE.test(env)) {
|
||||
throw new Error(`Invalid GBRAIN_SOURCE value "${env}". Must match [a-z0-9-]{1,32}.`);
|
||||
}
|
||||
@@ -315,8 +325,11 @@ export async function resolveSourceWithTier(
|
||||
explicit: string | null | undefined,
|
||||
cwd: string = process.cwd(),
|
||||
): Promise<{ source_id: string; tier: SourceTier; detail?: string }> {
|
||||
// 1. Explicit flag wins.
|
||||
// 1. Explicit flag wins. __all__ sentinel passes through verbatim (#1712).
|
||||
if (explicit) {
|
||||
if (explicit === ALL_SOURCES) {
|
||||
return { source_id: ALL_SOURCES, tier: 'flag', detail: `--source ${ALL_SOURCES} (spans all sources)` };
|
||||
}
|
||||
if (!SOURCE_ID_RE.test(explicit)) {
|
||||
throw new Error(`Invalid --source value "${explicit}". Must match [a-z0-9-]{1,32}.`);
|
||||
}
|
||||
@@ -324,9 +337,12 @@ export async function resolveSourceWithTier(
|
||||
return { source_id: explicit, tier: 'flag', detail: `--source ${explicit}` };
|
||||
}
|
||||
|
||||
// 2. Env var.
|
||||
// 2. Env var. Same __all__ pass-through (#2140).
|
||||
const env = process.env.GBRAIN_SOURCE;
|
||||
if (env && env.length > 0) {
|
||||
if (env === ALL_SOURCES) {
|
||||
return { source_id: ALL_SOURCES, tier: 'env', detail: `GBRAIN_SOURCE=${ALL_SOURCES} (spans all sources)` };
|
||||
}
|
||||
if (!SOURCE_ID_RE.test(env)) {
|
||||
throw new Error(`Invalid GBRAIN_SOURCE value "${env}". Must match [a-z0-9-]{1,32}.`);
|
||||
}
|
||||
|
||||
+13
-8
@@ -11,7 +11,7 @@
|
||||
* pathToSlug() → convert file paths to page slugs
|
||||
*/
|
||||
|
||||
import { CJK_SLUG_CHARS } from './cjk.ts';
|
||||
import { SLUG_WORD_CHARS } from './cjk.ts';
|
||||
// v0.37.7.0 #1169 submodule-detection helpers. Bottom-of-file already
|
||||
// aliases existsSync as `_existsSync` for other purposes; the top-of-file
|
||||
// import keeps the pruneDir helper's deps near its callsite.
|
||||
@@ -396,8 +396,10 @@ export function unsyncableReason(path: string, opts: SyncableOptions = {}): Sync
|
||||
|
||||
/**
|
||||
* Character class for the lowercase-canonical form of a slug segment after
|
||||
* slugifySegment() has run. Lowercase letters, digits, dots, underscores,
|
||||
* hyphens. Exposed so adjacent code (e.g. takes-fence holder validation,
|
||||
* slugifySegment() has run. Letters/numbers in any script (lowercase where
|
||||
* the script has case — #3417), dots, underscores, hyphens. Uses \p{...}
|
||||
* classes, so composed regexes need the `u` flag (this one carries it).
|
||||
* Exposed so adjacent code (e.g. takes-fence holder validation,
|
||||
* v0.32 EXP-4) can reuse the actual repo slug grammar instead of inventing
|
||||
* a stricter parallel one and emitting false-positive warnings on legitimate
|
||||
* `companies/acme.io` / `people/foo_bar` slugs (codex review #3).
|
||||
@@ -405,15 +407,18 @@ export function unsyncableReason(path: string, opts: SyncableOptions = {}): Sync
|
||||
* Pattern is the inner character class only (no anchors); callers wrap it
|
||||
* in `^...$` or compose it with prefixes like `(?:people|companies)/...`.
|
||||
*/
|
||||
export const SLUG_SEGMENT_PATTERN = new RegExp(`[a-z0-9._\\-${CJK_SLUG_CHARS}]+`);
|
||||
export const SLUG_SEGMENT_PATTERN = new RegExp(`[${SLUG_WORD_CHARS}._\\-]+`, 'u');
|
||||
|
||||
/**
|
||||
* Slugify a single path segment: lowercase, strip special chars, spaces → hyphens.
|
||||
* CJK ranges (Han / Hiragana / Katakana / Hangul Syllables) are preserved (v0.32.7).
|
||||
* NFC re-normalize after the NFD-strip-accents pass so Hangul Jamo recomposes back
|
||||
* into precomposed syllables that fall inside the whitelist.
|
||||
* Letters and numbers from EVERY script are preserved (#3417): previously only
|
||||
* Latin + CJK survived, so Hebrew/Arabic/Cyrillic/Greek/Thai/... filenames
|
||||
* collapsed to empty segments and distinct files silently merged onto one slug.
|
||||
* NFC re-normalize after the NFD-strip-accents pass so Hangul Jamo recomposes
|
||||
* back into precomposed syllables, and so NFD filenames (macOS) and NFC
|
||||
* filenames (Linux/git) of the same name produce the SAME slug.
|
||||
*/
|
||||
const SLUGIFY_KEEP_RE = new RegExp(`[^a-z0-9.\\s_\\-${CJK_SLUG_CHARS}]`, 'g');
|
||||
const SLUGIFY_KEEP_RE = new RegExp(`[^${SLUG_WORD_CHARS}.\\s_\\-]`, 'gu');
|
||||
|
||||
export function slugifySegment(segment: string): string {
|
||||
return segment
|
||||
|
||||
@@ -134,6 +134,7 @@ export const TAKES_FENCE_END = '<!--- gbrain:takes:end -->';
|
||||
import { SLUG_SEGMENT_PATTERN } from './sync.ts';
|
||||
export const HOLDER_REGEX = new RegExp(
|
||||
`^(?:world|brain|(?:people|companies)/${SLUG_SEGMENT_PATTERN.source}|${SLUG_SEGMENT_PATTERN.source})$`,
|
||||
'u', // required by SLUG_SEGMENT_PATTERN's \p{...} classes (#3417)
|
||||
);
|
||||
|
||||
/**
|
||||
|
||||
@@ -699,6 +699,17 @@ export interface SearchResult {
|
||||
* Absent when the page is clean.
|
||||
*/
|
||||
content_flag?: { reason: string; detail: string };
|
||||
/**
|
||||
* Extraction quarantine lane (issue #160): true when the result's page is
|
||||
* an unverified auto-extracted entity stub (frontmatter
|
||||
* `provenance: 'auto-extracted'` + `status: 'unverified'`). Such pages are
|
||||
* excluded from the compiled-truth authority boost and the namespace
|
||||
* source-boost — they rank as ordinary content — and this marker tells the
|
||||
* agent the page has NOT been reviewed by the owner. Stamped pre-fusion by
|
||||
* `stampUnverifiedExtractions` (hybrid.ts). Absent for reviewed/ordinary
|
||||
* pages.
|
||||
*/
|
||||
unverified?: boolean;
|
||||
/**
|
||||
* v0.36 (cross-modal wave): the chunk's modality discriminator from
|
||||
* content_chunks.modality. 'text' for the existing text-embedding rows,
|
||||
|
||||
@@ -110,6 +110,12 @@ export function rowToPage(row: Record<string, unknown>): Page {
|
||||
const sourceUri = row.source_uri === undefined ? undefined : (row.source_uri as string | null);
|
||||
const ingestedVia = row.ingested_via === undefined ? undefined : (row.ingested_via as string | null);
|
||||
const ingestedAt = readOptionalDate(row.ingested_at);
|
||||
// #3507: the CR tier the page was last embedded under (three-state, same
|
||||
// pattern as the provenance columns above). Re-embed paths (`embed --stale`
|
||||
// and friends) read this to reproduce the page's stored wrapping convention.
|
||||
const contextualRetrievalMode = row.contextual_retrieval_mode === undefined
|
||||
? undefined
|
||||
: (row.contextual_retrieval_mode as Page['contextual_retrieval_mode']);
|
||||
return {
|
||||
id: row.id as number,
|
||||
slug: row.slug as string,
|
||||
@@ -135,6 +141,7 @@ export function rowToPage(row: Record<string, unknown>): Page {
|
||||
...(sourceUri !== undefined && { source_uri: sourceUri }),
|
||||
...(ingestedVia !== undefined && { ingested_via: ingestedVia }),
|
||||
...(ingestedAt !== undefined && { ingested_at: ingestedAt }),
|
||||
...(contextualRetrievalMode !== undefined && { contextual_retrieval_mode: contextualRetrievalMode }),
|
||||
// v0.31.12: propagate source_id so downstream callers (embed, reconcile-links)
|
||||
// can thread it through getChunks / upsertChunks without defaulting to 'default'.
|
||||
// v0.32.8: Page.source_id is required. Every SELECT feeding rowToPage now
|
||||
|
||||
@@ -0,0 +1,68 @@
|
||||
/**
|
||||
* #3554 — resetGateway() must restore the test baseline, not unconfigure.
|
||||
*
|
||||
* The bunfig preload (test/helpers/legacy-embedding-preload.ts) pins the
|
||||
* gateway to openai:text-embedding-3-large @ 1536 at process start and
|
||||
* registers that config as the reset baseline. Before the fix,
|
||||
* resetGateway() wiped the pin to _config = null; the next file's beforeAll
|
||||
* engine-connect then reconfigured from the SHIPPED default (zembed-1 @
|
||||
* 1280) and every 1536-d fixture in that file failed with
|
||||
* `expected 1280 dimensions, not 1536`. Which file pairs collided depended
|
||||
* on shard bin-packing, so adding ANY test file reshuffled the mines.
|
||||
*
|
||||
* These assertions pin the contract so it cannot silently rot again.
|
||||
*/
|
||||
import { describe, test, expect, afterEach } from 'bun:test';
|
||||
import {
|
||||
configureGateway,
|
||||
resetGateway,
|
||||
__unconfigureGatewayForTests,
|
||||
__setChatTransportForTests,
|
||||
getEmbeddingModel,
|
||||
getEmbeddingDimensions,
|
||||
isAvailable,
|
||||
} from '../../src/core/ai/gateway.ts';
|
||||
|
||||
afterEach(() => resetGateway());
|
||||
|
||||
describe('resetGateway baseline restore (#3554)', () => {
|
||||
test('immediately after resetGateway(), the preload baseline is live', () => {
|
||||
resetGateway();
|
||||
expect(getEmbeddingModel()).toBe('openai:text-embedding-3-large');
|
||||
expect(getEmbeddingDimensions()).toBe(1536);
|
||||
});
|
||||
|
||||
test('resetGateway() overwrites a file-local config back to the baseline', () => {
|
||||
configureGateway({
|
||||
embedding_model: 'zeroentropyai:zembed-1',
|
||||
embedding_dimensions: 1280,
|
||||
env: {},
|
||||
});
|
||||
expect(getEmbeddingDimensions()).toBe(1280);
|
||||
resetGateway();
|
||||
expect(getEmbeddingModel()).toBe('openai:text-embedding-3-large');
|
||||
expect(getEmbeddingDimensions()).toBe(1536);
|
||||
});
|
||||
|
||||
test('resetGateway() still clears test transports (no stale transport leaks back)', () => {
|
||||
__setChatTransportForTests(async () => {
|
||||
throw new Error('should have been cleared');
|
||||
});
|
||||
resetGateway();
|
||||
// Baseline config sets no chat key in a keyless env, but the transport
|
||||
// seam itself must be gone: isAvailable('chat') short-circuits to true
|
||||
// whenever a chat transport is installed, so with a hard-unconfigured
|
||||
// gateway it can only be true if the transport survived the reset.
|
||||
__unconfigureGatewayForTests();
|
||||
expect(isAvailable('chat')).toBe(false);
|
||||
});
|
||||
|
||||
test('__unconfigureGatewayForTests() gives a genuinely unconfigured gateway', () => {
|
||||
__unconfigureGatewayForTests();
|
||||
expect(() => getEmbeddingDimensions()).toThrow(/not configured/);
|
||||
expect(isAvailable('embedding')).toBe(false);
|
||||
// And a plain reset brings the baseline back.
|
||||
resetGateway();
|
||||
expect(getEmbeddingDimensions()).toBe(1536);
|
||||
});
|
||||
});
|
||||
@@ -2,6 +2,7 @@ import { describe, test, expect, beforeEach, afterAll } from 'bun:test';
|
||||
import {
|
||||
configureGateway,
|
||||
resetGateway,
|
||||
__unconfigureGatewayForTests,
|
||||
isAvailable,
|
||||
embed,
|
||||
getEmbeddingModel,
|
||||
@@ -55,6 +56,9 @@ describe('gateway.isAvailable (silent-drop regression surface)', () => {
|
||||
beforeEach(() => resetGateway());
|
||||
|
||||
test('returns false when gateway not configured', () => {
|
||||
// resetGateway() restores the preload's test baseline (#3554); go
|
||||
// genuinely unconfigured for this one assertion.
|
||||
__unconfigureGatewayForTests();
|
||||
expect(isAvailable('embedding')).toBe(false);
|
||||
});
|
||||
|
||||
@@ -114,11 +118,12 @@ describe('gateway.isAvailable (silent-drop regression surface)', () => {
|
||||
// #1135 — an explicit expansion_model pointed at a chat-capable
|
||||
// OpenAI-compatible provider used to silently yield no expansion because
|
||||
// the recipe declared no expansion touchpoint.
|
||||
test('expansion available for chat-capable openai-compat providers (deepseek/groq/together)', () => {
|
||||
test('expansion available for chat-capable openai-compat providers (deepseek/groq/together/openrouter)', () => {
|
||||
const cases: Array<[string, Record<string, string>]> = [
|
||||
['deepseek:deepseek-chat', { DEEPSEEK_API_KEY: 'fake' }],
|
||||
['groq:llama-3.1-8b-instant', { GROQ_API_KEY: 'fake' }],
|
||||
['together:meta-llama/Llama-3.3-70B-Instruct-Turbo', { TOGETHER_API_KEY: 'fake' }],
|
||||
['openrouter:google/gemini-3-flash-preview', { OPENROUTER_API_KEY: 'fake' }],
|
||||
];
|
||||
for (const [model, env] of cases) {
|
||||
resetGateway();
|
||||
|
||||
@@ -10,7 +10,7 @@
|
||||
*/
|
||||
|
||||
import { afterAll, beforeAll, describe, expect, mock, test } from 'bun:test';
|
||||
import { configureGateway, resetGateway } from '../../src/core/ai/gateway.ts';
|
||||
import { capBatchItems, configureGateway, resetGateway } from '../../src/core/ai/gateway.ts';
|
||||
import { listRecipes, getRecipe } from '../../src/core/ai/recipes/index.ts';
|
||||
|
||||
describe('v0.32 #779: no_batch_cap suppresses the missing-max_batch_tokens warning', () => {
|
||||
@@ -49,6 +49,19 @@ describe('v0.32 #779: no_batch_cap suppresses the missing-max_batch_tokens warni
|
||||
expect(r!.touchpoints.embedding?.no_batch_cap).toBeUndefined();
|
||||
});
|
||||
|
||||
test('dashscope declares the documented 10-item embedding cap (max_batch_items: 10)', () => {
|
||||
// DashScope's OpenAI-compat /embeddings endpoint rejects >10-item batches
|
||||
// (documented Model Studio cap; concept from community PRs #2643/#2405).
|
||||
// max_batch_tokens stays as the aggregate token-size guard.
|
||||
const r = getRecipe('dashscope');
|
||||
expect(r, 'dashscope not registered').toBeDefined();
|
||||
expect(r!.touchpoints.embedding?.max_batch_items).toBe(10);
|
||||
expect(r!.touchpoints.embedding?.max_batch_tokens).toBe(8192);
|
||||
// 25 items pre-split into DashScope-sized groups of at most 10.
|
||||
const texts = Array.from({ length: 25 }, (_, i) => `t${i}`);
|
||||
expect(capBatchItems(texts, 10).map(b => b.length)).toEqual([10, 10, 5]);
|
||||
});
|
||||
|
||||
test('configureGateway does NOT warn for ollama/litellm/llama-server', () => {
|
||||
warnSpy.mockClear();
|
||||
resetGateway();
|
||||
|
||||
@@ -65,6 +65,18 @@ describe('recipe: openrouter', () => {
|
||||
).not.toThrow();
|
||||
});
|
||||
|
||||
test('3b. expansion reuses routed chat models and accepts arbitrary provider/model IDs', () => {
|
||||
const r = getRecipe('openrouter')!;
|
||||
expect(r.touchpoints.expansion).toBeDefined();
|
||||
expect(r.touchpoints.expansion!.models.length).toBeGreaterThanOrEqual(3);
|
||||
expect(() =>
|
||||
assertTouchpoint(r, 'expansion', 'some/provider-model'),
|
||||
).not.toThrow();
|
||||
expect(() =>
|
||||
assertTouchpoint(r, 'expansion', 'meta-llama/llama-future-2030'),
|
||||
).not.toThrow();
|
||||
});
|
||||
|
||||
test('4. chat models list — every entry matches provider/model shape (D5 regression)', () => {
|
||||
// Codex correction: pinning specific slugs creates false confidence (the
|
||||
// list is advisory; OR's catalog churns). The shape test catches the
|
||||
|
||||
@@ -0,0 +1,176 @@
|
||||
/**
|
||||
* #1712 (dupes #2289, #2140) — the `__all__` sentinel must work in EVERY
|
||||
* resolution tier, not just as a per-call `source_id` param.
|
||||
*
|
||||
* The bug: SOURCE_ID_RE forbids underscores, so `--source __all__` and
|
||||
* `GBRAIN_SOURCE=__all__` threw in the resolver; the CLI's makeContext
|
||||
* blanket-caught that and silently fell back to `sourceId: 'default'` —
|
||||
* making the documented span-everything flag STRICTLY NARROWER than passing
|
||||
* no flag at all (the catch also discarded the #2561/#3242 federated
|
||||
* widening). Meanwhile sourceScopeOpts treated a ctx.sourceId of '__all__'
|
||||
* as an unsatisfiable literal.
|
||||
*
|
||||
* Uses the literal '__all__' (not the ALL_SOURCES constant) so these tests
|
||||
* load and run behaviorally against pre-fix trees.
|
||||
*/
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { withEnv } from './helpers/with-env.ts';
|
||||
import {
|
||||
resolveSourceId,
|
||||
resolveSourceIdEngineFree,
|
||||
resolveSourceWithTier,
|
||||
} from '../src/core/source-resolver.ts';
|
||||
import {
|
||||
sourceScopeOpts,
|
||||
federatedSearchScope,
|
||||
type OperationContext,
|
||||
} from '../src/core/operations.ts';
|
||||
import type { BrainEngine } from '../src/core/engine.ts';
|
||||
|
||||
// Stub engine: registered sources + no local_path rows + no default config.
|
||||
function makeStub(registeredSources: string[]): BrainEngine {
|
||||
return {
|
||||
kind: 'pglite',
|
||||
executeRaw: async <T>(sql: string, params?: unknown[]): Promise<T[]> => {
|
||||
if (sql.includes('SELECT id FROM sources WHERE id = $1')) {
|
||||
const target = params?.[0];
|
||||
return registeredSources.includes(target as string)
|
||||
? [{ id: target } as unknown as T]
|
||||
: [];
|
||||
}
|
||||
return [];
|
||||
},
|
||||
getConfig: async () => null,
|
||||
} as unknown as BrainEngine;
|
||||
}
|
||||
|
||||
function ctxOf(overrides: Partial<OperationContext> = {}): OperationContext {
|
||||
return {
|
||||
engine: {} as any,
|
||||
config: {} as any,
|
||||
logger: console as any,
|
||||
dryRun: false,
|
||||
remote: true,
|
||||
sourceId: 'default',
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
// ── Resolver tiers pass the sentinel through verbatim ──────────────────
|
||||
|
||||
describe('source-resolver — __all__ sentinel pass-through', () => {
|
||||
test('resolveSourceId: explicit --source __all__ resolves (no regex throw, no existence check)', async () => {
|
||||
// '__all__' is deliberately NOT in the registered set — the sentinel
|
||||
// must skip assertSourceExists (it is not a source id).
|
||||
const id = await resolveSourceId(makeStub(['default']), '__all__', '/nonexistent');
|
||||
expect(id).toBe('__all__');
|
||||
});
|
||||
|
||||
test('resolveSourceId: GBRAIN_SOURCE=__all__ resolves (#2140)', async () => {
|
||||
await withEnv({ GBRAIN_SOURCE: '__all__' }, async () => {
|
||||
const id = await resolveSourceId(makeStub(['default']), null, '/nonexistent');
|
||||
expect(id).toBe('__all__');
|
||||
});
|
||||
});
|
||||
|
||||
test('resolveSourceIdEngineFree: explicit + env __all__ (thin-client path)', async () => {
|
||||
expect(resolveSourceIdEngineFree('__all__', '/nonexistent')).toBe('__all__');
|
||||
await withEnv({ GBRAIN_SOURCE: '__all__' }, () => {
|
||||
expect(resolveSourceIdEngineFree(null, '/nonexistent')).toBe('__all__');
|
||||
});
|
||||
});
|
||||
|
||||
test('resolveSourceWithTier: flag and env tiers carry the sentinel', async () => {
|
||||
const flag = await resolveSourceWithTier(makeStub(['default']), '__all__', '/nonexistent');
|
||||
expect(flag).toMatchObject({ source_id: '__all__', tier: 'flag' });
|
||||
await withEnv({ GBRAIN_SOURCE: '__all__' }, async () => {
|
||||
const env = await resolveSourceWithTier(makeStub(['default']), null, '/nonexistent');
|
||||
expect(env).toMatchObject({ source_id: '__all__', tier: 'env' });
|
||||
});
|
||||
});
|
||||
|
||||
test('a genuinely invalid --source still throws (SOURCE_ID_RE not loosened)', async () => {
|
||||
await expect(resolveSourceId(makeStub(['default']), 'my_source', '/nonexistent'))
|
||||
.rejects.toThrow(/Invalid --source/);
|
||||
expect(() => resolveSourceIdEngineFree('my_source', '/nonexistent'))
|
||||
.toThrow(/Invalid --source/);
|
||||
});
|
||||
});
|
||||
|
||||
// ── sourceScopeOpts — the single read-scope choke point ─────────────────
|
||||
|
||||
describe('sourceScopeOpts — __all__ sentinel', () => {
|
||||
test('trusted local (remote === false): spans the whole brain (empty scope)', () => {
|
||||
expect(sourceScopeOpts(ctxOf({ remote: false, sourceId: '__all__' }))).toEqual({});
|
||||
});
|
||||
|
||||
test('remote: keeps the unsatisfiable literal — fail-closed, never widens', () => {
|
||||
expect(sourceScopeOpts(ctxOf({ remote: true, sourceId: '__all__' })))
|
||||
.toEqual({ sourceId: '__all__' });
|
||||
});
|
||||
|
||||
test('anything not strictly remote === false is untrusted (fail-closed)', () => {
|
||||
// undefined / missing remote must behave like remote, per the trust rule.
|
||||
const ctx = ctxOf({ sourceId: '__all__' });
|
||||
(ctx as any).remote = undefined;
|
||||
expect(sourceScopeOpts(ctx)).toEqual({ sourceId: '__all__' });
|
||||
});
|
||||
|
||||
test('a federated grant always wins over the sentinel', () => {
|
||||
const ctx = ctxOf({
|
||||
remote: true,
|
||||
sourceId: '__all__',
|
||||
auth: { token: 't', clientId: 'c', scopes: [], allowedSources: ['a', 'b'] } as any,
|
||||
});
|
||||
expect(sourceScopeOpts(ctx)).toEqual({ sourceIds: ['a', 'b'] });
|
||||
});
|
||||
});
|
||||
|
||||
// ── Never narrower than passing no flag (#2561 regression shape) ────────
|
||||
|
||||
describe('__all__ is never narrower than an unqualified read', () => {
|
||||
test('local __all__ spans the brain even when federated widening exists', () => {
|
||||
// Unqualified read on a federated brain widens to the federated array…
|
||||
const unqualified = ctxOf({
|
||||
remote: false,
|
||||
sourceId: 'default',
|
||||
localFederatedSourceIds: ['default', 'src-a', 'src-b'],
|
||||
});
|
||||
expect(federatedSearchScope(unqualified)).toEqual({
|
||||
sourceIds: ['default', 'src-a', 'src-b'],
|
||||
});
|
||||
// …and __all__ must be a superset of that: the whole brain ({}).
|
||||
const all = ctxOf({ remote: false, sourceId: '__all__' });
|
||||
expect(federatedSearchScope(all)).toEqual({});
|
||||
});
|
||||
});
|
||||
|
||||
// ── makeContext — explicit --source failures error loudly ───────────────
|
||||
|
||||
describe('cli makeContext — no silent default fallback for explicit --source', () => {
|
||||
test('--source __all__ produces ctx.sourceId __all__ (was: silent default)', async () => {
|
||||
const { makeContext } = await import('../src/cli.ts');
|
||||
const ctx = await makeContext(makeStub(['default']), { source: '__all__' });
|
||||
expect(ctx.sourceId).toBe('__all__');
|
||||
expect(ctx.remote).toBe(false);
|
||||
});
|
||||
|
||||
test('an explicit --source that fails to resolve throws instead of becoming default', async () => {
|
||||
const { makeContext } = await import('../src/cli.ts');
|
||||
await expect(makeContext(makeStub(['default']), { source: 'ghost' }))
|
||||
.rejects.toThrow(/not found/);
|
||||
await expect(makeContext(makeStub(['default']), { source: 'my_source' }))
|
||||
.rejects.toThrow(/Invalid --source/);
|
||||
});
|
||||
|
||||
test('ambient resolution failure still falls back silently (pre-init brains)', async () => {
|
||||
const { makeContext } = await import('../src/cli.ts');
|
||||
const broken = {
|
||||
kind: 'pglite',
|
||||
executeRaw: async () => { throw new Error('relation "sources" does not exist'); },
|
||||
getConfig: async () => { throw new Error('relation "config" does not exist'); },
|
||||
} as unknown as BrainEngine;
|
||||
const ctx = await makeContext(broken, {});
|
||||
expect(ctx.sourceId).toBe('default');
|
||||
});
|
||||
});
|
||||
@@ -11,15 +11,34 @@ import { tmpdir } from 'os';
|
||||
import { execFileSync } from 'child_process';
|
||||
import { hardenBrainRepo } from '../src/core/brain-repo-durability.ts';
|
||||
|
||||
// #2943 root cause: `env: process.env` is REQUIRED here. Bun snapshots
|
||||
// process.env at startup, so without it the spawned git — and any post-commit
|
||||
// hook it fires — is blind to beforeEach's HOME/GBRAIN_HOME mutations (the
|
||||
// same Bun quirk as #2747, see resolveGbrainCliPath in brain-repo-durability).
|
||||
// Pre-fix, the hook under test resolved ${GBRAIN_HOME:-$HOME/.gbrain} to the
|
||||
// OPERATOR'S REAL ~/.gbrain: it wrote its log lines there (polluting the real
|
||||
// brain-push.log on every run), the LOCAL-ONLY test never saw them in the
|
||||
// temp log it polls, and the assertion only passed when the scaffolding push
|
||||
// from beforeEach (spawned by hardenBrainRepo WITH explicit env) happened to
|
||||
// still be in flight, lose the ref race, and retry AFTER the test had pointed
|
||||
// origin at the dead path — an accidental, load-dependent signal. That race
|
||||
// is the CI flake.
|
||||
function git(cwd: string, ...args: string[]): string {
|
||||
return execFileSync('git', ['-C', cwd, '-c', 'protocol.file.allow=always', ...args], {
|
||||
stdio: ['ignore', 'pipe', 'pipe'], encoding: 'utf-8',
|
||||
stdio: ['ignore', 'pipe', 'pipe'], encoding: 'utf-8', env: process.env,
|
||||
}).trim();
|
||||
}
|
||||
function originHead(bare: string): string {
|
||||
return git(bare, 'rev-parse', 'refs/heads/main');
|
||||
}
|
||||
async function waitForOrigin(bare: string, expectSha: string, ms = 8000): Promise<boolean> {
|
||||
// #2943: 30s poll deadlines (was 8s) for headroom under loaded CI shards —
|
||||
// the unreachable-origin path runs ~6 sequential process spawns after the
|
||||
// hook detaches. Every hook test also passes an explicit 60_000 third-arg
|
||||
// timeout: bun 1.3.14 IGNORES bunfig.toml's `timeout` key, so a bare
|
||||
// `bun test` enforces its 5000ms default and killed these tests before the
|
||||
// internal deadline could even elapse (the runner scripts pass --timeout
|
||||
// explicitly, which is why the inversion only bit direct local runs).
|
||||
async function waitForOrigin(bare: string, expectSha: string, ms = 30_000): Promise<boolean> {
|
||||
const deadline = Date.now() + ms;
|
||||
while (Date.now() < deadline) {
|
||||
try { if (originHead(bare) === expectSha) return true; } catch { /* */ }
|
||||
@@ -28,6 +47,24 @@ async function waitForOrigin(bare: string, expectSha: string, ms = 8000): Promis
|
||||
return false;
|
||||
}
|
||||
|
||||
/** #2943 (index.lock form): hardenBrainRepo installs the post-commit hook
|
||||
* BEFORE committing the scaffolding, so that commit fires the hook and
|
||||
* detaches a background brain_push. If that push loses the ref race against
|
||||
* hardenBrainRepo's own synchronous push, it falls back to `git pull
|
||||
* --rebase`, which takes .git/index.lock — racing the test body's first git
|
||||
* calls ("Unable to create '.../.git/index.lock': File exists"). Wait for the
|
||||
* detached push's terminal log line before handing the repo to the test. */
|
||||
async function waitForHookPushSettled(ms = 30_000): Promise<void> {
|
||||
const log = join(process.env.GBRAIN_HOME!, 'brain-push.log');
|
||||
const terminal = /\[push\] (ok|lock-timeout|LOCAL-ONLY)/;
|
||||
const deadline = Date.now() + ms;
|
||||
while (Date.now() < deadline) {
|
||||
if (existsSync(log) && terminal.test(readFileSync(log, 'utf-8'))) return;
|
||||
await new Promise(r => setTimeout(r, 150));
|
||||
}
|
||||
throw new Error(`detached hook push did not settle within ${ms}ms (${log})`);
|
||||
}
|
||||
|
||||
let root: string, work: string, bare: string;
|
||||
let oldHome: string | undefined, oldGbrainHome: string | undefined;
|
||||
|
||||
@@ -38,14 +75,15 @@ beforeEach(async () => {
|
||||
process.env.GBRAIN_HOME = join(process.env.HOME, '.gbrain');
|
||||
process.env.GBRAIN_GIT_ALLOW_FILE_TRANSPORT = '1';
|
||||
bare = mkdtempSync(join(root, 'origin-')) + '.git';
|
||||
execFileSync('git', ['init', '-q', '--bare', '-b', 'main', bare], { stdio: 'ignore' });
|
||||
execFileSync('git', ['init', '-q', '--bare', '-b', 'main', bare], { stdio: 'ignore', env: process.env });
|
||||
work = mkdtempSync(join(root, 'work-'));
|
||||
execFileSync('git', ['-c', 'protocol.file.allow=always', 'clone', '-q', bare, work], { stdio: 'ignore' });
|
||||
execFileSync('git', ['-c', 'protocol.file.allow=always', 'clone', '-q', bare, work], { stdio: 'ignore', env: process.env });
|
||||
git(work, 'config', 'user.email', 't@t.t'); git(work, 'config', 'user.name', 'tester');
|
||||
writeFileSync(join(work, 'README.md'), 'init\n');
|
||||
git(work, 'add', 'README.md'); git(work, 'commit', '-qm', 'init'); git(work, 'push', '-q', 'origin', 'main');
|
||||
git(work, 'remote', 'set-head', 'origin', 'main');
|
||||
await hardenBrainRepo({ repoPath: work, sourceId: 'wiki', pat: 'ghp_x', installCron: false });
|
||||
await waitForHookPushSettled();
|
||||
});
|
||||
afterEach(() => {
|
||||
if (oldHome === undefined) delete process.env.HOME; else process.env.HOME = oldHome;
|
||||
@@ -65,7 +103,7 @@ describe('brain-commit-push.sh (D13 guarantee)', () => {
|
||||
expect(originHead(bare)).toBe(git(work, 'rev-parse', 'HEAD'));
|
||||
// origin actually has the file
|
||||
const verify = mkdtempSync(join(root, 'verify-'));
|
||||
execFileSync('git', ['-c', 'protocol.file.allow=always', 'clone', '-q', bare, verify], { stdio: 'ignore' });
|
||||
execFileSync('git', ['-c', 'protocol.file.allow=always', 'clone', '-q', bare, verify], { stdio: 'ignore', env: process.env });
|
||||
expect(existsSync(join(verify, 'people', 'alice.md'))).toBe(true);
|
||||
});
|
||||
|
||||
@@ -102,7 +140,7 @@ describe('brain-commit-push.sh (D13 guarantee)', () => {
|
||||
rmSync(join(work, '.git', 'hooks', 'post-commit'));
|
||||
// Advance the remote from a second clone so a pull is genuinely needed.
|
||||
const other = mkdtempSync(join(root, 'other-'));
|
||||
execFileSync('git', ['-c', 'protocol.file.allow=always', 'clone', '-q', bare, other], { stdio: 'ignore' });
|
||||
execFileSync('git', ['-c', 'protocol.file.allow=always', 'clone', '-q', bare, other], { stdio: 'ignore', env: process.env });
|
||||
git(other, 'config', 'user.email', 'o@o.o'); git(other, 'config', 'user.name', 'other');
|
||||
writeFileSync(join(other, 'remote.md'), 'from other\n');
|
||||
git(other, 'add', 'remote.md'); git(other, 'commit', '-qm', 'remote change'); git(other, 'push', '-q', 'origin', 'main');
|
||||
@@ -128,26 +166,26 @@ describe('post-commit hook (D9 local, D7 self-contained)', () => {
|
||||
git(work, 'add', 'note.md'); git(work, 'commit', '-qm', 'note'); // fires .git/hooks/post-commit
|
||||
const head = git(work, 'rev-parse', 'HEAD');
|
||||
expect(await waitForOrigin(bare, head)).toBe(true);
|
||||
});
|
||||
}, 60_000);
|
||||
|
||||
test('the hook works even with the committed helper deleted (self-contained)', async () => {
|
||||
rmSync(join(work, 'scripts', 'brain-commit-push.sh'));
|
||||
git(work, 'add', '-A'); git(work, 'commit', '-qm', 'remove helper');
|
||||
const head = git(work, 'rev-parse', 'HEAD');
|
||||
expect(await waitForOrigin(bare, head)).toBe(true);
|
||||
});
|
||||
}, 60_000);
|
||||
|
||||
test('logs a clear LOCAL-ONLY line when origin is unreachable', async () => {
|
||||
git(work, 'remote', 'set-url', 'origin', join(root, 'gone2.git'));
|
||||
writeFileSync(join(work, 'orphan.md'), 'o\n');
|
||||
git(work, 'add', 'orphan.md'); git(work, 'commit', '-qm', 'orphan');
|
||||
const log = join(process.env.GBRAIN_HOME!, 'brain-push.log');
|
||||
const deadline = Date.now() + 8000;
|
||||
const deadline = Date.now() + 30_000;
|
||||
let found = false;
|
||||
while (Date.now() < deadline) {
|
||||
if (existsSync(log) && readFileSync(log, 'utf-8').includes('NEEDS ATTENTION')) { found = true; break; }
|
||||
await new Promise(r => setTimeout(r, 150));
|
||||
}
|
||||
expect(found).toBe(true);
|
||||
});
|
||||
}, 60_000);
|
||||
});
|
||||
|
||||
+159
-4
@@ -58,12 +58,23 @@ beforeEach(async () => {
|
||||
// Tiny gazetteer builder for pure-fn cases that don't need engine.
|
||||
function gazetteerFromEntries(entries: Omit<GazetteerEntry, 'tokens'>[]): Gazetteer {
|
||||
const TOKEN_RE = /[a-zA-Z0-9]+/g;
|
||||
const isCJK = (s: string): boolean => {
|
||||
const cp = s.codePointAt(0) ?? 0;
|
||||
return (cp >= 0x4e00 && cp <= 0x9fff) || (cp >= 0x3400 && cp <= 0x4dbf) ||
|
||||
(cp >= 0x3040 && cp <= 0x309f) || (cp >= 0x30a0 && cp <= 0x30ff) ||
|
||||
(cp >= 0xac00 && cp <= 0xd7af);
|
||||
};
|
||||
const hasCJKTitle = (s: string): boolean => [...s].some(isCJK);
|
||||
const tokenize = (s: string): string[] => {
|
||||
TOKEN_RE.lastIndex = 0;
|
||||
const out: string[] = [];
|
||||
let m: RegExpExecArray | null;
|
||||
while ((m = TOKEN_RE.exec(s)) !== null) out.push(m[0].toLowerCase());
|
||||
return out;
|
||||
if (!hasCJKTitle(s)) {
|
||||
const out: string[] = [];
|
||||
let m: RegExpExecArray | null;
|
||||
while ((m = TOKEN_RE.exec(s)) !== null) out.push(m[0].toLowerCase());
|
||||
return out;
|
||||
}
|
||||
// CJK: split into individual characters, lowercased.
|
||||
return [...s].map(c => isCJK(c) ? c.toLowerCase() : '').filter(Boolean);
|
||||
};
|
||||
const g: Gazetteer = new Map();
|
||||
for (const raw of entries) {
|
||||
@@ -259,6 +270,128 @@ describe('findMentionedEntities — pure cases', () => {
|
||||
});
|
||||
});
|
||||
|
||||
// ============================================================
|
||||
// CJK — entity extraction tests
|
||||
// ============================================================
|
||||
|
||||
describe('findMentionedEntities — CJK cases', () => {
|
||||
test('CJK single-name match — "纳瓦尔" in body → matched', () => {
|
||||
const g = gazetteerFromEntries([
|
||||
{ slug: 'people/naval', source_id: 'default', title: '纳瓦尔' },
|
||||
]);
|
||||
const mentions = findMentionedEntities('我最近读了纳瓦尔的书。', g, {
|
||||
fromSlug: 'writing/post-1', fromSourceId: 'default',
|
||||
});
|
||||
expect(mentions).toHaveLength(1);
|
||||
expect(mentions[0]!.slug).toBe('people/naval');
|
||||
expect(mentions[0]!.name).toBe('纳瓦尔');
|
||||
});
|
||||
|
||||
test('CJK multi-name — two different CJK entities in one body', () => {
|
||||
const g = gazetteerFromEntries([
|
||||
{ slug: 'people/naval', source_id: 'default', title: '纳瓦尔' },
|
||||
{ slug: 'people/shuang-xuetao', source_id: 'default', title: '双雪涛' },
|
||||
]);
|
||||
const mentions = findMentionedEntities('纳瓦尔和双雪涛都是作家。', g, {
|
||||
fromSlug: 'writing/post-1', fromSourceId: 'default',
|
||||
});
|
||||
expect(mentions).toHaveLength(2);
|
||||
const slugs = mentions.map(m => m.slug);
|
||||
expect(slugs).toContain('people/naval');
|
||||
expect(slugs).toContain('people/shuang-xuetao');
|
||||
});
|
||||
|
||||
test('CJK first-mention-only — repeated name → single link', () => {
|
||||
const g = gazetteerFromEntries([
|
||||
{ slug: 'people/naval', source_id: 'default', title: '纳瓦尔' },
|
||||
]);
|
||||
const mentions = findMentionedEntities('纳瓦尔说过。然后纳瓦尔又说过。', g, {
|
||||
fromSlug: 'writing/post-1', fromSourceId: 'default',
|
||||
});
|
||||
expect(mentions).toHaveLength(1);
|
||||
});
|
||||
|
||||
test('CJK self-link guard — entity page mentioning itself is skipped', () => {
|
||||
const g = gazetteerFromEntries([
|
||||
{ slug: 'people/naval', source_id: 'default', title: '纳瓦尔' },
|
||||
]);
|
||||
const mentions = findMentionedEntities('纳瓦尔是一位投资人。', g, {
|
||||
fromSlug: 'people/naval', fromSourceId: 'default',
|
||||
});
|
||||
expect(mentions).toEqual([]);
|
||||
});
|
||||
|
||||
test('CJK cross-source guard — entity in different source skipped', () => {
|
||||
const g = gazetteerFromEntries([
|
||||
{ slug: 'people/naval', source_id: 'team-b', title: '纳瓦尔' },
|
||||
]);
|
||||
const mentions = findMentionedEntities('纳瓦尔写了这本书。', g, {
|
||||
fromSlug: 'writing/post-1', fromSourceId: 'team-a',
|
||||
});
|
||||
expect(mentions).toEqual([]);
|
||||
});
|
||||
|
||||
test('CJK code-block stripping — CJK name inside ``` is skipped, outside matched', () => {
|
||||
const g = gazetteerFromEntries([
|
||||
{ slug: 'people/naval', source_id: 'default', title: '纳瓦尔' },
|
||||
]);
|
||||
// "纳瓦尔" only appears inside code block → should be skipped.
|
||||
const body = '```\n纳瓦尔\n```\n只有代码块里面有。';
|
||||
const mentions = findMentionedEntities(body, g, {
|
||||
fromSlug: 'writing/post-1', fromSourceId: 'default',
|
||||
});
|
||||
expect(mentions).toHaveLength(0);
|
||||
});
|
||||
|
||||
test('CJK determinism — same output across 10 calls', () => {
|
||||
const g = gazetteerFromEntries([
|
||||
{ slug: 'people/naval', source_id: 'default', title: '纳瓦尔' },
|
||||
{ slug: 'people/shuang-xuetao', source_id: 'default', title: '双雪涛' },
|
||||
]);
|
||||
const body = '纳瓦尔和双雪涛。纳瓦尔再说一次。';
|
||||
const refs = new Set<string>();
|
||||
for (let i = 0; i < 10; i++) {
|
||||
const mentions = findMentionedEntities(body, g, {
|
||||
fromSlug: 'writing/post-1', fromSourceId: 'default',
|
||||
});
|
||||
refs.add(JSON.stringify(mentions));
|
||||
}
|
||||
expect(refs.size).toBe(1);
|
||||
});
|
||||
|
||||
test('CJK mixed body — CJK entity matched in body with ASCII around it', () => {
|
||||
const g = gazetteerFromEntries([
|
||||
{ slug: 'people/naval', source_id: 'default', title: '纳瓦尔' },
|
||||
{ slug: 'companies/acme', source_id: 'default', title: 'Acme' },
|
||||
]);
|
||||
const mentions = findMentionedEntities('Acme was founded by 纳瓦尔 in 2020.', g, {
|
||||
fromSlug: 'writing/post-1', fromSourceId: 'default',
|
||||
});
|
||||
expect(mentions).toHaveLength(2);
|
||||
const slugs = mentions.map(m => m.slug);
|
||||
expect(slugs).toContain('people/naval');
|
||||
expect(slugs).toContain('companies/acme');
|
||||
});
|
||||
|
||||
test('CJK empty gazetteer — no false positives', () => {
|
||||
const g: Gazetteer = new Map();
|
||||
const mentions = findMentionedEntities('纳瓦尔和双雪涛。', g, {
|
||||
fromSlug: 'writing/post-1', fromSourceId: 'default',
|
||||
});
|
||||
expect(mentions).toEqual([]);
|
||||
});
|
||||
|
||||
test('CJK empty text → empty result', () => {
|
||||
const g = gazetteerFromEntries([
|
||||
{ slug: 'people/naval', source_id: 'default', title: '纳瓦尔' },
|
||||
]);
|
||||
const mentions = findMentionedEntities('', g, {
|
||||
fromSlug: 'writing/post-1', fromSourceId: 'default',
|
||||
});
|
||||
expect(mentions).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
// ============================================================
|
||||
// buildGazetteer — engine-backed tests
|
||||
// ============================================================
|
||||
@@ -366,4 +499,26 @@ describe('buildGazetteer — engine integration', () => {
|
||||
// forces a deliberate change (and a corresponding test update).
|
||||
expect(LINKABLE_ENTITY_TYPES).toEqual(['person', 'company', 'organization', 'entity']);
|
||||
});
|
||||
|
||||
// CJK — engine-backed tests
|
||||
test('CJK entity with 2-char title enters gazetteer with char-level tokens', async () => {
|
||||
await engine.putPage('people/naval', {
|
||||
type: 'person', title: '纳瓦尔', compiled_truth: 'b', timeline: '', frontmatter: {},
|
||||
});
|
||||
const g = await buildGazetteer(engine);
|
||||
// "纳瓦尔" tokenized as ["纳","瓦","尔"] → key is "纳"
|
||||
expect(g.has('纳')).toBe(true);
|
||||
const bucket = g.get('纳')!;
|
||||
expect(bucket.length).toBe(1);
|
||||
expect(bucket[0]!.tokens).toEqual(['纳', '瓦', '尔']);
|
||||
expect(bucket[0]!.slug).toBe('people/naval');
|
||||
});
|
||||
|
||||
test('CJK single-char title (cjkCharCount < 2) excluded from gazetteer', async () => {
|
||||
await engine.putPage('people/x', {
|
||||
type: 'person', title: '谢', compiled_truth: 'b', timeline: '', frontmatter: {},
|
||||
});
|
||||
const g = await buildGazetteer(engine);
|
||||
expect(g.size).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -1,8 +1,10 @@
|
||||
/**
|
||||
* Serial (stubs globalThis.fetch): exercises the self-upgrade cache REFRESH
|
||||
* orchestration end-to-end — `refreshUpdateCache()` fetches the latest release
|
||||
* and writes the correct marker to the shared cache file that the CLI startup
|
||||
* hook reads. Network is stubbed; the cache write + marker logic are real.
|
||||
* orchestration end-to-end — `refreshUpdateCache()` resolves the latest version
|
||||
* (from the VERSION file on master, #486 — the repo has zero GitHub releases,
|
||||
* so the old `releases/latest` API path could never succeed) and writes the
|
||||
* correct marker to the shared cache file that the CLI startup hook reads.
|
||||
* Network is stubbed; the cache write + marker logic are real.
|
||||
*
|
||||
* Quarantined as *.serial.test.ts because it reassigns the process-global
|
||||
* `fetch` (cross-file-unsafe under the parallel runner).
|
||||
@@ -13,10 +15,11 @@ import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { VERSION } from '../src/version.ts';
|
||||
import { parseSemver } from '../src/core/semver.ts';
|
||||
import { readUpdateCache } from '../src/core/self-upgrade.ts';
|
||||
import { refreshUpdateCache } from '../src/commands/check-update.ts';
|
||||
import { readUpdateCache, writeUpdateCache } from '../src/core/self-upgrade.ts';
|
||||
import { fetchLatestRelease, parseVersionFileBody, refreshUpdateCache, runCheckUpdate } from '../src/commands/check-update.ts';
|
||||
|
||||
const realFetch = globalThis.fetch;
|
||||
const realLog = console.log;
|
||||
let homeDir: string;
|
||||
let priorHome: string | undefined;
|
||||
|
||||
@@ -27,14 +30,13 @@ function bump(kind: 'minor' | 'patch' | 'micro'): string {
|
||||
return `${v[0]}.${v[1]}.${v[2]}.${v[3] + 1}`;
|
||||
}
|
||||
|
||||
function stubReleaseFetch(tag: string | null, ok = true): void {
|
||||
/** Stub the VERSION-file fetch. body === null → network throw. */
|
||||
function stubVersionFetch(body: string | null, status = 200): void {
|
||||
globalThis.fetch = (async (url: any) => {
|
||||
const u = String(url);
|
||||
if (u.includes('/releases/latest')) {
|
||||
if (tag === null) throw new Error('network down');
|
||||
return new Response(JSON.stringify({ tag_name: tag, published_at: '2026-01-01T00:00:00Z', html_url: 'https://x' }), {
|
||||
status: ok ? 200 : 500,
|
||||
});
|
||||
if (u.includes('/gbrain/master/VERSION')) {
|
||||
if (body === null) throw new Error('network down');
|
||||
return new Response(body, { status });
|
||||
}
|
||||
// Changelog fetch (only happens when update available) — return empty.
|
||||
return new Response('', { status: 200 });
|
||||
@@ -49,49 +51,155 @@ beforeEach(() => {
|
||||
|
||||
afterEach(() => {
|
||||
globalThis.fetch = realFetch;
|
||||
console.log = realLog;
|
||||
if (priorHome === undefined) delete process.env.GBRAIN_HOME;
|
||||
else process.env.GBRAIN_HOME = priorHome;
|
||||
rmSync(homeDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
describe('fetchLatestRelease — resolves from the VERSION file, discriminates failures', () => {
|
||||
test('bare version body → ok with that tag', async () => {
|
||||
stubVersionFetch('0.99.1.0\n');
|
||||
expect(await fetchLatestRelease()).toMatchObject({ ok: true, tag: '0.99.1.0' });
|
||||
});
|
||||
|
||||
test('network throw → network_error (NOT no_releases — offline users are not told "no releases exist")', async () => {
|
||||
stubVersionFetch(null);
|
||||
expect(await fetchLatestRelease()).toEqual({ ok: false, reason: 'network_error' });
|
||||
});
|
||||
|
||||
test('HTTP 404 → no_releases', async () => {
|
||||
stubVersionFetch('Not Found', 404);
|
||||
expect(await fetchLatestRelease()).toEqual({ ok: false, reason: 'no_releases' });
|
||||
});
|
||||
|
||||
test('garbage body → no_releases', async () => {
|
||||
stubVersionFetch('<html>rate limited</html>');
|
||||
expect(await fetchLatestRelease()).toEqual({ ok: false, reason: 'no_releases' });
|
||||
});
|
||||
});
|
||||
|
||||
describe('parseVersionFileBody — shape gate over the raw fetch body', () => {
|
||||
test('trailing newline, v prefix, 3-segment legacy, suffix channel', () => {
|
||||
expect(parseVersionFileBody('0.42.67.0\n')).toBe('0.42.67.0');
|
||||
expect(parseVersionFileBody('v0.42.67.0')).toBe('0.42.67.0');
|
||||
expect(parseVersionFileBody('0.31.3\n')).toBe('0.31.3'); // legacy 3-segment
|
||||
expect(parseVersionFileBody('0.31.1.1-fixwave\n')).toBe('0.31.1.1'); // suffix compares as base
|
||||
});
|
||||
|
||||
test('malformed / huge / injected bodies → null', () => {
|
||||
expect(parseVersionFileBody('')).toBeNull();
|
||||
expect(parseVersionFileBody('not a version')).toBeNull();
|
||||
expect(parseVersionFileBody('$(rm -rf /)')).toBeNull();
|
||||
expect(parseVersionFileBody('1.2')).toBeNull(); // 2-segment: not a gbrain version
|
||||
expect(parseVersionFileBody('9'.repeat(10_000_000))).toBeNull(); // bounded, no blowup
|
||||
});
|
||||
});
|
||||
|
||||
describe('refreshUpdateCache — full refresh orchestration (network stubbed)', () => {
|
||||
test('minor-bump release → writes upgrade_available marker', async () => {
|
||||
test('minor-bump VERSION on master → writes upgrade_available marker', async () => {
|
||||
const latest = bump('minor');
|
||||
stubReleaseFetch(`v${latest}`);
|
||||
stubVersionFetch(`${latest}\n`);
|
||||
await refreshUpdateCache();
|
||||
const entry = readUpdateCache();
|
||||
expect(entry?.marker).toEqual({ kind: 'upgrade_available', current: VERSION, latest });
|
||||
});
|
||||
|
||||
test('patch release → writes upgrade_available marker', async () => {
|
||||
test('patch bump → writes upgrade_available marker', async () => {
|
||||
const latest = bump('patch');
|
||||
stubReleaseFetch(`v${latest}`);
|
||||
stubVersionFetch(`${latest}\n`);
|
||||
await refreshUpdateCache();
|
||||
expect(readUpdateCache()?.marker).toEqual({ kind: 'upgrade_available', current: VERSION, latest });
|
||||
});
|
||||
|
||||
test('micro release → writes upgrade_available marker', async () => {
|
||||
test('micro bump → writes upgrade_available marker', async () => {
|
||||
const latest = bump('micro');
|
||||
stubReleaseFetch(`v${latest}`);
|
||||
stubVersionFetch(`${latest}\n`);
|
||||
await refreshUpdateCache();
|
||||
expect(readUpdateCache()?.marker).toEqual({ kind: 'upgrade_available', current: VERSION, latest });
|
||||
});
|
||||
|
||||
test('network failure → writes up_to_date marker (fail-open, TTL prevents hammering)', async () => {
|
||||
stubReleaseFetch(null);
|
||||
test('minor bump published as legacy 3-segment → still detected', async () => {
|
||||
const v = parseSemver(VERSION)!;
|
||||
const latest = `${v[0]}.${v[1] + 1}.0`; // 3-segment, no micro
|
||||
stubVersionFetch(`${latest}\n`);
|
||||
await refreshUpdateCache();
|
||||
expect(readUpdateCache()?.marker).toEqual({ kind: 'upgrade_available', current: VERSION, latest });
|
||||
});
|
||||
|
||||
test('suffix channel release (X.Y.Z.W-fixwave) → compares as numeric base', async () => {
|
||||
const latest = bump('micro');
|
||||
stubVersionFetch(`${latest}-fixwave\n`);
|
||||
await refreshUpdateCache();
|
||||
expect(readUpdateCache()?.marker).toEqual({ kind: 'upgrade_available', current: VERSION, latest });
|
||||
});
|
||||
|
||||
test('same version on master → up_to_date marker', async () => {
|
||||
stubVersionFetch(`${VERSION}\n`);
|
||||
await refreshUpdateCache();
|
||||
expect(readUpdateCache()?.marker).toEqual({ kind: 'up_to_date', current: VERSION });
|
||||
});
|
||||
|
||||
test('non-OK HTTP → fail-open up_to_date', async () => {
|
||||
stubReleaseFetch(`v${bump('minor')}`, false);
|
||||
// The #486 bug class: a failed check must never fabricate "you're current".
|
||||
test('network failure with NO prior cache → writes NOTHING (never a fabricated up_to_date)', async () => {
|
||||
stubVersionFetch(null);
|
||||
await refreshUpdateCache();
|
||||
expect(readUpdateCache()?.marker).toEqual({ kind: 'up_to_date', current: VERSION });
|
||||
expect(readUpdateCache()).toBeNull();
|
||||
});
|
||||
|
||||
test('garbage tag → fail-open up_to_date (forged/invalid version never cached as upgrade)', async () => {
|
||||
stubReleaseFetch('v$(rm -rf /)');
|
||||
test('network failure with prior upgrade_available → pending notice PRESERVED, not erased', async () => {
|
||||
const latest = bump('minor');
|
||||
writeUpdateCache({ kind: 'upgrade_available', current: VERSION, latest });
|
||||
stubVersionFetch(null);
|
||||
await refreshUpdateCache();
|
||||
expect(readUpdateCache()?.marker).toEqual({ kind: 'up_to_date', current: VERSION });
|
||||
expect(readUpdateCache()?.marker).toEqual({ kind: 'upgrade_available', current: VERSION, latest });
|
||||
});
|
||||
|
||||
test('non-OK HTTP → no fabricated up_to_date', async () => {
|
||||
stubVersionFetch('nope', 500);
|
||||
await refreshUpdateCache();
|
||||
expect(readUpdateCache()).toBeNull();
|
||||
});
|
||||
|
||||
test('garbage body → no fabricated marker (forged/invalid version never cached as upgrade)', async () => {
|
||||
stubVersionFetch('$(rm -rf /)');
|
||||
await refreshUpdateCache();
|
||||
expect(readUpdateCache()).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('runCheckUpdate --json — failure discrimination (#486)', () => {
|
||||
function capture(): string[] {
|
||||
const lines: string[] = [];
|
||||
console.log = (...a: unknown[]) => { lines.push(a.join(' ')); };
|
||||
return lines;
|
||||
}
|
||||
|
||||
test('offline → error: network_error (not "no releases exist")', async () => {
|
||||
stubVersionFetch(null);
|
||||
const lines = capture();
|
||||
await runCheckUpdate(['--json']);
|
||||
const out = JSON.parse(lines.join('\n'));
|
||||
expect(out.error).toBe('network_error');
|
||||
expect(out.update_available).toBe(false);
|
||||
expect(readUpdateCache()).toBeNull(); // and no fabricated up_to_date cache
|
||||
});
|
||||
|
||||
test('endpoint answers but no usable version → error: no_releases', async () => {
|
||||
stubVersionFetch('garbage');
|
||||
const lines = capture();
|
||||
await runCheckUpdate(['--json']);
|
||||
expect(JSON.parse(lines.join('\n')).error).toBe('no_releases');
|
||||
});
|
||||
|
||||
test('newer VERSION on master → update_available true with latest_version set', async () => {
|
||||
const latest = bump('minor');
|
||||
stubVersionFetch(`${latest}\n`);
|
||||
const lines = capture();
|
||||
await runCheckUpdate(['--json']);
|
||||
const out = JSON.parse(lines.join('\n'));
|
||||
expect(out.update_available).toBe(true);
|
||||
expect(out.latest_version).toBe(latest);
|
||||
expect(readUpdateCache()?.marker).toEqual({ kind: 'upgrade_available', current: VERSION, latest });
|
||||
});
|
||||
});
|
||||
|
||||
@@ -35,6 +35,13 @@ describe('isNewerVersion', () => {
|
||||
expect(isNewerVersion('0.42.66.0', '0.42.66.1')).toBe(true);
|
||||
});
|
||||
|
||||
test('orders legacy 3-segment against 4-segment: 0.42.67.0 > 0.42.66.1 > 0.42.66', () => {
|
||||
expect(isNewerVersion('0.42.66.1', '0.42.67.0')).toBe(true);
|
||||
expect(isNewerVersion('0.42.66', '0.42.66.1')).toBe(true);
|
||||
expect(isNewerVersion('0.42.66', '0.42.66.0')).toBe(false); // 3-segment == its .0 micro
|
||||
expect(isNewerVersion('0.42.67.0', '0.42.66.1')).toBe(false);
|
||||
});
|
||||
|
||||
test('rejects equal, older, and malformed versions', () => {
|
||||
expect(isNewerVersion('0.42.66.0', '0.42.66.0')).toBe(false);
|
||||
expect(isNewerVersion('0.42.66.1', '0.42.66.0')).toBe(false);
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
/**
|
||||
* capOversizedChunks — CJK-aware oversize measurement (follow-up to #1675,
|
||||
* shape requested in #3475's closing review).
|
||||
*
|
||||
* cl100k (estimateTokens) matches embedding-family tokenizers on pure-ASCII
|
||||
* source (measured identical on English prose and JSON vs Qwen3-Embedding),
|
||||
* but undercounts MIXED CJK+ASCII chunks — measured −31% on URL-dense Korean
|
||||
* text (#2826's failure shape). estimateEmbedTokens lifts only that class:
|
||||
* ASCII-only input short-circuits to estimateTokens verbatim, and max()
|
||||
* keeps CJK-dominant text at the cl100k count it gets today.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { chunkCodeText, estimateTokens, estimateEmbedTokens } from '../../src/core/chunkers/code.ts';
|
||||
|
||||
/** URL-dense Korean rollup lines — the measured −31% divergence shape. */
|
||||
function urlDenseKoreanMix(lines: number): string {
|
||||
return Array.from({ length: lines }, (_, i) =>
|
||||
`- 항목 ${i}: 검증용 한국어 문장 · 링크: https://docs.example.com/pages/${String(i).padStart(32, '0')}?v=abcdef0123456789&ref=sample`,
|
||||
).join('\n');
|
||||
}
|
||||
|
||||
function bigJsonWithKoreanValues(targetChars: number): string {
|
||||
const entries: string[] = [];
|
||||
let i = 0;
|
||||
let len = 0;
|
||||
while (len < targetChars) {
|
||||
const row =
|
||||
` "item_${i}": { "name": "예시-${i}", "url": "https://example.com/api/v2/items/${i}?token=abc${i}def", "qty": ${i % 100}, "memo": "한국어 값이 섞인 예시 데이터" }`;
|
||||
entries.push(row);
|
||||
len += row.length;
|
||||
i++;
|
||||
}
|
||||
return `{\n${entries.join(',\n')}\n}`;
|
||||
}
|
||||
|
||||
describe('estimateEmbedTokens — measurement gate', () => {
|
||||
test('ASCII-only input is bit-identical to estimateTokens (no CJK → short-circuit)', () => {
|
||||
const en = 'function ordinary() { return compute(42) + helper(); } '.repeat(80);
|
||||
const json = '{"item": {"name": "sample", "url": "https://example.com/a?b=c", "qty": 42}}, '.repeat(60);
|
||||
expect(estimateEmbedTokens(en)).toBe(estimateTokens(en));
|
||||
expect(estimateEmbedTokens(json)).toBe(estimateTokens(json));
|
||||
});
|
||||
|
||||
test('never estimates below estimateTokens (max composition)', () => {
|
||||
for (const s of [urlDenseKoreanMix(20), '이 문장은 순수 한국어 산문 예시입니다. '.repeat(40), 'plain ascii ', '']) {
|
||||
expect(estimateEmbedTokens(s)).toBeGreaterThanOrEqual(estimateTokens(s));
|
||||
}
|
||||
});
|
||||
|
||||
test('mixed CJK+ASCII (the measured divergence class) estimates strictly higher', () => {
|
||||
const mix = urlDenseKoreanMix(20);
|
||||
// Real Qwen3-Embedding count for this shape measures ~45% ABOVE cl100k;
|
||||
// the weighted form stays above the real count (+15% measured margin).
|
||||
expect(estimateEmbedTokens(mix)).toBeGreaterThan(estimateTokens(mix));
|
||||
});
|
||||
});
|
||||
|
||||
describe('capOversizedChunks with the CJK-aware estimate', () => {
|
||||
test('oversized json fence with Korean values re-splits under the default cap', async () => {
|
||||
const src = bigJsonWithKoreanValues(14_000);
|
||||
const chunks = await chunkCodeText(src, 'fence.json');
|
||||
expect(chunks.length).toBeGreaterThan(1);
|
||||
for (const c of chunks) {
|
||||
// Small slack for the "[JSON] fence.json:…" header buildChunk re-adds
|
||||
// after the body-level split.
|
||||
expect(estimateEmbedTokens(c.text)).toBeLessThanOrEqual(2000 + 60);
|
||||
}
|
||||
// Content preserved — spot-check first / last entries survive.
|
||||
const joined = chunks.map((c) => c.text).join('\n');
|
||||
expect(joined).toContain('"item_0"');
|
||||
expect(joined).toContain('한국어 값이 섞인 예시 데이터');
|
||||
});
|
||||
|
||||
test('hard-split fallback makes progress on whitespace-less CJK-mixed input and stays under cap', async () => {
|
||||
const blob = '한a민b국c'.repeat(3_000); // 18K chars, no whitespace
|
||||
const chunks = await chunkCodeText(`{"blob": "${blob}"}`, 'fence.json');
|
||||
expect(chunks.length).toBeGreaterThan(1);
|
||||
for (const c of chunks) {
|
||||
expect(estimateEmbedTokens(c.text)).toBeLessThanOrEqual(2000 + 60);
|
||||
}
|
||||
});
|
||||
|
||||
test('pure-ASCII chunks are measured by the identical estimator (cap decisions unchanged)', async () => {
|
||||
const entries = Array.from({ length: 120 }, (_, i) =>
|
||||
` "item_${i}": { "name": "sample-${i}", "url": "https://example.com/api/v2/items/${i}?token=abc${i}def", "qty": ${i % 100} }`,
|
||||
);
|
||||
const src = `{\n${entries.join(',\n')}\n}`;
|
||||
const chunks = await chunkCodeText(src, 'fence.json');
|
||||
expect(chunks.length).toBeGreaterThan(0);
|
||||
for (const c of chunks) {
|
||||
// For ASCII-only chunks the two estimators are identical (pinned
|
||||
// above), so cap decisions — and therefore boundaries — are unchanged.
|
||||
expect(estimateEmbedTokens(c.text)).toBe(estimateTokens(c.text));
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,173 @@
|
||||
/**
|
||||
* #3513: parseOpArgs' stdin read must never block forever.
|
||||
*
|
||||
* In a non-TTY with no piped input — a CI step, a cron job, an agent
|
||||
* harness that inherits a non-TTY stdin without writing to it — the old
|
||||
* inline `readFileSync(0)` never returned. The fix bounds the read with a
|
||||
* first-byte deadline (pipes/sockets only) and falls through to the
|
||||
* existing required-param usage error on timeout.
|
||||
*
|
||||
* The load-bearing regression test spawns the REAL CLI with a held-open,
|
||||
* never-written pipe: on pre-fix code it hangs until our observation window
|
||||
* kills it; on fixed code it exits 1 with the usage error well inside the
|
||||
* window. The stdin read + required-param check both run BEFORE engine
|
||||
* connect, so no brain/DB is touched.
|
||||
*/
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import { dirname, join } from 'path';
|
||||
|
||||
const REPO = dirname(import.meta.dir);
|
||||
const CLI = join(REPO, 'src', 'cli.ts');
|
||||
|
||||
interface CliRun {
|
||||
exited: boolean;
|
||||
exitCode: number | null;
|
||||
stderr: string;
|
||||
}
|
||||
|
||||
/** Narrow Bun's `number | FileSink` stdin union to the pipe sink. */
|
||||
function pipeSink(proc: { stdin: unknown }): { write(d: string): unknown; end(): unknown } {
|
||||
const s = proc.stdin;
|
||||
if (!s || typeof s === 'number') throw new Error('expected a piped stdin sink');
|
||||
return s as { write(d: string): unknown; end(): unknown };
|
||||
}
|
||||
|
||||
/**
|
||||
* Spawn the CLI with the given stdin wiring. `holdPipeOpen` keeps the write
|
||||
* end of the stdin pipe alive without ever writing — the #3513 repro. The
|
||||
* observation window kills the child if it hasn't exited (pre-fix hang).
|
||||
*/
|
||||
async function runCliWithStdin(
|
||||
args: string[],
|
||||
stdin: 'hold-open' | 'closed-empty' | { data: string } | { file: string },
|
||||
windowMs: number,
|
||||
): Promise<CliRun> {
|
||||
const proc = Bun.spawn(['bun', 'run', CLI, ...args], {
|
||||
cwd: REPO,
|
||||
env: { ...process.env, GBRAIN_STDIN_TIMEOUT_MS: '500' },
|
||||
stdin: typeof stdin === 'object' && 'file' in stdin ? Bun.file(stdin.file) : 'pipe',
|
||||
stdout: 'pipe',
|
||||
stderr: 'pipe',
|
||||
});
|
||||
if (typeof stdin === 'object' && 'data' in stdin) {
|
||||
pipeSink(proc).write(stdin.data);
|
||||
await pipeSink(proc).end();
|
||||
} else if (stdin === 'closed-empty') {
|
||||
await pipeSink(proc).end();
|
||||
}
|
||||
// 'hold-open': never write, never close — the CI/cron/agent-harness shape.
|
||||
|
||||
let exited = true;
|
||||
const killer = setTimeout(() => {
|
||||
exited = false;
|
||||
try { proc.kill('SIGKILL'); } catch { /* already dead */ }
|
||||
}, windowMs);
|
||||
const [exitCode, stderr] = await Promise.all([
|
||||
proc.exited,
|
||||
new Response(proc.stderr).text(),
|
||||
]);
|
||||
clearTimeout(killer);
|
||||
try { pipeSink(proc).end(); } catch { /* hold-open cleanup */ }
|
||||
return { exited, exitCode: exited ? exitCode : null, stderr };
|
||||
}
|
||||
|
||||
describe('#3513 — stdin-capable op with a non-TTY, never-written stdin', () => {
|
||||
test('exits fast with the usage error instead of blocking forever', async () => {
|
||||
// `put` declares stdin:'content' (required). No inline content, no piped
|
||||
// input → the bounded read times out at 500ms, content stays unset, and
|
||||
// the required-param check prints usage and exits 1. Pre-fix: readFileSync(0)
|
||||
// blocks until the 20s window kills the child.
|
||||
const run = await runCliWithStdin(['put', 'stdin-hang-test-slug'], 'hold-open', 20_000);
|
||||
expect(run.exited).toBe(true); // pre-#3513 this is false: the read never returns
|
||||
expect(run.exitCode).toBe(1);
|
||||
expect(run.stderr).toContain('Usage: gbrain put');
|
||||
}, 30_000);
|
||||
|
||||
test('a genuine pipe with data is still consumed (no hang, no crash)', async () => {
|
||||
// Piped content fills `content`; the missing positional slug then fails
|
||||
// the required check — proving the stream path read stdin and moved on.
|
||||
const run = await runCliWithStdin(['put'], { data: '# hello\n' }, 20_000);
|
||||
expect(run.exited).toBe(true);
|
||||
expect(run.exitCode).toBe(1);
|
||||
expect(run.stderr).toContain('Usage: gbrain put');
|
||||
}, 30_000);
|
||||
|
||||
test('empty-but-real input (`< /dev/null`) does not hang', async () => {
|
||||
const run = await runCliWithStdin(['put'], { file: '/dev/null' }, 20_000);
|
||||
expect(run.exited).toBe(true);
|
||||
expect(run.exitCode).toBe(1);
|
||||
}, 30_000);
|
||||
|
||||
test('an empty pipe that closes immediately does not hang', async () => {
|
||||
const run = await runCliWithStdin(['put'], 'closed-empty', 20_000);
|
||||
expect(run.exited).toBe(true);
|
||||
expect(run.exitCode).toBe(1);
|
||||
}, 30_000);
|
||||
});
|
||||
|
||||
describe('#3513 — applyStdinParam content preservation (subprocess driver)', () => {
|
||||
// Drive the exported helper in a child process so we control the child's
|
||||
// real fd 0 — bun test's own stdin is not a reliable fixture.
|
||||
const DRIVER = `
|
||||
const { applyStdinParam } = await import(${JSON.stringify(CLI)});
|
||||
const op = { name: 'put', params: { content: { type: 'string', required: true } }, cliHints: { stdin: 'content' } };
|
||||
const params = {};
|
||||
await applyStdinParam(op, params);
|
||||
console.log(JSON.stringify(params));
|
||||
process.exit(0);
|
||||
`;
|
||||
|
||||
async function runDriver(
|
||||
stdin: 'hold-open' | 'closed-empty' | { data: string } | { file: string },
|
||||
): Promise<{ exited: boolean; params: Record<string, unknown> | null }> {
|
||||
const proc = Bun.spawn(['bun', '-e', DRIVER], {
|
||||
cwd: REPO,
|
||||
env: { ...process.env, GBRAIN_STDIN_TIMEOUT_MS: '500' },
|
||||
stdin: typeof stdin === 'object' && 'file' in stdin ? Bun.file(stdin.file) : 'pipe',
|
||||
stdout: 'pipe',
|
||||
stderr: 'pipe',
|
||||
});
|
||||
if (typeof stdin === 'object' && 'data' in stdin) {
|
||||
pipeSink(proc).write(stdin.data);
|
||||
await pipeSink(proc).end();
|
||||
} else if (stdin === 'closed-empty') {
|
||||
await pipeSink(proc).end();
|
||||
}
|
||||
let exited = true;
|
||||
const killer = setTimeout(() => {
|
||||
exited = false;
|
||||
try { proc.kill('SIGKILL'); } catch { /* already dead */ }
|
||||
}, 20_000);
|
||||
const [stdout] = await Promise.all([new Response(proc.stdout).text(), proc.exited]);
|
||||
clearTimeout(killer);
|
||||
try { pipeSink(proc).end(); } catch { /* hold-open cleanup */ }
|
||||
const line = stdout.trim().split('\n').pop() ?? '';
|
||||
let params: Record<string, unknown> | null = null;
|
||||
try { params = JSON.parse(line); } catch { /* child killed before printing */ }
|
||||
return { exited, params };
|
||||
}
|
||||
|
||||
test('piped data lands in the stdin param verbatim', async () => {
|
||||
const { exited, params } = await runDriver({ data: '---\ntitle: x\n---\nbody' });
|
||||
expect(exited).toBe(true);
|
||||
expect(params?.content).toBe('---\ntitle: x\n---\nbody');
|
||||
}, 30_000);
|
||||
|
||||
test('/dev/null yields empty-string content (readable, empty — pre-fix parity)', async () => {
|
||||
const { exited, params } = await runDriver({ file: '/dev/null' });
|
||||
expect(exited).toBe(true);
|
||||
expect(params?.content).toBe('');
|
||||
}, 30_000);
|
||||
|
||||
test('empty closed pipe yields empty-string content', async () => {
|
||||
const { exited, params } = await runDriver('closed-empty');
|
||||
expect(exited).toBe(true);
|
||||
expect(params?.content).toBe('');
|
||||
}, 30_000);
|
||||
|
||||
test('held-open pipe times out and leaves the param unset', async () => {
|
||||
const { exited, params } = await runDriver('hold-open');
|
||||
expect(exited).toBe(true); // completes inside the window instead of hanging
|
||||
expect(params).toEqual({});
|
||||
}, 30_000);
|
||||
});
|
||||
@@ -0,0 +1,81 @@
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import {
|
||||
normalizeBlockConversation,
|
||||
looksLikeBlockConversation,
|
||||
} from '../src/core/conversation-parser/normalize-block.ts';
|
||||
import { parseConversation } from '../src/core/conversation-parser/parse.ts';
|
||||
|
||||
// A realistic Slack-collector page body (the format gbrain's own collector emits).
|
||||
const SLACK_DM = `# DM (group) with Hugh, Karyshma, Theo — 2026-06-15
|
||||
|
||||
- **Theo** (Mon 11:18)
|
||||
Hey everyone — quick note on the *real fiscal value* we surface after accounting.
|
||||
|
||||
It's a huge win at renewal and dents churn.
|
||||
- **Juan** (Mon 11:20)
|
||||
Agreed. Let's make sure we capture it for all ongoing customers.`;
|
||||
|
||||
describe('looksLikeBlockConversation', () => {
|
||||
test('detects the block header signature', () => {
|
||||
expect(looksLikeBlockConversation(SLACK_DM)).toBe(true);
|
||||
expect(looksLikeBlockConversation('- **Theo** (16:36)\n body')).toBe(true);
|
||||
expect(looksLikeBlockConversation('- **Theo** (11:18 AM)\n body')).toBe(true);
|
||||
});
|
||||
|
||||
test('is false for canonical single-line content (no false trigger)', () => {
|
||||
expect(looksLikeBlockConversation('**Theo** (11:18): hi there')).toBe(false);
|
||||
expect(looksLikeBlockConversation('**Theo** (2026-06-15 11:18): hi')).toBe(false);
|
||||
expect(looksLikeBlockConversation('just some prose with no chat at all')).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('normalizeBlockConversation', () => {
|
||||
test('collapses header + indented multi-paragraph body to one canonical line', () => {
|
||||
const out = normalizeBlockConversation(SLACK_DM).split('\n');
|
||||
expect(out).toEqual([
|
||||
"**Theo** (11:18): Hey everyone — quick note on the *real fiscal value* we surface after accounting. It's a huge win at renewal and dents churn.",
|
||||
"**Juan** (11:20): Agreed. Let's make sure we capture it for all ongoing customers.",
|
||||
]);
|
||||
});
|
||||
|
||||
test('drops the page-title line and leading blanks', () => {
|
||||
const out = normalizeBlockConversation(SLACK_DM);
|
||||
expect(out.startsWith('# DM')).toBe(false);
|
||||
expect(out.startsWith('**Theo**')).toBe(true);
|
||||
});
|
||||
|
||||
test('converts 12h am/pm to 24h', () => {
|
||||
expect(normalizeBlockConversation('- **A** (1:05 PM)\n hi')).toBe('**A** (13:05): hi');
|
||||
expect(normalizeBlockConversation('- **A** (12:00 AM)\n midnight')).toBe('**A** (00:00): midnight');
|
||||
expect(normalizeBlockConversation('- **A** (12:30 PM)\n noon-ish')).toBe('**A** (12:30): noon-ish');
|
||||
});
|
||||
|
||||
test('keeps day-of-week out of the emitted time', () => {
|
||||
expect(normalizeBlockConversation('- **A** (Tue 09:07)\n morning')).toBe('**A** (09:07): morning');
|
||||
});
|
||||
|
||||
test('is a strict no-op on canonical single-line content', () => {
|
||||
const canonical = '**Theo** (11:18): hi\n**Juan** (11:20): yo';
|
||||
expect(normalizeBlockConversation(canonical)).toBe(canonical);
|
||||
});
|
||||
|
||||
test('a message with no body emits an empty-body line', () => {
|
||||
expect(normalizeBlockConversation('- **A** (10:00)')).toBe('**A** (10:00): ');
|
||||
});
|
||||
});
|
||||
|
||||
describe('parseConversation integration — block format now yields messages', () => {
|
||||
test('Slack-collector body parses to 2 messages via the normalize pre-pass', () => {
|
||||
const res = parseConversation(SLACK_DM, { fallbackDate: '2026-06-15' });
|
||||
expect(res.messages.length).toBe(2);
|
||||
expect(res.messages[0].speaker).toBe('Theo');
|
||||
expect(res.messages[1].speaker).toBe('Juan');
|
||||
expect(res.phase).not.toBe('no_match');
|
||||
});
|
||||
|
||||
test('canonical content still parses unchanged (no regression)', () => {
|
||||
const res = parseConversation('**Theo** (11:18): hi there', { fallbackDate: '2026-06-15' });
|
||||
expect(res.messages.length).toBe(1);
|
||||
expect(res.messages[0].speaker).toBe('Theo');
|
||||
});
|
||||
});
|
||||
@@ -12,6 +12,7 @@ import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import {
|
||||
isSourceUnchangedSinceSync,
|
||||
probeSourceGitState,
|
||||
_setGitHeadProbeForTests,
|
||||
_setGitCleanProbeForTests,
|
||||
type GitHeadProbe,
|
||||
@@ -176,3 +177,60 @@ describe('isSourceUnchangedSinceSync — requireCleanWorkingTree (D7)', () => {
|
||||
expect(cleanCalls).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('probeSourceGitState — three-state verdict', () => {
|
||||
test('state 1: HEAD matches + clean → unchanged', () => {
|
||||
_setGitHeadProbeForTests(() => 'abc123');
|
||||
_setGitCleanProbeForTests(() => true);
|
||||
expect(probeSourceGitState('/tmp/repo', 'abc123', { requireCleanWorkingTree: 'ignore-untracked' }))
|
||||
.toBe('unchanged');
|
||||
});
|
||||
|
||||
test('state 2: HEAD probe null (clone missing / not a repo / git error) → unavailable', () => {
|
||||
_setGitHeadProbeForTests(() => null);
|
||||
expect(probeSourceGitState('/tmp/gone', 'abc123')).toBe('unavailable');
|
||||
});
|
||||
|
||||
test('state 3: HEAD mismatch → changed', () => {
|
||||
_setGitHeadProbeForTests(() => 'def456');
|
||||
expect(probeSourceGitState('/tmp/repo', 'abc123')).toBe('changed');
|
||||
});
|
||||
|
||||
test('state 4: dirty tree with readable HEAD → changed (NOT unavailable)', () => {
|
||||
_setGitHeadProbeForTests(() => 'abc123');
|
||||
_setGitCleanProbeForTests(() => false);
|
||||
expect(probeSourceGitState('/tmp/repo', 'abc123', { requireCleanWorkingTree: true }))
|
||||
.toBe('changed');
|
||||
});
|
||||
|
||||
test('state 5: clean-probe ERROR with readable HEAD → changed (fail toward work)', () => {
|
||||
_setGitHeadProbeForTests(() => 'abc123');
|
||||
_setGitCleanProbeForTests(() => null);
|
||||
expect(probeSourceGitState('/tmp/repo', 'abc123', { requireCleanWorkingTree: true }))
|
||||
.toBe('changed');
|
||||
});
|
||||
|
||||
test('state 6: NULL inputs → changed, head probe never called (case-4 contract)', () => {
|
||||
let probeCalls = 0;
|
||||
_setGitHeadProbeForTests(() => { probeCalls++; return 'abc'; });
|
||||
expect(probeSourceGitState(null, 'abc')).toBe('changed');
|
||||
expect(probeSourceGitState('/tmp/repo', null)).toBe('changed');
|
||||
expect(probeSourceGitState('', '')).toBe('changed');
|
||||
expect(probeCalls).toBe(0);
|
||||
});
|
||||
|
||||
test('state 7: boolean façade parity — isSourceUnchangedSinceSync === (state is unchanged)', () => {
|
||||
_setGitHeadProbeForTests(() => 'abc123');
|
||||
_setGitCleanProbeForTests(() => true);
|
||||
for (const [path, commit] of [
|
||||
['/tmp/repo', 'abc123'], // unchanged → true
|
||||
['/tmp/repo', 'other'], // changed → false
|
||||
] as const) {
|
||||
expect(isSourceUnchangedSinceSync(path, commit))
|
||||
.toBe(probeSourceGitState(path, commit) === 'unchanged');
|
||||
}
|
||||
_setGitHeadProbeForTests(() => null); // unavailable → false
|
||||
expect(isSourceUnchangedSinceSync('/tmp/gone', 'abc123'))
|
||||
.toBe(probeSourceGitState('/tmp/gone', 'abc123') === 'unchanged');
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,138 @@
|
||||
/**
|
||||
* #3502: docs must not reference nonexistent gbrain commands.
|
||||
*
|
||||
* `docs/tutorials/personal-brain.md` shipped a `gbrain install` step for two
|
||||
* months after the command it replaced was retired — every reader hit
|
||||
* "Unknown command: install". This guard scans README.md, docs/, and skills/
|
||||
* for `gbrain <verb>` invocations in code (fenced blocks + inline code spans)
|
||||
* and checks each verb against the live CLI surface: CLI_ONLY, operation
|
||||
* cliHints names (non-hidden), and aliases.
|
||||
*
|
||||
* Deliberately excluded (historical or speculative by design, per CLAUDE.md's
|
||||
* "historical docs are never rewritten" rule):
|
||||
* - docs/GBRAIN_V0.md — the v0 spec; documents v0's CLI
|
||||
* - docs/designs/, docs/plans/ — future/speculative design docs
|
||||
* - docs/migrations/, skills/migrations/ — per-release migration notes,
|
||||
* written against that release's CLI
|
||||
* - docs/UPGRADING_DOWNSTREAM_AGENTS.md — per-release upgrade chronicle
|
||||
*
|
||||
* Heuristics keep prose out: only fenced code + inline spans are scanned,
|
||||
* comment lines and diagram lines are skipped, and the verb must sit in
|
||||
* command position (start of command text, or after a shell operator).
|
||||
*/
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import { readdirSync, readFileSync, statSync } from 'fs';
|
||||
import { dirname, join, relative } from 'path';
|
||||
import { CLI_ONLY, cliAliases } from '../src/cli.ts';
|
||||
import { operations } from '../src/core/operations.ts';
|
||||
|
||||
const ROOT = dirname(import.meta.dir);
|
||||
|
||||
const EXCLUDED = [
|
||||
'docs/GBRAIN_V0.md',
|
||||
'docs/UPGRADING_DOWNSTREAM_AGENTS.md',
|
||||
'docs/designs/',
|
||||
'docs/plans/',
|
||||
'docs/migrations/',
|
||||
'skills/migrations/',
|
||||
];
|
||||
|
||||
/** Known-intentional references to commands that deliberately don't exist. */
|
||||
const ALLOWLIST: Record<string, string[]> = {
|
||||
// The doc explains that gbrain does NOT ship this command, on purpose.
|
||||
'docs/guides/rls-and-you.md': ['rls-exempt'],
|
||||
};
|
||||
|
||||
function validCommands(): Set<string> {
|
||||
const valid = new Set<string>(CLI_ONLY);
|
||||
for (const op of operations) {
|
||||
const name = op.cliHints?.name;
|
||||
if (name && !op.cliHints?.hidden) valid.add(name);
|
||||
}
|
||||
for (const alias of cliAliases.keys()) valid.add(alias);
|
||||
return valid;
|
||||
}
|
||||
|
||||
function* mdFiles(dir: string): Generator<string> {
|
||||
for (const entry of readdirSync(dir)) {
|
||||
const p = join(dir, entry);
|
||||
if (statSync(p).isDirectory()) yield* mdFiles(p);
|
||||
else if (p.endsWith('.md')) yield p;
|
||||
}
|
||||
}
|
||||
|
||||
interface CodeLine { code: string; line: number }
|
||||
|
||||
/** Fenced-block lines + inline code spans that START with `gbrain `. */
|
||||
function codeRegions(text: string): CodeLine[] {
|
||||
const out: CodeLine[] = [];
|
||||
const lines = text.split('\n');
|
||||
let inFence = false;
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
const l = lines[i];
|
||||
if (/^\s*(```|~~~)/.test(l)) { inFence = !inFence; continue; }
|
||||
if (inFence) {
|
||||
const t = l.trim();
|
||||
if (/^(#|\/\/|--|\*)/.test(t)) continue; // comment lines
|
||||
if (/[│┌┐└┘├┤─═╔╗╚╝]/.test(l)) continue; // ASCII-art diagrams
|
||||
out.push({ code: l, line: i + 1 });
|
||||
continue;
|
||||
}
|
||||
for (const m of l.matchAll(/`(gbrain [^`]+)`/g)) out.push({ code: m[1], line: i + 1 });
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/** True when `gbrain` sits at command position (not mid-prose). */
|
||||
function commandPosition(prefix: string): boolean {
|
||||
const p = prefix.trimEnd();
|
||||
return p === '' || /[|;&`(={[]$/.test(p) || /\$$/.test(p);
|
||||
}
|
||||
|
||||
function scan(): string[] {
|
||||
const valid = validCommands();
|
||||
const violations: string[] = [];
|
||||
const files = [
|
||||
join(ROOT, 'README.md'),
|
||||
...mdFiles(join(ROOT, 'docs')),
|
||||
...mdFiles(join(ROOT, 'skills')),
|
||||
];
|
||||
for (const file of files) {
|
||||
const rel = relative(ROOT, file);
|
||||
if (EXCLUDED.some((e) => rel === e || rel.startsWith(e))) continue;
|
||||
const text = readFileSync(file, 'utf-8');
|
||||
for (const { code, line } of codeRegions(text)) {
|
||||
for (const m of code.matchAll(/\bgbrain\s+([A-Za-z][\w-]*)/g)) {
|
||||
const verb = m[1];
|
||||
if (!/^[a-z][a-z0-9_-]{2,}$/.test(verb)) continue; // flags, <slots>, v0.x
|
||||
if (!commandPosition(code.slice(0, m.index))) continue;
|
||||
if (valid.has(verb)) continue;
|
||||
if (ALLOWLIST[rel]?.includes(verb)) continue;
|
||||
violations.push(`${rel}:${line}: \`gbrain ${verb}\` is not a real command — ${code.trim().slice(0, 90)}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
return violations;
|
||||
}
|
||||
|
||||
describe('#3502 — docs reference only real gbrain commands', () => {
|
||||
test('every `gbrain <verb>` in README/docs/skills resolves to a live command', () => {
|
||||
const violations = scan();
|
||||
expect(violations).toEqual([]);
|
||||
});
|
||||
|
||||
test('the sanity anchors: install is dead, init/put/skillpack are live', () => {
|
||||
const valid = validCommands();
|
||||
expect(valid.has('install')).toBe(false); // retired v0.36.0.0 — the #3502 bug
|
||||
expect(valid.has('init')).toBe(true);
|
||||
expect(valid.has('put')).toBe(true);
|
||||
expect(valid.has('skillpack')).toBe(true);
|
||||
});
|
||||
|
||||
test('pages + bench are dispatchable (documented surfaces; #2035 bug class)', () => {
|
||||
// `pages` had a live handleCliOnly case but was dropped from CLI_ONLY;
|
||||
// `bench` (bench-publish.ts) was documented but never wired at all.
|
||||
expect(CLI_ONLY.has('pages')).toBe(true);
|
||||
expect(CLI_ONLY.has('bench')).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -143,9 +143,10 @@ describe('checkEmbeddingWidthConsistency', () => {
|
||||
});
|
||||
|
||||
test('gateway unconfigured: skips with ok', async () => {
|
||||
// Reset gateway so requireConfig() throws.
|
||||
const { resetGateway } = await import('../src/core/ai/gateway.ts');
|
||||
resetGateway();
|
||||
// Hard-unconfigure so requireConfig() throws — resetGateway() would
|
||||
// restore the preload's test baseline (#3554).
|
||||
const { __unconfigureGatewayForTests } = await import('../src/core/ai/gateway.ts');
|
||||
__unconfigureGatewayForTests();
|
||||
const check = await checkEmbeddingWidthConsistency(engine);
|
||||
expect(check.status).toBe('ok');
|
||||
expect(check.message).toContain('gateway not configured');
|
||||
|
||||
@@ -1580,3 +1580,174 @@ describe('BUG 4 — in-progress sync via live lock, not stale freshness', () =>
|
||||
expect(result.status).toBe('fail');
|
||||
});
|
||||
});
|
||||
|
||||
// ============================================================================
|
||||
// sync_freshness — clone-unavailable content-lag fallback (stateless deploys)
|
||||
// ============================================================================
|
||||
// A container restart (Docker on EB / K8s / Fly) wipes federated clones;
|
||||
// each one is only re-materialized when that source's next sync job runs.
|
||||
// Until then the LOCAL git short-circuit cannot probe HEAD at all. That is
|
||||
// not evidence of pending work, so instead of falling through to raw
|
||||
// wall-clock age (which no-op syncs never advance → false stale/FAIL for
|
||||
// every quiet source after a restart), the check borrows the REMOTE path's
|
||||
// newest_content_at lag (v0.41.32.0). Contracts:
|
||||
// F1: clone unavailable + content at/before last sync → healthy (lag 0).
|
||||
// F2: clone unavailable + content NEWER than last sync → still stale
|
||||
// (wall-clock) — real missed work is never masked.
|
||||
// F3: clone unavailable + NULL newest_content_at → wall-clock fallback
|
||||
// (pre-migration parity with git short-circuit case 5).
|
||||
// F4: chunker mismatch disables the fallback (D7 — a pending re-chunk is
|
||||
// never masked).
|
||||
// F5: a READABLE clone that failed the short-circuit (HEAD moved) keeps
|
||||
// wall-clock even when newest_content_at is old — the fallback is
|
||||
// scoped to 'unavailable' only.
|
||||
// ============================================================================
|
||||
describe('sync_freshness — clone-unavailable content-lag fallback', () => {
|
||||
function makeStubEngine(rows: any[]): any {
|
||||
return { executeRaw: async () => rows };
|
||||
}
|
||||
function agoMs(ms: number): Date { return new Date(Date.now() - ms); }
|
||||
const HOURS = 60 * 60 * 1000;
|
||||
let currentChunkerVersion: string;
|
||||
|
||||
beforeEach(async () => {
|
||||
const { _setGitHeadProbeForTests, _setGitCleanProbeForTests } =
|
||||
await import('../src/core/git-head.ts');
|
||||
const { CHUNKER_VERSION } = await import('../src/core/chunkers/code.ts');
|
||||
currentChunkerVersion = String(CHUNKER_VERSION);
|
||||
_setGitHeadProbeForTests(null);
|
||||
_setGitCleanProbeForTests(null);
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
const { _setGitHeadProbeForTests, _setGitCleanProbeForTests } =
|
||||
await import('../src/core/git-head.ts');
|
||||
_setGitHeadProbeForTests(null);
|
||||
_setGitCleanProbeForTests(null);
|
||||
});
|
||||
|
||||
test('F1: quiet source, clone gone, content predates last sync → ok', async () => {
|
||||
const { checkSyncFreshness } = await import('../src/commands/doctor.ts');
|
||||
const { _setGitHeadProbeForTests, _setGitCleanProbeForTests } =
|
||||
await import('../src/core/git-head.ts');
|
||||
_setGitHeadProbeForTests(() => null); // clone not re-materialized yet
|
||||
_setGitCleanProbeForTests(() => true);
|
||||
|
||||
const result = await checkSyncFreshness(makeStubEngine([
|
||||
{ id: 'quiet-docs', name: '', local_path: '/tmp/quiet-docs',
|
||||
last_sync_at: agoMs(40 * HOURS),
|
||||
last_commit: 'abc', chunker_version: currentChunkerVersion,
|
||||
newest_content_at: agoMs(72 * HOURS) }, // content older than last sync
|
||||
]), { localOnly: true });
|
||||
|
||||
expect(result.status).toBe('ok');
|
||||
expect(result.details).toEqual({
|
||||
unchanged_count: 0, synced_recently_count: 1, stale_count: 0,
|
||||
});
|
||||
});
|
||||
|
||||
test('F2: clone gone but content NEWER than last sync → warn (real work not masked)', async () => {
|
||||
const { checkSyncFreshness } = await import('../src/commands/doctor.ts');
|
||||
const { _setGitHeadProbeForTests, _setGitCleanProbeForTests } =
|
||||
await import('../src/core/git-head.ts');
|
||||
_setGitHeadProbeForTests(() => null);
|
||||
_setGitCleanProbeForTests(() => true);
|
||||
|
||||
const result = await checkSyncFreshness(makeStubEngine([
|
||||
{ id: 'missed-work', name: '', local_path: '/tmp/missed-work',
|
||||
last_sync_at: agoMs(40 * HOURS),
|
||||
last_commit: 'abc', chunker_version: currentChunkerVersion,
|
||||
newest_content_at: agoMs(1 * HOURS) }, // content NEWER than last sync
|
||||
]), { localOnly: true });
|
||||
|
||||
expect(result.status).toBe('warn');
|
||||
expect(result.message).toMatch(/40h ago/);
|
||||
expect(result.details?.stale_count).toBe(1);
|
||||
});
|
||||
|
||||
test('F3: clone gone + NULL newest_content_at → wall-clock fallback (warn)', async () => {
|
||||
const { checkSyncFreshness } = await import('../src/commands/doctor.ts');
|
||||
const { _setGitHeadProbeForTests, _setGitCleanProbeForTests } =
|
||||
await import('../src/core/git-head.ts');
|
||||
_setGitHeadProbeForTests(() => null);
|
||||
_setGitCleanProbeForTests(() => true);
|
||||
|
||||
const result = await checkSyncFreshness(makeStubEngine([
|
||||
{ id: 'pre-migration', name: '', local_path: '/tmp/pre-migration',
|
||||
last_sync_at: agoMs(40 * HOURS),
|
||||
last_commit: 'abc', chunker_version: currentChunkerVersion,
|
||||
newest_content_at: null },
|
||||
]), { localOnly: true });
|
||||
|
||||
expect(result.status).toBe('warn');
|
||||
expect(result.details?.stale_count).toBe(1);
|
||||
});
|
||||
|
||||
test('F4: clone gone + chunker MISMATCH → fallback disabled, wall-clock warn', async () => {
|
||||
const { checkSyncFreshness } = await import('../src/commands/doctor.ts');
|
||||
const { _setGitHeadProbeForTests, _setGitCleanProbeForTests } =
|
||||
await import('../src/core/git-head.ts');
|
||||
_setGitHeadProbeForTests(() => null);
|
||||
_setGitCleanProbeForTests(() => true);
|
||||
|
||||
const result = await checkSyncFreshness(makeStubEngine([
|
||||
{ id: 'needs-rechunk', name: '', local_path: '/tmp/needs-rechunk',
|
||||
last_sync_at: agoMs(40 * HOURS),
|
||||
last_commit: 'abc',
|
||||
chunker_version: '0', // STALE — re-chunk pending
|
||||
newest_content_at: agoMs(72 * HOURS) },
|
||||
]), { localOnly: true });
|
||||
|
||||
expect(result.status).toBe('warn');
|
||||
expect(result.details?.stale_count).toBe(1);
|
||||
});
|
||||
|
||||
test('F5: readable clone, HEAD moved → wall-clock even with old content', async () => {
|
||||
const { checkSyncFreshness } = await import('../src/commands/doctor.ts');
|
||||
const { _setGitHeadProbeForTests, _setGitCleanProbeForTests } =
|
||||
await import('../src/core/git-head.ts');
|
||||
_setGitHeadProbeForTests(() => 'NEW-HEAD'); // clone readable, real work
|
||||
_setGitCleanProbeForTests(() => true);
|
||||
|
||||
const result = await checkSyncFreshness(makeStubEngine([
|
||||
{ id: 'has-commits', name: '', local_path: '/tmp/has-commits',
|
||||
last_sync_at: agoMs(40 * HOURS),
|
||||
last_commit: 'OLD-HEAD', chunker_version: currentChunkerVersion,
|
||||
newest_content_at: agoMs(72 * HOURS) },
|
||||
]), { localOnly: true });
|
||||
|
||||
expect(result.status).toBe('warn');
|
||||
expect(result.message).toMatch(/40h ago/);
|
||||
expect(result.details?.stale_count).toBe(1);
|
||||
});
|
||||
|
||||
test('F6: three-bucket invariant holds across rescued + unchanged + stale', async () => {
|
||||
const { checkSyncFreshness } = await import('../src/commands/doctor.ts');
|
||||
const { _setGitHeadProbeForTests, _setGitCleanProbeForTests } =
|
||||
await import('../src/core/git-head.ts');
|
||||
_setGitHeadProbeForTests((path) => path === '/tmp/frozen' ? 'frozen-sha' : null);
|
||||
_setGitCleanProbeForTests(() => true);
|
||||
|
||||
const result = await checkSyncFreshness(makeStubEngine([
|
||||
{ id: 'frozen', name: '', local_path: '/tmp/frozen', // unchanged bucket
|
||||
last_sync_at: agoMs(40 * HOURS),
|
||||
last_commit: 'frozen-sha', chunker_version: currentChunkerVersion,
|
||||
newest_content_at: agoMs(80 * HOURS) },
|
||||
{ id: 'rescued', name: '', local_path: '/tmp/rescued', // clone gone, quiet → healthy
|
||||
last_sync_at: agoMs(40 * HOURS),
|
||||
last_commit: 'abc', chunker_version: currentChunkerVersion,
|
||||
newest_content_at: agoMs(80 * HOURS) },
|
||||
{ id: 'stale', name: '', local_path: '/tmp/stale', // clone gone, content newer → stale
|
||||
last_sync_at: agoMs(5 * 24 * HOURS),
|
||||
last_commit: 'def', chunker_version: currentChunkerVersion,
|
||||
newest_content_at: agoMs(1 * HOURS) },
|
||||
]), { localOnly: true });
|
||||
|
||||
expect(result.status).toBe('fail');
|
||||
expect(result.message).toContain(`'stale'`);
|
||||
expect(result.message).not.toContain(`'rescued'`);
|
||||
expect(result.details).toEqual({
|
||||
unchanged_count: 1, synced_recently_count: 1, stale_count: 1,
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user