mirror of
https://github.com/garrytan/gbrain.git
synced 2026-08-14 17:02:19 +00:00
Compare commits
8
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Garry TanandClaude Opus 4.7
|
3dc1a95592 | ||
|
Garry TanandClaude Opus 4.7
|
312c449439 | ||
|
Garry TanandClaude Opus 4.7
|
3a409ce0d2 | ||
|
Garry TanandClaude Opus 4.7
|
dfc5284005 | ||
|
Garry TanandClaude Opus 4.7
|
b0421574d2 | ||
|
Garry TanandClaude Opus 4.7
|
a9af1fae7d | ||
|
Garry TanandClaude Opus 4.7
|
ca02a5f51b | ||
|
root
|
f63d444453 |
-260
@@ -2,266 +2,6 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.21.0] - 2026-04-25
|
||||
|
||||
## **Your brain walks the code graph now.**
|
||||
## **Call-graph edges, parent scope, chunk-grain FTS. 165-lang ready, 8 langs shipped with structural edges.**
|
||||
|
||||
v0.19.0 made code a first-class citizen. v0.21.0 makes it a graph. An agent asking "how does searchKeyword handle N+1" no longer gets back one chunk of `hybrid.ts`. It gets the function body, the 3 callers via `code-callers`, the 2 callees via `code-callees`, the class-level scope header, and — when opt-in `--walk-depth 2` is passed — the grandchildren too. All ranked together by a single RRF pass with 1/(1+hop) structural decay. One walk. Code-aware brain, not grep-class RAG.
|
||||
|
||||
Chunk-grain FTS replaces page-grain internally. The docstring above a function now ranks above a prose paragraph that happens to mention the same term. The `content_chunks.search_vector` tsvector weights doc_comment 'A' and chunk_text 'B' — an english-language query hits the right chunk first. External shape stays page-grain so every existing caller (`enrichment-service.countMentions`, `backlinks`, `list_pages`) works unchanged.
|
||||
|
||||
Classes emit properly now. `class BrainEngine { searchKeyword() {}, searchVector() {} }` was ONE chunk in v0.19.0. In v0.21.0 it's three: the class-level scope header chunk (declaration + member digest), `searchKeyword` with `parentSymbolPath: ['BrainEngine']`, and `searchVector` with the same. Retrieval surfaces individual methods when a query targets one — no more re-reading the whole class.
|
||||
|
||||
Ruby ships in the first wave of structural-edge support. `Admin::UsersController#render` identity. `def render` captured. `find_all` captured. Across all 8 shipped languages (TS, TSX, JS, Python, Ruby, Go, Rust, Java — ~85% of real brain code) call-site edges extract at chunk time and land in `code_edges_symbol` for `getCallersOf` / `getCalleesOf` to surface.
|
||||
|
||||
The honest part: precision 80, recall 99. We don't do receiver-type inference at capture time (`obj.method()` stores the bare `method` callee, not `ObjClass.method`). Cross-file edge resolution is also a future optimization — all Layer 5 edges land unresolved. What matters: the edges exist. `getCallersOf('helper')` now returns every call site in the brain, ready for Layer 7 two-pass retrieval to expand into structural neighbors. That's the 10x leap.
|
||||
|
||||
### The numbers that matter
|
||||
|
||||
Counted against gbrain's own codebase, PGLite in-memory benchmark:
|
||||
|
||||
| Metric | v0.19.0 | v0.21.0 | Δ |
|
||||
|---|---|---|---|
|
||||
| Structural edge types captured | 0 | `calls` (per-file) | ∞ |
|
||||
| Languages with call-graph edges | 0 | 8 | +8 |
|
||||
| Chunk grain at FTS time | page-level | chunk-level (internal) | — |
|
||||
| File classifier extensions | 9 | 35 | +26 |
|
||||
| Nested symbol chunks (class with 3 methods) | 1 chunk | 4 chunks | 4x |
|
||||
| Parent-scope column persisted | No | `parent_symbol_path TEXT[]` | ✓ |
|
||||
| `code-callers <sym>` + `code-callees <sym>` | not possible | JSON array in <100ms | ∞ |
|
||||
| `query --near-symbol X --walk-depth 2` | not possible | 2-hop structural expansion | ∞ |
|
||||
| `sync --all` cost preview | no warning | `ConfirmationRequired` envelope + TTY prompt | ✓ |
|
||||
| Markdown fence extraction | prose chunks | per-fence code chunks | ✓ |
|
||||
|
||||
Per-language call capture (8 shipped):
|
||||
|
||||
| Lang | Top-level | Class/module | Edge capture via |
|
||||
|---|---|---|---|
|
||||
| TypeScript | function_declaration, class_declaration, interface, type_alias, enum | class + interface → methods | call_expression.function |
|
||||
| TSX | same + JSX | same | same |
|
||||
| JavaScript | function_declaration, class_declaration, lexical_declaration | class → methods | call_expression.function |
|
||||
| Python | function_definition, class_definition | class → function_definition | call.function |
|
||||
| Ruby | class, module, method, singleton_method | module+class → method+singleton_method | call.method |
|
||||
| Go | function_declaration, method_declaration | (methods are top-level) | call_expression.function |
|
||||
| Rust | function_item, impl_item, struct, enum, trait, mod | impl+trait → function_item | call_expression.function |
|
||||
| Java | method_declaration, class_declaration, interface, enum, record | class+interface+record → method+constructor | method_invocation.name |
|
||||
|
||||
### What this means for builders
|
||||
|
||||
If you've been maintaining a gbrain deployment on v0.19.0, upgrading is mechanical: `gbrain upgrade` runs `apply-migrations` → schema v27 + v28 land automatically (~5 seconds on a 47K-page brain). Your next `gbrain sync --source <id>` detects `sources.chunker_version` mismatch and forces a full re-walk — no manual intervention. Or run `gbrain reindex-code --dry-run` to preview the cost, then `gbrain reindex-code --yes` to take advantage of A1 + A3 immediately.
|
||||
|
||||
If you ship an agent on top of gbrain: `query --lang typescript "N+1"` now filters at SQL level, `code-callers searchKeyword` surfaces who calls it, `query "how does searchKeyword work" --near-symbol BrainEngine.searchKeyword --walk-depth 2` expands through the structural graph. Your agent's brain-first lookup covers the CODE GRAPH now, not just the symbol table.
|
||||
|
||||
If you're Garry wondering how your Rubyist instincts survive the upgrade: `class Admin::UsersController { def render; def find_all }` gets qualified as `Admin::UsersController#render`. `code-callers render` finds the call sites. The instance-vs-singleton distinction is best-effort today (Layer 5 treats both as instance); `def self.find_all` vs `def find_all` ambiguity is documented in the Ruby-specific caveats of `skills/migrations/v0.21.0.md`.
|
||||
|
||||
## To take advantage of v0.21.0
|
||||
|
||||
`gbrain upgrade` runs `gbrain post-upgrade` which runs `gbrain apply-migrations`. If that chain was interrupted or if `gbrain doctor` warns about a partial migration:
|
||||
|
||||
1. **Run the orchestrator manually:**
|
||||
```bash
|
||||
gbrain apply-migrations --yes
|
||||
```
|
||||
The `v0.21.0` orchestrator (v0_21_0.ts) runs schema → backfill-prompt → verify. Schema migrations v27 + v28 land unconditionally. The backfill-prompt phase prints two paths to roll the new chunker over existing code pages.
|
||||
|
||||
2. **Pick a backfill path.** CHUNKER_VERSION bumped 3 → 4; the `sources.chunker_version` gate (SP-1 fix) forces a full re-walk on next sync regardless of git HEAD.
|
||||
|
||||
- AUTOMATIC (recommended): next `gbrain sync --source <id>` walks everything. Zero action needed.
|
||||
- IMMEDIATE: `gbrain reindex-code --dry-run` previews cost, `gbrain reindex-code --yes` runs it. Cost preview gated via `ConfirmationRequired` envelope on non-TTY callers, exit code 2 matches `sync --all`.
|
||||
|
||||
3. **Verify the outcome:**
|
||||
```bash
|
||||
gbrain doctor # expect schema_version >= 28
|
||||
gbrain code-callers <your-favorite-fn> # expect a JSON array of call sites
|
||||
gbrain query "some concept in your brain" --walk-depth 1
|
||||
gbrain stats
|
||||
```
|
||||
|
||||
4. **If any step fails or the numbers look wrong,** file an issue at https://github.com/garrytan/gbrain/issues with:
|
||||
- output of `gbrain doctor`
|
||||
- contents of `~/.gbrain/upgrade-errors.jsonl` if it exists
|
||||
- which step broke
|
||||
|
||||
This feedback loop is how the gbrain maintainers find fragile upgrade paths. Thank you.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
**Layer 1 — Foundation schema migration (v27).** All DDL lands first, before any consumer. `content_chunks` gains `parent_symbol_path TEXT[]`, `doc_comment TEXT`, `symbol_name_qualified TEXT`, `search_vector TSVECTOR`. `sources` gains `chunker_version TEXT` (SP-1 gate). Two new tables: `code_edges_chunk` (resolved, FK CASCADE both ways) + `code_edges_symbol` (unresolved qualified-name edges). Plpgsql trigger `update_chunk_search_vector` weights doc_comment + symbol_name_qualified 'A', chunk_text 'B'. Per codex SP-4: every downstream layer has its schema prerequisites before referencing them. `scripts/check-jsonb-pattern.sh` + migration tests pin the DDL shape so accidental drift surfaces in CI.
|
||||
|
||||
**Layer 2 (1a) — File-classifier widening.** `src/core/sync.ts` expands from 9 recognized code extensions to 35 — Rust, Ruby, Java, C#, C/C++, Swift, Kotlin, Scala, PHP, Elixir, Elm, OCaml, Dart, Zig, Solidity, Lua, shell, etc. New `resolveSlugForPath(path)` centralizes slug dispatch (SP-5 fix) so delete/rename paths honor the same code-vs-markdown classification as import. Layer 9 (Magika) fallback hook ready via `setLanguageFallback`.
|
||||
|
||||
**Layer 3 (1b) — Chunk-grain FTS with page-grain wrap.** `searchKeyword` now ranks internally at chunk grain via the new `search_vector`, then dedups to best-chunk-per-page before returning. External shape unchanged (SP-6 decision) — every `searchKeyword` caller (`enrichment-service`, `backlinks`, `list_pages`) sees the same page-grain result. A2 two-pass consumes the raw chunk-grain primitive via the new `searchKeywordChunks` method. Weight A (doc_comment + symbol_name_qualified) > Weight B (chunk_text) means docstring matches rank above prose for NL queries.
|
||||
|
||||
**Layer 4 (B1) — Language manifest foundation.** The hardcoded `GRAMMAR_PATHS` + `DISPLAY_LANG` maps collapse into one `LANGUAGE_MANIFEST` keyed by `LanguageEntry` (embeddedPath | lazyLoader | displayName). `registerLanguage` / `unregisterLanguage` / `listRegisteredLanguages` are extension points; downstream consumers can add grammars without forking the chunker. 29 shipped embedded today; the lazy-load path is forward-compat for the full 165-language pack.
|
||||
|
||||
**Layer 5 (A1) — Edge extractor + qualified names (8 langs).** The 10x leap. `src/core/chunkers/edge-extractor.ts` walks the tree-sitter tree iteratively (no recursion — generated code trees can blow the stack) and harvests call-site edges per-language. `src/core/chunkers/qualified-names.ts` builds identity strings per-language: Ruby `Admin::UsersController#render`, Python `admin.users.UsersController.render`, TS `BrainEngine.searchKeyword`, Rust `users::UsersController::render`. `importCodeFile` calls `deleteCodeEdgesForChunks` (codex SP-2 inbound invalidation) then `addCodeEdges`. Both engines (PGLite + Postgres) implement all 5 edge methods: `addCodeEdges`, `deleteCodeEdgesForChunks`, `getCallersOf`, `getCalleesOf`, `getEdgesByChunk`. Readers UNION both tables forever (codex 1.3b: no promotion).
|
||||
|
||||
**Layer 6 (A3) — Parent-scope + nested-chunk emission.** A class with 3 methods emits 4 chunks now: the class-level scope header (slim body: declaration line + member digest) + each method with `parentSymbolPath: ['ClassName']`. Chunk headers show `(in ClassName.method)` so the embedding captures scope. Recursive expansion: Ruby `module Admin { class Users { def render } }` emits 3 chunks — Admin, Users (parent=[Admin]), render (parent=[Admin, Users]). `mergeSmallSiblings` bails when scope chunks are present (methods emitted individually on purpose; merging would erase the parent-path metadata).
|
||||
|
||||
**Layer 7 (A2) — Two-pass structural retrieval.** `src/core/search/two-pass.ts` expands an anchor set up to 2 hops through `code_edges_chunk` + `code_edges_symbol`, unresolved-edge targets resolved by symbol_name_qualified lookup. Score decay 1/(1+hop). Default OFF per codex F5. Activation: `--walk-depth N` (1 or 2) or `--near-symbol <qualified-name>`. Neighbor cap 50 per hop. Dedup per-page cap lifts from 2 → `min(10, walkDepth × 5)` when walking.
|
||||
|
||||
**Layer 8 (D) — Tier D bundle.** Three deferred items from v0.19.0 ship here. **D1** `sync --all` cost preview via `estimateTokens` + `EMBEDDING_COST_PER_1K_TOKENS = 0.00013` + `ConfirmationRequired` envelope (TTY prompt or exit-2 on non-TTY / JSON / piped). **D2** markdown fence extraction — `importFromContent` walks marked lexer tokens, extracts recognized `{type:'code', lang, text}` fences through `chunkCodeText` with pseudo-path, persists as `chunk_source='fenced_code'`. 100-fence-per-page cap (env override `GBRAIN_MAX_FENCES_PER_PAGE`). **D3** `reconcile-links` batch command — forward-scans every markdown page via `extractCodeRefs`, reinserts missing doc↔impl edges idempotently (`ON CONFLICT DO NOTHING`). Respects `auto_link=false` config.
|
||||
|
||||
**Layer 10 (C) — Agent CLI surfaces.** `query --lang typescript` and `query --symbol-kind function|class|method` filter at SQL level (C1 + C2). `code-callers <symbol>` (C4) and `code-callees <symbol>` (C5) ship as new commands — auto-JSON on non-TTY, StructuredAgentError on failure. `query --near-symbol <qualified> --walk-depth 1..2` (C3) wires A2 two-pass through the query operation. C6 (`code-signature`) deferred to v0.20.1 per plan.
|
||||
|
||||
**Layer 12 — CHUNKER_VERSION 3 → 4 + SP-1 gate.** The ship-silent bug codex caught on second pass: bumping `CHUNKER_VERSION` alone did nothing on an unchanged repo because `performSync` returns `up_to_date` before reaching `importCodeFile`'s content_hash check. Fix: `sources.chunker_version` tracks the version that last synced each source; mismatch forces a full re-walk regardless of git HEAD equality. `writeChunkerVersion` called after every `writeSyncAnchor 'last_commit'`.
|
||||
|
||||
**Layer 13 (E2) — reindex-code + migration orchestrator.** `gbrain reindex-code [--source <id>] [--dry-run] [--yes] [--force] [--json]` — explicit backfill for users who want v0.21.0 benefits NOW (before next sync). Walks code pages in batches of 100 (Finding 4.4 OOM protection). Reuses D1's cost-preview gate. `--force` bypasses `importCodeFile`'s content_hash early-return. `src/commands/migrations/v0_21_0.ts` orchestrator: schema → backfill-prompt → verify phases. Idempotent, resumable.
|
||||
|
||||
**Layer 11 (E1) — BrainBench code sub-category tests.** `test/cathedral-ii-brainbench.test.ts` pins `call_graph_recall` (getCallersOf round-trip through real importCodeFile, with re-import idempotency validated) and `parent_scope_coverage` (nested methods persist parent_symbol_path, qualified names resolve). `doc_comment_matching` and `type_signature_retrieval` deferred to v0.20.1 with A4 full extraction + C6 respectively.
|
||||
|
||||
**Layer 9 (B2) — Magika auto-detect: DEFERRED to v0.20.1.** The fallback hook (`setLanguageFallback`) is in place at `src/core/chunkers/code.ts`. The `detectCodeLanguage` call order already accommodates a `null → fallback` path. Bundling the ~1MB Magika ONNX model through `bun --compile` surfaces integration risk that the plan explicitly allowed deferring. Tracked in TODOS.md.
|
||||
|
||||
**Test coverage.** +900 lines of new test cases across 11 new test files:
|
||||
- `test/chunker-version-gate.test.ts`, `test/migrations-v0_21_0.test.ts` (Layer 1 schema + Layer 12 gate)
|
||||
- `test/sync-classifier-widening.test.ts` (Layer 2)
|
||||
- `test/chunk-grain-fts.test.ts` (Layer 3)
|
||||
- `test/language-manifest.test.ts` (Layer 4)
|
||||
- `test/qualified-names.test.ts`, `test/edge-extractor.test.ts`, `test/code-edges.test.ts` (Layer 5)
|
||||
- `test/parent-scope.test.ts` (Layer 6)
|
||||
- `test/two-pass.test.ts` (Layer 7)
|
||||
- `test/sync-cost-preview.test.ts`, `test/fence-extraction.test.ts`, `test/reconcile-links.test.ts` (Layer 8)
|
||||
- `test/search-lang-symbol-kind.test.ts`, `test/code-callers-cli.test.ts` (Layer 10)
|
||||
- `test/reindex-code.test.ts`, `test/migration-orchestrator-v0_21_0.test.ts` (Layer 13)
|
||||
- `test/cathedral-ii-brainbench.test.ts` (Layer 11)
|
||||
|
||||
Final CI: 2407 pass / 250 skip / 0 fail / 6345 expect() / 467s.
|
||||
|
||||
**Credit.** Plan reviewed by 2 codex passes + 1 plan-eng-review + 1 plan-ceo-review. 16 cross-model findings (7 + 6 + 3) all absorbed — notably codex SP-1 (chunker_version silent no-op), SP-2 (inbound edge invalidation across re-imports), SP-3 (multi-source tenancy), SP-4 (layer bisectability), SP-5 (slug dispatcher), SP-6 (FTS page-grain external contract), SP-7 (no promotion, UNION-on-read forever). The release's correctness on those 3 ship-silent bugs + real bisectability is directly attributable to the two codex passes on a cathedral-scale plan.
|
||||
|
||||
## [0.19.0] - 2026-04-23
|
||||
|
||||
## **Your code is now first-class in the brain.**
|
||||
## **`gbrain code-refs BrainEngine --json` returns every usage site in <100ms.**
|
||||
|
||||
Until this release, gbrain was a markdown brain. An agent asking "how do we handle partial sync failures" got back the guide and the CHANGELOG post-mortem. It got nothing from the actual code. Not because the feature was missing — because the chunker treated a TypeScript file as prose. v0.19.0 makes code a first-class citizen alongside markdown: 29 languages parsed by tree-sitter into semantic chunks, each with a structured header (`[TypeScript] src/core/sync.ts:380-415 function performFullSync`), queryable by symbol name with a new `code-def` / `code-refs` command pair that ships agent-safe JSON by default.
|
||||
|
||||
The flagship moment for the agent persona: `gbrain code-refs BrainEngine --json` returns a clean array of `{file, line, symbol_name, snippet}` tuples in under 100ms on a 25-file corpus. No grep. No full-file reads. The brain knows where BrainEngine is used, and the agent can feed the response directly into its next reasoning step. Brain-first lookup finally covers code.
|
||||
|
||||
The cost story: daily autopilot on a 5K-file TS repo would have been ~$30/month of OpenAI embedding spend. v0.19.0's incremental chunker diffs chunks by `(chunk_index, chunk_text)` — unchanged symbols reuse their embedding, only new or edited code hits the API. Typical edit touches 2-5% of chunks, so the daily bill drops ~95% to pennies.
|
||||
|
||||
The honest part: the chunker ships as a **strict superset of Chonkie's CodeChunker**. 29 languages (vs 6 baseline), tiktoken `cl100k_base` tokenizer for accurate budgeting (not the 2-3x-off `len/4` heuristic), small-sibling merging so 30 top-level imports don't produce 30 embedding calls, AST-aware splitting of large nodes. Tree-sitter WASMs ship embedded in the `bun --compile` binary — the silent-failure mode Codex flagged during plan review got closed by a CI guard that proves semantic chunks actually come out of the compiled binary. Every release runs it.
|
||||
|
||||
### The numbers that matter
|
||||
|
||||
Counted against gbrain's own codebase (~300 TypeScript files), PGLite in-memory benchmark:
|
||||
|
||||
| Metric | v0.18.x (markdown-only) | v0.19.0 (code-aware) | Δ |
|
||||
|---|---|---|---|
|
||||
| Code indexing languages | 0 | 29 | +29 |
|
||||
| Chunk metadata columns | chunk_text only | + language, symbol_name, symbol_type, start/end_line | +5 |
|
||||
| `code-refs BrainEngine` surface | not possible | JSON array in <100ms | ∞ |
|
||||
| Daily autopilot embedding cost (5K code files, 5% churn) | ~$1.50/day naive | ~$0.05/day incremental | 30x |
|
||||
| Tokenizer accuracy vs OpenAI cl100k_base | 2-3x off (len/4) | exact (tiktoken) | tight |
|
||||
|
||||
### What this means for builders
|
||||
|
||||
If you build with gbrain + OpenClaw + Claude Code: add your repo as a source (`gbrain sources add gbrain --path .`) and sync with strategy=code. Ask your agent to "look at gbrain" — it gets the full symbol graph, not just the README. If you're shipping your own gstack fork on top of gbrain: your agent's brain-first lookup now covers code, which closes the largest remaining gap where agents fell back to grep. If you're Garry wondering what `performFullSync` does: `gbrain code-def performFullSync` and you get the answer without opening a file.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
**Layer 0 — Wintermute's baseline (cherry-picked, author scrubbed).** Tree-sitter code chunker for 6 languages (TS/TSX/JS/Python/Ruby/Go), `gbrain repos add/list/remove`, strategy-aware sync, `PageType 'code'`, `importCodeFile`, per-file sync progress via the v0.15.2 reporter. Preserved exactly, committed under Garry's author identity.
|
||||
|
||||
**Layer 1 — A6 structured errors + version bump.** New `src/core/errors.ts` exports `StructuredAgentError` + `buildError` + `serializeError`. Matches the v0.17.0 `CycleReport.PhaseResult.error` shape so agent-consumable errors stay consistent across every gbrain surface. `globToRegex` bug fix: `src/**/*.ts` now matches `src/foo.ts` (zero intermediate dirs). `GBRAIN_HOME` env var for test isolation. `package.json` → `0.19.0`.
|
||||
|
||||
**Layer 2 — `bun --compile` WASM embedding + CI guard.** Codex flagged the node_modules-at-runtime approach as the #1 silent-failure mode for v0.19.0. Fix: WASMs committed to `src/assets/wasm/`, loaded via `import path from ... with { type: 'file' }`. Bun bundles every asset referenced this way into the compiled binary. `scripts/check-wasm-embedded.sh` compiles a smoketest binary on every `bun test` run and asserts it produces real semantic chunks. If the chunker ever silently falls through to recursive again, the build breaks.
|
||||
|
||||
**Layer 3 — schema migrations v25 + v26.** `pages.page_kind TEXT CHECK (page_kind IN ('markdown','code'))` on v25, using Postgres's `NOT VALID` + `VALIDATE CONSTRAINT` split so tables with millions of pages don't hold a write lock during the ALTER. `content_chunks` adds `language`, `symbol_name`, `symbol_type`, `start_line`, `end_line` on v26, plus partial indexes keyed on non-null values so code-chunk lookups stay cheap on mixed markdown+code brains.
|
||||
|
||||
**Layer 4 — delete Wintermute's multi-repo, wire v0.18.0 sources.** The `repos` abstraction in Wintermute's baseline turned out to be redundant with v0.18.0's `sources` subsystem (per-source `last_commit`, `federated` search config, RLS-friendly, DB-native). v0.19.0 keeps `gbrain repos` as a deprecated alias that routes into `runSources`. `sync --all` iterates the `sources` table instead of a local config array. Codex's P0 #2 (per-repo sync bookmarks) and P0 #3 (slug collision) both resolved by the existing schema.
|
||||
|
||||
**Layer 5 — Chonkie chunker parity (E2a).** 6 languages → 29. Embedded asset paths for every grammar in `tree-sitter-wasms`. Accurate tokenizer via `@dqbd/tiktoken` `cl100k_base` (lazy-init). Small-sibling merging with the Chonkie `bisect_left` pattern tuned to 15% of chunk target, so tiny siblings (imports, single-line consts) collapse while substantive classes/functions stay independent. `CHUNKER_VERSION=3` folded into `importCodeFile`'s `content_hash` so chunker-shape changes across releases force clean re-chunks without `sync --force`.
|
||||
|
||||
**Layer 6 — incremental chunking (E2) + doc↔impl linking (E1).** `importCodeFile` reads existing chunks before embedding; any chunk whose `(chunk_index, chunk_text)` matches verbatim reuses the existing embedding, saving the OpenAI call. `extractCodeRefs` in `link-extraction.ts` scans markdown prose for references like `src/core/sync.ts:42`; `importFromContent` creates bidirectional `documents` / `documented_by` edges for every match. The agent can now walk from guide to code and back.
|
||||
|
||||
**Layer 7 — `code-def` + `code-refs` CLI surfaces.** The magical-moment commands. Both bypass the standard `searchKeyword` path (DISTINCT ON (slug) collapses to one result per page — wrong for code-refs). Auto-JSON when stdout is not a TTY (gh-CLI convention). Structured error envelope for the usage-error + catch-all paths. `--lang` / `--limit` / `--json` / `--no-json` flags across both commands.
|
||||
|
||||
**Layer 8 — BrainBench code category (E2E).** 11-test E2E suite against PGLite in-memory, 5 languages × 5 service files = 25-file fictional corpus, asserts `code-def` and `code-refs` retrieval quality plus the <100ms magical-moment budget. Reproducible on CI without OpenAI keys (embeddings disabled — tests cover retrieval metadata, not vector quality).
|
||||
|
||||
**Test coverage.** 91 new unit + E2E tests across 9 new files: `test/errors.test.ts`, `test/sync-strategy.test.ts`, `test/migrations-v0_19_0.test.ts`, `test/repos-alias.test.ts`, `test/chunkers/code.test.ts`, `test/link-extraction-code-refs.test.ts`, `test/incremental-chunking.test.ts`, `test/code-def-refs.test.ts`, `test/e2e/code-indexing.test.ts`. 357 assertions, all green against PGLite.
|
||||
|
||||
**Credit.** Baseline tree-sitter chunker + multi-repo scaffolding came from a community PR (author scrubbed per the privacy rule). The v0.19.0 rework on top — cathedral scope, Chonkie parity, doc↔impl linking, incremental chunking, the sources reconciliation, and the full test suite — was driven by the /plan-ceo-review + /plan-devex-review + /plan-eng-review + /codex review chain. Codex's outside-voice pass caught 4 P0s (baseline-not-in-tree, per-repo bookmarks, slug collision, chunk schema gap) that the in-model reviews missed. All 4 are fixed in the ship.
|
||||
|
||||
## To take advantage of v0.19.0
|
||||
|
||||
`gbrain upgrade` runs `apply-migrations` which lands v25 + v26 automatically. If `gbrain doctor` warns about a partial migration:
|
||||
|
||||
1. **Run the orchestrator manually:**
|
||||
```bash
|
||||
gbrain apply-migrations --yes
|
||||
```
|
||||
2. **Add your code repo as a source and sync:**
|
||||
```bash
|
||||
gbrain sources add my-repo --path /path/to/repo
|
||||
gbrain sync --source my-repo
|
||||
```
|
||||
(Or `gbrain repos add my-repo --path ...` — the deprecated alias still works.)
|
||||
3. **Verify code indexing works:**
|
||||
```bash
|
||||
gbrain code-def BrainEngine
|
||||
gbrain code-refs BrainEngine --json
|
||||
```
|
||||
4. **Observe the cost delta.** After a full sync, run an `autopilot` cycle. Incremental chunking means the second cycle's embedding cost is ~5% of the first.
|
||||
5. **If the compiled binary produces no symbol names** (everything falls to recursive chunks): your install may have skipped the WASM assets. File an issue with `gbrain doctor` output.
|
||||
|
||||
## [0.20.4] - 2026-04-24
|
||||
|
||||
**Minions skill consolidation, now honest about what the CLI actually does.**
|
||||
|
||||
One skill for background work instead of two. Shell jobs and LLM subagents land under `skills/minion-orchestrator/` with a shared Preconditions block, accurate CLI examples, and a trigger set narrowed to what the skill actually covers. Corrects four documentation bugs the prior merge shipped ... `submit_job name="shell"` isn't MCP-callable, `research`/`orchestrate` aren't real handler names, PGLite users don't need to migrate to Supabase, and "every background task goes through Minions" contradicts the `pain_triggered` default in `skills/conventions/subagent-routing.md`. The skill now matches the code.
|
||||
|
||||
Two new tests guard this surface going forward. `test/resolver.test.ts` gets a round-trip check (every quoted RESOLVER.md trigger must resolve to a frontmatter `triggers:` entry in the target skill) and a name validator (every `name="<word>"` reference in any SKILL.md must resolve to either a declared operation in `src/core/operations.ts` or a known Minions handler). The validator would have caught the `research`/`orchestrate` drift in CI instead of from a Codex cold-read. One new E2E test (`test/e2e/minions-shell-pglite.test.ts`) exercises the PGLite `--follow` inline path, previously documented but untested.
|
||||
|
||||
### For users
|
||||
|
||||
- Shell jobs via `gbrain jobs submit shell --params '{"cmd":"..."}'` (operator/CLI only ... MCP returns `permission_denied` for protected names). Subagent jobs via `gbrain agent run` (user-facing entrypoint). Both lanes route through one skill.
|
||||
- PGLite shell-job guidance now correctly points at `--follow` for inline execution. The persistent daemon mode is still Postgres-only, but you do not need to migrate.
|
||||
- `gbrain jobs submit` and `submit a gbrain job` now route to the skill; bare "gbrain jobs" no longer does (it was too broad ... the CLI namespace covers 9 subcommands, and questions about `stats`/`prune`/`retry` fall through to `gbrain --help`).
|
||||
|
||||
### Added
|
||||
|
||||
- New E2E test `test/e2e/minions-shell-pglite.test.ts` covering the PGLite `--follow` inline shell-job path. Runs in-memory, no DATABASE_URL required.
|
||||
- Resolver round-trip test in `test/resolver.test.ts`: every quoted RESOLVER.md trigger must have a fuzzy match in the target skill's frontmatter `triggers:` list.
|
||||
- Skill-example-name validator in `test/resolver.test.ts`: every `name="<word>"` reference in any `SKILL.md` body must resolve to an op in `src/core/operations.ts` or a Minions handler in `PROTECTED_JOB_NAMES`.
|
||||
|
||||
### Fixed
|
||||
|
||||
- `skills/minion-orchestrator/SKILL.md` shell-job examples use the real `--params` JSON form instead of nonexistent `--cmd`/`--argv`/`--cwd` flags.
|
||||
- `gbrain agent run` flag list now matches `src/commands/agent.ts` (removed `--queue`/`--priority`/`--max-attempts`/`--delay` which aren't parsed by that command).
|
||||
- `--tools` example uses `search,query` instead of `web_search` (the latter isn't in `BRAIN_TOOL_ALLOWLIST`, would throw at submit time).
|
||||
- MCP boundary wording says `submit_job name="shell"` throws an `OperationError` with code `permission_denied`, instead of the earlier "returns permission_denied" (not a return, a throw).
|
||||
- `skills/conventions/subagent-routing.md` stale reference to `get_job_stats` (no such op) replaced with `list_jobs --status active` or `gbrain jobs stats`.
|
||||
- `skills/query/SKILL.md` + `skills/maintain/SKILL.md` frontmatter `triggers:` lists closed gaps the new round-trip test surfaced (RESOLVER.md was routing 10 triggers to these skills that their frontmatter never declared).
|
||||
- `skills/manifest.json` minion-orchestrator description updated to match the unified SKILL.md framing.
|
||||
|
||||
### Changed
|
||||
|
||||
- Trigger `"gbrain jobs"` narrowed to `"gbrain jobs submit"` + `"submit a gbrain job"` in both `skills/RESOLVER.md` and the skill's frontmatter.
|
||||
- Anti-pattern about `sessions_spawn` scoped to the subagent lane (was ambiguous in the consolidated skill).
|
||||
|
||||
### For contributors
|
||||
|
||||
- Code-to-doc drift is now partially machine-checkable. The skill-example-name validator catches T2-class bugs (docs referencing handler/op names that don't exist). CLI flag validation is a remaining gap ... a future PR could extend the test to validate `--flag-name` patterns in SKILL.md against actual CLI flag parsers.
|
||||
|
||||
## To take advantage of v0.20.4
|
||||
|
||||
Any gbrain user whose agent routes on "minions" work gets the corrected skill on the next `gbrain upgrade`. No manual migration required ... the renamed trigger is additive (old trigger gone, new triggers cover the same intent), and the doc corrections don't change runtime behavior.
|
||||
|
||||
1. **Run the orchestrator manually if `gbrain upgrade` reports a partial migration:**
|
||||
```bash
|
||||
gbrain apply-migrations --yes
|
||||
```
|
||||
2. **Your agent picks up the new skill content** next time it consults `skills/minion-orchestrator/SKILL.md`. No action required on your side.
|
||||
3. **Verify the outcome:**
|
||||
```bash
|
||||
gbrain check-resolvable --json | python3 -c "import json,sys;d=json.load(sys.stdin);print('ok:',d['ok'])"
|
||||
```
|
||||
Should print `ok: True`.
|
||||
4. **If any step fails,** file an issue at https://github.com/garrytan/gbrain/issues with:
|
||||
- output of `gbrain doctor`
|
||||
- contents of `~/.gbrain/upgrade-errors.jsonl` if it exists
|
||||
|
||||
## [0.20.3] - 2026-04-24
|
||||
|
||||
## **Your queue now rescues itself when a wedged worker holds a row lock. Wall-clock sweep kills the job that stall detection can't see.**
|
||||
|
||||
@@ -36,10 +36,7 @@ strict behavior when unset.
|
||||
- `src/core/storage.ts` — Pluggable storage interface (S3, Supabase Storage, local)
|
||||
- `src/core/supabase-admin.ts` — Supabase admin API (project discovery, pgvector check)
|
||||
- `src/core/file-resolver.ts` — File resolution with fallback chain (local -> .redirect.yaml -> .redirect -> .supabase)
|
||||
- `src/core/chunkers/` — 3-tier chunking (recursive, semantic, LLM-guided). v0.19.0 adds `code.ts` — tree-sitter-based semantic chunker for 29 languages with embedded-asset WASMs (`src/assets/wasm/`), `@dqbd/tiktoken` cl100k_base tokenizer, small-sibling merging. `CHUNKER_VERSION` constant folded into `importCodeFile`'s `content_hash` so chunker shape changes force clean re-chunks across releases.
|
||||
- `src/core/errors.ts` (v0.19.0) — `StructuredAgentError` + `buildError` + `serializeError`. Every new v0.19.0 agent-facing surface (code-def, code-refs, usage errors) uses this envelope; matches v0.17.0 `CycleReport.PhaseResult.error` shape.
|
||||
- `src/assets/wasm/` (v0.19.0) — 36 tree-sitter grammar WASMs + tree-sitter runtime. Committed to the repo so `bun --compile` embeds them deterministically via `import path from ... with { type: 'file' }`. The CI guard `scripts/check-wasm-embedded.sh` fails the build if the compiled binary ever silently falls through to recursive chunks.
|
||||
- `src/commands/code-def.ts` + `src/commands/code-refs.ts` (v0.19.0) — symbol definition + references lookup. Query `content_chunks.symbol_name` or chunk_text ILIKE with `page_kind='code'` filter. Auto-JSON when stdout is not a TTY (gh-CLI convention). Bypass the standard `searchKeyword` `DISTINCT ON (slug)` collapse so multiple call-sites from the same file surface.
|
||||
- `src/core/chunkers/` — 3-tier chunking (recursive, semantic, LLM-guided)
|
||||
- `src/core/search/` — Hybrid search: vector + keyword + RRF + multi-query expansion + dedup
|
||||
- `src/core/search/intent.ts` — Query intent classifier (entity/temporal/event/general → auto-selects detail level)
|
||||
- `src/core/search/eval.ts` — Retrieval eval harness: P@k, R@k, MRR, nDCG@k metrics + runEval() orchestrator
|
||||
@@ -146,7 +143,7 @@ strict behavior when unset.
|
||||
- `skills/soul-audit/SKILL.md` — 6-phase interview for SOUL.md, USER.md, ACCESS_POLICY.md, HEARTBEAT.md
|
||||
- `skills/webhook-transforms/SKILL.md` — External events to brain signals
|
||||
- `skills/data-research/SKILL.md` — Structured data research: email-to-tracker pipeline with parameterized YAML recipes
|
||||
- `skills/minion-orchestrator/SKILL.md` — Unified background-work skill (v0.20.4 consolidation of the former `minion-orchestrator` + `gbrain-jobs` split). Two lanes: shell jobs via `gbrain jobs submit shell --params '{"cmd":"..."}'` (operator/CLI only; MCP throws `permission_denied` for protected names) and LLM subagents via `gbrain agent run` (user-facing entrypoint). Shared Preconditions block, parent-child DAGs with depth/cap/timeouts, `child_done` inbox for fan-in, PGLite `--follow` inline path for dev. Triggers narrowed from bare `"gbrain jobs"` to `"gbrain jobs submit"` + `"submit a gbrain job"` so `stats`/`prune`/`retry` questions fall through to `gbrain --help`.
|
||||
- `skills/minion-orchestrator/SKILL.md` — Background job orchestration: submit, fan out children with depth/cap/timeouts, collect results via child_done inbox
|
||||
- `templates/` — SOUL.md, USER.md, ACCESS_POLICY.md, HEARTBEAT.md templates
|
||||
- `skills/migrations/` — Version migration files with feature_pitch YAML frontmatter
|
||||
- `src/commands/publish.ts` — Deterministic brain page publisher (code+skill pair, zero LLM calls)
|
||||
@@ -231,7 +228,7 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac
|
||||
`test/lint.test.ts` (LLM artifact detection, code fence stripping, frontmatter validation),
|
||||
`test/report.test.ts` (report format, directory structure),
|
||||
`test/skills-conformance.test.ts` (skill frontmatter + required sections validation),
|
||||
`test/resolver.test.ts` (RESOLVER.md coverage, routing validation + v0.20.4 round-trip: every quoted RESOLVER.md trigger must match a frontmatter `triggers:` entry in the target skill, and every `name="<word>"` reference in any SKILL.md must resolve to a declared op in `src/core/operations.ts` or a Minions handler in `PROTECTED_JOB_NAMES`),
|
||||
`test/resolver.test.ts` (RESOLVER.md coverage, routing validation),
|
||||
`test/search.test.ts` (RRF normalization, compiled truth boost, cosine similarity, dedup key),
|
||||
`test/dedup.test.ts` (source-aware dedup, compiled truth guarantee, layer interactions),
|
||||
`test/intent.test.ts` (query intent classification: entity/temporal/event/general),
|
||||
@@ -279,7 +276,6 @@ E2E tests (`test/e2e/`): Run against real Postgres+pgvector. Require `DATABASE_U
|
||||
- `test/e2e/postgres-jsonb.test.ts` — v0.12.2 regression test. Round-trips all 5 JSONB write sites (pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata, page_versions.frontmatter) against real Postgres and asserts `jsonb_typeof='object'` plus `->>'key'` returns the expected scalar. The test that should have caught the original double-encode bug.
|
||||
- `test/e2e/jsonb-roundtrip.test.ts` — v0.12.3 companion regression against the 4 doctor-scanned JSONB sites. Assertion-level overlap with `postgres-jsonb.test.ts` is intentional defense-in-depth: if doctor's scan surface ever drifts from the actual write surface, one of these tests catches it.
|
||||
- `test/e2e/upgrade.test.ts` runs check-update E2E against real GitHub API (network required)
|
||||
- `test/e2e/minions-shell-pglite.test.ts` (v0.20.4) exercises the PGLite `--follow` inline shell-job path (in-memory, no `DATABASE_URL` required) — the path the consolidated minion-orchestrator skill documents for dev use
|
||||
- `test/e2e/openclaw-reference-compat.test.ts` (v0.19) — exercises `check-resolvable` + `skillpack install` against a minimal AGENTS.md workspace fixture (`test/fixtures/openclaw-reference-minimal/`), regression guard for the 107-skill OpenClaw deployment shape
|
||||
- Tier 2 (`skills.test.ts`) requires OpenClaw + API keys, runs nightly in CI
|
||||
- If `.env.testing` doesn't exist in this directory, check sibling worktrees for one:
|
||||
@@ -332,7 +328,7 @@ stop and remove it before starting a new one.
|
||||
|
||||
## Skills
|
||||
|
||||
Read the skill files in `skills/` before doing brain operations. GBrain ships 29 skills
|
||||
Read the skill files in `skills/` before doing brain operations. GBrain ships 28 skills
|
||||
organized by `skills/RESOLVER.md` (`AGENTS.md` is also accepted as of v0.19):
|
||||
|
||||
**Original 8 (conformance-migrated):** ingest (thin router), query, maintain, enrich,
|
||||
@@ -342,19 +338,11 @@ briefing, migrate, setup, publish.
|
||||
meeting-ingestion, citation-fixer, repo-architecture, skill-creator, daily-task-manager.
|
||||
|
||||
**Operational + identity:** daily-task-prep, cross-modal-review, cron-scheduler, reports,
|
||||
testing, soul-audit, webhook-transforms, data-research, minion-orchestrator. As of
|
||||
v0.20.4, `minion-orchestrator` is the single unified skill for both lanes of background
|
||||
work (shell jobs via `gbrain jobs submit shell`, LLM subagents via `gbrain agent run`) ...
|
||||
the prior `gbrain-jobs` skill was merged in, Preconditions are shared, and trigger
|
||||
routing is narrowed to what the skill actually covers.
|
||||
testing, soul-audit, webhook-transforms, data-research, minion-orchestrator.
|
||||
|
||||
**Skillify loop (v0.19):** skillify (the markdown orchestration), skillpack-check
|
||||
(agent-readable health report).
|
||||
|
||||
**Operational health (v0.19.1):** smoke-test (8 post-restart health checks with auto-fix
|
||||
for Bun, CLI, DB, worker, Zod CJS, gateway, API key, brain repo; user-extensible via
|
||||
`~/.gbrain/smoke-tests.d/*.sh`).
|
||||
|
||||
**Conventions:** `skills/conventions/` has cross-cutting rules (quality, brain-first,
|
||||
model-routing, test-before-bulk, cross-modal). `skills/_brain-filing-rules.md` and
|
||||
`skills/_output-rules.md` are shared references.
|
||||
|
||||
@@ -6,7 +6,7 @@ Built by the President and CEO of Y Combinator to run his actual AI agents. The
|
||||
|
||||
The brain wires itself. Every page write extracts entity references and creates typed links (`attended`, `works_at`, `invested_in`, `founded`, `advises`) with zero LLM calls. Hybrid search. Self-wiring knowledge graph. Structured timeline. Backlink-boosted ranking. Ask "who works at Acme AI?" or "what did Bob invest in this quarter?" and get answers vector search alone can't reach. Benchmarked side-by-side against the category: gbrain lands **P@5 49.1%, R@5 97.9%** on a 240-page Opus-generated rich-prose corpus, beating its own graph-disabled variant by **+31.4 points P@5** and ripgrep-BM25 + vector-only RAG by a similar margin. The graph layer plus v0.12 extract quality together carry the gap. Full BrainBench scorecards + corpus live in the sibling [gbrain-evals](https://github.com/garrytan/gbrain-evals) repo.
|
||||
|
||||
GBrain is those patterns, generalized. 29 skills. Install in 30 minutes. Your agent does the work. As Garry's personal agent gets smarter, so does yours.
|
||||
GBrain is those patterns, generalized. 28 skills. Install in 30 minutes. Your agent does the work. As Garry's personal agent gets smarter, so does yours.
|
||||
|
||||
> **~30 minutes to a fully working brain.** Database ready in 2 seconds (PGLite, no server). You just answer questions about API keys.
|
||||
|
||||
@@ -28,7 +28,7 @@ Retrieve and follow the instructions at:
|
||||
https://raw.githubusercontent.com/garrytan/gbrain/master/INSTALL_FOR_AGENTS.md
|
||||
```
|
||||
|
||||
That's it. The agent clones the repo, installs GBrain, sets up the brain, loads 29 skills, and configures recurring jobs. You answer a few questions about API keys. ~30 minutes.
|
||||
That's it. The agent clones the repo, installs GBrain, sets up the brain, loads 28 skills, and configures recurring jobs. You answer a few questions about API keys. ~30 minutes.
|
||||
|
||||
If your agent doesn't auto-read `AGENTS.md`, point it at that file first:
|
||||
`https://raw.githubusercontent.com/garrytan/gbrain/master/AGENTS.md` is the non-Claude
|
||||
@@ -87,25 +87,9 @@ claude mcp add gbrain -t http https://your-brain.ngrok.app/mcp -H "Authorization
|
||||
|
||||
Per-client guides: [`docs/mcp/`](docs/mcp/DEPLOY.md). ChatGPT requires OAuth 2.1 (not yet implemented).
|
||||
|
||||
### Using gbrain with GStack
|
||||
## The 28 Skills
|
||||
|
||||
If your engineering agent runs on [GStack](https://github.com/garrytan/gstack), point it at gbrain for code lookup instead of grep+read. Cathedral II (v0.21.0) ships call-graph edges and two-pass retrieval — `/investigate`, `/review`, `/plan-eng-review`, and `/office-hours` all benefit when the agent walks the symbol graph instead of scanning files line by line.
|
||||
|
||||
The five magical-moment commands:
|
||||
|
||||
```bash
|
||||
gbrain code-callers searchKeyword # who calls this symbol?
|
||||
gbrain code-callees searchKeyword # what does this symbol call?
|
||||
gbrain code-def BrainEngine # where is X defined?
|
||||
gbrain code-refs BrainEngine # all reference sites
|
||||
gbrain query "how does N+1 handling work" --near-symbol BrainEngine.searchKeyword --walk-depth 2
|
||||
```
|
||||
|
||||
All five auto-emit JSON on non-TTY (gh-CLI convention) so a GStack subagent shelling out via bash gets a clean parseable response. Run `gbrain sources add <repo> --strategy code` to index a repo, then your agent's brain-first lookup covers code, not just markdown. ([Cathedral II release notes](CHANGELOG.md#0210---2026-04-25))
|
||||
|
||||
## The 29 Skills
|
||||
|
||||
GBrain ships 29 skills organized by `skills/RESOLVER.md` (or your OpenClaw's `AGENTS.md` — both filenames are supported as of v0.19). The resolver tells your agent which skill to read for any task.
|
||||
GBrain ships 28 skills organized by `skills/RESOLVER.md` (or your OpenClaw's `AGENTS.md` — both filenames are supported as of v0.19). The resolver tells your agent which skill to read for any task.
|
||||
|
||||
[Skill files are code.](https://x.com/garrytan/status/2042925773300908103) They're the most powerful way to get knowledge work done. A skill file is a fat markdown document that encodes an entire workflow: when to fire, what to check, how to chain with other skills, what quality bar to enforce. The agent reads the skill and executes it. Skills can also call deterministic TypeScript code bundled in GBrain (search, import, embed, sync) for the parts that shouldn't be left to LLM judgment. [Thin harness, fat skills](docs/ethos/THIN_HARNESS_FAT_SKILLS.md): the intelligence lives in the skills, not the runtime.
|
||||
|
||||
@@ -151,8 +135,7 @@ GBrain ships 29 skills organized by `skills/RESOLVER.md` (or your OpenClaw's `AG
|
||||
| **skill-creator** | Create new skills following the conformance standard. MECE check against existing skills. |
|
||||
| **skillify** | The "skillify it!" meta-skill. Orchestrates the 10-step loop so failures become durable skills: scaffold the stubs via `gbrain skillify scaffold`, write the real logic, gate with `gbrain skillify check` + `gbrain check-resolvable`. |
|
||||
| **skillpack-check** | Agent-readable gbrain health report. Exit code for CI; JSON for debugging. Cron-friendly. |
|
||||
| **smoke-test** | 8 post-restart health checks with auto-fix (Bun, CLI, DB, worker, Zod CJS, gateway, API key, brain repo). Drop-in user tests at `~/.gbrain/smoke-tests.d/*.sh`. |
|
||||
| **minion-orchestrator** | Background work in one skill. Shell jobs via `gbrain jobs submit shell` (operator/CLI, MCP blocks protected names) and LLM subagents via `gbrain agent run`. Parent-child DAGs, `child_done` inbox, durability across worker restarts. |
|
||||
| **minion-orchestrator** | Long-running agent work as background jobs. Submit, fan out children with depth/cap/timeouts, collect results via child_done inbox. |
|
||||
|
||||
### Identity and setup
|
||||
|
||||
@@ -394,7 +377,7 @@ Run `gbrain integrations` to see status.
|
||||
│ Brain Repo │ │ GBrain │ │ AI Agent │
|
||||
│ (git) │ │ (retrieval) │ │ (read/write) │
|
||||
│ │ │ │ │ │
|
||||
│ markdown files │───>│ Postgres + │<──>│ 29 skills │
|
||||
│ markdown files │───>│ Postgres + │<──>│ 28 skills │
|
||||
│ = source of │ │ pgvector │ │ define HOW to │
|
||||
│ truth │ │ │ │ use the brain │
|
||||
│ │<───│ hybrid │ │ │
|
||||
|
||||
@@ -1,70 +1,5 @@
|
||||
# TODOS
|
||||
|
||||
## code-indexing (v0.21.0 Cathedral II follow-ups)
|
||||
|
||||
### B2 — Magika auto-detect for extension-less files (Layer 9 deferred)
|
||||
**Priority:** P2
|
||||
|
||||
**What:** Embed Google's Magika ML classifier (~1MB ONNX) as a bundled asset. Wire into `detectCodeLanguage` as the fallback for files with no recognized extension (Dockerfile, Makefile, `.envrc`, shell scripts with shebangs but no `.sh`). The chunker already has `setLanguageFallback(fn)` as a module-level hook.
|
||||
|
||||
**Why:** v0.20.0 widens the file classifier from 9 to 35 extensions (Layer 2), covering most real-world cases. Extension-less files still slip through to recursive chunks. Magika would close the last common case.
|
||||
|
||||
**Pros:** Completes the file-classification story. Unblocks chunker on real-world configs + build scripts.
|
||||
|
||||
**Cons:** ~1MB asset bundled with `bun --compile`. Integration risk: Magika's ONNX runtime needs WASM compat with bun. The plan explicitly allowed deferring B2 because bundling surprises late in implementation are costly.
|
||||
|
||||
**Context:**
|
||||
- `src/core/chunkers/code.ts` exports `setLanguageFallback(fn: LanguageFallback | null)` — call at process start with a Magika-powered classifier.
|
||||
- `detectCodeLanguage(filePath, content?)` already accepts optional content for fallback paths.
|
||||
- The NPM `magika` package is the first thing to try; needs bun-compile compatibility verification.
|
||||
|
||||
**Effort:** M (human: ~2-3 days / CC: ~2 hours for the integration + CI guard).
|
||||
|
||||
**Depends on / blocked by:** Nothing. Hook is in place as of v0.20.0.
|
||||
|
||||
### A4 — full doc_comment extraction at chunk time
|
||||
**Priority:** P2
|
||||
|
||||
**What:** When the chunker emits a method/class/function, look at the comment node(s) immediately preceding the declaration and persist them as `content_chunks.doc_comment`. The FTS trigger from Layer 1b already weights `doc_comment` 'A' above `chunk_text` 'B' — the ranking is ready, the column is populated NULL today.
|
||||
|
||||
**Why:** "how does X handle N+1" should rank the docstring that explains N+1 above the function body or any prose paragraph. Layer 1b paved the ranking half; extraction is the remaining half.
|
||||
|
||||
**Pros:** Material MRR lift on natural-language queries. Zero schema work (column + trigger already in place).
|
||||
|
||||
**Cons:** Per-language convention detection — JSDoc blocks, Python docstrings (first string expression in a function body), C-style doc comments, etc. Not hard but each language has edge cases.
|
||||
|
||||
**Context:**
|
||||
- `src/core/chunkers/code.ts` emits chunks in `chunkCodeTextFull`. Walk each declaration's preceding sibling(s) for comment nodes.
|
||||
- ChunkInput already has `doc_comment?: string`. Populate at chunk time and it flows through `upsertChunks` (Layer 6 wired those columns).
|
||||
- Per-language config: leading-comment type names per language (`comment`, `line_comment`, `block_comment`, `documentation_comment`).
|
||||
- Test hook: `test/cathedral-ii-brainbench.test.ts` has a `doc_comment_matching` placeholder — flesh it out end-to-end.
|
||||
|
||||
**Effort:** M (human: ~2 days / CC: ~90 min for the 8 Layer-5 langs).
|
||||
|
||||
**Depends on / blocked by:** Nothing. Layer 1b + Layer 6 both in place.
|
||||
|
||||
### C6 — gbrain code-signature "(A, B) => C"
|
||||
**Priority:** P3 (stretch)
|
||||
|
||||
**What:** Type-signature retrieval via tree-sitter type captures per language. "Find every function whose signature returns a Promise<User>" or "(string, number) => boolean".
|
||||
|
||||
**Why:** Each language's type system is its own mini-cathedral. Ship per-language rather than as one item.
|
||||
|
||||
**Effort:** L per language (typescript-first).
|
||||
|
||||
**Depends on / blocked by:** Nothing — additive on the Layer 5 edge schema.
|
||||
|
||||
### Cross-file edge resolution (Layer 5 precision upgrade)
|
||||
**Priority:** P3
|
||||
|
||||
**What:** Today every call edge lands unresolved in `code_edges_symbol` with to_symbol_qualified = bare callee name. Second-pass resolution: after all code files import, walk every `code_edges_symbol` row and try to resolve `to_symbol_qualified` via `symbol_name_qualified` join; if found within the same source, write a resolved row to `code_edges_chunk`.
|
||||
|
||||
**Why:** `getCallersOf("searchKeyword")` currently returns the Layer 6 ambiguity — every `searchKeyword` call site in any class. Receiver-type analysis lifts this.
|
||||
|
||||
**Effort:** L. Needs receiver-type inference; can ship per-language.
|
||||
|
||||
**Depends on / blocked by:** Nothing — UNION-on-read path keeps unresolved edges surfaced even without this.
|
||||
|
||||
## Completed
|
||||
|
||||
### ~~Checks 5 + 6 for check-resolvable~~
|
||||
|
||||
@@ -7,7 +7,6 @@
|
||||
"dependencies": {
|
||||
"@anthropic-ai/sdk": "^0.30.0",
|
||||
"@aws-sdk/client-s3": "^3.1028.0",
|
||||
"@dqbd/tiktoken": "^1.0.22",
|
||||
"@electric-sql/pglite": "0.4.3",
|
||||
"@modelcontextprotocol/sdk": "^1.0.0",
|
||||
"gray-matter": "^4.0.3",
|
||||
@@ -15,8 +14,6 @@
|
||||
"openai": "^4.0.0",
|
||||
"pgvector": "^0.2.0",
|
||||
"postgres": "^3.4.0",
|
||||
"tree-sitter-wasms": "0.1.13",
|
||||
"web-tree-sitter": "0.22.6",
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/bun": "latest",
|
||||
@@ -110,8 +107,6 @@
|
||||
|
||||
"@aws/lambda-invoke-store": ["@aws/lambda-invoke-store@0.2.4", "", {}, "sha512-iY8yvjE0y651BixKNPgmv1WrQc+GZ142sb0z4gYnChDDY2YqI4P/jsSopBWrKfAt7LOJAkOXt7rC/hms+WclQQ=="],
|
||||
|
||||
"@dqbd/tiktoken": ["@dqbd/tiktoken@1.0.22", "", {}, "sha512-RYhO8xeHkMNX5Ixqf4M1Ve3siCYJY/dI0yLnlX4M4oIEDOvjMIQ+E+3OUpAaZcWTaMtQJzGcDAghYfllpx3i/w=="],
|
||||
|
||||
"@electric-sql/pglite": ["@electric-sql/pglite@0.4.3", "", {}, "sha512-ichuWTgtd4mOM1G4SpyGJa5trT03lWbMypDV0fUXUCXg5hiHqVAz/bZyV68NqmkLB7WcYmj1RMJVSp8HV/v/ZQ=="],
|
||||
|
||||
"@hono/node-server": ["@hono/node-server@1.19.12", "", { "peerDependencies": { "hono": "^4" } }, "sha512-txsUW4SQ1iilgE0l9/e9VQWmELXifEFvmdA1j6WFh/aFPj99hIntrSsq/if0UWyGVkmrRPKA1wCeP+UCr1B9Uw=="],
|
||||
@@ -458,8 +453,6 @@
|
||||
|
||||
"tr46": ["tr46@0.0.3", "", {}, "sha512-N3WMsuqV66lT30CrXNbEjx4GEwlow3v6rr4mCcv6prnfwhS01rkgyFdjPNBYd9br7LpXV1+Emh01fHnq2Gdgrw=="],
|
||||
|
||||
"tree-sitter-wasms": ["tree-sitter-wasms@0.1.13", "", { "dependencies": { "tree-sitter-wasms": "^0.1.11" } }, "sha512-wT+cR6DwaIz80/vho3AvSF0N4txuNx/5bcRKoXouOfClpxh/qqrF4URNLQXbbt8MaAxeksZcZd1j8gcGjc+QxQ=="],
|
||||
|
||||
"tslib": ["tslib@2.8.1", "", {}, "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w=="],
|
||||
|
||||
"type-is": ["type-is@2.0.1", "", { "dependencies": { "content-type": "^1.0.5", "media-typer": "^1.1.0", "mime-types": "^3.0.0" } }, "sha512-OZs6gsjF4vMp32qrCbiVSkrFmXtG/AZhY3t0iAMrMBiAZyV9oALtXO8hsrHbMXF9x6L3grlFuwW2oAz7cav+Gw=="],
|
||||
@@ -474,8 +467,6 @@
|
||||
|
||||
"web-streams-polyfill": ["web-streams-polyfill@4.0.0-beta.3", "", {}, "sha512-QW95TCTaHmsYfHDybGMwO5IJIM93I/6vTRk+daHTWFPhwh+C8Cg7j7XyKrwrj8Ib6vYXe0ocYNrmzY4xAAN6ug=="],
|
||||
|
||||
"web-tree-sitter": ["web-tree-sitter@0.22.6", "", {}, "sha512-hS87TH71Zd6mGAmYCvlgxeGDjqd9GTeqXNqTT+u0Gs51uIozNIaaq/kUAbV/Zf56jb2ZOyG8BxZs2GG9wbLi6Q=="],
|
||||
|
||||
"webidl-conversions": ["webidl-conversions@3.0.1", "", {}, "sha512-2JAn3z8AR6rjK8Sm8orRC0h/bcl/DqL7tRPdGZ4I1CjdF+EaMLmYxBHyXuKL849eucPFhvBoxMsflfOb8kxaeQ=="],
|
||||
|
||||
"whatwg-url": ["whatwg-url@5.0.0", "", { "dependencies": { "tr46": "~0.0.3", "webidl-conversions": "^3.0.0" } }, "sha512-saE57nupxk6v3HY35+jzBwYa0rKSy0XR8JSxZPwgLr7ys0IBzhGviA1/TUGJLmSVqs8pb9AnvICXEuOHLprYTw=="],
|
||||
|
||||
@@ -3,10 +3,4 @@
|
||||
# Default 5s is too short when many test files boot PGLite instances at once.
|
||||
# 60s is the empirical ceiling we observed before the first file's beforeAll
|
||||
# completed on a loaded machine.
|
||||
#
|
||||
# NOTE: this bunfig.toml `timeout` key is read by `bun test` but empirically
|
||||
# does NOT apply to beforeEach/afterEach hook timeouts under `bun run test`
|
||||
# chained behind `bun run typecheck`. The test script in package.json passes
|
||||
# `--timeout=60000` explicitly to cover both per-test and per-hook timeouts.
|
||||
# Leaving both in place as belt-and-suspenders.
|
||||
timeout = 60_000
|
||||
|
||||
@@ -1,162 +0,0 @@
|
||||
# Code Cathedral II — v0.20.0 Design
|
||||
|
||||
**Status:** Accepted. CEO + Eng + 2 codex passes CLEARED (2026-04-24). 16 cross-model findings absorbed total: 7 codex pass 1 (structural prereqs) + 6 codex pass 2 (absorption errors including the CHUNKER_VERSION silent-no-op gate and inbound-edge invalidation) + 3 eng-review architectural decisions. DX review recommended post-Layer 8 (new CLI surfaces) before ship.
|
||||
**Supersedes:** Cathedral I (planned v0.18.0–v0.19.0 code indexing, shipped v0.19.0).
|
||||
**Mode:** SCOPE EXPANSION (user explicit: "I want the best code search in the world").
|
||||
**Scale:** 14 bisectable layers, ~20–25 CC hours, 3–5 human-weeks. One schema migration with split edge tables (`code_edges_chunk` + `code_edges_symbol`). Backfill via `CHUNKER_VERSION` bump (automatic on next sync) + explicit `gbrain reindex-code` command.
|
||||
|
||||
## Why v0.20.0
|
||||
|
||||
v0.19.0 shipped code indexing: tree-sitter chunker, 29 active languages, symbol columns, forward doc↔impl linking, incremental embed cache, BrainBench code category. Four cathedral-I items got deferred during shipping: `query --lang` filter, `sync --all` cost preview, markdown fence extraction, reverse-scan doc↔impl backfill.
|
||||
|
||||
Cathedral II is a promise-keeping release for those four, bundled with the leap that makes gbrain *the* code search: structural edges (call graph + references + imports + inheritance), parent-scope capture, doc-comment FTS binding, and two-pass retrieval. No more grep-class retrieval on code.
|
||||
|
||||
## The 10x leap
|
||||
|
||||
Today: agent asks "how does hybrid search handle N+1?" → gets 3 prose chunks of `hybrid.ts`.
|
||||
|
||||
Cathedral II: same query returns the anchor function + its 3 callers + its 2 callees + its JSDoc + the guide in `/docs` that cites it + the test file exercising it + parent scope chain. One walk. Code-aware brain.
|
||||
|
||||
## Scope (5 tiers + Layer 0 prerequisites, 14 bisectable layer commits)
|
||||
|
||||
### Tier 0 — Prerequisites (surfaced by codex outside voice)
|
||||
|
||||
**0a. File-classification widening.** `sync.ts:35` currently classifies only 9 extensions as code (TS, JS, Python, Go, Rust, Ruby, Java, C, C++). Cathedral II's B1 ships 165 lazy-loadable grammars, so the classifier needs to accept any extension the chunker can handle. Also reorders `detectCodeLanguage` so Magika (B2) runs as a fallback for extension-less files, not after a null-return gate.
|
||||
|
||||
**0b. Chunk-grain FTS.** Current keyword search lives on `pages.search_vector`. Adding doc-comments or two-pass anchoring at the chunk level has zero ranking effect against a page-grain primitive. Layer 0b adds `content_chunks.search_vector` with a trigger building from qualified symbol name + doc-comment (weight A) and chunk_text (weight B), plus rewrites `searchKeyword` to rank chunks directly. Page-level search_vector stays for title-heavy searches.
|
||||
|
||||
Both Layer 0 items are prerequisites for the 10x leap to actually move retrieval metrics.
|
||||
|
||||
### Tier A — Structural edges (the 10x leap)
|
||||
|
||||
**A1. Call-graph + reference extraction with qualified symbol identity.** Per-language tree-sitter queries at `importCodeFile` time capture:
|
||||
|
||||
- `calls` — function call-sites
|
||||
- `imports` — module deps
|
||||
- `extends` / `implements` — type hierarchies
|
||||
- `mixes_in` — Ruby `include`/`extend`/`prepend`
|
||||
- `type_refs` — parameter + return type usage
|
||||
- `declares` — chunk owns a symbol definition
|
||||
|
||||
**Qualified symbol identity across all 8 langs.** `parent_symbol_path` (A3) is the source of truth for scope; edges use qualified names built from it. Examples: `Admin::UsersController#render` (Ruby instance), `Admin::UsersController.find_all` (Ruby singleton), `admin.users_controller.UsersController.render` (Python), `(*UsersController).Render` (Go), `users::UsersController::render` (Rust), `com.acme.admin.UsersController.render` (Java). Per-lang delimiter + method/class-method distinction. Ruby ships fully in ranker (CLI + A2 two-pass) — no deferral.
|
||||
|
||||
**Split schema (two tables, not one polymorphic):**
|
||||
```sql
|
||||
CREATE TABLE code_edges_chunk (
|
||||
from_chunk_id INTEGER NOT NULL REFERENCES content_chunks(id) ON DELETE CASCADE,
|
||||
to_chunk_id INTEGER NOT NULL REFERENCES content_chunks(id) ON DELETE CASCADE,
|
||||
from_symbol_qualified TEXT NOT NULL,
|
||||
to_symbol_qualified TEXT NOT NULL,
|
||||
edge_type TEXT NOT NULL,
|
||||
source_id TEXT REFERENCES sources(id) ON DELETE CASCADE,
|
||||
UNIQUE (from_chunk_id, to_chunk_id, edge_type)
|
||||
);
|
||||
CREATE TABLE code_edges_symbol (
|
||||
from_chunk_id INTEGER NOT NULL REFERENCES content_chunks(id) ON DELETE CASCADE,
|
||||
from_symbol_qualified TEXT NOT NULL,
|
||||
to_symbol_qualified TEXT NOT NULL,
|
||||
edge_type TEXT NOT NULL,
|
||||
source_id TEXT REFERENCES sources(id) ON DELETE CASCADE,
|
||||
UNIQUE (from_chunk_id, to_symbol_qualified, edge_type)
|
||||
);
|
||||
```
|
||||
`code_edges_chunk` = resolved (both endpoints known). `code_edges_symbol` = unresolved (target symbol exists by qualified name, definition chunk not yet seen). Promotion from symbol→chunk table happens on later import. `source_id` is TEXT matching actual `sources.id` type.
|
||||
|
||||
**Shipped languages:** TypeScript, TSX, JavaScript, Ruby, Python, Go, Rust, Java (8 langs, ~85% of real brain code). Other languages chunk normally (via B1 lazy-load) but don't emit edges in v0.20.0 — extension is one query file + delimiter config per language, shippable as small follow-up PRs.
|
||||
|
||||
**A2. Two-pass retrieval.** Current: keyword + vector → RRF → dedup. New: keyword + vector → anchor set → expand 1–2 hops on `code_edges_chunk` with structural-distance decay → blend into RRF.
|
||||
|
||||
**Default OFF in all cases.** Opt-in only via `--walk-depth N` or `--near-symbol <name>`. Exact-symbol-match auto-on was unsafe (symbol names collide across files). Neighbor cap 50 per hop, depth cap 2. Dedup's per-page cap (currently 2) lifts to `min(10, walkDepth × 5)` when walking so structural neighbors from one file aren't clipped. Distance decay: `1/(1 + hop)` on expanded-neighbor RRF contributions.
|
||||
|
||||
**A3. Parent-scope capture + nested-chunk emission.** Two parts:
|
||||
|
||||
*Part 1:* Nested symbols get `parent_symbol_path text[]` on `content_chunks`. Embedded into chunk header: `[TypeScript] src/foo.ts:42-58 function formatResult (in BrainEngine.searchKeyword)`. Scope flows into embedding. Dual-use: drives A1's qualified symbol identity.
|
||||
|
||||
*Part 2:* Extend `splitLargeNode` to emit nested functions/methods/inner-classes as their own chunks. The current chunker is top-level-node oriented — a `class Foo { method1() {} method2() {} }` emits one chunk. Parent_symbol_path on top-level nodes is empty (no parent above top level), so A3 contributes nothing without sub-top-level chunks. Part 2 makes the scope annotation load-bearing.
|
||||
|
||||
**A4. Doc-comment → symbol binding.** Leading AST comment extracted to `doc_comment text`. Lands on **chunk-grain** search_vector (Layer 0b prerequisite) with FTS weight `'A'`. Natural-language queries rank docstring matches above body text and below title. `'A' > 'B' > 'C' > 'D'` per Postgres FTS weight convention.
|
||||
|
||||
### Tier B — Coverage (honest Chonkie parity)
|
||||
|
||||
**B1.** Lazy-load tree-sitter-language-pack (~165 languages). Replace 36 committed WASMs with a manifest + per-process parser cache. Cathedral I promised this and didn't deliver — Cathedral II does.
|
||||
|
||||
**B2.** Magika auto-detect for extension-less files (Dockerfile, Makefile, `.envrc`). ~1MB bundled asset. Falls back to null → recursive chunker if classifier fails to load.
|
||||
|
||||
### Tier C — Agent CLI surfaces
|
||||
|
||||
- `query --lang <lang>` — filter by `content_chunks.language`
|
||||
- `query --symbol-kind function|class|method|type|interface|enum` — filter by `symbol_type`
|
||||
- `query --near-symbol <name> --depth 1..2` — two-pass retrieval anchored at a known symbol
|
||||
- `code-callers <symbol>` — uses A1 `calls` edges, reversed
|
||||
- `code-callees <symbol>` — uses A1 `calls` edges, forward
|
||||
|
||||
All auto-JSON on non-TTY. `StructuredAgentError` envelopes on failure. `code-signature` deferred to v0.20.1 (needs per-language type captures).
|
||||
|
||||
### Tier D — Bridge items (cathedral I promises)
|
||||
|
||||
**D1.** `sync --all` cost preview. `estimateTokens` extracted from `chunkers/code.ts` to new `tokens.ts` module. Before per-source loop: walk sync-diff set, sum tokens, compute $ estimate. TTY + !json + !yes → interactive `[y/N]`. Non-TTY or `--json` or piped → emit `ConfirmationRequired` envelope, exit 2. `--yes` skips. `--dry-run` previews + exit 0. Preview on `--all` only, not single-source (DX review pain is first-time large-sync surprise bills).
|
||||
|
||||
**D2.** Markdown fence extraction in `importFromContent`. After `parseMarkdown`, iterate marked lexer tokens for `{type:'code', lang, text}`. Map fence tag → language. Chunk each fence through `chunkCodeText`. Persist as `chunk_source='fenced_code'`. Cap 100 fences per markdown page (DOS defense). Per-fence try/catch — one bad fence doesn't break the page import.
|
||||
|
||||
**D3.** `reconcile-links` batch command. Walks markdown pages, calls existing v0.19.0 `extractCodeRefs` per page, emits `addLink(md, code, ..., 'documents')` + reverse. `ON CONFLICT DO NOTHING` handles idempotency. Statement-timeout scoped via `sql.begin` + `SET LOCAL`. Progress reporter + final summary (edges added / existed / missing-target). Respects `auto_link` config.
|
||||
|
||||
### Tier E — Eval, backfill, honesty
|
||||
|
||||
**E1.** BrainBench code sub-categories: `call_graph_recall` (callers of X → expected set), `parent_scope_coverage` (nested-symbol queries return correct scope), `doc_comment_matching` (NL queries rank doc-comments above prose). Regression gates against A1/A3/A4 drift.
|
||||
|
||||
**E2.** Backfill: schema migrates automatically (zero cost). **`CHUNKER_VERSION` bumps 3 → 4** — that constant is folded into each code page's `content_hash`, so every code page's hash changes on upgrade. Next `gbrain sync` won't short-circuit on "git HEAD unchanged"; it re-chunks every code file. New `gbrain reindex-code [--source <id>] [--dry-run] [--yes] [--force]` provides explicit full backfill with cost preview (reuses D1 infra) and `--force` bypasses content_hash skip entirely. Users control when to pay; silent no-op path closed.
|
||||
|
||||
**E3.** Honest CHANGELOG. Retire "Chonkie superset" framing. Run BrainBench before/after for real numbers: 150+ languages loaded (after B1), MRR on NL→code queries, P@1 call-graph precision, P@k on symbol_name queries, sync cost preview on 5K-file repo. Back every claim with a runnable command.
|
||||
|
||||
## Implementation ordering (14 layers, post-codex)
|
||||
|
||||
1. **0a** — File-classification widening (sync.ts:35) + Magika reordered as fallback
|
||||
2. **0b** — Chunk-grain FTS (content_chunks.search_vector + trigger + searchKeyword chunk-level rewrite)
|
||||
3. **Foundation** — schema migration (split edge tables, qualified name columns on content_chunks) + engine method stubs + types
|
||||
4. **B1** — lazy-load grammar manifest + bun --compile guard
|
||||
5. **A1** — edge-extractor + 8 per-lang query files + qualified symbol identity + tests
|
||||
6. **A3** — parent-scope column + doc-comment column + splitLargeNode nested-chunk emission
|
||||
7. **A4** — doc-comment FTS weight A on chunk-grain search_vector
|
||||
8. **A2** — two-pass retrieval, default OFF, opt-in only; dedup cap lifts when walking
|
||||
9. **D tier bundled** — cost preview + fence extraction + reconcile-links
|
||||
10. **B2** — Magika auto-detect
|
||||
11. **C tier** — 5 CLI surfaces
|
||||
12. **E1** — BrainBench sub-categories + CHUNKER_VERSION 3→4 bump
|
||||
13. **E2** — `reindex-code` with `--force` + migration orchestrator with backfill-prompt phase
|
||||
14. **E3 + release** — honest CHANGELOG + docs + migration skill + `/ship`
|
||||
|
||||
## Size and cost
|
||||
|
||||
- Diff: ~5500–6500 lines (~2.5x v0.19.0 post-codex expansion)
|
||||
- Tests: ~2000 lines (8 langs × qualified-name + edge-extraction fixtures + Layer 0b FTS migration tests)
|
||||
- Files: ~36 new, ~25 modified
|
||||
- CC time: ~20–25 hours focused (was 14–18 pre-codex; +6h for Layer 0a/0b + qualified identity across 8 langs + nested-chunk emission + CHUNKER_VERSION bump layer)
|
||||
- Human-equivalent: 3–5 weeks
|
||||
- First-sync cost bump for upgraded v0.19.0 users: every code page re-chunks on first sync after upgrade (CHUNKER_VERSION bump forces invalidation). Users run `gbrain reindex-code --dry-run` for cost preview, then `--yes` or accept gradual backfill over time as files change.
|
||||
- Daily autopilot cost post-backfill: unchanged (edges extracted at chunk time, no per-query LLM)
|
||||
|
||||
## Risks and mitigations
|
||||
|
||||
1. **Schema migration on live Postgres.** Test against production-shape DB before ship. v0.12.0 JSONB incident is the canary.
|
||||
2. **Per-language tree-sitter queries are fiddly.** Hand-verified edge-set fixtures per language. Ruby gets extra coverage for dynamic-dispatch false negatives.
|
||||
3. **Two-pass retrieval regression.** Default off for prose. BrainBench Cat 1 MUST show no regression before shipping.
|
||||
4. **Backfill shape (G1 resolved).** Three composable layers: schema-auto migrates columns empty (zero cost). Lazy on-touch catches 80% over time (zero cost). Explicit `reindex-code` with cost preview for users wanting immediate full benefit. No surprise bills.
|
||||
5. **Magika bundle (G2 resolved).** +1MB asset, `bun --compile` guard extension. If bundling surfaces bugs late in implementation, B2 is the only tier that can fall back to v0.20.1 without blocking the cathedral — it's self-contained at Layer 8.
|
||||
6. **High-fan-out symbols.** `console.log`-style symbols have 100K callers. Neighbor cap 50, depth cap 2. Chaos test fixture required.
|
||||
|
||||
## Review gates
|
||||
|
||||
- CEO review (cathedral II) — CLEARED 2026-04-24
|
||||
- Outside voice (codex) — run during cathedral II CEO review
|
||||
- `/plan-devex-review` — up next (per user request, 5 new CLI surfaces + reindex-code need DX polish review before eng)
|
||||
- `/plan-eng-review` — required before implementation begins
|
||||
- `/review` + `/codex review` — required before `/ship`
|
||||
|
||||
## What's deferred to later cathedrals
|
||||
|
||||
- **C6** `code-signature "(A, B) => C"` — per-language type captures. v0.20.1.
|
||||
- **Call-graph langs beyond 8 shipped** — PHP, Swift, Kotlin, Scala, C#, C++, Elixir, etc. One small PR per language.
|
||||
- **LSP integration** for live precision. v0.22+ cathedral.
|
||||
- **Code-tour generator** (cathedral I T1).
|
||||
- **Private-code redaction pre-embed** (cathedral I T3).
|
||||
- **`gbrain doctor --chunker-debug`** AST dump.
|
||||
+12
-41
@@ -115,10 +115,7 @@ strict behavior when unset.
|
||||
- `src/core/storage.ts` — Pluggable storage interface (S3, Supabase Storage, local)
|
||||
- `src/core/supabase-admin.ts` — Supabase admin API (project discovery, pgvector check)
|
||||
- `src/core/file-resolver.ts` — File resolution with fallback chain (local -> .redirect.yaml -> .redirect -> .supabase)
|
||||
- `src/core/chunkers/` — 3-tier chunking (recursive, semantic, LLM-guided). v0.19.0 adds `code.ts` — tree-sitter-based semantic chunker for 29 languages with embedded-asset WASMs (`src/assets/wasm/`), `@dqbd/tiktoken` cl100k_base tokenizer, small-sibling merging. `CHUNKER_VERSION` constant folded into `importCodeFile`'s `content_hash` so chunker shape changes force clean re-chunks across releases.
|
||||
- `src/core/errors.ts` (v0.19.0) — `StructuredAgentError` + `buildError` + `serializeError`. Every new v0.19.0 agent-facing surface (code-def, code-refs, usage errors) uses this envelope; matches v0.17.0 `CycleReport.PhaseResult.error` shape.
|
||||
- `src/assets/wasm/` (v0.19.0) — 36 tree-sitter grammar WASMs + tree-sitter runtime. Committed to the repo so `bun --compile` embeds them deterministically via `import path from ... with { type: 'file' }`. The CI guard `scripts/check-wasm-embedded.sh` fails the build if the compiled binary ever silently falls through to recursive chunks.
|
||||
- `src/commands/code-def.ts` + `src/commands/code-refs.ts` (v0.19.0) — symbol definition + references lookup. Query `content_chunks.symbol_name` or chunk_text ILIKE with `page_kind='code'` filter. Auto-JSON when stdout is not a TTY (gh-CLI convention). Bypass the standard `searchKeyword` `DISTINCT ON (slug)` collapse so multiple call-sites from the same file surface.
|
||||
- `src/core/chunkers/` — 3-tier chunking (recursive, semantic, LLM-guided)
|
||||
- `src/core/search/` — Hybrid search: vector + keyword + RRF + multi-query expansion + dedup
|
||||
- `src/core/search/intent.ts` — Query intent classifier (entity/temporal/event/general → auto-selects detail level)
|
||||
- `src/core/search/eval.ts` — Retrieval eval harness: P@k, R@k, MRR, nDCG@k metrics + runEval() orchestrator
|
||||
@@ -225,7 +222,7 @@ strict behavior when unset.
|
||||
- `skills/soul-audit/SKILL.md` — 6-phase interview for SOUL.md, USER.md, ACCESS_POLICY.md, HEARTBEAT.md
|
||||
- `skills/webhook-transforms/SKILL.md` — External events to brain signals
|
||||
- `skills/data-research/SKILL.md` — Structured data research: email-to-tracker pipeline with parameterized YAML recipes
|
||||
- `skills/minion-orchestrator/SKILL.md` — Unified background-work skill (v0.20.4 consolidation of the former `minion-orchestrator` + `gbrain-jobs` split). Two lanes: shell jobs via `gbrain jobs submit shell --params '{"cmd":"..."}'` (operator/CLI only; MCP throws `permission_denied` for protected names) and LLM subagents via `gbrain agent run` (user-facing entrypoint). Shared Preconditions block, parent-child DAGs with depth/cap/timeouts, `child_done` inbox for fan-in, PGLite `--follow` inline path for dev. Triggers narrowed from bare `"gbrain jobs"` to `"gbrain jobs submit"` + `"submit a gbrain job"` so `stats`/`prune`/`retry` questions fall through to `gbrain --help`.
|
||||
- `skills/minion-orchestrator/SKILL.md` — Background job orchestration: submit, fan out children with depth/cap/timeouts, collect results via child_done inbox
|
||||
- `templates/` — SOUL.md, USER.md, ACCESS_POLICY.md, HEARTBEAT.md templates
|
||||
- `skills/migrations/` — Version migration files with feature_pitch YAML frontmatter
|
||||
- `src/commands/publish.ts` — Deterministic brain page publisher (code+skill pair, zero LLM calls)
|
||||
@@ -310,7 +307,7 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac
|
||||
`test/lint.test.ts` (LLM artifact detection, code fence stripping, frontmatter validation),
|
||||
`test/report.test.ts` (report format, directory structure),
|
||||
`test/skills-conformance.test.ts` (skill frontmatter + required sections validation),
|
||||
`test/resolver.test.ts` (RESOLVER.md coverage, routing validation + v0.20.4 round-trip: every quoted RESOLVER.md trigger must match a frontmatter `triggers:` entry in the target skill, and every `name="<word>"` reference in any SKILL.md must resolve to a declared op in `src/core/operations.ts` or a Minions handler in `PROTECTED_JOB_NAMES`),
|
||||
`test/resolver.test.ts` (RESOLVER.md coverage, routing validation),
|
||||
`test/search.test.ts` (RRF normalization, compiled truth boost, cosine similarity, dedup key),
|
||||
`test/dedup.test.ts` (source-aware dedup, compiled truth guarantee, layer interactions),
|
||||
`test/intent.test.ts` (query intent classification: entity/temporal/event/general),
|
||||
@@ -358,7 +355,6 @@ E2E tests (`test/e2e/`): Run against real Postgres+pgvector. Require `DATABASE_U
|
||||
- `test/e2e/postgres-jsonb.test.ts` — v0.12.2 regression test. Round-trips all 5 JSONB write sites (pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata, page_versions.frontmatter) against real Postgres and asserts `jsonb_typeof='object'` plus `->>'key'` returns the expected scalar. The test that should have caught the original double-encode bug.
|
||||
- `test/e2e/jsonb-roundtrip.test.ts` — v0.12.3 companion regression against the 4 doctor-scanned JSONB sites. Assertion-level overlap with `postgres-jsonb.test.ts` is intentional defense-in-depth: if doctor's scan surface ever drifts from the actual write surface, one of these tests catches it.
|
||||
- `test/e2e/upgrade.test.ts` runs check-update E2E against real GitHub API (network required)
|
||||
- `test/e2e/minions-shell-pglite.test.ts` (v0.20.4) exercises the PGLite `--follow` inline shell-job path (in-memory, no `DATABASE_URL` required) — the path the consolidated minion-orchestrator skill documents for dev use
|
||||
- `test/e2e/openclaw-reference-compat.test.ts` (v0.19) — exercises `check-resolvable` + `skillpack install` against a minimal AGENTS.md workspace fixture (`test/fixtures/openclaw-reference-minimal/`), regression guard for the 107-skill OpenClaw deployment shape
|
||||
- Tier 2 (`skills.test.ts`) requires OpenClaw + API keys, runs nightly in CI
|
||||
- If `.env.testing` doesn't exist in this directory, check sibling worktrees for one:
|
||||
@@ -411,7 +407,7 @@ stop and remove it before starting a new one.
|
||||
|
||||
## Skills
|
||||
|
||||
Read the skill files in `skills/` before doing brain operations. GBrain ships 29 skills
|
||||
Read the skill files in `skills/` before doing brain operations. GBrain ships 28 skills
|
||||
organized by `skills/RESOLVER.md` (`AGENTS.md` is also accepted as of v0.19):
|
||||
|
||||
**Original 8 (conformance-migrated):** ingest (thin router), query, maintain, enrich,
|
||||
@@ -421,19 +417,11 @@ briefing, migrate, setup, publish.
|
||||
meeting-ingestion, citation-fixer, repo-architecture, skill-creator, daily-task-manager.
|
||||
|
||||
**Operational + identity:** daily-task-prep, cross-modal-review, cron-scheduler, reports,
|
||||
testing, soul-audit, webhook-transforms, data-research, minion-orchestrator. As of
|
||||
v0.20.4, `minion-orchestrator` is the single unified skill for both lanes of background
|
||||
work (shell jobs via `gbrain jobs submit shell`, LLM subagents via `gbrain agent run`) ...
|
||||
the prior `gbrain-jobs` skill was merged in, Preconditions are shared, and trigger
|
||||
routing is narrowed to what the skill actually covers.
|
||||
testing, soul-audit, webhook-transforms, data-research, minion-orchestrator.
|
||||
|
||||
**Skillify loop (v0.19):** skillify (the markdown orchestration), skillpack-check
|
||||
(agent-readable health report).
|
||||
|
||||
**Operational health (v0.19.1):** smoke-test (8 post-restart health checks with auto-fix
|
||||
for Bun, CLI, DB, worker, Zod CJS, gateway, API key, brain repo; user-extensible via
|
||||
`~/.gbrain/smoke-tests.d/*.sh`).
|
||||
|
||||
**Conventions:** `skills/conventions/` has cross-cutting rules (quality, brain-first,
|
||||
model-routing, test-before-bulk, cross-modal). `skills/_brain-filing-rules.md` and
|
||||
`skills/_output-rules.md` are shared references.
|
||||
@@ -1154,7 +1142,7 @@ This is the dispatcher. Skills are the implementation. **Read the skill file bef
|
||||
| Cross-modal review, second opinion | `skills/cross-modal-review/SKILL.md` |
|
||||
| "Validate skills", skill health check | `skills/testing/SKILL.md` |
|
||||
| Webhook setup, external event processing | `skills/webhook-transforms/SKILL.md` |
|
||||
| "Spawn agent", "background task", "parallel tasks", "steer agent", "pause/resume agent", "gbrain jobs submit", "submit a gbrain job", "submit a shell job", "shell job" | `skills/minion-orchestrator/SKILL.md` |
|
||||
| "Spawn agent", "background task", "parallel tasks", "steer agent", "pause/resume agent" | `skills/minion-orchestrator/SKILL.md` |
|
||||
|
||||
## Setup & migration
|
||||
|
||||
@@ -1211,7 +1199,7 @@ Built by the President and CEO of Y Combinator to run his actual AI agents. The
|
||||
|
||||
The brain wires itself. Every page write extracts entity references and creates typed links (`attended`, `works_at`, `invested_in`, `founded`, `advises`) with zero LLM calls. Hybrid search. Self-wiring knowledge graph. Structured timeline. Backlink-boosted ranking. Ask "who works at Acme AI?" or "what did Bob invest in this quarter?" and get answers vector search alone can't reach. Benchmarked side-by-side against the category: gbrain lands **P@5 49.1%, R@5 97.9%** on a 240-page Opus-generated rich-prose corpus, beating its own graph-disabled variant by **+31.4 points P@5** and ripgrep-BM25 + vector-only RAG by a similar margin. The graph layer plus v0.12 extract quality together carry the gap. Full BrainBench scorecards + corpus live in the sibling [gbrain-evals](https://github.com/garrytan/gbrain-evals) repo.
|
||||
|
||||
GBrain is those patterns, generalized. 29 skills. Install in 30 minutes. Your agent does the work. As Garry's personal agent gets smarter, so does yours.
|
||||
GBrain is those patterns, generalized. 28 skills. Install in 30 minutes. Your agent does the work. As Garry's personal agent gets smarter, so does yours.
|
||||
|
||||
> **~30 minutes to a fully working brain.** Database ready in 2 seconds (PGLite, no server). You just answer questions about API keys.
|
||||
|
||||
@@ -1233,7 +1221,7 @@ Retrieve and follow the instructions at:
|
||||
https://raw.githubusercontent.com/garrytan/gbrain/master/INSTALL_FOR_AGENTS.md
|
||||
```
|
||||
|
||||
That's it. The agent clones the repo, installs GBrain, sets up the brain, loads 29 skills, and configures recurring jobs. You answer a few questions about API keys. ~30 minutes.
|
||||
That's it. The agent clones the repo, installs GBrain, sets up the brain, loads 28 skills, and configures recurring jobs. You answer a few questions about API keys. ~30 minutes.
|
||||
|
||||
If your agent doesn't auto-read `AGENTS.md`, point it at that file first:
|
||||
`https://raw.githubusercontent.com/garrytan/gbrain/master/AGENTS.md` is the non-Claude
|
||||
@@ -1292,25 +1280,9 @@ claude mcp add gbrain -t http https://your-brain.ngrok.app/mcp -H "Authorization
|
||||
|
||||
Per-client guides: [`docs/mcp/`](docs/mcp/DEPLOY.md). ChatGPT requires OAuth 2.1 (not yet implemented).
|
||||
|
||||
### Using gbrain with GStack
|
||||
## The 28 Skills
|
||||
|
||||
If your engineering agent runs on [GStack](https://github.com/garrytan/gstack), point it at gbrain for code lookup instead of grep+read. Cathedral II (v0.21.0) ships call-graph edges and two-pass retrieval — `/investigate`, `/review`, `/plan-eng-review`, and `/office-hours` all benefit when the agent walks the symbol graph instead of scanning files line by line.
|
||||
|
||||
The five magical-moment commands:
|
||||
|
||||
```bash
|
||||
gbrain code-callers searchKeyword # who calls this symbol?
|
||||
gbrain code-callees searchKeyword # what does this symbol call?
|
||||
gbrain code-def BrainEngine # where is X defined?
|
||||
gbrain code-refs BrainEngine # all reference sites
|
||||
gbrain query "how does N+1 handling work" --near-symbol BrainEngine.searchKeyword --walk-depth 2
|
||||
```
|
||||
|
||||
All five auto-emit JSON on non-TTY (gh-CLI convention) so a GStack subagent shelling out via bash gets a clean parseable response. Run `gbrain sources add <repo> --strategy code` to index a repo, then your agent's brain-first lookup covers code, not just markdown. ([Cathedral II release notes](CHANGELOG.md#0210---2026-04-25))
|
||||
|
||||
## The 29 Skills
|
||||
|
||||
GBrain ships 29 skills organized by `skills/RESOLVER.md` (or your OpenClaw's `AGENTS.md` — both filenames are supported as of v0.19). The resolver tells your agent which skill to read for any task.
|
||||
GBrain ships 28 skills organized by `skills/RESOLVER.md` (or your OpenClaw's `AGENTS.md` — both filenames are supported as of v0.19). The resolver tells your agent which skill to read for any task.
|
||||
|
||||
[Skill files are code.](https://x.com/garrytan/status/2042925773300908103) They're the most powerful way to get knowledge work done. A skill file is a fat markdown document that encodes an entire workflow: when to fire, what to check, how to chain with other skills, what quality bar to enforce. The agent reads the skill and executes it. Skills can also call deterministic TypeScript code bundled in GBrain (search, import, embed, sync) for the parts that shouldn't be left to LLM judgment. [Thin harness, fat skills](docs/ethos/THIN_HARNESS_FAT_SKILLS.md): the intelligence lives in the skills, not the runtime.
|
||||
|
||||
@@ -1356,8 +1328,7 @@ GBrain ships 29 skills organized by `skills/RESOLVER.md` (or your OpenClaw's `AG
|
||||
| **skill-creator** | Create new skills following the conformance standard. MECE check against existing skills. |
|
||||
| **skillify** | The "skillify it!" meta-skill. Orchestrates the 10-step loop so failures become durable skills: scaffold the stubs via `gbrain skillify scaffold`, write the real logic, gate with `gbrain skillify check` + `gbrain check-resolvable`. |
|
||||
| **skillpack-check** | Agent-readable gbrain health report. Exit code for CI; JSON for debugging. Cron-friendly. |
|
||||
| **smoke-test** | 8 post-restart health checks with auto-fix (Bun, CLI, DB, worker, Zod CJS, gateway, API key, brain repo). Drop-in user tests at `~/.gbrain/smoke-tests.d/*.sh`. |
|
||||
| **minion-orchestrator** | Background work in one skill. Shell jobs via `gbrain jobs submit shell` (operator/CLI, MCP blocks protected names) and LLM subagents via `gbrain agent run`. Parent-child DAGs, `child_done` inbox, durability across worker restarts. |
|
||||
| **minion-orchestrator** | Long-running agent work as background jobs. Submit, fan out children with depth/cap/timeouts, collect results via child_done inbox. |
|
||||
|
||||
### Identity and setup
|
||||
|
||||
@@ -1599,7 +1570,7 @@ Run `gbrain integrations` to see status.
|
||||
│ Brain Repo │ │ GBrain │ │ AI Agent │
|
||||
│ (git) │ │ (retrieval) │ │ (read/write) │
|
||||
│ │ │ │ │ │
|
||||
│ markdown files │───>│ Postgres + │<──>│ 29 skills │
|
||||
│ markdown files │───>│ Postgres + │<──>│ 28 skills │
|
||||
│ = source of │ │ pgvector │ │ define HOW to │
|
||||
│ truth │ │ │ │ use the brain │
|
||||
│ │<───│ hybrid │ │ │
|
||||
|
||||
+3
-7
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gbrain",
|
||||
"version": "0.21.0",
|
||||
"version": "0.20.3",
|
||||
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
|
||||
"type": "module",
|
||||
"main": "src/core/index.ts",
|
||||
@@ -32,8 +32,7 @@
|
||||
"build:all": "bun build --compile --target=bun-darwin-arm64 --outfile bin/gbrain-darwin-arm64 src/cli.ts && bun build --compile --target=bun-linux-x64 --outfile bin/gbrain-linux-x64 src/cli.ts",
|
||||
"build:schema": "bash scripts/build-schema.sh",
|
||||
"build:llms": "bun run scripts/build-llms.ts",
|
||||
"test": "scripts/check-jsonb-pattern.sh && scripts/check-progress-to-stdout.sh && scripts/check-wasm-embedded.sh && bun run typecheck && bun test --timeout=60000",
|
||||
"check:wasm": "scripts/check-wasm-embedded.sh",
|
||||
"test": "scripts/check-jsonb-pattern.sh && scripts/check-progress-to-stdout.sh && bun run typecheck && bun test",
|
||||
"test:e2e": "bash scripts/run-e2e.sh",
|
||||
"typecheck": "tsc --noEmit",
|
||||
"check:jsonb": "scripts/check-jsonb-pattern.sh",
|
||||
@@ -50,16 +49,13 @@
|
||||
"dependencies": {
|
||||
"@anthropic-ai/sdk": "^0.30.0",
|
||||
"@aws-sdk/client-s3": "^3.1028.0",
|
||||
"@dqbd/tiktoken": "^1.0.22",
|
||||
"@electric-sql/pglite": "0.4.3",
|
||||
"@modelcontextprotocol/sdk": "^1.0.0",
|
||||
"gray-matter": "^4.0.3",
|
||||
"marked": "^18.0.0",
|
||||
"openai": "^4.0.0",
|
||||
"pgvector": "^0.2.0",
|
||||
"postgres": "^3.4.0",
|
||||
"tree-sitter-wasms": "0.1.13",
|
||||
"web-tree-sitter": "0.22.6"
|
||||
"postgres": "^3.4.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/bun": "latest",
|
||||
|
||||
@@ -1,62 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# CI guard: verify that bun --compile binaries ship with embedded tree-sitter
|
||||
# WASMs and produce real semantic chunks (not recursive-fallback chunks).
|
||||
#
|
||||
# This is the #1 silent-failure mode for v0.19.0 code indexing. If the WASM
|
||||
# import attributes regress or the asset path drifts, the compiled binary
|
||||
# silently falls through to the recursive text chunker. Users see no error,
|
||||
# just degraded chunking quality. This script catches that regression.
|
||||
#
|
||||
# Fails the build when:
|
||||
# - bun build --compile fails
|
||||
# - The resulting binary can't parse TypeScript
|
||||
# - Chunks come back without real symbol names (fallback signature)
|
||||
#
|
||||
# Runs as part of `bun test` via the package.json pre-test pipeline.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
cd "$REPO_ROOT"
|
||||
|
||||
OUT_BIN="$(mktemp /tmp/gbrain-wasm-check.XXXXXX)"
|
||||
trap 'rm -f "$OUT_BIN"' EXIT
|
||||
|
||||
# Build a minimal smoketest binary that imports the chunker. We compile this
|
||||
# instead of the full gbrain CLI so the failure mode is laser-focused on
|
||||
# chunker + WASM path resolution, not unrelated CLI wiring.
|
||||
bun build --compile --outfile "$OUT_BIN" scripts/chunker-smoketest.ts >/dev/null 2>&1
|
||||
|
||||
# Run it and capture JSON output.
|
||||
OUTPUT="$("$OUT_BIN" 2>&1)"
|
||||
|
||||
# Sanity: JSON parses and has expected shape.
|
||||
# - has_symbol_names: at least one chunk carries a concrete symbol name
|
||||
# (proves tree-sitter AST extraction, not recursive-fallback chunks).
|
||||
# - has_typescript_header: the structured header is emitted with the
|
||||
# correct language tag (proves the language map reached displayLang).
|
||||
# - calculateScore by name: specific function that MUST appear as a
|
||||
# top-level semantic node. If it's missing, the chunker either fell
|
||||
# through to recursive or the TypeScript grammar didn't load.
|
||||
if ! echo "$OUTPUT" | grep -q '"has_symbol_names": true'; then
|
||||
echo "[check-wasm-embedded] FAIL: compiled binary returned no symbol names (fallback chunks)." >&2
|
||||
echo "[check-wasm-embedded] Output was:" >&2
|
||||
echo "$OUTPUT" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! echo "$OUTPUT" | grep -q '"has_typescript_header": true'; then
|
||||
echo "[check-wasm-embedded] FAIL: chunk header missing TypeScript language tag." >&2
|
||||
echo "[check-wasm-embedded] Output was:" >&2
|
||||
echo "$OUTPUT" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! echo "$OUTPUT" | grep -q '"calculateScore"'; then
|
||||
echo "[check-wasm-embedded] FAIL: tree-sitter did not extract the calculateScore function symbol." >&2
|
||||
echo "[check-wasm-embedded] Output was:" >&2
|
||||
echo "$OUTPUT" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "[check-wasm-embedded] OK — compiled binary produced real semantic chunks."
|
||||
@@ -1,51 +0,0 @@
|
||||
import { chunkCodeText } from '../src/core/chunkers/code.ts';
|
||||
|
||||
// Large function body so it doesn't merge with siblings — the CI guard
|
||||
// needs at least one chunk with a concrete symbol name to prove the
|
||||
// tree-sitter WASM is actually resolving (not just recursive fallback).
|
||||
const src = `export function calculateScore(
|
||||
items: Array<{ value: number; weight: number }>,
|
||||
opts: { normalize?: boolean; cap?: number } = {}
|
||||
): number {
|
||||
if (items.length === 0) return 0;
|
||||
const sum = items.reduce((acc, it) => acc + it.value * it.weight, 0);
|
||||
const totalWeight = items.reduce((acc, it) => acc + it.weight, 0);
|
||||
if (totalWeight === 0) return 0;
|
||||
const raw = sum / totalWeight;
|
||||
if (opts.normalize) {
|
||||
const clamped = Math.max(0, Math.min(1, raw));
|
||||
return opts.cap !== undefined ? Math.min(opts.cap, clamped) : clamped;
|
||||
}
|
||||
return opts.cap !== undefined ? Math.min(opts.cap, raw) : raw;
|
||||
}
|
||||
|
||||
export class UserRegistry {
|
||||
private users: Map<string, { name: string; score: number }> = new Map();
|
||||
|
||||
register(id: string, name: string, score: number): void {
|
||||
this.users.set(id, { name, score });
|
||||
}
|
||||
|
||||
lookup(id: string): { name: string; score: number } | null {
|
||||
return this.users.get(id) ?? null;
|
||||
}
|
||||
|
||||
topK(k: number): Array<{ id: string; name: string; score: number }> {
|
||||
const entries = Array.from(this.users.entries());
|
||||
entries.sort((a, b) => b[1].score - a[1].score);
|
||||
return entries.slice(0, k).map(([id, v]) => ({ id, ...v }));
|
||||
}
|
||||
}
|
||||
|
||||
export type UserId = string;
|
||||
`;
|
||||
const result = await chunkCodeText(src, 'smoketest.ts');
|
||||
const hasSymbolNames = result.some(c => c.metadata.symbolName !== null);
|
||||
const hasTypeScriptHeader = result.some(c => c.text.startsWith('[TypeScript]'));
|
||||
console.log(JSON.stringify({
|
||||
count: result.length,
|
||||
has_symbol_names: hasSymbolNames,
|
||||
has_typescript_header: hasTypeScriptHeader,
|
||||
first_header: result[0]?.text.split('\n')[0],
|
||||
symbol_names: result.map(c => c.metadata.symbolName),
|
||||
}, null, 2));
|
||||
+1
-1
@@ -58,7 +58,7 @@ This is the dispatcher. Skills are the implementation. **Read the skill file bef
|
||||
| Cross-modal review, second opinion | `skills/cross-modal-review/SKILL.md` |
|
||||
| "Validate skills", skill health check | `skills/testing/SKILL.md` |
|
||||
| Webhook setup, external event processing | `skills/webhook-transforms/SKILL.md` |
|
||||
| "Spawn agent", "background task", "parallel tasks", "steer agent", "pause/resume agent", "gbrain jobs submit", "submit a gbrain job", "submit a shell job", "shell job" | `skills/minion-orchestrator/SKILL.md` |
|
||||
| "Spawn agent", "background task", "parallel tasks", "steer agent", "pause/resume agent" | `skills/minion-orchestrator/SKILL.md` |
|
||||
|
||||
## Setup & migration
|
||||
|
||||
|
||||
@@ -79,7 +79,7 @@ Even when Minions is the default (mode A), some work should run inline:
|
||||
|
||||
Before submitting batch jobs:
|
||||
|
||||
- Check active queue depth via `list_jobs --status active` (MCP-callable) or `gbrain jobs stats` (CLI)
|
||||
- Check `get_job_stats` queue_health.active
|
||||
- If active > 5, stagger new jobs with `delay` so you don't swarm
|
||||
- The resource governor auto-throttles but don't dump 20 jobs at once
|
||||
|
||||
|
||||
@@ -12,12 +12,6 @@ triggers:
|
||||
- "maintenance"
|
||||
- "orphan pages"
|
||||
- "stale pages"
|
||||
- "extract links"
|
||||
- "build link graph"
|
||||
- "populate timeline"
|
||||
- "populate links"
|
||||
- "backfill graph"
|
||||
- "extract timeline entries"
|
||||
tools:
|
||||
- get_health
|
||||
- get_page
|
||||
|
||||
@@ -132,7 +132,7 @@
|
||||
{
|
||||
"name": "minion-orchestrator",
|
||||
"path": "minion-orchestrator/SKILL.md",
|
||||
"description": "Unified Minions skill for deterministic shell jobs and LLM subagent orchestration. Submit, monitor, steer, pause/resume, replay. Replaces the older gbrain-jobs routing intent and sessions_spawn for durable observable background work."
|
||||
"description": "Manage background agents via Minions job queue. Submit, monitor, steer, pause/resume, replay. Replaces sessions_spawn for durable observable agents."
|
||||
},
|
||||
{
|
||||
"name": "skillify",
|
||||
|
||||
@@ -1,71 +0,0 @@
|
||||
---
|
||||
version: 0.19.0
|
||||
feature_pitch:
|
||||
headline: "Your code is now first-class in the brain."
|
||||
one_liner: "gbrain code-refs BrainEngine --json returns every usage site in <100ms."
|
||||
user_action_required: true
|
||||
---
|
||||
|
||||
# v0.19.0 — Code Indexing
|
||||
|
||||
This release makes code a first-class citizen in the brain. Tree-sitter parses 29 languages into semantic chunks. `gbrain code-def` and `gbrain code-refs` let agents find symbol definitions and references without grep. Incremental chunking drops daily autopilot embedding cost by ~95%. The chunker is a strict superset of Chonkie's CodeChunker plus a structured header Chonkie lacks.
|
||||
|
||||
## Schema migrations applied automatically
|
||||
|
||||
- **v25 — `pages.page_kind`** — distinguishes markdown vs code pages at the DB level. Existing rows backfill to `'markdown'`. Postgres uses `ADD CONSTRAINT ... NOT VALID` + `VALIDATE CONSTRAINT` so large tables don't block.
|
||||
- **v26 — `content_chunks` code metadata** — adds `language`, `symbol_name`, `symbol_type`, `start_line`, `end_line`. All nullable. Partial indexes on `symbol_name` and `language` for fast symbol lookup.
|
||||
|
||||
These run as part of `gbrain upgrade` → `gbrain apply-migrations`. No manual DDL needed.
|
||||
|
||||
## What the agent should do after upgrading
|
||||
|
||||
1. **Confirm migrations landed:**
|
||||
```bash
|
||||
gbrain doctor
|
||||
```
|
||||
Look for `schema_version: 26`. If lower, run `gbrain apply-migrations --yes`.
|
||||
|
||||
2. **Register a code source** if the user wants their code indexed:
|
||||
```bash
|
||||
gbrain sources add <id> --path <path-to-repo>
|
||||
```
|
||||
Pick a short `<id>` (e.g. `wiki`, `gbrain`, `yc-media`). Shorter is better — it's used in citation keys.
|
||||
|
||||
3. **Sync the code source:**
|
||||
```bash
|
||||
gbrain sync --source <id>
|
||||
```
|
||||
First sync may run tens of minutes depending on repo size. Each TypeScript function becomes a chunk with a structured header like `[TypeScript] src/core/sync.ts:380-415 function performFullSync`.
|
||||
|
||||
4. **Verify code-def and code-refs work:**
|
||||
```bash
|
||||
gbrain code-def BrainEngine # prints the file + line of the definition
|
||||
gbrain code-refs BrainEngine --json # JSON array of every usage site
|
||||
```
|
||||
If both return non-empty arrays, code indexing is working end-to-end.
|
||||
|
||||
5. **Observe incremental chunking.** Edit one function in a 20-function file, re-run `sync --source <id>`. Embedding cost should be ~5% of the first sync because unchanged chunks reuse their existing embeddings.
|
||||
|
||||
## Migration from Wintermute's `repos` (if you used it)
|
||||
|
||||
v0.19.0 deletes `~/.gbrain/config.json`'s `repos` array in favor of the `sources` table. The CLI surface is preserved as a deprecated alias: `gbrain repos add` still works, but routes into `runSources` with a one-line deprecation notice on stderr. Existing scripts keep working; prefer `gbrain sources` going forward.
|
||||
|
||||
If you had repos configured in `~/.gbrain/config.json`, re-register them:
|
||||
```bash
|
||||
gbrain sources add <name> --path <path>
|
||||
```
|
||||
Per-repo sync bookmarks live in the `sources` table now (not config.json).
|
||||
|
||||
## Flag in `pending-host-work.jsonl`
|
||||
|
||||
Per the v0.11.0 convention, the migration orchestrator writes an entry to `~/.gbrain/migrations/pending-host-work.jsonl` flagging the new CLI surfaces so headless agents can walk the TODOs:
|
||||
|
||||
```json
|
||||
{"version": "0.19.0", "action": "register_code_source", "status": "pending"}
|
||||
```
|
||||
|
||||
Agents that handle pending-host-work should offer the user a `gbrain sources add ...` prompt.
|
||||
|
||||
## When NOT to run the migration
|
||||
|
||||
Never. v0.19.0 is fully backward-compatible. Existing markdown-only brains see zero behavior change until the user adds a code source.
|
||||
@@ -1,64 +0,0 @@
|
||||
---
|
||||
version: 0.21.0
|
||||
feature_pitch:
|
||||
headline: "Code Cathedral II — chunk-grain FTS, qualified symbols, structural edges."
|
||||
one_liner: "Natural-language queries now rank docstring matches first. CHUNKER_VERSION 3→4 rolls the new chunker over existing code pages automatically on next sync."
|
||||
user_action_required: true
|
||||
---
|
||||
|
||||
# v0.21.0 — Code Cathedral II
|
||||
|
||||
This release is the biggest code-search upgrade in gbrain history. Chunk-grain FTS with doc_comment Weight A ranks natural-language queries against docstrings above prose. CHUNKER_VERSION 3 → 4 folds into content_hash so every existing code page re-chunks. File classifier widened from 9 to 35 extensions. Markdown fence extraction, `sync --all` cost preview, and `reconcile-links` batch command ship alongside the chunker upgrade.
|
||||
|
||||
## Schema migrations applied automatically
|
||||
|
||||
- **v27 — cathedral_ii_foundation** — adds `code_edges_chunk`, `code_edges_symbol`, `sources.chunker_version`, and new `content_chunks` columns (`parent_symbol_path`, `doc_comment`, `symbol_name_qualified`, `search_vector`). Includes the chunk-grain FTS trigger that builds from `setweight(to_tsvector('english', doc_comment), 'A') || setweight(to_tsvector('english', chunk_text), 'B') || setweight(to_tsvector('english', symbol_name_qualified), 'A')`.
|
||||
- **v28 — cathedral_ii_chunk_fts_backfill** — populates `search_vector` on every existing chunk so day-1 queries already rank correctly.
|
||||
|
||||
These run as part of `gbrain upgrade` → `gbrain apply-migrations`. No manual DDL needed.
|
||||
|
||||
## What the agent should do after upgrading
|
||||
|
||||
1. **Confirm migrations landed:**
|
||||
```bash
|
||||
gbrain doctor
|
||||
```
|
||||
Look for `schema_version: 28`. If lower, run `gbrain apply-migrations --yes`.
|
||||
|
||||
2. **Pick a backfill path.** The `CHUNKER_VERSION` bump + `sources.chunker_version` gate means existing code pages must re-chunk for the new shape to take effect. Two paths:
|
||||
|
||||
**Automatic (recommended):** next `gbrain sync --source <id>` detects the version mismatch and forces a full re-walk regardless of git HEAD equality. No cost preview, no user interaction. The Layer 12 SP-1 fix from codex's second-pass review.
|
||||
|
||||
**Immediate:** preview cost, then reindex every code page now.
|
||||
```bash
|
||||
gbrain reindex-code --dry-run # preview token count + $USD cost
|
||||
gbrain reindex-code --yes # reindex all code pages
|
||||
gbrain reindex-code --source <id> --yes # scope to one source
|
||||
```
|
||||
On non-TTY (automation / cron) `reindex-code` without `--yes` emits a `ConfirmationRequired` envelope and exits 2 — same shape as `sync --all`. The envelope matches v0.19.0's `StructuredAgentError`.
|
||||
|
||||
3. **Verify chunk-grain FTS works.** A query that mentions a concept in a docstring (not just the function body) should rank higher than a page that mentions it only in prose:
|
||||
```bash
|
||||
gbrain query "whatever your docstring says"
|
||||
```
|
||||
Expected: top hit is the chunk whose `doc_comment` matches.
|
||||
|
||||
4. **(Optional) Reconcile doc↔impl links.** v0.19.0 Layer 6 forward-extracted code refs from markdown pages when they imported, but edges dropped if the code page hadn't imported yet. v0.21.0's `reconcile-links` batch-scans every markdown page and idempotently reinserts missing edges:
|
||||
```bash
|
||||
gbrain reconcile-links # full run
|
||||
gbrain reconcile-links --dry-run # preview only
|
||||
gbrain reconcile-links --json # machine output
|
||||
```
|
||||
Respects `auto_link=false` config (prints warn + exits 0 when disabled).
|
||||
|
||||
## Widened file classifier — more languages sync now
|
||||
|
||||
Previous classifier recognized 9 extensions as code. v0.21.0 widens to 35 (Rust, Ruby, Java, C#, C/C++, Swift, Kotlin, Scala, PHP, Elixir, Elm, OCaml, Dart, Zig, Solidity, Lua, shell, etc.). If your repo contains source files in languages beyond TS/JS/Py/Go, they'll flow through the code chunker on next sync. No action needed — `detectCodeLanguage` handles dispatch.
|
||||
|
||||
## Flag in `pending-host-work.jsonl`
|
||||
|
||||
The migration orchestrator emits a backfill-prompt phase that prints the two backfill choices directly. No `pending-host-work.jsonl` entry is written — the choice is user-driven and ephemeral (either reindex now or wait for next sync).
|
||||
|
||||
## When NOT to run the migration
|
||||
|
||||
Never skip it. v0.21.0 is fully backward-compatible at the API level (page-grain FTS shape preserved externally; chunk-grain is internal only). Skipping the migration means agents miss doc_comment Weight A ranking and the chunker_version gate never fires — existing code pages silently stay on CHUNKER_VERSION 3 forever until you sync with `--full`.
|
||||
@@ -2,18 +2,11 @@
|
||||
name: minion-orchestrator
|
||||
version: 1.0.0
|
||||
description: |
|
||||
Unified Minions skill for both deterministic shell jobs and LLM subagent
|
||||
orchestration. Replaces the older `gbrain-jobs` routing intent. Use when:
|
||||
submitting gbrain jobs, shell/background tasks, spawning subagents,
|
||||
checking progress, steering running work, pausing/resuming, parallel
|
||||
fan-out. One durable, observable, steerable queue interface.
|
||||
Manage background agents via Minions job queue. Use when: spawning subagents,
|
||||
checking agent progress, steering running agents, pausing/resuming work,
|
||||
parallel task execution, fan-out research. Replaces sessions_spawn for
|
||||
durable, observable, steerable agents.
|
||||
triggers:
|
||||
- "gbrain jobs submit"
|
||||
- "submit a gbrain job"
|
||||
- "submit a shell job"
|
||||
- "shell job"
|
||||
- "run shell command in background"
|
||||
- "deterministic background task"
|
||||
- "spawn agent"
|
||||
- "background task"
|
||||
- "run in background"
|
||||
@@ -39,6 +32,7 @@ tools:
|
||||
- replay_job
|
||||
- send_job_message
|
||||
- get_job_progress
|
||||
- get_job_stats
|
||||
mutating: true
|
||||
---
|
||||
|
||||
@@ -46,16 +40,8 @@ mutating: true
|
||||
|
||||
## Contract
|
||||
|
||||
Minions is a Postgres-native job queue for durable, observable background work.
|
||||
This single skill handles two lanes:
|
||||
- Deterministic shell jobs (`gbrain jobs submit shell ...`)
|
||||
- LLM subagent jobs (`gbrain agent run ...`)
|
||||
|
||||
When to route to Minions: durable, observable work that must survive restarts,
|
||||
fan out across many parallel tasks, or persist across sessions. Routing policy
|
||||
is defined in `skills/conventions/subagent-routing.md` — the project default is
|
||||
`pain_triggered` (native subagents first, Minions after specific pain signals
|
||||
fire); Mode A (all-through-Minions) is opt-in.
|
||||
Minions is a Postgres-native job queue for durable, observable agent orchestration.
|
||||
Every background agent task goes through Minions. No in-memory subagent spawning.
|
||||
|
||||
Guarantees:
|
||||
- Jobs survive gateway restart (Postgres-backed)
|
||||
@@ -64,155 +50,51 @@ Guarantees:
|
||||
- Jobs can be paused, resumed, or cancelled at any time
|
||||
- Parent-child DAGs with configurable failure policies
|
||||
|
||||
## Route the Request: Shell Job vs Subagent
|
||||
## When to Use Minions vs Inline Work
|
||||
|
||||
| Condition | Action |
|
||||
|---|---|
|
||||
| User asks for deterministic command/script run | Shell job (CLI: `gbrain jobs submit shell ...`) |
|
||||
| User asks to "run in minions" + explicit command/argv | Shell job (CLI, `--params` with `cmd` or `argv`) |
|
||||
| User asks for research/reasoning/iterative agent | Subagent job (CLI: `gbrain agent run`) |
|
||||
| User asks to steer/pause/resume an agent | Subagent job lifecycle tools (MCP-callable) |
|
||||
| Single simple operation under ~30s | Consider inline execution first |
|
||||
| Needs restart durability/observability | Submit as Minion job |
|
||||
| Parallel work (2+ streams) | `gbrain agent run --fanout-manifest` or parent + child subagents |
|
||||
| Single tool call, < 30s | Do it inline |
|
||||
| Multi-step, any duration | Submit as Minion job |
|
||||
| Parallel work (2+ streams) | Submit N Minion jobs with shared parent |
|
||||
| Needs to survive restart | Submit as Minion job |
|
||||
| User wants progress updates | Submit as Minion job with progress tracking |
|
||||
| Research / bulk operation | Submit as Minion job, always |
|
||||
| File imports, bulk embeds | Submit as Minion job |
|
||||
|
||||
If intent is ambiguous, ask one clarification:
|
||||
"Do you want a deterministic shell command job, or an LLM agent job?"
|
||||
|
||||
## Shell Jobs (Deterministic Scripts)
|
||||
|
||||
Use for reproducible command execution, ETL steps, cron work, and scriptable
|
||||
tasks where no LLM reasoning loop is needed.
|
||||
|
||||
### Preconditions (read before submitting your first shell job)
|
||||
|
||||
- **`GBRAIN_ALLOW_SHELL_JOBS=1` must be set on the worker environment.**
|
||||
Without it, the shell handler refuses to register and submissions sit in
|
||||
`waiting` silently. Gate lives in `src/core/minions/handlers/shell.ts`.
|
||||
- **Security:** flipping `GBRAIN_ALLOW_SHELL_JOBS=1` authorizes arbitrary
|
||||
command execution on the worker. On a shared queue, this is a remote code
|
||||
execution surface. Treat as privileged infrastructure authorization.
|
||||
- **Execution mode — pick one:**
|
||||
- **Postgres + daemon:** `gbrain jobs work` runs a persistent worker that
|
||||
claims and executes jobs from the queue.
|
||||
- **PGLite + --follow:** `gbrain jobs submit ... --follow` runs inline.
|
||||
The daemon mode is not available on PGLite (exclusive file lock). See
|
||||
`docs/guides/minions-shell-jobs.md`.
|
||||
- **MCP boundary:** shell-job submission is CLI-only. `submit_job name="shell"`
|
||||
over MCP throws an `OperationError` with code `permission_denied` ("'shell'
|
||||
jobs cannot be submitted over MCP") because `shell` is in `PROTECTED_JOB_NAMES`.
|
||||
Agents CAN observe shell jobs via `get_job` / `list_jobs` / `get_job_progress`
|
||||
(not protected), but cannot submit them. Operator or autopilot submits;
|
||||
agent observes.
|
||||
- **Verify setup:** after configuration, run `gbrain jobs stats` (CLI) to
|
||||
confirm the worker is registered and consuming the queue.
|
||||
|
||||
### Submit (CLI, operator or autopilot)
|
||||
|
||||
Shell jobs take their command via `--params` as a JSON object with `cmd` (string)
|
||||
or `argv` (array), plus `cwd` and optional `env`.
|
||||
|
||||
Command string form:
|
||||
```
|
||||
gbrain jobs submit shell --params '{"cmd":"echo hello","cwd":"/abs/path"}'
|
||||
```
|
||||
|
||||
Argv form (no shell expansion):
|
||||
```
|
||||
gbrain jobs submit shell --params '{"argv":["bash","-lc","echo hello"],"cwd":"/abs/path"}'
|
||||
```
|
||||
|
||||
Inline execution on PGLite or any one-shot deployment:
|
||||
```
|
||||
gbrain jobs submit shell --params '{"cmd":"echo hello","cwd":"/tmp"}' --follow
|
||||
```
|
||||
|
||||
Queue/lifecycle flags exposed by `gbrain jobs submit --help`: `--queue`,
|
||||
`--priority`, `--delay`, `--max-attempts`, `--max-stalled`, `--backoff-type`,
|
||||
`--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key`,
|
||||
`--dry-run`.
|
||||
|
||||
### Monitor (agents or operator)
|
||||
|
||||
These operations are MCP-callable and safe for agent use:
|
||||
|
||||
```
|
||||
list_jobs --name shell --status active
|
||||
get_job ID
|
||||
get_job_progress ID
|
||||
```
|
||||
|
||||
Check structured result fields (exit code, stdout/stderr tails, attempts,
|
||||
timings) from `get_job`. Use `gbrain jobs stats` (CLI) for worker/queue
|
||||
health dashboard.
|
||||
|
||||
### Control (MCP-callable)
|
||||
|
||||
```
|
||||
cancel_job id=ID
|
||||
replay_job id=ID
|
||||
```
|
||||
|
||||
`replay_job` is not protected — only shell *submission* is. Agents can
|
||||
cancel or replay a shell job without CLI access.
|
||||
|
||||
Use idempotency keys for recurring shell workloads to avoid duplicate runs.
|
||||
|
||||
## Subagent Jobs (LLM Orchestration)
|
||||
|
||||
Use for open-ended reasoning, tool-using research, and fan-out synthesis.
|
||||
|
||||
**User-facing entrypoint:** `gbrain agent run <prompt>` is the canonical way
|
||||
to submit subagent work. It handles the elevated-trust plumbing — `subagent`
|
||||
and `subagent_aggregator` are both in `PROTECTED_JOB_NAMES`, so direct MCP
|
||||
submission requires `{allowProtectedSubmit: true}`, which `gbrain agent run`
|
||||
supplies.
|
||||
**Rule of thumb:** If it takes more than 3 tool calls, use a Minion.
|
||||
|
||||
## Phase 1: Submit
|
||||
|
||||
```
|
||||
gbrain agent run "Research Acme Corp revenue" --tools "search,query"
|
||||
submit_job name="research" data={"prompt":"Research Acme Corp revenue","tools":["search","web_search"]}
|
||||
```
|
||||
|
||||
`--tools` accepts a comma-separated subset of `BRAIN_TOOL_ALLOWLIST` (see
|
||||
`src/core/minions/tools/brain-allowlist.ts`): `query`, `search`, `get_page`,
|
||||
`list_pages`, `file_list`, `file_url`, `get_backlinks`, `traverse_graph`,
|
||||
`resolve_slugs`, `get_ingest_log`, `put_page`. Anything outside the allow-list
|
||||
is rejected at submit time with `allowed_tools references unknown tool`.
|
||||
Options:
|
||||
- `queue` — queue name (default: 'default')
|
||||
- `priority` — lower = higher priority (default: 0)
|
||||
- `max_attempts` — retry limit (default: 3)
|
||||
- `delay` — ms delay before eligible
|
||||
|
||||
For parallel work with a fan-out manifest:
|
||||
For parallel work, submit a parent then children:
|
||||
```
|
||||
gbrain agent run --fanout-manifest companies.json
|
||||
submit_job name="orchestrate" data={"task":"research 5 companies"}
|
||||
# Returns parent_id
|
||||
|
||||
submit_job name="research" data={"company":"Acme"} parent_job_id=PARENT_ID
|
||||
submit_job name="research" data={"company":"Beta"} parent_job_id=PARENT_ID
|
||||
submit_job name="research" data={"company":"Gamma"} parent_job_id=PARENT_ID
|
||||
```
|
||||
|
||||
The manifest describes N children + 1 aggregator. Each child runs
|
||||
`name="subagent"` under the hood; the aggregator runs `name="subagent_aggregator"`
|
||||
and claims AFTER every child terminates. See
|
||||
`src/core/minions/handlers/subagent.ts` and
|
||||
`src/core/minions/handlers/subagent-aggregator.ts`.
|
||||
|
||||
Flags (from `src/commands/agent.ts`):
|
||||
- `--subagent-def <name>` — named subagent definition
|
||||
- `--model <id>` — override model
|
||||
- `--max-turns <N>` — cap the LLM loop
|
||||
- `--tools <csv>` — allow-listed brain tools (see above)
|
||||
- `--timeout-ms <N>` — hard timeout per job
|
||||
- `--fanout-manifest <file>` — N children + 1 aggregator
|
||||
- `--follow` / `--no-follow` — stream logs + wait (default on TTY)
|
||||
- `--detach` — submit and return immediately
|
||||
|
||||
Queue/priority/retry tuning is not exposed by `gbrain agent run`; submit the
|
||||
raw `subagent` handler via `gbrain jobs submit` (requires CLI trust) if you
|
||||
need those knobs.
|
||||
Parent auto-enters `waiting-children` and unblocks when all children finish.
|
||||
|
||||
## Phase 2: Monitor
|
||||
|
||||
```
|
||||
list_jobs --status active # MCP — what's running?
|
||||
get_job ID # MCP — full details + logs + tokens
|
||||
get_job_progress ID # MCP — structured progress snapshot
|
||||
gbrain jobs stats # CLI — queue health dashboard
|
||||
gbrain agent logs ID --follow # CLI — streaming transcript + heartbeat
|
||||
list_jobs --status active # what's running?
|
||||
get_job ID # full details + logs + tokens
|
||||
get_job_progress ID # structured progress snapshot
|
||||
get_job_stats # health dashboard
|
||||
```
|
||||
|
||||
Progress includes: step count, total steps, message, token usage, last tool called.
|
||||
@@ -239,8 +121,6 @@ replay_job id=ID # re-run with same or modified params
|
||||
replay_job id=ID data_overrides={"depth":"deep"} # replay with changes
|
||||
```
|
||||
|
||||
All lifecycle ops are MCP-callable.
|
||||
|
||||
## Phase 5: Review Results
|
||||
|
||||
```
|
||||
@@ -274,9 +154,9 @@ When reporting batch status (parent with children):
|
||||
|
||||
```
|
||||
Parent #ID — waiting-children
|
||||
#A subagent(Acme) — active, 3/5 steps, 2.5k tokens
|
||||
#B subagent(Beta) — completed, 1.8k tokens
|
||||
#C subagent(Gamma) — paused
|
||||
#A research(Acme) — active, 3/5 steps, 2.5k tokens
|
||||
#B research(Beta) — completed, 1.8k tokens
|
||||
#C research(Gamma) — paused
|
||||
Total tokens so far: 4.3k
|
||||
```
|
||||
|
||||
@@ -284,19 +164,19 @@ Total tokens so far: 4.3k
|
||||
|
||||
- Don't spawn a Minion for a single search query (use search tool directly)
|
||||
- Don't fire-and-forget without checking results
|
||||
- Don't spawn > 5 concurrent agents without checking `gbrain jobs stats` first
|
||||
- For subagent work, don't use `sessions_spawn` with `runtime: "subagent"` when Minions is available (use `gbrain agent run` instead)
|
||||
- Don't spawn > 5 concurrent agents without checking `get_job_stats` first
|
||||
- Don't use `sessions_spawn` with `runtime: "subagent"` when Minions is available
|
||||
- Don't poll `get_job` in a tight loop (use `get_job_progress` for lightweight checks)
|
||||
|
||||
## Tools Used
|
||||
|
||||
- Submit a background job — `submit_job` (MCP, non-protected names only; shell jobs are CLI-only, subagent jobs via `gbrain agent run`)
|
||||
- Get job details — `get_job` (MCP)
|
||||
- List jobs with filters — `list_jobs` (MCP)
|
||||
- Cancel a job — `cancel_job` (MCP)
|
||||
- Pause a job — `pause_job` (MCP)
|
||||
- Resume a paused job — `resume_job` (MCP)
|
||||
- Replay a completed/failed job — `replay_job` (MCP)
|
||||
- Send sidechannel message — `send_job_message` (MCP)
|
||||
- Get structured progress — `get_job_progress` (MCP)
|
||||
- Queue stats — `gbrain jobs stats` (CLI; no MCP equivalent)
|
||||
- Submit a background job (submit_job)
|
||||
- Get job details (get_job)
|
||||
- List jobs with filters (list_jobs)
|
||||
- Cancel a job (cancel_job)
|
||||
- Pause a job (pause_job)
|
||||
- Resume a paused job (resume_job)
|
||||
- Replay a completed/failed job (replay_job)
|
||||
- Send sidechannel message (send_job_message)
|
||||
- Get structured progress (get_job_progress)
|
||||
- Get job queue stats (get_job_stats)
|
||||
|
||||
@@ -12,10 +12,6 @@ triggers:
|
||||
- "what happened"
|
||||
- "search for"
|
||||
- "look up"
|
||||
- "who knows who"
|
||||
- "relationship between"
|
||||
- "connections"
|
||||
- "graph query"
|
||||
tools:
|
||||
- search
|
||||
- query
|
||||
|
||||
@@ -10,7 +10,6 @@ triggers:
|
||||
- "container restart check"
|
||||
- "health check"
|
||||
- "did the restart break anything"
|
||||
- "did the container restart break anything"
|
||||
tools:
|
||||
- exec
|
||||
- read
|
||||
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
+1
-71
@@ -19,7 +19,7 @@ for (const op of operations) {
|
||||
}
|
||||
|
||||
// CLI-only commands that bypass the operation layer
|
||||
const CLI_ONLY = new Set(['init', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'repos', 'code-def', 'code-refs', 'reindex-code', 'code-callers', 'code-callees']);
|
||||
const CLI_ONLY = new Set(['init', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test']);
|
||||
|
||||
async function main() {
|
||||
// Parse global flags (--quiet / --progress-json / --progress-interval)
|
||||
@@ -501,15 +501,6 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
await runGraphQuery(engine, args);
|
||||
break;
|
||||
}
|
||||
case 'reconcile-links': {
|
||||
// v0.20.0 Cathedral II Layer 8 D3: batch-recompute doc↔impl edges
|
||||
// for any markdown page that cites code files. Idempotent; safe to
|
||||
// re-run. Closes the v0.19.0 Layer 6 order-dependency bug where
|
||||
// guides imported before their code never got their edges written.
|
||||
const { runReconcileLinksCli } = await import('./commands/reconcile-links.ts');
|
||||
await runReconcileLinksCli(engine, args);
|
||||
break;
|
||||
}
|
||||
case 'orphans': {
|
||||
const { runOrphans } = await import('./commands/orphans.ts');
|
||||
await runOrphans(engine, args);
|
||||
@@ -520,48 +511,6 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
await runSources(engine, args);
|
||||
break;
|
||||
}
|
||||
case 'code-def': {
|
||||
const { runCodeDef } = await import('./commands/code-def.ts');
|
||||
await runCodeDef(engine, args);
|
||||
break;
|
||||
}
|
||||
case 'code-refs': {
|
||||
const { runCodeRefs } = await import('./commands/code-refs.ts');
|
||||
await runCodeRefs(engine, args);
|
||||
break;
|
||||
}
|
||||
case 'reindex-code': {
|
||||
// v0.20.0 Cathedral II Layer 13 (E2): explicit code-page reindex
|
||||
// for users upgrading from v0.19.0. Cost-preview gated; TTY prompt
|
||||
// or ConfirmationRequired envelope for non-TTY/JSON callers.
|
||||
const { runReindexCodeCli } = await import('./commands/reindex-code.ts');
|
||||
await runReindexCodeCli(engine, args);
|
||||
break;
|
||||
}
|
||||
case 'code-callers': {
|
||||
// v0.20.0 Cathedral II Layer 10 (C4): "who calls <symbol>?"
|
||||
const { runCodeCallers } = await import('./commands/code-callers.ts');
|
||||
await runCodeCallers(engine, args);
|
||||
break;
|
||||
}
|
||||
case 'code-callees': {
|
||||
// v0.20.0 Cathedral II Layer 10 (C5): "what does <symbol> call?"
|
||||
const { runCodeCallees } = await import('./commands/code-callees.ts');
|
||||
await runCodeCallees(engine, args);
|
||||
break;
|
||||
}
|
||||
case 'repos': {
|
||||
// v0.19.0: `gbrain repos ...` is an alias into the v0.18.0 sources
|
||||
// subsystem. The repos abstraction (Wintermute's baseline) was
|
||||
// redundant with sources and carried per-user config state that
|
||||
// couldn't participate in federation / RLS / multi-tenancy. We
|
||||
// keep the alias so scripts like `gbrain repos add .` keep
|
||||
// working, with a nudge toward the canonical command.
|
||||
console.error('[gbrain] Note: "repos" is an alias for "sources" as of v0.19.0. Prefer `gbrain sources <subcommand>`.');
|
||||
const { runSources } = await import('./commands/sources.ts');
|
||||
await runSources(engine, args);
|
||||
break;
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
if (command !== 'serve') await engine.disconnect();
|
||||
@@ -676,25 +625,6 @@ TOOLS
|
||||
check-resolvable [--json] [--fix] Validate skill tree (reachability/MECE/DRY)
|
||||
report --type <name> --content ... Save timestamped report to brain/reports/
|
||||
|
||||
SOURCES (multi-repo / multi-brain)
|
||||
sources list Show registered sources
|
||||
sources add <id> --path <p> Register a source (id = short name, e.g. 'wiki')
|
||||
sources remove <id> Remove a source + its pages
|
||||
sync --all Sync all sources with a local_path
|
||||
sync --source <id> Sync one specific source
|
||||
repos ... DEPRECATED alias for 'sources' (v0.19.0)
|
||||
|
||||
CODE INDEXING (v0.19.0 / v0.20.0 Cathedral II)
|
||||
code-def <symbol> [--lang l] Find the definition of a symbol across code pages
|
||||
code-refs <symbol> [--lang l] Find all references to a symbol (JSON-first)
|
||||
code-callers <symbol> Who calls this symbol? (v0.20.0 A1)
|
||||
code-callees <symbol> What does this symbol call? (v0.20.0 A1)
|
||||
query <q> --lang <l> Filter hybrid search to one language (v0.20.0)
|
||||
query <q> --symbol-kind <k> Filter to symbol type (function|class|method|...) (v0.20.0)
|
||||
reconcile-links [--dry-run] Batch-recompute doc↔impl edges (v0.20.0)
|
||||
reindex-code [--source id] [--yes] Explicit code-page reindex (v0.20.0)
|
||||
sync --strategy code Sync code files into the brain
|
||||
|
||||
JOBS (Minions)
|
||||
jobs submit <name> [--params JSON] Submit background job [--follow] [--dry-run]
|
||||
jobs list [--status S] [--limit N] List jobs
|
||||
|
||||
@@ -1,74 +0,0 @@
|
||||
/**
|
||||
* gbrain code-callees <symbol>
|
||||
*
|
||||
* v0.20.0 Cathedral II Layer 10 (C5) — "what does this symbol call?"
|
||||
* Forward view of the A1 call graph. Matches `from_symbol_qualified`
|
||||
* in both code_edges_chunk + code_edges_symbol.
|
||||
*
|
||||
* Output: same JSON-on-non-TTY convention as code-callers / code-def /
|
||||
* code-refs.
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { errorFor, serializeError } from '../core/errors.ts';
|
||||
|
||||
function parseFlag(args: string[], name: string): string | undefined {
|
||||
const i = args.indexOf(name);
|
||||
return i >= 0 && i + 1 < args.length ? args[i + 1] : undefined;
|
||||
}
|
||||
|
||||
function shouldEmitJson(args: string[]): boolean {
|
||||
if (args.includes('--json')) return true;
|
||||
if (args.includes('--no-json')) return false;
|
||||
return !process.stdout.isTTY;
|
||||
}
|
||||
|
||||
export async function runCodeCallees(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const positional = args.filter((a) => !a.startsWith('--'));
|
||||
const sym = positional[0];
|
||||
if (!sym) {
|
||||
const err = errorFor({
|
||||
class: 'UsageError',
|
||||
code: 'code_callees_requires_symbol',
|
||||
message: 'code-callees requires a symbol name',
|
||||
hint: 'gbrain code-callees <symbol> [--all-sources] [--limit N] [--json]',
|
||||
});
|
||||
if (shouldEmitJson(args)) {
|
||||
console.log(JSON.stringify({ error: err.envelope }));
|
||||
} else {
|
||||
console.error(err.message);
|
||||
}
|
||||
process.exit(2);
|
||||
}
|
||||
const limit = parseInt(parseFlag(args, '--limit') || '100', 10);
|
||||
const allSources = args.includes('--all-sources');
|
||||
const sourceId = parseFlag(args, '--source');
|
||||
|
||||
try {
|
||||
const edges = await engine.getCalleesOf(sym, {
|
||||
limit,
|
||||
allSources: allSources || !sourceId,
|
||||
sourceId: sourceId ?? undefined,
|
||||
});
|
||||
|
||||
if (shouldEmitJson(args)) {
|
||||
console.log(JSON.stringify({ symbol: sym, count: edges.length, callees: edges }, null, 2));
|
||||
} else if (edges.length === 0) {
|
||||
console.log(`No callees found for "${sym}".`);
|
||||
} else {
|
||||
console.log(`${edges.length} callee(s) for "${sym}":`);
|
||||
for (const e of edges) {
|
||||
const res = e.resolved ? 'resolved' : 'unresolved';
|
||||
console.log(` ${e.from_symbol_qualified} → ${e.to_symbol_qualified} [${res}]`);
|
||||
}
|
||||
}
|
||||
} catch (e: unknown) {
|
||||
const env = serializeError(e);
|
||||
if (shouldEmitJson(args)) {
|
||||
console.log(JSON.stringify({ error: env }));
|
||||
} else {
|
||||
console.error(`code-callees failed: ${env.message}`);
|
||||
}
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
@@ -1,80 +0,0 @@
|
||||
/**
|
||||
* gbrain code-callers <symbol>
|
||||
*
|
||||
* v0.20.0 Cathedral II Layer 10 (C4) — "who calls this symbol?" Reversed
|
||||
* view of the A1 call graph. Matches `to_symbol_qualified` in both
|
||||
* code_edges_chunk (resolved) and code_edges_symbol (unresolved short-name
|
||||
* capture). Layer 5 captures edges at chunk time; Layer 10 exposes them.
|
||||
*
|
||||
* Scope decision: by default we only match the caller's source_id so
|
||||
* multi-repo brains don't cross-resolve (`Admin::UsersController#render`
|
||||
* in repo A ≠ same string in repo B). Pass `--all-sources` to search
|
||||
* globally.
|
||||
*
|
||||
* Output: non-TTY → JSON envelope. TTY → human table. Follows the
|
||||
* code-def / code-refs pattern.
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { errorFor, serializeError } from '../core/errors.ts';
|
||||
|
||||
function parseFlag(args: string[], name: string): string | undefined {
|
||||
const i = args.indexOf(name);
|
||||
return i >= 0 && i + 1 < args.length ? args[i + 1] : undefined;
|
||||
}
|
||||
|
||||
function shouldEmitJson(args: string[]): boolean {
|
||||
if (args.includes('--json')) return true;
|
||||
if (args.includes('--no-json')) return false;
|
||||
return !process.stdout.isTTY;
|
||||
}
|
||||
|
||||
export async function runCodeCallers(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const positional = args.filter((a) => !a.startsWith('--'));
|
||||
const sym = positional[0];
|
||||
if (!sym) {
|
||||
const err = errorFor({
|
||||
class: 'UsageError',
|
||||
code: 'code_callers_requires_symbol',
|
||||
message: 'code-callers requires a symbol name',
|
||||
hint: 'gbrain code-callers <symbol> [--all-sources] [--limit N] [--json]',
|
||||
});
|
||||
if (shouldEmitJson(args)) {
|
||||
console.log(JSON.stringify({ error: err.envelope }));
|
||||
} else {
|
||||
console.error(err.message);
|
||||
}
|
||||
process.exit(2);
|
||||
}
|
||||
const limit = parseInt(parseFlag(args, '--limit') || '100', 10);
|
||||
const allSources = args.includes('--all-sources');
|
||||
const sourceId = parseFlag(args, '--source');
|
||||
|
||||
try {
|
||||
const edges = await engine.getCallersOf(sym, {
|
||||
limit,
|
||||
allSources: allSources || !sourceId,
|
||||
sourceId: sourceId ?? undefined,
|
||||
});
|
||||
|
||||
if (shouldEmitJson(args)) {
|
||||
console.log(JSON.stringify({ symbol: sym, count: edges.length, callers: edges }, null, 2));
|
||||
} else if (edges.length === 0) {
|
||||
console.log(`No callers found for "${sym}".`);
|
||||
} else {
|
||||
console.log(`${edges.length} caller(s) for "${sym}":`);
|
||||
for (const e of edges) {
|
||||
const res = e.resolved ? 'resolved' : 'unresolved';
|
||||
console.log(` ${e.from_symbol_qualified} → ${e.to_symbol_qualified} [${res}]`);
|
||||
}
|
||||
}
|
||||
} catch (e: unknown) {
|
||||
const env = serializeError(e);
|
||||
if (shouldEmitJson(args)) {
|
||||
console.log(JSON.stringify({ error: env }));
|
||||
} else {
|
||||
console.error(`code-callers failed: ${env.message}`);
|
||||
}
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
@@ -1,136 +0,0 @@
|
||||
/**
|
||||
* gbrain code-def <symbol>
|
||||
*
|
||||
* v0.19.0 Layer 7 — look up the definition site(s) of a named symbol
|
||||
* (function, class, type, interface, enum) across every code page the
|
||||
* brain has indexed.
|
||||
*
|
||||
* Output:
|
||||
* - TTY or --pretty: human-readable list of matches, one per line.
|
||||
* - non-TTY or --json: JSON array the agent consumes.
|
||||
*
|
||||
* Uses the content_chunks.symbol_name column (v0.19.0 migration v26).
|
||||
* No tree-sitter re-parsing needed — the metadata is already there.
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { errorFor, serializeError } from '../core/errors.ts';
|
||||
|
||||
export interface CodeDefResult {
|
||||
slug: string;
|
||||
file: string | null;
|
||||
language: string | null;
|
||||
symbol_type: string | null;
|
||||
start_line: number | null;
|
||||
end_line: number | null;
|
||||
snippet: string;
|
||||
}
|
||||
|
||||
export async function findCodeDef(
|
||||
engine: BrainEngine,
|
||||
symbol: string,
|
||||
opts: { limit?: number; language?: string } = {},
|
||||
): Promise<CodeDefResult[]> {
|
||||
const limit = opts.limit ?? 20;
|
||||
const DEF_TYPES = ['function', 'class', 'interface', 'type', 'enum', 'struct', 'trait', 'module', 'contract'];
|
||||
const params: unknown[] = [symbol, limit];
|
||||
let whereLang = '';
|
||||
if (opts.language) {
|
||||
params.splice(1, 0, opts.language);
|
||||
whereLang = 'AND cc.language = $2';
|
||||
}
|
||||
// Deterministic ordering: exact type matches first (functions before
|
||||
// export_statement wrappers), then page slug, then line number.
|
||||
const rows = await engine.executeRaw<{
|
||||
slug: string; file: string | null; language: string | null;
|
||||
symbol_type: string | null; start_line: number | null; end_line: number | null;
|
||||
chunk_text: string;
|
||||
}>(
|
||||
`SELECT p.slug, (p.frontmatter->>'file') AS file, cc.language, cc.symbol_type,
|
||||
cc.start_line, cc.end_line, cc.chunk_text
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.symbol_name = $1
|
||||
${whereLang}
|
||||
AND p.page_kind = 'code'
|
||||
AND cc.symbol_type IN ('${DEF_TYPES.join("','")}', 'export statement')
|
||||
ORDER BY
|
||||
CASE cc.symbol_type
|
||||
WHEN 'function' THEN 1 WHEN 'class' THEN 2 WHEN 'interface' THEN 3
|
||||
WHEN 'type' THEN 4 WHEN 'enum' THEN 5 WHEN 'struct' THEN 6
|
||||
ELSE 7
|
||||
END,
|
||||
p.slug, cc.start_line
|
||||
LIMIT $${params.length}`,
|
||||
params,
|
||||
);
|
||||
return rows.map((r) => ({
|
||||
slug: r.slug,
|
||||
file: r.file,
|
||||
language: r.language,
|
||||
symbol_type: r.symbol_type,
|
||||
start_line: r.start_line,
|
||||
end_line: r.end_line,
|
||||
// First 500 chars of chunk — enough for a preview without flooding output.
|
||||
snippet: r.chunk_text.slice(0, 500),
|
||||
}));
|
||||
}
|
||||
|
||||
function parseFlag(args: string[], name: string): string | undefined {
|
||||
const i = args.indexOf(name);
|
||||
return i >= 0 && i + 1 < args.length ? args[i + 1] : undefined;
|
||||
}
|
||||
|
||||
function shouldEmitJson(args: string[]): boolean {
|
||||
if (args.includes('--json')) return true;
|
||||
if (args.includes('--no-json')) return false;
|
||||
// Auto-detect: non-TTY stdout means an agent is piping us — default to JSON.
|
||||
return !process.stdout.isTTY;
|
||||
}
|
||||
|
||||
export async function runCodeDef(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const symbol = args.find((a) => !a.startsWith('--') && args.indexOf(a) > 0);
|
||||
// args[0] is the symbol when invoked as `gbrain code-def <symbol>`
|
||||
const positional = args.filter((a) => !a.startsWith('--'));
|
||||
const sym = positional[0];
|
||||
if (!sym) {
|
||||
const err = errorFor({
|
||||
class: 'UsageError',
|
||||
code: 'code_def_requires_symbol',
|
||||
message: 'code-def requires a symbol name',
|
||||
hint: 'gbrain code-def <symbol> [--lang <language>] [--json]',
|
||||
});
|
||||
if (shouldEmitJson(args)) {
|
||||
console.log(JSON.stringify({ error: err.envelope }));
|
||||
} else {
|
||||
console.error(err.message);
|
||||
}
|
||||
process.exit(2);
|
||||
}
|
||||
const limit = parseInt(parseFlag(args, '--limit') || '20', 10);
|
||||
const language = parseFlag(args, '--lang');
|
||||
try {
|
||||
const results = await findCodeDef(engine, sym, { limit, language });
|
||||
if (shouldEmitJson(args)) {
|
||||
console.log(JSON.stringify({ symbol: sym, count: results.length, results }, null, 2));
|
||||
} else {
|
||||
if (results.length === 0) {
|
||||
console.log(`No definitions found for "${sym}"`);
|
||||
} else {
|
||||
console.log(`Found ${results.length} definition(s) for "${sym}":`);
|
||||
for (const r of results) {
|
||||
const loc = r.start_line != null ? `:${r.start_line}` : '';
|
||||
console.log(` ${r.file || r.slug}${loc} (${r.symbol_type})`);
|
||||
}
|
||||
}
|
||||
}
|
||||
} catch (e: unknown) {
|
||||
const env = serializeError(e);
|
||||
if (shouldEmitJson(args)) {
|
||||
console.log(JSON.stringify({ error: env }));
|
||||
} else {
|
||||
console.error(`code-def failed: ${env.message}`);
|
||||
}
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
@@ -1,133 +0,0 @@
|
||||
/**
|
||||
* gbrain code-refs <symbol>
|
||||
*
|
||||
* v0.19.0 Layer 7 — find all usage sites of a named symbol across the
|
||||
* brain's code pages. The DX "magical moment" for v0.19.0: an agent
|
||||
* asks "what uses BrainEngine" and gets back a JSON array of
|
||||
* {file, line, snippet} tuples in one CLI call.
|
||||
*
|
||||
* Implementation: bypasses the standard searchKeyword path (which uses
|
||||
* DISTINCT ON (slug) to collapse to one result per page — wrong for
|
||||
* code-refs where a single file typically has many usage sites). Uses
|
||||
* a direct ILIKE scan over content_chunks + JOIN pages, returning every
|
||||
* matching chunk.
|
||||
*
|
||||
* Scope: simple substring match. Word-boundary precision is a follow-up
|
||||
* (would require either tsvector or regex). For v0.19.0 the heuristic
|
||||
* is good enough: symbol names are distinctive by design, and noisy
|
||||
* matches (e.g. 'foo' matching 'food') are rare in well-written code.
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { errorFor, serializeError } from '../core/errors.ts';
|
||||
|
||||
export interface CodeRefResult {
|
||||
slug: string;
|
||||
file: string | null;
|
||||
language: string | null;
|
||||
symbol_name: string | null;
|
||||
symbol_type: string | null;
|
||||
start_line: number | null;
|
||||
end_line: number | null;
|
||||
snippet: string;
|
||||
}
|
||||
|
||||
export async function findCodeRefs(
|
||||
engine: BrainEngine,
|
||||
symbol: string,
|
||||
opts: { limit?: number; language?: string } = {},
|
||||
): Promise<CodeRefResult[]> {
|
||||
const limit = opts.limit ?? 50;
|
||||
const params: unknown[] = [`%${symbol}%`];
|
||||
let whereLang = '';
|
||||
if (opts.language) {
|
||||
params.push(opts.language);
|
||||
whereLang = `AND cc.language = $${params.length}`;
|
||||
}
|
||||
params.push(limit);
|
||||
const rows = await engine.executeRaw<{
|
||||
slug: string; file: string | null; language: string | null;
|
||||
symbol_name: string | null; symbol_type: string | null;
|
||||
start_line: number | null; end_line: number | null;
|
||||
chunk_text: string;
|
||||
}>(
|
||||
`SELECT p.slug, (p.frontmatter->>'file') AS file, cc.language,
|
||||
cc.symbol_name, cc.symbol_type, cc.start_line, cc.end_line,
|
||||
cc.chunk_text
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE p.page_kind = 'code'
|
||||
AND cc.chunk_text ILIKE $1
|
||||
${whereLang}
|
||||
ORDER BY p.slug, cc.start_line NULLS LAST
|
||||
LIMIT $${params.length}`,
|
||||
params,
|
||||
);
|
||||
return rows.map((r) => ({
|
||||
slug: r.slug,
|
||||
file: r.file,
|
||||
language: r.language,
|
||||
symbol_name: r.symbol_name,
|
||||
symbol_type: r.symbol_type,
|
||||
start_line: r.start_line,
|
||||
end_line: r.end_line,
|
||||
snippet: r.chunk_text.slice(0, 500),
|
||||
}));
|
||||
}
|
||||
|
||||
function parseFlag(args: string[], name: string): string | undefined {
|
||||
const i = args.indexOf(name);
|
||||
return i >= 0 && i + 1 < args.length ? args[i + 1] : undefined;
|
||||
}
|
||||
|
||||
function shouldEmitJson(args: string[]): boolean {
|
||||
if (args.includes('--json')) return true;
|
||||
if (args.includes('--no-json')) return false;
|
||||
return !process.stdout.isTTY;
|
||||
}
|
||||
|
||||
export async function runCodeRefs(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const positional = args.filter((a) => !a.startsWith('--'));
|
||||
const sym = positional[0];
|
||||
if (!sym) {
|
||||
const err = errorFor({
|
||||
class: 'UsageError',
|
||||
code: 'code_refs_requires_symbol',
|
||||
message: 'code-refs requires a symbol name',
|
||||
hint: 'gbrain code-refs <symbol> [--lang <language>] [--json]',
|
||||
});
|
||||
if (shouldEmitJson(args)) {
|
||||
console.log(JSON.stringify({ error: err.envelope }));
|
||||
} else {
|
||||
console.error(err.message);
|
||||
}
|
||||
process.exit(2);
|
||||
}
|
||||
const limit = parseInt(parseFlag(args, '--limit') || '50', 10);
|
||||
const language = parseFlag(args, '--lang');
|
||||
try {
|
||||
const results = await findCodeRefs(engine, sym, { limit, language });
|
||||
if (shouldEmitJson(args)) {
|
||||
console.log(JSON.stringify({ symbol: sym, count: results.length, results }, null, 2));
|
||||
} else {
|
||||
if (results.length === 0) {
|
||||
console.log(`No references found for "${sym}"`);
|
||||
} else {
|
||||
console.log(`Found ${results.length} reference(s) to "${sym}":`);
|
||||
for (const r of results) {
|
||||
const loc = r.start_line != null ? `:${r.start_line}` : '';
|
||||
const sig = r.symbol_name ? ` in ${r.symbol_name}` : '';
|
||||
console.log(` ${r.file || r.slug}${loc}${sig}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
} catch (e: unknown) {
|
||||
const env = serializeError(e);
|
||||
if (shouldEmitJson(args)) {
|
||||
console.log(JSON.stringify({ error: env }));
|
||||
} else {
|
||||
console.error(`code-refs failed: ${env.message}`);
|
||||
}
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
@@ -20,7 +20,6 @@ import { v0_14_0 } from './v0_14_0.ts';
|
||||
import { v0_16_0 } from './v0_16_0.ts';
|
||||
import { v0_18_0 } from './v0_18_0.ts';
|
||||
import { v0_18_1 } from './v0_18_1.ts';
|
||||
import { v0_21_0 } from './v0_21_0.ts';
|
||||
|
||||
export const migrations: Migration[] = [
|
||||
v0_11_0,
|
||||
@@ -32,7 +31,6 @@ export const migrations: Migration[] = [
|
||||
v0_16_0,
|
||||
v0_18_0,
|
||||
v0_18_1,
|
||||
v0_21_0,
|
||||
];
|
||||
|
||||
/** Look up a migration by exact version string. */
|
||||
|
||||
@@ -1,155 +0,0 @@
|
||||
/**
|
||||
* v0.21.0 migration orchestrator — Code Cathedral II.
|
||||
*
|
||||
* Cathedral II ships 14 bisectable layers. The user-visible migration
|
||||
* surface is:
|
||||
* - Schema: v27 foundation (code_edges_chunk + code_edges_symbol + new
|
||||
* content_chunks columns + sources.chunker_version gate + chunk-grain
|
||||
* search_vector) and v28 (backfill existing chunks' search_vector).
|
||||
* Both run through the MIGRATIONS chain in src/core/migrate.ts.
|
||||
* - Data backfill: CHUNKER_VERSION bumped 3→4. Layer 12's
|
||||
* sources.chunker_version gate forces a full re-walk next sync on any
|
||||
* source whose tree hasn't drifted, so normal usage rolls the new
|
||||
* chunker shape over existing brains automatically. Users who want
|
||||
* the full reindex NOW run `gbrain reindex-code --yes`.
|
||||
*
|
||||
* Phases:
|
||||
* A. Schema — `gbrain init --migrate-only` applies v27 + v28.
|
||||
* B. Backfill-prompt — emit a pending-host-work notice telling the user
|
||||
* to choose between (1) `gbrain reindex-code --yes` for immediate full
|
||||
* backfill, or (2) accepting gradual sync-driven re-chunk via the
|
||||
* chunker_version gate. No DB side-effects; the orchestrator doesn't
|
||||
* decide for the user.
|
||||
* C. Verify — assert v27 column set exists + CHUNKER_VERSION=4.
|
||||
*
|
||||
* All phases are idempotent and safe to re-run.
|
||||
*/
|
||||
|
||||
import { execSync } from 'child_process';
|
||||
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
|
||||
import { childGlobalFlags } from '../../core/cli-options.ts';
|
||||
|
||||
// ── Phase A — Schema ────────────────────────────────────────
|
||||
|
||||
function phaseASchema(opts: OrchestratorOpts): OrchestratorPhaseResult {
|
||||
if (opts.dryRun) return { name: 'schema', status: 'skipped', detail: 'dry-run' };
|
||||
try {
|
||||
execSync('gbrain init --migrate-only' + childGlobalFlags(), {
|
||||
stdio: 'inherit',
|
||||
timeout: 600_000,
|
||||
env: process.env,
|
||||
});
|
||||
return { name: 'schema', status: 'complete' };
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
return { name: 'schema', status: 'failed', detail: msg };
|
||||
}
|
||||
}
|
||||
|
||||
// ── Phase B — Backfill prompt ───────────────────────────────
|
||||
|
||||
function phaseBBackfillPrompt(opts: OrchestratorOpts): OrchestratorPhaseResult {
|
||||
if (opts.dryRun) return { name: 'backfill_prompt', status: 'skipped', detail: 'dry-run' };
|
||||
|
||||
// Emit a clear console nudge about the two backfill choices. No DB work,
|
||||
// no prompt blocking — Cathedral II's chunker_version gate makes the
|
||||
// schema-level migration zero-cost, and reindex-code is opt-in.
|
||||
console.log('');
|
||||
console.log('=== v0.21.0 Cathedral II — code reindex options ===');
|
||||
console.log('');
|
||||
console.log('Schema migrated. CHUNKER_VERSION bumped 3 → 4 (folds into content_hash).');
|
||||
console.log('');
|
||||
console.log('Two ways to roll the new chunker over existing code pages:');
|
||||
console.log('');
|
||||
console.log(' 1. AUTOMATIC (recommended): next `gbrain sync` detects the version');
|
||||
console.log(' mismatch via sources.chunker_version and forces a full re-walk.');
|
||||
console.log(' No action needed.');
|
||||
console.log('');
|
||||
console.log(' 2. IMMEDIATE: `gbrain reindex-code --dry-run` to preview cost, then');
|
||||
console.log(' `gbrain reindex-code --yes` to reindex every code page now.');
|
||||
console.log('');
|
||||
console.log('Either way, the new chunker ships: qualified symbol identity, chunk-grain');
|
||||
console.log('FTS with doc_comment Weight A, parent scope capture (Layer 6 pending),');
|
||||
console.log('and structural edge resolution (Layer 5 pending).');
|
||||
console.log('');
|
||||
|
||||
return { name: 'backfill_prompt', status: 'complete' };
|
||||
}
|
||||
|
||||
// ── Phase C — Verify ────────────────────────────────────────
|
||||
|
||||
function phaseCVerify(opts: OrchestratorOpts): OrchestratorPhaseResult {
|
||||
if (opts.dryRun) return { name: 'verify', status: 'skipped', detail: 'dry-run' };
|
||||
try {
|
||||
// Round-trip the schema check through `gbrain doctor --json` if available,
|
||||
// but gracefully degrade: the real verification is the migration runner
|
||||
// reporting success on v27/v28 SQL — this is a belt-and-suspenders check.
|
||||
// Cheap and optional; a non-zero exit here does not fail the orchestrator.
|
||||
return { name: 'verify', status: 'complete', detail: 'schema migrations applied via phase A' };
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
return { name: 'verify', status: 'failed', detail: msg };
|
||||
}
|
||||
}
|
||||
|
||||
// ── Orchestrator ────────────────────────────────────────────
|
||||
|
||||
async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult> {
|
||||
console.log('');
|
||||
console.log('=== v0.21.0 — Code Cathedral II ===');
|
||||
if (opts.dryRun) console.log(' (dry-run; no side effects)');
|
||||
console.log('');
|
||||
|
||||
const phases: OrchestratorPhaseResult[] = [];
|
||||
|
||||
const a = phaseASchema(opts);
|
||||
phases.push(a);
|
||||
if (a.status === 'failed') return finalizeResult(phases, 'failed');
|
||||
|
||||
const b = phaseBBackfillPrompt(opts);
|
||||
phases.push(b);
|
||||
|
||||
const c = phaseCVerify(opts);
|
||||
phases.push(c);
|
||||
|
||||
const anyFailed = phases.some(p => p.status === 'failed');
|
||||
const status: OrchestratorResult['status'] = anyFailed ? 'partial' : 'complete';
|
||||
|
||||
return finalizeResult(phases, status);
|
||||
}
|
||||
|
||||
function finalizeResult(
|
||||
phases: OrchestratorPhaseResult[],
|
||||
status: 'complete' | 'partial' | 'failed',
|
||||
): OrchestratorResult {
|
||||
return {
|
||||
version: '0.21.0',
|
||||
status,
|
||||
phases,
|
||||
};
|
||||
}
|
||||
|
||||
// ── Export ──────────────────────────────────────────────────
|
||||
|
||||
export const v0_21_0: Migration = {
|
||||
version: '0.21.0',
|
||||
featurePitch: {
|
||||
headline: 'Code Cathedral II — chunk-grain FTS, qualified symbols, structural edges, 165-language lazy-load',
|
||||
description:
|
||||
'v0.21.0 ships the biggest code-search upgrade in gbrain history. Chunk-grain FTS ' +
|
||||
'with doc_comment Weight A ranks natural-language queries against docstrings above ' +
|
||||
'prose. CHUNKER_VERSION 3 → 4 folds into content_hash so every existing code page ' +
|
||||
're-chunks on next sync (via sources.chunker_version gate) or immediately via ' +
|
||||
'`gbrain reindex-code --yes`. File classifier widened to 35 extensions. Markdown ' +
|
||||
'fence extraction, sync --all cost preview, and reconcile-links batch command ' +
|
||||
'ship alongside the chunker upgrade.',
|
||||
},
|
||||
orchestrator,
|
||||
};
|
||||
|
||||
/** Exported for unit tests. */
|
||||
export const __testing = {
|
||||
phaseASchema,
|
||||
phaseBBackfillPrompt,
|
||||
phaseCVerify,
|
||||
};
|
||||
@@ -1,177 +0,0 @@
|
||||
/**
|
||||
* v0.20.0 Cathedral II Layer 8 D3 — reconcile-links batch command.
|
||||
*
|
||||
* Closes the v0.19.0 Layer 6 doc↔impl order-dependency bug. When a
|
||||
* markdown guide cites `src/core/sync.ts:42` but the code source
|
||||
* hasn't been synced yet, the forward-scan at import time inserts
|
||||
* nothing because `addLink`'s inner SELECT drops edges to missing
|
||||
* pages. The guide and the code eventually both exist, but the edge
|
||||
* never materialized.
|
||||
*
|
||||
* D3 fixes this batch-style: walk every markdown page, re-run
|
||||
* `extractCodeRefs`, and call `addLink(md, code, ..., 'documents')` +
|
||||
* reverse for each hit. ON CONFLICT DO NOTHING on the `links` table
|
||||
* makes the operation idempotent — edges that already exist stay,
|
||||
* new edges land.
|
||||
*
|
||||
* Why batch over per-import-reverse-scan: codex 2-phase review
|
||||
* flagged the per-import approach as O(N) ILIKE/JOIN queries per
|
||||
* code file imported. On a 47K-page brain first-syncing 5K code
|
||||
* files, that's 5K ILIKE scans. A user-triggered batch pass on an
|
||||
* already-synced brain is one walk, fully indexed via the existing
|
||||
* slug lookup in addLink.
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { extractCodeRefs } from '../core/link-extraction.ts';
|
||||
import { slugifyCodePath } from '../core/sync.ts';
|
||||
import { createProgress } from '../core/progress.ts';
|
||||
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
|
||||
|
||||
export interface ReconcileLinksResult {
|
||||
status: 'ok' | 'auto_link_disabled';
|
||||
markdownPagesScanned: number;
|
||||
codeRefsFound: number;
|
||||
edgesAttempted: number;
|
||||
edgesTargetsMissing: number;
|
||||
}
|
||||
|
||||
export interface ReconcileLinksOpts {
|
||||
sourceId?: string;
|
||||
dryRun?: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* Scan every markdown page for code-path references (e.g.
|
||||
* `src/core/sync.ts`, `lib/foo.py:42`) and create bidirectional
|
||||
* doc↔impl edges (`documents` + `documented_by`) for each hit
|
||||
* that resolves to a code page. Idempotent via ON CONFLICT DO
|
||||
* NOTHING in the underlying addLink path.
|
||||
*
|
||||
* Called by `gbrain reconcile-links` CLI surface. Respects the
|
||||
* `auto_link` config: if the user has disabled auto-linking on
|
||||
* put_page, reconcile-links doesn't silently re-populate those
|
||||
* edges either.
|
||||
*/
|
||||
export async function runReconcileLinks(
|
||||
engine: BrainEngine,
|
||||
opts: ReconcileLinksOpts = {},
|
||||
): Promise<ReconcileLinksResult> {
|
||||
// Respect auto_link config (same gate put_page uses). A user that
|
||||
// explicitly turned off auto-link doesn't want reconcile-links
|
||||
// writing edges back either.
|
||||
const autoLinkCfg = await engine.getConfig('auto_link');
|
||||
if (autoLinkCfg === 'false') {
|
||||
return {
|
||||
status: 'auto_link_disabled',
|
||||
markdownPagesScanned: 0,
|
||||
codeRefsFound: 0,
|
||||
edgesAttempted: 0,
|
||||
edgesTargetsMissing: 0,
|
||||
};
|
||||
}
|
||||
|
||||
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
|
||||
|
||||
// Walk all markdown slugs. listPages(markdown-only filter) isn't exposed,
|
||||
// so filter at call time via page_kind. Not using getAllSlugs because we
|
||||
// also need compiled_truth + timeline for extractCodeRefs.
|
||||
const mdSlugs = (await engine.executeRaw<{ slug: string }>(
|
||||
`SELECT slug FROM pages WHERE page_kind = 'markdown' ORDER BY slug`,
|
||||
)).map(r => r.slug);
|
||||
|
||||
progress.start('reconcile_links.scan', mdSlugs.length);
|
||||
|
||||
let codeRefsFound = 0;
|
||||
let edgesAttempted = 0;
|
||||
let edgesTargetsMissing = 0;
|
||||
|
||||
// Fetch pages one at a time via getPage (no bulk read helper exists yet).
|
||||
// On a 47K-page brain this is the slow path; a v0.20.x follow-up can add
|
||||
// getPagesBatch. For the typical 2K–5K markdown count it's fine.
|
||||
for (const mdSlug of mdSlugs) {
|
||||
const page = await engine.getPage(mdSlug);
|
||||
if (!page) {
|
||||
progress.tick(1, mdSlug);
|
||||
continue;
|
||||
}
|
||||
const haystack = (page.compiled_truth || '') + '\n' + (page.timeline || '');
|
||||
const refs = extractCodeRefs(haystack);
|
||||
if (refs.length === 0) {
|
||||
progress.tick(1, mdSlug);
|
||||
continue;
|
||||
}
|
||||
codeRefsFound += refs.length;
|
||||
|
||||
if (opts.dryRun) {
|
||||
progress.tick(1, `${mdSlug} (+${refs.length} refs)`);
|
||||
continue;
|
||||
}
|
||||
|
||||
for (const ref of refs) {
|
||||
const codeSlug = slugifyCodePath(ref.path);
|
||||
const ctx = ref.line ? `cited at ${ref.path}:${ref.line}` : ref.path;
|
||||
edgesAttempted++;
|
||||
try {
|
||||
// Forward: guide documents code. addLink's inner SELECT drops
|
||||
// silently if codeSlug isn't a page yet (benign — counted below).
|
||||
await engine.addLink(mdSlug, codeSlug, ctx, 'documents', 'markdown', mdSlug, 'compiled_truth');
|
||||
await engine.addLink(codeSlug, mdSlug, ref.path, 'documented_by', 'markdown', mdSlug, 'compiled_truth');
|
||||
} catch (e: unknown) {
|
||||
// Per-link errors don't abort the batch. Track them for the summary.
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
if (/not found|does not exist/i.test(msg)) {
|
||||
edgesTargetsMissing++;
|
||||
} else {
|
||||
// Real error — log but keep going. Agents can inspect progress events.
|
||||
console.warn(`[reconcile-links] ${mdSlug} → ${codeSlug}: ${msg}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
progress.tick(1, `${mdSlug} (+${refs.length} refs)`);
|
||||
}
|
||||
|
||||
progress.finish();
|
||||
|
||||
return {
|
||||
status: 'ok',
|
||||
markdownPagesScanned: mdSlugs.length,
|
||||
codeRefsFound,
|
||||
edgesAttempted,
|
||||
edgesTargetsMissing,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* CLI entry. Parses argv, runs runReconcileLinks, prints a summary.
|
||||
* --dry-run reports counts without writing. --json emits machine output.
|
||||
*/
|
||||
export async function runReconcileLinksCli(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const dryRun = args.includes('--dry-run');
|
||||
const jsonOut = args.includes('--json');
|
||||
|
||||
const result = await runReconcileLinks(engine, { dryRun });
|
||||
|
||||
if (jsonOut) {
|
||||
console.log(JSON.stringify(result));
|
||||
return;
|
||||
}
|
||||
|
||||
if (result.status === 'auto_link_disabled') {
|
||||
console.log(
|
||||
'[reconcile-links] auto_link is disabled in config; skipping. ' +
|
||||
'Set `gbrain config set auto_link true` to re-enable.',
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
const header = dryRun ? 'reconcile-links (dry run)' : 'reconcile-links';
|
||||
console.log(
|
||||
`${header}: scanned ${result.markdownPagesScanned} markdown pages, ` +
|
||||
`found ${result.codeRefsFound} code refs, ` +
|
||||
`attempted ${result.edgesAttempted} edges` +
|
||||
(result.edgesTargetsMissing > 0
|
||||
? ` (${result.edgesTargetsMissing} targets missing code page)`
|
||||
: ''),
|
||||
);
|
||||
}
|
||||
@@ -1,324 +0,0 @@
|
||||
/**
|
||||
* v0.21.0 Cathedral II Layer 13 (E2) — `gbrain reindex-code`.
|
||||
*
|
||||
* Explicit backfill for v0.19.0 → v0.21.0 brains. Layer 12's
|
||||
* `sources.chunker_version` gate forces a re-walk next sync on any source
|
||||
* whose working tree hasn't drifted, but users who want the benefits NOW
|
||||
* (before the next sync) get this: walk every page where type='code', read
|
||||
* compiled_truth + frontmatter.file, re-import via importCodeFile. Pages
|
||||
* flow through the same code path as normal sync (chunker + embeddings +
|
||||
* content_hash folding), so a reindex is bit-identical to a fresh sync.
|
||||
*
|
||||
* Flags:
|
||||
* --source <id> Scope to one sources row. Omit = all code pages.
|
||||
* --dry-run Preview cost + page count, exit 0.
|
||||
* --yes Skip interactive [y/N]. Required for non-TTY + non-JSON.
|
||||
* --json Machine-readable ConfirmationRequired / result envelope.
|
||||
* --force Bypass importCodeFile's content_hash early-return. Use
|
||||
* this for paranoid full reindex when content_hash equals
|
||||
* but you still want a re-chunk + re-embed pass.
|
||||
*
|
||||
* Batched in chunks of 100 pages to avoid OOM on 47K-page brains (codex
|
||||
* review Finding 4.4). Idempotent: re-running on already-reindexed pages
|
||||
* is a no-op unless --force is passed.
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { importCodeFile } from '../core/import-file.ts';
|
||||
import { estimateTokens } from '../core/chunkers/code.ts';
|
||||
import { EMBEDDING_MODEL, estimateEmbeddingCostUsd } from '../core/embedding.ts';
|
||||
import { errorFor, serializeError } from '../core/errors.ts';
|
||||
import { createInterface } from 'readline';
|
||||
import { createProgress } from '../core/progress.ts';
|
||||
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
|
||||
|
||||
export interface ReindexCodeOpts {
|
||||
sourceId?: string;
|
||||
dryRun?: boolean;
|
||||
yes?: boolean;
|
||||
json?: boolean;
|
||||
force?: boolean;
|
||||
noEmbed?: boolean;
|
||||
/** Page batch size. Default 100 (codex Finding 4.4 OOM protection). */
|
||||
batchSize?: number;
|
||||
}
|
||||
|
||||
export interface ReindexCodeResult {
|
||||
status: 'ok' | 'dry_run' | 'cancelled' | 'source_id_required';
|
||||
codePages: number;
|
||||
reindexed: number;
|
||||
skipped: number;
|
||||
failed: number;
|
||||
totalTokens: number;
|
||||
costUsd: number;
|
||||
model: string;
|
||||
failures?: Array<{ slug: string; error: string }>;
|
||||
}
|
||||
|
||||
interface CodePageRow {
|
||||
slug: string;
|
||||
compiled_truth: string;
|
||||
frontmatter: Record<string, unknown> | null;
|
||||
}
|
||||
|
||||
async function fetchCodePages(
|
||||
engine: BrainEngine,
|
||||
sourceId: string | undefined,
|
||||
batchSize: number,
|
||||
offset: number,
|
||||
): Promise<CodePageRow[]> {
|
||||
// Direct SQL: listPages doesn't expose source_id filtering, and we need
|
||||
// compiled_truth + frontmatter anyway (not just the Page shape).
|
||||
const sourceClause = sourceId ? `AND p.source_id = '${sourceId.replace(/'/g, "''")}'` : '';
|
||||
const rows = await engine.executeRaw<CodePageRow>(
|
||||
`SELECT p.slug, p.compiled_truth, p.frontmatter
|
||||
FROM pages p
|
||||
WHERE p.type = 'code' ${sourceClause}
|
||||
ORDER BY p.slug
|
||||
LIMIT ${batchSize} OFFSET ${offset}`,
|
||||
);
|
||||
return rows;
|
||||
}
|
||||
|
||||
async function countCodePages(engine: BrainEngine, sourceId: string | undefined): Promise<number> {
|
||||
const sourceClause = sourceId ? `AND p.source_id = '${sourceId.replace(/'/g, "''")}'` : '';
|
||||
const rows = await engine.executeRaw<{ n: string | number }>(
|
||||
`SELECT COUNT(*)::text AS n FROM pages p WHERE p.type = 'code' ${sourceClause}`,
|
||||
);
|
||||
if (rows.length === 0) return 0;
|
||||
const raw = rows[0]!.n;
|
||||
return typeof raw === 'string' ? parseInt(raw, 10) : raw;
|
||||
}
|
||||
|
||||
/**
|
||||
* Estimate total embedding cost for a reindex. Walks every code page's
|
||||
* compiled_truth and sums tokens. Conservative: does not try to detect
|
||||
* unchanged chunks (the incremental embedding cache in importCodeFile does
|
||||
* that; this estimate is the ceiling, not the floor).
|
||||
*/
|
||||
async function estimateReindexCost(
|
||||
engine: BrainEngine,
|
||||
sourceId: string | undefined,
|
||||
batchSize: number,
|
||||
): Promise<{ totalTokens: number; totalPages: number }> {
|
||||
let totalTokens = 0;
|
||||
let totalPages = 0;
|
||||
let offset = 0;
|
||||
while (true) {
|
||||
const batch = await fetchCodePages(engine, sourceId, batchSize, offset);
|
||||
if (batch.length === 0) break;
|
||||
for (const row of batch) {
|
||||
if (row.compiled_truth) totalTokens += estimateTokens(row.compiled_truth);
|
||||
totalPages++;
|
||||
}
|
||||
offset += batch.length;
|
||||
if (batch.length < batchSize) break;
|
||||
}
|
||||
return { totalTokens, totalPages };
|
||||
}
|
||||
|
||||
async function promptYesNo(question: string): Promise<boolean> {
|
||||
return new Promise((resolve) => {
|
||||
const rl = createInterface({ input: process.stdin, output: process.stdout });
|
||||
rl.question(question, (answer) => {
|
||||
rl.close();
|
||||
const a = answer.trim().toLowerCase();
|
||||
resolve(a === 'y' || a === 'yes');
|
||||
});
|
||||
rl.on('close', () => resolve(false));
|
||||
});
|
||||
}
|
||||
|
||||
export async function runReindexCode(
|
||||
engine: BrainEngine,
|
||||
opts: ReindexCodeOpts = {},
|
||||
): Promise<ReindexCodeResult> {
|
||||
const batchSize = opts.batchSize ?? 100;
|
||||
|
||||
const { totalTokens, totalPages } = await estimateReindexCost(engine, opts.sourceId, batchSize);
|
||||
const costUsd = estimateEmbeddingCostUsd(totalTokens);
|
||||
|
||||
if (opts.dryRun) {
|
||||
return {
|
||||
status: 'dry_run',
|
||||
codePages: totalPages,
|
||||
reindexed: 0,
|
||||
skipped: 0,
|
||||
failed: 0,
|
||||
totalTokens,
|
||||
costUsd,
|
||||
model: EMBEDDING_MODEL,
|
||||
};
|
||||
}
|
||||
|
||||
if (totalPages === 0) {
|
||||
return {
|
||||
status: 'ok',
|
||||
codePages: 0,
|
||||
reindexed: 0,
|
||||
skipped: 0,
|
||||
failed: 0,
|
||||
totalTokens: 0,
|
||||
costUsd: 0,
|
||||
model: EMBEDDING_MODEL,
|
||||
};
|
||||
}
|
||||
|
||||
// Walk every code page, re-run importCodeFile with compiled_truth as
|
||||
// the content source. relativePath comes from frontmatter.file (set by
|
||||
// the original importCodeFile call). Progress via stderr reporter.
|
||||
const reporter = createProgress(cliOptsToProgressOptions(getCliOptions()));
|
||||
reporter.start('reindex_code.pages', totalPages);
|
||||
|
||||
let reindexed = 0;
|
||||
let skipped = 0;
|
||||
let failed = 0;
|
||||
const failures: Array<{ slug: string; error: string }> = [];
|
||||
let offset = 0;
|
||||
|
||||
try {
|
||||
while (true) {
|
||||
const batch = await fetchCodePages(engine, opts.sourceId, batchSize, offset);
|
||||
if (batch.length === 0) break;
|
||||
|
||||
for (const row of batch) {
|
||||
const fm = row.frontmatter ?? {};
|
||||
const relPath = typeof fm.file === 'string' ? fm.file : null;
|
||||
if (!relPath) {
|
||||
failed++;
|
||||
failures.push({ slug: row.slug, error: 'missing frontmatter.file' });
|
||||
reporter.tick();
|
||||
continue;
|
||||
}
|
||||
if (!row.compiled_truth) {
|
||||
failed++;
|
||||
failures.push({ slug: row.slug, error: 'missing compiled_truth' });
|
||||
reporter.tick();
|
||||
continue;
|
||||
}
|
||||
try {
|
||||
const result = await importCodeFile(engine, relPath, row.compiled_truth, {
|
||||
noEmbed: opts.noEmbed,
|
||||
force: opts.force,
|
||||
});
|
||||
if (result.status === 'imported') reindexed++;
|
||||
else if (result.status === 'skipped') skipped++;
|
||||
else {
|
||||
failed++;
|
||||
failures.push({ slug: row.slug, error: result.error ?? result.status });
|
||||
}
|
||||
} catch (e: unknown) {
|
||||
failed++;
|
||||
failures.push({ slug: row.slug, error: e instanceof Error ? e.message : String(e) });
|
||||
}
|
||||
reporter.tick();
|
||||
}
|
||||
|
||||
offset += batch.length;
|
||||
if (batch.length < batchSize) break;
|
||||
}
|
||||
} finally {
|
||||
reporter.finish();
|
||||
}
|
||||
|
||||
return {
|
||||
status: 'ok',
|
||||
codePages: totalPages,
|
||||
reindexed,
|
||||
skipped,
|
||||
failed,
|
||||
totalTokens,
|
||||
costUsd,
|
||||
model: EMBEDDING_MODEL,
|
||||
failures: failures.length > 0 ? failures : undefined,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* CLI entrypoint. Parses argv, wires cost-preview gate + JSON/TTY branching,
|
||||
* delegates to runReindexCode. Exit codes: 0 on success/dry-run, 2 on
|
||||
* ConfirmationRequired (matches sync --all), 1 on runtime error.
|
||||
*/
|
||||
export async function runReindexCodeCli(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const sourceIdx = args.indexOf('--source');
|
||||
const sourceId = sourceIdx >= 0 ? args[sourceIdx + 1] : undefined;
|
||||
const dryRun = args.includes('--dry-run');
|
||||
const yes = args.includes('--yes') || args.includes('-y');
|
||||
const json = args.includes('--json');
|
||||
const force = args.includes('--force');
|
||||
const noEmbed = args.includes('--no-embed');
|
||||
|
||||
if (dryRun) {
|
||||
const result = await runReindexCode(engine, { sourceId, dryRun: true, yes, json, force, noEmbed });
|
||||
if (json) {
|
||||
console.log(JSON.stringify(result));
|
||||
} else {
|
||||
console.log(
|
||||
`reindex-code preview: ${result.codePages} code page(s), ` +
|
||||
`~${result.totalTokens.toLocaleString()} tokens, ` +
|
||||
`est. $${result.costUsd.toFixed(2)} on ${result.model}.`,
|
||||
);
|
||||
console.log('--dry-run: exit without reindexing.');
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
// Cost preview + gate, before touching the DB.
|
||||
if (!noEmbed) {
|
||||
const preview = await estimateReindexCost(engine, sourceId, 100);
|
||||
const costUsd = estimateEmbeddingCostUsd(preview.totalTokens);
|
||||
const previewMsg =
|
||||
`reindex-code: ${preview.totalPages} code page(s), ` +
|
||||
`~${preview.totalTokens.toLocaleString()} tokens, ` +
|
||||
`est. $${costUsd.toFixed(2)} on ${EMBEDDING_MODEL}.`;
|
||||
|
||||
if (preview.totalPages === 0) {
|
||||
if (json) {
|
||||
console.log(JSON.stringify({ status: 'ok', codePages: 0, reindexed: 0, skipped: 0, failed: 0, totalTokens: 0, costUsd: 0, model: EMBEDDING_MODEL }));
|
||||
} else {
|
||||
console.log('No code pages to reindex.');
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
if (!yes) {
|
||||
const isTTY = Boolean(process.stdout.isTTY) && Boolean(process.stdin.isTTY);
|
||||
if (!isTTY || json) {
|
||||
const envelope = serializeError(errorFor({
|
||||
class: 'ConfirmationRequired',
|
||||
code: 'cost_preview_requires_yes',
|
||||
message: previewMsg,
|
||||
hint: 'Pass --yes to proceed, or --dry-run to see the preview and exit 0.',
|
||||
}));
|
||||
console.log(JSON.stringify({ error: envelope, preview, costUsd, model: EMBEDDING_MODEL }));
|
||||
process.exit(2);
|
||||
}
|
||||
console.log(previewMsg);
|
||||
const answer = await promptYesNo('Proceed? [y/N] ');
|
||||
if (!answer) {
|
||||
console.log('Cancelled.');
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const result = await runReindexCode(engine, { sourceId, yes, json, force, noEmbed });
|
||||
if (json) {
|
||||
console.log(JSON.stringify(result));
|
||||
} else {
|
||||
console.log(
|
||||
`reindex-code: ${result.reindexed} reindexed, ${result.skipped} skipped, ${result.failed} failed ` +
|
||||
`(${result.codePages} total code pages, ~${result.totalTokens.toLocaleString()} tokens, ` +
|
||||
`est. $${result.costUsd.toFixed(2)}).`,
|
||||
);
|
||||
if (result.failures && result.failures.length > 0) {
|
||||
console.log(`\n${result.failures.length} failure(s):`);
|
||||
for (const f of result.failures.slice(0, 10)) {
|
||||
console.log(` ${f.slug}: ${f.error}`);
|
||||
}
|
||||
if (result.failures.length > 10) {
|
||||
console.log(` ... and ${result.failures.length - 10} more`);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
+17
-296
@@ -3,19 +3,14 @@ import { execFileSync } from 'child_process';
|
||||
import { join, relative } from 'path';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { importFile } from '../core/import-file.ts';
|
||||
import { readFileSync, statSync, readdirSync } from 'fs';
|
||||
import { createInterface } from 'readline';
|
||||
import {
|
||||
buildSyncManifest,
|
||||
isSyncable,
|
||||
resolveSlugForPath,
|
||||
pathToSlug,
|
||||
recordSyncFailures,
|
||||
unacknowledgedSyncFailures,
|
||||
acknowledgeSyncFailures,
|
||||
} from '../core/sync.ts';
|
||||
import { estimateTokens, CHUNKER_VERSION } from '../core/chunkers/code.ts';
|
||||
import { EMBEDDING_MODEL, estimateEmbeddingCostUsd } from '../core/embedding.ts';
|
||||
import { errorFor, serializeError } from '../core/errors.ts';
|
||||
import type { SyncManifest } from '../core/sync.ts';
|
||||
import { createProgress } from '../core/progress.ts';
|
||||
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
|
||||
@@ -35,107 +30,6 @@ export interface SyncResult {
|
||||
failedFiles?: number; // count of parse failures (Bug 9)
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.20.0 Cathedral II Layer 8 (D1) — walk each source's working tree and
|
||||
* sum tokens for every syncable file. This is a conservative overestimate
|
||||
* (full file content, not just the incremental diff) because `sync --all`
|
||||
* on a source that hasn't been synced yet WILL embed every file in the
|
||||
* working tree. For already-synced sources with only incremental changes,
|
||||
* the overestimate is the ceiling, not the floor — users never get
|
||||
* surprised by MORE cost than the preview claims. The false-high bias is
|
||||
* intentional: a lower estimate that undersells the real bill would be
|
||||
* worse than one that oversells.
|
||||
*/
|
||||
function estimateSyncAllCost(sources: Array<{ local_path: string | null; config: Record<string, unknown> }>): {
|
||||
totalTokens: number;
|
||||
totalFiles: number;
|
||||
activeSources: number;
|
||||
perSource: Array<{ path: string; tokens: number; files: number }>;
|
||||
} {
|
||||
let totalTokens = 0;
|
||||
let totalFiles = 0;
|
||||
let activeSources = 0;
|
||||
const perSource: Array<{ path: string; tokens: number; files: number }> = [];
|
||||
|
||||
for (const src of sources) {
|
||||
if (!src.local_path) continue;
|
||||
const cfg = (src.config || {}) as { syncEnabled?: boolean; strategy?: 'markdown' | 'code' | 'auto' };
|
||||
if (cfg.syncEnabled === false) continue;
|
||||
activeSources++;
|
||||
let sourceTokens = 0;
|
||||
let sourceFiles = 0;
|
||||
try {
|
||||
walkSyncableFiles(src.local_path, (filePath: string, content: string) => {
|
||||
sourceTokens += estimateTokens(content);
|
||||
sourceFiles++;
|
||||
}, cfg.strategy ?? 'markdown');
|
||||
} catch {
|
||||
// Best-effort: a source whose local_path is gone or unreadable just
|
||||
// contributes 0. The sync itself would have failed anyway; no point
|
||||
// blocking the preview on a pre-existing fault.
|
||||
}
|
||||
totalTokens += sourceTokens;
|
||||
totalFiles += sourceFiles;
|
||||
perSource.push({ path: src.local_path, tokens: sourceTokens, files: sourceFiles });
|
||||
}
|
||||
|
||||
return { totalTokens, totalFiles, activeSources, perSource };
|
||||
}
|
||||
|
||||
/**
|
||||
* Walk a repo's working tree and invoke `cb(path, content)` for each
|
||||
* syncable file. Honors the same strategy as `isSyncable` so the preview
|
||||
* and the real sync agree on what's in scope.
|
||||
*/
|
||||
function walkSyncableFiles(
|
||||
repoRoot: string,
|
||||
cb: (path: string, content: string) => void,
|
||||
strategy: 'markdown' | 'code' | 'auto',
|
||||
): void {
|
||||
const stack: string[] = [repoRoot];
|
||||
while (stack.length > 0) {
|
||||
const dir = stack.pop()!;
|
||||
let entries: import('fs').Dirent[];
|
||||
try {
|
||||
entries = readdirSync(dir, { withFileTypes: true }) as unknown as import('fs').Dirent[];
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
for (const entry of entries) {
|
||||
const name = typeof entry.name === 'string' ? entry.name : String(entry.name);
|
||||
// Skip hidden dirs, .git, node_modules (same rules isSyncable applies).
|
||||
if (name.startsWith('.') || name === 'node_modules' || name === 'ops') continue;
|
||||
const fullPath = `${dir}/${name}`;
|
||||
if (entry.isDirectory()) {
|
||||
stack.push(fullPath);
|
||||
} else if (entry.isFile()) {
|
||||
const relativePath = fullPath.slice(repoRoot.length + 1);
|
||||
if (!isSyncable(relativePath, { strategy })) continue;
|
||||
try {
|
||||
const stat = statSync(fullPath);
|
||||
if (stat.size > 5_000_000) continue; // skip large binaries
|
||||
const content = readFileSync(fullPath, 'utf-8');
|
||||
cb(fullPath, content);
|
||||
} catch {
|
||||
// Ignore files we can't read; consistent with sync's own tolerance.
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/** Interactive [y/N] prompt. Resolves false on non-y answers or EOF. */
|
||||
async function promptYesNo(question: string): Promise<boolean> {
|
||||
return new Promise((resolve) => {
|
||||
const rl = createInterface({ input: process.stdin, output: process.stdout });
|
||||
rl.question(question, (answer) => {
|
||||
rl.close();
|
||||
resolve(answer.trim().toLowerCase() === 'y' || answer.trim().toLowerCase() === 'yes');
|
||||
});
|
||||
rl.on('close', () => resolve(false));
|
||||
});
|
||||
}
|
||||
|
||||
export interface SyncOpts {
|
||||
repoPath?: string;
|
||||
dryRun?: boolean;
|
||||
@@ -155,8 +49,6 @@ export interface SyncOpts {
|
||||
* pre-v0.17 global-config path unchanged.
|
||||
*/
|
||||
sourceId?: string;
|
||||
/** Multi-repo: sync strategy override (markdown, code, auto). */
|
||||
strategy?: 'markdown' | 'code' | 'auto';
|
||||
}
|
||||
|
||||
function git(repoPath: string, ...args: string[]): string {
|
||||
@@ -212,43 +104,6 @@ async function writeSyncAnchor(
|
||||
await engine.setConfig(`sync.${which}`, value);
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.20.0 Cathedral II Layer 12 (SP-1 fix) — read/write the chunker version
|
||||
* last used to sync a given source. When it mismatches CURRENT_CHUNKER_VERSION,
|
||||
* `performSync` forces a full walk regardless of git HEAD equality. Without
|
||||
* this gate, bumping CHUNKER_VERSION does NOTHING on an unchanged repo
|
||||
* because sync short-circuits at `up_to_date` before reaching
|
||||
* `importCodeFile`'s content_hash check.
|
||||
*
|
||||
* Per-source storage matches writeSyncAnchor's shape — sources.chunker_version
|
||||
* TEXT column from the v27 migration. No global fallback: non-source syncs
|
||||
* (pre-v0.17 brains with no sources table) never had CHUNKER_VERSION
|
||||
* version-gating, so they keep the v0.19.0 behavior.
|
||||
*/
|
||||
async function readChunkerVersion(
|
||||
engine: BrainEngine,
|
||||
sourceId: string | undefined,
|
||||
): Promise<string | null> {
|
||||
if (!sourceId) return null;
|
||||
const rows = await engine.executeRaw<{ chunker_version: string | null }>(
|
||||
`SELECT chunker_version FROM sources WHERE id = $1`,
|
||||
[sourceId],
|
||||
);
|
||||
return rows[0]?.chunker_version ?? null;
|
||||
}
|
||||
|
||||
async function writeChunkerVersion(
|
||||
engine: BrainEngine,
|
||||
sourceId: string | undefined,
|
||||
version: string,
|
||||
): Promise<void> {
|
||||
if (!sourceId) return;
|
||||
await engine.executeRaw(
|
||||
`UPDATE sources SET chunker_version = $1 WHERE id = $2`,
|
||||
[version, sourceId],
|
||||
);
|
||||
}
|
||||
|
||||
export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise<SyncResult> {
|
||||
// Resolve repo path
|
||||
const repoPath = opts.repoPath || await readSyncAnchor(engine, opts.sourceId, 'repo_path');
|
||||
@@ -312,19 +167,8 @@ export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise<
|
||||
return performFullSync(engine, repoPath, headCommit, opts);
|
||||
}
|
||||
|
||||
// v0.20.0 Cathedral II Layer 12 (codex SP-1 fix): before returning
|
||||
// 'up_to_date' on git-HEAD equality, check the chunker version gate.
|
||||
// If sources.chunker_version mismatches CURRENT_CHUNKER_VERSION, force
|
||||
// a full re-walk so existing chunks get re-chunked under the new
|
||||
// pipeline (qualified symbol names, parent scope, doc-comment column
|
||||
// population, etc.). Without this, upgraded brains silently stay on
|
||||
// the old chunks — the whole reason we bumped the version.
|
||||
const storedVersion = await readChunkerVersion(engine, opts.sourceId);
|
||||
const currentVersion = String(CHUNKER_VERSION);
|
||||
const versionMismatch = storedVersion !== null && storedVersion !== currentVersion;
|
||||
const versionNeverSet = storedVersion === null && opts.sourceId !== undefined;
|
||||
|
||||
if (lastCommit === headCommit && !versionMismatch && !versionNeverSet) {
|
||||
// No changes
|
||||
if (lastCommit === headCommit) {
|
||||
return {
|
||||
status: 'up_to_date',
|
||||
fromCommit: lastCommit,
|
||||
@@ -336,38 +180,22 @@ export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise<
|
||||
};
|
||||
}
|
||||
|
||||
if ((versionMismatch || versionNeverSet) && lastCommit === headCommit) {
|
||||
console.log(
|
||||
`[sync] chunker_version gate: stored=${storedVersion ?? 'unset'}, current=${currentVersion}. ` +
|
||||
`Forcing full re-chunk pass (git HEAD unchanged but pipeline version advanced).`,
|
||||
);
|
||||
const result = await performFullSync(engine, repoPath, headCommit, opts);
|
||||
await writeChunkerVersion(engine, opts.sourceId, currentVersion);
|
||||
return result;
|
||||
}
|
||||
|
||||
// Diff using git diff (net result, not per-commit)
|
||||
const diffOutput = git(repoPath, 'diff', '--name-status', '-M', `${lastCommit}..${headCommit}`);
|
||||
const manifest = buildSyncManifest(diffOutput);
|
||||
|
||||
// Filter to syncable files (strategy-aware)
|
||||
const syncOpts = opts.strategy ? { strategy: opts.strategy } : undefined;
|
||||
// Filter to syncable files
|
||||
const filtered: SyncManifest = {
|
||||
added: manifest.added.filter(p => isSyncable(p, syncOpts)),
|
||||
modified: manifest.modified.filter(p => isSyncable(p, syncOpts)),
|
||||
deleted: manifest.deleted.filter(p => isSyncable(p, syncOpts)),
|
||||
renamed: manifest.renamed.filter(r => isSyncable(r.to, syncOpts)),
|
||||
added: manifest.added.filter(p => isSyncable(p)),
|
||||
modified: manifest.modified.filter(p => isSyncable(p)),
|
||||
deleted: manifest.deleted.filter(p => isSyncable(p)),
|
||||
renamed: manifest.renamed.filter(r => isSyncable(r.to)),
|
||||
};
|
||||
|
||||
// Delete pages that became un-syncable (modified but filtered out).
|
||||
// v0.20.0 Cathedral II SP-5: resolveSlugForPath picks the right slug shape
|
||||
// (markdown vs code) based on the chunker's classifier, so a Rust file that
|
||||
// became un-syncable (e.g., moved under `.gitignore` or filtered by
|
||||
// strategy=markdown) deletes the actual code-slug page, not a ghost
|
||||
// markdown-slug that never existed.
|
||||
const unsyncableModified = manifest.modified.filter(p => !isSyncable(p, syncOpts));
|
||||
// Delete pages that became un-syncable (modified but filtered out)
|
||||
const unsyncableModified = manifest.modified.filter(p => !isSyncable(p));
|
||||
for (const path of unsyncableModified) {
|
||||
const slug = resolveSlugForPath(path);
|
||||
const slug = pathToSlug(path);
|
||||
try {
|
||||
const existing = await engine.getPage(slug);
|
||||
if (existing) {
|
||||
@@ -406,7 +234,6 @@ export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise<
|
||||
// Update sync state even with no syncable changes (git advanced)
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'last_commit', headCommit);
|
||||
await engine.setConfig('sync.last_run', new Date().toISOString());
|
||||
await writeChunkerVersion(engine, opts.sourceId, String(CHUNKER_VERSION));
|
||||
return {
|
||||
status: 'up_to_date',
|
||||
fromCommit: lastCommit,
|
||||
@@ -431,12 +258,11 @@ export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise<
|
||||
// Phases: sync.deletes, sync.renames, sync.imports.
|
||||
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
|
||||
|
||||
// Process deletes first (prevents slug conflicts). SP-5: resolveSlugForPath
|
||||
// dispatches to the right slug shape so code file deletes hit the real page.
|
||||
// Process deletes first (prevents slug conflicts)
|
||||
if (filtered.deleted.length > 0) {
|
||||
progress.start('sync.deletes', filtered.deleted.length);
|
||||
for (const path of filtered.deleted) {
|
||||
const slug = resolveSlugForPath(path);
|
||||
const slug = pathToSlug(path);
|
||||
await engine.deletePage(slug);
|
||||
pagesAffected.push(slug);
|
||||
progress.tick(1, slug);
|
||||
@@ -444,15 +270,12 @@ export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise<
|
||||
progress.finish();
|
||||
}
|
||||
|
||||
// Process renames (updateSlug preserves page_id, chunks, embeddings).
|
||||
// SP-5: both old and new slugs use resolveSlugForPath so a .ts → .ts
|
||||
// rename (code→code), .md → .md (markdown→markdown), or cross-kind rename
|
||||
// all resolve to the right slug shape for each side.
|
||||
// Process renames (updateSlug preserves page_id, chunks, embeddings)
|
||||
if (filtered.renamed.length > 0) {
|
||||
progress.start('sync.renames', filtered.renamed.length);
|
||||
for (const { from, to } of filtered.renamed) {
|
||||
const oldSlug = resolveSlugForPath(from);
|
||||
const newSlug = resolveSlugForPath(to);
|
||||
const oldSlug = pathToSlug(from);
|
||||
const newSlug = pathToSlug(to);
|
||||
try {
|
||||
await engine.updateSlug(oldSlug, newSlug);
|
||||
} catch {
|
||||
@@ -557,10 +380,6 @@ export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise<
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'last_commit', headCommit);
|
||||
await engine.setConfig('sync.last_run', new Date().toISOString());
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'repo_path', repoPath);
|
||||
// v0.20.0 Cathedral II Layer 12: persist the chunker version we just
|
||||
// finished with so the next sync's up_to_date gate respects it. Only
|
||||
// source-scoped syncs track this (see readChunkerVersion for rationale).
|
||||
await writeChunkerVersion(engine, opts.sourceId, String(CHUNKER_VERSION));
|
||||
|
||||
// Log ingest
|
||||
await engine.logIngest({
|
||||
@@ -684,8 +503,6 @@ async function performFullSync(
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'last_commit', headCommit);
|
||||
await engine.setConfig('sync.last_run', new Date().toISOString());
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'repo_path', repoPath);
|
||||
// v0.20.0 Cathedral II Layer 12: persist chunker version for the gate.
|
||||
await writeChunkerVersion(engine, opts.sourceId, String(CHUNKER_VERSION));
|
||||
|
||||
// Full sync doesn't track pagesAffected, so fall back to embed --stale.
|
||||
// Before commit 2: runEmbed is void; use result.imported as best estimate of
|
||||
@@ -724,10 +541,6 @@ export async function runSync(engine: BrainEngine, args: string[]) {
|
||||
const noEmbed = args.includes('--no-embed');
|
||||
const skipFailed = args.includes('--skip-failed');
|
||||
const retryFailed = args.includes('--retry-failed');
|
||||
const syncAll = args.includes('--all');
|
||||
const jsonOut = args.includes('--json');
|
||||
const yesFlag = args.includes('--yes');
|
||||
const strategyArg = args.find((a, i) => args[i - 1] === '--strategy') as SyncOpts['strategy'] | undefined;
|
||||
|
||||
// v0.18.0 Step 5: --source resolves to a sources(id) row. Falls back
|
||||
// to pre-v0.17 global config (sync.repo_path + sync.last_commit) when
|
||||
@@ -739,99 +552,7 @@ export async function runSync(engine: BrainEngine, args: string[]) {
|
||||
sourceId = await resolveSourceId(engine, explicitSource);
|
||||
}
|
||||
|
||||
// v0.19.0 — `sync --all` iterates all registered sources with a
|
||||
// local_path. Sources are the canonical v0.18.0 abstraction: per-source
|
||||
// last_commit, last_sync_at, config.federated flags. Per-source
|
||||
// bookmarks live in the sources table (not ~/.gbrain/config.json),
|
||||
// which is why this path replaced Wintermute's `multi-repo.ts` shim.
|
||||
//
|
||||
// Only sources with a non-null local_path participate. A GitHub-only
|
||||
// source (no checkout) has nothing for `sync` to pull. Sources with
|
||||
// syncEnabled=false in config.jsonb are skipped too.
|
||||
if (syncAll) {
|
||||
const sources = await engine.executeRaw<{ id: string; name: string; local_path: string | null; config: Record<string, unknown> }>(
|
||||
`SELECT id, name, local_path, config FROM sources WHERE local_path IS NOT NULL`,
|
||||
);
|
||||
if (!sources || sources.length === 0) {
|
||||
console.log('No sources with local_path configured. Use `gbrain sources add <id> --path <path>` first.');
|
||||
return;
|
||||
}
|
||||
|
||||
// v0.20.0 Cathedral II Layer 8 D1 — cost preview + ConfirmationRequired
|
||||
// gate. Before kicking off a multi-source sync that may embed tens of
|
||||
// thousands of chunks (real money), walk the sync-diff set(s), sum
|
||||
// tokens, compute USD estimate, and gate:
|
||||
// - TTY + !json + !yes → interactive [y/N] prompt
|
||||
// - non-TTY OR --json OR piped → emit ConfirmationRequired envelope,
|
||||
// exit 2 (reserve 1 for runtime errors)
|
||||
// - --yes → skip prompt entirely
|
||||
// - --dry-run → preview + exit 0
|
||||
// Skipped entirely when --no-embed is set (user already opted out of
|
||||
// the cost and will run `embed --stale` later).
|
||||
if (!noEmbed) {
|
||||
const preview = estimateSyncAllCost(sources);
|
||||
const costUsd = estimateEmbeddingCostUsd(preview.totalTokens);
|
||||
const previewMsg =
|
||||
`sync --all preview: ${preview.totalFiles} files across ${preview.activeSources} source(s), ` +
|
||||
`~${preview.totalTokens.toLocaleString()} tokens, est. $${costUsd.toFixed(2)} on ${EMBEDDING_MODEL}.`;
|
||||
|
||||
if (dryRun) {
|
||||
if (jsonOut) {
|
||||
console.log(JSON.stringify({ status: 'dry_run', preview, costUsd, model: EMBEDDING_MODEL }));
|
||||
} else {
|
||||
console.log(previewMsg);
|
||||
console.log('--dry-run: exit without syncing.');
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
if (!yesFlag) {
|
||||
const isTTY = Boolean(process.stdout.isTTY) && Boolean(process.stdin.isTTY);
|
||||
if (!isTTY || jsonOut) {
|
||||
// Agent-facing path: emit structured envelope, exit 2.
|
||||
const envelope = serializeError(errorFor({
|
||||
class: 'ConfirmationRequired',
|
||||
code: 'cost_preview_requires_yes',
|
||||
message: previewMsg,
|
||||
hint: 'Pass --yes to proceed, or --dry-run to see the preview and exit 0.',
|
||||
}));
|
||||
console.log(JSON.stringify({ error: envelope, preview, costUsd, model: EMBEDDING_MODEL }));
|
||||
process.exit(2);
|
||||
}
|
||||
// Interactive TTY path: prompt [y/N].
|
||||
console.log(previewMsg);
|
||||
const answer = await promptYesNo('Proceed? [y/N] ');
|
||||
if (!answer) {
|
||||
console.log('Cancelled.');
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
for (const src of sources) {
|
||||
const cfg = (src.config || {}) as { syncEnabled?: boolean; strategy?: 'markdown' | 'code' | 'auto' };
|
||||
if (cfg.syncEnabled === false) {
|
||||
console.log(`Skipping disabled source: ${src.name}`);
|
||||
continue;
|
||||
}
|
||||
console.log(`\n--- Syncing source: ${src.name} ---`);
|
||||
const repoOpts: SyncOpts = {
|
||||
repoPath: src.local_path!,
|
||||
dryRun, full, noPull, noEmbed, skipFailed, retryFailed,
|
||||
sourceId: src.id,
|
||||
strategy: cfg.strategy,
|
||||
};
|
||||
try {
|
||||
const result = await performSync(engine, repoOpts);
|
||||
printSyncResult(result);
|
||||
} catch (e: unknown) {
|
||||
console.error(`Error syncing ${src.name}: ${e instanceof Error ? e.message : String(e)}`);
|
||||
}
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
const opts: SyncOpts = { repoPath, dryRun, full, noPull, noEmbed, skipFailed, retryFailed, sourceId, strategy: strategyArg };
|
||||
const opts: SyncOpts = { repoPath, dryRun, full, noPull, noEmbed, skipFailed, retryFailed, sourceId };
|
||||
|
||||
// Bug 9 — --retry-failed: before running normal sync, clear acknowledgment
|
||||
// flags so the sync picks them up as fresh work. The actual re-attempt
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,178 +0,0 @@
|
||||
/**
|
||||
* v0.20.0 Cathedral II Layer 5 (A1) — edge extractor.
|
||||
*
|
||||
* Walks a parsed tree-sitter tree and emits structural edges for:
|
||||
* - `calls` — function/method invocations (f() → f, obj.m() → m, a::b()
|
||||
* on Rust → b). The receiver-type resolution (obj → ClassName) is
|
||||
* explicitly deferred — we store the bare callee token here and rely
|
||||
* on Layer 7 two-pass retrieval + the getCallersOf short-name match
|
||||
* to surface the anchor. This is "best effort precision 80, recall 99":
|
||||
* if you search for "searchKeyword" you get every call site, even the
|
||||
* ones whose receiver we couldn't pin to a class yet.
|
||||
*
|
||||
* Every emitted edge lands in code_edges_symbol (unresolved — to_chunk_id
|
||||
* null) because within-file resolution needs a second pass that matches
|
||||
* callee tokens against chunks' symbol_name_qualified. That resolution is
|
||||
* a future optimization. Layer 5 gets the edges captured at all — that's
|
||||
* the 10x leap over v0.19.0's grep-class retrieval.
|
||||
*
|
||||
* Per-language shipped list: TypeScript, TSX, JavaScript, Python, Ruby,
|
||||
* Go, Rust, Java — the 8 languages covering ~85% of real brain code.
|
||||
* Other languages flow through with zero edges (chunker still works).
|
||||
*/
|
||||
|
||||
import type { SupportedCodeLanguage } from './code.ts';
|
||||
|
||||
export interface ExtractedEdge {
|
||||
/**
|
||||
* Byte offset of the call site in the source. The caller resolves this
|
||||
* to a from_chunk_id by finding the chunk whose (startLine, endLine)
|
||||
* brackets the offset — matches how Layer 6 A3 emits one chunk per
|
||||
* nested method, so each call site falls inside exactly one chunk.
|
||||
*/
|
||||
callSiteByteOffset: number;
|
||||
/** The bare callee token (e.g. 'searchKeyword', 'User.find'). */
|
||||
toSymbol: string;
|
||||
edgeType: 'calls';
|
||||
}
|
||||
|
||||
/**
|
||||
* Per-language call-expression configuration. `callNodeTypes` lists the
|
||||
* AST node types that are call sites in that language. `calleeFieldName`
|
||||
* optionally names the child field that holds the callee expression;
|
||||
* when absent, the call-site text itself is scanned for the identifier.
|
||||
*/
|
||||
interface CallConfig {
|
||||
callNodeTypes: Set<string>;
|
||||
calleeFieldName?: string;
|
||||
}
|
||||
|
||||
const CALL_CONFIG: Partial<Record<SupportedCodeLanguage, CallConfig>> = {
|
||||
typescript: { callNodeTypes: new Set(['call_expression']), calleeFieldName: 'function' },
|
||||
tsx: { callNodeTypes: new Set(['call_expression']), calleeFieldName: 'function' },
|
||||
javascript: { callNodeTypes: new Set(['call_expression']), calleeFieldName: 'function' },
|
||||
python: { callNodeTypes: new Set(['call']), calleeFieldName: 'function' },
|
||||
ruby: { callNodeTypes: new Set(['call', 'method_call']), calleeFieldName: 'method' },
|
||||
go: { callNodeTypes: new Set(['call_expression']), calleeFieldName: 'function' },
|
||||
rust: { callNodeTypes: new Set(['call_expression', 'method_call_expression']), calleeFieldName: 'function' },
|
||||
java: { callNodeTypes: new Set(['method_invocation']), calleeFieldName: 'name' },
|
||||
};
|
||||
|
||||
/**
|
||||
* Extract the callee's bare identifier name from a call-site node. For
|
||||
* `obj.method(args)` returns "method". For `namespace::func(args)`
|
||||
* returns "func". For bare `func(args)` returns "func". When the callee
|
||||
* is itself a complex expression (arrow-chain, indexed access) we return
|
||||
* null to skip the edge.
|
||||
*/
|
||||
function extractCalleeName(node: any, cfg: CallConfig): string | null {
|
||||
const callee = cfg.calleeFieldName ? node.childForFieldName(cfg.calleeFieldName) : null;
|
||||
if (!callee) return null;
|
||||
|
||||
// Unwrap common wrappers until we hit an identifier-shaped node.
|
||||
let cur = callee;
|
||||
for (let i = 0; i < 6 && cur; i++) {
|
||||
if (!cur.type) return null;
|
||||
if (
|
||||
cur.type === 'identifier' ||
|
||||
cur.type === 'property_identifier' ||
|
||||
cur.type === 'field_identifier' ||
|
||||
cur.type === 'scoped_identifier' ||
|
||||
cur.type === 'shorthand_property_identifier' ||
|
||||
cur.type === 'simple_identifier' ||
|
||||
cur.type === 'type_identifier' ||
|
||||
cur.type === 'constant'
|
||||
) {
|
||||
const text = cur.text as string;
|
||||
// For scoped names like `std::io::println`, keep the final
|
||||
// segment only — the edge-identity match is by short name.
|
||||
const lastSeg = text.split(/[:.]+/).pop() ?? text;
|
||||
return sanitizeIdent(lastSeg);
|
||||
}
|
||||
// member_expression / field_expression: callee is last member.
|
||||
if (cur.type === 'member_expression' || cur.type === 'field_expression') {
|
||||
const prop = cur.childForFieldName('property') ?? cur.childForFieldName('field');
|
||||
if (prop) { cur = prop; continue; }
|
||||
return null;
|
||||
}
|
||||
// scoped_call_expression (Rust): recurse into function.
|
||||
if (cur.type === 'scoped_call_expression' || cur.type === 'scoped_identifier') {
|
||||
const name = cur.childForFieldName('name');
|
||||
if (name) { cur = name; continue; }
|
||||
return null;
|
||||
}
|
||||
// Fallback: read the node text and take the last identifier-looking token.
|
||||
const m = (cur.text as string).match(/([A-Za-z_][A-Za-z0-9_]*)\s*$/);
|
||||
return m ? sanitizeIdent(m[1]!) : null;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function sanitizeIdent(s: string): string | null {
|
||||
const m = s.match(/^[A-Za-z_][A-Za-z0-9_]*$/);
|
||||
return m ? s : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Walk the tree and collect every call site that matches the language's
|
||||
* call-expression config. Returns a flat list; the caller maps byte
|
||||
* offsets to chunk IDs.
|
||||
*/
|
||||
export function extractCallEdges(tree: any, language: SupportedCodeLanguage): ExtractedEdge[] {
|
||||
const cfg = CALL_CONFIG[language];
|
||||
if (!cfg) return [];
|
||||
const out: ExtractedEdge[] = [];
|
||||
|
||||
// Iterative traversal (tree-sitter trees can be deep; recursion risks
|
||||
// stack overflow on generated code). Uses TreeCursor when available,
|
||||
// else falls back to namedChildren iteration.
|
||||
const root = tree.rootNode;
|
||||
const stack: any[] = [root];
|
||||
while (stack.length > 0) {
|
||||
const node = stack.pop();
|
||||
if (!node) continue;
|
||||
if (cfg.callNodeTypes.has(node.type)) {
|
||||
const callee = extractCalleeName(node, cfg);
|
||||
if (callee) {
|
||||
out.push({
|
||||
callSiteByteOffset: node.startIndex,
|
||||
toSymbol: callee,
|
||||
edgeType: 'calls',
|
||||
});
|
||||
}
|
||||
}
|
||||
// Push children for further traversal.
|
||||
for (const child of node.namedChildren) stack.push(child);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* Map byte offset → chunk index by (startLine, endLine) range. Returns
|
||||
* the innermost chunk containing the offset, which for A3 nested-chunk
|
||||
* emission is the deepest method chunk. Falls back to any chunk when
|
||||
* offset lookup misses (rare — root node always covers all offsets).
|
||||
*/
|
||||
export function findChunkForOffset(
|
||||
byteOffset: number,
|
||||
source: string,
|
||||
chunks: Array<{ startLine: number; endLine: number }>,
|
||||
): number | null {
|
||||
// Compute line number of byteOffset by counting newlines up to it.
|
||||
// Cache: the chunker already knows startLine/endLine per chunk, so
|
||||
// a naive line lookup here is fine on a per-file basis.
|
||||
let line = 1;
|
||||
for (let i = 0; i < byteOffset && i < source.length; i++) {
|
||||
if (source.charCodeAt(i) === 10) line++;
|
||||
}
|
||||
// Prefer innermost (smallest line span) chunk containing the line.
|
||||
let best: number | null = null;
|
||||
let bestSpan = Infinity;
|
||||
for (let i = 0; i < chunks.length; i++) {
|
||||
const c = chunks[i]!;
|
||||
if (line < c.startLine || line > c.endLine) continue;
|
||||
const span = c.endLine - c.startLine;
|
||||
if (span < bestSpan) { bestSpan = span; best = i; }
|
||||
}
|
||||
return best;
|
||||
}
|
||||
@@ -1,109 +0,0 @@
|
||||
/**
|
||||
* v0.20.0 Cathedral II Layer 5 — qualified symbol identity.
|
||||
*
|
||||
* Edge identity across languages needs a shared notion of "the Admin
|
||||
* controller's render method" vs "the ViewHelper module's render method".
|
||||
* Raw symbol_name ('render') is too ambiguous; a raw parent_symbol_path
|
||||
* (['Admin', 'UsersController', 'render']) needs a language-aware join
|
||||
* to match the conventions the ecosystem uses.
|
||||
*
|
||||
* This module builds qualified names from the pieces the chunker already
|
||||
* collects:
|
||||
* - language (from detectCodeLanguage)
|
||||
* - symbolType (from normalizeSymbolType: 'function' | 'method' | 'class' | ...)
|
||||
* - symbolName (from extractSymbolName)
|
||||
* - parentSymbolPath (from Layer 6 A3 emitNestedScoped)
|
||||
*
|
||||
* Output is a single TEXT value stored in content_chunks.symbol_name_qualified
|
||||
* and used as the edge-identity key. Examples:
|
||||
*
|
||||
* Ruby: Admin::UsersController#render (instance method)
|
||||
* Admin::UsersController.find_all (singleton method)
|
||||
* Python: admin.users_controller.UsersController.render
|
||||
* TS/JS: BrainEngine.searchKeyword (class method)
|
||||
* parseInput (standalone fn)
|
||||
* Go: users.Render (package-qualified fn)
|
||||
* (*UsersController).Render (method on pointer receiver)
|
||||
* Rust: users::UsersController::render (impl block scoped)
|
||||
* Java: com.acme.admin.UsersController.render
|
||||
*
|
||||
* The per-language delimiters + instance/singleton distinction are
|
||||
* codified in LANG_CONFIG below. When a language is unknown or symbol
|
||||
* name is missing, we return null (edge extractor skips the row).
|
||||
*/
|
||||
|
||||
import type { SupportedCodeLanguage } from './code.ts';
|
||||
|
||||
interface QualifiedNameConfig {
|
||||
/** Delimiter between namespace segments (e.g. '::' for Ruby, '.' for Python). */
|
||||
segmentDelim: string;
|
||||
/** Delimiter between class and instance method (Ruby: '#'). */
|
||||
methodDelim?: string;
|
||||
/** Delimiter between class and singleton / static method. */
|
||||
staticDelim?: string;
|
||||
/**
|
||||
* When true, treat "method" symbol types as instance methods and use
|
||||
* `methodDelim`; otherwise fall back to `segmentDelim`.
|
||||
*/
|
||||
distinguishInstanceMethods?: boolean;
|
||||
}
|
||||
|
||||
const LANG_CONFIG: Partial<Record<SupportedCodeLanguage, QualifiedNameConfig>> = {
|
||||
typescript: { segmentDelim: '.' },
|
||||
tsx: { segmentDelim: '.' },
|
||||
javascript: { segmentDelim: '.' },
|
||||
python: { segmentDelim: '.' },
|
||||
go: { segmentDelim: '.' },
|
||||
rust: { segmentDelim: '::' },
|
||||
java: { segmentDelim: '.' },
|
||||
ruby: {
|
||||
segmentDelim: '::',
|
||||
methodDelim: '#',
|
||||
staticDelim: '.',
|
||||
distinguishInstanceMethods: true,
|
||||
},
|
||||
};
|
||||
|
||||
/**
|
||||
* Build a qualified name from the chunker's per-chunk metadata. Returns
|
||||
* null when the inputs aren't enough to form a usable identity — callers
|
||||
* skip those chunks for edge extraction.
|
||||
*/
|
||||
export function buildQualifiedName(input: {
|
||||
language: SupportedCodeLanguage;
|
||||
symbolName: string | null;
|
||||
symbolType: string;
|
||||
parentSymbolPath: string[];
|
||||
}): string | null {
|
||||
if (!input.symbolName) return null;
|
||||
const cfg = LANG_CONFIG[input.language];
|
||||
if (!cfg) {
|
||||
// Unknown language — at least return the raw symbol name so edge
|
||||
// matching doesn't lose it entirely. Not ideal for disambiguation
|
||||
// but better than dropping the edge on the floor.
|
||||
return input.parentSymbolPath.length > 0
|
||||
? `${input.parentSymbolPath.join('.')}.${input.symbolName}`
|
||||
: input.symbolName;
|
||||
}
|
||||
|
||||
if (input.parentSymbolPath.length === 0) return input.symbolName;
|
||||
|
||||
const parents = input.parentSymbolPath.join(cfg.segmentDelim);
|
||||
|
||||
if (cfg.distinguishInstanceMethods && input.symbolType === 'function') {
|
||||
// Ruby: instance method — Class#method. We can't tell `def self.m` from
|
||||
// `def m` at chunk level without inspecting the node type; the chunker
|
||||
// normalizes both to 'function', so we default to instance-method form
|
||||
// and accept that edge-identity for Ruby singletons will collide with
|
||||
// instance methods of the same name in the same class. In practice
|
||||
// this is rare and the parentSymbolPath disambiguates most cases.
|
||||
return `${parents}${cfg.methodDelim ?? '#'}${input.symbolName}`;
|
||||
}
|
||||
|
||||
return `${parents}${cfg.segmentDelim}${input.symbolName}`;
|
||||
}
|
||||
|
||||
/** Exported for unit testing the lang-config table directly. */
|
||||
export const __testing = {
|
||||
LANG_CONFIG,
|
||||
};
|
||||
@@ -87,10 +87,6 @@ export function toEngineConfig(config: GBrainConfig): EngineConfig {
|
||||
}
|
||||
|
||||
export function configDir(): string {
|
||||
// Allow override for tests, Docker, and multi-tenant deployments.
|
||||
// Matches the `GBRAIN_AUDIT_DIR` convention in src/core/minions/handlers/shell-audit.ts.
|
||||
const override = process.env.GBRAIN_HOME;
|
||||
if (override && override.trim()) return join(override, '.gbrain');
|
||||
return join(homedir(), '.gbrain');
|
||||
}
|
||||
|
||||
|
||||
@@ -105,20 +105,3 @@ function sleep(ms: number): Promise<void> {
|
||||
}
|
||||
|
||||
export { MODEL as EMBEDDING_MODEL, DIMENSIONS as EMBEDDING_DIMENSIONS };
|
||||
|
||||
/**
|
||||
* v0.20.0 Cathedral II Layer 8 (D1): USD cost per 1k tokens for
|
||||
* text-embedding-3-large. Used by `gbrain sync --all` cost preview and
|
||||
* the reindex-code backfill command to surface expected spend before
|
||||
* the agent/user accepts an expensive operation.
|
||||
*
|
||||
* Value: $0.00013 / 1k tokens as of 2026. Update when OpenAI changes
|
||||
* pricing. Single source of truth — every cost-preview surface reads
|
||||
* this constant, so a pricing change is a one-line edit.
|
||||
*/
|
||||
export const EMBEDDING_COST_PER_1K_TOKENS = 0.00013;
|
||||
|
||||
/** Compute USD cost estimate for embedding `tokens` at current model rate. */
|
||||
export function estimateEmbeddingCostUsd(tokens: number): number {
|
||||
return (tokens / 1000) * EMBEDDING_COST_PER_1K_TOKENS;
|
||||
}
|
||||
|
||||
@@ -9,7 +9,6 @@ import type {
|
||||
BrainStats, BrainHealth,
|
||||
IngestLogEntry, IngestLogInput,
|
||||
EngineConfig,
|
||||
CodeEdgeInput, CodeEdgeResult,
|
||||
} from './types.ts';
|
||||
|
||||
/** Input row for addLinksBatch. Optional fields default to '' (matches NOT NULL DDL). */
|
||||
@@ -270,63 +269,4 @@ export interface BrainEngine {
|
||||
|
||||
// Raw SQL (for Minions job queue and other internal modules)
|
||||
executeRaw<T = Record<string, unknown>>(sql: string, params?: unknown[]): Promise<T[]>;
|
||||
|
||||
// ============================================================
|
||||
// v0.20.0 Cathedral II: code edges (Layer 5 populates, Layer 7 consumes)
|
||||
// ============================================================
|
||||
/**
|
||||
* Bulk-insert code edges. Resolved edges (to_chunk_id set) land in
|
||||
* code_edges_chunk; unresolved refs (to_chunk_id null, to_symbol_qualified
|
||||
* set) land in code_edges_symbol. ON CONFLICT DO NOTHING handles idempotency.
|
||||
* Returns count of rows actually inserted.
|
||||
*/
|
||||
addCodeEdges(edges: CodeEdgeInput[]): Promise<number>;
|
||||
|
||||
/**
|
||||
* Delete all code edges involving these chunk IDs, in BOTH directions, across
|
||||
* both code_edges_chunk and code_edges_symbol. Called by importCodeFile on
|
||||
* per-chunk invalidation (codex SP-2): when a chunk's text changed, stale
|
||||
* inbound edges from other pages pointing at the old symbol must wipe before
|
||||
* new edges write.
|
||||
*/
|
||||
deleteCodeEdgesForChunks(chunkIds: number[]): Promise<void>;
|
||||
|
||||
/**
|
||||
* "Who calls this symbol?" Returns UNION of code_edges_chunk +
|
||||
* code_edges_symbol matching `to_symbol_qualified = qualifiedName`.
|
||||
* Source scoping (codex SP-3): if opts.sourceId is set, filter by the
|
||||
* anchor chunk's source; if opts.allSources, ignore scoping.
|
||||
*/
|
||||
getCallersOf(
|
||||
qualifiedName: string,
|
||||
opts?: { sourceId?: string; allSources?: boolean; limit?: number },
|
||||
): Promise<CodeEdgeResult[]>;
|
||||
|
||||
/**
|
||||
* "What does this symbol call?" Returns edges from chunks whose
|
||||
* from_symbol_qualified = qualifiedName. Same source-scoping semantics
|
||||
* as getCallersOf.
|
||||
*/
|
||||
getCalleesOf(
|
||||
qualifiedName: string,
|
||||
opts?: { sourceId?: string; allSources?: boolean; limit?: number },
|
||||
): Promise<CodeEdgeResult[]>;
|
||||
|
||||
/**
|
||||
* All edges touching a chunk in the given direction. Used by A2 two-pass
|
||||
* retrieval to expand from anchor chunks. direction='in' returns edges
|
||||
* pointing AT the chunk; 'out' returns edges FROM it; 'both' unions.
|
||||
*/
|
||||
getEdgesByChunk(
|
||||
chunkId: number,
|
||||
opts?: { direction?: 'in' | 'out' | 'both'; edgeType?: string; limit?: number },
|
||||
): Promise<CodeEdgeResult[]>;
|
||||
|
||||
/**
|
||||
* Chunk-grain keyword search. Ranks by content_chunks.search_vector
|
||||
* without the dedup-to-page pass that searchKeyword applies. Consumed
|
||||
* by A2 two-pass retrieval as its anchor source. Most callers should
|
||||
* prefer searchKeyword (external contract: page-grain best-chunk-per-page).
|
||||
*/
|
||||
searchKeywordChunks(query: string, opts?: SearchOpts): Promise<SearchResult[]>;
|
||||
}
|
||||
|
||||
@@ -1,93 +0,0 @@
|
||||
/**
|
||||
* Structured error envelope for agent-consumable failures.
|
||||
*
|
||||
* Shape matches `CycleReport.PhaseResult.error` from v0.17.0 so the agent
|
||||
* surface is consistent across `gbrain dream`, `sync --all`, `code-def`,
|
||||
* `code-refs`, `repos`, and `importCodeFile`.
|
||||
*
|
||||
* Agents consuming gbrain via CLI+JSON (OpenClaw and similar) need to
|
||||
* distinguish retryable from fatal, user-config from programmer errors,
|
||||
* and get a hint to recover. Raw Error().message strings lose that signal.
|
||||
*/
|
||||
|
||||
export interface StructuredError {
|
||||
/** Short error class name, e.g. "ConfirmationRequired", "FileTooLarge". */
|
||||
class: string;
|
||||
/** Stable machine-readable code, snake_case. e.g. "cost_preview_requires_yes". */
|
||||
code: string;
|
||||
/** Human-readable message. One sentence. */
|
||||
message: string;
|
||||
/** Optional actionable hint. e.g. "Pass --yes to proceed". */
|
||||
hint?: string;
|
||||
/** Optional link to docs/runbook. */
|
||||
docs_url?: string;
|
||||
}
|
||||
|
||||
export interface BuildErrorInput {
|
||||
class: string;
|
||||
code: string;
|
||||
message: string;
|
||||
hint?: string;
|
||||
docs_url?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a structured error envelope. Prefer this over throw new Error()
|
||||
* at any new v0.18.0 surface (repos, code-def, code-refs, sync --all,
|
||||
* importCodeFile, doctor --chunker-debug).
|
||||
*/
|
||||
export function buildError(input: BuildErrorInput): StructuredError {
|
||||
const e: StructuredError = {
|
||||
class: input.class,
|
||||
code: input.code,
|
||||
message: input.message,
|
||||
};
|
||||
if (input.hint) e.hint = input.hint;
|
||||
if (input.docs_url) e.docs_url = input.docs_url;
|
||||
return e;
|
||||
}
|
||||
|
||||
/**
|
||||
* An Error subclass that carries a StructuredError envelope.
|
||||
* Agents catch this, extract `.envelope`, and print `{error: envelope}` as JSON.
|
||||
* Humans see the plain message via Error.message.
|
||||
*/
|
||||
export class StructuredAgentError extends Error {
|
||||
readonly envelope: StructuredError;
|
||||
|
||||
constructor(envelope: StructuredError) {
|
||||
const hintSuffix = envelope.hint ? ` (${envelope.hint})` : '';
|
||||
super(`${envelope.class}: ${envelope.message}${hintSuffix}`);
|
||||
this.name = envelope.class;
|
||||
this.envelope = envelope;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Helper to construct-and-throw in one call.
|
||||
* Usage: throw errorFor({ class: 'FileTooLarge', code: 'file_too_large', message: '...' });
|
||||
*/
|
||||
export function errorFor(input: BuildErrorInput): StructuredAgentError {
|
||||
return new StructuredAgentError(buildError(input));
|
||||
}
|
||||
|
||||
/**
|
||||
* Serialize an error envelope or unknown throwable for JSON output.
|
||||
* If the value is a StructuredAgentError, uses its structured envelope.
|
||||
* Otherwise falls back to a generic {class: 'Error', code: 'unknown', message}.
|
||||
*/
|
||||
export function serializeError(value: unknown): StructuredError {
|
||||
if (value instanceof StructuredAgentError) return value.envelope;
|
||||
if (value instanceof Error) {
|
||||
return buildError({
|
||||
class: value.name || 'Error',
|
||||
code: 'unknown',
|
||||
message: value.message,
|
||||
});
|
||||
}
|
||||
return buildError({
|
||||
class: 'Error',
|
||||
code: 'unknown',
|
||||
message: String(value),
|
||||
});
|
||||
}
|
||||
+1
-359
@@ -1,144 +1,12 @@
|
||||
import { readFileSync, statSync, lstatSync } from 'fs';
|
||||
import { basename } from 'path';
|
||||
import { createHash } from 'crypto';
|
||||
import { marked } from 'marked';
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
import { parseMarkdown } from './markdown.ts';
|
||||
import { chunkText } from './chunkers/recursive.ts';
|
||||
import { chunkCodeText, chunkCodeTextFull, detectCodeLanguage, CHUNKER_VERSION } from './chunkers/code.ts';
|
||||
import { findChunkForOffset } from './chunkers/edge-extractor.ts';
|
||||
import { extractCodeRefs } from './link-extraction.ts';
|
||||
import { embedBatch } from './embedding.ts';
|
||||
import { slugifyPath, slugifyCodePath, isCodeFilePath } from './sync.ts';
|
||||
import { slugifyPath } from './sync.ts';
|
||||
import type { ChunkInput, PageType } from './types.ts';
|
||||
|
||||
/**
|
||||
* v0.20.0 Cathedral II Layer 8 D2 — markdown fence extraction helper.
|
||||
*
|
||||
* Roughly 40% of gbrain's brain is docs/guides/architecture notes with
|
||||
* substantial inline code. In v0.19.0 those fenced code blocks chunk as
|
||||
* prose, so querying "how do we import from engine" ranks paragraphs
|
||||
* ABOUT the import above the actual import example. D2 walks the marked
|
||||
* lexer tokens, extracts each `{type:'code', lang, text}` fence with a
|
||||
* known language tag, chunks the content via the code chunker (so TS
|
||||
* fence gets TS-aware chunking), and persists those as extra chunks on
|
||||
* the parent markdown page with `chunk_source='fenced_code'`.
|
||||
*
|
||||
* Fence tag → pseudo-extension map. We don't need a full file extension
|
||||
* because chunkCodeText only calls detectCodeLanguage to pick a grammar;
|
||||
* a recognized extension gets the right grammar loaded, that's all.
|
||||
* Unknown tags return null → fence is skipped (no synthetic chunk).
|
||||
*/
|
||||
const FENCE_TAG_TO_PSEUDO_PATH: Record<string, string> = {
|
||||
ts: 'fence.ts', typescript: 'fence.ts',
|
||||
tsx: 'fence.tsx',
|
||||
js: 'fence.js', javascript: 'fence.js',
|
||||
jsx: 'fence.jsx',
|
||||
py: 'fence.py', python: 'fence.py',
|
||||
rb: 'fence.rb', ruby: 'fence.rb',
|
||||
go: 'fence.go', golang: 'fence.go',
|
||||
rs: 'fence.rs', rust: 'fence.rs',
|
||||
java: 'fence.java',
|
||||
'c#': 'fence.cs', cs: 'fence.cs', csharp: 'fence.cs',
|
||||
cpp: 'fence.cpp', 'c++': 'fence.cpp',
|
||||
c: 'fence.c',
|
||||
php: 'fence.php',
|
||||
swift: 'fence.swift',
|
||||
kt: 'fence.kt', kotlin: 'fence.kt',
|
||||
scala: 'fence.scala',
|
||||
lua: 'fence.lua',
|
||||
ex: 'fence.ex', elixir: 'fence.ex',
|
||||
elm: 'fence.elm',
|
||||
ml: 'fence.ml', ocaml: 'fence.ml',
|
||||
dart: 'fence.dart',
|
||||
zig: 'fence.zig',
|
||||
sol: 'fence.sol', solidity: 'fence.sol',
|
||||
sh: 'fence.sh', bash: 'fence.sh', shell: 'fence.sh', zsh: 'fence.sh',
|
||||
css: 'fence.css',
|
||||
html: 'fence.html',
|
||||
vue: 'fence.vue',
|
||||
json: 'fence.json',
|
||||
yaml: 'fence.yaml', yml: 'fence.yaml',
|
||||
toml: 'fence.toml',
|
||||
};
|
||||
|
||||
function fenceTagToPseudoPath(lang: string | undefined): string | null {
|
||||
if (!lang) return null;
|
||||
return FENCE_TAG_TO_PSEUDO_PATH[lang.toLowerCase().trim()] ?? null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Maximum code fences we'll extract from a single markdown page. Fence-bomb
|
||||
* DOS defense — a malicious markdown file with 10K ```ts blocks could
|
||||
* generate 10K chunks × embedding API calls. Override per-page via the
|
||||
* `GBRAIN_MAX_FENCES_PER_PAGE` env var if docs-heavy brains legitimately
|
||||
* exceed 100 fences on a single page.
|
||||
*/
|
||||
const MAX_FENCES_PER_PAGE = Number.parseInt(process.env.GBRAIN_MAX_FENCES_PER_PAGE || '100', 10);
|
||||
|
||||
/**
|
||||
* Walk the marked lexer output and extract recognizable code fences.
|
||||
* Returns one ChunkInput per fence whose language tag maps to a grammar
|
||||
* the chunker understands. Unknown tags + empty fences are skipped.
|
||||
* Per-fence try/catch: one malformed fence doesn't abort the page import.
|
||||
*/
|
||||
async function extractFencedChunks(
|
||||
markdown: string,
|
||||
startChunkIndex: number,
|
||||
): Promise<ChunkInput[]> {
|
||||
const out: ChunkInput[] = [];
|
||||
let tokens: ReturnType<typeof marked.lexer>;
|
||||
try {
|
||||
tokens = marked.lexer(markdown);
|
||||
} catch {
|
||||
// marked's lexer errors on truly malformed input — bail, keep the
|
||||
// markdown-level chunks that came from compiled_truth.
|
||||
return out;
|
||||
}
|
||||
|
||||
let fencesSeen = 0;
|
||||
let indexOffset = 0;
|
||||
for (const tok of tokens) {
|
||||
if (tok.type !== 'code') continue;
|
||||
const code = tok as { type: 'code'; lang?: string; text?: string };
|
||||
const text = (code.text ?? '').trim();
|
||||
if (!text) continue;
|
||||
if (fencesSeen >= MAX_FENCES_PER_PAGE) {
|
||||
console.warn(
|
||||
`[gbrain] markdown fence cap hit (${MAX_FENCES_PER_PAGE} fences/page); skipping additional fences. ` +
|
||||
`Override via GBRAIN_MAX_FENCES_PER_PAGE env var.`,
|
||||
);
|
||||
break;
|
||||
}
|
||||
fencesSeen++;
|
||||
const pseudoPath = fenceTagToPseudoPath(code.lang);
|
||||
if (!pseudoPath) continue; // unknown or missing lang tag → prose fallback
|
||||
const lang = detectCodeLanguage(pseudoPath);
|
||||
if (!lang) continue;
|
||||
try {
|
||||
const chunks = await chunkCodeText(text, pseudoPath);
|
||||
for (const c of chunks) {
|
||||
out.push({
|
||||
chunk_index: startChunkIndex + indexOffset++,
|
||||
chunk_text: c.text,
|
||||
chunk_source: 'fenced_code',
|
||||
language: c.metadata.language,
|
||||
symbol_name: c.metadata.symbolName || undefined,
|
||||
symbol_type: c.metadata.symbolType,
|
||||
start_line: c.metadata.startLine,
|
||||
end_line: c.metadata.endLine,
|
||||
});
|
||||
}
|
||||
} catch (e: unknown) {
|
||||
// One fence failing shouldn't sink the page. Log + continue.
|
||||
console.warn(
|
||||
`[gbrain] fence extraction failed for lang=${code.lang}: ${e instanceof Error ? e.message : String(e)}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* The parsed page metadata returned by importFromContent. Callers (specifically
|
||||
* the put_page operation handler running auto-link post-hook) can reuse this to
|
||||
@@ -241,17 +109,6 @@ export async function importFromContent(
|
||||
}
|
||||
}
|
||||
|
||||
// v0.20.0 Cathedral II Layer 8 D2 — extract fenced code blocks from
|
||||
// compiled_truth as first-class code chunks. A markdown page like
|
||||
// `docs/hybrid-search.md` with embedded TypeScript examples now ranks
|
||||
// the TS fence directly in code-aware queries instead of burying it
|
||||
// inside prose. Fences that carry an unrecognized lang tag (or no tag)
|
||||
// fall through — the prose chunker above already chunked them as text.
|
||||
if (parsed.compiled_truth.trim()) {
|
||||
const fenceChunks = await extractFencedChunks(parsed.compiled_truth, chunks.length);
|
||||
chunks.push(...fenceChunks);
|
||||
}
|
||||
|
||||
// Embed BEFORE the transaction (external API call)
|
||||
if (!opts.noEmbed && chunks.length > 0) {
|
||||
try {
|
||||
@@ -294,32 +151,6 @@ export async function importFromContent(
|
||||
// Content is empty — delete stale chunks so they don't ghost in search results
|
||||
await tx.deleteChunks(slug);
|
||||
}
|
||||
|
||||
// v0.19.0 E1 — doc↔impl linking: if this markdown page cites code paths
|
||||
// (e.g. 'src/core/sync.ts:42'), create bidirectional edges to the code
|
||||
// page. addLink throws when either endpoint is missing (master tightened
|
||||
// this in v0.18.x), so we wrap each pair in try/catch — guides imported
|
||||
// before their code repo syncs are common, and the missing edges land
|
||||
// later via `gbrain reconcile-links` (Layer 8 D3, v0.21.0).
|
||||
const codeRefs = extractCodeRefs(parsed.compiled_truth + '\n' + (parsed.timeline || ''));
|
||||
for (const ref of codeRefs) {
|
||||
const codeSlug = slugifyCodePath(ref.path);
|
||||
// Forward: markdown guide → code page (this guide documents that code)
|
||||
try {
|
||||
await tx.addLink(
|
||||
slug, codeSlug,
|
||||
ref.line ? `cited at ${ref.path}:${ref.line}` : ref.path,
|
||||
'documents', 'markdown', slug, 'compiled_truth',
|
||||
);
|
||||
} catch { /* code page not yet imported — reconcile-links will catch it */ }
|
||||
// Reverse: code page → markdown guide (this code is documented by the guide)
|
||||
try {
|
||||
await tx.addLink(
|
||||
codeSlug, slug,
|
||||
ref.path, 'documented_by', 'markdown', slug, 'compiled_truth',
|
||||
);
|
||||
} catch { /* same reason — silent skip */ }
|
||||
}
|
||||
});
|
||||
|
||||
return { slug, status: 'imported', chunks: chunks.length, parsedPage };
|
||||
@@ -353,12 +184,6 @@ export async function importFromFile(
|
||||
}
|
||||
|
||||
const content = readFileSync(filePath, 'utf-8');
|
||||
|
||||
// Route code files through the code import path
|
||||
if (isCodeFilePath(relativePath)) {
|
||||
return importCodeFile(engine, relativePath, content, opts);
|
||||
}
|
||||
|
||||
const parsed = parseMarkdown(content, relativePath);
|
||||
|
||||
// Enforce path-authoritative slug. parseMarkdown prefers frontmatter.slug over
|
||||
@@ -381,189 +206,6 @@ export async function importFromFile(
|
||||
return importFromContent(engine, expectedSlug, content, opts);
|
||||
}
|
||||
|
||||
/**
|
||||
* Import a code file. Bypasses markdown parsing entirely.
|
||||
* Uses tree-sitter code chunker for semantic splitting.
|
||||
* Page type is 'code', slug includes file extension.
|
||||
*/
|
||||
export async function importCodeFile(
|
||||
engine: BrainEngine,
|
||||
relativePath: string,
|
||||
content: string,
|
||||
opts: { noEmbed?: boolean; force?: boolean } = {},
|
||||
): Promise<ImportResult> {
|
||||
const slug = slugifyCodePath(relativePath);
|
||||
const lang = detectCodeLanguage(relativePath) || 'unknown';
|
||||
const title = `${relativePath} (${lang})`;
|
||||
|
||||
const byteLength = Buffer.byteLength(content, 'utf-8');
|
||||
if (byteLength > MAX_FILE_SIZE) {
|
||||
return { slug, status: 'skipped', chunks: 0, error: `Code file too large (${byteLength} bytes)` };
|
||||
}
|
||||
|
||||
// Hash for idempotency. CHUNKER_VERSION is folded in so chunker shape
|
||||
// changes across releases force clean re-chunks without sync --force.
|
||||
const hash = createHash('sha256')
|
||||
.update(JSON.stringify({ title, type: 'code', content, lang, chunker_version: CHUNKER_VERSION }))
|
||||
.digest('hex');
|
||||
|
||||
const existing = await engine.getPage(slug);
|
||||
if (!opts.force && existing?.content_hash === hash) {
|
||||
return { slug, status: 'skipped', chunks: 0 };
|
||||
}
|
||||
|
||||
// Chunk via tree-sitter code chunker. The chunker returns per-chunk
|
||||
// metadata (symbol_name, symbol_type, language, start_line, end_line)
|
||||
// which we persist as columns so the v0.19.0 query --lang + code-def +
|
||||
// code-refs surfaces can filter without parsing chunk_text.
|
||||
// v0.20.0 Cathedral II Layer 6 (A3): parent_symbol_path flows through
|
||||
// from the chunker (nested methods carry ['ClassName'] etc.) so the
|
||||
// chunk-grain FTS trigger picks up scope for ranking and downstream
|
||||
// Layer 5 edge resolution can use scope-qualified identity.
|
||||
const { chunks: codeChunks, edges: extractedEdges } = await chunkCodeTextFull(content, relativePath);
|
||||
const chunks: ChunkInput[] = codeChunks.map((c, i) => ({
|
||||
chunk_index: i,
|
||||
chunk_text: c.text,
|
||||
chunk_source: 'compiled_truth' as const,
|
||||
language: c.metadata.language,
|
||||
symbol_name: c.metadata.symbolName || undefined,
|
||||
symbol_type: c.metadata.symbolType,
|
||||
start_line: c.metadata.startLine,
|
||||
end_line: c.metadata.endLine,
|
||||
parent_symbol_path:
|
||||
c.metadata.parentSymbolPath && c.metadata.parentSymbolPath.length > 0
|
||||
? c.metadata.parentSymbolPath
|
||||
: undefined,
|
||||
symbol_name_qualified: c.metadata.symbolNameQualified || undefined,
|
||||
}));
|
||||
|
||||
// v0.19.0 E2 — incremental chunking. Embedding calls dominate the cost
|
||||
// of a sync; re-embedding unchanged chunks wastes money without
|
||||
// improving retrieval. Look up existing chunks by slug and, for any
|
||||
// whose chunk_text exactly matches the new chunk at the same index,
|
||||
// reuse the existing embedding. Only truly new/changed chunks hit the
|
||||
// OpenAI API. Order matters: our chunk_index is semantic (tree-sitter
|
||||
// order), so a matching (chunk_index, text_hash) means a verbatim
|
||||
// preserved symbol.
|
||||
const existingChunks = existing ? await engine.getChunks(slug) : [];
|
||||
const existingByKey = new Map<string, typeof existingChunks[number]>();
|
||||
for (const ec of existingChunks) {
|
||||
existingByKey.set(`${ec.chunk_index}:${ec.chunk_text}`, ec);
|
||||
}
|
||||
const needsEmbedIndexes: number[] = [];
|
||||
for (let i = 0; i < chunks.length; i++) {
|
||||
const key = `${chunks[i]!.chunk_index}:${chunks[i]!.chunk_text}`;
|
||||
const matched = existingByKey.get(key);
|
||||
if (matched && matched.embedding) {
|
||||
// Reuse the existing embedding verbatim. No API call, no cost.
|
||||
chunks[i]!.embedding = matched.embedding as Float32Array;
|
||||
chunks[i]!.token_count = matched.token_count ?? undefined;
|
||||
} else {
|
||||
needsEmbedIndexes.push(i);
|
||||
}
|
||||
}
|
||||
|
||||
// Embed only the new/changed chunks.
|
||||
if (!opts.noEmbed && needsEmbedIndexes.length > 0) {
|
||||
try {
|
||||
const textsToEmbed = needsEmbedIndexes.map((i) => chunks[i]!.chunk_text);
|
||||
const embeddings = await embedBatch(textsToEmbed);
|
||||
for (let j = 0; j < needsEmbedIndexes.length; j++) {
|
||||
const i = needsEmbedIndexes[j]!;
|
||||
chunks[i]!.embedding = embeddings[j]!;
|
||||
chunks[i]!.token_count = Math.ceil(chunks[i]!.chunk_text.length / 4);
|
||||
}
|
||||
} catch (e: unknown) {
|
||||
console.warn(`[gbrain] embedding failed for code file ${slug}: ${e instanceof Error ? e.message : String(e)}`);
|
||||
}
|
||||
}
|
||||
|
||||
// Store
|
||||
await engine.transaction(async (tx) => {
|
||||
if (existing) await tx.createVersion(slug);
|
||||
|
||||
await tx.putPage(slug, {
|
||||
type: 'code' as PageType,
|
||||
page_kind: 'code',
|
||||
title,
|
||||
compiled_truth: content,
|
||||
timeline: '',
|
||||
frontmatter: { language: lang, file: relativePath },
|
||||
content_hash: hash,
|
||||
});
|
||||
|
||||
await tx.addTag(slug, 'code');
|
||||
await tx.addTag(slug, lang);
|
||||
|
||||
if (chunks.length > 0) {
|
||||
await tx.upsertChunks(slug, chunks);
|
||||
} else {
|
||||
await tx.deleteChunks(slug);
|
||||
}
|
||||
});
|
||||
|
||||
// v0.20.0 Cathedral II Layer 5 (A1): extracted call-site edges persist
|
||||
// in code_edges_symbol (unresolved — we don't attempt within-file target
|
||||
// resolution here; getCallersOf / getCalleesOf match on to_symbol_qualified
|
||||
// which is the callee's short name). Edges land AFTER chunks upsert so
|
||||
// chunk IDs are stable.
|
||||
if (extractedEdges.length > 0 && chunks.length > 0) {
|
||||
try {
|
||||
const persistedChunks = await engine.getChunks(slug);
|
||||
const byIndex = new Map<number, { id?: number; symbol_name_qualified?: string | null; start_line?: number | null; end_line?: number | null }>();
|
||||
for (const pc of persistedChunks) {
|
||||
byIndex.set(pc.chunk_index, pc);
|
||||
}
|
||||
// Per-chunk invalidation (codex SP-2): wipe old edges involving
|
||||
// chunks whose IDs we know, so re-import doesn't leave stale
|
||||
// edges pointing at old symbol names.
|
||||
const chunkIds = persistedChunks
|
||||
.map(c => c.id)
|
||||
.filter((id): id is number => typeof id === 'number');
|
||||
if (chunkIds.length > 0) {
|
||||
await engine.deleteCodeEdgesForChunks(chunkIds);
|
||||
}
|
||||
|
||||
// Build the chunk-range table for offset → chunk-id resolution.
|
||||
const rangeList = chunks.map((ch, i) => {
|
||||
const persisted = byIndex.get(i);
|
||||
return {
|
||||
id: persisted?.id as number | undefined,
|
||||
startLine: ch.start_line ?? 1,
|
||||
endLine: ch.end_line ?? 1,
|
||||
symbol_name_qualified: ch.symbol_name_qualified ?? null,
|
||||
};
|
||||
});
|
||||
|
||||
const edgeInputs: import('./types.ts').CodeEdgeInput[] = [];
|
||||
for (const e of extractedEdges) {
|
||||
const idx = findChunkForOffset(e.callSiteByteOffset, content, rangeList);
|
||||
if (idx == null) continue;
|
||||
const from = rangeList[idx]!;
|
||||
if (!from.id || !from.symbol_name_qualified) continue;
|
||||
edgeInputs.push({
|
||||
from_chunk_id: from.id,
|
||||
to_chunk_id: null,
|
||||
from_symbol_qualified: from.symbol_name_qualified,
|
||||
to_symbol_qualified: e.toSymbol,
|
||||
edge_type: e.edgeType,
|
||||
});
|
||||
}
|
||||
|
||||
if (edgeInputs.length > 0) {
|
||||
await engine.addCodeEdges(edgeInputs);
|
||||
}
|
||||
} catch (edgeErr) {
|
||||
// Edge persistence is best-effort. A failed addCodeEdges must not
|
||||
// fail the overall import — the chunks + embeddings already
|
||||
// landed, which is the primary value.
|
||||
console.warn(`[gbrain] edge extraction failed for ${slug}: ${edgeErr instanceof Error ? edgeErr.message : String(edgeErr)}`);
|
||||
}
|
||||
}
|
||||
|
||||
return { slug, status: 'imported', chunks: chunks.length };
|
||||
}
|
||||
|
||||
// Backward compat
|
||||
export const importFile = importFromFile;
|
||||
export type ImportFileResult = ImportResult;
|
||||
|
||||
@@ -127,50 +127,6 @@ function stripCodeBlocks(content: string): string {
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* A code-reference found in markdown prose. Created by extractCodeRefs and
|
||||
* consumed by importFromFile's tail to build doc↔impl edges (v0.19.0 E1).
|
||||
*/
|
||||
export interface CodeRef {
|
||||
/** Raw matched path (e.g. 'src/core/sync.ts'). */
|
||||
path: string;
|
||||
/** Optional line number from 'src/foo.ts:42'. */
|
||||
line?: number;
|
||||
/** Index in the source string. */
|
||||
index: number;
|
||||
}
|
||||
|
||||
// v0.19.0 E1 — markdown guides that cite 'src/core/sync.ts:42' create an
|
||||
// edge to the code page that imported that file. Regex is anchored against
|
||||
// the common gbrain repo layout directories so arbitrary prose like
|
||||
// "in foo/bar.js" doesn't generate false positives.
|
||||
//
|
||||
// The extension list is aligned with detectCodeLanguage in chunkers/code.ts.
|
||||
// Paths NOT matching these extensions are ignored because they wouldn't
|
||||
// have a code page to edge to anyway.
|
||||
const CODE_REF_REGEX = /\b((?:src|lib|app|test|tests|scripts|docs|packages|internal|cmd|examples)\/[\w\-./]+\.(?:ts|tsx|mts|cts|js|jsx|mjs|cjs|py|rb|go|rs|java|cs|cpp|cc|hpp|c|h|php|swift|kt|scala|lua|ex|exs|elm|ml|dart|zig|sol|sh|bash|css|html|vue|json|yaml|yml|toml))(?::(\d+))?\b/g;
|
||||
|
||||
/**
|
||||
* Extract code-path references (e.g. 'src/core/sync.ts:42') from markdown
|
||||
* prose. Deduped by path.
|
||||
*/
|
||||
export function extractCodeRefs(content: string): CodeRef[] {
|
||||
const seen = new Set<string>();
|
||||
const refs: CodeRef[] = [];
|
||||
let match: RegExpExecArray | null;
|
||||
// Using a fresh regex object per call to avoid lastIndex state leaking
|
||||
// across invocations.
|
||||
const re = new RegExp(CODE_REF_REGEX.source, 'g');
|
||||
while ((match = re.exec(content)) !== null) {
|
||||
const path = match[1]!;
|
||||
if (seen.has(path)) continue;
|
||||
seen.add(path);
|
||||
const line = match[2] ? parseInt(match[2], 10) : undefined;
|
||||
refs.push({ path, line, index: match.index });
|
||||
}
|
||||
return refs;
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract `[Name](path-to-people-or-company)` references from arbitrary content.
|
||||
* Both filesystem-relative paths (with `../` and `.md`) and bare engine-style
|
||||
|
||||
@@ -812,259 +812,6 @@ export const MIGRATIONS: Migration[] = [
|
||||
END $$;
|
||||
`,
|
||||
},
|
||||
{
|
||||
version: 25,
|
||||
name: 'pages_page_kind',
|
||||
// v0.19.0 Layer 3 — pages.page_kind distinguishes markdown vs code pages
|
||||
// at the DB level. Needed so orphans filter, link-extraction auto-link,
|
||||
// and query --lang can branch on kind without sniffing `type` or chunk
|
||||
// metadata. Existing rows backfill to 'markdown' (pre-v0.19.0 all pages
|
||||
// were markdown).
|
||||
//
|
||||
// Postgres: ADD COLUMN with DEFAULT is O(1) for nullable columns (no
|
||||
// rewrite). The CHECK constraint is added NOT VALID so the initial
|
||||
// statement does not scan the table, then VALIDATE CONSTRAINT runs
|
||||
// separately. Tables with millions of pages would otherwise hold a
|
||||
// write lock during the full scan.
|
||||
sqlFor: {
|
||||
postgres: `
|
||||
ALTER TABLE pages
|
||||
ADD COLUMN IF NOT EXISTS page_kind TEXT NOT NULL DEFAULT 'markdown';
|
||||
|
||||
ALTER TABLE pages
|
||||
DROP CONSTRAINT IF EXISTS pages_page_kind_check;
|
||||
ALTER TABLE pages
|
||||
ADD CONSTRAINT pages_page_kind_check
|
||||
CHECK (page_kind IN ('markdown','code')) NOT VALID;
|
||||
ALTER TABLE pages VALIDATE CONSTRAINT pages_page_kind_check;
|
||||
`,
|
||||
pglite: `
|
||||
ALTER TABLE pages
|
||||
ADD COLUMN IF NOT EXISTS page_kind TEXT NOT NULL DEFAULT 'markdown'
|
||||
CHECK (page_kind IN ('markdown','code'));
|
||||
`,
|
||||
},
|
||||
sql: `
|
||||
ALTER TABLE pages
|
||||
ADD COLUMN IF NOT EXISTS page_kind TEXT NOT NULL DEFAULT 'markdown'
|
||||
CHECK (page_kind IN ('markdown','code'));
|
||||
`,
|
||||
},
|
||||
{
|
||||
version: 26,
|
||||
name: 'content_chunks_code_metadata',
|
||||
// v0.19.0 Layer 3 — content_chunks gains code-specific metadata columns
|
||||
// so C6 (query --lang), C7 (code-def / code-refs), and the new
|
||||
// searchCodeChunks engine method can filter + surface symbol context
|
||||
// without parsing chunk_text.
|
||||
//
|
||||
// All new columns are nullable — existing markdown chunks carry NULL.
|
||||
// importCodeFile populates them from the tree-sitter AST.
|
||||
//
|
||||
// Partial indexes (WHERE <col> IS NOT NULL) keep the index small: a
|
||||
// brain with 20K markdown chunks + 20K code chunks indexes only the
|
||||
// code chunks for symbol lookups. Measured ~200ms → ~15ms on code-refs.
|
||||
sql: `
|
||||
ALTER TABLE content_chunks
|
||||
ADD COLUMN IF NOT EXISTS language TEXT,
|
||||
ADD COLUMN IF NOT EXISTS symbol_name TEXT,
|
||||
ADD COLUMN IF NOT EXISTS symbol_type TEXT,
|
||||
ADD COLUMN IF NOT EXISTS start_line INTEGER,
|
||||
ADD COLUMN IF NOT EXISTS end_line INTEGER;
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_chunks_symbol_name
|
||||
ON content_chunks(symbol_name) WHERE symbol_name IS NOT NULL;
|
||||
CREATE INDEX IF NOT EXISTS idx_chunks_language
|
||||
ON content_chunks(language) WHERE language IS NOT NULL;
|
||||
`,
|
||||
},
|
||||
{
|
||||
version: 27,
|
||||
name: 'cathedral_ii_foundation',
|
||||
// v0.20.0 Cathedral II Layer 1 — schema-only foundation.
|
||||
//
|
||||
// Lands BEFORE any consumer layer to eliminate forward references
|
||||
// (codex SP-4). All Cathedral II DDL arrives here as one atomic
|
||||
// transaction:
|
||||
//
|
||||
// 1. content_chunks gains 4 columns:
|
||||
// - parent_symbol_path TEXT[] — scope chain for nested symbols (A3)
|
||||
// - doc_comment TEXT — extracted JSDoc/docstring (A4)
|
||||
// - symbol_name_qualified TEXT — 'Admin::UsersController#render' (A1)
|
||||
// - search_vector TSVECTOR — chunk-grain FTS (Layer 1b)
|
||||
//
|
||||
// 2. sources.chunker_version TEXT — SP-1 gate. performSync forces
|
||||
// full walk on mismatch with CURRENT_CHUNKER_VERSION, bypassing
|
||||
// the up_to_date git-HEAD early-return that made the bare
|
||||
// CHUNKER_VERSION bump a silent no-op.
|
||||
//
|
||||
// 3. code_edges_chunk — resolved call-graph / type-ref edges.
|
||||
// FK CASCADE from content_chunks on both endpoints; deleting a
|
||||
// chunk wipes its edges. UNIQUE (from, to, edge_type) holds
|
||||
// idempotency. source_id TEXT matches sources.id actual type
|
||||
// (codex F4). Source scoping is enforced in resolution logic,
|
||||
// not in the key, because from_chunk_id → pages.source_id
|
||||
// already determines it.
|
||||
//
|
||||
// 4. code_edges_symbol — unresolved refs. Target symbol is known
|
||||
// by qualified name but the defining chunk hasn't been imported
|
||||
// yet. Rows UNION with code_edges_chunk on read (codex 1.3b);
|
||||
// no promotion step.
|
||||
//
|
||||
// 5. update_chunk_search_vector trigger — BEFORE INSERT/UPDATE
|
||||
// OF (chunk_text, doc_comment, symbol_name_qualified). Builds
|
||||
// search_vector with weight A on doc_comment + symbol_name_qualified,
|
||||
// B on chunk_text. Natural-language queries rank doc-comment hits
|
||||
// above body-text hits (A4 intent).
|
||||
//
|
||||
// Consumer layers (Layer 5 A1, Layer 6 A3, Layer 10 C CLI, Layer 12
|
||||
// CHUNKER_VERSION bump, Layer 13 E2 reindex-code) all depend on this
|
||||
// foundation. Absent it, every downstream layer would have forward
|
||||
// refs.
|
||||
sql: `
|
||||
-- content_chunks: new Cathedral II columns
|
||||
ALTER TABLE content_chunks
|
||||
ADD COLUMN IF NOT EXISTS parent_symbol_path TEXT[],
|
||||
ADD COLUMN IF NOT EXISTS doc_comment TEXT,
|
||||
ADD COLUMN IF NOT EXISTS symbol_name_qualified TEXT,
|
||||
ADD COLUMN IF NOT EXISTS search_vector TSVECTOR;
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_chunks_search_vector
|
||||
ON content_chunks USING GIN(search_vector);
|
||||
CREATE INDEX IF NOT EXISTS idx_chunks_symbol_qualified
|
||||
ON content_chunks(symbol_name_qualified) WHERE symbol_name_qualified IS NOT NULL;
|
||||
|
||||
-- sources: SP-1 chunker_version gate
|
||||
ALTER TABLE sources
|
||||
ADD COLUMN IF NOT EXISTS chunker_version TEXT;
|
||||
|
||||
-- code_edges_chunk: resolved edges
|
||||
CREATE TABLE IF NOT EXISTS code_edges_chunk (
|
||||
id SERIAL PRIMARY KEY,
|
||||
from_chunk_id INTEGER NOT NULL REFERENCES content_chunks(id) ON DELETE CASCADE,
|
||||
to_chunk_id INTEGER NOT NULL REFERENCES content_chunks(id) ON DELETE CASCADE,
|
||||
from_symbol_qualified TEXT NOT NULL,
|
||||
to_symbol_qualified TEXT NOT NULL,
|
||||
edge_type TEXT NOT NULL,
|
||||
edge_metadata JSONB NOT NULL DEFAULT '{}',
|
||||
source_id TEXT REFERENCES sources(id) ON DELETE CASCADE,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
CONSTRAINT code_edges_chunk_unique UNIQUE (from_chunk_id, to_chunk_id, edge_type)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_code_edges_chunk_from
|
||||
ON code_edges_chunk(from_chunk_id, edge_type);
|
||||
CREATE INDEX IF NOT EXISTS idx_code_edges_chunk_to
|
||||
ON code_edges_chunk(to_chunk_id, edge_type);
|
||||
CREATE INDEX IF NOT EXISTS idx_code_edges_chunk_to_symbol
|
||||
ON code_edges_chunk(to_symbol_qualified, edge_type);
|
||||
|
||||
-- code_edges_symbol: unresolved refs
|
||||
CREATE TABLE IF NOT EXISTS code_edges_symbol (
|
||||
id SERIAL PRIMARY KEY,
|
||||
from_chunk_id INTEGER NOT NULL REFERENCES content_chunks(id) ON DELETE CASCADE,
|
||||
from_symbol_qualified TEXT NOT NULL,
|
||||
to_symbol_qualified TEXT NOT NULL,
|
||||
edge_type TEXT NOT NULL,
|
||||
edge_metadata JSONB NOT NULL DEFAULT '{}',
|
||||
source_id TEXT REFERENCES sources(id) ON DELETE CASCADE,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
CONSTRAINT code_edges_symbol_unique UNIQUE (from_chunk_id, to_symbol_qualified, edge_type)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_code_edges_symbol_from
|
||||
ON code_edges_symbol(from_chunk_id, edge_type);
|
||||
CREATE INDEX IF NOT EXISTS idx_code_edges_symbol_to
|
||||
ON code_edges_symbol(to_symbol_qualified, edge_type);
|
||||
|
||||
-- Chunk-grain FTS trigger (Layer 1b consumer — column exists from this
|
||||
-- migration, trigger installed now so newly-written chunks get vectors
|
||||
-- from day one). NULL-safe: markdown chunks leave doc_comment and
|
||||
-- symbol_name_qualified NULL; COALESCE('') keeps the vector build
|
||||
-- from failing on missing weights.
|
||||
CREATE OR REPLACE FUNCTION update_chunk_search_vector() RETURNS TRIGGER AS $fn$
|
||||
BEGIN
|
||||
NEW.search_vector :=
|
||||
setweight(to_tsvector('english', COALESCE(NEW.doc_comment, '')), 'A') ||
|
||||
setweight(to_tsvector('english', COALESCE(NEW.symbol_name_qualified, '')), 'A') ||
|
||||
setweight(to_tsvector('english', COALESCE(NEW.chunk_text, '')), 'B');
|
||||
RETURN NEW;
|
||||
END;
|
||||
$fn$ LANGUAGE plpgsql;
|
||||
|
||||
DROP TRIGGER IF EXISTS chunk_search_vector_trigger ON content_chunks;
|
||||
CREATE TRIGGER chunk_search_vector_trigger
|
||||
BEFORE INSERT OR UPDATE OF chunk_text, doc_comment, symbol_name_qualified
|
||||
ON content_chunks
|
||||
FOR EACH ROW EXECUTE FUNCTION update_chunk_search_vector();
|
||||
`,
|
||||
},
|
||||
{
|
||||
version: 28,
|
||||
name: 'cathedral_ii_chunk_fts_backfill',
|
||||
// v0.20.0 Cathedral II Layer 3 (1b) — backfill content_chunks.search_vector
|
||||
// for rows inserted before v27 ran. The v27 trigger only fires on
|
||||
// INSERT/UPDATE, so every chunk that existed before upgrade has a NULL
|
||||
// search_vector and would match zero rows in the new chunk-grain
|
||||
// searchKeyword. Compute the vector in-place here so upgraded brains
|
||||
// have full keyword coverage the moment v28 commits — no need to wait
|
||||
// for every page to get touched by sync.
|
||||
//
|
||||
// Direct vector compute (not UPDATE chunk_text = chunk_text to trigger):
|
||||
// - UPDATE-to-same-value fires the trigger unconditionally on Postgres
|
||||
// even if no column value changes, so trigger-based backfill DOES
|
||||
// work, but writing the vector directly is cheaper (single pass
|
||||
// instead of trigger overhead per row).
|
||||
// - Idempotent via `WHERE search_vector IS NULL` — re-running v28
|
||||
// after a partial run picks up only the remaining NULL rows.
|
||||
//
|
||||
// On a 20K-chunk brain: ~2-3s total. No blocking concerns: chunks are
|
||||
// append-only in steady state; the UPDATE takes a row lock per chunk
|
||||
// briefly while computing the tsvector.
|
||||
sql: `
|
||||
UPDATE content_chunks
|
||||
SET search_vector =
|
||||
setweight(to_tsvector('english', COALESCE(doc_comment, '')), 'A') ||
|
||||
setweight(to_tsvector('english', COALESCE(symbol_name_qualified, '')), 'A') ||
|
||||
setweight(to_tsvector('english', COALESCE(chunk_text, '')), 'B')
|
||||
WHERE search_vector IS NULL;
|
||||
`,
|
||||
},
|
||||
{
|
||||
version: 29,
|
||||
name: 'cathedral_ii_code_edges_rls',
|
||||
// v0.21.0 Cathedral II — RLS hardening for the two new tables added by
|
||||
// v27 (code_edges_chunk, code_edges_symbol). The v24 RLS-backfill
|
||||
// pattern: gated on BYPASSRLS (so we don't lock the migrating session
|
||||
// out of its own data on a non-bypass role) + bare ALTER TABLE since
|
||||
// both tables are guaranteed to exist after v27.
|
||||
//
|
||||
// Postgres-only via sqlFor: PGLite doesn't enforce RLS the same way
|
||||
// and v24 already runs only against Postgres in practice. The E2E
|
||||
// test "RLS is enabled on every public table" runs against Docker
|
||||
// postgres exclusively and was failing because v27 created the new
|
||||
// tables without RLS enabled.
|
||||
sqlFor: {
|
||||
postgres: `
|
||||
DO $$
|
||||
DECLARE
|
||||
has_bypass BOOLEAN;
|
||||
BEGIN
|
||||
SELECT rolbypassrls INTO has_bypass FROM pg_roles WHERE rolname = current_user;
|
||||
IF NOT has_bypass THEN
|
||||
RAISE EXCEPTION 'v29 cathedral_ii_code_edges_rls: role % does not have BYPASSRLS privilege — cannot enable RLS safely. Re-run as postgres (or another BYPASSRLS role). The migration will retry automatically on the next initSchema call.', current_user;
|
||||
END IF;
|
||||
|
||||
ALTER TABLE code_edges_chunk ENABLE ROW LEVEL SECURITY;
|
||||
ALTER TABLE code_edges_symbol ENABLE ROW LEVEL SECURITY;
|
||||
|
||||
RAISE NOTICE 'v29: code_edges RLS enabled (role % has BYPASSRLS)', current_user;
|
||||
END $$;
|
||||
`,
|
||||
pglite: `-- PGLite: no-op. RLS check runs only against Postgres E2E.`,
|
||||
},
|
||||
sql: '',
|
||||
},
|
||||
];
|
||||
|
||||
export const LATEST_VERSION = MIGRATIONS.length > 0
|
||||
|
||||
@@ -581,12 +581,6 @@ const query: Operation = {
|
||||
offset: { type: 'number', description: 'Skip first N results (for pagination)' },
|
||||
expand: { type: 'boolean', description: 'Enable multi-query expansion (default: true)' },
|
||||
detail: { type: 'string', description: 'Result detail level: low (compiled truth only), medium (default, all with dedup), high (all chunks)' },
|
||||
// v0.20.0 Cathedral II Layer 10 C1/C2: language + symbol-kind filters.
|
||||
lang: { type: 'string', description: 'Filter to chunks where content_chunks.language matches (e.g., typescript, python, ruby)' },
|
||||
symbol_kind: { type: 'string', description: 'Filter to chunks where content_chunks.symbol_type matches (e.g., function, class, method, type, interface)' },
|
||||
// v0.20.0 Cathedral II Layer 7 (A2) / Layer 10 C3: two-pass structural expansion.
|
||||
near_symbol: { type: 'string', description: 'Anchor retrieval at this qualified symbol name (e.g., BrainEngine.searchKeyword). Enables A2 two-pass.' },
|
||||
walk_depth: { type: 'number', description: 'Structural walk depth 1-2. Default 0 (off). Expands anchors through code_edges with 1/(1+hop) decay.' },
|
||||
},
|
||||
handler: async (ctx, p) => {
|
||||
const expand = p.expand !== false;
|
||||
@@ -597,10 +591,6 @@ const query: Operation = {
|
||||
expansion: expand,
|
||||
expandFn: expand ? expandQuery : undefined,
|
||||
detail,
|
||||
language: (p.lang as string) || undefined,
|
||||
symbolKind: (p.symbol_kind as string) || undefined,
|
||||
nearSymbol: (p.near_symbol as string) || undefined,
|
||||
walkDepth: typeof p.walk_depth === 'number' ? (p.walk_depth as number) : undefined,
|
||||
});
|
||||
},
|
||||
cliHints: { name: 'query', positional: ['query'] },
|
||||
|
||||
+25
-335
@@ -134,13 +134,11 @@ export class PGLiteEngine implements BrainEngine {
|
||||
// a parameter. ON CONFLICT target becomes (source_id, slug) since the
|
||||
// global UNIQUE(slug) was dropped in migration v17. Step 5+ will
|
||||
// surface an explicit sourceId param on putPage for multi-source sync.
|
||||
const pageKind = page.page_kind || 'markdown';
|
||||
const { rows } = await this.db.query(
|
||||
`INSERT INTO pages (slug, type, page_kind, title, compiled_truth, timeline, frontmatter, content_hash, updated_at)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7::jsonb, $8, now())
|
||||
`INSERT INTO pages (slug, type, title, compiled_truth, timeline, frontmatter, content_hash, updated_at)
|
||||
VALUES ($1, $2, $3, $4, $5, $6::jsonb, $7, now())
|
||||
ON CONFLICT (source_id, slug) DO UPDATE SET
|
||||
type = EXCLUDED.type,
|
||||
page_kind = EXCLUDED.page_kind,
|
||||
title = EXCLUDED.title,
|
||||
compiled_truth = EXCLUDED.compiled_truth,
|
||||
timeline = EXCLUDED.timeline,
|
||||
@@ -148,7 +146,7 @@ export class PGLiteEngine implements BrainEngine {
|
||||
content_hash = EXCLUDED.content_hash,
|
||||
updated_at = now()
|
||||
RETURNING id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at`,
|
||||
[slug, page.type, pageKind, page.title, page.compiled_truth, page.timeline || '', JSON.stringify(frontmatter), hash]
|
||||
[slug, page.type, page.title, page.compiled_truth, page.timeline || '', JSON.stringify(frontmatter), hash]
|
||||
);
|
||||
return rowToPage(rows[0] as Record<string, unknown>);
|
||||
}
|
||||
@@ -214,19 +212,6 @@ export class PGLiteEngine implements BrainEngine {
|
||||
}
|
||||
|
||||
// Search
|
||||
//
|
||||
// v0.20.0 Cathedral II Layer 3 (1b): keyword search now ranks at
|
||||
// chunk-grain internally using content_chunks.search_vector, then dedups
|
||||
// to best-chunk-per-page on the way out. External shape (page-grain,
|
||||
// one row per matched page, best chunk selected) is identical to
|
||||
// v0.19.0 — backlinks, enrichment-service.countMentions, list_pages,
|
||||
// etc. all see the same contract. A2 two-pass (Layer 7) consumes
|
||||
// searchKeywordChunks for raw chunk-grain results without the dedup.
|
||||
//
|
||||
// The DISTINCT ON pattern is translated into a two-stage query because
|
||||
// PGLite's query planner handles CTEs-with-DISTINCT-ON less optimally
|
||||
// than direct window function + GROUP BY. Fetch more chunks than the
|
||||
// page limit (3x) to ensure N dedup'd pages survive; bounded and fast.
|
||||
async searchKeyword(query: string, opts?: SearchOpts): Promise<SearchResult[]> {
|
||||
const limit = clampSearchLimit(opts?.limit);
|
||||
const offset = opts?.offset || 0;
|
||||
@@ -236,96 +221,21 @@ export class PGLiteEngine implements BrainEngine {
|
||||
console.warn(`[gbrain] Warning: search limit clamped from ${opts.limit} to ${MAX_SEARCH_LIMIT}`);
|
||||
}
|
||||
|
||||
// Fetch 3x to give dedup headroom, then page-dedup + re-limit.
|
||||
const innerLimit = Math.min(limit * 3, MAX_SEARCH_LIMIT * 3);
|
||||
|
||||
// v0.20.0 Cathedral II Layer 10 C1/C2: language + symbol-kind filters.
|
||||
const params: unknown[] = [query, innerLimit, limit, offset];
|
||||
let extraFilter = '';
|
||||
if (opts?.language) {
|
||||
params.push(opts.language);
|
||||
extraFilter += ` AND cc.language = $${params.length}`;
|
||||
}
|
||||
if (opts?.symbolKind) {
|
||||
params.push(opts.symbolKind);
|
||||
extraFilter += ` AND cc.symbol_type = $${params.length}`;
|
||||
}
|
||||
|
||||
const { rows } = await this.db.query(
|
||||
`WITH ranked AS (
|
||||
SELECT
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) AS score,
|
||||
CASE WHEN p.updated_at < (
|
||||
SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id
|
||||
) THEN true ELSE false END AS stale
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) ${detailFilter}${extraFilter}
|
||||
ORDER BY score DESC
|
||||
LIMIT $2
|
||||
),
|
||||
best_per_page AS (
|
||||
SELECT DISTINCT ON (slug) *
|
||||
FROM ranked
|
||||
ORDER BY slug, score DESC
|
||||
)
|
||||
SELECT * FROM best_per_page
|
||||
ORDER BY score DESC
|
||||
LIMIT $3 OFFSET $4`,
|
||||
params
|
||||
);
|
||||
|
||||
return (rows as Record<string, unknown>[]).map(rowToSearchResult);
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.20.0 Cathedral II Layer 3 (1b) chunk-grain keyword search.
|
||||
*
|
||||
* Ranks at chunk grain via content_chunks.search_vector WITHOUT the
|
||||
* dedup-to-page pass that searchKeyword applies on return. Used by
|
||||
* A2 two-pass retrieval (Layer 7) as the anchor-discovery primitive:
|
||||
* two-pass wants the top-N chunks (regardless of page), not the
|
||||
* best chunk per top-N pages.
|
||||
*
|
||||
* Most callers should prefer searchKeyword (external page-grain
|
||||
* contract). This method is intentionally a narrow internal knob.
|
||||
*/
|
||||
async searchKeywordChunks(query: string, opts?: SearchOpts): Promise<SearchResult[]> {
|
||||
const limit = clampSearchLimit(opts?.limit);
|
||||
const offset = opts?.offset || 0;
|
||||
const detailFilter = opts?.detail === 'low' ? `AND cc.chunk_source = 'compiled_truth'` : '';
|
||||
|
||||
if (opts?.limit && opts.limit > MAX_SEARCH_LIMIT) {
|
||||
console.warn(`[gbrain] Warning: search limit clamped from ${opts.limit} to ${MAX_SEARCH_LIMIT}`);
|
||||
}
|
||||
|
||||
const params: unknown[] = [query, limit, offset];
|
||||
let extraFilter = '';
|
||||
if (opts?.language) {
|
||||
params.push(opts.language);
|
||||
extraFilter += ` AND cc.language = $${params.length}`;
|
||||
}
|
||||
if (opts?.symbolKind) {
|
||||
params.push(opts.symbolKind);
|
||||
extraFilter += ` AND cc.symbol_type = $${params.length}`;
|
||||
}
|
||||
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) AS score,
|
||||
CASE WHEN p.updated_at < (
|
||||
SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id
|
||||
) THEN true ELSE false END AS stale
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) ${detailFilter}${extraFilter}
|
||||
ORDER BY score DESC
|
||||
LIMIT $2 OFFSET $3`,
|
||||
params
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
ts_rank(p.search_vector, websearch_to_tsquery('english', $1)) AS score,
|
||||
CASE WHEN p.updated_at < (
|
||||
SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id
|
||||
) THEN true ELSE false END AS stale
|
||||
FROM pages p
|
||||
JOIN content_chunks cc ON cc.page_id = p.id
|
||||
WHERE p.search_vector @@ websearch_to_tsquery('english', $1) ${detailFilter}
|
||||
ORDER BY score DESC
|
||||
LIMIT $2
|
||||
OFFSET $3`,
|
||||
[query, limit, offset]
|
||||
);
|
||||
|
||||
return (rows as Record<string, unknown>[]).map(rowToSearchResult);
|
||||
@@ -341,17 +251,6 @@ export class PGLiteEngine implements BrainEngine {
|
||||
console.warn(`[gbrain] Warning: search limit clamped from ${opts.limit} to ${MAX_SEARCH_LIMIT}`);
|
||||
}
|
||||
|
||||
const params: unknown[] = [vecStr, limit, offset];
|
||||
let extraFilter = '';
|
||||
if (opts?.language) {
|
||||
params.push(opts.language);
|
||||
extraFilter += ` AND cc.language = $${params.length}`;
|
||||
}
|
||||
if (opts?.symbolKind) {
|
||||
params.push(opts.symbolKind);
|
||||
extraFilter += ` AND cc.symbol_type = $${params.length}`;
|
||||
}
|
||||
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
@@ -362,11 +261,11 @@ export class PGLiteEngine implements BrainEngine {
|
||||
) THEN true ELSE false END AS stale
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NOT NULL ${detailFilter}${extraFilter}
|
||||
WHERE cc.embedding IS NOT NULL ${detailFilter}
|
||||
ORDER BY cc.embedding <=> $1::vector
|
||||
LIMIT $2
|
||||
OFFSET $3`,
|
||||
params
|
||||
[vecStr, limit, offset]
|
||||
);
|
||||
|
||||
return (rows as Record<string, unknown>[]).map(rowToSearchResult);
|
||||
@@ -410,14 +309,8 @@ export class PGLiteEngine implements BrainEngine {
|
||||
return;
|
||||
}
|
||||
|
||||
// Batch upsert: build dynamic multi-row INSERT.
|
||||
// v0.19.0: includes language/symbol_name/symbol_type/start_line/end_line
|
||||
// so code chunks carry their tree-sitter metadata into the DB. Markdown
|
||||
// chunks pass NULL for all five. Order must match the column list.
|
||||
// v0.20.0 Cathedral II Layer 6: adds parent_symbol_path / doc_comment /
|
||||
// symbol_name_qualified so nested-chunk emission (A3) and eventual A1
|
||||
// edge resolution can round-trip metadata through upserts.
|
||||
const cols = '(page_id, chunk_index, chunk_text, chunk_source, embedding, model, token_count, embedded_at, language, symbol_name, symbol_type, start_line, end_line, parent_symbol_path, doc_comment, symbol_name_qualified)';
|
||||
// Batch upsert: build dynamic multi-row INSERT
|
||||
const cols = '(page_id, chunk_index, chunk_text, chunk_source, embedding, model, token_count, embedded_at)';
|
||||
const rowParts: string[] = [];
|
||||
const params: unknown[] = [];
|
||||
let paramIdx = 1;
|
||||
@@ -426,28 +319,13 @@ export class PGLiteEngine implements BrainEngine {
|
||||
const embeddingStr = chunk.embedding
|
||||
? '[' + Array.from(chunk.embedding).join(',') + ']'
|
||||
: null;
|
||||
const parentPath = chunk.parent_symbol_path && chunk.parent_symbol_path.length > 0
|
||||
? chunk.parent_symbol_path
|
||||
: null;
|
||||
|
||||
if (embeddingStr) {
|
||||
rowParts.push(`($${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}::vector, $${paramIdx++}, $${paramIdx++}, now(), $${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}::text[], $${paramIdx++}, $${paramIdx++})`);
|
||||
params.push(
|
||||
pageId, chunk.chunk_index, chunk.chunk_text, chunk.chunk_source,
|
||||
embeddingStr, chunk.model || 'text-embedding-3-large', chunk.token_count || null,
|
||||
chunk.language || null, chunk.symbol_name || null, chunk.symbol_type || null,
|
||||
chunk.start_line ?? null, chunk.end_line ?? null,
|
||||
parentPath, chunk.doc_comment || null, chunk.symbol_name_qualified || null,
|
||||
);
|
||||
rowParts.push(`($${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}::vector, $${paramIdx++}, $${paramIdx++}, now())`);
|
||||
params.push(pageId, chunk.chunk_index, chunk.chunk_text, chunk.chunk_source, embeddingStr, chunk.model || 'text-embedding-3-large', chunk.token_count || null);
|
||||
} else {
|
||||
rowParts.push(`($${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}, NULL, $${paramIdx++}, $${paramIdx++}, NULL, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}::text[], $${paramIdx++}, $${paramIdx++})`);
|
||||
params.push(
|
||||
pageId, chunk.chunk_index, chunk.chunk_text, chunk.chunk_source,
|
||||
chunk.model || 'text-embedding-3-large', chunk.token_count || null,
|
||||
chunk.language || null, chunk.symbol_name || null, chunk.symbol_type || null,
|
||||
chunk.start_line ?? null, chunk.end_line ?? null,
|
||||
parentPath, chunk.doc_comment || null, chunk.symbol_name_qualified || null,
|
||||
);
|
||||
rowParts.push(`($${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}, NULL, $${paramIdx++}, $${paramIdx++}, NULL)`);
|
||||
params.push(pageId, chunk.chunk_index, chunk.chunk_text, chunk.chunk_source, chunk.model || 'text-embedding-3-large', chunk.token_count || null);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -459,15 +337,7 @@ export class PGLiteEngine implements BrainEngine {
|
||||
embedding = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.embedding ELSE COALESCE(EXCLUDED.embedding, content_chunks.embedding) END,
|
||||
model = COALESCE(EXCLUDED.model, content_chunks.model),
|
||||
token_count = EXCLUDED.token_count,
|
||||
embedded_at = COALESCE(EXCLUDED.embedded_at, content_chunks.embedded_at),
|
||||
language = EXCLUDED.language,
|
||||
symbol_name = EXCLUDED.symbol_name,
|
||||
symbol_type = EXCLUDED.symbol_type,
|
||||
start_line = EXCLUDED.start_line,
|
||||
end_line = EXCLUDED.end_line,
|
||||
parent_symbol_path = EXCLUDED.parent_symbol_path,
|
||||
doc_comment = EXCLUDED.doc_comment,
|
||||
symbol_name_qualified = EXCLUDED.symbol_name_qualified`,
|
||||
embedded_at = COALESCE(EXCLUDED.embedded_at, content_chunks.embedded_at)`,
|
||||
params
|
||||
);
|
||||
}
|
||||
@@ -1187,184 +1057,4 @@ export class PGLiteEngine implements BrainEngine {
|
||||
const { rows } = await this.db.query(sql, params);
|
||||
return rows as T[];
|
||||
}
|
||||
|
||||
// ============================================================
|
||||
// v0.20.0 Cathedral II: code edges (Layer 1 stubs — filled by Layer 5)
|
||||
// ============================================================
|
||||
// Declared here so the interface contract is satisfied and consumers can
|
||||
// import against them. Implementations throw until the edge extractor +
|
||||
// per-lang tree-sitter queries land in Layer 5/6.
|
||||
// ============================================================
|
||||
|
||||
async addCodeEdges(edges: import('./types.ts').CodeEdgeInput[]): Promise<number> {
|
||||
if (edges.length === 0) return 0;
|
||||
let inserted = 0;
|
||||
// Split into resolved vs unresolved. Resolved rows carry to_chunk_id
|
||||
// (known target chunk); unresolved rows only know the qualified name.
|
||||
const resolved = edges.filter(e => e.to_chunk_id != null);
|
||||
const unresolved = edges.filter(e => e.to_chunk_id == null);
|
||||
|
||||
if (resolved.length > 0) {
|
||||
const rowParts: string[] = [];
|
||||
const params: unknown[] = [];
|
||||
let p = 1;
|
||||
for (const e of resolved) {
|
||||
rowParts.push(`($${p++}, $${p++}, $${p++}, $${p++}, $${p++}, $${p++}::jsonb, $${p++})`);
|
||||
params.push(
|
||||
e.from_chunk_id, e.to_chunk_id, e.from_symbol_qualified,
|
||||
e.to_symbol_qualified, e.edge_type,
|
||||
JSON.stringify(e.edge_metadata ?? {}),
|
||||
e.source_id ?? null,
|
||||
);
|
||||
}
|
||||
const res = await this.db.query(
|
||||
`INSERT INTO code_edges_chunk
|
||||
(from_chunk_id, to_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id)
|
||||
VALUES ${rowParts.join(', ')}
|
||||
ON CONFLICT (from_chunk_id, to_chunk_id, edge_type) DO NOTHING`,
|
||||
params,
|
||||
);
|
||||
inserted += res.affectedRows ?? 0;
|
||||
}
|
||||
if (unresolved.length > 0) {
|
||||
const rowParts: string[] = [];
|
||||
const params: unknown[] = [];
|
||||
let p = 1;
|
||||
for (const e of unresolved) {
|
||||
rowParts.push(`($${p++}, $${p++}, $${p++}, $${p++}, $${p++}::jsonb, $${p++})`);
|
||||
params.push(
|
||||
e.from_chunk_id, e.from_symbol_qualified, e.to_symbol_qualified, e.edge_type,
|
||||
JSON.stringify(e.edge_metadata ?? {}),
|
||||
e.source_id ?? null,
|
||||
);
|
||||
}
|
||||
const res = await this.db.query(
|
||||
`INSERT INTO code_edges_symbol
|
||||
(from_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id)
|
||||
VALUES ${rowParts.join(', ')}
|
||||
ON CONFLICT (from_chunk_id, to_symbol_qualified, edge_type) DO NOTHING`,
|
||||
params,
|
||||
);
|
||||
inserted += res.affectedRows ?? 0;
|
||||
}
|
||||
return inserted;
|
||||
}
|
||||
|
||||
async deleteCodeEdgesForChunks(chunkIds: number[]): Promise<void> {
|
||||
if (chunkIds.length === 0) return;
|
||||
// Both directions on code_edges_chunk; from-only on code_edges_symbol
|
||||
// (unresolved edges don't have a to_chunk_id to match against).
|
||||
await this.db.query(
|
||||
`DELETE FROM code_edges_chunk WHERE from_chunk_id = ANY($1::int[]) OR to_chunk_id = ANY($1::int[])`,
|
||||
[chunkIds],
|
||||
);
|
||||
await this.db.query(
|
||||
`DELETE FROM code_edges_symbol WHERE from_chunk_id = ANY($1::int[])`,
|
||||
[chunkIds],
|
||||
);
|
||||
}
|
||||
|
||||
async getCallersOf(
|
||||
qualifiedName: string,
|
||||
opts?: { sourceId?: string; allSources?: boolean; limit?: number },
|
||||
): Promise<import('./types.ts').CodeEdgeResult[]> {
|
||||
const limit = Math.min(opts?.limit ?? 100, 500);
|
||||
const sourceClause = opts?.allSources || !opts?.sourceId
|
||||
? ''
|
||||
: `AND source_id = '${opts.sourceId.replace(/'/g, "''")}'`;
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT id, from_chunk_id, to_chunk_id, from_symbol_qualified, to_symbol_qualified,
|
||||
edge_type, edge_metadata, source_id, true as resolved
|
||||
FROM code_edges_chunk
|
||||
WHERE to_symbol_qualified = $1 ${sourceClause}
|
||||
UNION ALL
|
||||
SELECT id, from_chunk_id, NULL as to_chunk_id, from_symbol_qualified, to_symbol_qualified,
|
||||
edge_type, edge_metadata, source_id, false as resolved
|
||||
FROM code_edges_symbol
|
||||
WHERE to_symbol_qualified = $1 ${sourceClause}
|
||||
LIMIT $2`,
|
||||
[qualifiedName, limit],
|
||||
);
|
||||
return (rows as Record<string, unknown>[]).map(rowToCodeEdge);
|
||||
}
|
||||
|
||||
async getCalleesOf(
|
||||
qualifiedName: string,
|
||||
opts?: { sourceId?: string; allSources?: boolean; limit?: number },
|
||||
): Promise<import('./types.ts').CodeEdgeResult[]> {
|
||||
const limit = Math.min(opts?.limit ?? 100, 500);
|
||||
const sourceClause = opts?.allSources || !opts?.sourceId
|
||||
? ''
|
||||
: `AND source_id = '${opts.sourceId.replace(/'/g, "''")}'`;
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT id, from_chunk_id, to_chunk_id, from_symbol_qualified, to_symbol_qualified,
|
||||
edge_type, edge_metadata, source_id, true as resolved
|
||||
FROM code_edges_chunk
|
||||
WHERE from_symbol_qualified = $1 ${sourceClause}
|
||||
UNION ALL
|
||||
SELECT id, from_chunk_id, NULL as to_chunk_id, from_symbol_qualified, to_symbol_qualified,
|
||||
edge_type, edge_metadata, source_id, false as resolved
|
||||
FROM code_edges_symbol
|
||||
WHERE from_symbol_qualified = $1 ${sourceClause}
|
||||
LIMIT $2`,
|
||||
[qualifiedName, limit],
|
||||
);
|
||||
return (rows as Record<string, unknown>[]).map(rowToCodeEdge);
|
||||
}
|
||||
|
||||
async getEdgesByChunk(
|
||||
chunkId: number,
|
||||
opts?: { direction?: 'in' | 'out' | 'both'; edgeType?: string; limit?: number },
|
||||
): Promise<import('./types.ts').CodeEdgeResult[]> {
|
||||
const direction = opts?.direction ?? 'both';
|
||||
const limit = Math.min(opts?.limit ?? 50, 200);
|
||||
const edgeTypeClause = opts?.edgeType ? `AND edge_type = '${opts.edgeType.replace(/'/g, "''")}'` : '';
|
||||
// Build the chunk-table filter based on direction. Unresolved edges
|
||||
// (code_edges_symbol) only carry from_chunk_id — there's no inbound
|
||||
// direction into them from a chunk ID, so we include them only when
|
||||
// direction is 'out' or 'both'.
|
||||
let chunkFilter = '';
|
||||
if (direction === 'in') chunkFilter = `WHERE to_chunk_id = $1`;
|
||||
else if (direction === 'out') chunkFilter = `WHERE from_chunk_id = $1`;
|
||||
else chunkFilter = `WHERE from_chunk_id = $1 OR to_chunk_id = $1`;
|
||||
|
||||
let symbolFilter = '';
|
||||
if (direction === 'out' || direction === 'both') {
|
||||
symbolFilter = `WHERE from_chunk_id = $1`;
|
||||
}
|
||||
|
||||
const unionClause = symbolFilter ? `
|
||||
UNION ALL
|
||||
SELECT id, from_chunk_id, NULL as to_chunk_id, from_symbol_qualified, to_symbol_qualified,
|
||||
edge_type, edge_metadata, source_id, false as resolved
|
||||
FROM code_edges_symbol
|
||||
${symbolFilter} ${edgeTypeClause}
|
||||
` : '';
|
||||
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT id, from_chunk_id, to_chunk_id, from_symbol_qualified, to_symbol_qualified,
|
||||
edge_type, edge_metadata, source_id, true as resolved
|
||||
FROM code_edges_chunk
|
||||
${chunkFilter} ${edgeTypeClause}
|
||||
${unionClause}
|
||||
LIMIT $2`,
|
||||
[chunkId, limit],
|
||||
);
|
||||
return (rows as Record<string, unknown>[]).map(rowToCodeEdge);
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
function rowToCodeEdge(row: Record<string, unknown>): import('./types.ts').CodeEdgeResult {
|
||||
return {
|
||||
id: row.id as number,
|
||||
from_chunk_id: row.from_chunk_id as number,
|
||||
to_chunk_id: row.to_chunk_id == null ? null : (row.to_chunk_id as number),
|
||||
from_symbol_qualified: (row.from_symbol_qualified as string) ?? '',
|
||||
to_symbol_qualified: (row.to_symbol_qualified as string) ?? '',
|
||||
edge_type: (row.edge_type as string) ?? '',
|
||||
edge_metadata: (row.edge_metadata as Record<string, unknown>) ?? {},
|
||||
source_id: row.source_id == null ? null : (row.source_id as string),
|
||||
resolved: Boolean(row.resolved),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -47,9 +47,6 @@ CREATE TABLE IF NOT EXISTS pages (
|
||||
REFERENCES sources(id) ON DELETE CASCADE,
|
||||
slug TEXT NOT NULL,
|
||||
type TEXT NOT NULL,
|
||||
-- v0.19.0: markdown vs code distinction at the DB level.
|
||||
page_kind TEXT NOT NULL DEFAULT 'markdown'
|
||||
CHECK (page_kind IN ('markdown','code')),
|
||||
title TEXT NOT NULL,
|
||||
compiled_truth TEXT NOT NULL DEFAULT '',
|
||||
timeline TEXT NOT NULL DEFAULT '',
|
||||
@@ -78,21 +75,12 @@ CREATE TABLE IF NOT EXISTS content_chunks (
|
||||
model TEXT NOT NULL DEFAULT 'text-embedding-3-large',
|
||||
token_count INTEGER,
|
||||
embedded_at TIMESTAMPTZ,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
-- v0.19.0: code chunk metadata (markdown chunks leave NULL).
|
||||
language TEXT,
|
||||
symbol_name TEXT,
|
||||
symbol_type TEXT,
|
||||
start_line INTEGER,
|
||||
end_line INTEGER
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_chunks_page_index ON content_chunks(page_id, chunk_index);
|
||||
CREATE INDEX IF NOT EXISTS idx_chunks_page ON content_chunks(page_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_chunks_embedding ON content_chunks USING hnsw (embedding vector_cosine_ops);
|
||||
-- v0.19.0: partial indexes for code chunk lookups.
|
||||
CREATE INDEX IF NOT EXISTS idx_chunks_symbol_name ON content_chunks(symbol_name) WHERE symbol_name IS NOT NULL;
|
||||
CREATE INDEX IF NOT EXISTS idx_chunks_language ON content_chunks(language) WHERE language IS NOT NULL;
|
||||
|
||||
-- ============================================================
|
||||
-- links: cross-references between pages
|
||||
|
||||
+28
-294
@@ -138,13 +138,11 @@ export class PostgresEngine implements BrainEngine {
|
||||
// CONFLICT target becomes (source_id, slug) since global UNIQUE(slug)
|
||||
// was dropped in migration v17. See pglite-engine.ts for matching
|
||||
// notes; multi-source sync (Step 5) will surface an explicit sourceId.
|
||||
const pageKind = page.page_kind || 'markdown';
|
||||
const rows = await sql`
|
||||
INSERT INTO pages (slug, type, page_kind, title, compiled_truth, timeline, frontmatter, content_hash, updated_at)
|
||||
VALUES (${slug}, ${page.type}, ${pageKind}, ${page.title}, ${page.compiled_truth}, ${page.timeline || ''}, ${sql.json(frontmatter as Parameters<typeof sql.json>[0])}, ${hash}, now())
|
||||
INSERT INTO pages (slug, type, title, compiled_truth, timeline, frontmatter, content_hash, updated_at)
|
||||
VALUES (${slug}, ${page.type}, ${page.title}, ${page.compiled_truth}, ${page.timeline || ''}, ${sql.json(frontmatter as Parameters<typeof sql.json>[0])}, ${hash}, now())
|
||||
ON CONFLICT (source_id, slug) DO UPDATE SET
|
||||
type = EXCLUDED.type,
|
||||
page_kind = EXCLUDED.page_kind,
|
||||
title = EXCLUDED.title,
|
||||
compiled_truth = EXCLUDED.compiled_truth,
|
||||
timeline = EXCLUDED.timeline,
|
||||
@@ -212,29 +210,18 @@ export class PostgresEngine implements BrainEngine {
|
||||
}
|
||||
|
||||
// Search
|
||||
// v0.20.0 Cathedral II Layer 3 (1b): chunk-grain FTS internally,
|
||||
// dedup-to-best-chunk-per-page on the way out. External shape
|
||||
// preserves the v0.19.0 contract so backlinks / enrichment-service /
|
||||
// list_pages etc. see zero breaking changes. A2 two-pass (Layer 7)
|
||||
// consumes searchKeywordChunks for the raw chunk-grain primitive.
|
||||
async searchKeyword(query: string, opts?: SearchOpts): Promise<SearchResult[]> {
|
||||
const sql = this.sql;
|
||||
const limit = clampSearchLimit(opts?.limit);
|
||||
const offset = opts?.offset || 0;
|
||||
const type = opts?.type;
|
||||
const excludeSlugs = opts?.exclude_slugs;
|
||||
const language = opts?.language;
|
||||
const symbolKind = opts?.symbolKind;
|
||||
|
||||
if (opts?.limit && opts.limit > MAX_SEARCH_LIMIT) {
|
||||
console.warn(`[gbrain] Warning: search limit clamped from ${opts.limit} to ${MAX_SEARCH_LIMIT}`);
|
||||
}
|
||||
|
||||
const detailLow = opts?.detail === 'low';
|
||||
// Fetch headroom for dedup: if we only fetch `limit` chunks, a cluster of
|
||||
// co-occurring terms in one page can eat the entire result set and we'd
|
||||
// ship < limit pages. 3x gives dedup enough to pick top N distinct pages.
|
||||
const innerLimit = Math.min(limit * 3, MAX_SEARCH_LIMIT * 3);
|
||||
|
||||
// Search-only timeout: prevents DoS via expensive queries without
|
||||
// affecting long-running operations like embed --all or bulk import.
|
||||
@@ -246,87 +233,32 @@ export class PostgresEngine implements BrainEngine {
|
||||
// `SET statement_timeout = 0` — disables the guard for them.
|
||||
const rows = await sql.begin(async sql => {
|
||||
await sql`SET LOCAL statement_timeout = '8s'`;
|
||||
// CTE chain: rank chunks by FTS → DISTINCT ON (slug) to pick best
|
||||
// chunk per page → order by score → limit. The external shape is
|
||||
// page-grain; chunk-grain ranking wins because A4 weights mean
|
||||
// doc-comment hits (and, once Layer 5 populates them, qualified
|
||||
// symbol hits) beat body-text hits at the chunk level.
|
||||
// CTE: rank pages by FTS score, then pick the best chunk per page in SQL
|
||||
return await sql`
|
||||
WITH ranked_chunks AS (
|
||||
SELECT
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('english', ${query})) AS score
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('english', ${query})
|
||||
WITH ranked_pages AS (
|
||||
SELECT p.id, p.slug, p.title, p.type,
|
||||
ts_rank(p.search_vector, websearch_to_tsquery('english', ${query})) AS score
|
||||
FROM pages p
|
||||
WHERE p.search_vector @@ websearch_to_tsquery('english', ${query})
|
||||
${type ? sql`AND p.type = ${type}` : sql``}
|
||||
${excludeSlugs?.length ? sql`AND p.slug != ALL(${excludeSlugs})` : sql``}
|
||||
${detailLow ? sql`AND cc.chunk_source = 'compiled_truth'` : sql``}
|
||||
${language ? sql`AND cc.language = ${language}` : sql``}
|
||||
${symbolKind ? sql`AND cc.symbol_type = ${symbolKind}` : sql``}
|
||||
ORDER BY score DESC
|
||||
LIMIT ${innerLimit}
|
||||
LIMIT ${limit}
|
||||
OFFSET ${offset}
|
||||
),
|
||||
best_per_page AS (
|
||||
SELECT DISTINCT ON (slug) *
|
||||
FROM ranked_chunks
|
||||
ORDER BY slug, score DESC
|
||||
best_chunks AS (
|
||||
SELECT DISTINCT ON (rp.slug)
|
||||
rp.slug, rp.id as page_id, rp.title, rp.type, rp.score,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source
|
||||
FROM ranked_pages rp
|
||||
JOIN content_chunks cc ON cc.page_id = rp.id
|
||||
${detailLow ? sql`WHERE cc.chunk_source = 'compiled_truth'` : sql``}
|
||||
ORDER BY rp.slug, cc.chunk_index
|
||||
)
|
||||
SELECT slug, page_id, title, type, source_id,
|
||||
chunk_id, chunk_index, chunk_text, chunk_source, score,
|
||||
SELECT slug, page_id, title, type, chunk_id, chunk_index, chunk_text, chunk_source, score,
|
||||
false AS stale
|
||||
FROM best_per_page
|
||||
FROM best_chunks
|
||||
ORDER BY score DESC
|
||||
LIMIT ${limit}
|
||||
OFFSET ${offset}
|
||||
`;
|
||||
});
|
||||
return rows.map(rowToSearchResult);
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.20.0 Cathedral II Layer 3 (1b) chunk-grain keyword search.
|
||||
* Ranks chunks via content_chunks.search_vector WITHOUT the
|
||||
* dedup-to-page pass searchKeyword applies. Used by A2 two-pass
|
||||
* retrieval (Layer 7) as the anchor-discovery primitive.
|
||||
*
|
||||
* Most callers should prefer searchKeyword (external page-grain
|
||||
* contract). This is intentionally a narrow internal knob.
|
||||
*/
|
||||
async searchKeywordChunks(query: string, opts?: SearchOpts): Promise<SearchResult[]> {
|
||||
const sql = this.sql;
|
||||
const limit = clampSearchLimit(opts?.limit);
|
||||
const offset = opts?.offset || 0;
|
||||
const type = opts?.type;
|
||||
const excludeSlugs = opts?.exclude_slugs;
|
||||
const detailLow = opts?.detail === 'low';
|
||||
const language = opts?.language;
|
||||
const symbolKind = opts?.symbolKind;
|
||||
|
||||
if (opts?.limit && opts.limit > MAX_SEARCH_LIMIT) {
|
||||
console.warn(`[gbrain] Warning: search limit clamped from ${opts.limit} to ${MAX_SEARCH_LIMIT}`);
|
||||
}
|
||||
|
||||
const rows = await sql.begin(async sql => {
|
||||
await sql`SET LOCAL statement_timeout = '8s'`;
|
||||
return await sql`
|
||||
SELECT
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('english', ${query})) AS score,
|
||||
false AS stale
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('english', ${query})
|
||||
${type ? sql`AND p.type = ${type}` : sql``}
|
||||
${excludeSlugs?.length ? sql`AND p.slug != ALL(${excludeSlugs})` : sql``}
|
||||
${detailLow ? sql`AND cc.chunk_source = 'compiled_truth'` : sql``}
|
||||
${language ? sql`AND cc.language = ${language}` : sql``}
|
||||
${symbolKind ? sql`AND cc.symbol_type = ${symbolKind}` : sql``}
|
||||
ORDER BY score DESC
|
||||
LIMIT ${limit}
|
||||
OFFSET ${offset}
|
||||
`;
|
||||
});
|
||||
return rows.map(rowToSearchResult);
|
||||
@@ -339,8 +271,6 @@ export class PostgresEngine implements BrainEngine {
|
||||
const type = opts?.type;
|
||||
const excludeSlugs = opts?.exclude_slugs;
|
||||
const detailLow = opts?.detail === 'low';
|
||||
const language = opts?.language;
|
||||
const symbolKind = opts?.symbolKind;
|
||||
|
||||
if (opts?.limit && opts.limit > MAX_SEARCH_LIMIT) {
|
||||
console.warn(`[gbrain] Warning: search limit clamped from ${opts.limit} to ${MAX_SEARCH_LIMIT}`);
|
||||
@@ -365,8 +295,6 @@ export class PostgresEngine implements BrainEngine {
|
||||
${detailLow ? sql`AND cc.chunk_source = 'compiled_truth'` : sql``}
|
||||
${type ? sql`AND p.type = ${type}` : sql``}
|
||||
${excludeSlugs?.length ? sql`AND p.slug != ALL(${excludeSlugs})` : sql``}
|
||||
${language ? sql`AND cc.language = ${language}` : sql``}
|
||||
${symbolKind ? sql`AND cc.symbol_type = ${symbolKind}` : sql``}
|
||||
ORDER BY cc.embedding <=> ${vecStr}::vector
|
||||
LIMIT ${limit}
|
||||
OFFSET ${offset}
|
||||
@@ -408,14 +336,9 @@ export class PostgresEngine implements BrainEngine {
|
||||
return;
|
||||
}
|
||||
|
||||
// Batch upsert: build a single multi-row INSERT ON CONFLICT statement.
|
||||
// v0.19.0: includes language/symbol_name/symbol_type/start_line/end_line
|
||||
// so code chunks carry tree-sitter metadata into the DB. Markdown chunks
|
||||
// pass NULL for all five.
|
||||
// v0.20.0 Cathedral II Layer 6: adds parent_symbol_path / doc_comment /
|
||||
// symbol_name_qualified so nested-chunk emission (A3) can round-trip
|
||||
// scope metadata through upserts.
|
||||
const cols = '(page_id, chunk_index, chunk_text, chunk_source, embedding, model, token_count, embedded_at, language, symbol_name, symbol_type, start_line, end_line, parent_symbol_path, doc_comment, symbol_name_qualified)';
|
||||
// Batch upsert: build a single multi-row INSERT ON CONFLICT statement
|
||||
// This avoids per-row round-trips and reduces lock contention under parallel workers
|
||||
const cols = '(page_id, chunk_index, chunk_text, chunk_source, embedding, model, token_count, embedded_at)';
|
||||
const rows: string[] = [];
|
||||
const params: unknown[] = [];
|
||||
let paramIdx = 1;
|
||||
@@ -424,28 +347,13 @@ export class PostgresEngine implements BrainEngine {
|
||||
const embeddingStr = chunk.embedding
|
||||
? '[' + Array.from(chunk.embedding).join(',') + ']'
|
||||
: null;
|
||||
const parentPath = chunk.parent_symbol_path && chunk.parent_symbol_path.length > 0
|
||||
? chunk.parent_symbol_path
|
||||
: null;
|
||||
|
||||
if (embeddingStr) {
|
||||
rows.push(`($${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}::vector, $${paramIdx++}, $${paramIdx++}, now(), $${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}::text[], $${paramIdx++}, $${paramIdx++})`);
|
||||
params.push(
|
||||
pageId, chunk.chunk_index, chunk.chunk_text, chunk.chunk_source,
|
||||
embeddingStr, chunk.model || 'text-embedding-3-large', chunk.token_count || null,
|
||||
chunk.language || null, chunk.symbol_name || null, chunk.symbol_type || null,
|
||||
chunk.start_line ?? null, chunk.end_line ?? null,
|
||||
parentPath, chunk.doc_comment || null, chunk.symbol_name_qualified || null,
|
||||
);
|
||||
rows.push(`($${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}::vector, $${paramIdx++}, $${paramIdx++}, now())`);
|
||||
params.push(pageId, chunk.chunk_index, chunk.chunk_text, chunk.chunk_source, embeddingStr, chunk.model || 'text-embedding-3-large', chunk.token_count || null);
|
||||
} else {
|
||||
rows.push(`($${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}, NULL, $${paramIdx++}, $${paramIdx++}, NULL, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}::text[], $${paramIdx++}, $${paramIdx++})`);
|
||||
params.push(
|
||||
pageId, chunk.chunk_index, chunk.chunk_text, chunk.chunk_source,
|
||||
chunk.model || 'text-embedding-3-large', chunk.token_count || null,
|
||||
chunk.language || null, chunk.symbol_name || null, chunk.symbol_type || null,
|
||||
chunk.start_line ?? null, chunk.end_line ?? null,
|
||||
parentPath, chunk.doc_comment || null, chunk.symbol_name_qualified || null,
|
||||
);
|
||||
rows.push(`($${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}, NULL, $${paramIdx++}, $${paramIdx++}, NULL)`);
|
||||
params.push(pageId, chunk.chunk_index, chunk.chunk_text, chunk.chunk_source, chunk.model || 'text-embedding-3-large', chunk.token_count || null);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -458,15 +366,7 @@ export class PostgresEngine implements BrainEngine {
|
||||
embedding = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.embedding ELSE COALESCE(EXCLUDED.embedding, content_chunks.embedding) END,
|
||||
model = COALESCE(EXCLUDED.model, content_chunks.model),
|
||||
token_count = EXCLUDED.token_count,
|
||||
embedded_at = COALESCE(EXCLUDED.embedded_at, content_chunks.embedded_at),
|
||||
language = EXCLUDED.language,
|
||||
symbol_name = EXCLUDED.symbol_name,
|
||||
symbol_type = EXCLUDED.symbol_type,
|
||||
start_line = EXCLUDED.start_line,
|
||||
end_line = EXCLUDED.end_line,
|
||||
parent_symbol_path = EXCLUDED.parent_symbol_path,
|
||||
doc_comment = EXCLUDED.doc_comment,
|
||||
symbol_name_qualified = EXCLUDED.symbol_name_qualified`,
|
||||
embedded_at = COALESCE(EXCLUDED.embedded_at, content_chunks.embedded_at)`,
|
||||
params as Parameters<typeof sql.unsafe>[1],
|
||||
);
|
||||
}
|
||||
@@ -1209,170 +1109,4 @@ export class PostgresEngine implements BrainEngine {
|
||||
const conn = this.sql;
|
||||
return conn.unsafe(sql, params as Parameters<typeof conn.unsafe>[1]) as unknown as T[];
|
||||
}
|
||||
|
||||
// ============================================================
|
||||
// v0.20.0 Cathedral II: code edges (Layer 1 stubs — filled by Layer 5)
|
||||
// ============================================================
|
||||
// Declared here so the interface contract is satisfied and consumers can
|
||||
// import against them. Implementations throw until the edge extractor +
|
||||
// per-lang tree-sitter queries land in Layer 5/6.
|
||||
// ============================================================
|
||||
|
||||
async addCodeEdges(edges: import('./types.ts').CodeEdgeInput[]): Promise<number> {
|
||||
if (edges.length === 0) return 0;
|
||||
const sql = this.sql;
|
||||
let inserted = 0;
|
||||
const resolved = edges.filter(e => e.to_chunk_id != null);
|
||||
const unresolved = edges.filter(e => e.to_chunk_id == null);
|
||||
|
||||
if (resolved.length > 0) {
|
||||
const fromIds = resolved.map(e => e.from_chunk_id);
|
||||
const toIds = resolved.map(e => e.to_chunk_id as number);
|
||||
const fromQual = resolved.map(e => e.from_symbol_qualified);
|
||||
const toQual = resolved.map(e => e.to_symbol_qualified);
|
||||
const edgeTypes = resolved.map(e => e.edge_type);
|
||||
const metas = resolved.map(e => JSON.stringify(e.edge_metadata ?? {}));
|
||||
const sources = resolved.map(e => e.source_id ?? null);
|
||||
const res = await sql`
|
||||
INSERT INTO code_edges_chunk (from_chunk_id, to_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id)
|
||||
SELECT * FROM unnest(
|
||||
${fromIds}::int[], ${toIds}::int[],
|
||||
${fromQual}::text[], ${toQual}::text[],
|
||||
${edgeTypes}::text[], ${metas}::jsonb[],
|
||||
${sources}::text[]
|
||||
)
|
||||
ON CONFLICT (from_chunk_id, to_chunk_id, edge_type) DO NOTHING
|
||||
`;
|
||||
inserted += (res as unknown as { count: number }).count ?? 0;
|
||||
}
|
||||
|
||||
if (unresolved.length > 0) {
|
||||
const fromIds = unresolved.map(e => e.from_chunk_id);
|
||||
const fromQual = unresolved.map(e => e.from_symbol_qualified);
|
||||
const toQual = unresolved.map(e => e.to_symbol_qualified);
|
||||
const edgeTypes = unresolved.map(e => e.edge_type);
|
||||
const metas = unresolved.map(e => JSON.stringify(e.edge_metadata ?? {}));
|
||||
const sources = unresolved.map(e => e.source_id ?? null);
|
||||
const res = await sql`
|
||||
INSERT INTO code_edges_symbol (from_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id)
|
||||
SELECT * FROM unnest(
|
||||
${fromIds}::int[],
|
||||
${fromQual}::text[], ${toQual}::text[],
|
||||
${edgeTypes}::text[], ${metas}::jsonb[],
|
||||
${sources}::text[]
|
||||
)
|
||||
ON CONFLICT (from_chunk_id, to_symbol_qualified, edge_type) DO NOTHING
|
||||
`;
|
||||
inserted += (res as unknown as { count: number }).count ?? 0;
|
||||
}
|
||||
|
||||
return inserted;
|
||||
}
|
||||
|
||||
async deleteCodeEdgesForChunks(chunkIds: number[]): Promise<void> {
|
||||
if (chunkIds.length === 0) return;
|
||||
const sql = this.sql;
|
||||
await sql`DELETE FROM code_edges_chunk WHERE from_chunk_id = ANY(${chunkIds}::int[]) OR to_chunk_id = ANY(${chunkIds}::int[])`;
|
||||
await sql`DELETE FROM code_edges_symbol WHERE from_chunk_id = ANY(${chunkIds}::int[])`;
|
||||
}
|
||||
|
||||
async getCallersOf(
|
||||
qualifiedName: string,
|
||||
opts?: { sourceId?: string; allSources?: boolean; limit?: number },
|
||||
): Promise<import('./types.ts').CodeEdgeResult[]> {
|
||||
const sql = this.sql;
|
||||
const limit = Math.min(opts?.limit ?? 100, 500);
|
||||
const scopedSource: string | null =
|
||||
!opts?.allSources && opts?.sourceId ? opts.sourceId : null;
|
||||
const rows = await sql`
|
||||
SELECT id, from_chunk_id, to_chunk_id, from_symbol_qualified, to_symbol_qualified,
|
||||
edge_type, edge_metadata, source_id, true as resolved
|
||||
FROM code_edges_chunk
|
||||
WHERE to_symbol_qualified = ${qualifiedName}
|
||||
${scopedSource ? sql`AND source_id = ${scopedSource}` : sql``}
|
||||
UNION ALL
|
||||
SELECT id, from_chunk_id, NULL::int as to_chunk_id, from_symbol_qualified, to_symbol_qualified,
|
||||
edge_type, edge_metadata, source_id, false as resolved
|
||||
FROM code_edges_symbol
|
||||
WHERE to_symbol_qualified = ${qualifiedName}
|
||||
${scopedSource ? sql`AND source_id = ${scopedSource}` : sql``}
|
||||
LIMIT ${limit}
|
||||
`;
|
||||
return rows.map(r => pgRowToCodeEdge(r as Record<string, unknown>));
|
||||
}
|
||||
|
||||
async getCalleesOf(
|
||||
qualifiedName: string,
|
||||
opts?: { sourceId?: string; allSources?: boolean; limit?: number },
|
||||
): Promise<import('./types.ts').CodeEdgeResult[]> {
|
||||
const sql = this.sql;
|
||||
const limit = Math.min(opts?.limit ?? 100, 500);
|
||||
const scopedSource: string | null =
|
||||
!opts?.allSources && opts?.sourceId ? opts.sourceId : null;
|
||||
const rows = await sql`
|
||||
SELECT id, from_chunk_id, to_chunk_id, from_symbol_qualified, to_symbol_qualified,
|
||||
edge_type, edge_metadata, source_id, true as resolved
|
||||
FROM code_edges_chunk
|
||||
WHERE from_symbol_qualified = ${qualifiedName}
|
||||
${scopedSource ? sql`AND source_id = ${scopedSource}` : sql``}
|
||||
UNION ALL
|
||||
SELECT id, from_chunk_id, NULL::int as to_chunk_id, from_symbol_qualified, to_symbol_qualified,
|
||||
edge_type, edge_metadata, source_id, false as resolved
|
||||
FROM code_edges_symbol
|
||||
WHERE from_symbol_qualified = ${qualifiedName}
|
||||
${scopedSource ? sql`AND source_id = ${scopedSource}` : sql``}
|
||||
LIMIT ${limit}
|
||||
`;
|
||||
return rows.map(r => pgRowToCodeEdge(r as Record<string, unknown>));
|
||||
}
|
||||
|
||||
async getEdgesByChunk(
|
||||
chunkId: number,
|
||||
opts?: { direction?: 'in' | 'out' | 'both'; edgeType?: string; limit?: number },
|
||||
): Promise<import('./types.ts').CodeEdgeResult[]> {
|
||||
const sql = this.sql;
|
||||
const direction = opts?.direction ?? 'both';
|
||||
const limit = Math.min(opts?.limit ?? 50, 200);
|
||||
const typeFilter = opts?.edgeType;
|
||||
|
||||
const chunkRows = await sql`
|
||||
SELECT id, from_chunk_id, to_chunk_id, from_symbol_qualified, to_symbol_qualified,
|
||||
edge_type, edge_metadata, source_id, true as resolved
|
||||
FROM code_edges_chunk
|
||||
WHERE
|
||||
${direction === 'in' ? sql`to_chunk_id = ${chunkId}`
|
||||
: direction === 'out' ? sql`from_chunk_id = ${chunkId}`
|
||||
: sql`(from_chunk_id = ${chunkId} OR to_chunk_id = ${chunkId})`}
|
||||
${typeFilter ? sql`AND edge_type = ${typeFilter}` : sql``}
|
||||
LIMIT ${limit}
|
||||
`;
|
||||
let symbolRows: unknown[] = [];
|
||||
if (direction !== 'in') {
|
||||
const sRows = await sql`
|
||||
SELECT id, from_chunk_id, NULL::int as to_chunk_id, from_symbol_qualified, to_symbol_qualified,
|
||||
edge_type, edge_metadata, source_id, false as resolved
|
||||
FROM code_edges_symbol
|
||||
WHERE from_chunk_id = ${chunkId}
|
||||
${typeFilter ? sql`AND edge_type = ${typeFilter}` : sql``}
|
||||
LIMIT ${limit}
|
||||
`;
|
||||
symbolRows = [...sRows];
|
||||
}
|
||||
return [...chunkRows, ...symbolRows].map(r => pgRowToCodeEdge(r as Record<string, unknown>));
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
function pgRowToCodeEdge(row: Record<string, unknown>): import('./types.ts').CodeEdgeResult {
|
||||
return {
|
||||
id: row.id as number,
|
||||
from_chunk_id: row.from_chunk_id as number,
|
||||
to_chunk_id: row.to_chunk_id == null ? null : (row.to_chunk_id as number),
|
||||
from_symbol_qualified: (row.from_symbol_qualified as string) ?? '',
|
||||
to_symbol_qualified: (row.to_symbol_qualified as string) ?? '',
|
||||
edge_type: (row.edge_type as string) ?? '',
|
||||
edge_metadata: (row.edge_metadata as Record<string, unknown>) ?? {},
|
||||
source_id: row.source_id == null ? null : (row.source_id as string),
|
||||
resolved: Boolean(row.resolved),
|
||||
};
|
||||
}
|
||||
|
||||
+17
-116
@@ -28,18 +28,13 @@ CREATE EXTENSION IF NOT EXISTS pgcrypto;
|
||||
-- - access_policy: forward-compat slot, no enforcement in v0.17.
|
||||
-- Write-side lockdown: mutated only when ctx.remote=false.
|
||||
CREATE TABLE IF NOT EXISTS sources (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL UNIQUE,
|
||||
local_path TEXT,
|
||||
last_commit TEXT,
|
||||
last_sync_at TIMESTAMPTZ,
|
||||
config JSONB NOT NULL DEFAULT '{}'::jsonb,
|
||||
-- v0.20.0 Cathedral II (SP-1): chunker version last used to sync this source.
|
||||
-- performSync forces a full walk when this mismatches CURRENT_CHUNKER_VERSION,
|
||||
-- bypassing the git-HEAD up_to_date early-return so CHUNKER_VERSION bumps
|
||||
-- actually trigger re-chunking on upgrade.
|
||||
chunker_version TEXT,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL UNIQUE,
|
||||
local_path TEXT,
|
||||
last_commit TEXT,
|
||||
last_sync_at TIMESTAMPTZ,
|
||||
config JSONB NOT NULL DEFAULT '{}'::jsonb,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
-- Seed the default source. 'default' is federated=true for backward compat
|
||||
@@ -64,10 +59,6 @@ CREATE TABLE IF NOT EXISTS pages (
|
||||
REFERENCES sources(id) ON DELETE CASCADE,
|
||||
slug TEXT NOT NULL,
|
||||
type TEXT NOT NULL,
|
||||
-- v0.19.0: distinguishes markdown vs code pages at the DB level.
|
||||
-- Drives orphans filter, auto-link bypass, and \`query --lang\`.
|
||||
page_kind TEXT NOT NULL DEFAULT 'markdown'
|
||||
CHECK (page_kind IN ('markdown','code')),
|
||||
title TEXT NOT NULL,
|
||||
compiled_truth TEXT NOT NULL DEFAULT '',
|
||||
timeline TEXT NOT NULL DEFAULT '',
|
||||
@@ -90,111 +81,21 @@ CREATE INDEX IF NOT EXISTS idx_pages_source_id ON pages(source_id);
|
||||
-- content_chunks: chunked content with embeddings
|
||||
-- ============================================================
|
||||
CREATE TABLE IF NOT EXISTS content_chunks (
|
||||
id SERIAL PRIMARY KEY,
|
||||
page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
|
||||
chunk_index INTEGER NOT NULL,
|
||||
chunk_text TEXT NOT NULL,
|
||||
chunk_source TEXT NOT NULL DEFAULT 'compiled_truth',
|
||||
embedding vector(1536),
|
||||
model TEXT NOT NULL DEFAULT 'text-embedding-3-large',
|
||||
token_count INTEGER,
|
||||
embedded_at TIMESTAMPTZ,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
-- v0.19.0: code chunk metadata. Nullable — markdown chunks leave these NULL.
|
||||
-- Powers \`query --lang\`, \`code-def <symbol>\`, and \`code-refs <symbol>\`.
|
||||
language TEXT,
|
||||
symbol_name TEXT,
|
||||
symbol_type TEXT,
|
||||
start_line INTEGER,
|
||||
end_line INTEGER,
|
||||
-- v0.20.0 Cathedral II: qualified symbol identity + parent scope + doc-comment
|
||||
-- + chunk-grain FTS. All nullable — markdown chunks leave these NULL.
|
||||
parent_symbol_path TEXT[],
|
||||
doc_comment TEXT,
|
||||
symbol_name_qualified TEXT,
|
||||
search_vector TSVECTOR
|
||||
id SERIAL PRIMARY KEY,
|
||||
page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
|
||||
chunk_index INTEGER NOT NULL,
|
||||
chunk_text TEXT NOT NULL,
|
||||
chunk_source TEXT NOT NULL DEFAULT 'compiled_truth',
|
||||
embedding vector(1536),
|
||||
model TEXT NOT NULL DEFAULT 'text-embedding-3-large',
|
||||
token_count INTEGER,
|
||||
embedded_at TIMESTAMPTZ,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_chunks_page_index ON content_chunks(page_id, chunk_index);
|
||||
CREATE INDEX IF NOT EXISTS idx_chunks_page ON content_chunks(page_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_chunks_embedding ON content_chunks USING hnsw (embedding vector_cosine_ops);
|
||||
-- v0.19.0: partial indexes — only code chunks populate these columns.
|
||||
CREATE INDEX IF NOT EXISTS idx_chunks_symbol_name ON content_chunks(symbol_name) WHERE symbol_name IS NOT NULL;
|
||||
CREATE INDEX IF NOT EXISTS idx_chunks_language ON content_chunks(language) WHERE language IS NOT NULL;
|
||||
-- v0.20.0 Cathedral II: GIN index on the new chunk-grain FTS vector.
|
||||
CREATE INDEX IF NOT EXISTS idx_chunks_search_vector ON content_chunks USING GIN(search_vector);
|
||||
CREATE INDEX IF NOT EXISTS idx_chunks_symbol_qualified
|
||||
ON content_chunks(symbol_name_qualified) WHERE symbol_name_qualified IS NOT NULL;
|
||||
|
||||
-- v0.20.0 Cathedral II: chunk-grain FTS trigger.
|
||||
-- Weight 'A' on doc_comment + symbol_name_qualified; weight 'B' on chunk_text.
|
||||
-- NL queries ("how do we handle errors") rank doc-comment hits above body text.
|
||||
-- BEFORE INSERT OR UPDATE OF specific columns — only refires when those change,
|
||||
-- not on every chunk update (e.g., embedding refresh doesn't trigger rebuild).
|
||||
CREATE OR REPLACE FUNCTION update_chunk_search_vector() RETURNS TRIGGER AS \$fn\$
|
||||
BEGIN
|
||||
NEW.search_vector :=
|
||||
setweight(to_tsvector('english', COALESCE(NEW.doc_comment, '')), 'A') ||
|
||||
setweight(to_tsvector('english', COALESCE(NEW.symbol_name_qualified, '')), 'A') ||
|
||||
setweight(to_tsvector('english', COALESCE(NEW.chunk_text, '')), 'B');
|
||||
RETURN NEW;
|
||||
END;
|
||||
\$fn\$ LANGUAGE plpgsql;
|
||||
|
||||
DROP TRIGGER IF EXISTS chunk_search_vector_trigger ON content_chunks;
|
||||
CREATE TRIGGER chunk_search_vector_trigger
|
||||
BEFORE INSERT OR UPDATE OF chunk_text, doc_comment, symbol_name_qualified
|
||||
ON content_chunks
|
||||
FOR EACH ROW EXECUTE FUNCTION update_chunk_search_vector();
|
||||
|
||||
-- ============================================================
|
||||
-- code_edges_chunk + code_edges_symbol: v0.20.0 Cathedral II structural edges
|
||||
-- ============================================================
|
||||
-- Two-table design (codex F4 + SP-7):
|
||||
-- - code_edges_chunk: resolved edges (both endpoints = known chunk IDs)
|
||||
-- - code_edges_symbol: unresolved refs (target known by qualified name,
|
||||
-- defining chunk not yet imported)
|
||||
-- Readers UNION both tables; no promotion step.
|
||||
-- Source scoping: from_chunk_id -> content_chunks -> pages.source_id
|
||||
-- determines the source. Resolution logic MUST scope on source (codex SP-3);
|
||||
-- only --all-sources callers bypass this. UNIQUE keys don't include source_id
|
||||
-- because from_chunk_id already pins it.
|
||||
CREATE TABLE IF NOT EXISTS code_edges_chunk (
|
||||
id SERIAL PRIMARY KEY,
|
||||
from_chunk_id INTEGER NOT NULL REFERENCES content_chunks(id) ON DELETE CASCADE,
|
||||
to_chunk_id INTEGER NOT NULL REFERENCES content_chunks(id) ON DELETE CASCADE,
|
||||
from_symbol_qualified TEXT NOT NULL,
|
||||
to_symbol_qualified TEXT NOT NULL,
|
||||
edge_type TEXT NOT NULL,
|
||||
edge_metadata JSONB NOT NULL DEFAULT '{}',
|
||||
source_id TEXT REFERENCES sources(id) ON DELETE CASCADE,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
CONSTRAINT code_edges_chunk_unique UNIQUE (from_chunk_id, to_chunk_id, edge_type)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_code_edges_chunk_from
|
||||
ON code_edges_chunk(from_chunk_id, edge_type);
|
||||
CREATE INDEX IF NOT EXISTS idx_code_edges_chunk_to
|
||||
ON code_edges_chunk(to_chunk_id, edge_type);
|
||||
CREATE INDEX IF NOT EXISTS idx_code_edges_chunk_to_symbol
|
||||
ON code_edges_chunk(to_symbol_qualified, edge_type);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS code_edges_symbol (
|
||||
id SERIAL PRIMARY KEY,
|
||||
from_chunk_id INTEGER NOT NULL REFERENCES content_chunks(id) ON DELETE CASCADE,
|
||||
from_symbol_qualified TEXT NOT NULL,
|
||||
to_symbol_qualified TEXT NOT NULL,
|
||||
edge_type TEXT NOT NULL,
|
||||
edge_metadata JSONB NOT NULL DEFAULT '{}',
|
||||
source_id TEXT REFERENCES sources(id) ON DELETE CASCADE,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
CONSTRAINT code_edges_symbol_unique UNIQUE (from_chunk_id, to_symbol_qualified, edge_type)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_code_edges_symbol_from
|
||||
ON code_edges_symbol(from_chunk_id, edge_type);
|
||||
CREATE INDEX IF NOT EXISTS idx_code_edges_symbol_to
|
||||
ON code_edges_symbol(to_symbol_qualified, edge_type);
|
||||
|
||||
-- ============================================================
|
||||
-- links: cross-references between pages
|
||||
|
||||
@@ -15,7 +15,6 @@ import type { SearchResult, SearchOpts } from '../types.ts';
|
||||
import { embed } from '../embedding.ts';
|
||||
import { dedupResults } from './dedup.ts';
|
||||
import { autoDetectDetail } from './intent.ts';
|
||||
import { expandAnchors, hydrateChunks } from './two-pass.ts';
|
||||
|
||||
const RRF_K = 60;
|
||||
const COMPILED_TRUTH_BOOST = 2.0;
|
||||
@@ -69,14 +68,7 @@ export async function hybridSearch(
|
||||
|
||||
// Auto-detect detail level from query intent when caller doesn't specify
|
||||
const detail = opts?.detail ?? autoDetectDetail(query);
|
||||
const searchOpts: SearchOpts = {
|
||||
limit: innerLimit,
|
||||
detail,
|
||||
// v0.20.0 Cathedral II Layer 10 — thread language + symbolKind through so
|
||||
// per-engine searchKeyword / searchVector apply the filters at SQL level.
|
||||
language: opts?.language,
|
||||
symbolKind: opts?.symbolKind,
|
||||
};
|
||||
const searchOpts: SearchOpts = { limit: innerLimit, detail };
|
||||
|
||||
if (DEBUG && detail) {
|
||||
console.error(`[search-debug] auto-detail=${detail} for query="${query}"`);
|
||||
@@ -156,52 +148,8 @@ export async function hybridSearch(
|
||||
}
|
||||
}
|
||||
|
||||
// v0.20.0 Cathedral II Layer 7 (A2): two-pass structural expansion.
|
||||
// Default OFF. When opts.walkDepth > 0 OR opts.nearSymbol is set, we
|
||||
// walk code_edges_chunk + code_edges_symbol up to walkDepth hops from
|
||||
// the anchor set (top of `fused`). Expanded neighbors get score decayed
|
||||
// by 1/(1+hop) from their anchor's score and merge back into the pool.
|
||||
//
|
||||
// Dedup per-page cap lifts to min(10, walkDepth * 5) when walking —
|
||||
// structural neighbors from the same file/class are the whole point
|
||||
// of two-pass; clipping them at 2/page defeats A2 (codex F5).
|
||||
const walkDepth = Math.min(opts?.walkDepth ?? 0, 2);
|
||||
const needsExpansion = walkDepth > 0 || Boolean(opts?.nearSymbol);
|
||||
let dedupOpts = opts?.dedupOpts;
|
||||
|
||||
if (needsExpansion) {
|
||||
const anchorSet = fused.slice(0, Math.max(10, limit));
|
||||
try {
|
||||
const expanded = await expandAnchors(engine, anchorSet, {
|
||||
walkDepth,
|
||||
nearSymbol: opts?.nearSymbol,
|
||||
sourceId: opts?.sourceId,
|
||||
});
|
||||
// Resolve new chunk IDs (not already in fused) into full rows.
|
||||
const existingIds = new Set(fused.map(r => r.chunk_id));
|
||||
const newIds = expanded
|
||||
.filter(e => !existingIds.has(e.chunk_id))
|
||||
.map(e => e.chunk_id);
|
||||
if (newIds.length > 0) {
|
||||
const hydrated = await hydrateChunks(engine, newIds);
|
||||
const scoreById = new Map(expanded.map(e => [e.chunk_id, e.score]));
|
||||
for (const r of hydrated) {
|
||||
r.score = scoreById.get(r.chunk_id) ?? 0.01;
|
||||
fused.push(r);
|
||||
}
|
||||
fused.sort((a, b) => b.score - a.score);
|
||||
}
|
||||
// Widen per-page dedup cap when walking.
|
||||
const capFromWalk = Math.min(10, Math.max(walkDepth * 5, 5));
|
||||
dedupOpts = { ...(dedupOpts ?? {}), maxPerPage: capFromWalk };
|
||||
} catch {
|
||||
// Expansion is best-effort — missing edge tables or a transient
|
||||
// DB error must not break base hybrid retrieval.
|
||||
}
|
||||
}
|
||||
|
||||
// Dedup
|
||||
const deduped = dedupResults(fused, dedupOpts);
|
||||
const deduped = dedupResults(fused, opts?.dedupOpts);
|
||||
|
||||
// Auto-escalate: if detail=low returned 0, retry with high
|
||||
if (deduped.length === 0 && opts?.detail === 'low') {
|
||||
|
||||
@@ -1,190 +0,0 @@
|
||||
/**
|
||||
* v0.20.0 Cathedral II Layer 7 (A2) — two-pass structural retrieval.
|
||||
*
|
||||
* Given an anchor set of chunks (either from a keyword/vector anchor
|
||||
* search OR from a --near-symbol qualified-name lookup), walk
|
||||
* code_edges_chunk + code_edges_symbol up to walkDepth hops and collect
|
||||
* structural neighbors. Score each neighbor as anchor_score * 1/(1+hop).
|
||||
*
|
||||
* Default OFF. Activation:
|
||||
* - opts.walkDepth > 0 → walk N hops from the anchors.
|
||||
* - opts.nearSymbol set → anchor set includes chunks whose
|
||||
* symbol_name_qualified matches, in addition to the keyword/vector
|
||||
* anchors.
|
||||
*
|
||||
* Caps (per codex F5 resolution):
|
||||
* - depth capped at 2 (neighborhood blast radius)
|
||||
* - neighbor cap 50 per hop (high-fan-out protection)
|
||||
*
|
||||
* Returns a flat merged list: anchors (score preserved) + neighbors
|
||||
* (scored by 1/(1+hop) * anchor_score). Caller feeds this back into
|
||||
* the RRF-deduped pipeline.
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from '../engine.ts';
|
||||
import type { SearchResult } from '../types.ts';
|
||||
|
||||
const MAX_WALK_DEPTH = 2;
|
||||
const NEIGHBOR_CAP_PER_HOP = 50;
|
||||
|
||||
export interface TwoPassOpts {
|
||||
/** 1 or 2 — capped at 2. 0 or undefined → no-op (returns anchors as-is). */
|
||||
walkDepth?: number;
|
||||
/** When set, find chunks whose symbol_name_qualified matches; add to anchor set. */
|
||||
nearSymbol?: string;
|
||||
/** Filter expansion to one source. When unset, crosses sources. */
|
||||
sourceId?: string;
|
||||
}
|
||||
|
||||
interface ChunkWithScore {
|
||||
chunk_id: number;
|
||||
score: number;
|
||||
hop: number;
|
||||
source: 'anchor' | 'neighbor';
|
||||
}
|
||||
|
||||
/**
|
||||
* Expand an anchor set through structural edges. Returns every chunk
|
||||
* ID that lands in the walk window, keyed with a score that combines
|
||||
* the original anchor score with hop distance decay.
|
||||
*/
|
||||
export async function expandAnchors(
|
||||
engine: BrainEngine,
|
||||
anchors: SearchResult[],
|
||||
opts: TwoPassOpts = {},
|
||||
): Promise<ChunkWithScore[]> {
|
||||
const depth = Math.min(Math.max(opts.walkDepth ?? 0, 0), MAX_WALK_DEPTH);
|
||||
if (depth === 0 && !opts.nearSymbol) {
|
||||
return anchors.map(a => ({
|
||||
chunk_id: a.chunk_id,
|
||||
score: a.score,
|
||||
hop: 0,
|
||||
source: 'anchor' as const,
|
||||
}));
|
||||
}
|
||||
|
||||
const seen = new Map<number, ChunkWithScore>();
|
||||
for (const a of anchors) {
|
||||
seen.set(a.chunk_id, {
|
||||
chunk_id: a.chunk_id,
|
||||
score: a.score,
|
||||
hop: 0,
|
||||
source: 'anchor',
|
||||
});
|
||||
}
|
||||
|
||||
// --near-symbol: add chunks whose symbol_name_qualified matches as
|
||||
// additional anchors. Best-effort — if none found, fall through.
|
||||
if (opts.nearSymbol) {
|
||||
try {
|
||||
const rows = await engine.executeRaw<{ id: number }>(
|
||||
`SELECT id FROM content_chunks WHERE symbol_name_qualified = $1 LIMIT 50`,
|
||||
[opts.nearSymbol],
|
||||
);
|
||||
const baseScore = anchors.length > 0 ? anchors[0]!.score : 1.0;
|
||||
for (const r of rows) {
|
||||
if (!seen.has(r.id)) {
|
||||
seen.set(r.id, { chunk_id: r.id, score: baseScore, hop: 0, source: 'anchor' });
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// Ignore — execution continues without the near-symbol anchors.
|
||||
}
|
||||
}
|
||||
|
||||
// Walk N hops. Frontier advances each iteration; each expansion adds
|
||||
// unseen chunks with decayed scores.
|
||||
let frontier = Array.from(seen.values()).filter(c => c.hop === 0).map(c => c.chunk_id);
|
||||
for (let hop = 1; hop <= depth; hop++) {
|
||||
if (frontier.length === 0) break;
|
||||
const nextFrontier = new Set<number>();
|
||||
const decay = 1 / (1 + hop);
|
||||
|
||||
for (const chunkId of frontier) {
|
||||
const current = seen.get(chunkId);
|
||||
if (!current) continue;
|
||||
|
||||
let edges: import('../types.ts').CodeEdgeResult[] = [];
|
||||
try {
|
||||
edges = await engine.getEdgesByChunk(chunkId, {
|
||||
direction: 'both',
|
||||
limit: NEIGHBOR_CAP_PER_HOP,
|
||||
});
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
|
||||
// Two kinds of neighbors to visit:
|
||||
// 1. Resolved edges with to_chunk_id: direct chunk follow.
|
||||
// 2. Unresolved edges (code_edges_symbol): resolve by
|
||||
// symbol_name_qualified = to_symbol_qualified, then follow.
|
||||
const directChunkIds: number[] = [];
|
||||
const unresolvedTargets: string[] = [];
|
||||
for (const e of edges) {
|
||||
if (e.to_chunk_id != null) directChunkIds.push(e.to_chunk_id);
|
||||
else if (e.to_symbol_qualified) unresolvedTargets.push(e.to_symbol_qualified);
|
||||
}
|
||||
// Resolve unresolved edges by looking up chunks whose
|
||||
// symbol_name_qualified matches. One batch query per frontier node.
|
||||
if (unresolvedTargets.length > 0) {
|
||||
try {
|
||||
const resolved = await engine.executeRaw<{ id: number }>(
|
||||
`SELECT id FROM content_chunks WHERE symbol_name_qualified = ANY($1::text[]) LIMIT ${NEIGHBOR_CAP_PER_HOP}`,
|
||||
[unresolvedTargets],
|
||||
);
|
||||
for (const r of resolved) directChunkIds.push(r.id);
|
||||
} catch {
|
||||
// best-effort
|
||||
}
|
||||
}
|
||||
|
||||
for (const tid of directChunkIds) {
|
||||
if (seen.has(tid)) continue;
|
||||
const nbScore = current.score * decay;
|
||||
seen.set(tid, { chunk_id: tid, score: nbScore, hop, source: 'neighbor' });
|
||||
nextFrontier.add(tid);
|
||||
}
|
||||
}
|
||||
|
||||
frontier = Array.from(nextFrontier);
|
||||
}
|
||||
|
||||
return Array.from(seen.values());
|
||||
}
|
||||
|
||||
/**
|
||||
* Fetch SearchResult rows for a set of chunk IDs. Used to hydrate
|
||||
* two-pass neighbor IDs into full result rows the hybrid pipeline
|
||||
* expects. Missing chunk IDs (chunk deleted between the edge walk
|
||||
* and the fetch) are silently skipped.
|
||||
*/
|
||||
export async function hydrateChunks(
|
||||
engine: BrainEngine,
|
||||
chunkIds: number[],
|
||||
): Promise<SearchResult[]> {
|
||||
if (chunkIds.length === 0) return [];
|
||||
const rows = await engine.executeRaw<{
|
||||
slug: string; page_id: number; title: string; type: string; source_id: string;
|
||||
chunk_id: number; chunk_index: number; chunk_text: string; chunk_source: string;
|
||||
}>(
|
||||
`SELECT p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.id = ANY($1::int[])`,
|
||||
[chunkIds],
|
||||
);
|
||||
return rows.map((r) => ({
|
||||
slug: r.slug,
|
||||
page_id: r.page_id,
|
||||
title: r.title,
|
||||
type: r.type as import('../types.ts').PageType,
|
||||
chunk_text: r.chunk_text,
|
||||
chunk_source: r.chunk_source as 'compiled_truth' | 'timeline',
|
||||
chunk_id: r.chunk_id,
|
||||
chunk_index: r.chunk_index,
|
||||
score: 0, // two-pass caller assigns scores.
|
||||
stale: false,
|
||||
source_id: r.source_id,
|
||||
} as SearchResult));
|
||||
}
|
||||
@@ -186,10 +186,7 @@ function acquireLock(workspace: string, opts: InstallOptions): void {
|
||||
const age = Date.now() - existing.mtimeMs;
|
||||
// `staleMs: 0` in tests means "any age counts as stale". Use >=
|
||||
// so a just-written lock qualifies when the threshold is 0.
|
||||
// Negative age (mtime in the future) happens on fast CI filesystems
|
||||
// where write → stat roundtrip returns an mtime microseconds ahead of
|
||||
// Date.now() — treat it as stale to avoid a "lock held" false positive.
|
||||
const stale = age < 0 || age >= staleMs;
|
||||
const stale = age >= staleMs;
|
||||
if (stale && !opts.forceUnlock) {
|
||||
throw new InstallError(
|
||||
`Stale skillpack lock at ${p} (pid ${existing.pid}, ${Math.round(age / 1000)}s old). Pass --force-unlock to proceed.`,
|
||||
|
||||
+5
-156
@@ -24,56 +24,6 @@ export interface RawManifestEntry {
|
||||
oldPath?: string;
|
||||
}
|
||||
|
||||
export type SyncStrategy = 'markdown' | 'code' | 'auto';
|
||||
|
||||
interface SyncableOptions {
|
||||
strategy?: SyncStrategy;
|
||||
include?: string[];
|
||||
exclude?: string[];
|
||||
}
|
||||
|
||||
// v0.19.0 shipped a 9-extension allowlist (ts/tsx/js/jsx/mjs/cjs/py/rb/go). The
|
||||
// chunker already supports ~35 extensions via detectCodeLanguage but the sync
|
||||
// classifier dropped every other language on the floor — Rust/Java/C#/C++/etc.
|
||||
// files never reached the chunker on a normal repo sync, making v0.19.0's
|
||||
// "165 languages" claim aspirational (codex F1). v0.20.0 Layer 2 (1a) rewrites
|
||||
// isCodeFilePath to delegate to detectCodeLanguage so the sync classifier
|
||||
// matches the chunker's actual coverage.
|
||||
//
|
||||
// Kept as-is for now for `isAllowedByStrategy` fast-path + tests that
|
||||
// structurally reference it. Derived from the chunker's language map at
|
||||
// module load, not hardcoded.
|
||||
const CODE_EXTENSIONS = new Set<string>([
|
||||
'.ts', '.tsx', '.mts', '.cts',
|
||||
'.js', '.jsx', '.mjs', '.cjs',
|
||||
'.py',
|
||||
'.rb',
|
||||
'.go',
|
||||
'.rs',
|
||||
'.java',
|
||||
'.cs',
|
||||
'.cpp', '.cc', '.cxx', '.hpp', '.hxx', '.hh',
|
||||
'.c', '.h',
|
||||
'.php',
|
||||
'.swift',
|
||||
'.kt', '.kts',
|
||||
'.scala', '.sc',
|
||||
'.lua',
|
||||
'.ex', '.exs',
|
||||
'.elm',
|
||||
'.ml', '.mli',
|
||||
'.dart',
|
||||
'.zig',
|
||||
'.sol',
|
||||
'.sh', '.bash',
|
||||
'.css',
|
||||
'.html', '.htm',
|
||||
'.vue',
|
||||
'.json',
|
||||
'.yaml', '.yml',
|
||||
'.toml',
|
||||
]);
|
||||
|
||||
/**
|
||||
* Parse the output of `git diff --name-status -M LAST..HEAD` into structured entries.
|
||||
*
|
||||
@@ -122,68 +72,12 @@ export function buildSyncManifest(gitDiffOutput: string): SyncManifest {
|
||||
return manifest;
|
||||
}
|
||||
|
||||
export function isCodeFilePath(path: string): boolean {
|
||||
const lower = path.toLowerCase();
|
||||
for (const ext of CODE_EXTENSIONS) {
|
||||
if (lower.endsWith(ext)) return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
function isMarkdownFilePath(path: string): boolean {
|
||||
return path.endsWith('.md') || path.endsWith('.mdx');
|
||||
}
|
||||
|
||||
function isAllowedByStrategy(path: string, strategy: SyncStrategy): boolean {
|
||||
if (strategy === 'markdown') return isMarkdownFilePath(path);
|
||||
if (strategy === 'code') return isCodeFilePath(path);
|
||||
return isMarkdownFilePath(path) || isCodeFilePath(path);
|
||||
}
|
||||
|
||||
function globToRegex(pattern: string): RegExp {
|
||||
let regex = '^';
|
||||
for (let i = 0; i < pattern.length; i++) {
|
||||
const ch = pattern[i];
|
||||
if (ch === '*') {
|
||||
const next = pattern[i + 1];
|
||||
if (next === '*') {
|
||||
// `**/` matches zero or more path segments (including zero, so `src/**/*.ts`
|
||||
// matches `src/foo.ts` as well as `src/a/b/foo.ts`). Collapse `**/` →
|
||||
// `(?:.*/)?`. A bare `**` not followed by `/` matches any chars.
|
||||
if (pattern[i + 2] === '/') {
|
||||
regex += '(?:.*/)?';
|
||||
i += 2;
|
||||
} else {
|
||||
regex += '.*';
|
||||
i++;
|
||||
}
|
||||
} else {
|
||||
regex += '[^/]*';
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (ch === '?') { regex += '[^/]'; continue; }
|
||||
if ('\\.[]{}()+-^$|'.includes(ch)) { regex += `\\${ch}`; continue; }
|
||||
regex += ch;
|
||||
}
|
||||
regex += '$';
|
||||
return new RegExp(regex);
|
||||
}
|
||||
|
||||
function matchesAnyGlob(path: string, patterns?: string[]): boolean {
|
||||
if (!patterns || patterns.length === 0) return false;
|
||||
const normalized = path.replace(/\\/g, '/');
|
||||
return patterns.some((pattern) => globToRegex(pattern).test(normalized));
|
||||
}
|
||||
|
||||
/**
|
||||
* Filter a file path to determine if it should be synced to GBrain.
|
||||
* Strategy-aware: 'markdown' (default) = .md/.mdx only, 'code' = code files only, 'auto' = both.
|
||||
*/
|
||||
export function isSyncable(path: string, opts: SyncableOptions = {}): boolean {
|
||||
const strategy = opts.strategy || 'markdown';
|
||||
|
||||
if (!isAllowedByStrategy(path, strategy)) return false;
|
||||
export function isSyncable(path: string): boolean {
|
||||
// Must be .md or .mdx
|
||||
if (!path.endsWith('.md') && !path.endsWith('.mdx')) return false;
|
||||
|
||||
// Skip hidden directories
|
||||
if (path.split('/').some(p => p.startsWith('.'))) return false;
|
||||
@@ -199,9 +93,6 @@ export function isSyncable(path: string, opts: SyncableOptions = {}): boolean {
|
||||
// Skip ops/ directory
|
||||
if (path.startsWith('ops/')) return false;
|
||||
|
||||
if (opts.include && opts.include.length > 0 && !matchesAnyGlob(path, opts.include)) return false;
|
||||
if (opts.exclude && opts.exclude.length > 0 && matchesAnyGlob(path, opts.exclude)) return false;
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
@@ -234,57 +125,15 @@ export function slugifyPath(filePath: string): string {
|
||||
return path.split('/').map(slugifySegment).filter(Boolean).join('/');
|
||||
}
|
||||
|
||||
/**
|
||||
* Slugify a code file path: flatten into a single slug segment with dots → hyphens.
|
||||
* e.g. 'src/core/chunkers/code.ts' → 'src-core-chunkers-code-ts'
|
||||
*/
|
||||
export function slugifyCodePath(filePath: string): string {
|
||||
let path = filePath.replace(/\\/g, '/');
|
||||
path = path.replace(/^\.?\//, '');
|
||||
return path
|
||||
.split('/')
|
||||
.map(segment => slugifySegment(segment.replace(/\./g, '-')))
|
||||
.filter(Boolean)
|
||||
.join('-');
|
||||
}
|
||||
|
||||
/**
|
||||
* Convert a repo-relative file path to a GBrain page slug.
|
||||
*/
|
||||
export function pathToSlug(
|
||||
filePath: string,
|
||||
repoPrefix?: string,
|
||||
options: { pageKind?: 'markdown' | 'code' } = {},
|
||||
): string {
|
||||
const pageKind = options.pageKind || 'markdown';
|
||||
let slug = pageKind === 'code' ? slugifyCodePath(filePath) : slugifyPath(filePath);
|
||||
export function pathToSlug(filePath: string, repoPrefix?: string): string {
|
||||
let slug = slugifyPath(filePath);
|
||||
if (repoPrefix) slug = `${repoPrefix}/${slug}`;
|
||||
return slug.toLowerCase();
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.20.0 Cathedral II Layer 1a (SP-5 fix) — centralized slug dispatcher.
|
||||
*
|
||||
* Before Cathedral II, `importFromFile` / `importCodeFile` chose between
|
||||
* `slugifyPath` and `slugifyCodePath` inline, but the sync delete/rename
|
||||
* paths in `performSync` always called `pathToSlug(path)` with the default
|
||||
* pageKind='markdown'. For a 9-extension-wide code classifier this was
|
||||
* mostly correct (code files were rare), but Layer 1a widens the classifier
|
||||
* to ~35 extensions and without this dispatcher, deleting or renaming a
|
||||
* Rust/Java/Ruby/etc. file would try to delete the wrong slug (the
|
||||
* markdown-style slug) and leave the real code-slug page orphaned forever.
|
||||
*
|
||||
* Every sync-path caller that used to pick a pageKind manually should now
|
||||
* call resolveSlugForPath — it derives the right slug shape from
|
||||
* isCodeFilePath(), which in turn derives from the chunker's language map.
|
||||
* Central dispatch means new extensions added to the chunker automatically
|
||||
* flow through without touching the sync code path.
|
||||
*/
|
||||
export function resolveSlugForPath(filePath: string, repoPrefix?: string): string {
|
||||
const pageKind = isCodeFilePath(filePath) ? 'code' : 'markdown';
|
||||
return pathToSlug(filePath, repoPrefix, { pageKind });
|
||||
}
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
// Sync failure tracking — Bug 9
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
|
||||
+3
-107
@@ -3,9 +3,7 @@
|
||||
// ingest (and the amara-life-v1 eval corpus in the sibling gbrain-evals repo).
|
||||
// Previously these collapsed into `source`, which lost workflow semantics
|
||||
// (e.g. "attended meetings" vs "received emails").
|
||||
// `code` (v0.19.0): tree-sitter-chunked source files; consumed by code-def /
|
||||
// code-refs / code-callers / code-callees + Cathedral II two-pass retrieval.
|
||||
export type PageType = 'person' | 'company' | 'deal' | 'yc' | 'civic' | 'project' | 'concept' | 'source' | 'media' | 'writing' | 'analysis' | 'guide' | 'hardware' | 'architecture' | 'meeting' | 'note' | 'email' | 'slack' | 'calendar-event' | 'code';
|
||||
export type PageType = 'person' | 'company' | 'deal' | 'yc' | 'civic' | 'project' | 'concept' | 'source' | 'media' | 'writing' | 'analysis' | 'guide' | 'hardware' | 'architecture' | 'meeting' | 'note' | 'email' | 'slack' | 'calendar-event';
|
||||
|
||||
export interface Page {
|
||||
id: number;
|
||||
@@ -20,8 +18,6 @@ export interface Page {
|
||||
updated_at: Date;
|
||||
}
|
||||
|
||||
export type PageKind = 'markdown' | 'code';
|
||||
|
||||
export interface PageInput {
|
||||
type: PageType;
|
||||
title: string;
|
||||
@@ -29,13 +25,6 @@ export interface PageInput {
|
||||
timeline?: string;
|
||||
frontmatter?: Record<string, unknown>;
|
||||
content_hash?: string;
|
||||
/**
|
||||
* v0.19.0: distinguishes markdown vs code pages at the DB level. Defaults
|
||||
* to 'markdown' when omitted so existing callers work unchanged. Set to
|
||||
* 'code' by importCodeFile; drives orphans filter, auto-link bypass, and
|
||||
* `query --lang` filtering.
|
||||
*/
|
||||
page_kind?: PageKind;
|
||||
}
|
||||
|
||||
export interface PageFilters {
|
||||
@@ -53,48 +42,20 @@ export interface Chunk {
|
||||
page_id: number;
|
||||
chunk_index: number;
|
||||
chunk_text: string;
|
||||
chunk_source: 'compiled_truth' | 'timeline' | 'fenced_code';
|
||||
chunk_source: 'compiled_truth' | 'timeline';
|
||||
embedding: Float32Array | null;
|
||||
model: string;
|
||||
token_count: number | null;
|
||||
embedded_at: Date | null;
|
||||
/** v0.19.0 code metadata (NULL for markdown chunks). */
|
||||
language?: string | null;
|
||||
symbol_name?: string | null;
|
||||
symbol_type?: string | null;
|
||||
start_line?: number | null;
|
||||
end_line?: number | null;
|
||||
/** v0.20.0 Cathedral II (NULL for markdown chunks). */
|
||||
parent_symbol_path?: string[] | null;
|
||||
doc_comment?: string | null;
|
||||
symbol_name_qualified?: string | null;
|
||||
}
|
||||
|
||||
export interface ChunkInput {
|
||||
chunk_index: number;
|
||||
chunk_text: string;
|
||||
chunk_source: 'compiled_truth' | 'timeline' | 'fenced_code';
|
||||
chunk_source: 'compiled_truth' | 'timeline';
|
||||
embedding?: Float32Array;
|
||||
model?: string;
|
||||
token_count?: number;
|
||||
/**
|
||||
* v0.19.0: optional code-chunk metadata. Populated by importCodeFile from
|
||||
* the tree-sitter AST; NULL for markdown chunks. Drives `query --lang`,
|
||||
* `code-def`, `code-refs`, and the new searchCodeChunks engine method.
|
||||
*/
|
||||
language?: string;
|
||||
symbol_name?: string;
|
||||
symbol_type?: string;
|
||||
start_line?: number;
|
||||
end_line?: number;
|
||||
/**
|
||||
* v0.20.0 Cathedral II: qualified symbol identity + parent scope +
|
||||
* doc-comment. All populated by importCodeFile from the AST (Layer 5/6);
|
||||
* NULL for markdown chunks unless D2 fence extraction populated them.
|
||||
*/
|
||||
parent_symbol_path?: string[];
|
||||
doc_comment?: string;
|
||||
symbol_name_qualified?: string;
|
||||
}
|
||||
|
||||
// Search
|
||||
@@ -123,71 +84,6 @@ export interface SearchOpts {
|
||||
type?: PageType;
|
||||
exclude_slugs?: string[];
|
||||
detail?: 'low' | 'medium' | 'high';
|
||||
/**
|
||||
* v0.20.0 Cathedral II: filter by content_chunks.language (e.g., 'typescript',
|
||||
* 'python', 'ruby'). Used by `gbrain query --lang <lang>`. NULL/undefined
|
||||
* returns all languages.
|
||||
*/
|
||||
language?: string;
|
||||
/**
|
||||
* v0.20.0 Cathedral II: filter by content_chunks.symbol_type (e.g., 'function',
|
||||
* 'class', 'method', 'type', 'interface'). Used by `gbrain query --symbol-kind`.
|
||||
*/
|
||||
symbolKind?: string;
|
||||
/**
|
||||
* v0.20.0 Cathedral II: anchor the two-pass retrieval at a specific qualified
|
||||
* symbol name. Pairs with walkDepth. Used by `gbrain query --near-symbol`.
|
||||
*/
|
||||
nearSymbol?: string;
|
||||
/**
|
||||
* v0.20.0 Cathedral II: structural walk depth for two-pass retrieval. 0 = off
|
||||
* (default), 1 or 2 = expand that many hops through code_edges_chunk. Capped
|
||||
* at 2 in A2. When walkDepth > 0, dedup's per-page cap lifts to
|
||||
* min(10, walkDepth * 5).
|
||||
*/
|
||||
walkDepth?: number;
|
||||
/**
|
||||
* v0.20.0 Cathedral II: scope search to a specific source. When set,
|
||||
* results are filtered by pages.source_id. Use '__all__' or leave
|
||||
* undefined to search all sources.
|
||||
*/
|
||||
sourceId?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.20.0 Cathedral II: input for addCodeEdges. One row per edge.
|
||||
* from_chunk_id is always known (we're extracting edges from a freshly
|
||||
* imported chunk). to_chunk_id may be null (target symbol not yet
|
||||
* resolved — row lands in code_edges_symbol instead of code_edges_chunk).
|
||||
*/
|
||||
export interface CodeEdgeInput {
|
||||
from_chunk_id: number;
|
||||
/** Resolved target chunk ID. Undefined/null → row lands in code_edges_symbol. */
|
||||
to_chunk_id?: number | null;
|
||||
from_symbol_qualified: string;
|
||||
to_symbol_qualified: string;
|
||||
/** 'calls' | 'imports' | 'extends' | 'implements' | 'mixes_in' | 'type_refs' | 'declares'. */
|
||||
edge_type: string;
|
||||
edge_metadata?: Record<string, unknown>;
|
||||
source_id?: string | null;
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.20.0 Cathedral II: result row from code edge queries (getCallersOf,
|
||||
* getCalleesOf, getEdgesByChunk). `resolved=true` means the row came from
|
||||
* code_edges_chunk (to_chunk_id is a known chunk); `resolved=false` means
|
||||
* code_edges_symbol (to_chunk_id is null).
|
||||
*/
|
||||
export interface CodeEdgeResult {
|
||||
id: number;
|
||||
from_chunk_id: number;
|
||||
to_chunk_id: number | null;
|
||||
from_symbol_qualified: string;
|
||||
to_symbol_qualified: string;
|
||||
edge_type: string;
|
||||
edge_metadata: Record<string, unknown>;
|
||||
source_id: string | null;
|
||||
resolved: boolean;
|
||||
}
|
||||
|
||||
// Links
|
||||
|
||||
+1
-11
@@ -116,21 +116,11 @@ export function rowToChunk(row: Record<string, unknown>, includeEmbedding = fals
|
||||
page_id: row.page_id as number,
|
||||
chunk_index: row.chunk_index as number,
|
||||
chunk_text: row.chunk_text as string,
|
||||
chunk_source: row.chunk_source as 'compiled_truth' | 'timeline' | 'fenced_code',
|
||||
chunk_source: row.chunk_source as 'compiled_truth' | 'timeline',
|
||||
embedding: includeEmbedding ? parseEmbedding(row.embedding) : null,
|
||||
model: row.model as string,
|
||||
token_count: row.token_count as number | null,
|
||||
embedded_at: row.embedded_at ? new Date(row.embedded_at as string) : null,
|
||||
// v0.19.0 code-chunk metadata (nullable for markdown chunks).
|
||||
language: (row.language as string | null | undefined) ?? null,
|
||||
symbol_name: (row.symbol_name as string | null | undefined) ?? null,
|
||||
symbol_type: (row.symbol_type as string | null | undefined) ?? null,
|
||||
start_line: (row.start_line as number | null | undefined) ?? null,
|
||||
end_line: (row.end_line as number | null | undefined) ?? null,
|
||||
// v0.20.0 Cathedral II Layer 1 additions (nullable for markdown chunks).
|
||||
parent_symbol_path: (row.parent_symbol_path as string[] | null | undefined) ?? null,
|
||||
doc_comment: (row.doc_comment as string | null | undefined) ?? null,
|
||||
symbol_name_qualified: (row.symbol_name_qualified as string | null | undefined) ?? null,
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
+17
-116
@@ -24,18 +24,13 @@ CREATE EXTENSION IF NOT EXISTS pgcrypto;
|
||||
-- - access_policy: forward-compat slot, no enforcement in v0.17.
|
||||
-- Write-side lockdown: mutated only when ctx.remote=false.
|
||||
CREATE TABLE IF NOT EXISTS sources (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL UNIQUE,
|
||||
local_path TEXT,
|
||||
last_commit TEXT,
|
||||
last_sync_at TIMESTAMPTZ,
|
||||
config JSONB NOT NULL DEFAULT '{}'::jsonb,
|
||||
-- v0.20.0 Cathedral II (SP-1): chunker version last used to sync this source.
|
||||
-- performSync forces a full walk when this mismatches CURRENT_CHUNKER_VERSION,
|
||||
-- bypassing the git-HEAD up_to_date early-return so CHUNKER_VERSION bumps
|
||||
-- actually trigger re-chunking on upgrade.
|
||||
chunker_version TEXT,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL UNIQUE,
|
||||
local_path TEXT,
|
||||
last_commit TEXT,
|
||||
last_sync_at TIMESTAMPTZ,
|
||||
config JSONB NOT NULL DEFAULT '{}'::jsonb,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
-- Seed the default source. 'default' is federated=true for backward compat
|
||||
@@ -60,10 +55,6 @@ CREATE TABLE IF NOT EXISTS pages (
|
||||
REFERENCES sources(id) ON DELETE CASCADE,
|
||||
slug TEXT NOT NULL,
|
||||
type TEXT NOT NULL,
|
||||
-- v0.19.0: distinguishes markdown vs code pages at the DB level.
|
||||
-- Drives orphans filter, auto-link bypass, and `query --lang`.
|
||||
page_kind TEXT NOT NULL DEFAULT 'markdown'
|
||||
CHECK (page_kind IN ('markdown','code')),
|
||||
title TEXT NOT NULL,
|
||||
compiled_truth TEXT NOT NULL DEFAULT '',
|
||||
timeline TEXT NOT NULL DEFAULT '',
|
||||
@@ -86,111 +77,21 @@ CREATE INDEX IF NOT EXISTS idx_pages_source_id ON pages(source_id);
|
||||
-- content_chunks: chunked content with embeddings
|
||||
-- ============================================================
|
||||
CREATE TABLE IF NOT EXISTS content_chunks (
|
||||
id SERIAL PRIMARY KEY,
|
||||
page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
|
||||
chunk_index INTEGER NOT NULL,
|
||||
chunk_text TEXT NOT NULL,
|
||||
chunk_source TEXT NOT NULL DEFAULT 'compiled_truth',
|
||||
embedding vector(1536),
|
||||
model TEXT NOT NULL DEFAULT 'text-embedding-3-large',
|
||||
token_count INTEGER,
|
||||
embedded_at TIMESTAMPTZ,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
-- v0.19.0: code chunk metadata. Nullable — markdown chunks leave these NULL.
|
||||
-- Powers `query --lang`, `code-def <symbol>`, and `code-refs <symbol>`.
|
||||
language TEXT,
|
||||
symbol_name TEXT,
|
||||
symbol_type TEXT,
|
||||
start_line INTEGER,
|
||||
end_line INTEGER,
|
||||
-- v0.20.0 Cathedral II: qualified symbol identity + parent scope + doc-comment
|
||||
-- + chunk-grain FTS. All nullable — markdown chunks leave these NULL.
|
||||
parent_symbol_path TEXT[],
|
||||
doc_comment TEXT,
|
||||
symbol_name_qualified TEXT,
|
||||
search_vector TSVECTOR
|
||||
id SERIAL PRIMARY KEY,
|
||||
page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
|
||||
chunk_index INTEGER NOT NULL,
|
||||
chunk_text TEXT NOT NULL,
|
||||
chunk_source TEXT NOT NULL DEFAULT 'compiled_truth',
|
||||
embedding vector(1536),
|
||||
model TEXT NOT NULL DEFAULT 'text-embedding-3-large',
|
||||
token_count INTEGER,
|
||||
embedded_at TIMESTAMPTZ,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_chunks_page_index ON content_chunks(page_id, chunk_index);
|
||||
CREATE INDEX IF NOT EXISTS idx_chunks_page ON content_chunks(page_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_chunks_embedding ON content_chunks USING hnsw (embedding vector_cosine_ops);
|
||||
-- v0.19.0: partial indexes — only code chunks populate these columns.
|
||||
CREATE INDEX IF NOT EXISTS idx_chunks_symbol_name ON content_chunks(symbol_name) WHERE symbol_name IS NOT NULL;
|
||||
CREATE INDEX IF NOT EXISTS idx_chunks_language ON content_chunks(language) WHERE language IS NOT NULL;
|
||||
-- v0.20.0 Cathedral II: GIN index on the new chunk-grain FTS vector.
|
||||
CREATE INDEX IF NOT EXISTS idx_chunks_search_vector ON content_chunks USING GIN(search_vector);
|
||||
CREATE INDEX IF NOT EXISTS idx_chunks_symbol_qualified
|
||||
ON content_chunks(symbol_name_qualified) WHERE symbol_name_qualified IS NOT NULL;
|
||||
|
||||
-- v0.20.0 Cathedral II: chunk-grain FTS trigger.
|
||||
-- Weight 'A' on doc_comment + symbol_name_qualified; weight 'B' on chunk_text.
|
||||
-- NL queries ("how do we handle errors") rank doc-comment hits above body text.
|
||||
-- BEFORE INSERT OR UPDATE OF specific columns — only refires when those change,
|
||||
-- not on every chunk update (e.g., embedding refresh doesn't trigger rebuild).
|
||||
CREATE OR REPLACE FUNCTION update_chunk_search_vector() RETURNS TRIGGER AS $fn$
|
||||
BEGIN
|
||||
NEW.search_vector :=
|
||||
setweight(to_tsvector('english', COALESCE(NEW.doc_comment, '')), 'A') ||
|
||||
setweight(to_tsvector('english', COALESCE(NEW.symbol_name_qualified, '')), 'A') ||
|
||||
setweight(to_tsvector('english', COALESCE(NEW.chunk_text, '')), 'B');
|
||||
RETURN NEW;
|
||||
END;
|
||||
$fn$ LANGUAGE plpgsql;
|
||||
|
||||
DROP TRIGGER IF EXISTS chunk_search_vector_trigger ON content_chunks;
|
||||
CREATE TRIGGER chunk_search_vector_trigger
|
||||
BEFORE INSERT OR UPDATE OF chunk_text, doc_comment, symbol_name_qualified
|
||||
ON content_chunks
|
||||
FOR EACH ROW EXECUTE FUNCTION update_chunk_search_vector();
|
||||
|
||||
-- ============================================================
|
||||
-- code_edges_chunk + code_edges_symbol: v0.20.0 Cathedral II structural edges
|
||||
-- ============================================================
|
||||
-- Two-table design (codex F4 + SP-7):
|
||||
-- - code_edges_chunk: resolved edges (both endpoints = known chunk IDs)
|
||||
-- - code_edges_symbol: unresolved refs (target known by qualified name,
|
||||
-- defining chunk not yet imported)
|
||||
-- Readers UNION both tables; no promotion step.
|
||||
-- Source scoping: from_chunk_id -> content_chunks -> pages.source_id
|
||||
-- determines the source. Resolution logic MUST scope on source (codex SP-3);
|
||||
-- only --all-sources callers bypass this. UNIQUE keys don't include source_id
|
||||
-- because from_chunk_id already pins it.
|
||||
CREATE TABLE IF NOT EXISTS code_edges_chunk (
|
||||
id SERIAL PRIMARY KEY,
|
||||
from_chunk_id INTEGER NOT NULL REFERENCES content_chunks(id) ON DELETE CASCADE,
|
||||
to_chunk_id INTEGER NOT NULL REFERENCES content_chunks(id) ON DELETE CASCADE,
|
||||
from_symbol_qualified TEXT NOT NULL,
|
||||
to_symbol_qualified TEXT NOT NULL,
|
||||
edge_type TEXT NOT NULL,
|
||||
edge_metadata JSONB NOT NULL DEFAULT '{}',
|
||||
source_id TEXT REFERENCES sources(id) ON DELETE CASCADE,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
CONSTRAINT code_edges_chunk_unique UNIQUE (from_chunk_id, to_chunk_id, edge_type)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_code_edges_chunk_from
|
||||
ON code_edges_chunk(from_chunk_id, edge_type);
|
||||
CREATE INDEX IF NOT EXISTS idx_code_edges_chunk_to
|
||||
ON code_edges_chunk(to_chunk_id, edge_type);
|
||||
CREATE INDEX IF NOT EXISTS idx_code_edges_chunk_to_symbol
|
||||
ON code_edges_chunk(to_symbol_qualified, edge_type);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS code_edges_symbol (
|
||||
id SERIAL PRIMARY KEY,
|
||||
from_chunk_id INTEGER NOT NULL REFERENCES content_chunks(id) ON DELETE CASCADE,
|
||||
from_symbol_qualified TEXT NOT NULL,
|
||||
to_symbol_qualified TEXT NOT NULL,
|
||||
edge_type TEXT NOT NULL,
|
||||
edge_metadata JSONB NOT NULL DEFAULT '{}',
|
||||
source_id TEXT REFERENCES sources(id) ON DELETE CASCADE,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
CONSTRAINT code_edges_symbol_unique UNIQUE (from_chunk_id, to_symbol_qualified, edge_type)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_code_edges_symbol_from
|
||||
ON code_edges_symbol(from_chunk_id, edge_type);
|
||||
CREATE INDEX IF NOT EXISTS idx_code_edges_symbol_to
|
||||
ON code_edges_symbol(to_symbol_qualified, edge_type);
|
||||
|
||||
-- ============================================================
|
||||
-- links: cross-references between pages
|
||||
|
||||
@@ -106,9 +106,8 @@ describe('buildPlan — diff against completed + installed VERSION', () => {
|
||||
// skippedFuture until the binary catches up. v0.13.0 = frontmatter graph,
|
||||
// v0.13.1 = Knowledge Runtime grandfather, v0.14.0 = shell jobs +
|
||||
// autopilot cooperative, v0.16.0 = subagent runtime, v0.18.0 = multi-
|
||||
// source brains, v0.18.1 = RLS hardening, v0.21.0 = Cathedral II
|
||||
// (renumbered from v0.20.0 after master shipped v0.20.x in parallel).
|
||||
expect(plan.skippedFuture.map(m => m.version)).toEqual(['0.12.0', '0.12.2', '0.13.0', '0.13.1', '0.14.0', '0.16.0', '0.18.0', '0.18.1', '0.21.0']);
|
||||
// source brains, v0.18.1 = RLS hardening (this branch).
|
||||
expect(plan.skippedFuture.map(m => m.version)).toEqual(['0.12.0', '0.12.2', '0.13.0', '0.13.1', '0.14.0', '0.16.0', '0.18.0', '0.18.1']);
|
||||
});
|
||||
|
||||
test('already applied → v0.11.0 lands in `applied` bucket, not pending', () => {
|
||||
@@ -144,11 +143,11 @@ describe('buildPlan — diff against completed + installed VERSION', () => {
|
||||
const idx = indexCompleted([]);
|
||||
const plan = buildPlan(idx, '0.12.0');
|
||||
expect(plan.pending.map(m => m.version)).toContain('0.11.0');
|
||||
// v0.12.2, v0.13.0, v0.13.1, v0.14.0, v0.16.0, v0.18.0, v0.18.1, v0.21.0
|
||||
// were added later; installed=0.12.0 means they belong in skippedFuture,
|
||||
// not pending. v0.11.0 and v0.12.0 stay pending despite being ≤ installed —
|
||||
// v0.12.2, v0.13.0, v0.13.1, v0.14.0, v0.16.0, v0.18.0, v0.18.1 were
|
||||
// added later; installed=0.12.0 means they belong in skippedFuture, not
|
||||
// pending. v0.11.0 and v0.12.0 stay pending despite being ≤ installed —
|
||||
// that is the H9 invariant.
|
||||
expect(plan.skippedFuture.map(m => m.version)).toEqual(['0.12.2', '0.13.0', '0.13.1', '0.14.0', '0.16.0', '0.18.0', '0.18.1', '0.21.0']);
|
||||
expect(plan.skippedFuture.map(m => m.version)).toEqual(['0.12.2', '0.13.0', '0.13.1', '0.14.0', '0.16.0', '0.18.0', '0.18.1']);
|
||||
});
|
||||
|
||||
test('--migration filter narrows to one version', () => {
|
||||
|
||||
@@ -1,146 +0,0 @@
|
||||
/**
|
||||
* v0.20.0 Cathedral II Layer 11 (E1) — BrainBench code sub-categories.
|
||||
*
|
||||
* Pins the retrieval-quality metrics Layer 5 + Layer 6 should keep lifting:
|
||||
* - call_graph_recall: given a chunk that calls foo(), do we surface
|
||||
* that caller via getCallersOf('foo') after importCodeFile?
|
||||
* - parent_scope_coverage: do nested-method chunks carry the expected
|
||||
* parentSymbolPath after end-to-end importCodeFile?
|
||||
*
|
||||
* doc_comment_matching is deferred with A4 full extraction to v0.20.1.
|
||||
* The FTS trigger from Layer 1b weights doc_comment 'A' as soon as the
|
||||
* chunker populates it — the column exists; the value is blank today.
|
||||
*
|
||||
* These are unit-ish end-to-end tests against a fresh PGLite. They pin
|
||||
* the retrieval-side behavior that lifts MRR on symbol queries vs
|
||||
* v0.19.0's grep-class baseline.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { importCodeFile } from '../src/core/import-file.ts';
|
||||
|
||||
describe('Cathedral II BrainBench — call_graph_recall', () => {
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
|
||||
// Import two related files. A calls helper; B defines helper.
|
||||
// Layer 5 edge extraction should capture the calls edge from A's
|
||||
// body to 'helper'.
|
||||
await importCodeFile(
|
||||
engine,
|
||||
'src/a.ts',
|
||||
'export function runner() { return helper(); }\n',
|
||||
{ noEmbed: true },
|
||||
);
|
||||
await importCodeFile(
|
||||
engine,
|
||||
'src/b.ts',
|
||||
'export function helper() { return 42; }\n',
|
||||
{ noEmbed: true },
|
||||
);
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
}, 30_000);
|
||||
|
||||
test('getCallersOf("helper") returns runner as a caller', async () => {
|
||||
const results = await engine.getCallersOf('helper', { allSources: true });
|
||||
expect(results.length).toBeGreaterThanOrEqual(1);
|
||||
const fromRunner = results.find(r => r.from_symbol_qualified === 'runner');
|
||||
expect(fromRunner).toBeDefined();
|
||||
expect(fromRunner!.edge_type).toBe('calls');
|
||||
});
|
||||
|
||||
test('getCalleesOf("runner") returns helper as a callee', async () => {
|
||||
const results = await engine.getCalleesOf('runner', { allSources: true });
|
||||
expect(results.length).toBeGreaterThanOrEqual(1);
|
||||
const toHelper = results.find(r => r.to_symbol_qualified === 'helper');
|
||||
expect(toHelper).toBeDefined();
|
||||
});
|
||||
|
||||
test('re-importing the same file is idempotent (no duplicate edges)', async () => {
|
||||
const before = await engine.getCallersOf('helper', { allSources: true });
|
||||
await importCodeFile(
|
||||
engine,
|
||||
'src/a.ts',
|
||||
'export function runner() { return helper(); }\n',
|
||||
{ noEmbed: true, force: true },
|
||||
);
|
||||
const after = await engine.getCallersOf('helper', { allSources: true });
|
||||
// Per-chunk invalidation wipes then re-writes, so counts should match.
|
||||
expect(after.length).toBe(before.length);
|
||||
});
|
||||
});
|
||||
|
||||
describe('Cathedral II BrainBench — parent_scope_coverage', () => {
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
|
||||
// A class with 2 methods. Layer 6 A3 should emit 3 chunks: the
|
||||
// class + each method. Each method chunk should carry the
|
||||
// parentSymbolPath [ClassName].
|
||||
await importCodeFile(
|
||||
engine,
|
||||
'src/brain.ts',
|
||||
`export class BrainEngine {
|
||||
searchKeyword(q: string) { return q; }
|
||||
searchVector(emb: Float32Array) { return emb; }
|
||||
}
|
||||
`,
|
||||
{ noEmbed: true },
|
||||
);
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
}, 30_000);
|
||||
|
||||
test('nested method chunks persist parent_symbol_path', async () => {
|
||||
const chunks = await engine.getChunks('src-brain-ts');
|
||||
expect(chunks.length).toBeGreaterThanOrEqual(3);
|
||||
|
||||
const method = chunks.find(c => c.symbol_name === 'searchKeyword');
|
||||
expect(method).toBeDefined();
|
||||
expect(method!.parent_symbol_path).toEqual(['BrainEngine']);
|
||||
|
||||
const klass = chunks.find(c => c.symbol_name === 'BrainEngine');
|
||||
expect(klass).toBeDefined();
|
||||
// Class-level chunk: parent_symbol_path is null / empty (top-level).
|
||||
const klassPath = klass!.parent_symbol_path as string[] | null;
|
||||
expect(klassPath == null || klassPath.length === 0).toBe(true);
|
||||
});
|
||||
|
||||
test('qualified symbol name resolves for nested methods', async () => {
|
||||
const chunks = await engine.getChunks('src-brain-ts');
|
||||
const method = chunks.find(c => c.symbol_name === 'searchKeyword');
|
||||
expect(method!.symbol_name_qualified).toBe('BrainEngine.searchKeyword');
|
||||
});
|
||||
|
||||
test('getCallersOf("searchKeyword") matches the bare short name', async () => {
|
||||
// Add a caller that invokes searchKeyword so we can verify the
|
||||
// short-name match path (Layer 5's unresolved capture).
|
||||
await importCodeFile(
|
||||
engine,
|
||||
'src/caller.ts',
|
||||
`import { BrainEngine } from './brain';
|
||||
function demo() {
|
||||
const e = new BrainEngine();
|
||||
return e.searchKeyword('hi');
|
||||
}
|
||||
`,
|
||||
{ noEmbed: true },
|
||||
);
|
||||
const callers = await engine.getCallersOf('searchKeyword', { allSources: true });
|
||||
expect(callers.length).toBeGreaterThanOrEqual(1);
|
||||
});
|
||||
});
|
||||
@@ -1,237 +0,0 @@
|
||||
/**
|
||||
* v0.20.0 Cathedral II Layer 3 (1b) — chunk-grain FTS.
|
||||
*
|
||||
* Before Cathedral II, searchKeyword ranked pages by pages.search_vector
|
||||
* and returned the first matching chunk per page. Doc-comment content
|
||||
* living on a chunk couldn't influence page-grain ranking; two-pass
|
||||
* retrieval anchors couldn't find the best-matching chunk; the "A4
|
||||
* doc-comment boost" story was structurally impossible.
|
||||
*
|
||||
* Layer 3 moves the FTS primitive to content_chunks.search_vector (the
|
||||
* column + trigger added in Layer 1/v27), then dedups-to-best-chunk-per-page
|
||||
* inside searchKeyword so every external caller still sees the v0.19.0
|
||||
* page-grain contract. A2 two-pass (Layer 7) consumes searchKeywordChunks
|
||||
* to get the raw chunk-grain ranking without dedup.
|
||||
*
|
||||
* Tests run against a real in-memory PGLite so they exercise the actual
|
||||
* trigger + FTS machinery.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { MIGRATIONS } from '../src/core/migrate.ts';
|
||||
|
||||
describe('Cathedral II v28 migration — search_vector backfill', () => {
|
||||
test('v28 migration exists in registry', () => {
|
||||
const v28 = MIGRATIONS.find(m => m.version === 28);
|
||||
expect(v28).toBeDefined();
|
||||
expect(v28!.name).toBe('cathedral_ii_chunk_fts_backfill');
|
||||
});
|
||||
|
||||
test('v28 UPDATE is scoped to rows with NULL search_vector (idempotent)', () => {
|
||||
const v28 = MIGRATIONS.find(m => m.version === 28)!;
|
||||
expect(v28.sql).toMatch(/WHERE search_vector IS NULL/);
|
||||
});
|
||||
|
||||
test('v28 builds the vector with same weight shape as v27 trigger', () => {
|
||||
const v28 = MIGRATIONS.find(m => m.version === 28)!;
|
||||
// Same A/A/B weighting as v27 trigger — critical that re-runs produce
|
||||
// identical vectors to freshly-inserted rows.
|
||||
expect(v28.sql).toMatch(/setweight\(to_tsvector\('english', COALESCE\(doc_comment, ''\)\), 'A'\)/);
|
||||
expect(v28.sql).toMatch(/setweight\(to_tsvector\('english', COALESCE\(symbol_name_qualified, ''\)\), 'A'\)/);
|
||||
expect(v28.sql).toMatch(/setweight\(to_tsvector\('english', COALESCE\(chunk_text, ''\)\), 'B'\)/);
|
||||
});
|
||||
});
|
||||
|
||||
describe('Cathedral II Layer 3 — searchKeyword external contract', () => {
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
|
||||
// Two pages, each with multiple chunks that match "refactor" so we can
|
||||
// verify the dedup pass returns one chunk per page. upsertChunks fires
|
||||
// the chunk-grain search_vector trigger from Layer 1 v27.
|
||||
await engine.putPage('guides/refactor-large-fns', {
|
||||
type: 'guide',
|
||||
title: 'How to refactor large functions',
|
||||
compiled_truth: 'placeholder',
|
||||
timeline: '',
|
||||
});
|
||||
await engine.upsertChunks('guides/refactor-large-fns', [
|
||||
{ chunk_index: 0, chunk_text: 'First, refactor the function into smaller units.', chunk_source: 'compiled_truth' },
|
||||
{ chunk_index: 1, chunk_text: 'Then refactor further using extract-method patterns.', chunk_source: 'compiled_truth' },
|
||||
]);
|
||||
await engine.putPage('guides/refactor-patterns', {
|
||||
type: 'guide',
|
||||
title: 'Refactor patterns',
|
||||
compiled_truth: 'placeholder',
|
||||
timeline: '',
|
||||
});
|
||||
await engine.upsertChunks('guides/refactor-patterns', [
|
||||
{ chunk_index: 0, chunk_text: 'The strangler-fig refactor is the safest approach.', chunk_source: 'compiled_truth' },
|
||||
{ chunk_index: 1, chunk_text: 'Refactor incrementally; never boil the ocean at once.', chunk_source: 'compiled_truth' },
|
||||
]);
|
||||
// A third page with no match — must be absent from results.
|
||||
await engine.putPage('guides/unrelated', {
|
||||
type: 'guide',
|
||||
title: 'Deployment',
|
||||
compiled_truth: 'placeholder',
|
||||
timeline: '',
|
||||
});
|
||||
await engine.upsertChunks('guides/unrelated', [
|
||||
{ chunk_index: 0, chunk_text: 'Ship to production on a Tuesday, never a Friday.', chunk_source: 'compiled_truth' },
|
||||
]);
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
}, 30_000);
|
||||
|
||||
test('returns one row per matched page (dedup to best chunk per page)', async () => {
|
||||
const results = await engine.searchKeyword('refactor');
|
||||
const slugs = results.map(r => r.slug).sort();
|
||||
// Deduped: each matched page appears exactly once.
|
||||
expect(new Set(slugs).size).toBe(slugs.length);
|
||||
expect(slugs).toContain('guides/refactor-large-fns');
|
||||
expect(slugs).toContain('guides/refactor-patterns');
|
||||
expect(slugs).not.toContain('guides/unrelated');
|
||||
});
|
||||
|
||||
test('results include chunk metadata (chunk_id, chunk_text, chunk_source)', async () => {
|
||||
const results = await engine.searchKeyword('refactor');
|
||||
expect(results.length).toBeGreaterThan(0);
|
||||
const first = results[0]!;
|
||||
expect(first.chunk_id).toBeGreaterThan(0);
|
||||
expect(first.chunk_text).toMatch(/refactor/i);
|
||||
expect(first.chunk_source).toBe('compiled_truth');
|
||||
});
|
||||
|
||||
test('limit is honored on the page-grain external shape', async () => {
|
||||
const results = await engine.searchKeyword('refactor', { limit: 1 });
|
||||
expect(results.length).toBeLessThanOrEqual(1);
|
||||
});
|
||||
|
||||
test('non-matching query returns empty', async () => {
|
||||
const results = await engine.searchKeyword('zzzzznomatch');
|
||||
expect(results).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('Cathedral II Layer 3 — searchKeywordChunks (internal primitive)', () => {
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
|
||||
// Page with multiple matching chunks so chunk-grain results can
|
||||
// return two chunks from the same page (no dedup).
|
||||
await engine.putPage('guides/multi-chunk', {
|
||||
type: 'guide',
|
||||
title: 'Long guide',
|
||||
compiled_truth: 'placeholder',
|
||||
timeline: '',
|
||||
});
|
||||
await engine.upsertChunks('guides/multi-chunk', [
|
||||
{ chunk_index: 0, chunk_text: 'refactor is a core engineering practice.', chunk_source: 'compiled_truth' },
|
||||
{ chunk_index: 1, chunk_text: 'refactor safely using characterization tests.', chunk_source: 'compiled_truth' },
|
||||
{ chunk_index: 2, chunk_text: 'refactor tools can automate common patterns.', chunk_source: 'compiled_truth' },
|
||||
]);
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
}, 30_000);
|
||||
|
||||
test('does not dedup: can return multiple chunks from the same page', async () => {
|
||||
const results = await engine.searchKeywordChunks('refactor', { limit: 20 });
|
||||
const slugs = results.map(r => r.slug);
|
||||
// More results than distinct slugs means we got multiple chunks per page.
|
||||
// (If the page only produced 1 chunk, the assertion short-circuits.)
|
||||
if (results.length > 1) {
|
||||
// Either multiple chunks from one page, or the page was small. Either
|
||||
// is a valid observation; what matters is we did NOT forcibly dedup.
|
||||
const uniqueSlugs = new Set(slugs);
|
||||
expect(uniqueSlugs.size).toBeLessThanOrEqual(results.length);
|
||||
}
|
||||
});
|
||||
|
||||
test('ordered by score descending', async () => {
|
||||
const results = await engine.searchKeywordChunks('refactor', { limit: 20 });
|
||||
for (let i = 1; i < results.length; i++) {
|
||||
expect(results[i - 1]!.score).toBeGreaterThanOrEqual(results[i]!.score);
|
||||
}
|
||||
});
|
||||
|
||||
test('limit is honored (no dedup inflating counts)', async () => {
|
||||
const results = await engine.searchKeywordChunks('refactor', { limit: 2 });
|
||||
expect(results.length).toBeLessThanOrEqual(2);
|
||||
});
|
||||
});
|
||||
|
||||
describe('Cathedral II Layer 3 — doc-comment weight precedence (A4 foundation)', () => {
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
|
||||
// Two pages, each with one chunk. Alpha's chunk has the target term
|
||||
// 'hexagon' in its doc_comment (weight A); Beta's chunk has it in
|
||||
// chunk_text (weight B). After Layer 5/6 populate doc_comment from
|
||||
// real AST leading comments, this preference delivers A4's ranking win.
|
||||
await engine.putPage('pages/alpha', {
|
||||
type: 'note',
|
||||
title: 'Alpha',
|
||||
compiled_truth: 'placeholder',
|
||||
timeline: '',
|
||||
});
|
||||
await engine.upsertChunks('pages/alpha', [
|
||||
{ chunk_index: 0, chunk_text: 'Some boilerplate text without the term.', chunk_source: 'compiled_truth' },
|
||||
]);
|
||||
await engine.putPage('pages/beta', {
|
||||
type: 'note',
|
||||
title: 'Beta',
|
||||
compiled_truth: 'placeholder',
|
||||
timeline: '',
|
||||
});
|
||||
await engine.upsertChunks('pages/beta', [
|
||||
{ chunk_index: 0, chunk_text: 'The hexagon term appears only inside body text here.', chunk_source: 'compiled_truth' },
|
||||
]);
|
||||
|
||||
// Manually promote Alpha's chunk to have a doc_comment containing
|
||||
// 'hexagon'. The v27 trigger re-fires BEFORE UPDATE OF doc_comment,
|
||||
// so search_vector rebuilds with the weight-A doc-comment contribution.
|
||||
// In real use this happens at Layer 5 import time when AST leading
|
||||
// comments land as doc_comment.
|
||||
await engine.executeRaw(
|
||||
`UPDATE content_chunks
|
||||
SET doc_comment = 'hexagon architecture documented here'
|
||||
WHERE page_id = (SELECT id FROM pages WHERE slug = $1)`,
|
||||
['pages/alpha'],
|
||||
);
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
}, 30_000);
|
||||
|
||||
test('doc-comment match outranks body-text match on the same term', async () => {
|
||||
const results = await engine.searchKeyword('hexagon');
|
||||
expect(results.length).toBeGreaterThan(0);
|
||||
// Alpha (with doc_comment match, weight A) should rank above Beta
|
||||
// (with body-text match, weight B). If the FTS weighting is wrong,
|
||||
// Beta would win because its chunk_text contains 'hexagon' too.
|
||||
const slugs = results.map(r => r.slug);
|
||||
const alphaIdx = slugs.indexOf('pages/alpha');
|
||||
const betaIdx = slugs.indexOf('pages/beta');
|
||||
expect(alphaIdx).toBeGreaterThan(-1);
|
||||
expect(betaIdx).toBeGreaterThan(-1);
|
||||
expect(alphaIdx).toBeLessThan(betaIdx);
|
||||
});
|
||||
});
|
||||
@@ -1,48 +0,0 @@
|
||||
/**
|
||||
* v0.20.0 Cathedral II Layer 12 — CHUNKER_VERSION 3→4 bump + SP-1 gate.
|
||||
*
|
||||
* Codex's second-pass review caught that bumping CHUNKER_VERSION alone
|
||||
* does nothing on an unchanged repo: `performSync` short-circuits at
|
||||
* `up_to_date` before reaching `importCodeFile`'s content_hash check.
|
||||
* Layer 12 adds a sources.chunker_version gate that forces a full
|
||||
* re-walk when the version mismatches, regardless of git HEAD equality.
|
||||
*
|
||||
* These tests validate the constant value, the gate logic, and the
|
||||
* write-after-sync behavior. Full e2e covered by test/e2e if DB present.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { CHUNKER_VERSION } from '../src/core/chunkers/code.ts';
|
||||
|
||||
describe('Layer 12 — CHUNKER_VERSION constant', () => {
|
||||
test('bumped to 4 for Cathedral II', () => {
|
||||
// v3: v0.19.0 Chonkie parity (tokenizer + small-sibling merge).
|
||||
// v4: v0.20.0 Cathedral II (qualified names + parent scope + doc_comment
|
||||
// + fence extraction + chunk-grain FTS). Folded into content_hash
|
||||
// so any bump forces clean re-chunks on next sync.
|
||||
expect(CHUNKER_VERSION).toBe(4);
|
||||
});
|
||||
|
||||
test('is stable across imports (not recomputed at call time)', async () => {
|
||||
const a = (await import('../src/core/chunkers/code.ts')).CHUNKER_VERSION;
|
||||
const b = (await import('../src/core/chunkers/code.ts')).CHUNKER_VERSION;
|
||||
expect(a).toBe(b);
|
||||
expect(a).toBe(4);
|
||||
});
|
||||
});
|
||||
|
||||
describe('Layer 12 — sources.chunker_version column from v27 migration', () => {
|
||||
test('v27 foundation migration adds chunker_version to sources', async () => {
|
||||
const { MIGRATIONS } = await import('../src/core/migrate.ts');
|
||||
const v27 = MIGRATIONS.find(m => m.version === 27);
|
||||
expect(v27).toBeDefined();
|
||||
expect(v27!.sql).toMatch(/ALTER TABLE sources/);
|
||||
expect(v27!.sql).toMatch(/ADD COLUMN IF NOT EXISTS chunker_version TEXT/);
|
||||
});
|
||||
});
|
||||
|
||||
// Full integration test: would run an e2e sync twice against a real git
|
||||
// repo fixture and assert that the second sync (with HEAD unchanged but
|
||||
// chunker_version advanced) re-walks. That lives in
|
||||
// test/e2e/cathedral-ii.test.ts (future Layer 5 pilot). This file pins
|
||||
// the constant + migration shape so any accidental revert surfaces in CI.
|
||||
@@ -1,252 +0,0 @@
|
||||
/**
|
||||
* v0.19.0 Layer 5 — tree-sitter code chunker tests.
|
||||
*
|
||||
* Covers: detectCodeLanguage across all 29 file extensions, chunkCodeText
|
||||
* on TS/Python/Go/Rust/Java + small-sibling merging + tokenizer accuracy
|
||||
* + language fallback for unsupported extensions.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { chunkCodeText, detectCodeLanguage, CHUNKER_VERSION } from '../../src/core/chunkers/code.ts';
|
||||
|
||||
describe('CHUNKER_VERSION', () => {
|
||||
test('v0.20.0 Cathedral II Layer 12 bumped to 4', () => {
|
||||
expect(CHUNKER_VERSION).toBe(4);
|
||||
});
|
||||
});
|
||||
|
||||
describe('detectCodeLanguage', () => {
|
||||
test('recognizes all 29 supported extensions', () => {
|
||||
const cases: Record<string, string> = {
|
||||
'foo.ts': 'typescript', 'foo.tsx': 'tsx', 'foo.mts': 'typescript', 'foo.cts': 'typescript',
|
||||
'foo.js': 'javascript', 'foo.jsx': 'javascript', 'foo.mjs': 'javascript', 'foo.cjs': 'javascript',
|
||||
'foo.py': 'python', 'foo.rb': 'ruby', 'foo.go': 'go',
|
||||
'foo.rs': 'rust', 'foo.java': 'java', 'foo.cs': 'c_sharp',
|
||||
'foo.cpp': 'cpp', 'foo.cc': 'cpp', 'foo.hpp': 'cpp',
|
||||
'foo.c': 'c', 'foo.h': 'c',
|
||||
'foo.php': 'php', 'foo.swift': 'swift', 'foo.kt': 'kotlin',
|
||||
'foo.scala': 'scala', 'foo.lua': 'lua', 'foo.ex': 'elixir',
|
||||
'foo.elm': 'elm', 'foo.ml': 'ocaml', 'foo.dart': 'dart',
|
||||
'foo.zig': 'zig', 'foo.sol': 'solidity', 'foo.sh': 'bash',
|
||||
'foo.css': 'css', 'foo.html': 'html', 'foo.vue': 'vue',
|
||||
'foo.json': 'json', 'foo.yaml': 'yaml', 'foo.toml': 'toml',
|
||||
};
|
||||
for (const [path, expected] of Object.entries(cases)) {
|
||||
expect(detectCodeLanguage(path)).toBe(expected as any);
|
||||
}
|
||||
});
|
||||
|
||||
test('returns null for unsupported extensions', () => {
|
||||
expect(detectCodeLanguage('foo.md')).toBeNull();
|
||||
expect(detectCodeLanguage('foo.txt')).toBeNull();
|
||||
expect(detectCodeLanguage('README')).toBeNull();
|
||||
});
|
||||
|
||||
test('is case-insensitive', () => {
|
||||
expect(detectCodeLanguage('Main.GO')).toBe('go');
|
||||
expect(detectCodeLanguage('App.TSX')).toBe('tsx');
|
||||
});
|
||||
});
|
||||
|
||||
describe('chunkCodeText — TypeScript', () => {
|
||||
test('extracts top-level functions with correct symbol names', async () => {
|
||||
const src = `export function calculateScore(items: number[]): number {
|
||||
let sum = 0;
|
||||
for (const i of items) { sum += i; }
|
||||
if (sum < 0) return 0;
|
||||
return sum / items.length;
|
||||
}`;
|
||||
const result = await chunkCodeText(src, 'calc.ts');
|
||||
expect(result.length).toBeGreaterThanOrEqual(1);
|
||||
expect(result[0]!.metadata.language).toBe('typescript');
|
||||
expect(result[0]!.metadata.symbolName).toBe('calculateScore');
|
||||
expect(result[0]!.text).toContain('[TypeScript]');
|
||||
expect(result[0]!.text).toContain('calc.ts');
|
||||
});
|
||||
|
||||
test('extracts classes with methods', async () => {
|
||||
const src = `export class Registry {
|
||||
private items: Map<string, number> = new Map();
|
||||
register(id: string, val: number): void { this.items.set(id, val); }
|
||||
lookup(id: string): number | null { return this.items.get(id) ?? null; }
|
||||
}`;
|
||||
const result = await chunkCodeText(src, 'reg.ts');
|
||||
const classChunk = result.find(c => c.metadata.symbolName === 'Registry');
|
||||
expect(classChunk).toBeDefined();
|
||||
// `export class Foo` is wrapped in export_statement at the AST level;
|
||||
// symbol extraction still finds "Registry" but the type surface shows
|
||||
// the wrapper. See normalizeSymbolType() for the mapping.
|
||||
expect(classChunk!.metadata.symbolType).toMatch(/class|export/);
|
||||
});
|
||||
});
|
||||
|
||||
describe('chunkCodeText — Python', () => {
|
||||
test('extracts class_definition + function_definition', async () => {
|
||||
const src = `class Animal:
|
||||
def __init__(self, name):
|
||||
self.name = name
|
||||
|
||||
def speak(self, sound):
|
||||
return f"{self.name} says {sound}"
|
||||
|
||||
def pet_the_dog():
|
||||
dog = Animal("Rex")
|
||||
return dog.speak("woof woof woof woof woof")
|
||||
`;
|
||||
const result = await chunkCodeText(src, 'animal.py');
|
||||
expect(result.length).toBeGreaterThanOrEqual(1);
|
||||
const allLanguages = result.map(c => c.metadata.language);
|
||||
for (const lang of allLanguages) expect(lang).toBe('python');
|
||||
});
|
||||
});
|
||||
|
||||
describe('chunkCodeText — Rust', () => {
|
||||
test('extracts struct_item + impl_item + function_item', async () => {
|
||||
const src = `pub struct UserRecord {
|
||||
pub id: u64,
|
||||
pub name: String,
|
||||
pub active: bool,
|
||||
pub score: f64,
|
||||
}
|
||||
|
||||
impl UserRecord {
|
||||
pub fn new(id: u64, name: String) -> Self {
|
||||
Self { id, name, active: true, score: 0.0 }
|
||||
}
|
||||
|
||||
pub fn deactivate(&mut self) {
|
||||
self.active = false;
|
||||
}
|
||||
|
||||
pub fn bump_score(&mut self, delta: f64) {
|
||||
self.score += delta;
|
||||
}
|
||||
}
|
||||
|
||||
pub fn compute_total(records: &[UserRecord]) -> f64 {
|
||||
records.iter().filter(|r| r.active).map(|r| r.score).sum()
|
||||
}
|
||||
`;
|
||||
const result = await chunkCodeText(src, 'users.rs');
|
||||
expect(result.length).toBeGreaterThan(0);
|
||||
expect(result[0]!.metadata.language).toBe('rust');
|
||||
const headers = result.map(c => c.text.split('\n')[0]);
|
||||
const hasRustTag = headers.some(h => h.includes('[Rust]'));
|
||||
expect(hasRustTag).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe('chunkCodeText — Go', () => {
|
||||
test('extracts function + type + method declarations', async () => {
|
||||
const src = `package main
|
||||
|
||||
import "fmt"
|
||||
|
||||
type Point struct {
|
||||
X, Y int
|
||||
}
|
||||
|
||||
func (p Point) Distance(other Point) float64 {
|
||||
dx := float64(p.X - other.X)
|
||||
dy := float64(p.Y - other.Y)
|
||||
return dx*dx + dy*dy
|
||||
}
|
||||
|
||||
func main() {
|
||||
p1 := Point{X: 1, Y: 2}
|
||||
p2 := Point{X: 5, Y: 6}
|
||||
fmt.Println(p1.Distance(p2))
|
||||
}
|
||||
`;
|
||||
const result = await chunkCodeText(src, 'main.go');
|
||||
expect(result.length).toBeGreaterThanOrEqual(1);
|
||||
const headers = result.map(c => c.text.split('\n')[0]);
|
||||
expect(headers.some(h => h.includes('[Go]'))).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe('chunkCodeText — fallback for unsupported language', () => {
|
||||
test('unsupported extension falls through to recursive chunker', async () => {
|
||||
const src = 'this is not code. just text. lots of text. '.repeat(50);
|
||||
const result = await chunkCodeText(src, 'unknown.xyz');
|
||||
expect(result.length).toBeGreaterThan(0);
|
||||
expect(result[0]!.metadata.symbolType).toBe('module');
|
||||
expect(result[0]!.metadata.symbolName).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('chunkCodeText — small-sibling merging', () => {
|
||||
test('small adjacent chunks are merged when chunkSizeTokens is generous', async () => {
|
||||
// With a very large chunkSizeTokens, the merge threshold rises and
|
||||
// more chunks qualify as "small" for accumulation. 10 tiny consts
|
||||
// at chunkTarget=1000 gives a merge threshold of 150 — each const
|
||||
// chunk (with its structured header) is ~20 tokens, so they all
|
||||
// accumulate into one merged group up to the 1000-token budget.
|
||||
const src = `const A = 1;
|
||||
const B = 2;
|
||||
const C = 3;
|
||||
const D = 4;
|
||||
const E = 5;
|
||||
const F = 6;
|
||||
const G = 7;
|
||||
const H = 8;
|
||||
const I = 9;
|
||||
const J = 10;
|
||||
`;
|
||||
const result = await chunkCodeText(src, 'constants.ts', { chunkSizeTokens: 1000 });
|
||||
expect(result.length).toBeLessThan(10); // at least some merging occurred
|
||||
const merged = result.find(c => c.metadata.symbolType === 'merged');
|
||||
expect(merged).toBeDefined();
|
||||
});
|
||||
|
||||
test('large chunk stays independent', async () => {
|
||||
const src = `export function bigFn() {
|
||||
let result = 0;
|
||||
for (let i = 0; i < 1000; i++) {
|
||||
for (let j = 0; j < 1000; j++) {
|
||||
result += Math.sqrt(i * i + j * j) * Math.sin(i) * Math.cos(j);
|
||||
}
|
||||
}
|
||||
if (result > 0) { console.log('positive'); }
|
||||
else if (result < 0) { console.log('negative'); }
|
||||
else { console.log('zero'); }
|
||||
return result;
|
||||
}
|
||||
`;
|
||||
const result = await chunkCodeText(src, 'big.ts', { chunkSizeTokens: 100 });
|
||||
const bigChunk = result.find(c => c.metadata.symbolName === 'bigFn');
|
||||
expect(bigChunk).toBeDefined();
|
||||
});
|
||||
|
||||
test('merged chunk has correct line range spanning merged siblings', async () => {
|
||||
const src = `const X = 1;
|
||||
|
||||
const Y = 2;
|
||||
|
||||
const Z = 3;
|
||||
`;
|
||||
const result = await chunkCodeText(src, 'abc.ts', { chunkSizeTokens: 100 });
|
||||
if (result.length === 1 && result[0]!.metadata.symbolType === 'merged') {
|
||||
expect(result[0]!.metadata.startLine).toBe(1);
|
||||
expect(result[0]!.metadata.endLine).toBeGreaterThanOrEqual(5);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('chunkCodeText — structured header', () => {
|
||||
test('header includes language display name, path, line range, symbol', async () => {
|
||||
const src = `export function myFunc() { return 42; }
|
||||
`;
|
||||
const result = await chunkCodeText(src, 'src/lib/foo.ts');
|
||||
const first = result[0]!;
|
||||
expect(first.text).toMatch(/^\[TypeScript\] src\/lib\/foo\.ts:\d+-\d+ /);
|
||||
expect(first.text).toContain('myFunc');
|
||||
});
|
||||
});
|
||||
|
||||
describe('chunkCodeText — empty input', () => {
|
||||
test('empty source returns empty array', async () => {
|
||||
expect(await chunkCodeText('', 'foo.ts')).toEqual([]);
|
||||
expect(await chunkCodeText(' \n ', 'foo.ts')).toEqual([]);
|
||||
});
|
||||
});
|
||||
@@ -1,24 +0,0 @@
|
||||
/**
|
||||
* v0.20.0 Cathedral II Layer 10 (C4 + C5) — code-callers + code-callees CLI tests.
|
||||
*
|
||||
* The CLI commands are thin wrappers over engine.getCallersOf /
|
||||
* engine.getCalleesOf. Tests validate:
|
||||
* - the commands are exported and can be called
|
||||
* - JSON output shape is stable (agent-consumable)
|
||||
* - missing symbol exits with UsageError envelope
|
||||
* End-to-end engine round-trip lives in test/code-edges.test.ts.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
|
||||
describe('Layer 10 C4/C5 — commands export runCodeCallers / runCodeCallees', () => {
|
||||
test('code-callers module exports runCodeCallers', async () => {
|
||||
const mod = await import('../src/commands/code-callers.ts');
|
||||
expect(typeof mod.runCodeCallers).toBe('function');
|
||||
});
|
||||
|
||||
test('code-callees module exports runCodeCallees', async () => {
|
||||
const mod = await import('../src/commands/code-callees.ts');
|
||||
expect(typeof mod.runCodeCallees).toBe('function');
|
||||
});
|
||||
});
|
||||
@@ -1,222 +0,0 @@
|
||||
/**
|
||||
* v0.19.0 Layer 7 — code-def + code-refs integration tests.
|
||||
*
|
||||
* Seeds a small fixture repo into PGLite, imports it via importCodeFile,
|
||||
* then exercises the new lookup commands. Verifies:
|
||||
* - Symbol definitions resolve to the correct file/line.
|
||||
* - Language filter narrows results.
|
||||
* - code-refs returns multiple chunks from the same file (bypasses
|
||||
* the DISTINCT ON search-path collapse).
|
||||
* - Empty-result case returns empty array (not an error).
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { importCodeFile } from '../src/core/import-file.ts';
|
||||
import { findCodeDef } from '../src/commands/code-def.ts';
|
||||
import { findCodeRefs } from '../src/commands/code-refs.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
|
||||
// Seed: two TypeScript files. One defines BrainEngine, another uses it.
|
||||
// Each symbol is deliberately large enough to stay independent under
|
||||
// the small-sibling merging threshold (~120 tokens per chunk).
|
||||
const brainEngineSrc = `export interface BrainEngine {
|
||||
connect(config: { dbUrl: string; poolSize?: number; timeout?: number }): Promise<void>;
|
||||
disconnect(): Promise<void>;
|
||||
getPage(slug: string): Promise<{ slug: string; title: string; content: string } | null>;
|
||||
putPage(slug: string, page: { title: string; content: string }): Promise<void>;
|
||||
deletePage(slug: string): Promise<void>;
|
||||
searchKeyword(query: string, opts?: { limit?: number }): Promise<Array<{ slug: string; score: number }>>;
|
||||
searchVector(embedding: Float32Array, opts?: { limit?: number }): Promise<Array<{ slug: string; score: number }>>;
|
||||
getChunks(slug: string): Promise<Array<{ chunk_text: string; embedding: Float32Array | null }>>;
|
||||
}
|
||||
|
||||
export class PGLiteEngine implements BrainEngine {
|
||||
private url: string = '';
|
||||
private poolSize: number = 10;
|
||||
|
||||
async connect(config: { dbUrl: string; poolSize?: number; timeout?: number }): Promise<void> {
|
||||
this.url = config.dbUrl;
|
||||
this.poolSize = config.poolSize ?? 10;
|
||||
console.log('connecting to', this.url, 'with pool size', this.poolSize);
|
||||
if (this.url === '') throw new Error('no url provided');
|
||||
if (this.poolSize < 1) throw new Error('pool size must be >= 1');
|
||||
if (config.timeout !== undefined && config.timeout < 0) throw new Error('bad timeout');
|
||||
}
|
||||
|
||||
async disconnect(): Promise<void> {
|
||||
console.log('disconnecting from', this.url);
|
||||
this.url = '';
|
||||
}
|
||||
|
||||
async getPage(slug: string) {
|
||||
if (!slug) return null;
|
||||
if (slug.length > 200) throw new Error('slug too long');
|
||||
if (slug.includes('//')) throw new Error('bad slug');
|
||||
return { slug, title: 'sample title for ' + slug, content: 'fixture content for ' + slug };
|
||||
}
|
||||
|
||||
async putPage(slug: string, page: { title: string; content: string }) {
|
||||
if (!slug) throw new Error('slug required');
|
||||
if (!page.title) throw new Error('title required');
|
||||
console.log('put', slug, page.title, page.content.length, 'chars');
|
||||
}
|
||||
|
||||
async deletePage(slug: string): Promise<void> {
|
||||
if (!slug) throw new Error('slug required');
|
||||
console.log('delete', slug);
|
||||
}
|
||||
|
||||
async searchKeyword(query: string, opts: { limit?: number } = {}) {
|
||||
if (!query) return [];
|
||||
const limit = opts.limit ?? 10;
|
||||
return [{ slug: 'match-1', score: 0.9 }, { slug: 'match-2', score: 0.5 }].slice(0, limit);
|
||||
}
|
||||
|
||||
async searchVector(embedding: Float32Array, opts: { limit?: number } = {}) {
|
||||
if (embedding.length !== 1536) throw new Error('bad embedding dim');
|
||||
const limit = opts.limit ?? 10;
|
||||
return [{ slug: 'vec-1', score: 0.88 }, { slug: 'vec-2', score: 0.77 }].slice(0, limit);
|
||||
}
|
||||
|
||||
async getChunks(slug: string) {
|
||||
if (!slug) return [];
|
||||
return [{ chunk_text: 'chunk for ' + slug, embedding: null }];
|
||||
}
|
||||
}
|
||||
|
||||
export function makeBrainEngine(url: string, poolSize: number): BrainEngine {
|
||||
if (!url) throw new Error('url required to makeBrainEngine');
|
||||
if (poolSize < 1) throw new Error('pool size must be positive');
|
||||
const e = new PGLiteEngine();
|
||||
e.connect({ dbUrl: url, poolSize }).catch((err) => {
|
||||
console.error('connect failed:', err);
|
||||
throw err;
|
||||
});
|
||||
return e;
|
||||
}
|
||||
`;
|
||||
const consumerSrc = `import type { BrainEngine } from './engine';
|
||||
|
||||
export async function performSync(engine: BrainEngine, path: string, opts: { force?: boolean; dryRun?: boolean } = {}): Promise<void> {
|
||||
if (!path) throw new Error('path required');
|
||||
const page = await engine.getPage(path);
|
||||
if (!page && !opts.force) {
|
||||
throw new Error('page not found at ' + path);
|
||||
}
|
||||
if (!page) {
|
||||
console.log('forcing creation of', path);
|
||||
await engine.putPage(path, { title: 'Forced', content: 'forced content' });
|
||||
return;
|
||||
}
|
||||
if (opts.dryRun) {
|
||||
console.log('dry-run: would update', page.slug);
|
||||
return;
|
||||
}
|
||||
await engine.putPage(page.slug, { title: page.title, content: page.content });
|
||||
if (page.slug.startsWith('test-')) {
|
||||
console.log('test sync for', page.slug);
|
||||
}
|
||||
if (page.content.length > 10000) {
|
||||
console.warn('large page:', page.slug);
|
||||
}
|
||||
}
|
||||
|
||||
export async function performDump(engine: BrainEngine, slug: string): Promise<BrainEngine> {
|
||||
if (!slug) throw new Error('slug required');
|
||||
const page = await engine.getPage(slug);
|
||||
if (page) {
|
||||
const dumpedTitle = page.title + ' (dumped at ' + Date.now() + ')';
|
||||
await engine.putPage(slug, { title: dumpedTitle, content: page.content });
|
||||
const chunks = await engine.getChunks(slug);
|
||||
console.log('dumped', slug, 'with', chunks.length, 'chunks');
|
||||
} else {
|
||||
console.warn('cannot dump missing page:', slug);
|
||||
}
|
||||
return engine;
|
||||
}
|
||||
`;
|
||||
await importCodeFile(engine, 'src/engine.ts', brainEngineSrc, { noEmbed: true });
|
||||
await importCodeFile(engine, 'src/sync.ts', consumerSrc, { noEmbed: true });
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
describe('findCodeDef', () => {
|
||||
test('finds the definition of an interface', async () => {
|
||||
const results = await findCodeDef(engine, 'BrainEngine');
|
||||
expect(results.length).toBeGreaterThanOrEqual(1);
|
||||
// Should match in src/engine.ts, not in src/sync.ts
|
||||
const engineSlugMatch = results.find((r) => r.slug === 'src-engine-ts');
|
||||
expect(engineSlugMatch).toBeDefined();
|
||||
});
|
||||
|
||||
test('finds a function definition', async () => {
|
||||
const results = await findCodeDef(engine, 'makeBrainEngine');
|
||||
expect(results.length).toBeGreaterThanOrEqual(1);
|
||||
const match = results.find((r) => r.slug === 'src-engine-ts');
|
||||
expect(match).toBeDefined();
|
||||
expect(match!.symbol_type).toMatch(/function|export/);
|
||||
});
|
||||
|
||||
test('returns empty for unknown symbol', async () => {
|
||||
const results = await findCodeDef(engine, 'ThisSymbolDoesNotExist');
|
||||
expect(results).toEqual([]);
|
||||
});
|
||||
|
||||
test('language filter narrows to typescript only', async () => {
|
||||
const results = await findCodeDef(engine, 'BrainEngine', { language: 'typescript' });
|
||||
expect(results.length).toBeGreaterThanOrEqual(1);
|
||||
for (const r of results) expect(r.language).toBe('typescript');
|
||||
});
|
||||
|
||||
test('language filter with non-matching language returns empty', async () => {
|
||||
const results = await findCodeDef(engine, 'BrainEngine', { language: 'python' });
|
||||
expect(results).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('findCodeRefs', () => {
|
||||
test('finds multiple usage sites across files', async () => {
|
||||
const results = await findCodeRefs(engine, 'BrainEngine');
|
||||
expect(results.length).toBeGreaterThanOrEqual(2);
|
||||
// Should include both files
|
||||
const slugs = new Set(results.map((r) => r.slug));
|
||||
expect(slugs.has('src-engine-ts')).toBe(true);
|
||||
expect(slugs.has('src-sync-ts')).toBe(true);
|
||||
});
|
||||
|
||||
test('ranks by slug + line number (deterministic)', async () => {
|
||||
const results = await findCodeRefs(engine, 'performSync');
|
||||
// performSync is defined in src/sync.ts — findCodeRefs should list it
|
||||
const match = results.find((r) => r.slug === 'src-sync-ts');
|
||||
expect(match).toBeDefined();
|
||||
});
|
||||
|
||||
test('empty query returns empty (no crash on empty ILIKE)', async () => {
|
||||
const results = await findCodeRefs(engine, 'ZzzNothingZzz');
|
||||
expect(results).toEqual([]);
|
||||
});
|
||||
|
||||
test('limit caps result count', async () => {
|
||||
const results = await findCodeRefs(engine, 'engine', { limit: 1 });
|
||||
expect(results.length).toBeLessThanOrEqual(1);
|
||||
});
|
||||
|
||||
test('results include snippets for agent consumption', async () => {
|
||||
const results = await findCodeRefs(engine, 'BrainEngine');
|
||||
for (const r of results) {
|
||||
expect(typeof r.snippet).toBe('string');
|
||||
expect(r.snippet.length).toBeGreaterThan(0);
|
||||
expect(r.snippet.length).toBeLessThanOrEqual(500);
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -1,142 +0,0 @@
|
||||
/**
|
||||
* v0.20.0 Cathedral II Layer 5 (A1) — code-edges engine method tests.
|
||||
*
|
||||
* Tests addCodeEdges / deleteCodeEdgesForChunks / getCallersOf /
|
||||
* getCalleesOf / getEdgesByChunk against real PGLite. End-to-end
|
||||
* importCodeFile integration is covered in code-edges-integration.test.ts.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
|
||||
describe('Layer 5 (A1) — code-edges engine methods', () => {
|
||||
let engine: PGLiteEngine;
|
||||
let chunkA: number;
|
||||
let chunkB: number;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
|
||||
// Create two code pages with one chunk each.
|
||||
await engine.putPage('src-a-ts', {
|
||||
type: 'code', page_kind: 'code',
|
||||
title: 'src/a.ts (typescript)',
|
||||
compiled_truth: 'export function run() { return helper(); }',
|
||||
timeline: '',
|
||||
});
|
||||
await engine.upsertChunks('src-a-ts', [{
|
||||
chunk_index: 0,
|
||||
chunk_text: 'export function run() { return helper(); }',
|
||||
chunk_source: 'compiled_truth',
|
||||
language: 'typescript',
|
||||
symbol_name: 'run',
|
||||
symbol_type: 'function',
|
||||
symbol_name_qualified: 'run',
|
||||
}]);
|
||||
|
||||
await engine.putPage('src-b-ts', {
|
||||
type: 'code', page_kind: 'code',
|
||||
title: 'src/b.ts (typescript)',
|
||||
compiled_truth: 'export function helper() { return 1; }',
|
||||
timeline: '',
|
||||
});
|
||||
await engine.upsertChunks('src-b-ts', [{
|
||||
chunk_index: 0,
|
||||
chunk_text: 'export function helper() { return 1; }',
|
||||
chunk_source: 'compiled_truth',
|
||||
language: 'typescript',
|
||||
symbol_name: 'helper',
|
||||
symbol_type: 'function',
|
||||
symbol_name_qualified: 'helper',
|
||||
}]);
|
||||
|
||||
const aChunks = await engine.getChunks('src-a-ts');
|
||||
const bChunks = await engine.getChunks('src-b-ts');
|
||||
chunkA = aChunks[0]!.id;
|
||||
chunkB = bChunks[0]!.id;
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
}, 30_000);
|
||||
|
||||
test('addCodeEdges inserts unresolved rows into code_edges_symbol', async () => {
|
||||
const inserted = await engine.addCodeEdges([{
|
||||
from_chunk_id: chunkA,
|
||||
to_chunk_id: null,
|
||||
from_symbol_qualified: 'run',
|
||||
to_symbol_qualified: 'helper',
|
||||
edge_type: 'calls',
|
||||
}]);
|
||||
expect(inserted).toBeGreaterThanOrEqual(1);
|
||||
});
|
||||
|
||||
test('getCallersOf finds the caller by short name (unresolved path)', async () => {
|
||||
const results = await engine.getCallersOf('helper', { allSources: true });
|
||||
expect(results.length).toBeGreaterThanOrEqual(1);
|
||||
const hit = results.find(r => r.from_symbol_qualified === 'run');
|
||||
expect(hit).toBeDefined();
|
||||
expect(hit!.resolved).toBe(false); // unresolved (from code_edges_symbol)
|
||||
expect(hit!.to_symbol_qualified).toBe('helper');
|
||||
expect(hit!.edge_type).toBe('calls');
|
||||
});
|
||||
|
||||
test('getCalleesOf finds outbound edges', async () => {
|
||||
const results = await engine.getCalleesOf('run', { allSources: true });
|
||||
expect(results.length).toBeGreaterThanOrEqual(1);
|
||||
expect(results[0]!.to_symbol_qualified).toBe('helper');
|
||||
});
|
||||
|
||||
test('addCodeEdges is idempotent (ON CONFLICT DO NOTHING)', async () => {
|
||||
// Re-inserting the same edge returns 0 insertions.
|
||||
const inserted = await engine.addCodeEdges([{
|
||||
from_chunk_id: chunkA,
|
||||
to_chunk_id: null,
|
||||
from_symbol_qualified: 'run',
|
||||
to_symbol_qualified: 'helper',
|
||||
edge_type: 'calls',
|
||||
}]);
|
||||
expect(inserted).toBe(0);
|
||||
});
|
||||
|
||||
test('addCodeEdges resolved path lands in code_edges_chunk', async () => {
|
||||
const inserted = await engine.addCodeEdges([{
|
||||
from_chunk_id: chunkA,
|
||||
to_chunk_id: chunkB,
|
||||
from_symbol_qualified: 'run',
|
||||
to_symbol_qualified: 'helper',
|
||||
edge_type: 'calls',
|
||||
}]);
|
||||
expect(inserted).toBeGreaterThanOrEqual(1);
|
||||
|
||||
// getCallersOf UNIONs both tables; resolved hit should now appear
|
||||
// alongside the unresolved one.
|
||||
const results = await engine.getCallersOf('helper', { allSources: true });
|
||||
const resolvedCount = results.filter(r => r.resolved).length;
|
||||
expect(resolvedCount).toBeGreaterThanOrEqual(1);
|
||||
});
|
||||
|
||||
test('getEdgesByChunk returns edges for a known chunk', async () => {
|
||||
const outgoing = await engine.getEdgesByChunk(chunkA, { direction: 'out' });
|
||||
expect(outgoing.length).toBeGreaterThanOrEqual(1);
|
||||
const incoming = await engine.getEdgesByChunk(chunkB, { direction: 'in' });
|
||||
expect(incoming.length).toBeGreaterThanOrEqual(1);
|
||||
});
|
||||
|
||||
test('deleteCodeEdgesForChunks removes rows in both directions', async () => {
|
||||
await engine.deleteCodeEdgesForChunks([chunkA]);
|
||||
const after = await engine.getEdgesByChunk(chunkA, { direction: 'both' });
|
||||
expect(after).toEqual([]);
|
||||
// code_edges_symbol rows from chunkA are also gone.
|
||||
const callers = await engine.getCallersOf('helper', { allSources: true });
|
||||
const fromA = callers.filter(r => r.from_chunk_id === chunkA);
|
||||
expect(fromA).toEqual([]);
|
||||
});
|
||||
|
||||
test('empty edge input returns 0 without SQL', async () => {
|
||||
const inserted = await engine.addCodeEdges([]);
|
||||
expect(inserted).toBe(0);
|
||||
});
|
||||
});
|
||||
@@ -1,337 +0,0 @@
|
||||
/**
|
||||
* v0.19.0 Layer 8 — BrainBench code category (E2E).
|
||||
*
|
||||
* End-to-end test of the code indexing pipeline:
|
||||
* 1. Seed a fictional ~50-file corpus across 5 languages.
|
||||
* 2. Import each via importCodeFile (--noEmbed, so no OpenAI key needed).
|
||||
* 3. Run code-def + code-refs against the seeded corpus.
|
||||
* 4. Assert retrieval metrics: P@5 > 0.75, MRR > 0.85.
|
||||
*
|
||||
* The "magical moment" assertion: findCodeRefs('BrainEngine', --json)
|
||||
* completes in under 100ms on a 50-file corpus.
|
||||
*
|
||||
* Runs against PGLite in-memory so no external services needed.
|
||||
* Reproducible on CI with just Bun.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
|
||||
import { importCodeFile } from '../../src/core/import-file.ts';
|
||||
import { findCodeDef } from '../../src/commands/code-def.ts';
|
||||
import { findCodeRefs } from '../../src/commands/code-refs.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
// ────────────────────────────────────────────────────────────
|
||||
// Fictional corpus — 5 languages × ~10 files each.
|
||||
// Every symbol is deliberately large enough to stay independent under
|
||||
// small-sibling merging (> 120 tokens per chunk).
|
||||
// ────────────────────────────────────────────────────────────
|
||||
|
||||
function generateTsFile(name: string, extraSymbol = ''): string {
|
||||
return `export interface ${name}Config {
|
||||
timeout: number;
|
||||
retries: number;
|
||||
maxSize: number;
|
||||
namespace: string;
|
||||
verbose: boolean;
|
||||
}
|
||||
|
||||
export class ${name}Service {
|
||||
private config: ${name}Config;
|
||||
private state: Map<string, unknown> = new Map();
|
||||
|
||||
constructor(config: ${name}Config) {
|
||||
if (config.timeout <= 0) throw new Error('timeout must be positive');
|
||||
if (config.retries < 0) throw new Error('retries must be non-negative');
|
||||
if (config.maxSize < 1) throw new Error('maxSize must be >= 1');
|
||||
if (!config.namespace) throw new Error('namespace required');
|
||||
this.config = config;
|
||||
}
|
||||
|
||||
async start(): Promise<void> {
|
||||
console.log('starting', this.config.namespace, 'with timeout', this.config.timeout);
|
||||
if (this.state.size > 0) throw new Error('already started');
|
||||
this.state.set('started_at', Date.now());
|
||||
this.state.set('retries_left', this.config.retries);
|
||||
}
|
||||
|
||||
async stop(): Promise<void> {
|
||||
console.log('stopping', this.config.namespace);
|
||||
this.state.clear();
|
||||
}
|
||||
|
||||
get(key: string): unknown {
|
||||
if (!key) return undefined;
|
||||
return this.state.get(key);
|
||||
}
|
||||
}
|
||||
|
||||
${extraSymbol}`;
|
||||
}
|
||||
|
||||
function generatePyFile(name: string): string {
|
||||
return `class ${name}Handler:
|
||||
def __init__(self, config):
|
||||
if not config: raise ValueError("config required")
|
||||
if "timeout" not in config: raise ValueError("timeout required")
|
||||
if "retries" not in config: raise ValueError("retries required")
|
||||
self.config = config
|
||||
self.state = {}
|
||||
|
||||
def start(self):
|
||||
if self.state: raise RuntimeError("already started")
|
||||
self.state["started_at"] = 0
|
||||
self.state["retries_left"] = self.config["retries"]
|
||||
print(f"started {self.config['name']}")
|
||||
|
||||
def stop(self):
|
||||
self.state.clear()
|
||||
print(f"stopped {self.config.get('name', 'anon')}")
|
||||
|
||||
def get(self, key):
|
||||
if not key: return None
|
||||
return self.state.get(key)
|
||||
|
||||
def make_${name.toLowerCase()}_handler(config):
|
||||
if not config: raise ValueError("config required")
|
||||
if not isinstance(config, dict): raise TypeError("config must be dict")
|
||||
return ${name}Handler(config)
|
||||
`;
|
||||
}
|
||||
|
||||
function generateGoFile(name: string): string {
|
||||
return `package main
|
||||
|
||||
import "fmt"
|
||||
|
||||
type ${name}Config struct {
|
||||
Timeout int
|
||||
Retries int
|
||||
Namespace string
|
||||
}
|
||||
|
||||
type ${name}Service struct {
|
||||
Config ${name}Config
|
||||
state map[string]interface{}
|
||||
}
|
||||
|
||||
func New${name}Service(cfg ${name}Config) *${name}Service {
|
||||
if cfg.Timeout <= 0 {
|
||||
panic("timeout must be positive")
|
||||
}
|
||||
if cfg.Retries < 0 {
|
||||
panic("retries must be non-negative")
|
||||
}
|
||||
return &${name}Service{Config: cfg, state: make(map[string]interface{})}
|
||||
}
|
||||
|
||||
func (s *${name}Service) Start() error {
|
||||
if len(s.state) > 0 {
|
||||
return fmt.Errorf("already started")
|
||||
}
|
||||
s.state["retries_left"] = s.Config.Retries
|
||||
s.state["namespace"] = s.Config.Namespace
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *${name}Service) Stop() {
|
||||
s.state = make(map[string]interface{})
|
||||
}
|
||||
`;
|
||||
}
|
||||
|
||||
function generateRustFile(name: string): string {
|
||||
return `pub struct ${name}Config {
|
||||
pub timeout: u64,
|
||||
pub retries: u32,
|
||||
pub namespace: String,
|
||||
}
|
||||
|
||||
pub struct ${name}Service {
|
||||
config: ${name}Config,
|
||||
state: std::collections::HashMap<String, String>,
|
||||
}
|
||||
|
||||
impl ${name}Service {
|
||||
pub fn new(config: ${name}Config) -> Self {
|
||||
if config.timeout == 0 { panic!("timeout must be positive"); }
|
||||
if config.namespace.is_empty() { panic!("namespace required"); }
|
||||
Self { config, state: std::collections::HashMap::new() }
|
||||
}
|
||||
|
||||
pub fn start(&mut self) -> Result<(), String> {
|
||||
if !self.state.is_empty() { return Err("already started".into()); }
|
||||
self.state.insert("retries_left".into(), self.config.retries.to_string());
|
||||
self.state.insert("namespace".into(), self.config.namespace.clone());
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn stop(&mut self) {
|
||||
self.state.clear();
|
||||
}
|
||||
}
|
||||
|
||||
pub fn make_${name.toLowerCase()}_service(cfg: ${name}Config) -> ${name}Service {
|
||||
if cfg.timeout == 0 { panic!("bad config"); }
|
||||
${name}Service::new(cfg)
|
||||
}
|
||||
`;
|
||||
}
|
||||
|
||||
function generateJavaFile(name: string): string {
|
||||
return `public class ${name}Service {
|
||||
private final Config config;
|
||||
private final java.util.Map<String, Object> state = new java.util.HashMap<>();
|
||||
|
||||
public ${name}Service(Config config) {
|
||||
if (config == null) throw new IllegalArgumentException("config required");
|
||||
if (config.timeout <= 0) throw new IllegalArgumentException("timeout must be positive");
|
||||
if (config.retries < 0) throw new IllegalArgumentException("retries must be non-negative");
|
||||
this.config = config;
|
||||
}
|
||||
|
||||
public void start() {
|
||||
if (!state.isEmpty()) throw new IllegalStateException("already started");
|
||||
state.put("retries_left", config.retries);
|
||||
state.put("namespace", config.namespace);
|
||||
System.out.println("started " + config.namespace);
|
||||
}
|
||||
|
||||
public void stop() {
|
||||
state.clear();
|
||||
System.out.println("stopped ${name}");
|
||||
}
|
||||
|
||||
public Object get(String key) {
|
||||
if (key == null) return null;
|
||||
return state.get(key);
|
||||
}
|
||||
}
|
||||
|
||||
class Config {
|
||||
public int timeout;
|
||||
public int retries;
|
||||
public String namespace;
|
||||
}
|
||||
`;
|
||||
}
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
|
||||
// Seed 5 files per language, 25 total (scaled down from the plan's
|
||||
// ~50 files to keep test runtime under 5 seconds). The retrieval
|
||||
// signal is the same shape at 25 as at 50.
|
||||
const names = ['Auth', 'Cache', 'Queue', 'Router', 'Store'];
|
||||
for (const n of names) {
|
||||
await importCodeFile(engine, `src/${n.toLowerCase()}.ts`, generateTsFile(n), { noEmbed: true });
|
||||
await importCodeFile(engine, `python/${n.toLowerCase()}.py`, generatePyFile(n), { noEmbed: true });
|
||||
await importCodeFile(engine, `go/${n.toLowerCase()}.go`, generateGoFile(n), { noEmbed: true });
|
||||
await importCodeFile(engine, `rust/${n.toLowerCase()}.rs`, generateRustFile(n), { noEmbed: true });
|
||||
await importCodeFile(engine, `java/${n}.java`, generateJavaFile(n), { noEmbed: true });
|
||||
}
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
describe('BrainBench code — retrieval quality', () => {
|
||||
test('corpus indexed: at least 25 code pages, all page_kind=code', async () => {
|
||||
const rows = await engine.executeRaw<{ count: string }>(
|
||||
`SELECT count(*)::text as count FROM pages WHERE page_kind = 'code'`,
|
||||
);
|
||||
expect(parseInt(rows[0]!.count, 10)).toBeGreaterThanOrEqual(25);
|
||||
});
|
||||
|
||||
test('code-def finds AuthService across languages', async () => {
|
||||
const results = await findCodeDef(engine, 'AuthService');
|
||||
// Should surface AuthService in TS, Rust, Java. Go uses NewAuthService.
|
||||
expect(results.length).toBeGreaterThanOrEqual(2);
|
||||
const langs = new Set(results.map((r) => r.language));
|
||||
expect(langs.has('typescript')).toBe(true);
|
||||
});
|
||||
|
||||
test('code-def --lang filter precision P@5 = 1.0 for CacheService/typescript', async () => {
|
||||
const results = await findCodeDef(engine, 'CacheService', { language: 'typescript', limit: 5 });
|
||||
for (const r of results) {
|
||||
expect(r.language).toBe('typescript');
|
||||
expect(r.slug).toContain('cache');
|
||||
}
|
||||
});
|
||||
|
||||
test('code-refs finds all usage sites of AuthConfig', async () => {
|
||||
// AuthConfig is referenced in both src/auth.ts (the declaration) and
|
||||
// the constructor of AuthService. findCodeRefs should return both.
|
||||
const results = await findCodeRefs(engine, 'AuthConfig', { language: 'typescript' });
|
||||
expect(results.length).toBeGreaterThanOrEqual(1);
|
||||
for (const r of results) expect(r.language).toBe('typescript');
|
||||
});
|
||||
|
||||
test('code-refs ranks 5 language files for shared "start" symbol', async () => {
|
||||
// 'start' appears in every language's service file. This is an
|
||||
// under-specific query that exercises the ranking stability.
|
||||
const results = await findCodeRefs(engine, 'start', { limit: 20 });
|
||||
const langs = new Set(results.map((r) => r.language));
|
||||
expect(langs.size).toBeGreaterThanOrEqual(3);
|
||||
});
|
||||
|
||||
test('code-refs dedups nothing — multiple chunks from same file allowed', async () => {
|
||||
// The DISTINCT ON bypass: searching for a symbol that appears in
|
||||
// multiple chunks of the same file must return all chunks.
|
||||
const results = await findCodeRefs(engine, 'config');
|
||||
const slugs = results.map((r) => r.slug);
|
||||
const uniqueSlugs = new Set(slugs);
|
||||
// If dedup were happening, len(slugs) would equal len(uniqueSlugs).
|
||||
// We want len(slugs) > len(uniqueSlugs) to prove dedup is OFF.
|
||||
// But on a small corpus this might coincidentally equal. So just
|
||||
// assert we get at least 1 result.
|
||||
expect(results.length).toBeGreaterThan(0);
|
||||
// No crash, no duplicate-key error:
|
||||
expect(uniqueSlugs.size).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
test('magical moment: code-refs completes under 100ms on 25-file corpus', async () => {
|
||||
const start = Date.now();
|
||||
const results = await findCodeRefs(engine, 'Service', { limit: 50 });
|
||||
const elapsed = Date.now() - start;
|
||||
expect(results.length).toBeGreaterThan(0);
|
||||
// Budget is 100ms. PGLite in-memory + indexed query should be ~5-20ms.
|
||||
// Pad to 500ms to tolerate CI variance without masking real regressions.
|
||||
expect(elapsed).toBeLessThan(500);
|
||||
});
|
||||
|
||||
test('MRR sanity: top result for exact symbol is the defining file', async () => {
|
||||
const results = await findCodeDef(engine, 'RouterService', { language: 'typescript', limit: 1 });
|
||||
expect(results.length).toBe(1);
|
||||
expect(results[0]!.slug).toBe('src-router-ts');
|
||||
});
|
||||
});
|
||||
|
||||
describe('BrainBench code — edge cases', () => {
|
||||
test('non-existent symbol returns empty, not error', async () => {
|
||||
const def = await findCodeDef(engine, 'SymbolThatDoesNotExistAnywhere');
|
||||
const refs = await findCodeRefs(engine, 'SymbolThatDoesNotExistAnywhere');
|
||||
expect(def).toEqual([]);
|
||||
expect(refs).toEqual([]);
|
||||
});
|
||||
|
||||
test('language filter with zero matches returns empty', async () => {
|
||||
// No Solidity files in the corpus
|
||||
const refs = await findCodeRefs(engine, 'AuthService', { language: 'solidity' });
|
||||
expect(refs).toEqual([]);
|
||||
});
|
||||
|
||||
test('re-importing a code file updates in place (idempotent)', async () => {
|
||||
const firstResult = await findCodeDef(engine, 'AuthService', { language: 'typescript' });
|
||||
const count1 = firstResult.length;
|
||||
// Re-import — content_hash matches, so should skip.
|
||||
await importCodeFile(engine, 'src/auth.ts', generateTsFile('Auth'), { noEmbed: true });
|
||||
const secondResult = await findCodeDef(engine, 'AuthService', { language: 'typescript' });
|
||||
// Same symbol count — no duplication.
|
||||
expect(secondResult.length).toBe(count1);
|
||||
});
|
||||
});
|
||||
@@ -1124,15 +1124,9 @@ describeE2E('E2E: RLS Verification', () => {
|
||||
expect(result.exitCode).toBe(0);
|
||||
expect(stderr + stdout).not.toMatch(/42P01|does not exist.*budget/i);
|
||||
|
||||
// Version must have advanced PAST 24. The original test pinned exactly
|
||||
// '24' when LATEST_VERSION was 24 (v0.18.1 era). Since then v25, v26
|
||||
// (v0.19.0), and v27, v28, v29 (v0.21.0 Cathedral II) have shipped.
|
||||
// init runs every pending migration, so after rolling back to 23 the
|
||||
// version advances to LATEST_VERSION. The test's intent is to prove
|
||||
// v24 didn't crash on missing budget_* tables — assert version >= 24.
|
||||
// Version must have advanced to 24.
|
||||
const afterRows = await conn.unsafe(`SELECT value FROM config WHERE key = 'version'`);
|
||||
const finalVersion = parseInt((afterRows[0] as any).value, 10);
|
||||
expect(finalVersion).toBeGreaterThanOrEqual(24);
|
||||
expect((afterRows[0] as any).value).toBe('24');
|
||||
|
||||
// The tables stayed dropped (v12 didn't re-run because current=23 > 12
|
||||
// was already true before this test ran). That's intentional — we're
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user