mirror of
https://github.com/garrytan/gbrain.git
synced 2026-08-17 02:12:40 +00:00
Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
7bee2e48f3 | ||
|
|
3aa064bcc6 | ||
|
|
53bb974eaf | ||
|
|
6136e13997 | ||
|
|
b3b43d0f91 | ||
|
|
5b9a87f1a3 | ||
|
|
661f1f05cc | ||
|
|
3fec2123d2 | ||
|
|
176836f84d | ||
|
|
539d015cc5 | ||
|
|
91464564cd | ||
|
|
bd049d2969 | ||
|
|
784358f5fd | ||
|
|
d58bb2b0bb | ||
|
|
b252acfce3 | ||
|
|
bdd23cdede | ||
|
|
45689dd1bd | ||
|
|
6920744dd8 | ||
|
|
3df20f9f18 | ||
|
|
e58abd652c | ||
|
|
a104f98dca | ||
|
|
2ac6959b46 | ||
|
|
18ec732e1b | ||
|
|
fd8be831c5 | ||
|
|
9664cad329 | ||
|
|
fcc6e670f2 | ||
|
|
2a17a4dab5 | ||
|
|
ddd66e1d25 | ||
|
|
faf5cdba54 | ||
|
|
e9fa962929 | ||
|
|
0413c93e72 | ||
|
|
56aac51a08 | ||
|
|
0bbaed2e48 | ||
|
|
5dfd2696d1 | ||
|
|
ae8753c872 | ||
|
|
b30f0aa7cb | ||
|
|
2c758e23e8 | ||
|
|
4320527785 | ||
|
|
3a28d2612a | ||
|
|
bf4cf8a6dd | ||
|
|
0ce4064d13 | ||
|
|
032af6e5f7 | ||
|
|
29dd67c8ae | ||
|
|
d2ac2aef49 | ||
|
|
10079efe40 | ||
|
|
16782aee7f | ||
|
|
3126b8fdfc | ||
|
|
b6c75d802f | ||
|
|
07901b1886 | ||
|
|
d7c9625395 | ||
|
|
7a65f182aa | ||
|
|
9690140bf3 | ||
|
|
d014707e3c | ||
|
|
dde1bd9353 | ||
|
|
f0a28eb276 | ||
|
|
5ecab70a21 | ||
|
|
70beb16b8b | ||
|
|
7efb1694cc | ||
|
|
4beafbae46 | ||
|
|
5f84fb8813 | ||
|
|
14f0674bcf | ||
|
|
4871ae0c05 | ||
|
|
d9ac24744c | ||
|
|
c19a8808b4 | ||
|
|
ea08effd02 | ||
|
|
3fafb69b07 |
@@ -0,0 +1,16 @@
|
||||
# Line-ending policy.
|
||||
#
|
||||
# Shell scripts MUST be checked out with LF endings on every platform.
|
||||
# Git for Windows installs with `core.autocrlf=true` by default, which
|
||||
# rewrites LF -> CRLF on checkout. A strict bash (WSL, Linux CI, macOS)
|
||||
# then chokes on the trailing CR:
|
||||
#
|
||||
# scripts/run-unit-parallel.sh: line 23: $'\r': command not found
|
||||
# scripts/run-unit-parallel.sh: line 24: set: pipefail : invalid option name
|
||||
# scripts/run-unit-parallel.sh: line 32: syntax error near unexpected token `$'{\r''
|
||||
#
|
||||
# That silently disabled `bun run test`, `bun run verify`, `bun run ci:local`
|
||||
# and `bun run test:e2e` for Windows contributors, since all four dispatch
|
||||
# through bash. `eol=lf` pins the checkout regardless of the user's
|
||||
# core.autocrlf setting.
|
||||
*.sh text eol=lf
|
||||
+5
-2
@@ -1,4 +1,7 @@
|
||||
node_modules/
|
||||
# No trailing slash: a bare `node_modules/` pattern matches directories only,
|
||||
# so a *symlink* named node_modules slips past it and can be committed
|
||||
# (that's how the /tmp-pointing symlink in faf5cdba got in). Match any type.
|
||||
node_modules
|
||||
bin/
|
||||
.DS_Store
|
||||
*.log
|
||||
@@ -15,7 +18,7 @@ supabase/.temp/
|
||||
# self-contained binaries (the bun --compile path embeds it via
|
||||
# `import path from 'admin/dist/index.html' with { type: 'file' }`).
|
||||
# Build via: cd admin && bun install && bun run build.
|
||||
admin/node_modules/
|
||||
admin/node_modules
|
||||
.idea
|
||||
eval/reports/
|
||||
eval/data/world-v1/world.html
|
||||
|
||||
+152
@@ -2,6 +2,158 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.42.67.0] - 2026-07-28
|
||||
|
||||
**If you develop GBrain on Windows, the test and check commands now actually run. Until this release they were quietly doing almost nothing.**
|
||||
|
||||
`bun run test`, `bun run verify`, `bun run ci:local` and `bun run test:e2e` all hand off to shell scripts, and on Windows that hand-off was broken in two separate places. The commands did not stop with an obvious error. They reported a result, so a run could look finished when barely any of the checks had actually inspected anything. On a clean Windows clone, `bun run verify` got 1 check to pass and 31 to fail. It now gets 25 to pass and 7 to fail, and none of the 7 are caused by this change.
|
||||
|
||||
The first problem was line endings. Git for Windows installs with `core.autocrlf=true`, which rewrites shell scripts to Windows line endings when you clone or check out. Bash refuses to run those, so a script died on its second line before doing any work. The scripts stored in the repository were always correct; only the copy on your disk was wrong. A new `.gitattributes` pins every `.sh` file to Unix line endings at checkout, no matter how your Git is configured.
|
||||
|
||||
The second problem was how the checks were started. Thirty three of them pointed straight at a `.sh` file. On macOS and Linux the shell reads the `#!/usr/bin/env bash` line at the top of the script and runs it correctly. Bun on Windows does not do that, so those commands failed the moment they were called. They now go through `bash` explicitly, the same way the other eleven were already written.
|
||||
|
||||
Nothing changes for macOS and Linux. No stored file content moves, and no check behaves differently on those platforms.
|
||||
|
||||
## To take advantage of v0.42.67.0
|
||||
|
||||
Only Windows contributors need to do anything, and only once. `.gitattributes` applies at checkout time, so shell scripts already sitting on your disk keep their old line endings until you refresh them.
|
||||
|
||||
1. **Refresh the working copy** from the repository root:
|
||||
```bash
|
||||
git rm --cached -r . -q
|
||||
git reset --hard
|
||||
```
|
||||
2. **Confirm bash can read the scripts:**
|
||||
```bash
|
||||
bash -n scripts/run-unit-parallel.sh
|
||||
```
|
||||
Silence means it worked. `$'\r': command not found` means step 1 did not take effect.
|
||||
3. **Run the gate:**
|
||||
```bash
|
||||
bun run verify
|
||||
```
|
||||
|
||||
### Itemized changes
|
||||
|
||||
- New root `.gitattributes` pins `*.sh text eol=lf`, so shell scripts check out with Unix line endings regardless of the contributor's `core.autocrlf` setting. All 59 tracked `.sh` files were already stored with Unix endings, so `git add --renormalize .` reports nothing to do and no stored content changes.
|
||||
- `package.json` now routes the remaining 33 `.sh` check commands through `bash`, matching the 11 that already did. Every tracked `.sh` file carries a bash shebang (52 `#!/usr/bin/env bash` and 7 `#!/bin/bash`), so the treatment is uniform across all of them.
|
||||
- The five `scripts/*.ts` entries still run under bun and are untouched.
|
||||
- `CONTRIBUTING.md` gains a Windows section covering the one-time working-copy refresh and the `bash scripts/<name>.sh` convention for new checks.
|
||||
- `docs/TESTING.md` records how the test commands dispatch through bash, and notes that three tree-walking checks plus `typecheck` can exceed the 120s per-check cap on Windows while passing on Linux and macOS.
|
||||
|
||||
## [0.42.66.1] - 2026-07-27
|
||||
|
||||
### Fixed
|
||||
|
||||
- `gbrain doctor` now treats embedding columns wider than pgvector's HNSW limit as healthy exact-scan configurations instead of prescribing an index PostgreSQL cannot build.
|
||||
- Local CI now passes an empty Docker mount list correctly and compiles the embedded-WASM smoke binary from container-local storage on Docker Desktop.
|
||||
|
||||
## [0.42.66.0] - 2026-07-24
|
||||
|
||||
**54 verified fixes from the community backlog: background enrichment stops wasting money on dead pages, autopilot stops killing its own healthy runs, and search respects your settings.**
|
||||
|
||||
This release is the second big sweep through the open pull-request backlog, with every change reviewed and tested individually before merging. The theme is trust in the background machinery. The overnight "dream" cycle now remembers which pages produced nothing and stops re-reading them every night, meters its small-model calls against your spend caps, and keeps claim proposals from silently overwriting each other. Long consolidation runs get a 30-minute deadline instead of being killed at 10 minutes mid-work. A wedged server boot now releases its database lock instead of blocking every later command.
|
||||
|
||||
Search behaves the way you configured it: the recency-decay setting now actually applies to hybrid search, a local `list_pages` call returns as many rows as you asked for, and when a listing is cut short it says so instead of looking complete. Slack conversation exports parse cleanly, with an optional AI fallback for formats the parser does not know.
|
||||
|
||||
New provider recipes: DashScope reranking, OpenRouter reranking, and a claude-cli recipe for dispatching subagents through the gateway.
|
||||
|
||||
## To take advantage of v0.42.66.0
|
||||
|
||||
`gbrain upgrade` should do this automatically. One schema migration ships in this release (v125, take-proposal idempotency); it is idempotent and needs no manual action.
|
||||
|
||||
1. **Upgrade and verify:**
|
||||
```bash
|
||||
gbrain upgrade
|
||||
gbrain doctor
|
||||
gbrain stats
|
||||
```
|
||||
2. **If `gbrain doctor` warns about a partial migration**, run the orchestrator manually:
|
||||
```bash
|
||||
gbrain apply-migrations --yes
|
||||
```
|
||||
3. **If any step fails,** please file an issue at https://github.com/garrytan/gbrain/issues with the output of `gbrain doctor` and `~/.gbrain/upgrade-errors.jsonl` if it exists.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
#### Dream cycle, takes, and spend control
|
||||
|
||||
- Pages whose extraction yields zero claims are memoized, so the cycle stops re-spending on them every night. (#2514, #3319, contributed by @ivandebot)
|
||||
- Zero-yield pages are tombstoned so `extract_atoms` stops rediscovering them. (#2144, #3304, contributed by @ChenyqThu)
|
||||
- `extract_atoms` Haiku calls are metered against the cost gate. (#2371, #3329, contributed by @TheRealMrSystem)
|
||||
- `extract_atoms` stamps concepts so `synthesize_concepts` has material to work with. (#2123, #3308, contributed by @ChenyqThu)
|
||||
- `extract_facts` requires a live backing page, not just a non-NULL entity slug. (#2497, #3321, contributed by @javieraldape)
|
||||
- Multi-claim pages keep every proposal instead of only the first (migration v125 makes the idempotency key per claim). (#3297, contributed by @rp-agent-bot)
|
||||
- Superseding a take now queries the active row first. (#3275, contributed by @arisgysel-design)
|
||||
- Takes keyword search matches words inside long claims via `word_similarity`. (#3267)
|
||||
- Dream-generated orphan pages stay scoped to their source. (#2368, #3344, contributed by @snvtac)
|
||||
- Drift detection is wired into the dream cycle, report-only for now. (#2653, #3317)
|
||||
|
||||
#### Autopilot, jobs, and serve
|
||||
|
||||
- Full consolidation cycles get a 30-minute timeout floor; lighter dispatches keep the interval-derived budget. (#2852, #3338, contributed by @sanchalr)
|
||||
- The cron wrapper exports `~/.bun/bin` onto PATH so autopilot survives minimal environments. (#2013, #3305, contributed by @klampatech)
|
||||
- Dead or cancelled jobs no longer block idempotent re-submission. (#2253, #3306, contributed by @rafaelreis-r)
|
||||
- Contextual reindex jobs get a default timeout. (#2611, #3323, contributed by @spiky02plateau)
|
||||
- Onboarding stops repeating the same auto-remediation within a single run. (#2854, #3342, contributed by @sanchalr)
|
||||
- A wedged `gbrain serve` boot hits a readiness deadline and releases the PGLite lock. (#3335)
|
||||
|
||||
#### Search, retrieval, and health
|
||||
|
||||
- The recency-decay config is honored on the hybrid search path. (#2386, #3312, contributed by @rwbaker)
|
||||
- `list_pages` honors explicit limits for local callers, warns on remote clamping, and threads `offset`. (#2591, #3322, contributed by @deacon-botdoctor)
|
||||
- Truncated `list_pages` results say so instead of silently capping. (#2865, #3341, contributed by @paul-0320)
|
||||
- Negative metrics no longer invert trajectory regression signals. (#2621, #3324, contributed by @morluto)
|
||||
- Per-chunk synopsis generation in contextual retrieval is concurrency-bounded. (#2628, #3326, contributed by @spiky02plateau)
|
||||
- Graph health metrics count `entity` pages. (#2639, #3330, contributed by @tylr-r)
|
||||
|
||||
#### Ingestion, extraction, and links
|
||||
|
||||
- Conversation parsing gains an opt-in LLM fallback for unknown formats. (#2247, #3371, contributed by @danwiggins)
|
||||
- Normalized Slack markdown parses into conversations. (#3289, #3372, contributed by @danwiggins)
|
||||
- Conversation backfill outcomes are durable, so completed pages skip on the next run. (#3293, #3373, contributed by @danwiggins)
|
||||
- Reference-style wikilinks are recognized during extraction. (#2071, #3303, contributed by @mzkarami)
|
||||
- `[[wikilink]]` frontmatter values resolve via global basename lookup. (#2406, #3313, contributed by @spiky02plateau)
|
||||
- Incremental push syncs extract links. (#2850, #3337, contributed by @patentsong)
|
||||
- `<think>` reasoning tags in extractor output are handled. (#2559, #3318, contributed by @qaz8545355)
|
||||
- Tiktoken special tokens no longer crash code-chunker token estimates. (#2453, #3315, contributed by @Jiglet)
|
||||
- Source config stops re-wrapping into a growing JSON string scalar. (#2829, #3334, contributed by @1alessio)
|
||||
|
||||
#### Providers and recipes
|
||||
|
||||
- DashScope reranking recipe (DashScope serves a plural `/reranks` endpoint under its compatible API). (#2644, #3328, contributed by @YiconZiwei)
|
||||
- OpenRouter reranking touchpoint. (#2164, #3302, contributed by @Hippityy)
|
||||
- claude-cli recipe for native gateway-based subagent dispatch. (#2277, #3310, contributed by @brettdavies)
|
||||
- Prefixed model IDs work on the openai-compatible embedding-dimensions path. (#2325, #3309, contributed by @noetherly)
|
||||
- Embeddings stamp the gateway-resolved model in `content_chunks.model`, not the compiled default. (#2846, #3343, contributed by @SailorJoe6)
|
||||
- Bun-on-Windows write-through EEXIST fixed, non-Anthropic `--max-cost` pricing works, dream pages excluded from enrich. (#2407, #3316, contributed by @nguyenchiviet)
|
||||
- Supabase signed URLs prepend `/storage/v1`. (#2565, #3320, contributed by @danwiggins)
|
||||
|
||||
#### Sources, auth, and multi-brain
|
||||
|
||||
- Federated-source pages are visible to `get_page`, `list_pages`, `resolve_slugs`, and no-grant MCP callers. (#3242, #3301)
|
||||
- Admin-gated rescope surface for DCR clients stuck on a default scope. (#3299)
|
||||
- `whoami` exposes OAuth source grants. (#3279, #3332, contributed by @boundless-forest)
|
||||
- Thin-client `--source` maps onto `source_id` for remote-routed operations. (#3086)
|
||||
|
||||
#### CLI, doctor, and init
|
||||
|
||||
- `gbrain doctor` stops claiming "Brain is at target" when the target is unreachable. (#2151, #3339, contributed by @brettdavies)
|
||||
- Doctor gains a raw-source persistence guarantee for synthesized pages, warn-only for now. (#3300)
|
||||
- Doctor timeline labels disambiguate entity coverage from the brain-score component. (#2298, #3073, contributed by @TurgutKural)
|
||||
- Unknown `gbrain init` flags are rejected before migrations run. (#2201, #3307, contributed by @caioribeiroclw-pixel)
|
||||
- The init soul-audit hint points at the conversational skill, not a nonexistent CLI verb. (#2486, #3314, contributed by @SeanGearin)
|
||||
- `--force` retry escapes completed migration-ledger entries. (#2616, #3325, contributed by @spiky02plateau)
|
||||
- PGLite data-dir lock contention gets a clear error message. (#2658, #3336, contributed by @zaycruz)
|
||||
- Frontmatter validation derives slugs from the brain root, not the absolute path. (#2340, #3311, contributed by @alessioalionco)
|
||||
|
||||
#### For contributors
|
||||
|
||||
- Docker network isolation guidance for co-located self-hosted Postgres. (#3270, #3331)
|
||||
- `CLAUDE.local.md` / `AGENTS.local.md` are gitignored. (#3290, contributed by @igbymyboy)
|
||||
- The hybrid-reranker integration test isolates `GBRAIN_HOME`. (#1527, #3327, contributed by @Willisbest)
|
||||
- Test-shard scripts capture the real exit code before watchdog teardown in the no-timeout fallback. (#2864, #3340, contributed by @paul-0320)
|
||||
|
||||
## [0.42.65.0] - 2026-07-23
|
||||
|
||||
**A large maintenance release: 93 verified fixes and small features merged since v0.42.64.0, most of them community contributions.**
|
||||
|
||||
@@ -11,6 +11,28 @@ bun test
|
||||
|
||||
Requires Bun 1.0+.
|
||||
|
||||
### Windows
|
||||
|
||||
`bun run test`, `verify`, `ci:local` and `test:e2e` all dispatch through bash, so
|
||||
the shell scripts under `scripts/` must be checked out with Unix line endings.
|
||||
The root `.gitattributes` pins `*.sh text eol=lf`, which overrides the
|
||||
`core.autocrlf=true` that Git for Windows installs by default. A fresh clone is
|
||||
correct with no extra steps.
|
||||
|
||||
If you cloned before that pin existed, your working copy still has the old
|
||||
Windows line endings and bash will fail with `$'\r': command not found`. Refresh
|
||||
it once, from the repository root:
|
||||
|
||||
```bash
|
||||
git rm --cached -r . -q
|
||||
git reset --hard
|
||||
bash -n scripts/run-unit-parallel.sh # silence means bash can read the scripts
|
||||
```
|
||||
|
||||
Every `check:*` entry in `package.json` invokes its script as `bash scripts/<name>.sh`
|
||||
rather than relying on the shebang, because bun on Windows cannot exec a `.sh`
|
||||
directly. Keep that prefix when you add a new shell-script check.
|
||||
|
||||
## Project structure
|
||||
|
||||
```
|
||||
@@ -163,6 +185,14 @@ host port with `GBRAIN_CI_PG_PORT=5435 bun run ci:local` if 5434 collides.
|
||||
Fail-closed selector: an unmapped `src/` change runs all 29 E2E files. Hand-tune
|
||||
narrower mappings via `scripts/e2e-test-map.ts`.
|
||||
|
||||
### PR-side security checks
|
||||
|
||||
Besides the test gate, PRs may trigger three security workflows: Semgrep CE
|
||||
SAST (every PR — **advisory/non-blocking** while the baseline is tuned, so a
|
||||
Semgrep finding won't fail your PR), OSV-Scanner (only when `package.json` or
|
||||
`bun.lock` change), and actionlint (only when `.github/workflows/**` change).
|
||||
See `SECURITY.md` → "Automated security scanning" for details.
|
||||
|
||||
## Building
|
||||
|
||||
```bash
|
||||
|
||||
+24
@@ -8,6 +8,30 @@ on GitHub.
|
||||
|
||||
Do not open a public issue for security vulnerabilities.
|
||||
|
||||
## Automated security scanning
|
||||
|
||||
CI runs three automated security checks alongside secret scanning (Gitleaks):
|
||||
|
||||
- **Dependency vulnerabilities** — OSV-Scanner
|
||||
(`.github/workflows/osv-scanner.yml`) runs weekly and on any PR that touches
|
||||
`package.json` or `bun.lock`.
|
||||
- **Static analysis (SAST)** — Semgrep CE (`.github/workflows/semgrep.yml`)
|
||||
runs on every PR and weekly. It is currently **advisory (non-blocking)**
|
||||
while the finding baseline is tuned; the graduation path to a blocking check
|
||||
is documented in the workflow file.
|
||||
- **Release binary provenance** — release builds
|
||||
(`.github/workflows/release.yml`) attest each compiled binary with
|
||||
[GitHub artifact attestations](https://docs.github.com/en/actions/security-for-github-actions/using-artifact-attestations).
|
||||
Verify a downloaded release binary with:
|
||||
|
||||
```bash
|
||||
gh attestation verify ./gbrain-darwin-arm64 -R garrytan/gbrain
|
||||
gh attestation verify ./gbrain-linux-x64 -R garrytan/gbrain
|
||||
```
|
||||
|
||||
All security workflows use SHA-pinned actions and least-privilege permissions,
|
||||
enforced structurally by actionlint on every workflow change.
|
||||
|
||||
## Remote MCP Security
|
||||
|
||||
### ⚠️ Do NOT use open OAuth client registration for remote MCP
|
||||
|
||||
@@ -1,10 +1,31 @@
|
||||
# TODOS
|
||||
|
||||
## v0.42.67.0 follow-ups (Windows build tooling)
|
||||
|
||||
Filed as follow-ups from v0.42.67.0 (`.gitattributes` LF pin for `*.sh` +
|
||||
`bash` prefix on the 33 `package.json` check commands). Both items are newly
|
||||
observable: before that release these checks never executed on Windows at all,
|
||||
so nothing about their runtime was measurable.
|
||||
|
||||
- [ ] **P2 — three guard scripts exceed the 120s `run-verify-parallel.sh` cap on Windows.**
|
||||
With the dispatch fixed, `bun run verify` on Windows gets 25 passes and 7 failures, and
|
||||
`check:privacy`, `check:test-names` and `check:test-isolation` are timeouts rather than
|
||||
real failures (they pass on Linux and macOS well inside the cap). They walk the tree with
|
||||
per-file shell loops, which is far slower under Windows process creation. Either raise the
|
||||
cap for these three, or replace the per-file loop with a single `grep -r` pass. Same cap
|
||||
swallows `typecheck`, though standalone `bun run typecheck` exits 0.
|
||||
- [ ] **P3 — `check:wasm` cannot create its `node_modules` symlink on Windows.**
|
||||
`scripts/check-wasm-embedded.sh` fails with `ln: failed to create symbolic link
|
||||
'/tmp/gbrain-wasm-check.XXXX/node_modules': No such file or directory`. Unprivileged
|
||||
Windows accounts cannot create symlinks without developer mode. Consider a junction, a
|
||||
copy, or skipping the check with a clear message when symlink creation is unavailable.
|
||||
|
||||
## community fix-wave follow-ups (filed v0.42.60.0)
|
||||
|
||||
- [ ] **P2 — cherry-pick #2112's uncovered doctor.ts hunk.** Fix-wave A (#2820) superseded
|
||||
- [x] **P2 — cherry-pick #2112's uncovered doctor.ts hunk.** Fix-wave A (#2820) superseded
|
||||
most of #2112 but not its `checkSubagentCapability` fix (check explicit `models.subagent`
|
||||
before `models.tier.subagent`). Refile or cherry-pick; the rest of that PR is covered.
|
||||
before `models.tier.subagent`). Implemented: `checkSubagentCapability` now resolves
|
||||
`models.subagent` before tier/default fallbacks and has regression coverage.
|
||||
|
||||
## v0.42.59.0 follow-ups (five-fix rollup #2735–#2739)
|
||||
|
||||
@@ -61,17 +82,20 @@ Deferred from the provider-agnostic plumbing wave (#1249/#1250/#1292/#2271/#2209
|
||||
Plan + review trail at `~/.claude/plans/system-instruction-you-are-working-keen-newell.md`.
|
||||
The eng-review + Codex outside-voice narrowed the wave to these deferrals:
|
||||
|
||||
- [ ] **P2 — Capability-aware query expansion on OpenAI-compat providers (#2372).**
|
||||
- [x] **P2 — Capability-aware query expansion on OpenAI-compat providers (#2372).**
|
||||
Expansion only runs for recipes that declare an `expansion` touchpoint, and only the
|
||||
native providers (anthropic/openai/google) do. To make expansion work on
|
||||
litellm/openrouter/groq/together/deepseek you must ADD expansion touchpoints to those
|
||||
chat-capable recipes AND add a `generateObject`→`generateText` capability fallback for
|
||||
backends without strict structured outputs. Feature-shaped; overlaps the general
|
||||
OpenAI-compat proxy story (`docs/designs/COMMUNITY_IDEAS.md`). Community PR #2373 is a
|
||||
starting point. Where: `src/core/ai/gateway.ts:expand`, recipe files, `types.ts` (ExpansionTouchpoint).
|
||||
- [ ] **P2 — LiteLLM as a chat/expansion backend.** `litellm-proxy` declares ONLY an
|
||||
starting point. Implemented by #2373 plus the DeepSeek/Groq/Together recipe wave,
|
||||
LiteLLM chat/expansion support, and the OpenRouter expansion touchpoint. Where:
|
||||
`src/core/ai/gateway.ts:expand`, recipe files, `types.ts` (ExpansionTouchpoint).
|
||||
- [x] **P2 — LiteLLM as a chat/expansion backend.** `litellm-proxy` declares ONLY an
|
||||
embedding touchpoint, so `think`/chat on LiteLLM is dead. Add chat (and expansion) so a
|
||||
LiteLLM proxy is a full LLM backend, not embedding-only. The general OpenAI-compat proxy story.
|
||||
LiteLLM proxy is a full LLM backend, not embedding-only. Implemented by #2208.
|
||||
The general OpenAI-compat proxy story.
|
||||
- [ ] **P3 — Per-model embedding dims metadata on `EmbeddingTouchpoint`.** `default_dims`
|
||||
is recipe-wide, so a recipe (ollama) can't carry different native dims per model. This
|
||||
wave added the modern ollama model NAMES + a `trust_custom_dims` passthrough (user supplies
|
||||
|
||||
Vendored
-56
File diff suppressed because one or more lines are too long
Vendored
+56
File diff suppressed because one or more lines are too long
Vendored
+1
-1
@@ -7,7 +7,7 @@
|
||||
<link rel="preconnect" href="https://fonts.googleapis.com" />
|
||||
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
|
||||
<link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600&family=JetBrains+Mono:wght@400;500&display=swap" rel="stylesheet" />
|
||||
<script type="module" crossorigin src="/admin/assets/index-CoGEje3-.js"></script>
|
||||
<script type="module" crossorigin src="/admin/assets/index-CviJXT-1.js"></script>
|
||||
<link rel="stylesheet" crossorigin href="/admin/assets/index-GxkWX7v3.css">
|
||||
</head>
|
||||
<body>
|
||||
|
||||
+12
-2
@@ -39,11 +39,21 @@ export const api = {
|
||||
stats: () => apiFetch('/admin/api/stats'),
|
||||
health: () => apiFetch('/admin/api/health-indicators'),
|
||||
agents: () => apiFetch('/admin/api/agents'),
|
||||
sources: () => apiFetch('/admin/api/sources'),
|
||||
requests: (page = 1, qs = '') => apiFetch(`/admin/api/requests?page=${page}${qs}`),
|
||||
apiKeys: () => apiFetch('/admin/api/api-keys'),
|
||||
createApiKey: (name: string) => apiFetch('/admin/api/api-keys', { method: 'POST', body: JSON.stringify({ name }) }),
|
||||
revokeApiKey: (name: string) => apiFetch('/admin/api/api-keys/revoke', { method: 'POST', body: JSON.stringify({ name }) }),
|
||||
createApiKey(keyName: string) {
|
||||
return apiFetch('/admin/api/api-keys', { method: 'POST', body: JSON.stringify({ name: keyName }) });
|
||||
},
|
||||
revokeApiKey(keyName: string) {
|
||||
return apiFetch('/admin/api/api-keys/revoke', { method: 'POST', body: JSON.stringify({ name: keyName }) });
|
||||
},
|
||||
updateClientTtl: (clientId: string, tokenTtl: number | null) => apiFetch('/admin/api/update-client-ttl', { method: 'POST', body: JSON.stringify({ clientId, tokenTtl }) }),
|
||||
rescopeClient: (clientId: string, sourceId: string, federatedRead: string[]) =>
|
||||
apiFetch('/admin/api/rescope-client', {
|
||||
method: 'POST',
|
||||
body: JSON.stringify({ clientId, sourceId, federatedRead }),
|
||||
}),
|
||||
revokeClient: (clientId: string) => apiFetch('/admin/api/revoke-client', { method: 'POST', body: JSON.stringify({ clientId }) }),
|
||||
// v0.36.1.0 (T15 / E6) — calibration endpoints.
|
||||
calibrationProfile: (holder?: string) =>
|
||||
|
||||
+169
-4
@@ -18,6 +18,8 @@ interface Agent {
|
||||
client_name?: string; // compat
|
||||
grant_types: string[];
|
||||
scope: string;
|
||||
source_id: string | null;
|
||||
federated_read: string[];
|
||||
created_at: string;
|
||||
last_used_at: string | null;
|
||||
total_requests: number;
|
||||
@@ -26,6 +28,12 @@ interface Agent {
|
||||
status: 'active' | 'revoked';
|
||||
}
|
||||
|
||||
interface Source {
|
||||
id: string;
|
||||
name: string;
|
||||
federated: boolean;
|
||||
}
|
||||
|
||||
interface ApiKey {
|
||||
id: string;
|
||||
name: string;
|
||||
@@ -36,6 +44,7 @@ interface ApiKey {
|
||||
|
||||
export function AgentsPage() {
|
||||
const [agents, setAgents] = useState<Agent[]>([]);
|
||||
const [sources, setSources] = useState<Source[]>([]);
|
||||
const [hideRevoked, setHideRevoked] = useState(true);
|
||||
const [showRegister, setShowRegister] = useState(false);
|
||||
const [showCredentials, setShowCredentials] = useState<{ clientId: string; clientSecret: string; name: string } | null>(null);
|
||||
@@ -43,7 +52,10 @@ export function AgentsPage() {
|
||||
const [showApiKeyToken, setShowApiKeyToken] = useState<{ name: string; token: string } | null>(null);
|
||||
const [selectedAgent, setSelectedAgent] = useState<Agent | null>(null);
|
||||
|
||||
useEffect(() => { loadAgents(); }, []);
|
||||
useEffect(() => {
|
||||
loadAgents();
|
||||
api.sources().then(setSources).catch(() => {});
|
||||
}, []);
|
||||
|
||||
const loadAgents = () => { api.agents().then(setAgents).catch(() => {}); };
|
||||
|
||||
@@ -88,6 +100,7 @@ export function AgentsPage() {
|
||||
<th>Name</th>
|
||||
<th>Type</th>
|
||||
<th>Scopes</th>
|
||||
<th>Sources</th>
|
||||
<th>Status</th>
|
||||
<th>Requests</th>
|
||||
<th>Last Used</th>
|
||||
@@ -108,6 +121,11 @@ export function AgentsPage() {
|
||||
<span key={s} className={`badge badge-${s}`} style={{ marginRight: 4 }}>{s}</span>
|
||||
))}
|
||||
</td>
|
||||
<td style={{ color: 'var(--text-secondary)', fontSize: 12 }}>
|
||||
{a.auth_type === 'oauth'
|
||||
? `${a.source_id || 'none'} · ${(a.federated_read || []).length} readable`
|
||||
: 'Unscoped'}
|
||||
</td>
|
||||
<td>
|
||||
<span className={`badge ${a.status === 'active' ? 'badge-success' : 'badge-danger'}`}>{a.status}</span>
|
||||
</td>
|
||||
@@ -144,7 +162,21 @@ export function AgentsPage() {
|
||||
)}
|
||||
|
||||
{selectedAgent && (
|
||||
<AgentDrawer agent={selectedAgent} onClose={() => setSelectedAgent(null)} onRevoked={loadAgents} />
|
||||
<AgentDrawer
|
||||
key={selectedAgent.id}
|
||||
agent={selectedAgent}
|
||||
sources={sources}
|
||||
onClose={() => setSelectedAgent(null)}
|
||||
onRevoked={loadAgents}
|
||||
onRescoped={({ sourceId, federatedRead }) => {
|
||||
setSelectedAgent(current => current ? {
|
||||
...current,
|
||||
source_id: sourceId,
|
||||
federated_read: federatedRead,
|
||||
} : current);
|
||||
loadAgents();
|
||||
}}
|
||||
/>
|
||||
)}
|
||||
|
||||
{showApiKeyCreate && (
|
||||
@@ -381,7 +413,127 @@ function CredentialsModal({ credentials, onClose }: {
|
||||
);
|
||||
}
|
||||
|
||||
function AgentDrawer({ agent, onClose, onRevoked }: { agent: Agent; onClose: () => void; onRevoked: () => void }) {
|
||||
function SourceAccessEditor({ clientId, agent, sources, onRescoped }: {
|
||||
clientId: string;
|
||||
agent: Agent;
|
||||
sources: Source[];
|
||||
onRescoped: (scope: { sourceId: string; federatedRead: string[] }) => void;
|
||||
}) {
|
||||
const [writeSource, setWriteSource] = useState(agent.source_id || 'default');
|
||||
const [readSources, setReadSources] = useState<string[]>(agent.federated_read || []);
|
||||
const [saving, setSaving] = useState(false);
|
||||
const [error, setError] = useState('');
|
||||
const [saved, setSaved] = useState(false);
|
||||
const readableSet = new Set(readSources);
|
||||
const activeSourceIds = new Set(sources.map(source => source.id));
|
||||
const unavailableReadSources = readSources.filter(sourceId => !activeSourceIds.has(sourceId));
|
||||
const primaryUnavailable = !activeSourceIds.has(writeSource);
|
||||
|
||||
const save = async () => {
|
||||
if (readSources.length === 0) {
|
||||
setError('Select at least one readable source.');
|
||||
return;
|
||||
}
|
||||
setSaving(true);
|
||||
setError('');
|
||||
setSaved(false);
|
||||
try {
|
||||
const result = await api.rescopeClient(clientId, writeSource, readSources) as {
|
||||
sourceId: string;
|
||||
federatedRead: string[];
|
||||
};
|
||||
setWriteSource(result.sourceId);
|
||||
setReadSources(result.federatedRead);
|
||||
setSaved(true);
|
||||
onRescoped(result);
|
||||
} catch (e) {
|
||||
setError(e instanceof Error ? e.message : 'Failed to save source access');
|
||||
} finally {
|
||||
setSaving(false);
|
||||
}
|
||||
};
|
||||
|
||||
return (
|
||||
<>
|
||||
<div className="section-title">Source Access</div>
|
||||
<div style={{ color: 'var(--text-secondary)', fontSize: 12, lineHeight: 1.5, marginBottom: 12 }}>
|
||||
The primary source is the write destination. Read access is an explicit allowlist and does not widen automatically.
|
||||
</div>
|
||||
<div style={{ marginBottom: 14 }}>
|
||||
<label htmlFor="agent-write-source">Primary / write source</label>
|
||||
<select
|
||||
id="agent-write-source"
|
||||
value={writeSource}
|
||||
onChange={e => { setWriteSource(e.target.value); setSaved(false); }}
|
||||
style={{ width: '100%', background: 'var(--bg-secondary)', color: 'var(--text-primary)', border: '1px solid var(--border)', borderRadius: 6, padding: '6px 10px', fontSize: 14 }}
|
||||
>
|
||||
{primaryUnavailable && (
|
||||
<option value={writeSource} disabled>{writeSource} · unavailable</option>
|
||||
)}
|
||||
{sources.map(source => (
|
||||
<option key={source.id} value={source.id}>{source.name} ({source.id})</option>
|
||||
))}
|
||||
</select>
|
||||
</div>
|
||||
<fieldset style={{ border: 0, padding: 0, margin: '0 0 14px' }}>
|
||||
<legend>Readable sources</legend>
|
||||
<div className="checkbox-group" style={{ marginTop: 6 }}>
|
||||
{sources.map(source => (
|
||||
<label key={source.id} className="checkbox-label">
|
||||
<input
|
||||
type="checkbox"
|
||||
checked={readableSet.has(source.id)}
|
||||
onChange={e => {
|
||||
setSaved(false);
|
||||
setReadSources(current => e.target.checked
|
||||
? [...current, source.id]
|
||||
: current.filter(id => id !== source.id));
|
||||
}}
|
||||
/>
|
||||
{source.name} ({source.id}){source.federated ? ' · federated' : ' · private'}
|
||||
</label>
|
||||
))}
|
||||
{unavailableReadSources.map(sourceId => (
|
||||
<label key={sourceId} className="checkbox-label" style={{ color: 'var(--warning)' }}>
|
||||
<input
|
||||
type="checkbox"
|
||||
checked
|
||||
onChange={() => {
|
||||
setSaved(false);
|
||||
setReadSources(current => current.filter(id => id !== sourceId));
|
||||
}}
|
||||
/>
|
||||
{sourceId} · unavailable (clear to remove grant)
|
||||
</label>
|
||||
))}
|
||||
</div>
|
||||
</fieldset>
|
||||
{(primaryUnavailable || unavailableReadSources.length > 0) && (
|
||||
<div style={{ color: 'var(--warning)', fontSize: 13, marginBottom: 10 }}>
|
||||
This client references unavailable or archived sources. Choose an active primary source and clear unavailable read grants before saving.
|
||||
</div>
|
||||
)}
|
||||
{error && <div style={{ color: 'var(--error)', fontSize: 13, marginBottom: 10 }}>{error}</div>}
|
||||
{saved && <div style={{ color: 'var(--success)', fontSize: 13, marginBottom: 10 }}>Source access saved.</div>}
|
||||
<button
|
||||
type="button"
|
||||
className="btn btn-primary"
|
||||
disabled={saving || readSources.length === 0 || sources.length === 0 || primaryUnavailable || unavailableReadSources.length > 0}
|
||||
onClick={save}
|
||||
>
|
||||
{saving ? 'Saving...' : 'Save Source Access'}
|
||||
</button>
|
||||
</>
|
||||
);
|
||||
}
|
||||
|
||||
function AgentDrawer({ agent, sources, onClose, onRevoked, onRescoped }: {
|
||||
agent: Agent;
|
||||
sources: Source[];
|
||||
onClose: () => void;
|
||||
onRevoked: () => void;
|
||||
onRescoped: (scope: { sourceId: string; federatedRead: string[] }) => void;
|
||||
}) {
|
||||
const [tab, setTab] = useState<'claude-code' | 'chatgpt' | 'claude-cowork' | 'perplexity' | 'cursor' | 'json'>('claude-code');
|
||||
const copy = (text: string) => navigator.clipboard.writeText(text);
|
||||
const serverUrl = window.location.origin;
|
||||
@@ -553,6 +705,15 @@ function AgentDrawer({ agent, onClose, onRevoked }: { agent: Agent; onClose: ()
|
||||
<span>{agent.token_ttl ? (agent.token_ttl >= 31536000 ? 'No expiry' : agent.token_ttl >= 86400 ? `${Math.floor(agent.token_ttl / 86400)}d` : agent.token_ttl >= 3600 ? `${Math.floor(agent.token_ttl / 3600)}h` : `${agent.token_ttl}s`) : '1h (default)'}</span>
|
||||
</div>
|
||||
|
||||
{isOAuth && (
|
||||
<SourceAccessEditor
|
||||
clientId={cid}
|
||||
agent={agent}
|
||||
sources={sources}
|
||||
onRescoped={onRescoped}
|
||||
/>
|
||||
)}
|
||||
|
||||
{/*
|
||||
Config Export visible for both auth_type=oauth AND auth_type=api_key.
|
||||
Claude Code + Cursor + JSON tabs render real snippets regardless
|
||||
@@ -579,7 +740,11 @@ function AgentDrawer({ agent, onClose, onRevoked }: { agent: Agent; onClose: ()
|
||||
{(() => {
|
||||
const oauthOnlyTabs = new Set(['chatgpt', 'claude-cowork', 'perplexity']);
|
||||
if (!isOAuth && oauthOnlyTabs.has(tab)) {
|
||||
const clientName = { chatgpt: 'ChatGPT', 'claude-cowork': 'Claude.ai', perplexity: 'Perplexity' }[tab] || tab;
|
||||
const clientName = tab === 'chatgpt'
|
||||
? 'ChatGPT'
|
||||
: tab === 'claude-cowork'
|
||||
? 'Claude.ai'
|
||||
: 'Perplexity';
|
||||
return (
|
||||
<div style={{
|
||||
background: 'rgba(255, 200, 100, 0.08)',
|
||||
|
||||
+2
-1
@@ -39,10 +39,11 @@ gbrain migrate --to pglite # Postgres → PGLite (rare)
|
||||
|
||||
For shared / large / multi-machine deployments (a team or company brain with multiple users hitting one server over HTTP MCP with OAuth scoping per user), follow the dedicated walkthrough: **[Tutorial: set up GBrain as your company brain](tutorials/company-brain.md)**.
|
||||
|
||||
API keys live in `~/.gbrain/config.json` (file plane) or env vars (`OPENAI_API_KEY`, `ZEROENTROPY_API_KEY`, `VOYAGE_API_KEY`, `ANTHROPIC_API_KEY`). Set via CLI:
|
||||
API keys live in `~/.gbrain/config.json` (file plane) or env vars (`OPENAI_API_KEY`, `OPENROUTER_API_KEY`, `ZEROENTROPY_API_KEY`, `VOYAGE_API_KEY`, `ANTHROPIC_API_KEY`). Set via CLI:
|
||||
|
||||
```bash
|
||||
gbrain config set zeroentropy_api_key sk-...
|
||||
gbrain config set openrouter_api_key sk-or-...
|
||||
gbrain config set anthropic_api_key sk-ant-...
|
||||
```
|
||||
|
||||
|
||||
@@ -19,6 +19,29 @@ Seven test command tiers, each with a clear scope:
|
||||
| `bun run test:e2e` | Real Postgres E2E. Requires Docker + `DATABASE_URL`. Sequential. | ~5-10min | Pre-ship; nightly. |
|
||||
| `bun run check:all` | The historical pre-check scripts (22, chained sequentially in package.json). Overlaps `verify` heavily but is NOT a superset — `verify`'s `CHECKS` array in `scripts/run-verify-parallel.sh` (~30 entries incl. typecheck) is the authoritative gate; `check:all` keeps a few local-only extras (trailing-newline, exports-count, no-legacy-getconnection). | ~10s | Local-only sweep for the extras. |
|
||||
|
||||
### Shell dispatch and Windows
|
||||
|
||||
All four of `test`, `verify`, `ci:local` and `test:e2e` hand off to shell scripts
|
||||
under `scripts/`, so every `check:*` entry in `package.json` invokes its script as
|
||||
`bash scripts/<name>.sh` instead of relying on the shebang — bun on Windows cannot
|
||||
exec a `.sh` directly. Add a new shell-script check with that same prefix. The
|
||||
`scripts/*.ts` entries run under bun and take no prefix.
|
||||
|
||||
The scripts must also be on disk with Unix line endings. A strict bash (WSL, Linux
|
||||
CI, macOS) rejects CRLF and dies on the script's first meaningful line; the Cygwin
|
||||
bash that ships with Git for Windows tolerates it, so a green local run is not by
|
||||
itself evidence that a script is CRLF-clean.
|
||||
The root `.gitattributes` pins `*.sh text eol=lf`, which overrides the
|
||||
`core.autocrlf=true` default that Git for Windows installs. Working copies cloned
|
||||
before that pin need a one-time `git rm --cached -r . -q && git reset --hard` to
|
||||
pick it up; see the Windows section of `CONTRIBUTING.md`.
|
||||
|
||||
Wallclock figures in the table above are from a Mac dev box. Windows is
|
||||
substantially slower because each check pays full process-creation cost, and three
|
||||
tree-walking checks (`check:privacy`, `check:test-names`, `check:test-isolation`)
|
||||
plus `typecheck` can exceed the 120s per-check cap in `run-verify-parallel.sh`
|
||||
there even though they pass on Linux and macOS.
|
||||
|
||||
### CI vs local: intentionally divergent file sets
|
||||
|
||||
- **CI matrix** (`.github/workflows/test.yml`) runs `scripts/test-shard.sh` across 10 matrix shards partitioned by weight-aware LPT bin-packing (`scripts/sharding.ts`) and INCLUDES `*.slow.test.ts` (the two outlier slow files run as dedicated jobs alongside the matrix). CI EXCLUDES `*.serial.test.ts` from the shards and runs them in a dedicated job via `bun run test:serial`, one bun process per file — keeping serial files out of the shard processes is what preserves the `mock.module` quarantine (a top-level mock in one file leaks into every other file sharing its process). `bun run verify` gets its own job too. CI is the ground truth for "did everything pass."
|
||||
@@ -192,6 +215,7 @@ Unit tests and what they cover:
|
||||
- `test/sync-pull-failed-anchor.serial.test.ts` — #3068 regression: a failed internal `git pull` (local-path origin vs `protocol.file.allow=never`) with zero imports returns `partial`/`pull_failed` (not `up_to_date`), freezes `last_commit` + `last_sync_at`, recovers after a manual pull; fall-through import of local commits preserved. Serial: pins `GBRAIN_HOME` to a temp dir for the whole file.
|
||||
- `test/sync-concurrency.test.ts` — `autoConcurrency()` thresholds + PGLite-forces-serial + explicit-override clamping; `shouldRunParallel()` explicit-bypasses-floor contract; `parseWorkers()` validation rejecting `'0'`/`'-3'`/`'foo'`/`'1.5'`/trailing chars.
|
||||
- `test/sync-parallel.test.ts` — PGLite-routed coverage of the bookmark gate under concurrency, head-drift gate, vanished-file failure capture, PGLite-stays-serial, and the `gbrain-sync` writer-lock contract.
|
||||
- `test/sync-all-missing-path.test.ts` — `sync --all --missing-path <fail|skip>` pure helpers: `parseMissingPathMode` (default fail, explicit values, loud rejection of bad/dangling values, never swallows a following flag) and `partitionMissingPathSources` (classification driven only by the injected pathExists predicate — no fs; null `local_path` passes through runnable; order preserved).
|
||||
- `test/sync-failures.test.ts` — `classifyErrorCode` regex coverage for all 12 codes against literal production message strings from `markdown.ts` and `import-file.ts`; `summarizeFailuresByCode` sort + pre-classified-honor; `recordSyncFailures` code-field persistence; `acknowledgeSyncFailures` `AcknowledgeResult` shape + backfill on legacy entries.
|
||||
- `test/doctor.test.ts` — doctor command; assertions that `jsonb_integrity` scans the four JSONB write sites and `markdown_body_completeness` is present.
|
||||
- `test/utils.test.ts` — shared SQL utilities + `tryParseEmbedding` null-return and single-warn semantics.
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -87,6 +87,15 @@ embedding proximity. Four layers, added after the incident in
|
||||
deciding "is this page already here, safe to NOT write a duplicate?" keys off
|
||||
`create_safety`, not a raw blended score.
|
||||
|
||||
**Extraction quarantine lane (issue #160):** pages carrying the unverified
|
||||
auto-extracted markers (frontmatter `provenance: auto-extracted` +
|
||||
`status: unverified`, see `src/core/extraction-review.ts`) rank as ordinary
|
||||
content — they are skipped by the compiled-truth fusion boost and by the
|
||||
`people/`/`companies/` namespace source-boost, and every search result from
|
||||
such a page carries `unverified: true` so agents can label the provenance.
|
||||
Promote or reject them via `gbrain extraction-pending` / `gbrain
|
||||
extraction-review`.
|
||||
|
||||
The `search` MCP/CLI op is **cheap-hybrid** (vector + keyword + RRF + pool +
|
||||
title + alias, expansion off); `query` is the full-control variant. NamedThingBench
|
||||
(`gbrain eval retrieval-quality`) gates these families on every PR. Diagnose a
|
||||
|
||||
@@ -229,7 +229,7 @@ add `GBRAIN_AUDIT_FULL=1` (v0.43+ TODO; not yet wired).
|
||||
- Per-source pack-upgrade (the handler accepts `sourceId` but
|
||||
`findPackSuccessors` doesn't yet pass it through)
|
||||
- Cross-brain federated mounts that disagree on canonical packs
|
||||
- Automatic rollback (today: manual SQL or `gbrain pages restore`)
|
||||
- Automatic rollback (today: manual SQL or `gbrain restore`)
|
||||
- LLM-assisted mapping_rules codegen from production data (`gbrain
|
||||
schema detect-mappings`; deferred to v0.43+)
|
||||
|
||||
|
||||
@@ -214,7 +214,7 @@ gbrain schema downgrade
|
||||
|
||||
1. `git revert <merge-commit>` — restores the code.
|
||||
2. `gbrain schema downgrade --to gbrain-base` — restores config.
|
||||
3. (Optional) `gbrain pages purge-deleted --older-than 0h` — drops
|
||||
3. (Optional) `gbrain purge-deleted --older-than 0h` — drops
|
||||
v0.39-typed pages that no longer have a matching type in the active
|
||||
pack.
|
||||
|
||||
|
||||
@@ -19,11 +19,13 @@ entire DB from scratch.
|
||||
|
||||
This means:
|
||||
|
||||
- **Disaster recovery is one command.** If your DB volume corrupts, if
|
||||
Postgres eats itself, if PGLite's WASM lock wedges — you don't need
|
||||
a backup. You wipe the DB, re-import from your brain repo, and the
|
||||
derived state regenerates. v0.32.3 ships `gbrain rebuild
|
||||
--confirm-destructive` as the documented one-liner.
|
||||
- **Disaster recovery is a short, boring sequence.** If your DB volume
|
||||
corrupts, if Postgres eats itself, if PGLite's WASM lock wedges — you
|
||||
don't need a backup. You wipe the derived tables (on PGLite,
|
||||
`gbrain reinit-pglite` wipes the whole embedded DB), re-import from
|
||||
your brain repo with `gbrain sync`, and `gbrain extract all`
|
||||
regenerates the derived state. See "Disaster recovery" below for the
|
||||
exact commands.
|
||||
- **Multi-machine sync is git.** Your brain is a repo. Push from one
|
||||
machine, pull from another, and the second machine's DB rebuilds on
|
||||
its next sync. No "back up the database" step.
|
||||
@@ -146,11 +148,9 @@ The promise the rule makes:
|
||||
# Snapshot what's there
|
||||
gbrain stats > /tmp/before.txt
|
||||
|
||||
# Wipe and rebuild
|
||||
gbrain rebuild --confirm-destructive # v0.32.3 — deletes derived tables
|
||||
# (pages + content_chunks survive
|
||||
# the CASCADE-safe design)
|
||||
# OR manually for v0.32.2:
|
||||
# Wipe and rebuild — delete the derived tables (pages + content_chunks
|
||||
# survive the CASCADE-safe design), then re-derive from the repo.
|
||||
# On PGLite, `gbrain reinit-pglite` wipes the whole embedded DB instead.
|
||||
psql -c 'DELETE FROM facts; DELETE FROM takes; DELETE FROM links; DELETE FROM timeline_entries;'
|
||||
gbrain sync
|
||||
gbrain extract all
|
||||
|
||||
@@ -108,8 +108,8 @@ Every primitive ships with a documented rollback:
|
||||
| Operation | Rollback |
|
||||
|-----------|----------|
|
||||
| Retype | `frontmatter.legacy_type = <original>` preserved on every page (D8). One SQL UPDATE restores types: `UPDATE pages SET type = frontmatter->>'legacy_type' WHERE frontmatter ? 'legacy_type'`. |
|
||||
| Page-to-link | Source page soft-deleted with 72h TTL. `gbrain pages restore <slug>` within 72h. Link row stays harmless if source restored. |
|
||||
| Page-to-alias | Source page soft-deleted with 72h TTL. `gbrain pages restore <slug>` within 72h. Alias row stays harmless (or `DELETE FROM slug_aliases WHERE alias_slug = <slug>` to clean up). |
|
||||
| Page-to-link | Source page soft-deleted with 72h TTL. `gbrain restore <slug>` within 72h. Link row stays harmless if source restored. |
|
||||
| Page-to-alias | Source page soft-deleted with 72h TTL. `gbrain restore <slug>` within 72h. Alias row stays harmless (or `DELETE FROM slug_aliases WHERE alias_slug = <slug>` to clean up). |
|
||||
| Active-pack flip | `gbrain schema use gbrain-base` reverses the flip. |
|
||||
|
||||
## What if my brain doesn't fit?
|
||||
|
||||
@@ -183,6 +183,6 @@ This also means the best AI agent setups will be open source by default. Closed,
|
||||
|
||||
Software distribution reimagined: the package is a markdown file, the runtime is a sufficiently smart model, the package manager is your AI agent, and the app store is a git repo.
|
||||
|
||||
`gbrain install voice-agent`
|
||||
`gbrain skillpack scaffold voice-agent`
|
||||
|
||||
That's it.
|
||||
|
||||
@@ -159,7 +159,8 @@ proxy for worker env.
|
||||
If a brain DB ever traverses a trust boundary, secrets stay out.
|
||||
- **Free-form names.** `inherit:` accepts any snake_case config-key on your
|
||||
worker — `database_url`, `anthropic_api_key`, `openai_api_key`,
|
||||
`voyage_api_key`, `groq_api_key`, `zeroentropy_api_key`, or any custom
|
||||
`openrouter_api_key`, `voyage_api_key`, `groq_api_key`,
|
||||
`zeroentropy_api_key`, or any custom
|
||||
field you stuff into `~/.gbrain/config.json`. The agent picks what it
|
||||
needs.
|
||||
- **`env:` still works** for non-secret values, or for cases where you
|
||||
|
||||
@@ -69,7 +69,7 @@ update_brain_page(slug, new_info, source):
|
||||
page = gbrain get {slug}
|
||||
|
||||
// TIMELINE: always APPEND (never edit existing entries)
|
||||
gbrain add_timeline_entry {slug} {
|
||||
gbrain timeline-add {slug} {
|
||||
date: today,
|
||||
summary: new_info.summary,
|
||||
detail: new_info.detail,
|
||||
|
||||
@@ -46,10 +46,10 @@ on user_shares_media(url_or_file):
|
||||
|
||||
# Step 4: Extract and cross-reference entities
|
||||
for person in transcript.mentioned_people:
|
||||
gbrain add_link <slug> <person_slug>
|
||||
gbrain add_link <person_slug> <slug>
|
||||
gbrain add_timeline_entry <person_slug> \
|
||||
--entry "Discussed in {video_title}: {what_was_said}" \
|
||||
gbrain link <slug> <person_slug>
|
||||
gbrain link <person_slug> <slug>
|
||||
gbrain timeline-add <person_slug> {date} \
|
||||
"Discussed in {video_title}: {what_was_said}" \
|
||||
--source "YouTube: {url}"
|
||||
|
||||
# PATTERN 2: Social Media Bundles
|
||||
@@ -80,8 +80,8 @@ on user_shares_media(url_or_file):
|
||||
|
||||
# Extract entities and cross-reference
|
||||
for entity in bundle.mentioned_entities:
|
||||
gbrain add_link <slug> <entity_slug>
|
||||
gbrain add_link <entity_slug> <slug>
|
||||
gbrain link <slug> <entity_slug>
|
||||
gbrain link <entity_slug> <slug>
|
||||
|
||||
# PATTERN 3: PDFs and Documents
|
||||
elif media.type == "pdf" or media.type == "document":
|
||||
@@ -109,8 +109,8 @@ on user_shares_media(url_or_file):
|
||||
"""
|
||||
|
||||
for entity in document.mentioned_entities:
|
||||
gbrain add_link <slug> <entity_slug>
|
||||
gbrain add_link <entity_slug> <slug>
|
||||
gbrain link <slug> <entity_slug>
|
||||
gbrain link <entity_slug> <slug>
|
||||
|
||||
# Always sync after ingestion
|
||||
gbrain sync
|
||||
@@ -127,7 +127,7 @@ on user_shares_media(url_or_file):
|
||||
## How to Verify
|
||||
|
||||
1. Ingest a YouTube video. Run `gbrain get media/youtube/{slug}`. Confirm the page has: the agent's analysis (not just a summary), key quotes with speaker attribution, and the full diarized transcript.
|
||||
2. Run `gbrain get_links media/youtube/{slug}`. Confirm back-links exist to brain pages for every person and company mentioned in the video.
|
||||
2. Run `gbrain call get_links '{"slug": "media/youtube/{slug}"}'`. Confirm back-links exist to brain pages for every person and company mentioned in the video.
|
||||
3. Pick a person mentioned in the video. Run `gbrain get <person_slug>`. Confirm their timeline has a new entry referencing the video with specific context.
|
||||
4. Ingest a tweet. Confirm the brain page includes the thread context, linked article summaries, and entity cross-references -- not just the tweet text.
|
||||
5. Run `gbrain search "{topic_from_video}"`. Confirm the media page appears in search results (verifies the content is indexed and searchable).
|
||||
|
||||
@@ -0,0 +1,138 @@
|
||||
# Embedding migration — moving a brain to another embedding provider
|
||||
|
||||
`gbrain migrate embeddings` re-embeds an entire brain onto a different
|
||||
embedding provider/model, safely and resumably. It is the forward path off a
|
||||
sunsetting provider (for example ZeroEntropy's hosted API, which shuts down
|
||||
2026-09-04 and is the shipped default for brains that never picked a model) —
|
||||
but it is provider-agnostic: any configured `provider:model` works as a
|
||||
target.
|
||||
|
||||
Also reachable as `gbrain retrieval-upgrade` (the name `doctor` and the
|
||||
README reference).
|
||||
|
||||
## Quick start
|
||||
|
||||
```bash
|
||||
# Preview the work + cost. Changes nothing.
|
||||
gbrain migrate embeddings --to openai:text-embedding-3-small --dry-run
|
||||
|
||||
# Run it (interactive confirm shows chunk count + $ estimate first).
|
||||
gbrain migrate embeddings --to openai:text-embedding-3-small
|
||||
|
||||
# Non-interactive (cron / scripts): --yes is required, else exit 2.
|
||||
gbrain migrate embeddings --to voyage:voyage-3-large --yes
|
||||
```
|
||||
|
||||
`--dim <N>` overrides the target width; it defaults to the provider recipe's
|
||||
declared width and is required for recipes that don't declare one (litellm,
|
||||
llama-server, and other bring-your-own-model providers).
|
||||
|
||||
## What it does, in order
|
||||
|
||||
1. **Plan.** Counts every chunk not already in the target embedding space —
|
||||
including chunks on pages with **no recorded embedding signature**
|
||||
(pages embedded before the v108 provenance stamp). Prices the re-embed
|
||||
from the pricing table; unknown providers print "estimate unavailable"
|
||||
instead of a fabricated number.
|
||||
2. **Consent gate.** Prints the plan; requires an interactive `y` or `--yes`.
|
||||
Non-TTY without `--yes` refuses with exit 2 (mirrors the `reindex-code`
|
||||
gate in [spend-controls](../operations/spend-controls.md)). Unlike the pure
|
||||
cost gates there, `spend.posture=tokenmax` does **not** bypass this one:
|
||||
posture waives the spend *ceiling*, and this gate also guards a
|
||||
destructive schema rebuild. Under `tokenmax` the dollar figure is marked
|
||||
informational and the confirmation is still asked. `--yes` is the single
|
||||
scripted bypass.
|
||||
3. **Live probe.** One tiny embed against the TARGET provider before any
|
||||
mutation — validates the API key, model id, and dimension support in a
|
||||
single call. A bad key fails here, with nothing changed.
|
||||
4. **Env-override gate.** Refuses when `GBRAIN_EMBEDDING_MODEL` /
|
||||
`GBRAIN_EMBEDDING_DIMENSIONS` would silently defeat the switch at
|
||||
runtime (the same guard `ze-switch` uses). `--ignore-env-override` for
|
||||
people running deliberate experiments.
|
||||
5. **Apply.** When the target width differs from the actual column width,
|
||||
runs the same atomic schema transition `ze-switch` uses, in one
|
||||
transaction. It rebuilds **all three dim-pinned text-embedding-space
|
||||
columns** — `content_chunks.embedding`, `query_cache.embedding`, and
|
||||
`facts.embedding` — at the new width, preserving each column's type
|
||||
(`vector` vs `halfvec`) and recreating its HNSW index. Missing any of the
|
||||
three leaves it silently broken: a narrow `query_cache.embedding` makes
|
||||
every cache write and read fail *by design* (the cache swallows errors so
|
||||
it can never break search) for a permanent 0% hit rate, and a narrow
|
||||
`facts.embedding` fails every per-fact embed write. The image/multimodal
|
||||
columns ARE deliberately untouched — they use separate models whose
|
||||
dimensions are independent of the text embedding model.
|
||||
Writes `embedding_model` + `embedding_dimensions` to BOTH config planes
|
||||
(file plane for the runtime gateway, DB plane for doctor), invalidates
|
||||
every chunk still in the old space — **including NULL-signature pages** —
|
||||
and purges the semantic query cache so stale cached results can't be
|
||||
served across the swap.
|
||||
6. **Re-embed.** The standard embed pipeline (`embed --stale --catch-up`)
|
||||
with per-source single-flight locks, rate-limit backoff, stderr progress,
|
||||
and optional DB-contention pacing (`--pace[=mode]`).
|
||||
|
||||
## What the rebuild deletes
|
||||
|
||||
The dimension change **deletes every stored embedding vector** in the brain —
|
||||
they are in the old model's space and unusable. They are not recoverable:
|
||||
going back to the previous provider means paying for a second full re-embed.
|
||||
`content_chunks` vectors are rebuilt by the re-embed pass, the query cache
|
||||
refills on the next query, and fact embeddings are rewritten on their next
|
||||
write (or a `gbrain extract` pass).
|
||||
|
||||
## Resume after a kill
|
||||
|
||||
The NULL-embedding column is the checkpoint. If the run is killed (or some
|
||||
pages fail to embed), re-run the **same command**: chunks already embedded on
|
||||
the target are never re-embedded, the schema/config steps no-op, and the run
|
||||
continues where it stopped. An in-flight marker (`embedding_migration.state`
|
||||
in DB config) records the target; it is cleared only when the backlog drains
|
||||
to zero.
|
||||
|
||||
A page whose chunks straddle two stale batches is embedded correctly but not
|
||||
stamped by the embed loop (which only stamps all-or-nothing per batch), so the
|
||||
migration runs one reconcile pass after the drain that stamps every
|
||||
fully-embedded page. Without it a large brain would report "incomplete" and the
|
||||
re-run would pay again for those pages. `--batch-size N` tunes the batch
|
||||
(default 2000).
|
||||
|
||||
`--no-embed` applies schema + config + invalidation and stops, so you can run
|
||||
the (potentially long) re-embed later or in the background:
|
||||
|
||||
```bash
|
||||
gbrain migrate embeddings --to openai:text-embedding-3-small --yes --no-embed
|
||||
gbrain embed --stale --catch-up --include-null-signature --background
|
||||
```
|
||||
|
||||
## During the migration
|
||||
|
||||
While the re-embed runs, semantic search returns degraded (lexical-arm-only)
|
||||
results for not-yet-re-embedded content. Pick a quiet window for large
|
||||
brains, or use `--pace` to keep the DB responsive.
|
||||
|
||||
## Pages without an embedding signature (#3391)
|
||||
|
||||
Pages embedded before provenance stamping have `embedding_signature IS NULL`
|
||||
and are grandfathered by the routine stale sweep (so an upgrade never
|
||||
surprise-re-embeds a whole corpus). After a provider swap that grandfather
|
||||
clause would silently leave those pages in the OLD embedding space — mixed
|
||||
vector spaces in one index, degrading retrieval with nothing in the logs.
|
||||
|
||||
- `gbrain migrate embeddings` always includes them.
|
||||
- Plain `gbrain embed --stale` warns when a model swap leaves NULL-signature
|
||||
pages behind, and `gbrain embed --stale --include-null-signature` re-embeds
|
||||
them.
|
||||
|
||||
## Reranker
|
||||
|
||||
Migrating embeddings does not touch the reranker. If
|
||||
`search.reranker.model` points at the outgoing provider, the plan prints a
|
||||
warning; disable it (`gbrain config set search.reranker.enabled false`) or
|
||||
point it at another provider.
|
||||
|
||||
## Self-hosting instead of migrating
|
||||
|
||||
If the outgoing model's weights are available (zembed-1's are Apache-2.0),
|
||||
serving them locally via `llama-server` / `ollama` / a LiteLLM proxy
|
||||
preserves your existing vectors — no re-embed at all. Point
|
||||
`embedding_model` at the local recipe and keep the same dimensions. The
|
||||
migration command is for when you'd rather move to a hosted provider.
|
||||
@@ -49,23 +49,23 @@ on enrich(entity, trigger):
|
||||
data["contacts"] = google_contacts(entity.email) # Contact data
|
||||
|
||||
# Step 5: Store raw data (auditable, re-processable)
|
||||
gbrain put_raw_data <entity_slug> \
|
||||
--data '{"sources": {"crustdata": {"fetched_at": "...", "data": {...}}, ...}}'
|
||||
gbrain call put_raw_data \
|
||||
'{"slug": "<entity_slug>", "data": {"sources": {"crustdata": {"fetched_at": "...", "data": {...}}, ...}}}'
|
||||
# Overwrite on re-enrichment, don't append
|
||||
|
||||
# Step 6: Write to brain page
|
||||
if path == "CREATE":
|
||||
gbrain put <entity_slug> --content "<compiled_truth_from_all_sources>"
|
||||
gbrain add_timeline_entry <entity_slug> --entry "Page created via enrichment"
|
||||
gbrain timeline-add <entity_slug> {date} "Page created via enrichment"
|
||||
elif path == "UPDATE":
|
||||
# Append timeline, update compiled truth ONLY if materially new
|
||||
gbrain add_timeline_entry <entity_slug> --entry "Enriched: {new_signal}"
|
||||
gbrain timeline-add <entity_slug> {date} "Enriched: {new_signal}"
|
||||
# Flag contradictions -- don't silently resolve them
|
||||
|
||||
# Step 7: Cross-reference the graph
|
||||
gbrain add_link <person_slug> <company_slug> # person -> company
|
||||
gbrain add_link <company_slug> <person_slug> # company -> person
|
||||
gbrain add_link <person_slug> <deal_slug> # person -> deal
|
||||
gbrain link <person_slug> <company_slug> # person -> company
|
||||
gbrain link <company_slug> <person_slug> # company -> person
|
||||
gbrain link <person_slug> <deal_slug> # person -> deal
|
||||
# Every entity page links to every other entity page that references it
|
||||
|
||||
# People page sections (not a LinkedIn profile -- a living portrait):
|
||||
@@ -94,8 +94,8 @@ on enrich(entity, trigger):
|
||||
## How to Verify
|
||||
|
||||
1. Enrich a Tier 1 person. Run `gbrain get <slug>` and confirm the page has Executive Summary, State, What They Believe, Contact, and Timeline sections populated from multiple sources.
|
||||
2. Run `gbrain get_raw_data <slug>`. Confirm raw API responses are stored with `sources.{provider}.fetched_at` timestamps.
|
||||
3. Run `gbrain get_links <slug>`. Confirm cross-reference links exist to the person's company page, deal pages, and related entities.
|
||||
2. Run `gbrain call get_raw_data '{"slug": "<slug>"}'`. Confirm raw API responses are stored with `sources.{provider}.fetched_at` timestamps.
|
||||
3. Run `gbrain call get_links '{"slug": "<slug>"}'`. Confirm cross-reference links exist to the person's company page, deal pages, and related entities.
|
||||
4. Check a page that was enriched AND has a user-written Assessment. Confirm the Assessment section was preserved, not overwritten by API data.
|
||||
5. Try to re-enrich the same person. Confirm the system checks the `fetched_at` timestamp and skips if less than a week old.
|
||||
|
||||
|
||||
@@ -53,7 +53,7 @@ on upcoming_meeting(meeting):
|
||||
"last_interaction": page.timeline[0], # most recent
|
||||
"open_threads": page.open_threads,
|
||||
"relationship_temperature": page.relationship,
|
||||
"relevant_deals": gbrain get_links <attendee_slug>,
|
||||
"relevant_deals": gbrain call get_links '{"slug": "<attendee_slug>"}',
|
||||
}
|
||||
else:
|
||||
briefing[attendee] = "No brain page -- consider enriching"
|
||||
@@ -67,14 +67,14 @@ on inbox_cleared():
|
||||
for email in processed_emails:
|
||||
if email.contained_new_information:
|
||||
# Update the sender's brain page with new signal
|
||||
gbrain add_timeline_entry <sender_slug> \
|
||||
--entry "Email re: {subject}. Key info: {extracted_signal}" \
|
||||
gbrain timeline-add <sender_slug> {date} \
|
||||
"Email re: {subject}. Key info: {extracted_signal}" \
|
||||
--source "email from {sender} re {subject}, {date}"
|
||||
|
||||
# Update any mentioned entity pages too
|
||||
for entity in email.mentioned_entities:
|
||||
gbrain add_timeline_entry <entity_slug> \
|
||||
--entry "{what_was_said_about_them}" \
|
||||
gbrain timeline-add <entity_slug> {date} \
|
||||
"{what_was_said_about_them}" \
|
||||
--source "email from {sender}, {date}"
|
||||
|
||||
# WORKFLOW 4: Scheduling Nudges
|
||||
|
||||
@@ -32,15 +32,15 @@ on new_meeting_transcript(meeting):
|
||||
|
||||
# Step 3: Propagate to ALL entity pages (MANDATORY -- most agents skip this)
|
||||
for person in meeting.attendees + meeting.mentioned_people:
|
||||
gbrain add_timeline_entry <person_slug> \
|
||||
--entry "Met in '{meeting.title}' on {date}. Key points: ..." \
|
||||
gbrain timeline-add <person_slug> {date} \
|
||||
"Met in '{meeting.title}' on {date}. Key points: ..." \
|
||||
--source "Meeting notes '{meeting.title}', {date}"
|
||||
# Update their State section if new information surfaced
|
||||
# Update company pages for each person's company if relevant
|
||||
|
||||
for company in meeting.mentioned_companies:
|
||||
gbrain add_timeline_entry <company_slug> \
|
||||
--entry "Discussed in '{meeting.title}': {what_was_said}" \
|
||||
gbrain timeline-add <company_slug> {date} \
|
||||
"Discussed in '{meeting.title}': {what_was_said}" \
|
||||
--source "Meeting notes '{meeting.title}', {date}"
|
||||
|
||||
# Step 4: Extract action items
|
||||
@@ -49,8 +49,8 @@ on new_meeting_transcript(meeting):
|
||||
|
||||
# Step 5: Back-link everything (bidirectional graph)
|
||||
for entity in all_entities_mentioned:
|
||||
gbrain add_link <slug> <entity_slug> # meeting -> entity
|
||||
gbrain add_link <entity_slug> <slug> # entity -> meeting
|
||||
gbrain link <slug> <entity_slug> # meeting -> entity
|
||||
gbrain link <entity_slug> <slug> # entity -> meeting
|
||||
|
||||
# Step 6: Sync so new pages are immediately searchable
|
||||
gbrain sync
|
||||
@@ -73,7 +73,7 @@ on new_meeting_transcript(meeting):
|
||||
1. After ingesting a meeting, run `gbrain get meetings/{date}-{slug}`. Confirm the page has the agent's analysis above the bar and the full diarized transcript below it.
|
||||
2. For each attendee, run `gbrain get <attendee_slug>`. Check that their timeline has a new entry referencing the meeting with specific insights (not just "attended meeting").
|
||||
3. Pick a company mentioned in the meeting. Run `gbrain get <company_slug>`. Confirm a timeline entry exists referencing what was discussed about the company.
|
||||
4. Run `gbrain get_links meetings/{date}-{slug}`. Verify back-links exist to all attendee and entity pages.
|
||||
4. Run `gbrain call get_links '{"slug": "meetings/{date}-{slug}"}'`. Verify back-links exist to all attendee and entity pages.
|
||||
5. Run `gbrain search "{meeting_topic}"`. Confirm the meeting page appears in search results (verifies sync ran).
|
||||
|
||||
---
|
||||
|
||||
@@ -155,6 +155,7 @@ child-spawn time:
|
||||
- `inherit: ["database_url"]` → child env `GBRAIN_DATABASE_URL`
|
||||
- `inherit: ["anthropic_api_key"]` → child env `ANTHROPIC_API_KEY`
|
||||
- `inherit: ["openai_api_key"]` → child env `OPENAI_API_KEY`
|
||||
- `inherit: ["openrouter_api_key"]` → child env `OPENROUTER_API_KEY`
|
||||
- `inherit: ["voyage_api_key"]` → child env `VOYAGE_API_KEY`
|
||||
- `inherit: ["groq_api_key", "zeroentropy_api_key"]` → both injected
|
||||
- Or any arbitrary config-key your worker has (`my_custom_field` →
|
||||
|
||||
@@ -91,7 +91,7 @@ first):
|
||||
6. The seeded `default` source.
|
||||
|
||||
So inside `~/.gstack/plans/` on a brain that pinned `gstack` to
|
||||
`~/.gstack` via `.gbrain-source`, `gbrain put-page` implicitly writes to
|
||||
`~/.gstack` via `.gbrain-source`, `gbrain put` implicitly writes to
|
||||
the `gstack` source. Outside any registered directory with no env/dotfile
|
||||
set, it writes to the default.
|
||||
|
||||
@@ -188,10 +188,10 @@ citations keep working.
|
||||
|
||||
```bash
|
||||
# Pass --source explicitly
|
||||
gbrain put-page topics/ai ... --source wiki
|
||||
gbrain put topics/ai ... --source wiki
|
||||
|
||||
# Or rely on the dotfile / env / CWD match
|
||||
cd ~/.gstack && gbrain put-page plans/multi-repo ...
|
||||
cd ~/.gstack && gbrain put plans/multi-repo ...
|
||||
# → source auto-resolves to gstack
|
||||
```
|
||||
|
||||
|
||||
@@ -20,8 +20,8 @@ on every_inbound_message(message):
|
||||
for entity in entities:
|
||||
existing = gbrain search "{entity.name}"
|
||||
if existing:
|
||||
gbrain add_timeline_entry <entity_slug> \
|
||||
--entry "{what_was_said}" \
|
||||
gbrain timeline-add <entity_slug> {date} \
|
||||
"{what_was_said}" \
|
||||
--source "User, direct message, {timestamp}"
|
||||
# else: flag for enrichment if important enough
|
||||
|
||||
@@ -64,13 +64,13 @@ on nightly_schedule("02:00"):
|
||||
# The brain COMPOUNDS overnight.
|
||||
|
||||
# 5a: Entity sweep -- find unlinked mentions
|
||||
pages = gbrain list_pages
|
||||
pages = gbrain list
|
||||
for page in pages:
|
||||
mentions = extract_entity_mentions(page.content)
|
||||
existing_links = gbrain get_links <page.slug>
|
||||
existing_links = gbrain call get_links '{"slug": "<page.slug>"}'
|
||||
for mention in mentions:
|
||||
if mention not in existing_links:
|
||||
gbrain add_link <page.slug> <mention_slug> # fix broken graph
|
||||
gbrain link <page.slug> <mention_slug> # fix broken graph
|
||||
|
||||
# 5b: Citation audit -- find facts without sources
|
||||
for page in pages:
|
||||
@@ -80,7 +80,7 @@ on nightly_schedule("02:00"):
|
||||
|
||||
# 5c: Memory consolidation -- update compiled truth from timeline
|
||||
for page in stale_pages(older_than="7d"):
|
||||
timeline = gbrain get_timeline <page.slug>
|
||||
timeline = gbrain timeline <page.slug>
|
||||
if timeline.has_new_entries_since_last_consolidation:
|
||||
# Re-synthesize compiled truth from accumulated timeline
|
||||
updated_truth = consolidate(page.compiled_truth, timeline.new_entries)
|
||||
@@ -110,11 +110,11 @@ on nightly_schedule("02:00"):
|
||||
|
||||
## How to Verify
|
||||
|
||||
1. Send a message mentioning a person with a brain page. Confirm the agent detects the entity and adds a timeline entry to their page (`gbrain get_timeline <slug>`).
|
||||
1. Send a message mentioning a person with a brain page. Confirm the agent detects the entity and adds a timeline entry to their page (`gbrain timeline <slug>`).
|
||||
2. Ask the agent about someone in the brain. Confirm it runs `gbrain search` or `gbrain get` BEFORE reaching for external APIs (check the tool call order).
|
||||
3. Write a new page with `gbrain put`, then immediately run `gbrain search` for it. Confirm it appears in results (verifies sync ran).
|
||||
4. Run `gbrain doctor`. Confirm it returns a health report with database status, page count, and any flagged issues.
|
||||
5. After a dream cycle runs, check a page that had unlinked entity mentions. Confirm new links were added (`gbrain get_links <slug>`).
|
||||
5. After a dream cycle runs, check a page that had unlinked entity mentions. Confirm new links were added (`gbrain call get_links '{"slug": "<slug>"}'`).
|
||||
|
||||
---
|
||||
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*
|
||||
|
||||
@@ -47,8 +47,8 @@ on user_message(message):
|
||||
|
||||
# Step 3: Cross-link to everything that shaped the thinking
|
||||
for entity in idea.influences:
|
||||
gbrain add_link originals/{slug} <entity_slug>
|
||||
gbrain add_link <entity_slug> originals/{slug}
|
||||
gbrain link originals/{slug} <entity_slug>
|
||||
gbrain link <entity_slug> originals/{slug}
|
||||
|
||||
# Step 4: Sync
|
||||
gbrain sync
|
||||
@@ -79,7 +79,7 @@ on user_message(message):
|
||||
|
||||
1. Generate an original idea in conversation (e.g., "I call this the 'ambition debt' problem -- every year you delay going big, the compound interest works against you"). Confirm a new page appears at `brain/originals/ambition-debt` with `gbrain get originals/ambition-debt`.
|
||||
2. Check that the page uses the user's exact phrasing for the title and slug -- not a sanitized version.
|
||||
3. Run `gbrain get_links originals/ambition-debt`. Confirm cross-links exist to related people, meetings, or other originals.
|
||||
3. Run `gbrain call get_links '{"slug": "originals/ambition-debt"}'`. Confirm cross-links exist to related people, meetings, or other originals.
|
||||
4. Express a take on someone else's idea (e.g., "I think Thiel's contrarian question is wrong because..."). Confirm it goes to `originals/` (synthesis is original), not `concepts/`.
|
||||
5. Run `gbrain search "ambition debt"`. Confirm the originals page appears in search results and is discoverable.
|
||||
|
||||
|
||||
@@ -87,7 +87,7 @@ expect it.
|
||||
| `version` | string | yes | Your plugin's semver. Informational. |
|
||||
| `plugin_version` | string | yes | Contract lock. Must equal `"gbrain-plugin-v1"` for v0.15. |
|
||||
| `subagents` | string | no | Subdir name (default `subagents`). Escape-attempts are rejected. |
|
||||
| `description` | string | no | Shown in future `gbrain plugin list`. |
|
||||
| `description` | string | no | Shown in a future plugin-listing command. |
|
||||
|
||||
## Subagent definition files
|
||||
|
||||
|
||||
@@ -103,7 +103,7 @@ For GCP service-account / Vertex AI auth (production deployments), see the v0.32
|
||||
|
||||
### OpenRouter
|
||||
|
||||
Single OpenAI-compatible API for fan-out to OpenAI, Anthropic, Google, DeepSeek, Meta Llama, Qwen, and dozens of other hosted providers. One key, many models. Set `OPENROUTER_API_KEY` and use `openrouter:<provider>/<model>` (e.g. `openrouter:openai/gpt-5.2`, `openrouter:anthropic/claude-sonnet-4.6`).
|
||||
Single OpenAI-compatible API for fan-out to OpenAI, Anthropic, Google, DeepSeek, Meta Llama, Qwen, and dozens of other hosted providers. One key, many models. Set `OPENROUTER_API_KEY` or `openrouter_api_key` in `~/.gbrain/config.json`, then use `openrouter:<provider>/<model>` (e.g. `openrouter:openai/gpt-5.2`, `openrouter:anthropic/claude-sonnet-4.6`).
|
||||
|
||||
**Embedding**: `openai/text-embedding-3-small` (1536d default, Matryoshka shrink to 512/768/1024). OR's embedding catalog also includes `text-embedding-3-large`, `google/gemini-embedding-2-preview`, `qwen/qwen3-embedding-8b`, `bge-m3` — opt in via `--embedding-model openrouter:<id>`. Pricing matches the upstream provider (OR adds a small markup).
|
||||
|
||||
|
||||
+1
-1
@@ -250,7 +250,7 @@ All 30 GBrain operations are available remotely, including `sync_brain` and
|
||||
directory where `gbrain serve` was launched. Symlinks, `..` traversal, and absolute
|
||||
paths outside cwd are rejected. Page slugs and filenames are allowlist-validated
|
||||
(alphanumeric + hyphens; no control chars, RTL overrides, or backslashes). Local
|
||||
CLI callers (`gbrain file upload ...`) keep unrestricted filesystem access since
|
||||
CLI callers (`gbrain files upload ...`) keep unrestricted filesystem access since
|
||||
the user owns the machine.
|
||||
|
||||
## Deployment Options
|
||||
|
||||
@@ -49,6 +49,7 @@ The USD-limit knobs accept `off`, `unlimited`, or `none` (case-insensitive) to m
|
||||
| Backfill per-job budget | `embed.backfill_max_usd` | `10` | caps the job's tracker | `off` (`0` → default) | uncapped (still ledgered) |
|
||||
| Backfill cooldown | `embed.backfill_cooldown_min` | `10` | skips re-submission inside window | — (latency knob, not spend) | **not** bypassed |
|
||||
| `reindex-code` cost gate | — (preview before re-embed) | — | TTY prompt / non-TTY refuse + exit 2 | `--max-cost off` | informational |
|
||||
| `migrate embeddings` consent gate | — (plan + estimate before provider migration) | — | TTY y/N prompt / non-TTY refuse + exit 2 | `--yes` | estimate marked informational, but **still prompts** (guards a destructive schema rebuild, not just spend) |
|
||||
| `enrich` / `onboard --auto` | `--max-usd` (per-call) | — | refuse without a cap (non-TTY) | `--max-usd off` | runs uncapped (still ledgered) |
|
||||
|
||||
### Sync inline-embed cost gate
|
||||
|
||||
@@ -140,6 +140,9 @@ Stable phase names shipped in v0.15.2:
|
||||
- `import.files`
|
||||
- `sync.deletes`, `sync.renames`, `sync.imports`
|
||||
- `migrate.copy_pages`, `migrate.copy_links`
|
||||
- `migrate.reembed` (the re-embed pass of `gbrain migrate embeddings`; total is the
|
||||
stale-chunk backlog at the start of the pass, so it can grow slightly if a
|
||||
writer adds chunks mid-run)
|
||||
- `repair_jsonb.run`, `repair_jsonb.<table>.<column>`
|
||||
- `backlinks.scan`
|
||||
- `lint.pages`
|
||||
|
||||
@@ -13,7 +13,7 @@ Step-by-step walkthroughs that take you from zero to a working outcome. Concrete
|
||||
|
||||
These are the next tutorials on the roadmap. Open an issue if one of them is the one you need most; that's how we'll prioritize.
|
||||
|
||||
- **Set up GBrain for VC dealflow** — the operator's recipe. People pages for founders, companies with typed Facts fence carrying ARR / team-size / runway across dates, meetings auto-ingested, deal pages linking everything. Shows `gbrain whoknows`, `gbrain find_trajectory`, and `gbrain founder scorecard` on real workflows.
|
||||
- **Set up GBrain for VC dealflow** — the operator's recipe. People pages for founders, companies with typed Facts fence carrying ARR / team-size / runway across dates, meetings auto-ingested, deal pages linking everything. Shows `gbrain whoknows`, `gbrain find-trajectory`, and `gbrain founder scorecard` on real workflows.
|
||||
|
||||
- **Migrate your existing vault into GBrain** — for Notion / Obsidian / Roam users with a vault that doesn't match GBrain's default layout. Walks through `gbrain schema detect` → `suggest` → `review-candidates` so the brain learns your shape instead of forcing you to learn its.
|
||||
|
||||
|
||||
@@ -554,7 +554,7 @@ What to do next:
|
||||
|
||||
- **Wire ingestion** from external systems (Granola, Linear, Slack) using the [ingestion source contract](../skillpack-anatomy.md). Most companies want their meetings auto-ingested so the brain stays current without anyone typing notes.
|
||||
- **Set up team-specific dashboards** through the admin UI. Each team lead can have their own view of brain health and activity.
|
||||
- **Explore the rest of the brain layer.** `gbrain whoknows` (find the expert on a topic), `gbrain find_trajectory` (how a metric changed over time), `gbrain founder scorecard` (especially useful for VC and ops teams), the contradiction-detection cycle that surfaces conflicts between different people's notes.
|
||||
- **Explore the rest of the brain layer.** `gbrain whoknows` (find the expert on a topic), `gbrain find-trajectory` (how a metric changed over time), `gbrain founder scorecard` (especially useful for VC and ops teams), the contradiction-detection cycle that surfaces conflicts between different people's notes.
|
||||
|
||||
If you're building in this space (which YC has flagged as the [company-brain category in its Request for Startups](https://www.ycombinator.com/rfs#company-brain)), you might as well build on this. Everything described above is open source, MIT licensed, and what I run in production behind my own AI agents.
|
||||
|
||||
|
||||
@@ -115,21 +115,21 @@ You can use the same keys across multiple agents.
|
||||
|
||||
## Step 6: Install GBrain
|
||||
|
||||
Once OpenClaw is running:
|
||||
Once OpenClaw is running, installation is two commands — one in the brain repo, one in the agent workspace:
|
||||
|
||||
```bash
|
||||
gbrain install
|
||||
# In the BRAIN repo (the git repo that holds your markdown pages):
|
||||
gbrain init --supabase
|
||||
|
||||
# In the AGENT WORKSPACE repo (where OpenClaw runs):
|
||||
gbrain skillpack scaffold --all
|
||||
```
|
||||
|
||||
This installs:
|
||||
`gbrain init --supabase` walks a short wizard that asks for your Supabase connection string and creates the schema. You'll get that connection string in Step 7 — read 7a and 7b first so you paste the right one (the transaction pooler, not the direct connection). If you'd rather try things locally before paying for a database, `gbrain init --pglite` gives you a zero-config embedded engine instead; you can migrate to Supabase later with `gbrain migrate --to supabase`.
|
||||
|
||||
- About 60 skills
|
||||
- About 9 skill packs
|
||||
- Default brain structure
|
||||
- MCP server configuration
|
||||
- Supabase connection (for embeddings and search)
|
||||
`gbrain skillpack scaffold --all` copies the ~43 bundled skills into your agent workspace as first-class files you can edit freely. (The old managed-install model was retired in v0.36.0.0; see `docs/INSTALL.md` if you're upgrading from an older release.)
|
||||
|
||||
GBrain populates the brain repo with its default directory structure, skill files, and configuration. From this point, the agent has working memory and access to every skill.
|
||||
From this point, the agent has working memory and access to every skill.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -32,7 +32,7 @@ gbrain schema sync --apply
|
||||
The sync backfills `page.type = 'meeting'` on all 4000 pages in 1000-row batches. Now:
|
||||
|
||||
- `gbrain whoknows "Q3 roadmap discussion"` routes through the meeting type, ranking by `expert_routing` signal (attendees, recency, salience) instead of raw text.
|
||||
- `gbrain extract-facts` runs on every meeting page automatically (because `extractable: true`), pulling typed facts like `attended_by=alice-example`, `date=2026-05-23`.
|
||||
- The `extract_facts` cycle runs on every meeting page automatically (because `extractable: true`), pulling typed facts like `attended_by=alice-example`, `date=2026-05-23`.
|
||||
- The downstream `think` skill can now answer "what did we decide about pricing in the last three roadmap meetings" by querying the meeting graph instead of grep'ing 4000 files.
|
||||
|
||||
One command. 4000 pages went from invisible to queryable. The content didn't change. The structure did.
|
||||
@@ -62,7 +62,7 @@ gbrain schema add-link-type led-by --page-type deal --target-type inves
|
||||
gbrain schema sync --apply
|
||||
```
|
||||
|
||||
Now `gbrain whoknows "Series A SaaS"` routes through `investor` and `portco` types specifically, not the noisy general type set. `gbrain graph-query alice-example --type intro-from --depth 2` walks two hops of intros to surface "Alice introduced you to Bob who introduced you to Charlie." `gbrain extract-facts` starts producing typed claims from the fence in your deal pages: `(deals/acme-seed, raise=2000000, valuation=15000000, lead=widget-vc, closed_at=2026-05-23)`.
|
||||
Now `gbrain whoknows "Series A SaaS"` routes through `investor` and `portco` types specifically, not the noisy general type set. `gbrain graph-query alice-example --type intro-from --depth 2` walks two hops of intros to surface "Alice introduced you to Bob who introduced you to Charlie." The `extract_facts` cycle starts producing typed claims from the fence in your deal pages: `(deals/acme-seed, raise=2000000, valuation=15000000, lead=widget-vc, closed_at=2026-05-23)`.
|
||||
|
||||
The CRM you've been promising yourself you'll set up next quarter? You just shipped it in 4 commands. It's downstream of your notes, not parallel to them.
|
||||
|
||||
@@ -143,7 +143,7 @@ Re-run the same `whoknows` query. Top-3 should shift, because the new type is no
|
||||
|
||||
Three things gbrain does that generic note systems can't:
|
||||
|
||||
**1. The brain knows the difference between a person and an idea.** Page-type matters at query time. `gbrain whoknows` only considers `expert_routing: true` types. `gbrain extract-facts` only runs on `extractable: true` types. `gbrain graph-query` walks declared link verbs. None of that works on a flat tag system because tags don't have semantics — they're labels. Types are first-class citizens with rules attached.
|
||||
**1. The brain knows the difference between a person and an idea.** Page-type matters at query time. `gbrain whoknows` only considers `expert_routing: true` types. The `extract_facts` cycle only runs on `extractable: true` types. `gbrain graph-query` walks declared link verbs. None of that works on a flat tag system because tags don't have semantics — they're labels. Types are first-class citizens with rules attached.
|
||||
|
||||
**2. Untyped content is invisible content.** If your meetings are typed as `note`, expert routing skips them, facts extraction ignores them, link inference doesn't fire. They exist on disk and they're indexed for text search, but the structural surfaces (whoknows, find_experts, recall, think) treat them as second-class. Adding a type isn't cosmetic; it's structural promotion.
|
||||
|
||||
|
||||
+4
-4
@@ -2316,7 +2316,7 @@ gbrain schema sync --apply
|
||||
The sync backfills `page.type = 'meeting'` on all 4000 pages in 1000-row batches. Now:
|
||||
|
||||
- `gbrain whoknows "Q3 roadmap discussion"` routes through the meeting type, ranking by `expert_routing` signal (attendees, recency, salience) instead of raw text.
|
||||
- `gbrain extract-facts` runs on every meeting page automatically (because `extractable: true`), pulling typed facts like `attended_by=alice-example`, `date=2026-05-23`.
|
||||
- The `extract_facts` cycle runs on every meeting page automatically (because `extractable: true`), pulling typed facts like `attended_by=alice-example`, `date=2026-05-23`.
|
||||
- The downstream `think` skill can now answer "what did we decide about pricing in the last three roadmap meetings" by querying the meeting graph instead of grep'ing 4000 files.
|
||||
|
||||
One command. 4000 pages went from invisible to queryable. The content didn't change. The structure did.
|
||||
@@ -2346,7 +2346,7 @@ gbrain schema add-link-type led-by --page-type deal --target-type inves
|
||||
gbrain schema sync --apply
|
||||
```
|
||||
|
||||
Now `gbrain whoknows "Series A SaaS"` routes through `investor` and `portco` types specifically, not the noisy general type set. `gbrain graph-query alice-example --type intro-from --depth 2` walks two hops of intros to surface "Alice introduced you to Bob who introduced you to Charlie." `gbrain extract-facts` starts producing typed claims from the fence in your deal pages: `(deals/acme-seed, raise=2000000, valuation=15000000, lead=widget-vc, closed_at=2026-05-23)`.
|
||||
Now `gbrain whoknows "Series A SaaS"` routes through `investor` and `portco` types specifically, not the noisy general type set. `gbrain graph-query alice-example --type intro-from --depth 2` walks two hops of intros to surface "Alice introduced you to Bob who introduced you to Charlie." The `extract_facts` cycle starts producing typed claims from the fence in your deal pages: `(deals/acme-seed, raise=2000000, valuation=15000000, lead=widget-vc, closed_at=2026-05-23)`.
|
||||
|
||||
The CRM you've been promising yourself you'll set up next quarter? You just shipped it in 4 commands. It's downstream of your notes, not parallel to them.
|
||||
|
||||
@@ -2427,7 +2427,7 @@ Re-run the same `whoknows` query. Top-3 should shift, because the new type is no
|
||||
|
||||
Three things gbrain does that generic note systems can't:
|
||||
|
||||
**1. The brain knows the difference between a person and an idea.** Page-type matters at query time. `gbrain whoknows` only considers `expert_routing: true` types. `gbrain extract-facts` only runs on `extractable: true` types. `gbrain graph-query` walks declared link verbs. None of that works on a flat tag system because tags don't have semantics — they're labels. Types are first-class citizens with rules attached.
|
||||
**1. The brain knows the difference between a person and an idea.** Page-type matters at query time. `gbrain whoknows` only considers `expert_routing: true` types. The `extract_facts` cycle only runs on `extractable: true` types. `gbrain graph-query` walks declared link verbs. None of that works on a flat tag system because tags don't have semantics — they're labels. Types are first-class citizens with rules attached.
|
||||
|
||||
**2. Untyped content is invisible content.** If your meetings are typed as `note`, expert routing skips them, facts extraction ignores them, link inference doesn't fire. They exist on disk and they're indexed for text search, but the structural surfaces (whoknows, find_experts, recall, think) treat them as second-class. Adding a type isn't cosmetic; it's structural promotion.
|
||||
|
||||
@@ -3897,7 +3897,7 @@ All 30 GBrain operations are available remotely, including `sync_brain` and
|
||||
directory where `gbrain serve` was launched. Symlinks, `..` traversal, and absolute
|
||||
paths outside cwd are rejected. Page slugs and filenames are allowlist-validated
|
||||
(alphanumeric + hyphens; no control chars, RTL overrides, or backslashes). Local
|
||||
CLI callers (`gbrain file upload ...`) keep unrestricted filesystem access since
|
||||
CLI callers (`gbrain files upload ...`) keep unrestricted filesystem access since
|
||||
the user owns the machine.
|
||||
|
||||
## Deployment Options
|
||||
|
||||
+35
-33
@@ -23,6 +23,7 @@
|
||||
"./backoff": "./src/core/backoff.ts",
|
||||
"./search/hybrid": "./src/core/search/hybrid.ts",
|
||||
"./search/expansion": "./src/core/search/expansion.ts",
|
||||
"./think": "./src/core/think/index.ts",
|
||||
"./ai/gateway": "./src/core/ai/gateway.ts",
|
||||
"./extract": "./src/commands/extract.ts",
|
||||
"./ingestion": "./src/core/ingestion/index.ts",
|
||||
@@ -41,20 +42,20 @@
|
||||
"eval:autocut": "bun test test/search/autocut-eval.test.ts",
|
||||
"test:full": "bun run verify && bash scripts/run-unit-parallel.sh && bun run test:slow && ([ -n \"$DATABASE_URL\" ] && bash scripts/run-e2e.sh || echo '[test:full] skipped E2E (no DATABASE_URL); run docker-compose -f docker-compose.ci.yml up + bun run test:e2e to include' 1>&2)",
|
||||
"verify": "bash scripts/run-verify-parallel.sh",
|
||||
"check:source-config-leak": "scripts/check-source-config-leak.sh",
|
||||
"check:no-pii-agent-voice": "scripts/check-no-pii-in-agent-voice.sh",
|
||||
"check:synthetic-corpus-privacy": "scripts/check-synthetic-corpus-privacy.sh",
|
||||
"check:system-of-record": "scripts/check-system-of-record.sh",
|
||||
"check:admin-scope-drift": "scripts/check-admin-scope-drift.sh",
|
||||
"check:cli-exec": "scripts/check-cli-executable.sh",
|
||||
"check:all": "scripts/check-privacy.sh && scripts/check-proposal-pii.sh && scripts/check-test-real-names.sh && scripts/check-jsonb-pattern.sh && scripts/check-source-id-projection.sh && scripts/check-source-config-leak.sh && scripts/check-progress-to-stdout.sh && scripts/check-no-legacy-getconnection.sh && scripts/check-test-isolation.sh && scripts/check-trailing-newline.sh && scripts/check-wasm-embedded.sh && scripts/check-exports-count.sh && scripts/check-admin-build.sh && scripts/check-admin-scope-drift.sh && scripts/check-cli-executable.sh && scripts/check-skill-brain-first.sh && scripts/check-operations-filter-bypass.sh && scripts/check-gateway-routed-no-direct-anthropic.sh && scripts/check-worker-pool-atomicity.sh && scripts/check-key-files-current-state.sh && scripts/check-no-double-retry.sh && scripts/check-batch-audit-site.sh",
|
||||
"check:gateway-routed": "scripts/check-gateway-routed-no-direct-anthropic.sh",
|
||||
"check:worker-pool-atomicity": "scripts/check-worker-pool-atomicity.sh",
|
||||
"check:doc-history": "scripts/check-key-files-current-state.sh",
|
||||
"check:source-config-leak": "bash scripts/check-source-config-leak.sh",
|
||||
"check:no-pii-agent-voice": "bash scripts/check-no-pii-in-agent-voice.sh",
|
||||
"check:synthetic-corpus-privacy": "bash scripts/check-synthetic-corpus-privacy.sh",
|
||||
"check:system-of-record": "bash scripts/check-system-of-record.sh",
|
||||
"check:admin-scope-drift": "bash scripts/check-admin-scope-drift.sh",
|
||||
"check:cli-exec": "bash scripts/check-cli-executable.sh",
|
||||
"check:all": "bash scripts/check-privacy.sh && bash scripts/check-proposal-pii.sh && bash scripts/check-test-real-names.sh && bash scripts/check-jsonb-pattern.sh && bash scripts/check-source-id-projection.sh && bash scripts/check-source-config-leak.sh && bash scripts/check-progress-to-stdout.sh && bash scripts/check-no-tracked-symlinks.sh && bash scripts/check-no-legacy-getconnection.sh && bash scripts/check-test-isolation.sh && bash scripts/check-trailing-newline.sh && bash scripts/check-wasm-embedded.sh && bash scripts/check-exports-count.sh && bash scripts/check-admin-build.sh && bash scripts/check-admin-scope-drift.sh && bash scripts/check-cli-executable.sh && bash scripts/check-skill-brain-first.sh && bash scripts/check-operations-filter-bypass.sh && bash scripts/check-gateway-routed-no-direct-anthropic.sh && bash scripts/check-worker-pool-atomicity.sh && bash scripts/check-key-files-current-state.sh && bash scripts/check-no-double-retry.sh && bash scripts/check-batch-audit-site.sh",
|
||||
"check:gateway-routed": "bash scripts/check-gateway-routed-no-direct-anthropic.sh",
|
||||
"check:worker-pool-atomicity": "bash scripts/check-worker-pool-atomicity.sh",
|
||||
"check:doc-history": "bash scripts/check-key-files-current-state.sh",
|
||||
"check:resolver": "bun src/cli.ts check-resolvable --strict --skills-dir skills/",
|
||||
"check:skill-brain-first": "scripts/check-skill-brain-first.sh",
|
||||
"check:wasm": "scripts/check-wasm-embedded.sh",
|
||||
"check:newlines": "scripts/check-trailing-newline.sh",
|
||||
"check:skill-brain-first": "bash scripts/check-skill-brain-first.sh",
|
||||
"check:wasm": "bash scripts/check-wasm-embedded.sh",
|
||||
"check:newlines": "bash scripts/check-trailing-newline.sh",
|
||||
"test:e2e": "bash scripts/run-e2e.sh",
|
||||
"test:slow": "bash scripts/run-slow-tests.sh",
|
||||
"test:heavy": "bash scripts/run-heavy.sh",
|
||||
@@ -64,26 +65,27 @@
|
||||
"ci:local:diff": "bash scripts/ci-local.sh --diff",
|
||||
"ci:select-e2e": "bun run scripts/select-e2e.ts",
|
||||
"typecheck": "tsc --noEmit",
|
||||
"check:jsonb": "scripts/check-jsonb-pattern.sh",
|
||||
"check:search-path": "scripts/check-search-path.sh",
|
||||
"check:no-double-retry": "scripts/check-no-double-retry.sh",
|
||||
"check:batch-audit-site": "scripts/check-batch-audit-site.sh",
|
||||
"check:worker-lock-renewal-shape": "scripts/check-worker-lock-renewal-shape.sh",
|
||||
"check:source-id-projection": "scripts/check-source-id-projection.sh",
|
||||
"check:privacy": "scripts/check-privacy.sh",
|
||||
"check:proposal-pii": "scripts/check-proposal-pii.sh",
|
||||
"check:eval-glossary": "scripts/check-eval-glossary-fresh.sh",
|
||||
"check:test-names": "scripts/check-test-real-names.sh",
|
||||
"check:progress": "scripts/check-progress-to-stdout.sh",
|
||||
"check:exports-count": "scripts/check-exports-count.sh",
|
||||
"check:admin-build": "scripts/check-admin-build.sh",
|
||||
"check:admin-embedded": "scripts/check-admin-embedded.sh",
|
||||
"check:test-isolation": "scripts/check-test-isolation.sh",
|
||||
"check:fuzz-purity": "scripts/check-fuzz-purity.sh",
|
||||
"check:operations-filter-bypass": "scripts/check-operations-filter-bypass.sh",
|
||||
"check:fixture-privacy": "scripts/check-fixture-privacy.sh",
|
||||
"check:jsonb": "bash scripts/check-jsonb-pattern.sh",
|
||||
"check:search-path": "bash scripts/check-search-path.sh",
|
||||
"check:no-double-retry": "bash scripts/check-no-double-retry.sh",
|
||||
"check:batch-audit-site": "bash scripts/check-batch-audit-site.sh",
|
||||
"check:worker-lock-renewal-shape": "bash scripts/check-worker-lock-renewal-shape.sh",
|
||||
"check:source-id-projection": "bash scripts/check-source-id-projection.sh",
|
||||
"check:privacy": "bash scripts/check-privacy.sh",
|
||||
"check:proposal-pii": "bash scripts/check-proposal-pii.sh",
|
||||
"check:eval-glossary": "bash scripts/check-eval-glossary-fresh.sh",
|
||||
"check:test-names": "bash scripts/check-test-real-names.sh",
|
||||
"check:progress": "bash scripts/check-progress-to-stdout.sh",
|
||||
"check:no-tracked-symlinks": "bash scripts/check-no-tracked-symlinks.sh",
|
||||
"check:exports-count": "bash scripts/check-exports-count.sh",
|
||||
"check:admin-build": "bash scripts/check-admin-build.sh",
|
||||
"check:admin-embedded": "bash scripts/check-admin-embedded.sh",
|
||||
"check:test-isolation": "bash scripts/check-test-isolation.sh",
|
||||
"check:fuzz-purity": "bash scripts/check-fuzz-purity.sh",
|
||||
"check:operations-filter-bypass": "bash scripts/check-operations-filter-bypass.sh",
|
||||
"check:fixture-privacy": "bash scripts/check-fixture-privacy.sh",
|
||||
"check:conversation-parser": "bun src/cli.ts eval conversation-parser test/fixtures/conversation-formats/all.jsonl --no-llm",
|
||||
"check:source-scope-onboard": "scripts/check-source-scope-onboard.sh",
|
||||
"check:source-scope-onboard": "bash scripts/check-source-scope-onboard.sh",
|
||||
"postinstall": "bun run scripts/postinstall.ts",
|
||||
"prepublish:clawhub": "bun run build:all",
|
||||
"publish:clawhub": "clawhub package publish . --family bundle-plugin"
|
||||
@@ -144,7 +146,7 @@
|
||||
"bun": ">=1.3.10"
|
||||
},
|
||||
"license": "MIT",
|
||||
"version": "0.42.65.0",
|
||||
"version": "0.42.67.0",
|
||||
"overrides": {
|
||||
"@hono/node-server": "^2.0.5",
|
||||
"fast-uri": "^3.1.4",
|
||||
|
||||
@@ -19,7 +19,7 @@
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
EXPECTED_COUNT=20
|
||||
EXPECTED_COUNT=21
|
||||
|
||||
# Count top-level keys in the exports object. `node -e` parses JSON
|
||||
# reliably without needing jq (which isn't in every CI environment).
|
||||
|
||||
Executable
+67
@@ -0,0 +1,67 @@
|
||||
#!/usr/bin/env bash
|
||||
# CI guard: fail if any symlink is tracked in git.
|
||||
#
|
||||
# A symlink committed from a build sandbox points at a path that exists on
|
||||
# exactly one machine. Everywhere else the checkout produces a dangling
|
||||
# link, and anything that opens it fails. That is not hypothetical: commit
|
||||
# faf5cdba landed `node_modules -> /tmp/fleet/repo/node_modules`, which made
|
||||
# `bun install` abort with `ENOENT: could not open the "node_modules"
|
||||
# directory` on every fresh clone, and took `gbrain upgrade`'s bun-link path
|
||||
# down with it (the auto-upgrade runs `bun install`, so the printed manual
|
||||
# fallback failed the same way).
|
||||
#
|
||||
# .gitignore alone does not prevent this. A `node_modules/` pattern with a
|
||||
# trailing slash matches directories ONLY, so a symlink of the same name is
|
||||
# never ignored. Dropping the slash closes that hole, but `git add -f` still
|
||||
# walks straight past it. This guard is the backstop.
|
||||
#
|
||||
# The repo has no legitimate tracked symlinks, so the allowlist starts
|
||||
# empty. If you ever need one, add its exact repo-relative path to ALLOWLIST
|
||||
# below and explain why — a relative link that resolves inside the repo is
|
||||
# defensible; an absolute one almost never is.
|
||||
#
|
||||
# Usage: scripts/check-no-tracked-symlinks.sh
|
||||
# Exit: 0 when clean, 1 when a tracked symlink is found.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
ROOT="$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
|
||||
cd "$ROOT"
|
||||
|
||||
# Paths permitted to be tracked symlinks. Empty by design.
|
||||
ALLOWLIST=()
|
||||
|
||||
# Git records symlinks with mode 120000. Field 4 of `ls-files -s` is the path
|
||||
# (tab-separated from the stage number), so cut on the tab to keep paths with
|
||||
# spaces intact.
|
||||
found="$(git ls-files -s | awk '$1 == "120000"' | cut -f2- || true)"
|
||||
|
||||
if [ -n "$found" ]; then
|
||||
filtered="$found"
|
||||
for f in "${ALLOWLIST[@]:-}"; do
|
||||
[ -z "$f" ] && continue
|
||||
filtered="$(echo "$filtered" | grep -vxF "$f" || true)"
|
||||
done
|
||||
|
||||
if [ -n "$filtered" ]; then
|
||||
echo "ERROR: symlink(s) tracked in git:"
|
||||
echo
|
||||
while IFS= read -r path; do
|
||||
[ -z "$path" ] && continue
|
||||
target="$(git cat-file blob ":$path" 2>/dev/null || echo '<unreadable>')"
|
||||
echo " $path -> $target"
|
||||
done <<< "$filtered"
|
||||
echo
|
||||
echo "A committed symlink resolves on the machine that created it and"
|
||||
echo "nowhere else. Untrack it:"
|
||||
echo
|
||||
echo " git rm --cached <path>"
|
||||
echo
|
||||
echo "If the path is build output (node_modules, dist, bin), also confirm"
|
||||
echo "it is covered by .gitignore WITHOUT a trailing slash — a trailing"
|
||||
echo "slash matches directories only and lets the symlink through."
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
echo "check-no-tracked-symlinks: OK (no tracked symlinks)"
|
||||
@@ -19,13 +19,25 @@ set -euo pipefail
|
||||
REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
cd "$REPO_ROOT"
|
||||
|
||||
OUT_BIN="$(mktemp /tmp/gbrain-wasm-check.XXXXXX)"
|
||||
trap 'rm -f "$OUT_BIN"' EXIT
|
||||
# Build from a container-local copy. On Docker Desktop, Bun canonicalizes a
|
||||
# bind-mounted input to /run/host_virtiofs but keeps /app as the output path;
|
||||
# its final atomic rename then fails with ENOENT even though both names refer
|
||||
# to the same mount. Keeping inputs and output under /tmp avoids that alias.
|
||||
BUILD_DIR="$(mktemp -d /tmp/gbrain-wasm-check.XXXXXX)"
|
||||
OUT_BIN="$BUILD_DIR/chunker-smoketest"
|
||||
trap 'rm -rf "$BUILD_DIR"' EXIT
|
||||
mkdir -p "$BUILD_DIR/scripts"
|
||||
cp -R "$REPO_ROOT/src" "$BUILD_DIR/src"
|
||||
cp "$REPO_ROOT/scripts/chunker-smoketest.ts" "$BUILD_DIR/scripts/chunker-smoketest.ts"
|
||||
ln -s "$REPO_ROOT/node_modules" "$BUILD_DIR/node_modules"
|
||||
|
||||
# Build a minimal smoketest binary that imports the chunker. We compile this
|
||||
# instead of the full gbrain CLI so the failure mode is laser-focused on
|
||||
# chunker + WASM path resolution, not unrelated CLI wiring.
|
||||
bun build --compile --outfile "$OUT_BIN" scripts/chunker-smoketest.ts >/dev/null 2>&1
|
||||
if ! (cd "$BUILD_DIR" && bun build --compile --outfile "$OUT_BIN" scripts/chunker-smoketest.ts >/dev/null); then
|
||||
echo "[check-wasm-embedded] FAIL: bun could not compile the smoketest binary." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Run it and capture JSON output.
|
||||
OUTPUT="$("$OUT_BIN" 2>&1)"
|
||||
|
||||
+1
-1
@@ -350,7 +350,7 @@ if [ -f .git ]; then
|
||||
fi
|
||||
|
||||
echo "[ci-local] Running checks inside runner container..."
|
||||
docker compose -f "$COMPOSE_FILE" run --rm "${EXTRA_MOUNTS[@]:-}" runner bash -c "$INNER_CMD"
|
||||
docker compose -f "$COMPOSE_FILE" run --rm "${EXTRA_MOUNTS[@]}" runner bash -c "$INNER_CMD"
|
||||
|
||||
echo ""
|
||||
echo "[ci-local] All checks passed."
|
||||
|
||||
+15
-2
@@ -42,8 +42,19 @@ export const E2E_TEST_MAP: Record<string, string[]> = {
|
||||
// phase, extract, integrity, embed, or migrate-engine change.
|
||||
"src/core/cycle/extract-takes.ts": ["test/e2e/multi-source-bug-class.test.ts"],
|
||||
"src/core/cycle/patterns.ts": ["test/e2e/multi-source-bug-class.test.ts"],
|
||||
"src/core/cycle/synthesize.ts": ["test/e2e/multi-source-bug-class.test.ts"],
|
||||
"src/commands/embed.ts": ["test/e2e/multi-source-bug-class.test.ts"],
|
||||
"src/core/cycle/synthesize.ts": [
|
||||
"test/e2e/multi-source-bug-class.test.ts",
|
||||
"test/e2e/synthesize-bigint-job-id-postgres.test.ts",
|
||||
],
|
||||
"src/commands/embed.ts": [
|
||||
"test/e2e/multi-source-bug-class.test.ts",
|
||||
// #3391: the NULL-signature stale predicates differ per engine.
|
||||
"test/e2e/migrate-embeddings-postgres.test.ts",
|
||||
],
|
||||
// #3390: runSchemaTransition's DDL path + the stale predicates behave
|
||||
// differently on real pgvector than on PGLite.
|
||||
"src/core/embedding-migration.ts": ["test/e2e/migrate-embeddings-postgres.test.ts"],
|
||||
"src/core/retrieval-upgrade-planner.ts": ["test/e2e/migrate-embeddings-postgres.test.ts"],
|
||||
"src/commands/extract.ts": ["test/e2e/multi-source-bug-class.test.ts"],
|
||||
"src/commands/migrate-engine.ts": ["test/e2e/multi-source-bug-class.test.ts"],
|
||||
// Any minions queue/worker/handler change exercises all minion E2E.
|
||||
@@ -61,6 +72,8 @@ export const E2E_TEST_MAP: Record<string, string[]> = {
|
||||
"test/e2e/jsonb-roundtrip.test.ts",
|
||||
"test/e2e/engine-parity.test.ts",
|
||||
"test/e2e/schema-drift.test.ts",
|
||||
// #3391: includeNullSignature stale predicates (engine parity).
|
||||
"test/e2e/migrate-embeddings-postgres.test.ts",
|
||||
],
|
||||
// PGLite bootstrap path + parity guard.
|
||||
"src/core/pglite-engine.ts": [
|
||||
|
||||
@@ -42,6 +42,7 @@ CHECKS=(
|
||||
"check:source-id-projection"
|
||||
"check:source-config-leak"
|
||||
"check:progress"
|
||||
"check:no-tracked-symlinks"
|
||||
"check:test-isolation"
|
||||
"check:wasm"
|
||||
"check:admin-build"
|
||||
|
||||
@@ -248,7 +248,7 @@ before submission.
|
||||
After the brain page is written, render to PDF using `skills/brain-pdf`:
|
||||
|
||||
```bash
|
||||
gbrain put_page # already done by the CLI; nothing to add here
|
||||
gbrain put # already done by the CLI; nothing to add here
|
||||
# Then invoke brain-pdf:
|
||||
# (see skills/brain-pdf/SKILL.md for the make-pdf invocation)
|
||||
```
|
||||
|
||||
@@ -73,13 +73,13 @@ stock worker auto-loads on startup) registers handlers before `start()`.
|
||||
Users who set `minion_mode: off` in `~/.gbrain/preferences.json` keep
|
||||
using `agentTurn`. Respect that. No auto-rewrite.
|
||||
|
||||
## Forward note (v0.12.0)
|
||||
## Forward note
|
||||
|
||||
GBrain v0.12.0 ships `gbrain cron`: a scheduler loop inside
|
||||
`gbrain jobs work` that owns cron expressions natively — no more
|
||||
handing off to host schedulers. Until v0.12.0 lands, the host
|
||||
scheduler keeps firing on schedule; v0.11.1 only replaces the execution
|
||||
layer (what the cron trigger *does*), not the scheduling layer.
|
||||
A native scheduler loop inside `gbrain jobs work` (owning cron
|
||||
expressions directly, with no host-scheduler hand-off) has been on the
|
||||
roadmap since v0.11.1 but has not shipped. The host scheduler keeps
|
||||
firing on schedule; this convention only replaces the execution layer
|
||||
(what the cron trigger *does*), not the scheduling layer.
|
||||
|
||||
## Related
|
||||
|
||||
|
||||
@@ -54,8 +54,8 @@ Ask the user what they want to track. Either:
|
||||
- Define a custom recipe with: source queries, classification rules, extraction schema,
|
||||
tracker page path, tracker format
|
||||
|
||||
Recipes are YAML files at `~/.gbrain/recipes/{name}.yaml`. Use `gbrain research init`
|
||||
to scaffold a new one.
|
||||
Recipes are YAML files at `~/.gbrain/recipes/{name}.yaml`. Scaffold a new one by
|
||||
copying a built-in recipe file and editing its fields.
|
||||
|
||||
### Phase 2: Search Sources
|
||||
|
||||
|
||||
@@ -201,7 +201,7 @@ Use the brain page template. MUST include:
|
||||
|
||||
### 4b. Entity pages (people, companies)
|
||||
For each entity mentioned:
|
||||
- Check if a brain page exists (`gbrain search "<name>"` or `gbrain get_page people/<slug>`).
|
||||
- Check if a brain page exists (`gbrain search "<name>"` or `gbrain get people/<slug>`).
|
||||
- If exists: update State, append Timeline entry citing this research.
|
||||
- If not: create with enrichment.
|
||||
|
||||
|
||||
@@ -112,7 +112,7 @@ gbrain query "<topic keywords>"
|
||||
# -d '{"model": "sonar-pro", "messages": [{"role":"user","content":"..."}]}'
|
||||
|
||||
# 4. Write the structured research page via put_page:
|
||||
gbrain put_page research/<slug> # via the put_page operation
|
||||
gbrain put research/<slug> # via the put_page operation
|
||||
|
||||
# 5. Cross-link entities mentioned (people, companies) per Iron Law.
|
||||
```
|
||||
|
||||
@@ -11,7 +11,7 @@ tools:
|
||||
- gbrain schema active
|
||||
- gbrain schema use
|
||||
- gbrain schema stats
|
||||
- gbrain pages restore
|
||||
- gbrain restore
|
||||
- mcp:run_onboard
|
||||
triggers:
|
||||
- "unify my types"
|
||||
@@ -143,7 +143,7 @@ WHERE source_id = 'default' AND frontmatter->>'legacy_type' IS NOT NULL;
|
||||
Page-to-alias and page-to-link source pages soft-delete with 72h TTL. Restore within that window:
|
||||
|
||||
```bash
|
||||
gbrain pages restore <slug>
|
||||
gbrain restore <slug>
|
||||
```
|
||||
|
||||
Revert the active pack flip:
|
||||
@@ -197,7 +197,7 @@ Outputs:
|
||||
- Active pack flipped to `gbrain-base-v2` atomically at end of successful run.
|
||||
|
||||
Side effects:
|
||||
- Source pages soft-deleted with 72h restore TTL (`gbrain pages restore <slug>`).
|
||||
- Source pages soft-deleted with 72h restore TTL (`gbrain restore <slug>`).
|
||||
- One-time cache invalidation on KNOBS_HASH_VERSION bump (5→6); self-healing in `cache.ttl_seconds`.
|
||||
- Query-time `--type X` alias-expands via `expandTypeFilter` (D14 back-compat).
|
||||
|
||||
@@ -212,7 +212,7 @@ DON'T:
|
||||
- Submit `unify-types` directly via the MCP `submit_job` op without `--allow-protected`. PROTECTED handlers require trusted local callers; remote MCP rejection is the intentional trust boundary.
|
||||
- Edit `mapping_rules` in `gbrain-base-v2.yaml` to skip clusters you don't trust. Fork the pack instead (`gbrain schema fork`) so the source-of-truth migration stays consistent across brains.
|
||||
- Run `unify-types` from inside an autopilot tick. The check is `manual_only` per D17 — autopilot deliberately never auto-fires it because pack upgrades are one-time consenting taxonomy decisions.
|
||||
- Hard-delete soft-deleted source pages before the 72h restore window. Use `gbrain pages restore <slug>` first if rollback is needed.
|
||||
- Hard-delete soft-deleted source pages before the 72h restore window. Use `gbrain restore <slug>` first if rollback is needed.
|
||||
- Assume `frontmatter.legacy_type` survives every roundtrip. The marker is canonical for the immediate post-migration window; downstream re-imports may overwrite it.
|
||||
|
||||
## Output Format
|
||||
|
||||
@@ -60,7 +60,14 @@ Before skillifying, check:
|
||||
- Is there >20 lines of logic? (Trivial helpers don't need full infrastructure)
|
||||
- Does it have a clear trigger phrase a user would actually say?
|
||||
|
||||
If no to all three, it's a script, not a skill. Move on.
|
||||
If ANY answer is no, it's a script, not a skill — stop here. Do not scaffold, write a SKILL.md, run evals, or write tests for it. Tell the user why and move on.
|
||||
|
||||
Scope check (upper bound): one skill = one capability = one coherent trigger
|
||||
family. If the target spans multiple distinct intents users would invoke
|
||||
separately ("run the build" / "roll back the deploy" / "notify the team" are
|
||||
three intents, not one), do NOT build one skill covering them all. Stop,
|
||||
propose splitting into separate skillify targets, and ask the user which one
|
||||
to skillify first.
|
||||
|
||||
## Phase 1: Audit
|
||||
|
||||
|
||||
@@ -43,8 +43,9 @@ The Analysis section can interpret; the transcript section is sacred.
|
||||
|
||||
The user sends an audio or voice message via any channel (Telegram, voice
|
||||
memo upload, openclaw audio attachment). The host agent typically provides
|
||||
the transcript text. If not, transcribe via `gbrain transcription` (Groq
|
||||
Whisper by default; OpenAI fallback for audio > 25MB segmented via ffmpeg).
|
||||
the transcript text. If not, transcribe it with your host's transcription
|
||||
tool (Groq Whisper is fast and cheap; OpenAI Whisper works too — segment
|
||||
audio > 25MB via ffmpeg first).
|
||||
|
||||
## The pipeline
|
||||
|
||||
@@ -52,8 +53,9 @@ Whisper by default; OpenAI fallback for audio > 25MB segmented via ffmpeg).
|
||||
1. STORE → Upload original audio to gbrain storage backend
|
||||
(S3 / Supabase Storage / local — pluggable per
|
||||
src/core/storage.ts).
|
||||
2. TRANSCRIBE → Use the agent-provided transcript verbatim, OR call
|
||||
gbrain transcription if no transcript was supplied.
|
||||
2. TRANSCRIBE → Use the agent-provided transcript verbatim, OR
|
||||
transcribe the audio yourself (see "When to invoke")
|
||||
if no transcript was supplied.
|
||||
3. ROUTE → Apply the decision tree (below) to find the right
|
||||
destination directory.
|
||||
4. WRITE → Create / update the destination brain page; preserve the
|
||||
|
||||
@@ -1,13 +1,13 @@
|
||||
// AUTO-GENERATED — do not edit by hand.
|
||||
// Run `bun run scripts/build-admin-embedded.ts` to regenerate.
|
||||
// Source: admin/dist/ at 2026-05-27.
|
||||
// Source: admin/dist/ at 2026-07-24.
|
||||
//
|
||||
// Bun resolves the file: imports to a path that works at runtime even
|
||||
// inside a compiled binary (`bun build --compile`). The manifest maps
|
||||
// the request path the express handler sees to (resolved-path, mime).
|
||||
|
||||
// @ts-ignore — type: 'file' is Bun ESM, not in lib.d.ts
|
||||
import A_0_assets_index_CoGEje3__js from '../admin/dist/assets/index-CoGEje3-.js' with { type: 'file' };
|
||||
import A_0_assets_index_CviJXT_1_js from '../admin/dist/assets/index-CviJXT-1.js' with { type: 'file' };
|
||||
// @ts-ignore — type: 'file' is Bun ESM, not in lib.d.ts
|
||||
import A_1_assets_index_GxkWX7v3_css from '../admin/dist/assets/index-GxkWX7v3.css' with { type: 'file' };
|
||||
// @ts-ignore — type: 'file' is Bun ESM, not in lib.d.ts
|
||||
@@ -19,7 +19,7 @@ export interface AdminAsset {
|
||||
}
|
||||
|
||||
export const ADMIN_ASSETS: Record<string, AdminAsset> = {
|
||||
"/admin/assets/index-CoGEje3-.js": { path: A_0_assets_index_CoGEje3__js as unknown as string, mime: "application/javascript; charset=utf-8" },
|
||||
"/admin/assets/index-CviJXT-1.js": { path: A_0_assets_index_CviJXT_1_js as unknown as string, mime: "application/javascript; charset=utf-8" },
|
||||
"/admin/assets/index-GxkWX7v3.css": { path: A_1_assets_index_GxkWX7v3_css as unknown as string, mime: "text/css; charset=utf-8" },
|
||||
"/admin/index.html": { path: A_2_index_html as unknown as string, mime: "text/html; charset=utf-8" },
|
||||
};
|
||||
|
||||
+157
-14
@@ -9,7 +9,7 @@ installSigchldHandler();
|
||||
import { installSignalHandlers as installCleanupSignalHandlers } from './core/process-cleanup.ts';
|
||||
installCleanupSignalHandlers();
|
||||
|
||||
import { readFileSync, existsSync, unlinkSync } from 'fs';
|
||||
import { readFileSync, existsSync, unlinkSync, fstatSync } from 'fs';
|
||||
import { spawn } from 'child_process';
|
||||
import {
|
||||
readUpdateCache,
|
||||
@@ -55,12 +55,17 @@ export function bigintToStringReplacer(_key: string, value: unknown): unknown {
|
||||
}
|
||||
|
||||
// CLI-only commands that bypass the operation layer
|
||||
export const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'maintain', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'reconcile-links', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'calibration', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'advisor', 'watch', 'reindex-search-vector']);
|
||||
export const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'maintain', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'reconcile-links', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'calibration', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'retrieval-upgrade', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'advisor', 'watch', 'reindex-search-vector', 'pages', 'bench', 'backfill']);
|
||||
// CLI-only commands whose handlers print their own --help text. These are
|
||||
// excluded from the generic short-circuit so detailed per-command and
|
||||
// per-subcommand usage stays reachable.
|
||||
const CLI_ONLY_SELF_HELP = new Set([
|
||||
'upgrade', 'post-upgrade', 'check-update',
|
||||
// #3502 sweep: pages + bench print their own usage (pages.ts printHelp,
|
||||
// bench-publish.ts printHelp). Both were documented but undispatchable —
|
||||
// `pages` had a live handleCliOnly case but was missing from CLI_ONLY
|
||||
// (the #2035 calibration bug class); `bench` was never wired at all.
|
||||
'pages', 'bench',
|
||||
'embed', 'config',
|
||||
'skillpack', 'skillpack-check',
|
||||
'integrations', 'friction',
|
||||
@@ -107,6 +112,10 @@ const CLI_ONLY_SELF_HELP = new Set([
|
||||
// `gbrain connect --help` prints its own usage (flags + examples) from
|
||||
// runConnect; route around the generic one-line short-circuit.
|
||||
'connect',
|
||||
// #3390 — `gbrain migrate embeddings --help` / `gbrain retrieval-upgrade
|
||||
// --help` print the migration flags from runMigrateEmbeddings. `migrate`
|
||||
// (engine transfer) keeps its own dispatch too.
|
||||
'migrate', 'retrieval-upgrade',
|
||||
]);
|
||||
|
||||
// v114 (#1941): alias -> operation lookup, kept separate from `cliOps` so
|
||||
@@ -340,6 +349,11 @@ async function main() {
|
||||
// them out of the engine try/catch is safe and unlocks routing.
|
||||
const params = parseOpArgs(op, subArgs);
|
||||
|
||||
// #3513: stdin fill moved out of parseOpArgs so a non-TTY stdin with no
|
||||
// piped input can't block the parse forever — the bounded read leaves the
|
||||
// param unset on timeout and the required-param check below fails fast.
|
||||
await applyStdinParam(op, params);
|
||||
|
||||
// v0.27.1 (`gbrain query --image <path>`): swap the `image` param from
|
||||
// a filesystem path into base64 bytes + mime. The op accepts base64; the
|
||||
// CLI accepts a path. Helper is exported so tests can exercise the
|
||||
@@ -800,18 +814,99 @@ export function parseOpArgs(op: Operation, args: string[]): Record<string, unkno
|
||||
}
|
||||
}
|
||||
|
||||
// Read stdin for content params
|
||||
return params;
|
||||
}
|
||||
|
||||
/**
|
||||
* #3513: read stdin into an op's stdin-capable param without ever blocking
|
||||
* forever. The old inline `readFileSync(0)` in parseOpArgs assumed non-TTY
|
||||
* implies piped content; a non-TTY stdin with NO input (CI step, cron job,
|
||||
* agent harness holding an unwritten pipe open) blocked the read until kill.
|
||||
*
|
||||
* Strategy by fd kind (fstat):
|
||||
* - TTY: skip, as before (interactive input is not an op-param source).
|
||||
* - regular file / /dev/null / anything not a pipe or socket: readFileSync
|
||||
* returns without blocking (`gbrain put x < file`, `< /dev/null` → '').
|
||||
* - FIFO/socket: stream-read with a deadline on the FIRST byte only. A real
|
||||
* pipe (`echo foo | gbrain put x`, heredocs) delivers its first byte
|
||||
* within milliseconds; once any data arrives the deadline is lifted and
|
||||
* we read to EOF like readFileSync did (slow producers stay supported).
|
||||
* An empty-but-closed pipe (`: | gbrain put x`) EOFs immediately → ''.
|
||||
* A pipe that never delivers a byte times out → param stays unset, so
|
||||
* the existing required-param usage error fires (fail fast, exit 1).
|
||||
*
|
||||
* GBRAIN_STDIN_TIMEOUT_MS overrides the first-byte deadline (default 5000).
|
||||
* Exported for tests; called by the op dispatch right after parseOpArgs.
|
||||
*/
|
||||
export async function applyStdinParam(
|
||||
op: Operation,
|
||||
params: Record<string, unknown>,
|
||||
): Promise<void> {
|
||||
// Branch shape (stdin hint + missing param + `!process.stdin.isTTY` gate +
|
||||
// 5MB cap) is pinned by the R4 regression test for PR #1325's Windows fix
|
||||
// (test/cycle/regression-pr-wave-r1-r2-r4.test.ts) — keep the spelling.
|
||||
if (op.cliHints?.stdin && !params[op.cliHints.stdin] && !process.stdin.isTTY) {
|
||||
const stdinContent = readFileSync(0, 'utf-8');
|
||||
const content = await readStdinBounded();
|
||||
if (content === null) return; // no input arrived — let the required-param check fail fast
|
||||
const MAX_STDIN = 5_000_000; // 5MB
|
||||
if (Buffer.byteLength(stdinContent, 'utf-8') > MAX_STDIN) {
|
||||
if (Buffer.byteLength(content, 'utf-8') > MAX_STDIN) {
|
||||
console.error(`Error: stdin content exceeds ${MAX_STDIN} bytes. Split into smaller inputs.`);
|
||||
process.exit(1);
|
||||
}
|
||||
params[op.cliHints.stdin] = stdinContent;
|
||||
params[op.cliHints.stdin] = content;
|
||||
}
|
||||
}
|
||||
|
||||
return params;
|
||||
/** First-byte deadline for pipe/socket stdin (#3513). Env-overridable escape hatch. */
|
||||
function stdinFirstByteTimeoutMs(): number {
|
||||
const n = Number(process.env.GBRAIN_STDIN_TIMEOUT_MS);
|
||||
return Number.isFinite(n) && n > 0 ? n : 5000;
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns the full stdin content, '' for a readable-but-empty stdin, or
|
||||
* null when stdin is a pipe/socket that never delivered a byte within the
|
||||
* first-byte deadline (or the fd is closed/unreadable).
|
||||
*/
|
||||
export async function readStdinBounded(): Promise<string | null> {
|
||||
let isPipeOrSocket: boolean;
|
||||
try {
|
||||
const st = fstatSync(0);
|
||||
isPipeOrSocket = st.isFIFO() || st.isSocket();
|
||||
} catch {
|
||||
return null; // closed/invalid fd — treat as no input
|
||||
}
|
||||
if (!isPipeOrSocket) {
|
||||
// Regular file redirect, /dev/null, etc. — read returns without blocking.
|
||||
try {
|
||||
return readFileSync(0, 'utf-8');
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
return await new Promise<string | null>((resolve) => {
|
||||
const chunks: Buffer[] = [];
|
||||
let gotData = false;
|
||||
const timer = setTimeout(() => {
|
||||
if (!gotData) {
|
||||
process.stdin.destroy();
|
||||
resolve(null);
|
||||
}
|
||||
}, stdinFirstByteTimeoutMs());
|
||||
const finish = () => {
|
||||
clearTimeout(timer);
|
||||
resolve(Buffer.concat(chunks).toString('utf-8'));
|
||||
};
|
||||
process.stdin.on('data', (c: Buffer) => {
|
||||
if (!gotData) {
|
||||
gotData = true;
|
||||
clearTimeout(timer); // deadline applies to the FIRST byte only
|
||||
}
|
||||
chunks.push(c);
|
||||
});
|
||||
process.stdin.once('end', finish);
|
||||
process.stdin.once('error', finish);
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -868,7 +963,8 @@ export function applyThinClientSourceScope(
|
||||
params.source_id = resolved;
|
||||
}
|
||||
|
||||
async function makeContext(engine: BrainEngine, params: Record<string, unknown>): Promise<OperationContext> {
|
||||
// Exported for tests (same import-safety contract as applyThinClientSourceScope).
|
||||
export async function makeContext(engine: BrainEngine, params: Record<string, unknown>): Promise<OperationContext> {
|
||||
// v0.31.8 (D11): resolve sourceId via the canonical 6-tier chain. Honors
|
||||
// --source / GBRAIN_SOURCE / .gbrain-source / path-match / brain default /
|
||||
// 'default'. Wrapped in try/catch so a doctor / single-source brain that
|
||||
@@ -880,16 +976,21 @@ async function makeContext(engine: BrainEngine, params: Record<string, unknown>)
|
||||
// trusted local boundary) and consumed by federatedSearchScope in
|
||||
// operations.ts, which additionally gates on ctx.remote === false.
|
||||
let localFederated: string[] | undefined;
|
||||
// params.source is set when a CLI flag was parsed for the op (rare; most
|
||||
// CLI ops don't take --source). Falls through to env/dotfile/path-match.
|
||||
const explicit = (params.source as string | undefined) ?? null;
|
||||
try {
|
||||
const { resolveSourceWithTier, localFederatedSourceIds } = await import('./core/source-resolver.ts');
|
||||
// params.source is set when a CLI flag was parsed for the op (rare; most
|
||||
// CLI ops don't take --source). Falls through to env/dotfile/path-match.
|
||||
const explicit = (params.source as string | undefined) ?? null;
|
||||
const resolved = await resolveSourceWithTier(engine, explicit);
|
||||
sourceId = resolved.source_id;
|
||||
localFederated = await localFederatedSourceIds(engine, resolved.source_id, resolved.tier);
|
||||
} catch {
|
||||
// Source resolution failed (e.g. sources table doesn't exist on a fresh
|
||||
} catch (err) {
|
||||
// #1712: an EXPLICIT --source that fails to resolve (invalid id, or a
|
||||
// source that doesn't exist) must error loudly — the blanket swallow
|
||||
// turned `--source __all__` and typos into a silent `default` scope,
|
||||
// which is how three bug reports became debugging sessions.
|
||||
if (explicit) throw err;
|
||||
// Ambient resolution failed (e.g. sources table doesn't exist on a fresh
|
||||
// pre-init brain). Leave sourceId unset; engine read methods fall through
|
||||
// to the cross-source view (D16 back-compat path).
|
||||
sourceId = undefined;
|
||||
@@ -1055,7 +1156,7 @@ export function formatResult(opName: string, result: unknown): string {
|
||||
* `runRemoteDoctor` for thin-client installs.
|
||||
*/
|
||||
const THIN_CLIENT_REFUSED_COMMANDS = new Set([
|
||||
'sync', 'embed', 'extract', 'extract-conversation-facts', 'enrich', 'migrate', 'apply-migrations',
|
||||
'sync', 'embed', 'extract', 'extract-conversation-facts', 'enrich', 'migrate', 'retrieval-upgrade', 'apply-migrations',
|
||||
'repair-jsonb', 'orphans', 'integrity', 'serve',
|
||||
// v0.43 (#2095): watch streams against a LOCAL engine; thin clients get
|
||||
// the volunteer_context MCP op instead.
|
||||
@@ -1102,6 +1203,7 @@ const THIN_CLIENT_REFUSE_HINTS: Record<string, string> = {
|
||||
'extract-conversation-facts': 'extract-conversation-facts runs on the host (requires local engine + chat gateway). Run on the host machine.',
|
||||
enrich: 'enrich runs on the host (requires local engine + chat gateway for grounded synthesis). Run on the host machine.',
|
||||
migrate: "migrate runs on the host's local engine. Run on the host machine.",
|
||||
'retrieval-upgrade': "retrieval-upgrade (embedding migration) rebuilds the host brain's schema + re-embeds. Run on the host machine.",
|
||||
'apply-migrations': 'schema migrations run on the host. SSH and run there.',
|
||||
'repair-jsonb': 'repair-jsonb operates on the local DB only.',
|
||||
integrity: 'integrity scans local files. Run on the host machine.',
|
||||
@@ -1168,6 +1270,20 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
await runInit(args);
|
||||
return;
|
||||
}
|
||||
if (command === 'bench') {
|
||||
// #3502 sweep: `gbrain bench publish` was documented (docs/eval-bench.md,
|
||||
// KEY_FILES.md, and eval-gate's own --help text) but never dispatched —
|
||||
// the promised-but-unwired class retrieval-upgrade (#3390) fixed before.
|
||||
// Pure file-in/file-out (NDJSON → baseline); no DB, no engine.
|
||||
if (args[0] === 'publish') {
|
||||
const { runBenchPublish } = await import('./commands/bench-publish.ts');
|
||||
await runBenchPublish(args.slice(1));
|
||||
return;
|
||||
}
|
||||
console.error('Usage: gbrain bench publish --from <captured.ndjson> --to <X.baseline.ndjson> [flags]');
|
||||
console.error('Run `gbrain bench publish --help` for the full flag list.');
|
||||
process.exit(args[0] === '--help' || args[0] === '-h' ? 0 : 2);
|
||||
}
|
||||
// v0.37 fix wave (deferred TODO, shipped): one-command wipe-and-reinit.
|
||||
// Spawns its own engine internally so no pre-bound engine needed.
|
||||
if (command === 'reinit-pglite') {
|
||||
@@ -1752,10 +1868,33 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
}
|
||||
// doctor is handled before connectEngine() above
|
||||
case 'migrate': {
|
||||
// #3390: `gbrain migrate embeddings --to <provider:model>` — the
|
||||
// provider-agnostic embedding migration. Everything else stays the
|
||||
// engine-transfer path (`migrate --to <supabase|pglite>`).
|
||||
if (args[0] === 'embeddings') {
|
||||
const { runMigrateEmbeddings } = await import('./commands/migrate-embeddings.ts');
|
||||
await runMigrateEmbeddings(engine, args.slice(1));
|
||||
break;
|
||||
}
|
||||
if (args.includes('--help') || args.includes('-h')) {
|
||||
console.log('Usage: gbrain migrate --to <supabase|pglite> [--url <url>] [--path <path>] [--force]');
|
||||
console.log(' gbrain migrate embeddings --to <provider:model> [--dim N] [--dry-run] [--yes]');
|
||||
console.log('');
|
||||
console.log('The first form transfers the brain between engines; the second re-embeds');
|
||||
console.log('onto a different embedding provider (run `gbrain migrate embeddings --help`).');
|
||||
break;
|
||||
}
|
||||
const { runMigrateEngine } = await import('./commands/migrate-engine.ts');
|
||||
await runMigrateEngine(engine, args);
|
||||
break;
|
||||
}
|
||||
case 'retrieval-upgrade': {
|
||||
// The command README.md + doctor.ts promised since v0.36 but never
|
||||
// dispatched. Alias for `migrate embeddings` (#3390).
|
||||
const { runMigrateEmbeddings } = await import('./commands/migrate-embeddings.ts');
|
||||
await runMigrateEmbeddings(engine, args);
|
||||
break;
|
||||
}
|
||||
case 'eval': {
|
||||
// v0.32 EXP-5: `eval takes-quality {run,trend,regress}` requires a
|
||||
// brain (samples takes from DB / reads runs table). `replay` was
|
||||
@@ -2352,6 +2491,7 @@ USAGE
|
||||
SETUP
|
||||
init [--pglite|--supabase|--url] Create brain (PGLite default, no server)
|
||||
migrate --to <supabase|pglite> Transfer brain between engines
|
||||
migrate embeddings --to <p:model> Re-embed onto another embedding provider
|
||||
upgrade Self-update
|
||||
check-update [--json] Check for new versions
|
||||
doctor [--json] [--fast] Health check (resolver, skills, pgvector, RLS, embeddings)
|
||||
@@ -2373,6 +2513,8 @@ IMPORT/EXPORT
|
||||
sync [--repo <path>] [flags] Git-to-brain incremental sync
|
||||
sync --watch [--interval N] Continuous sync (loops until stopped)
|
||||
See also: autopilot --install (continuous daemon).
|
||||
sync --all --missing-path skip Classify sources whose local_path is absent
|
||||
on this machine as skipped, not failed
|
||||
export [--dir ./out/] Export to markdown
|
||||
export --restore-only [--repo <p>] Restore missing supabase-only files
|
||||
[--type T] [--slug-prefix S] With optional filters
|
||||
@@ -2417,6 +2559,7 @@ TOOLS
|
||||
publish <page.md> [--password] Shareable HTML (strips private data, optional AES-256)
|
||||
check-backlinks <check|fix> [dir] Find/fix missing back-links across brain
|
||||
lint <dir|file> [--fix] Catch LLM artifacts, placeholder dates, bad frontmatter
|
||||
backfill <kind|list> v0.30.1: run a registered backfill (effective-date, ...)
|
||||
orphans [--json] [--count] Find pages with no inbound wikilinks
|
||||
salience [--days N] [--kind P] v0.29: pages ranked by emotional + activity salience
|
||||
anomalies [--since D] [--sigma N] v0.29: cohort-based statistical anomalies (tag, type)
|
||||
|
||||
+19
-4
@@ -66,7 +66,9 @@ USAGE
|
||||
SUBMITTING
|
||||
gbrain agent run <prompt>
|
||||
--subagent-def <name> Named plugin subagent (from GBRAIN_PLUGIN_PATH)
|
||||
--model <id> Anthropic model id (defaults to sonnet)
|
||||
--model <id> Model id as provider:model (default: subagent tier model,
|
||||
anthropic:claude-sonnet-4-6). Non-Anthropic providers need
|
||||
agent.use_gateway_loop enabled — see NOTES below.
|
||||
--max-turns <n> Max assistant turns (default 20)
|
||||
--tools a,b,c Subset of registered tool names (comma list)
|
||||
--timeout-ms <n> Per-job wall-clock timeout
|
||||
@@ -87,9 +89,22 @@ VIEWING
|
||||
--since <spec> ISO-8601 timestamp OR relative ("5m","1h","2d")
|
||||
|
||||
NOTES
|
||||
Submitting subagent jobs is trusted-only; MCP submitters receive
|
||||
permission_denied. The worker needs ANTHROPIC_API_KEY set, or the
|
||||
first LLM turn of a claimed job fails.
|
||||
This CLI path is trusted-only. (Remote MCP callers reach subagents through
|
||||
the scoped submit_agent operation, not through this command.)
|
||||
|
||||
By default the worker runs the legacy Anthropic-direct path, which needs an
|
||||
Anthropic key — from ANTHROPIC_API_KEY or from anthropic_api_key in
|
||||
~/.gbrain/config.json — or the first LLM turn of a claimed job fails.
|
||||
|
||||
To run --model on a non-Anthropic provider, enable the provider-neutral
|
||||
gateway loop first, then supply whatever credential that provider needs
|
||||
(an API key for most; some recipes use OAuth or a local endpoint):
|
||||
gbrain config set agent.use_gateway_loop true
|
||||
Accepted values: true / 1 / yes / on.
|
||||
|
||||
The gateway loop needs a provider whose recipe supports chat WITH tool
|
||||
calling — not every recipe under src/core/ai/recipes/ qualifies. A model
|
||||
that cannot call tools is refused at job start with the reason named.
|
||||
`);
|
||||
}
|
||||
|
||||
|
||||
@@ -19,7 +19,7 @@
|
||||
|
||||
import { existsSync, readFileSync, writeFileSync, mkdirSync, appendFileSync, utimesSync, unlinkSync, chmodSync } from 'fs';
|
||||
import { setCliExitVerdict } from '../core/cli-force-exit.ts';
|
||||
import { join } from 'path';
|
||||
import { join, dirname } from 'path';
|
||||
import { execSync } from 'child_process';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { loadPreferences } from '../core/preferences.ts';
|
||||
@@ -1312,6 +1312,17 @@ function writeWrapperScript(repoPath: string): string {
|
||||
const gbrainPath = resolveGbrainCliPath();
|
||||
const safeRepoPath = repoPath.replace(/'/g, "'\\''");
|
||||
const safeGbrainPath = gbrainPath.replace(/'/g, "'\\''");
|
||||
// Bake the dir of the bun runtime actually executing this install onto PATH,
|
||||
// so the wrapper finds bun wherever it lives — Homebrew (/opt/homebrew/bin),
|
||||
// npm -g, Docker (/usr/local/bin), a custom BUN_INSTALL, or nix — not just
|
||||
// ~/.bun/bin (which #3305 hardcoded, covering only the default bun.sh installer).
|
||||
// dirname('') === '.', so guard the degenerate/empty case — otherwise a missing
|
||||
// execPath would prepend '.' (cwd) onto a cron PATH. Empty prefix falls back to
|
||||
// the #3305 behavior exactly.
|
||||
const runtimeDir = dirname(process.execPath || '');
|
||||
const runtimePathPrefix = runtimeDir && runtimeDir !== '.'
|
||||
? `'${runtimeDir.replace(/'/g, "'\\''")}':`
|
||||
: '';
|
||||
const wrapper = `#!/bin/bash
|
||||
# Auto-generated by gbrain autopilot --install
|
||||
# Sources shell profile for API keys, then runs autopilot.
|
||||
@@ -1326,10 +1337,11 @@ source ~/.zshrc 2>/dev/null || source ~/.bashrc 2>/dev/null || true
|
||||
# cron/systemd/launchd — so its PATH exports never reach this subprocess.
|
||||
# Without bun on PATH, the exec'd gbrain (a \`#!/usr/bin/env bun\` script) fails
|
||||
# silently with "env: bun: No such file or directory" and leaves a stale
|
||||
# lockfile that blocks every subsequent tick. Prepending ~/.bun/bin here
|
||||
# keeps the wrapper self-contained regardless of which init file the OS
|
||||
# loaded.
|
||||
export PATH="$HOME/.bun/bin:$PATH"
|
||||
# lockfile that blocks every subsequent tick. Prepending the running bun's own
|
||||
# dir (derived from process.execPath at install time), with ~/.bun/bin kept as a
|
||||
# fallback, keeps the wrapper self-contained regardless of where bun is installed
|
||||
# or which init file the OS loaded.
|
||||
export PATH=${runtimePathPrefix}"$HOME/.bun/bin:$PATH"
|
||||
exec '${safeGbrainPath}' autopilot --repo '${safeRepoPath}'
|
||||
`;
|
||||
writeFileSync(wrapperPath, wrapper, { mode: 0o755 });
|
||||
@@ -1756,7 +1768,10 @@ function showStatus(json: boolean) {
|
||||
} else {
|
||||
try {
|
||||
const crontab = execSync('crontab -l 2>/dev/null || true', { encoding: 'utf-8' });
|
||||
installed = crontab.includes('gbrain autopilot');
|
||||
// The installed cron line invokes the generated wrapper (…/autopilot-run.sh);
|
||||
// older installs called `gbrain autopilot` directly. Match either so status
|
||||
// isn't a false negative after the wrapper indirection landed.
|
||||
installed = crontab.includes('autopilot-run.sh') || crontab.includes('gbrain autopilot');
|
||||
} catch { /* no crontab */ }
|
||||
}
|
||||
|
||||
|
||||
@@ -2,12 +2,13 @@ import { VERSION } from '../version.ts';
|
||||
import { detectInstallMethod } from './upgrade.ts';
|
||||
import {
|
||||
isMinorOrMajorBump,
|
||||
isNewerVersion,
|
||||
isValidVersionString,
|
||||
parseSemver,
|
||||
semverGt,
|
||||
semverLte,
|
||||
} from '../core/semver.ts';
|
||||
import { writeUpdateCache, type UpdateMarker } from '../core/self-upgrade.ts';
|
||||
import { readUpdateCache, writeUpdateCache, type UpdateMarker } from '../core/self-upgrade.ts';
|
||||
|
||||
/** Best-effort cache write — a read-only ~/.gbrain must never make the check throw. */
|
||||
function safeWriteCache(marker: UpdateMarker): void {
|
||||
@@ -21,7 +22,7 @@ function safeWriteCache(marker: UpdateMarker): void {
|
||||
// Back-compat re-exports: these used to live here; moved to ../core/semver.ts
|
||||
// so the self-upgrade decision module can depend on them without an import
|
||||
// cycle. Existing importers (`test/check-update.test.ts`, etc.) keep working.
|
||||
export { parseSemver, isMinorOrMajorBump };
|
||||
export { parseSemver, isMinorOrMajorBump, isNewerVersion };
|
||||
|
||||
interface CheckUpdateResult {
|
||||
current_version: string;
|
||||
@@ -44,26 +45,53 @@ function upgradeCommandForMethod(method: string): string {
|
||||
}
|
||||
}
|
||||
|
||||
/** Where the latest version is resolved from. gbrain publishes NO GitHub
|
||||
* releases (the `releases/latest` API is a permanent 404), so the release
|
||||
* train's source of truth is the `VERSION` file on master — same trusted host
|
||||
* `fetchChangelog` already uses. An npm fallback was rejected: the `gbrain`
|
||||
* package on npm is an unrelated GPU library (#505), so it would produce false
|
||||
* upgrade prompts pointing at a stranger's package. */
|
||||
const VERSION_SOURCE_URL = 'https://raw.githubusercontent.com/garrytan/gbrain/master/VERSION';
|
||||
const RELEASE_NOTES_URL = 'https://github.com/garrytan/gbrain/blob/master/CHANGELOG.md';
|
||||
|
||||
/** Extract a version from the raw VERSION file body: first line, optional `v`
|
||||
* prefix, optional `-suffix` channel tag (`0.31.1.1-fixwave` compares as its
|
||||
* numeric base — fail-safe: a suffix-only bump never prompts). Body is bounded
|
||||
* before parsing so a malformed/huge response can't blow up the check. */
|
||||
export function parseVersionFileBody(body: string): string | null {
|
||||
const firstLine = body.slice(0, 256).trim().split('\n')[0].trim();
|
||||
const m = firstLine.match(/^v?(\d+\.\d+\.\d+(?:\.\d+)?)(?:[-+][0-9A-Za-z.-]+)?$/);
|
||||
return m && isValidVersionString(m[1]) ? m[1] : null;
|
||||
}
|
||||
|
||||
export type LatestReleaseResult =
|
||||
| { ok: true; tag: string; published_at: string; url: string }
|
||||
| { ok: false; reason: 'network_error' | 'no_releases' };
|
||||
|
||||
/**
|
||||
* Fetch the latest GitHub release. Exported (v0.42) so the self-upgrade refresh
|
||||
* path and tests can reuse it. 5s timeout (was 10s) — this runs on the detached
|
||||
* refresh, never the hot path, but a tight bound keeps the refresh cheap.
|
||||
* Resolve the latest published gbrain version (from VERSION on master — see
|
||||
* VERSION_SOURCE_URL). Exported (v0.42) so the self-upgrade refresh path and
|
||||
* tests can reuse it. 5s timeout — this runs on the detached refresh, never the
|
||||
* hot path. Failures are discriminated: `network_error` (offline/timeout) vs
|
||||
* `no_releases` (endpoint answered but no usable version).
|
||||
*/
|
||||
export async function fetchLatestRelease(): Promise<{ tag: string; published_at: string; url: string } | null> {
|
||||
export async function fetchLatestRelease(): Promise<LatestReleaseResult> {
|
||||
let res: Response;
|
||||
try {
|
||||
const res = await fetch('https://api.github.com/repos/garrytan/gbrain/releases/latest', {
|
||||
res = await fetch(VERSION_SOURCE_URL, {
|
||||
headers: { 'User-Agent': `gbrain/${VERSION}` },
|
||||
signal: AbortSignal.timeout(5_000),
|
||||
});
|
||||
if (!res.ok) return null;
|
||||
const data = await res.json() as any;
|
||||
return {
|
||||
tag: data.tag_name || '',
|
||||
published_at: data.published_at || '',
|
||||
url: data.html_url || '',
|
||||
};
|
||||
} catch {
|
||||
return null;
|
||||
return { ok: false, reason: 'network_error' };
|
||||
}
|
||||
try {
|
||||
if (!res.ok) return { ok: false, reason: 'no_releases' };
|
||||
const tag = parseVersionFileBody(await res.text());
|
||||
if (!tag) return { ok: false, reason: 'no_releases' };
|
||||
return { ok: true, tag, published_at: '', url: RELEASE_NOTES_URL };
|
||||
} catch {
|
||||
return { ok: false, reason: 'network_error' };
|
||||
}
|
||||
}
|
||||
|
||||
@@ -117,21 +145,37 @@ export function extractChangelogBetween(changelog: string, from: string, to: str
|
||||
}
|
||||
|
||||
/**
|
||||
* Fetch the latest release and write the self-upgrade cache (the marker line
|
||||
* read by the CLI startup hook). Fail-open: on any network failure we cache
|
||||
* `UP_TO_DATE <current>` so the TTL prevents hammering GitHub on every
|
||||
* invocation. Returns the resolved marker for callers that want it. This is the
|
||||
* function the detached single-flight refresh (`gbrain check-update
|
||||
* --refresh-cache`) invokes.
|
||||
* A failed check must NEVER write `up_to_date` — that was #486: the fetch
|
||||
* failed permanently (dead releases API) and every user was told "you're
|
||||
* current" forever. Instead, re-write the last-known-good marker (bumping its
|
||||
* mtime so the cache TTL still throttles retries and a network blip can't
|
||||
* erase a pending upgrade_available notice). No prior marker → write nothing;
|
||||
* the next invocation retries.
|
||||
*/
|
||||
function preserveCacheOnFailedCheck(): void {
|
||||
try {
|
||||
const prior = readUpdateCache();
|
||||
if (prior) safeWriteCache(prior.marker);
|
||||
} catch {
|
||||
/* best-effort */
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Fetch the latest version and write the self-upgrade cache (the marker line
|
||||
* read by the CLI startup hook). On fetch failure the last-known-good marker is
|
||||
* preserved (see preserveCacheOnFailedCheck) — never a fabricated `up_to_date`.
|
||||
* This is the function the detached single-flight refresh (`gbrain
|
||||
* check-update --refresh-cache`) invokes.
|
||||
*/
|
||||
export async function refreshUpdateCache(): Promise<void> {
|
||||
const release = await fetchLatestRelease();
|
||||
if (!release) {
|
||||
safeWriteCache({ kind: 'up_to_date', current: VERSION });
|
||||
if (!release.ok) {
|
||||
preserveCacheOnFailedCheck();
|
||||
return;
|
||||
}
|
||||
const latestVersion = release.tag.replace(/^v/, '');
|
||||
if (!isValidVersionString(latestVersion) || !isMinorOrMajorBump(VERSION, latestVersion)) {
|
||||
if (!isValidVersionString(latestVersion) || !isNewerVersion(VERSION, latestVersion)) {
|
||||
safeWriteCache({ kind: 'up_to_date', current: VERSION });
|
||||
return;
|
||||
}
|
||||
@@ -140,7 +184,7 @@ export async function refreshUpdateCache(): Promise<void> {
|
||||
|
||||
export async function runCheckUpdate(args: string[]) {
|
||||
if (args.includes('--help') || args.includes('-h')) {
|
||||
console.log('Usage: gbrain check-update [--json] [--refresh-cache]\n\nCheck for new GBrain versions.\n\nOnly reports minor/major version bumps (v0.X.0), not patches.\nFails silently on network errors.\n\n--refresh-cache Fetch + update the self-upgrade cache, print nothing (used by\n the CLI startup hook\'s detached refresh).');
|
||||
console.log('Usage: gbrain check-update [--json] [--refresh-cache]\n\nCheck for new GBrain versions.\n\nReports any strictly newer release, including patch and micro updates.\nFails silently on network errors.\n\n--refresh-cache Fetch + update the self-upgrade cache, print nothing (used by\n the CLI startup hook\'s detached refresh).');
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -165,9 +209,8 @@ export async function runCheckUpdate(args: string[]) {
|
||||
|
||||
const release = await fetchLatestRelease();
|
||||
|
||||
if (!release) {
|
||||
// Warm the cache fail-open so the startup hook doesn't re-fetch every call.
|
||||
safeWriteCache({ kind: 'up_to_date', current: VERSION });
|
||||
if (!release.ok) {
|
||||
preserveCacheOnFailedCheck();
|
||||
if (json) {
|
||||
console.log(JSON.stringify({
|
||||
current_version: VERSION,
|
||||
@@ -178,16 +221,18 @@ export async function runCheckUpdate(args: string[]) {
|
||||
release_url: '',
|
||||
changelog_diff: '',
|
||||
published_at: '',
|
||||
error: 'no_releases',
|
||||
error: release.reason,
|
||||
}, null, 2));
|
||||
} else if (release.reason === 'network_error') {
|
||||
console.log(`GBrain ${VERSION} — could not check for updates (network unavailable).`);
|
||||
} else {
|
||||
console.log(`GBrain ${VERSION} — could not check for updates (no releases found or network unavailable).`);
|
||||
console.log(`GBrain ${VERSION} — could not determine the latest published version.`);
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
const latestVersion = release.tag.replace(/^v/, '');
|
||||
const updateAvailable = isValidVersionString(latestVersion) && isMinorOrMajorBump(VERSION, latestVersion);
|
||||
const updateAvailable = isValidVersionString(latestVersion) && isNewerVersion(VERSION, latestVersion);
|
||||
|
||||
// Warm the self-upgrade cache so the next `gbrain <cmd>` startup hook can emit
|
||||
// the marker without a network call.
|
||||
|
||||
+125
-27
@@ -1,4 +1,5 @@
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { REPAIR_SOURCE_CONFIG_SQL } from '../core/source-config-sql.ts';
|
||||
import { setCliExitVerdict } from '../core/cli-force-exit.ts';
|
||||
import * as db from '../core/db.ts';
|
||||
import { LATEST_VERSION, getIdleBlockers } from '../core/migrate.ts';
|
||||
@@ -39,7 +40,7 @@ import {
|
||||
buildBasenameIndex,
|
||||
queryBasenameIndex,
|
||||
} from '../core/link-extraction.ts';
|
||||
import { isSourceUnchangedSinceSync } from '../core/git-head.ts';
|
||||
import { probeSourceGitState } from '../core/git-head.ts';
|
||||
// v0.41.32.0: remote staleness reads the stored newest_content_at column via
|
||||
// this pure comparator (no git subprocess on the HTTP MCP doctor path).
|
||||
import { lagFromContentMs } from '../core/source-health.ts';
|
||||
@@ -52,6 +53,8 @@ import { isUndefinedColumnError } from '../core/utils.ts';
|
||||
// drift from what search actually filters.
|
||||
import { resolveHardExcludes, DEFAULT_HARD_EXCLUDES } from '../core/search/source-boost.ts';
|
||||
import { escapeLikePattern, buildVisibilityClause } from '../core/search/sql-ranking.ts';
|
||||
import { unverifiedExtractionFragment } from '../core/extraction-review.ts';
|
||||
import { hnswIndexExpected, hnswMaxDimsForType } from '../core/vector-index.ts';
|
||||
|
||||
export interface Check {
|
||||
name: string;
|
||||
@@ -615,10 +618,8 @@ export async function checkSourceConfigShape(engine: BrainEngine): Promise<Check
|
||||
`${rows.length} source(s) have a non-object config — a JSON string/scalar ` +
|
||||
`instead of an object (the #2829 re-wrapping bug): ${affected}. ` +
|
||||
`Federation and ACL settings on these sources won't be read correctly. ` +
|
||||
`Repair by running any 'gbrain sources' config write (self-heals up to 10 ` +
|
||||
`nested layers), or in SQL: ` +
|
||||
`UPDATE sources SET config = (config #>> '{}')::jsonb ` +
|
||||
`WHERE jsonb_typeof(config) <> 'object';`,
|
||||
`Repair by running any 'gbrain sources' config write (self-heals nested ` +
|
||||
`strings and recoverable arrays), or in SQL: ${REPAIR_SOURCE_CONFIG_SQL}`,
|
||||
};
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
@@ -878,8 +879,8 @@ export async function doctorReportRemote(engine: BrainEngine): Promise<DoctorRep
|
||||
checks.push(await checkEmbeddingEnvOverride(engine));
|
||||
|
||||
// v0.31.12 subagent runtime enforcement (Layer 3 of 3 — Codex F13).
|
||||
// The subagent loop is Anthropic-only. If models.tier.subagent or
|
||||
// models.default is explicitly set to a non-Anthropic provider, warn here
|
||||
// The subagent loop requires native tool-calling. If models.subagent,
|
||||
// models.tier.subagent, or models.default resolves to a limited provider, warn here
|
||||
// so the user sees it at the next `gbrain doctor` run instead of at the
|
||||
// next subagent job submission. (Layers 1+2 also enforce — this is the
|
||||
// surfacing layer.)
|
||||
@@ -3053,6 +3054,7 @@ async function checkEmbeddingEnvOverride(engine: BrainEngine): Promise<Check> {
|
||||
export async function checkSubagentCapability(engine: BrainEngine): Promise<Check> {
|
||||
try {
|
||||
const { classifyCapabilities } = await import('../core/ai/capabilities.ts');
|
||||
const modelsSubagent = await engine.getConfig('models.subagent');
|
||||
const tierSubagent = await engine.getConfig('models.tier.subagent');
|
||||
const modelsDefault = await engine.getConfig('models.default');
|
||||
|
||||
@@ -3093,12 +3095,23 @@ export async function checkSubagentCapability(engine: BrainEngine): Promise<Chec
|
||||
return null;
|
||||
};
|
||||
|
||||
if (tierSubagent) {
|
||||
const issue = explain(tierSubagent, 'models.tier.subagent');
|
||||
let resolvedSource: string | null = null;
|
||||
let resolvedModel: string | null = null;
|
||||
if (modelsSubagent) {
|
||||
resolvedSource = 'models.subagent';
|
||||
resolvedModel = modelsSubagent;
|
||||
const issue = explain(modelsSubagent, resolvedSource);
|
||||
if (issue) return issue;
|
||||
} else if (modelsDefault) {
|
||||
resolvedSource = 'models.default';
|
||||
resolvedModel = modelsDefault;
|
||||
const issue = explain(modelsDefault, 'models.default');
|
||||
if (issue) return issue;
|
||||
} else if (tierSubagent) {
|
||||
resolvedSource = 'models.tier.subagent';
|
||||
resolvedModel = tierSubagent;
|
||||
const issue = explain(tierSubagent, resolvedSource);
|
||||
if (issue) return issue;
|
||||
}
|
||||
// v0.37 (T10 / D7) + v0.38 (D7 capability rename): warn when the configured
|
||||
// chat_model is non-Anthropic AND ANTHROPIC_API_KEY isn't set. With
|
||||
@@ -3111,9 +3124,9 @@ export async function checkSubagentCapability(engine: BrainEngine): Promise<Chec
|
||||
const { loadConfig } = await import('../core/config.ts');
|
||||
const cfg = loadConfig();
|
||||
const chatModel = cfg?.chat_model;
|
||||
const { isConfigTruthy } = await import('../core/config.ts');
|
||||
const gatewayLoopRaw = await engine.getConfig('agent.use_gateway_loop').catch(() => null);
|
||||
const gatewayLoopEnabled = typeof gatewayLoopRaw === 'string'
|
||||
&& ['true', '1', 'yes', 'on'].includes(gatewayLoopRaw.trim().toLowerCase());
|
||||
const gatewayLoopEnabled = isConfigTruthy(gatewayLoopRaw);
|
||||
const { isAnthropicProvider } = await import('../core/model-config.ts');
|
||||
if (chatModel && !isAnthropicProvider(chatModel) && !process.env.ANTHROPIC_API_KEY && !gatewayLoopEnabled) {
|
||||
return {
|
||||
@@ -3131,8 +3144,8 @@ export async function checkSubagentCapability(engine: BrainEngine): Promise<Chec
|
||||
return {
|
||||
name: 'subagent_capability',
|
||||
status: 'ok',
|
||||
message: tierSubagent
|
||||
? `Subagent tier resolves to "${tierSubagent}" with full tool-loop capability`
|
||||
message: resolvedModel && resolvedSource
|
||||
? `Subagent model resolves via ${resolvedSource} to "${resolvedModel}" with full tool-loop capability`
|
||||
: `Subagent tier resolves to default (claude-sonnet-4-6) — full tool-loop capability`,
|
||||
};
|
||||
} catch (e) {
|
||||
@@ -3608,6 +3621,52 @@ export async function checkLinksExtractionLag(
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* issue #160 — unverified_extractions doctor check.
|
||||
*
|
||||
* The extraction quarantine lane parks auto-extracted entity stubs
|
||||
* (frontmatter `provenance: 'auto-extracted'` + `status: 'unverified'`)
|
||||
* until the owner promotes or rejects them. A queue nobody reviews decays
|
||||
* into invisible clutter, so this check counts stubs older than N days
|
||||
* (default 7) and nudges toward the review surface. Exported for direct
|
||||
* testing (mirrors checkLinksExtractionLag).
|
||||
*/
|
||||
export async function checkUnverifiedExtractions(
|
||||
engine: BrainEngine,
|
||||
opts?: { sourceId?: string; days?: number },
|
||||
): Promise<Check> {
|
||||
const name = 'unverified_extractions';
|
||||
const days = opts?.days ?? 7;
|
||||
const sourceId = opts?.sourceId;
|
||||
try {
|
||||
const params: unknown[] = [String(days)];
|
||||
let srcClause = '';
|
||||
if (sourceId) {
|
||||
params.push(sourceId);
|
||||
srcClause = 'AND p.source_id = $2';
|
||||
}
|
||||
const rows = await engine.executeRaw<{ n: string | number }>(
|
||||
`SELECT COUNT(*)::int AS n FROM pages p
|
||||
WHERE p.deleted_at IS NULL
|
||||
AND ${unverifiedExtractionFragment('p')}
|
||||
AND p.created_at < now() - ($1 || ' days')::interval
|
||||
${srcClause}`,
|
||||
params,
|
||||
);
|
||||
const n = Number(rows[0]?.n ?? 0);
|
||||
return {
|
||||
name,
|
||||
status: n > 0 ? 'warn' : 'ok',
|
||||
message: n > 0
|
||||
? `${n} unverified auto-extracted entity stub(s) older than ${days} days awaiting review. List with 'gbrain extraction-pending'; promote/reject with 'gbrain extraction-review <promote|reject> --slugs <slug,...>'.`
|
||||
: 'No stale unverified extraction stubs',
|
||||
details: { count: n, days, source_id: sourceId ?? null },
|
||||
};
|
||||
} catch (e) {
|
||||
return { name, status: 'warn', message: `Could not check unverified_extractions: ${(e as Error).message}` };
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* issue #1678 — extract_atoms_backlog doctor check.
|
||||
*
|
||||
@@ -4003,29 +4062,51 @@ export async function checkSyncFreshness(
|
||||
// All four must hold; otherwise fall through to the time-based check.
|
||||
// The chunker version match is computed here (not in the helper)
|
||||
// because it depends on engine state, not git state.
|
||||
//
|
||||
// Clone-unavailable fallback: on stateless deploys (Docker on EB /
|
||||
// K8s / Fly — the platforms the cloud recipes produce), a container
|
||||
// restart wipes `local_path` and each clone is only re-materialized
|
||||
// when that source's next sync job runs. Until then the HEAD probe
|
||||
// cannot run at all ('unavailable'), which previously fell through to
|
||||
// raw wall-clock age — and since a no-op sync doesn't advance
|
||||
// `last_sync_at`, every QUIET source read as stale/FAIL after a
|
||||
// restart (score-sinking alert storm; observed live: 16-source brain,
|
||||
// 12 clones gone after a config-update restart, doctor 70→30).
|
||||
// 'unavailable' + chunker match now reuses the v0.41.32.0 REMOTE lag
|
||||
// signal (newest_content_at) below — DB-only, no subprocess, and it
|
||||
// still reports staleness whenever content really is newer than the
|
||||
// last sync. 'changed' (readable clone with real work) keeps
|
||||
// wall-clock exactly as before, and a chunker mismatch is never
|
||||
// masked (D7): it disables the fallback too.
|
||||
let cloneUnavailable = false;
|
||||
if (localOnly) {
|
||||
const gitUnchanged = isSourceUnchangedSinceSync(
|
||||
const gitState = probeSourceGitState(
|
||||
source.local_path,
|
||||
source.last_commit,
|
||||
{ requireCleanWorkingTree: 'ignore-untracked' },
|
||||
);
|
||||
const chunkerMatch = source.chunker_version === currentChunkerVersion;
|
||||
if (gitUnchanged && chunkerMatch) {
|
||||
if (gitState === 'unchanged' && chunkerMatch) {
|
||||
unchanged_count++;
|
||||
continue;
|
||||
}
|
||||
cloneUnavailable = gitState === 'unavailable' && chunkerMatch;
|
||||
}
|
||||
|
||||
// v0.41.32.0: REMOTE path (doctorReportRemote, !localOnly) computes lag
|
||||
// from the stored newest_content_at column — NO git subprocess on a
|
||||
// DB-supplied local_path (preserves the v0.41.27.0 trust boundary). A
|
||||
// quiet repo whose newest commit predates its last sync reports 0; NULL
|
||||
// column → wall-clock fallback. LOCAL fall-through keeps wall-clock: the
|
||||
// short-circuit already failed, so the source genuinely has work and
|
||||
// "hours since last sync" is the right staleness measure. The `ageMs < 0`
|
||||
// skew check above still runs on raw wall-clock for both paths (A1).
|
||||
// column → wall-clock fallback. LOCAL fall-through keeps wall-clock when
|
||||
// the clone is READABLE: the short-circuit failed on real evidence
|
||||
// (HEAD moved / dirty tree), so the source genuinely has work and
|
||||
// "hours since last sync" is the right staleness measure. A local clone
|
||||
// that is UNAVAILABLE (not yet re-materialized, see above) carries no
|
||||
// evidence either way, so it borrows this same DB-only lag. The
|
||||
// `ageMs < 0` skew check above still runs on raw wall-clock for both
|
||||
// paths (A1).
|
||||
let thresholdAgeMs = ageMs;
|
||||
if (!localOnly) {
|
||||
if (!localOnly || cloneUnavailable) {
|
||||
const contentMs = source.newest_content_at
|
||||
? new Date(source.newest_content_at).getTime()
|
||||
: null;
|
||||
@@ -4643,11 +4724,10 @@ export async function buildChecks(
|
||||
const checks: Check[] = [];
|
||||
let autoFixReport: AutoFixReport | null = null;
|
||||
|
||||
// Progress reporter. `--json` is doctor's own JSON output (list of checks);
|
||||
// progress events stay on stderr regardless, gated by the global --quiet /
|
||||
// --progress-json flags. On a 52K-page brain the DB checks can take minutes,
|
||||
// and without a heartbeat agents can't tell doctor from a hang.
|
||||
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
|
||||
// Progress reporter. `--json` is doctor's machine-readable output, so plain
|
||||
// progress must not leak to stderr unless the caller explicitly asks for
|
||||
// structured progress with --progress-json.
|
||||
const progress = createProgress(doctorProgressOptions(jsonOutput));
|
||||
|
||||
// --- Filesystem checks (always run, no DB needed) ---
|
||||
|
||||
@@ -5912,7 +5992,7 @@ export async function buildChecks(
|
||||
// that doesn't match the gateway's resolved default. Empty-brain vs
|
||||
// non-empty-brain branching determines the repair hint:
|
||||
// - empty brain (no embedded chunks) → `gbrain init --force --embedding-model …`
|
||||
// - non-empty brain → `gbrain retrieval-upgrade --to … --reindex`
|
||||
// - non-empty brain → `gbrain migrate embeddings --to … --dim …` (#3390)
|
||||
// The bug-reporter's `rm -rf ~/.gbrain` recovery is never the right answer.
|
||||
let surfacedUnconfiguredDrift = false;
|
||||
try {
|
||||
@@ -5943,7 +6023,7 @@ export async function buildChecks(
|
||||
if (totalChunks > 0) {
|
||||
const fix = embeddedCount === 0
|
||||
? `No embeddings yet — drop the empty schema and re-init at the right dim:\n gbrain init --force --pglite --embedding-model ${configuredModel} --embedding-dimensions ${configuredDims}`
|
||||
: `Non-empty brain (${embeddedCount} embedded chunks). Migrate cleanly:\n gbrain retrieval-upgrade --to ${configuredModel} --reindex`;
|
||||
: `Non-empty brain (${embeddedCount} embedded chunks). Migrate cleanly:\n gbrain migrate embeddings --to ${configuredModel} --dim ${configuredDims}`;
|
||||
|
||||
checks.push({
|
||||
name: 'embedding_provider',
|
||||
@@ -6147,6 +6227,12 @@ export async function buildChecks(
|
||||
continue;
|
||||
}
|
||||
if (engine.kind === 'postgres' && haveIndex.get(colName) === false) {
|
||||
if (!hnswIndexExpected(entry.type, entry.dimensions)) {
|
||||
okColumns.push(
|
||||
`${colName} (exact scan: ${entry.type}(${entry.dimensions}) exceeds HNSW cap ${hnswMaxDimsForType(entry.type)})`,
|
||||
);
|
||||
continue;
|
||||
}
|
||||
issues.push(
|
||||
`${colName}: no HNSW index. Search works but uses sequential scan. ` +
|
||||
`Fix: CREATE INDEX IF NOT EXISTS idx_chunks_${colName} ON content_chunks USING hnsw (${quoteIdentifier(colName)} ${entry.type}_cosine_ops);`,
|
||||
@@ -6874,6 +6960,10 @@ export async function buildChecks(
|
||||
checks.push({ name: 'flagged_pages', status: 'ok', message: `Skipped (${msg})` });
|
||||
}
|
||||
|
||||
// issue #160: extraction quarantine lane review nudge.
|
||||
progress.heartbeat('unverified_extractions');
|
||||
checks.push(await checkUnverifiedExtractions(engine, { sourceId: orphanRatioSourceId }));
|
||||
|
||||
// 11a. Frontmatter integrity (v0.22.4, hardened in v0.38.2.0).
|
||||
// scanBrainSources walks every registered source's local_path on disk
|
||||
// (not from the DB), invoking parseMarkdown(..., {validate:true}) per
|
||||
@@ -7670,6 +7760,14 @@ export async function runDoctor(
|
||||
// Helpers
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export function doctorProgressOptions(jsonOutput: boolean) {
|
||||
const cliOpts = getCliOptions();
|
||||
if (jsonOutput && !cliOpts.quiet && !cliOpts.progressJson) {
|
||||
return { mode: 'quiet' as const };
|
||||
}
|
||||
return cliOptsToProgressOptions(cliOpts);
|
||||
}
|
||||
|
||||
/** Print the auto-fix report in human-readable form. JSON output goes through
|
||||
* outputResults alongside the check list; this is the pretty-print path. */
|
||||
function printAutoFixReport(report: AutoFixReport, dryRun: boolean, jsonOutput: boolean): void {
|
||||
|
||||
+104
-7
@@ -19,6 +19,26 @@ import {
|
||||
} from '../core/pace-mode.ts';
|
||||
import { tryAcquireDbLock, type DbLockHandle } from '../core/db-lock.ts';
|
||||
import { embedBackfillLockId } from '../core/embed-backfill-lock.ts';
|
||||
import { wrapChunkTextsForStoredMode } from '../core/embedding-context.ts';
|
||||
import { titleTierCorpusGeneration } from '../core/contextual-retrieval-service.ts';
|
||||
import type { Page } from '../core/types.ts';
|
||||
|
||||
/**
|
||||
* #3507 — after a plain re-embed fully re-embedded a `per_chunk_synopsis`
|
||||
* page at the title-only tier (see wrapChunkTextsForStoredMode), restamp the
|
||||
* page's CR state to 'title' so `contextual_retrieval_mode` keeps describing
|
||||
* the vectors actually in the column. The reindex sweep restores the synopsis
|
||||
* tier later. No-op for every other mode.
|
||||
*/
|
||||
export async function restampIfDemotedToTitleTier(
|
||||
engine: BrainEngine,
|
||||
page: Pick<Page, 'contextual_retrieval_mode'> | null | undefined,
|
||||
slug: string,
|
||||
sourceId: string,
|
||||
): Promise<void> {
|
||||
if (page?.contextual_retrieval_mode !== 'per_chunk_synopsis') return;
|
||||
await engine.updatePageContextualRetrievalState(slug, sourceId, 'title', titleTierCorpusGeneration());
|
||||
}
|
||||
|
||||
export interface EmbedOpts {
|
||||
/** Embed ALL pages (every chunk). */
|
||||
@@ -115,6 +135,16 @@ export interface EmbedOpts {
|
||||
* Errors/warnings still go to stderr regardless.
|
||||
*/
|
||||
quiet?: boolean;
|
||||
/**
|
||||
* #3391: widen signature-drift invalidation to pages with NO recorded
|
||||
* embedding_signature (pre-v108). By default those are grandfathered
|
||||
* (never invalidated) so a routine upgrade doesn't surprise-re-embed a
|
||||
* whole corpus — but after a provider/model swap the grandfather clause
|
||||
* silently leaves them in the OLD embedding space, mixing two vector
|
||||
* spaces in one index. `gbrain migrate embeddings` and
|
||||
* `gbrain embed --stale --include-null-signature` set this.
|
||||
*/
|
||||
includeNullSignature?: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -356,6 +386,7 @@ export async function runEmbedCore(engine: BrainEngine, opts: EmbedOpts): Promis
|
||||
pacer,
|
||||
paceMaxConcurrency,
|
||||
quiet: opts.quiet,
|
||||
includeNullSignature: opts.includeNullSignature,
|
||||
}, opts.signal);
|
||||
} finally {
|
||||
// E1: surface pacing telemetry (human + structured) when pacing was on.
|
||||
@@ -469,6 +500,8 @@ export async function runEmbed(engine: BrainEngine, args: string[]): Promise<Emb
|
||||
const priorityRaw = priorityIdx >= 0 ? args[priorityIdx + 1] : undefined;
|
||||
const priority = priorityRaw === 'recent' ? 'recent' as const : undefined;
|
||||
const catchUp = args.includes('--catch-up');
|
||||
// #3391: re-embed pages that predate the embedding_signature stamp too.
|
||||
const includeNullSignature = args.includes('--include-null-signature');
|
||||
const pace = parsePaceArgs(args);
|
||||
|
||||
let opts: EmbedOpts;
|
||||
@@ -476,11 +509,11 @@ export async function runEmbed(engine: BrainEngine, args: string[]): Promise<Emb
|
||||
opts = { slugs: args.slice(slugsIdx + 1).filter(a => !a.startsWith('--')), dryRun, sourceId, batchSize, priority, catchUp };
|
||||
} else if (all || stale) {
|
||||
// E-2: CLI-only single-flight for stale runs (the minion path locks itself).
|
||||
opts = { all, stale, dryRun, sourceId, batchSize, priority, catchUp, ...(pace && { pace }), ...(stale && { singleFlight: true }) };
|
||||
opts = { all, stale, dryRun, sourceId, batchSize, priority, catchUp, ...(pace && { pace }), ...(stale && { singleFlight: true }), ...(includeNullSignature && { includeNullSignature: true }) };
|
||||
} else {
|
||||
const slug = args.find(a => !a.startsWith('--'));
|
||||
if (!slug) {
|
||||
serr('Usage: gbrain embed [<slug>|--all|--stale|--slugs s1 s2 ...] [--dry-run] [--batch-size N] [--priority recent] [--catch-up]');
|
||||
serr('Usage: gbrain embed [<slug>|--all|--stale|--slugs s1 s2 ...] [--dry-run] [--batch-size N] [--priority recent] [--catch-up] [--include-null-signature]');
|
||||
process.exit(1);
|
||||
}
|
||||
opts = { slug, dryRun, sourceId, batchSize, priority, catchUp };
|
||||
@@ -586,7 +619,11 @@ async function embedPage(
|
||||
return;
|
||||
}
|
||||
|
||||
const embeddings = await embedBatch(toEmbed.map(c => c.chunk_text), { abortSignal: signal });
|
||||
// #3507: embed with the page's STORED wrapping convention (title-tier
|
||||
// contextual prefix when the page was embedded wrapped), not raw
|
||||
// chunk_text — otherwise a re-embed silently strips the contextual
|
||||
// prefixes the sync path applied. fenced_code chunks stay unwrapped.
|
||||
const embeddings = await embedBatch(wrapChunkTextsForStoredMode(page, toEmbed), { abortSignal: signal });
|
||||
const embeddingMap = new Map<number, Float32Array>();
|
||||
for (let j = 0; j < toEmbed.length; j++) {
|
||||
embeddingMap.set(toEmbed[j].chunk_index, embeddings[j]);
|
||||
@@ -609,6 +646,9 @@ async function embedPage(
|
||||
// such a page and then stamps it.
|
||||
if (toEmbed.length === chunks.length) {
|
||||
await engine.setPageEmbeddingSignature(slug, { sourceId, signature: currentEmbeddingSignature() });
|
||||
// #3507: a fully re-embedded per_chunk_synopsis page landed at the
|
||||
// title tier — keep the stamped mode honest.
|
||||
await restampIfDemotedToTitleTier(engine, page, slug, page.source_id);
|
||||
}
|
||||
result.embedded += toEmbed.length;
|
||||
result.pages_processed++;
|
||||
@@ -657,6 +697,8 @@ async function embedAll(
|
||||
paceMaxConcurrency?: number;
|
||||
/** #394: suppress human stdout summaries (structured-output callers). */
|
||||
quiet?: boolean;
|
||||
/** #3391: lift the NULL-signature grandfather clause (see EmbedOpts). */
|
||||
includeNullSignature?: boolean;
|
||||
},
|
||||
signal?: AbortSignal,
|
||||
) {
|
||||
@@ -748,7 +790,8 @@ async function embedAll(
|
||||
}
|
||||
|
||||
try {
|
||||
const embeddings = await embedBatch(toEmbed.map(c => c.chunk_text));
|
||||
// #3507: reproduce the page's stored wrapping convention (see embedPage).
|
||||
const embeddings = await embedBatch(wrapChunkTextsForStoredMode(page, toEmbed));
|
||||
// Build a map of new embeddings by chunk_index
|
||||
const embeddingMap = new Map<number, Float32Array>();
|
||||
for (let j = 0; j < toEmbed.length; j++) {
|
||||
@@ -770,6 +813,11 @@ async function embedAll(
|
||||
await observed(pacer, () =>
|
||||
engine.setPageEmbeddingSignature(page.slug, { sourceId: pageSourceId, signature }),
|
||||
);
|
||||
// #3507: --all fully re-embeds; a per_chunk_synopsis page landed at
|
||||
// the title tier — keep the stamped mode honest.
|
||||
await observed(pacer, () =>
|
||||
restampIfDemotedToTitleTier(engine, page, page.slug, pageSourceId),
|
||||
);
|
||||
result.embedded += toEmbed.length;
|
||||
} catch (e: unknown) {
|
||||
serr(`\n Error embedding ${page.slug}: ${e instanceof Error ? e.message : e}`);
|
||||
@@ -845,6 +893,8 @@ async function embedAllStale(
|
||||
paceMaxConcurrency?: number;
|
||||
/** #394: suppress human stdout summaries (structured-output callers). */
|
||||
quiet?: boolean;
|
||||
/** #3391: lift the NULL-signature grandfather clause (see EmbedOpts). */
|
||||
includeNullSignature?: boolean;
|
||||
},
|
||||
signature?: string,
|
||||
externalSignal?: AbortSignal,
|
||||
@@ -852,6 +902,7 @@ async function embedAllStale(
|
||||
// D7: thread sourceId so source-scoped runs only count + visit
|
||||
// that source's NULL embeddings.
|
||||
const sourceOpt = sourceId ? { sourceId } : undefined;
|
||||
const includeNullSig = !!staleOpts?.includeNullSignature;
|
||||
|
||||
// v0.41.31: re-embed pages whose embedding_signature drifted (model/dims
|
||||
// swap). dry-run must NOT mutate, so it counts signature-stale via the
|
||||
@@ -861,16 +912,46 @@ async function embedAllStale(
|
||||
const invalidated = await engine.invalidateStaleSignatureEmbeddings({
|
||||
signature,
|
||||
...(sourceId && { sourceId }),
|
||||
...(includeNullSig && { includeNullSignature: true }),
|
||||
});
|
||||
if (invalidated > 0 && !staleOpts?.quiet) {
|
||||
slog(`[embed] invalidated ${invalidated} chunk(s) embedded under a prior model signature`);
|
||||
}
|
||||
// #3391: the grandfather clause keeps NULL-signature pages on their OLD
|
||||
// vectors — two embedding spaces mixed in one index. Loud stderr warning
|
||||
// with the fix, instead of silent retrieval degradation.
|
||||
//
|
||||
// Deliberately NOT gated on `invalidated > 0`: the original bug report's
|
||||
// shape is a brain where EVERY embedded page predates the signature stamp,
|
||||
// so nothing drifts, nothing is invalidated — and pre-fix that brain got
|
||||
// no warning AND no work, the exact silent case #3391 is about. The probe
|
||||
// below computes the left-behind count directly, which is 0 on a healthy
|
||||
// brain, so an unaffected run stays quiet.
|
||||
if (!includeNullSig) {
|
||||
try {
|
||||
const wide = await engine.countStaleChunks({ ...sourceOpt, signature, includeNullSignature: true });
|
||||
const narrow = await engine.countStaleChunks({ ...sourceOpt, signature });
|
||||
const leftBehind = wide - narrow;
|
||||
if (leftBehind > 0) {
|
||||
serr(
|
||||
` [embed] WARNING: ${leftBehind} embedded chunk(s) sit on pages with no recorded ` +
|
||||
`embedding signature and were NOT invalidated — they remain in the previous model's ` +
|
||||
`embedding space. Re-run with --include-null-signature (or use ` +
|
||||
`\`gbrain migrate embeddings\`) to re-embed them.`,
|
||||
);
|
||||
}
|
||||
} catch {
|
||||
// The warning probe is best-effort; never break the embed run.
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Pre-flight: 0 stale chunks → nothing to do, no further DB reads.
|
||||
// dry-run includes signature-drift in the count without mutating.
|
||||
const staleCount = await engine.countStaleChunks(
|
||||
dryRun && signature ? { ...sourceOpt, signature } : sourceOpt,
|
||||
dryRun && signature
|
||||
? { ...sourceOpt, signature, ...(includeNullSig && { includeNullSignature: true }) }
|
||||
: sourceOpt,
|
||||
);
|
||||
if (staleCount === 0) {
|
||||
if (!staleOpts?.quiet) {
|
||||
@@ -1050,7 +1131,13 @@ async function embedAllStale(
|
||||
const keySourceId = stale[0]?.source_id ?? 'default';
|
||||
const slug = stale[0].slug;
|
||||
try {
|
||||
const embeddings = await embedBatchWithBackoff(stale.map(c => c.chunk_text), { abortSignal: effectiveSignal });
|
||||
// #3507: fetch the page row for its title + stored CR mode so the
|
||||
// re-embed reproduces the page's wrapping convention instead of
|
||||
// silently stripping contextual prefixes — `embed --stale` is the
|
||||
// NORMAL post-model-migration path, so raw-text embedding here
|
||||
// quietly converted whole corpora to the unwrapped convention.
|
||||
const pageRow = await observed(pacer, () => engine.getPage(slug, { sourceId: keySourceId }));
|
||||
const embeddings = await embedBatchWithBackoff(wrapChunkTextsForStoredMode(pageRow, stale), { abortSignal: effectiveSignal });
|
||||
// Re-fetch existing chunks and merge to avoid deleting non-stale chunks.
|
||||
const existing = await observed(pacer, () => engine.getChunks(slug, { sourceId: keySourceId }));
|
||||
const staleIdxToEmbedding = new Map<number, Float32Array>();
|
||||
@@ -1078,6 +1165,14 @@ async function embedAllStale(
|
||||
engine.setPageEmbeddingSignature(slug, { sourceId: keySourceId, signature }),
|
||||
);
|
||||
}
|
||||
// #3507: a FULLY re-embedded per_chunk_synopsis page landed at the
|
||||
// title tier — keep the stamped mode honest. Partially-stale pages
|
||||
// stay stamped as-is (mixed provenance; reindex sweeps fix them).
|
||||
if (stale.length === existing.length) {
|
||||
await observed(pacer, () =>
|
||||
restampIfDemotedToTitleTier(engine, pageRow, slug, keySourceId),
|
||||
);
|
||||
}
|
||||
result.embedded += stale.length;
|
||||
} catch (e: unknown) {
|
||||
// Budget/abort-fired cancellations are expected on the way out; don't
|
||||
@@ -1138,7 +1233,9 @@ async function embedAllStale(
|
||||
// as a clean run — re-running won't help until the underlying failure is fixed.
|
||||
if (staleOpts?.catchUp && !effectiveSignal.aborted && embedFailures > 0) {
|
||||
const remaining = await engine.countStaleChunks(
|
||||
signature ? { signature, ...(sourceId ? { sourceId } : {}) } : (sourceId ? { sourceId } : undefined),
|
||||
signature
|
||||
? { signature, ...(sourceId ? { sourceId } : {}), ...(includeNullSig && { includeNullSignature: true }) }
|
||||
: (sourceId ? { sourceId } : undefined),
|
||||
);
|
||||
if (remaining > 0) {
|
||||
serr(`\n [embed] catch-up finished but ${remaining} chunk(s) remain stale after ${embedFailures} embed failure(s). These are not embeddable as-is; re-running won't clear them until the underlying error is resolved.`);
|
||||
|
||||
@@ -149,6 +149,40 @@ export const ALLOWED_TYPES = [
|
||||
] as const;
|
||||
export type AllowedType = (typeof ALLOWED_TYPES)[number];
|
||||
|
||||
/**
|
||||
* Granular collector page-types that alias into each canonical conversation
|
||||
* bucket. The v2 type-consolidation pack retypes these to the canonical names
|
||||
* (`slack-dm-day`/`slack-thread` → `slack`, `email-digest` → `email`), but a
|
||||
* brain that hasn't run that pack still carries the collector's granular types
|
||||
* in `pages.type`. Without this expansion, `listPages({ type: 'slack' })`
|
||||
* matches zero rows on such brains and the whole comms corpus is silently
|
||||
* skipped (facts stay empty → `find_trajectory` returns nothing). The canonical
|
||||
* name is always included first so consolidated brains keep working unchanged.
|
||||
*/
|
||||
export const ALLOWED_TYPE_ALIASES: Record<AllowedType, readonly string[]> = {
|
||||
conversation: ['conversation'],
|
||||
meeting: ['meeting'],
|
||||
slack: ['slack', 'slack-dm-day', 'slack-thread'],
|
||||
email: ['email', 'email-digest'],
|
||||
imessage: ['imessage'],
|
||||
'imessage-daily': ['imessage-daily'],
|
||||
};
|
||||
|
||||
/**
|
||||
* Expand the requested logical types to the concrete `pages.type` values to
|
||||
* enumerate, canonical-first and de-duplicated. Unknown types pass through
|
||||
* unchanged so an explicit override is never dropped.
|
||||
*/
|
||||
export function pageTypesForAllowed(types: readonly AllowedType[]): string[] {
|
||||
const out: string[] = [];
|
||||
for (const t of types) {
|
||||
for (const concrete of ALLOWED_TYPE_ALIASES[t] ?? [t]) {
|
||||
if (!out.includes(concrete)) out.push(concrete);
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* Pagination batch size for listPages enumeration. Per-batch memory
|
||||
* worst case = BATCH × MAX_PAGE_BODY_BYTES = 250MB at default 10
|
||||
@@ -1264,13 +1298,18 @@ export async function runExtractConversationFactsCore(
|
||||
}
|
||||
};
|
||||
|
||||
// Expand logical types (conversation/meeting/slack/email) to the concrete
|
||||
// `pages.type` values to enumerate, so brains on the granular collector
|
||||
// types are not silently skipped (see ALLOWED_TYPE_ALIASES).
|
||||
const concreteTypes = pageTypesForAllowed(types);
|
||||
|
||||
if (opts.slug) {
|
||||
const page = await engine.getPage(opts.slug, { sourceId });
|
||||
if (!page) {
|
||||
result.pages_skipped_disappeared++;
|
||||
return;
|
||||
}
|
||||
if (!types.includes(page.type as AllowedType)) {
|
||||
if (!concreteTypes.includes(page.type)) {
|
||||
result.pages_skipped++;
|
||||
return;
|
||||
}
|
||||
@@ -1284,7 +1323,7 @@ export async function runExtractConversationFactsCore(
|
||||
// honors AbortSignal at each claim boundary and threads
|
||||
// BudgetExhausted abort (D13) automatically.
|
||||
let processedPagesCount = 0;
|
||||
pageLoop: for (const type of types) {
|
||||
pageLoop: for (const type of concreteTypes) {
|
||||
let offset = 0;
|
||||
// eslint-disable-next-line no-constant-condition
|
||||
while (true) {
|
||||
|
||||
+23
-4
@@ -35,7 +35,7 @@ import type { BrainEngine, LinkBatchInput, TimelineBatchInput } from '../core/en
|
||||
import type { PageType } from '../core/types.ts';
|
||||
import { parseMarkdown } from '../core/markdown.ts';
|
||||
import {
|
||||
extractPageLinks, parseTimelineEntries, inferLinkType, makeResolver,
|
||||
extractPageLinks, parseTimelineEntries, deriveTimelineAnchor, inferLinkType, makeResolver,
|
||||
extractFrontmatterLinks, isGlobalBasenameEnabled, LINK_EXTRACTOR_VERSION_TS,
|
||||
WIKILINK_BASENAME_LINK_TYPE,
|
||||
buildBasenameIndex, queryBasenameIndex, stripCodeBlocks,
|
||||
@@ -749,6 +749,12 @@ export async function runExtract(engine: BrainEngine, args: string[]) {
|
||||
// v0.41.18.0 (A11, T8): --from-meetings extracts timeline entries from
|
||||
// meeting pages onto each discussed entity. Timeline subcommand only.
|
||||
const fromMeetings = args.includes('--from-meetings');
|
||||
// --infer-dates: for pages whose body has NO parseable timeline line, anchor
|
||||
// one entry at the page's computed effective_date (frontmatter / filename date,
|
||||
// never the updated_at fallback). Default OFF for back-compat — comms/calendar
|
||||
// brains opt in to populate timeline from slug/frontmatter dates. DB-source only
|
||||
// (needs the full Page.effective_date, which getPage projects).
|
||||
const inferDates = args.includes('--infer-dates');
|
||||
// v0.41.17.0 (T7, D9): --workers N parsed via the shared validator.
|
||||
// Honored on the fs-walk inner loops only; DB-source paths stay
|
||||
// serial in v0.41.17.0 (see ExtractOpts.workers doc).
|
||||
@@ -963,7 +969,7 @@ Status (v0.42):
|
||||
result.pages_processed = r.pages;
|
||||
}
|
||||
if (subcommand === 'timeline' || subcommand === 'all') {
|
||||
const r = await extractTimelineFromDB(engine, dryRun, jsonMode, typeFilter, since, { sourceIdFilter });
|
||||
const r = await extractTimelineFromDB(engine, dryRun, jsonMode, typeFilter, since, { sourceIdFilter, inferDates });
|
||||
result.timeline_entries_created = r.created;
|
||||
result.pages_processed = Math.max(result.pages_processed, r.pages);
|
||||
}
|
||||
@@ -1583,7 +1589,7 @@ async function extractTimelineFromDB(
|
||||
jsonMode: boolean,
|
||||
typeFilter: PageType | undefined,
|
||||
since: string | undefined,
|
||||
opts?: { sourceIdFilter?: string },
|
||||
opts?: { sourceIdFilter?: string; inferDates?: boolean },
|
||||
): Promise<{ created: number; pages: number }> {
|
||||
// v0.32.8: listAllPageRefs enumerates (slug, source_id) pairs so we can
|
||||
// thread sourceId to getPage and addTimelineEntriesBatch. Pre-fix used
|
||||
@@ -1592,6 +1598,7 @@ async function extractTimelineFromDB(
|
||||
// v0.37.7.0 #1204: when sourceIdFilter is set, scope the walk to one
|
||||
// source so federated brain users can extract per-source.
|
||||
const sourceIdFilter = opts?.sourceIdFilter;
|
||||
const inferDates = opts?.inferDates ?? false;
|
||||
const allRefs = sourceIdFilter
|
||||
? (await engine.listAllPageRefs()).filter(r => r.source_id === sourceIdFilter)
|
||||
: await engine.listAllPageRefs();
|
||||
@@ -1631,7 +1638,19 @@ async function extractTimelineFromDB(
|
||||
}
|
||||
|
||||
const fullContent = page.compiled_truth + '\n' + page.timeline;
|
||||
const entries = parseTimelineEntries(fullContent);
|
||||
let entries = parseTimelineEntries(fullContent);
|
||||
// --infer-dates: pages with no in-body timeline line but a trustworthy
|
||||
// content date (frontmatter / filename) get one anchor entry at that date.
|
||||
// Applied ONLY on the zero-entry path so it never shadows a real timeline.
|
||||
if (entries.length === 0 && inferDates) {
|
||||
const anchor = deriveTimelineAnchor({
|
||||
slug,
|
||||
title: page.title,
|
||||
effectiveDate: page.effective_date,
|
||||
effectiveDateSource: page.effective_date_source,
|
||||
});
|
||||
if (anchor) entries = [anchor];
|
||||
}
|
||||
|
||||
for (const entry of entries) {
|
||||
if (dryRunSeen) {
|
||||
|
||||
+13
-4
@@ -59,6 +59,11 @@ export async function runImport(
|
||||
* Threaded by performFullSync for `gbrain sync --exclude`.
|
||||
*/
|
||||
exclude?: string[];
|
||||
/**
|
||||
* Opt out of the git-visible fast path and walk the filesystem directly,
|
||||
* so markdown/code files matched by .gitignore can still be imported.
|
||||
*/
|
||||
includeGitignored?: boolean;
|
||||
/**
|
||||
* #753/#774 monorepo subdir-source support: when set, slugs and
|
||||
* `source_path` are computed relative to this root (the git repo root)
|
||||
@@ -71,6 +76,7 @@ export async function runImport(
|
||||
const noEmbed = args.includes('--no-embed');
|
||||
const fresh = args.includes('--fresh');
|
||||
const jsonOutput = args.includes('--json');
|
||||
const includeGitignored = args.includes('--include-gitignored') || opts.includeGitignored === true;
|
||||
|
||||
// T7 (D9): refuse cleanly when init persisted the deferred-setup sentinel,
|
||||
// unless the user is explicitly skipping embedding via `--no-embed` (in
|
||||
@@ -185,7 +191,7 @@ export async function runImport(
|
||||
const dirArg = args.find((a, i) => !a.startsWith('--') && !flagValues.has(i));
|
||||
|
||||
if (!dirArg) {
|
||||
console.error('Usage: gbrain import <dir> [--no-embed] [--workers N] [--fresh] [--source-id <id>] [--json]');
|
||||
console.error('Usage: gbrain import <dir> [--no-embed] [--workers N] [--fresh] [--source-id <id>] [--include-gitignored] [--json]');
|
||||
process.exit(1);
|
||||
}
|
||||
// #1728: capture the import target ONCE as an absolute real path. Every
|
||||
@@ -209,7 +215,7 @@ export async function runImport(
|
||||
const strategy: SyncStrategy = opts.strategy ?? 'markdown';
|
||||
const _walkT0 = Date.now();
|
||||
console.error(`[gbrain phase] import.collect_files start dir=${dir} strategy=${strategy}`);
|
||||
let allFiles = collectSyncableFiles(dir, { strategy });
|
||||
let allFiles = collectSyncableFiles(dir, { strategy, includeGitignored });
|
||||
console.error(
|
||||
`[gbrain phase] import.collect_files done ${Date.now() - _walkT0}ms files=${allFiles.length}`,
|
||||
);
|
||||
@@ -545,6 +551,7 @@ function resolveMaxWalkDepth(): number {
|
||||
|
||||
interface CollectOpts {
|
||||
strategy?: SyncStrategy;
|
||||
includeGitignored?: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -675,8 +682,10 @@ export function collectSyncableFiles(dir: string, opts: CollectOpts = {}): strin
|
||||
// vendored data/fixtures). `--cached --others --exclude-standard` = tracked
|
||||
// PLUS untracked-not-ignored, so uncommitted source is still indexed. Non-git
|
||||
// dirs (or git unavailable) fall through to the FS walk below.
|
||||
const gitFiles = gitListSyncableFiles(dir, strategy, multimodalOn);
|
||||
if (gitFiles) return gitFiles;
|
||||
if (!opts.includeGitignored) {
|
||||
const gitFiles = gitListSyncableFiles(dir, strategy, multimodalOn);
|
||||
if (gitFiles) return gitFiles;
|
||||
}
|
||||
|
||||
const maxDepth = resolveMaxWalkDepth();
|
||||
const visitedInodes = new Map<string, true>();
|
||||
|
||||
+19
-10
@@ -337,7 +337,9 @@ async function resolveAIOptions(opts: ResolveAIOptionsArgs): Promise<ResolvedAIO
|
||||
process.exit(1);
|
||||
}
|
||||
out.embedding_model = `${shorthand}:${firstModel}`;
|
||||
out.embedding_dimensions = recipe.touchpoints.embedding!.default_dims;
|
||||
// #2051: width follows the model actually chosen, not the recipe default.
|
||||
const { embeddingDimsForModel } = await import('../core/ai/model-resolver.ts');
|
||||
out.embedding_dimensions = embeddingDimsForModel(recipe, firstModel);
|
||||
}
|
||||
|
||||
if (dimsArg !== null && !Number.isNaN(dimsArg) && dimsArg > 0) {
|
||||
@@ -361,8 +363,13 @@ async function resolveAIOptions(opts: ResolveAIOptionsArgs): Promise<ResolvedAIO
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
if (recipe?.touchpoints.embedding?.default_dims) {
|
||||
out.embedding_dimensions = recipe.touchpoints.embedding.default_dims;
|
||||
// #2051: resolve the width from the SPECIFIC model, not the recipe-wide
|
||||
// default. `--embedding-model ollama:bge-m3` must yield 1024, not Ollama's
|
||||
// nomic-shaped 768.
|
||||
if (recipe) {
|
||||
const { embeddingDimsForModel } = await import('../core/ai/model-resolver.ts');
|
||||
const dims = embeddingDimsForModel(recipe, out.embedding_model);
|
||||
if (dims > 0) out.embedding_dimensions = dims;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -525,9 +532,11 @@ async function resolveEmbeddingByEnv(out: ResolvedAIOptions, nonInteractive: boo
|
||||
// legacy OpenAI 1536), not the recipe's 2560.
|
||||
const { DEFAULT_EMBEDDING_MODEL, DEFAULT_EMBEDDING_DIMENSIONS } =
|
||||
await import('../core/ai/defaults.ts');
|
||||
const { embeddingDimsForModel } = await import('../core/ai/model-resolver.ts');
|
||||
// #2051: non-canonical models resolve per-model, not recipe-wide.
|
||||
const dims = fullModel === DEFAULT_EMBEDDING_MODEL
|
||||
? DEFAULT_EMBEDDING_DIMENSIONS
|
||||
: tp.default_dims;
|
||||
: embeddingDimsForModel(r, model);
|
||||
out.embedding_model = fullModel;
|
||||
out.embedding_dimensions = dims;
|
||||
console.error(
|
||||
@@ -1108,12 +1117,12 @@ async function initPostgres(opts: {
|
||||
|
||||
// v0.37.10.0 T6 (D11) + v0.37.11.0 Lane B.2: ALWAYS configure gateway BEFORE
|
||||
// initSchema. Same preflight contract as PGLite. Refuse to call initSchema
|
||||
// until the gateway-resolved dim is validated. Schema substitution in
|
||||
// src/schema.sql is currently a static `vector(1536)` for Postgres (unlike
|
||||
// PGLite's templated dim), so a Voyage/ZE-configured Postgres brain will
|
||||
// still need a future schema rewrite path — preflight makes the
|
||||
// not-yet-supported case fail loud rather than silently produce a stuck
|
||||
// 1536d column.
|
||||
// until the gateway-resolved dim is validated. PostgresEngine.initSchema()
|
||||
// passes the resolved model and dimensions through getPostgresSchema(),
|
||||
// which templates the static `vector(1536)` source before executing it.
|
||||
// Preflight therefore prevents an invalid dimension from reaching schema
|
||||
// generation, while the post-init assertion below guards against templating
|
||||
// drift.
|
||||
let resolvedDim: number | undefined;
|
||||
let resolvedModel: string | undefined;
|
||||
if (opts.aiOpts?.noEmbedding) {
|
||||
|
||||
+37
-5
@@ -143,6 +143,31 @@ export function resolveWorkerConcurrency(args: string[], env: NodeJS.ProcessEnv
|
||||
return parsed;
|
||||
}
|
||||
|
||||
/**
|
||||
* #3026: the thin-client `list`/`get` branches receive jobs as parsed JSON
|
||||
* off the MCP wire, where every timestamp is an ISO string — but formatJob /
|
||||
* formatJobDetail (and the stalled-detection comparison) hold a Date
|
||||
* contract, hydrated locally by MinionQueue.rowToJob. Rehydrate once at the
|
||||
* unpack boundary so both paths hand the formatters real Dates. Exported for
|
||||
* unit tests.
|
||||
*/
|
||||
const JOB_DATE_FIELDS = [
|
||||
'created_at', 'updated_at', 'started_at', 'finished_at', 'lock_until', 'delay_until',
|
||||
] as const;
|
||||
|
||||
export function rehydrateJobDates<T>(job: T): T {
|
||||
if (!job || typeof job !== 'object') return job;
|
||||
const rec = job as { [k: string]: unknown };
|
||||
for (const field of JOB_DATE_FIELDS) {
|
||||
const v = rec[field];
|
||||
if (typeof v === 'string') {
|
||||
const d = new Date(v);
|
||||
if (!Number.isNaN(d.getTime())) rec[field] = d;
|
||||
}
|
||||
}
|
||||
return job;
|
||||
}
|
||||
|
||||
function formatJob(job: MinionJob): string {
|
||||
const dur = job.finished_at && job.started_at
|
||||
? `${((job.finished_at.getTime() - job.started_at.getTime()) / 1000).toFixed(1)}s`
|
||||
@@ -208,7 +233,7 @@ USAGE
|
||||
gbrain jobs get <id>
|
||||
gbrain jobs cancel <id>
|
||||
gbrain jobs retry <id>
|
||||
gbrain jobs prune [--older-than 30d]
|
||||
gbrain jobs prune [--older-than 30d] [--dry-run]
|
||||
gbrain jobs delete <id>
|
||||
gbrain jobs stats
|
||||
gbrain jobs smoke
|
||||
@@ -496,7 +521,7 @@ HANDLER TYPES (built in)
|
||||
const raw = await callRemoteTool(cfg!, 'list_jobs', {
|
||||
status, queue: queueName, limit,
|
||||
}, { timeoutMs: 30_000 });
|
||||
jobs = unpackToolResult<MinionJob[]>(raw);
|
||||
jobs = unpackToolResult<MinionJob[]>(raw).map((j) => rehydrateJobDates(j));
|
||||
} else {
|
||||
try { await queue.ensureSchema(); }
|
||||
catch (e) { console.error(e instanceof Error ? e.message : String(e)); process.exit(1); }
|
||||
@@ -525,7 +550,7 @@ HANDLER TYPES (built in)
|
||||
if (isThinClient(cfg)) {
|
||||
try {
|
||||
const raw = await callRemoteTool(cfg!, 'get_job', { id }, { timeoutMs: 30_000 });
|
||||
job = unpackToolResult<MinionJob | null>(raw);
|
||||
job = rehydrateJobDates(unpackToolResult<MinionJob | null>(raw));
|
||||
} catch (e) {
|
||||
// The remote op throws `invalid_params` on not-found; surface as
|
||||
// the same "Job not found" exit-1 the local path produces.
|
||||
@@ -608,8 +633,15 @@ HANDLER TYPES (built in)
|
||||
try { await queue.ensureSchema(); }
|
||||
catch (e) { console.error(e instanceof Error ? e.message : String(e)); process.exit(1); }
|
||||
|
||||
const count = await queue.prune({ olderThan: new Date(Date.now() - days * 86400000) });
|
||||
console.log(`Pruned ${count} jobs older than ${days} days.`);
|
||||
// #2712: --dry-run previews the count without deleting. It used to be
|
||||
// silently ignored (the destructive default ran anyway).
|
||||
const dryRun = hasFlag(args, '--dry-run');
|
||||
const count = await queue.prune({ olderThan: new Date(Date.now() - days * 86400000), dryRun });
|
||||
if (dryRun) {
|
||||
console.log(`[dry-run] Would prune ${count} jobs older than ${days} days. Nothing deleted.`);
|
||||
} else {
|
||||
console.log(`Pruned ${count} jobs older than ${days} days.`);
|
||||
}
|
||||
break;
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,402 @@
|
||||
/**
|
||||
* `gbrain migrate embeddings --to <provider:model>` (#3390) — the
|
||||
* provider-agnostic forward migration off any embedding provider, built for
|
||||
* the ZeroEntropy 2026-09-04 sunset but not keyed to it.
|
||||
*
|
||||
* Also reachable as `gbrain retrieval-upgrade` — the command README.md and
|
||||
* doctor.ts have promised since v0.36 but which never had a dispatch branch.
|
||||
*
|
||||
* Flow (everything heavy is reused, see src/core/embedding-migration.ts):
|
||||
* 1. plan — chunk/char counts via the widened stale predicates,
|
||||
* cost estimate from embedding-pricing.ts
|
||||
* 2. preflight— print estimate; require --yes or interactive confirm
|
||||
* (non-TTY without --yes refuses with exit 2, mirroring the
|
||||
* reindex-code cost gate in docs/operations/spend-controls.md)
|
||||
* 3. probe — one live embed against the TARGET provider BEFORE any
|
||||
* mutation (validates key + model + dims in one shot)
|
||||
* 4. apply — schema transition (dim change), config (DB + file plane),
|
||||
* #3391 NULL-signature-inclusive invalidation, cache purge
|
||||
* 5. re-embed — runEmbedCore --stale --catch-up with single-flight locks,
|
||||
* pacing (--pace), progress reporting. Resumable: a killed
|
||||
* run re-runs the SAME command; the NULL-embedding cursor is
|
||||
* the checkpoint and steps 3-4 no-op on the second pass.
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { serr, slog } from '../core/console-prefix.ts';
|
||||
import {
|
||||
planEmbeddingMigration,
|
||||
applyEmbeddingMigration,
|
||||
completeEmbeddingMigration,
|
||||
reconcilePageSignatures,
|
||||
MIGRATION_STATE_KEY,
|
||||
type EmbeddingMigrationPlan,
|
||||
} from '../core/embedding-migration.ts';
|
||||
import { formatEnvOverrideWarning } from '../core/retrieval-upgrade-planner.ts';
|
||||
import { parsePaceArgs, runEmbedCore } from './embed.ts';
|
||||
|
||||
export interface MigrateEmbeddingsFlags {
|
||||
to?: string;
|
||||
dim?: number;
|
||||
yes: boolean;
|
||||
dryRun: boolean;
|
||||
json: boolean;
|
||||
noEmbed: boolean;
|
||||
ignoreEnvOverride: boolean;
|
||||
batchSize?: number;
|
||||
pace?: ReturnType<typeof parsePaceArgs>;
|
||||
}
|
||||
|
||||
export function parseMigrateEmbeddingsFlags(args: string[]): MigrateEmbeddingsFlags {
|
||||
const toIdx = args.indexOf('--to');
|
||||
const dimIdx = args.indexOf('--dim');
|
||||
const dimRaw = dimIdx >= 0 ? parseInt(args[dimIdx + 1] ?? '', 10) : NaN;
|
||||
const bsIdx = args.indexOf('--batch-size');
|
||||
const bsRaw = bsIdx >= 0 ? parseInt(args[bsIdx + 1] ?? '', 10) : NaN;
|
||||
const batchSize = Number.isFinite(bsRaw) && bsRaw > 0 ? Math.min(10_000, bsRaw) : undefined;
|
||||
return {
|
||||
to: toIdx >= 0 ? args[toIdx + 1] : undefined,
|
||||
dim: Number.isFinite(dimRaw) && dimRaw > 0 ? dimRaw : undefined,
|
||||
yes: args.includes('--yes') || args.includes('--non-interactive'),
|
||||
dryRun: args.includes('--dry-run'),
|
||||
json: args.includes('--json'),
|
||||
noEmbed: args.includes('--no-embed'),
|
||||
ignoreEnvOverride: args.includes('--ignore-env-override'),
|
||||
...(batchSize !== undefined && { batchSize }),
|
||||
pace: parsePaceArgs(args),
|
||||
};
|
||||
}
|
||||
|
||||
function printHelp(): void {
|
||||
process.stdout.write(`Usage: gbrain migrate embeddings --to <provider:model> [flags]
|
||||
|
||||
Re-embed the whole brain onto a different embedding provider/model. Handles
|
||||
dimension changes (schema transition), pages without a recorded embedding
|
||||
signature (#3391), the query cache, and resume-after-kill. The forward path
|
||||
off a sunsetting provider.
|
||||
|
||||
Flags:
|
||||
--to <provider:model> Target embedding model (e.g. openai:text-embedding-3-small).
|
||||
--dim <N> Target dimensions. Defaults to the provider recipe's
|
||||
declared width; required when the recipe declares none.
|
||||
--dry-run Plan + cost estimate only; change nothing.
|
||||
--yes Skip the confirm prompt (required non-interactively).
|
||||
--json Machine-readable envelope on stdout.
|
||||
--no-embed Apply schema + config + invalidation, but skip the
|
||||
re-embed pass (run \`gbrain embed --stale --include-null-signature\`
|
||||
or \`... --background\` yourself).
|
||||
--batch-size <N> Stale-chunk batch size for the re-embed (default 2000).
|
||||
--pace[=mode] DB-contention pacing for the re-embed (off|gentle|balanced|aggressive).
|
||||
--ignore-env-override Proceed even when GBRAIN_EMBEDDING_* env vars would
|
||||
override the target at runtime (you know why).
|
||||
--help Show this help.
|
||||
|
||||
A killed run is resumable: re-run the same command. Already-migrated chunks
|
||||
are never re-embedded twice.
|
||||
`);
|
||||
}
|
||||
|
||||
function renderPlan(plan: EmbeddingMigrationPlan): string {
|
||||
const lines: string[] = [];
|
||||
lines.push('Embedding migration plan');
|
||||
lines.push(` From: ${plan.from_model} (${plan.from_dims}d${plan.column_dims !== null && plan.column_dims !== plan.from_dims ? `; column is actually ${plan.column_dims}d` : ''})`);
|
||||
lines.push(` To: ${plan.to_model} (${plan.to_dims}d)`);
|
||||
if (plan.dim_change) {
|
||||
lines.push(` DESTRUCTIVE: the embedding column is rebuilt at ${plan.to_dims}d, which DELETES`);
|
||||
lines.push(' every stored embedding vector in this brain. They are not recoverable —');
|
||||
lines.push(' going back to the old provider means paying for a second full re-embed.');
|
||||
lines.push(' Until the re-embed finishes, semantic search is degraded to lexical-only.');
|
||||
lines.push(` The query cache and fact embeddings are rebuilt at ${plan.to_dims}d too`);
|
||||
lines.push(' (cache refills on next query; facts re-embed on their next write).');
|
||||
}
|
||||
lines.push(` Chunks to re-embed: ${plan.chunks_to_embed}${plan.null_signature_chunks > 0 ? ` (includes ${plan.null_signature_chunks} on pages with no recorded embedding signature)` : ''}`);
|
||||
lines.push(
|
||||
plan.price_known
|
||||
? ` Estimated cost: $${plan.est_cost_usd.toFixed(2)} (${plan.total_chars} chars at the ${plan.to_model} rate)`
|
||||
: ` Estimated cost: unknown — no pricing entry for ${plan.to_model}. Check the provider's pricing before proceeding.`,
|
||||
);
|
||||
if (plan.resuming) {
|
||||
lines.push(' Resuming: a prior migration to this target was interrupted; continuing it.');
|
||||
}
|
||||
if (plan.reranker_warning) {
|
||||
lines.push(` WARNING: ${plan.reranker_warning}`);
|
||||
}
|
||||
return lines.join('\n');
|
||||
}
|
||||
|
||||
/** Single-keypress y/N confirm on stdin. Injectable for tests. */
|
||||
async function defaultConfirm(question: string): Promise<boolean> {
|
||||
process.stderr.write(`${question} [y/N] `);
|
||||
const stdin = process.stdin;
|
||||
stdin.setRawMode?.(true);
|
||||
stdin.resume();
|
||||
const key: string = await new Promise((resolve) => {
|
||||
stdin.once('data', (d) => resolve(d.toString()));
|
||||
});
|
||||
stdin.setRawMode?.(false);
|
||||
stdin.pause();
|
||||
process.stderr.write('\n');
|
||||
return key.trim().toLowerCase().startsWith('y');
|
||||
}
|
||||
|
||||
/**
|
||||
* One tiny embed against the TARGET provider, BEFORE any mutation: validates
|
||||
* the API key, the model id, and dimension support in a single call, so a bad
|
||||
* target fails with the brain untouched instead of after the column is
|
||||
* dropped. Shared by the CLI and the `migrate_embeddings` op (the op used to
|
||||
* skip it, which let `yes:true` drop the column against a bad key).
|
||||
*/
|
||||
export async function probeTargetProvider(
|
||||
toModel: string,
|
||||
toDims: number,
|
||||
): Promise<{ ok: true } | { ok: false; message: string }> {
|
||||
try {
|
||||
const { embed } = await import('../core/ai/gateway.ts');
|
||||
const vecs = await embed(['gbrain embedding migration probe'], {
|
||||
embeddingModel: toModel,
|
||||
dimensions: toDims,
|
||||
});
|
||||
const got = vecs[0]?.length ?? 0;
|
||||
if (got !== toDims) {
|
||||
return {
|
||||
ok: false,
|
||||
message: `Target provider returned ${got}-dim vectors, expected ${toDims}. Pass a valid --dim for ${toModel}.`,
|
||||
};
|
||||
}
|
||||
return { ok: true };
|
||||
} catch (e) {
|
||||
return {
|
||||
ok: false,
|
||||
message: `Preflight embed against ${toModel} failed — nothing was changed:\n ${e instanceof Error ? e.message : String(e)}`,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Persist the target model+dims to the FILE plane and reconfigure the
|
||||
* in-process gateway. The gateway reads file/env config, not the DB plane —
|
||||
* without this the re-embed would silently run against the OLD provider.
|
||||
* Shared by the CLI command and the `migrate_embeddings` op handler.
|
||||
*/
|
||||
export async function persistEmbeddingFileConfig(
|
||||
toModel: string,
|
||||
toDims: number,
|
||||
): Promise<void> {
|
||||
const { loadConfig, saveConfig } = await import('../core/config.ts');
|
||||
const { configureGateway } = await import('../core/ai/gateway.ts');
|
||||
const { buildGatewayConfig } = await import('../core/ai/build-gateway-config.ts');
|
||||
const cfg = loadConfig();
|
||||
if (!cfg) {
|
||||
// REFUSE rather than warn-and-proceed. Without a file plane to write, the
|
||||
// switch would not survive this process: the next `gbrain` invocation
|
||||
// reads file/env config, sees the OLD provider, and re-embeds the brain
|
||||
// back into the old space (paying twice) — or fails outright against a
|
||||
// column that is now the new width. Thrown from inside
|
||||
// applyEmbeddingMigration's try, so it surfaces as status: 'failed'
|
||||
// BEFORE the config/cache steps and the caller exits non-zero.
|
||||
throw new Error(
|
||||
'No ~/.gbrain/config.json found — refusing to migrate.\n' +
|
||||
' The embed pipeline reads file/env config, so without a file plane this switch\n' +
|
||||
' would not survive the process and the next run would re-embed into the old space.\n' +
|
||||
' Fix: run `gbrain init` (or set GBRAIN_EMBEDDING_MODEL + GBRAIN_EMBEDDING_DIMENSIONS\n' +
|
||||
' in the environment of every gbrain process) and re-run.',
|
||||
);
|
||||
}
|
||||
cfg.embedding_model = toModel;
|
||||
cfg.embedding_dimensions = toDims;
|
||||
saveConfig(cfg);
|
||||
configureGateway(buildGatewayConfig(cfg));
|
||||
}
|
||||
|
||||
export interface RunMigrateEmbeddingsOpts {
|
||||
/** Test seams. */
|
||||
confirm?: (question: string) => Promise<boolean>;
|
||||
isTTY?: boolean;
|
||||
exit?: (code: number) => never;
|
||||
}
|
||||
|
||||
export async function runMigrateEmbeddings(
|
||||
engine: BrainEngine,
|
||||
args: string[],
|
||||
opts: RunMigrateEmbeddingsOpts = {},
|
||||
): Promise<void> {
|
||||
// Explicit `never` annotation so TS control-flow analysis treats every
|
||||
// exit() call as terminal (required for narrowing after the guard blocks).
|
||||
const exit: (code: number) => never = opts.exit ?? ((code: number) => process.exit(code));
|
||||
if (args.includes('--help') || args.includes('-h')) {
|
||||
printHelp();
|
||||
exit(0);
|
||||
}
|
||||
const flags = parseMigrateEmbeddingsFlags(args);
|
||||
if (!flags.to) {
|
||||
serr('Missing --to <provider:model>. Example: gbrain migrate embeddings --to openai:text-embedding-3-small');
|
||||
serr('Run with --help for all flags.');
|
||||
exit(1);
|
||||
}
|
||||
|
||||
// From-state as the gateway resolved it (file/env config + defaults) —
|
||||
// the truth for what embeds run under TODAY.
|
||||
let fromModel: string | undefined;
|
||||
let fromDims: number | undefined;
|
||||
try {
|
||||
const { getEmbeddingModel, getEmbeddingDimensions } = await import('../core/ai/gateway.ts');
|
||||
fromModel = getEmbeddingModel();
|
||||
fromDims = getEmbeddingDimensions();
|
||||
} catch {
|
||||
// Gateway unconfigured — plan falls back to shipped defaults.
|
||||
}
|
||||
|
||||
let plan: EmbeddingMigrationPlan;
|
||||
try {
|
||||
plan = await planEmbeddingMigration(engine, {
|
||||
to: flags.to!,
|
||||
...(flags.dim !== undefined && { dim: flags.dim }),
|
||||
...(fromModel !== undefined && { fromModel }),
|
||||
...(fromDims !== undefined && { fromDims }),
|
||||
});
|
||||
} catch (e) {
|
||||
serr(e instanceof Error ? e.message : String(e));
|
||||
exit(1);
|
||||
return; // unreachable; keeps TS happy for injected exit seams
|
||||
}
|
||||
|
||||
if (flags.json) {
|
||||
// Human plan goes to stderr so stdout stays JSON-clean.
|
||||
serr(renderPlan(plan));
|
||||
} else {
|
||||
console.log(renderPlan(plan));
|
||||
}
|
||||
|
||||
if (plan.chunks_to_embed === 0 && !plan.dim_change && plan.from_model === plan.to_model) {
|
||||
if (flags.json) console.log(JSON.stringify({ status: 'skipped_no_work', plan }, null, 2));
|
||||
else console.log('Nothing to migrate — brain is already on the target model.');
|
||||
exit(0);
|
||||
}
|
||||
|
||||
if (flags.dryRun) {
|
||||
if (flags.json) console.log(JSON.stringify({ status: 'planned', plan }, null, 2));
|
||||
exit(0);
|
||||
}
|
||||
|
||||
// ── Consent gate. Unlike the pure cost gates in
|
||||
// docs/operations/spend-controls.md, `spend.posture=tokenmax` does NOT
|
||||
// bypass this one: posture waives the SPEND ceiling, and this gate also
|
||||
// guards a destructive schema rebuild (existing vectors are dropped, and
|
||||
// retrieval is degraded until the re-embed finishes). We honor the posture
|
||||
// by marking the dollar figure informational, and still ask.
|
||||
if (!flags.yes) {
|
||||
const { resolveSpendPosture } = await import('../core/spend-posture.ts');
|
||||
const posture = await resolveSpendPosture(engine);
|
||||
if (posture === 'tokenmax') {
|
||||
serr(' [migrate] spend.posture=tokenmax: the cost estimate above is informational.');
|
||||
serr(' [migrate] Confirmation is still required — this rebuilds the embedding column (destructive, not just costly).');
|
||||
}
|
||||
const isTTY = opts.isTTY ?? Boolean(process.stdin.isTTY);
|
||||
if (!isTTY) {
|
||||
serr('Refusing to migrate without confirmation in a non-TTY environment. Re-run with --yes.');
|
||||
exit(2);
|
||||
}
|
||||
const confirm = opts.confirm ?? defaultConfirm;
|
||||
const priceNote = plan.price_known ? `~$${plan.est_cost_usd.toFixed(2)}` : 'an UNKNOWN amount';
|
||||
const ok = await confirm(`Re-embed ${plan.chunks_to_embed} chunks (${priceNote})?`);
|
||||
if (!ok) {
|
||||
serr('Aborted. Nothing was changed.');
|
||||
exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
// ── Live probe BEFORE any mutation: one tiny embed against the TARGET
|
||||
// provider validates API key, model id, and dimension support in one call.
|
||||
const probe = await probeTargetProvider(plan.to_model, plan.to_dims);
|
||||
if (!probe.ok) {
|
||||
serr(probe.message);
|
||||
exit(1);
|
||||
}
|
||||
|
||||
// ── Apply: schema + config + invalidation + cache purge.
|
||||
const applied = await applyEmbeddingMigration(engine, plan, {
|
||||
ignoreEnvOverride: flags.ignoreEnvOverride,
|
||||
persistConfig: (toModel, toDims) => persistEmbeddingFileConfig(toModel, toDims),
|
||||
});
|
||||
|
||||
if (applied.status === 'refused') {
|
||||
if (flags.json) console.log(JSON.stringify(applied, null, 2));
|
||||
else serr(formatEnvOverrideWarning(applied.warning));
|
||||
exit(1);
|
||||
}
|
||||
if (applied.status === 'failed') {
|
||||
if (flags.json) console.log(JSON.stringify(applied, null, 2));
|
||||
else serr(`Migration apply failed: ${applied.reason}`);
|
||||
exit(1);
|
||||
}
|
||||
|
||||
serr(` [migrate] schema ${applied.schema_transitioned ? `rebuilt at ${plan.to_dims}d` : 'unchanged'}; ` +
|
||||
`${applied.invalidated} chunk(s) invalidated; query cache purged (${applied.cache_cleared} row(s)).`);
|
||||
|
||||
if (flags.noEmbed) {
|
||||
const msg = 'Config + schema migrated. Re-embed deferred — run: gbrain embed --stale --catch-up --include-null-signature';
|
||||
if (flags.json) console.log(JSON.stringify({ ...applied, status: 'applied_no_embed', plan }, null, 2));
|
||||
else console.log(msg);
|
||||
exit(0);
|
||||
}
|
||||
|
||||
// ── Re-embed. All the machinery (locks, pacing, backoff, progress,
|
||||
// signature stamping) is the standard embed pipeline.
|
||||
const { createProgress } = await import('../core/progress.ts');
|
||||
const { getCliOptions, cliOptsToProgressOptions } = await import('../core/cli-options.ts');
|
||||
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
|
||||
let progressStarted = false;
|
||||
const embedResult = await runEmbedCore(engine, {
|
||||
stale: true,
|
||||
catchUp: true,
|
||||
singleFlight: true,
|
||||
includeNullSignature: true,
|
||||
quiet: flags.json,
|
||||
...(flags.batchSize !== undefined && { batchSize: flags.batchSize }),
|
||||
...(flags.pace && { pace: flags.pace }),
|
||||
onProgress: (done, total) => {
|
||||
if (!progressStarted) {
|
||||
progress.start('migrate.reembed', total);
|
||||
progressStarted = true;
|
||||
}
|
||||
progress.tick(1);
|
||||
},
|
||||
});
|
||||
if (progressStarted) progress.finish();
|
||||
|
||||
// Reconcile signatures BEFORE the completion probe: pages straddling a
|
||||
// stale-batch boundary are embedded correctly but left unstamped by the
|
||||
// embed loop's all-or-nothing stamp rule. Without this the probe would call
|
||||
// a fully-migrated brain "incomplete" and the re-run would pay again.
|
||||
const reconciled = await reconcilePageSignatures(engine, plan);
|
||||
if (reconciled > 0) {
|
||||
serr(` [migrate] reconciled the embedding signature on ${reconciled} fully-embedded page(s) (batch-boundary pages).`);
|
||||
}
|
||||
|
||||
const remaining = await engine.countStaleChunks({
|
||||
signature: `${plan.to_model}:${plan.to_dims}`,
|
||||
includeNullSignature: true,
|
||||
});
|
||||
|
||||
if (remaining === 0) {
|
||||
await completeEmbeddingMigration(engine, plan);
|
||||
if (flags.json) {
|
||||
console.log(JSON.stringify({ status: 'completed', plan, embedded: embedResult.embedded, remaining: 0 }, null, 2));
|
||||
} else {
|
||||
slog(`Migration complete: ${embedResult.embedded} chunk(s) embedded on ${plan.to_model} (${plan.to_dims}d).`);
|
||||
if (plan.reranker_warning) serr(` [migrate] reminder: ${plan.reranker_warning}`);
|
||||
}
|
||||
exit(0);
|
||||
} else {
|
||||
if (flags.json) {
|
||||
console.log(JSON.stringify({ status: 'incomplete', plan, embedded: embedResult.embedded, remaining }, null, 2));
|
||||
} else {
|
||||
serr(`Migration incomplete: ${remaining} chunk(s) still stale (embed failures or an interrupted run).`);
|
||||
serr('Re-run the same command to resume — completed chunks are never re-embedded.');
|
||||
}
|
||||
exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
/** Re-export for the op handler + tests. */
|
||||
export { MIGRATION_STATE_KEY };
|
||||
@@ -134,7 +134,7 @@ EXAMPLES
|
||||
gbrain providers list
|
||||
gbrain providers test --model openai:text-embedding-3-large
|
||||
gbrain providers test --touchpoint chat --model anthropic:claude-haiku-4-5
|
||||
gbrain providers test --touchpoint chat --model deepseek:deepseek-chat
|
||||
gbrain providers test --touchpoint chat --model deepseek:deepseek-v4-flash
|
||||
gbrain providers env ollama
|
||||
gbrain providers explain --json
|
||||
`);
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { VERSION } from '../version.ts';
|
||||
import { isMinorOrMajorBump, isValidVersionString } from '../core/semver.ts';
|
||||
import { isNewerVersion, isValidVersionString } from '../core/semver.ts';
|
||||
import { fetchChangelog, fetchLatestRelease } from './check-update.ts';
|
||||
import { detectInstallMethod, runUpgrade } from './upgrade.ts';
|
||||
import { writeUpdateCache } from '../core/self-upgrade.ts';
|
||||
@@ -35,9 +35,10 @@ export async function runSelfUpgrade(args: string[]): Promise<void> {
|
||||
const force = args.includes('--force');
|
||||
const json = args.includes('--json');
|
||||
|
||||
const release = await fetchLatestRelease();
|
||||
const result = await fetchLatestRelease();
|
||||
const release = result.ok ? result : null;
|
||||
const latest = release ? release.tag.replace(/^v/, '') : null;
|
||||
const behind = !!latest && isValidVersionString(latest) && isMinorOrMajorBump(VERSION, latest);
|
||||
const behind = !!latest && isValidVersionString(latest) && isNewerVersion(VERSION, latest);
|
||||
|
||||
// Warm the cache so the next invocation's startup hook can emit without a fetch.
|
||||
try {
|
||||
|
||||
@@ -1156,7 +1156,8 @@ export async function runServeHttp(engine: BrainEngine, options: ServeHttpOption
|
||||
// Unified view: OAuth clients + legacy API keys
|
||||
const oauthClients = await sql`
|
||||
SELECT c.client_id as id, c.client_name as name, 'oauth' as auth_type,
|
||||
c.grant_types, c.scope, c.created_at, c.token_ttl,
|
||||
c.grant_types, c.scope, c.source_id, c.federated_read,
|
||||
c.created_at, c.token_ttl,
|
||||
CASE WHEN c.deleted_at IS NOT NULL THEN 'revoked' ELSE 'active' END as status,
|
||||
(SELECT max(created_at) FROM mcp_request_log WHERE token_name = c.client_id) as last_used_at,
|
||||
(SELECT count(*)::int FROM mcp_request_log WHERE token_name = c.client_id) as total_requests,
|
||||
@@ -1172,12 +1173,25 @@ export async function runServeHttp(engine: BrainEngine, options: ServeHttpOption
|
||||
(SELECT count(*)::int FROM mcp_request_log WHERE token_name = a.name AND created_at > now() - interval '24 hours') as requests_today
|
||||
FROM access_tokens a ORDER BY a.created_at DESC
|
||||
`;
|
||||
res.json([...oauthClients, ...legacyKeys]);
|
||||
res.json([
|
||||
...oauthClients,
|
||||
...legacyKeys.map((key) => ({ ...key, source_id: null, federated_read: [] })),
|
||||
]);
|
||||
} catch (e) {
|
||||
res.status(503).json({ error: 'service_unavailable' });
|
||||
}
|
||||
});
|
||||
|
||||
app.get('/admin/api/sources', requireAdmin, async (_req: Request, res: Response) => {
|
||||
try {
|
||||
const { listSources } = await import('../core/sources-ops.ts');
|
||||
const sources = await listSources(engine);
|
||||
res.json(sources.map(({ id, name, federated }) => ({ id, name, federated })));
|
||||
} catch {
|
||||
res.status(503).json({ error: 'service_unavailable' });
|
||||
}
|
||||
});
|
||||
|
||||
// v0.38 Slice 4 — per-OAuth-client agent spend viewer. Pre-computes today's
|
||||
// spend (committed + pending reservations) per client so the Agents tab
|
||||
// can render a "$X / $Y today" cell. Read-side endpoint only — no mutation.
|
||||
|
||||
+195
-25
@@ -1,6 +1,6 @@
|
||||
import { existsSync, readFileSync, writeFileSync, statSync, realpathSync } from 'fs';
|
||||
import { execFileSync } from 'child_process';
|
||||
import { join, relative } from 'path';
|
||||
import { isAbsolute, join, relative, sep } from 'path';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { DELETE_BATCH_SIZE } from '../core/engine-constants.ts';
|
||||
import { importFile } from '../core/import-file.ts';
|
||||
@@ -239,11 +239,12 @@ export interface SyncResult {
|
||||
export function estimateSourceTreeTokens(
|
||||
localPath: string,
|
||||
strategy: 'markdown' | 'code' | 'auto',
|
||||
opts: { includeGitignored?: boolean } = {},
|
||||
): { tokens: number; files: number } {
|
||||
let tokens = 0;
|
||||
let files = 0;
|
||||
try {
|
||||
const fileList = collectSyncableFiles(localPath, { strategy });
|
||||
const fileList = collectSyncableFiles(localPath, { strategy, includeGitignored: opts.includeGitignored });
|
||||
for (const fullPath of fileList) {
|
||||
try {
|
||||
const stat = statSync(fullPath);
|
||||
@@ -376,6 +377,7 @@ export function estimateInlineNewTokens(
|
||||
chunker_version: string | null;
|
||||
}>,
|
||||
currentChunkerVersion: string,
|
||||
opts: { forceFullTree?: boolean } = {},
|
||||
): InlineEstimate {
|
||||
let tokens = 0;
|
||||
let changedSources = 0;
|
||||
@@ -398,6 +400,14 @@ export function estimateInlineNewTokens(
|
||||
const strategy = cfg.strategy ?? 'markdown';
|
||||
const localPath = src.local_path;
|
||||
|
||||
if (opts.forceFullTree) {
|
||||
tokens += estimateSourceTreeTokens(localPath, strategy, { includeGitignored: true }).tokens;
|
||||
changedSources++;
|
||||
hadCeiling = true;
|
||||
ceilingReasons.push('include_gitignored');
|
||||
continue;
|
||||
}
|
||||
|
||||
// Rung 2: chunker drift forces a full re-chunk → full re-embed. CEILING.
|
||||
if (src.chunker_version !== currentChunkerVersion) {
|
||||
ceiling(localPath, strategy, 'chunker_drift');
|
||||
@@ -542,6 +552,7 @@ interface CostGateContext {
|
||||
jsonOut: boolean;
|
||||
yesFlag: boolean;
|
||||
full: boolean;
|
||||
includeGitignored?: boolean;
|
||||
/** Message prefix ('sync --all' | 'sync'). */
|
||||
label: string;
|
||||
}
|
||||
@@ -626,7 +637,9 @@ async function runInlineCostGate(
|
||||
}
|
||||
|
||||
// ── Inline path ───────────────────────────────────────────────
|
||||
const inline = estimateInlineNewTokens(sources, String(CHUNKER_VERSION));
|
||||
const inline = estimateInlineNewTokens(sources, String(CHUNKER_VERSION), {
|
||||
forceFullTree: ctx.includeGitignored === true,
|
||||
});
|
||||
// D7A: `--full` runs `performFullSync` → `runEmbedCore({stale:true})`, which
|
||||
// sweeps the pre-existing stale backlog INLINE on top of the delta. Price it.
|
||||
const costUsd = estimateEmbeddingCostUsd(inline.tokens) + (full ? staleCostUsd : 0);
|
||||
@@ -764,6 +777,11 @@ export interface SyncOpts {
|
||||
* matching the #1433 metafile posture).
|
||||
*/
|
||||
exclude?: string[];
|
||||
/**
|
||||
* Include files matched by .gitignore. Git cannot report untracked ignored
|
||||
* changes in diffs, so sync uses the full filesystem walker when this is set.
|
||||
*/
|
||||
includeGitignored?: boolean;
|
||||
/**
|
||||
* Number of parallel workers for the import phase. When > 1, each worker
|
||||
* gets its own small Postgres connection pool and files are dispatched via
|
||||
@@ -1152,6 +1170,20 @@ function createSyncBaselineCommit(repoPath: string): void {
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* True when `childReal` is `rootReal` itself or lives inside it. Both arguments
|
||||
* must already be realpath-resolved. Containment is decided by `relative()`
|
||||
* rather than a string prefix, so it holds on Windows too: `realpathSync`
|
||||
* returns backslash paths there, and a literal `rootReal + '/'` prefix can
|
||||
* never match one. A sibling (`root-evil`) is rejected because `relative`
|
||||
* yields `../root-evil`, and a cross-drive path because it yields an absolute.
|
||||
*/
|
||||
export function isWithinRoot(childReal: string, rootReal: string): boolean {
|
||||
if (childReal === rootReal) return true;
|
||||
const rel = relative(rootReal, childReal);
|
||||
return rel !== '' && rel !== '..' && !rel.startsWith('..' + sep) && !isAbsolute(rel);
|
||||
}
|
||||
|
||||
/**
|
||||
* #774 NAV-1 TOCTOU: true only if filePath realpath-resolves inside gitRoot.
|
||||
* Guards symlink escape at the per-file level (a committed symlink whose
|
||||
@@ -1159,9 +1191,7 @@ function createSyncBaselineCommit(repoPath: string): void {
|
||||
*/
|
||||
function isPathSafe(filePath: string, gitRoot: string): boolean {
|
||||
try {
|
||||
const real = realpathSync(filePath);
|
||||
const rootReal = realpathSync(gitRoot);
|
||||
return real === rootReal || real.startsWith(rootReal + '/');
|
||||
return isWithinRoot(realpathSync(filePath), realpathSync(gitRoot));
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
@@ -1932,7 +1962,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// NAV-1/NAV-2 scope-entry guard: the realpath-resolved scope must live
|
||||
// inside the realpath-resolved git root. Catches `--src-subpath ../escape`
|
||||
// AND a symlinked subdir pointing outside the repo, before any git op runs.
|
||||
if (syncScopeRoot !== gitContextRoot && !syncScopeRoot.startsWith(gitContextRoot + '/')) {
|
||||
if (!isWithinRoot(syncScopeRoot, gitContextRoot)) {
|
||||
throw new Error(
|
||||
`Sync scope ${syncScopeRoot} resolves outside git repo ${gitContextRoot}. ` +
|
||||
`Refusing to sync: possible path traversal via --src-subpath.`,
|
||||
@@ -2158,6 +2188,14 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
return performFullSync(engine, fullSyncRoots, headCommit, opts);
|
||||
}
|
||||
|
||||
if (opts.includeGitignored) {
|
||||
slog(
|
||||
`[sync] --include-gitignored: running full filesystem reconcile because ` +
|
||||
`git diff cannot report untracked ignored files.`,
|
||||
);
|
||||
return performFullSync(engine, fullSyncRoots, headCommit, opts);
|
||||
}
|
||||
|
||||
// v0.42.x (#1794): resumable incremental sync — resolve the PINNED target.
|
||||
// last_commit advances only at FULL import completion, so a killed run keeps
|
||||
// lastCommit fixed and the checkpoint key stable across every resume even as
|
||||
@@ -3557,7 +3595,10 @@ async function performFullSync(
|
||||
// code --dry-run` always reported zero files even when ~1500 code
|
||||
// files were waiting.
|
||||
if (opts.dryRun) {
|
||||
let allFiles = collectSyncableFiles(syncScopeRoot, { strategy: opts.strategy ?? 'markdown' });
|
||||
let allFiles = collectSyncableFiles(syncScopeRoot, {
|
||||
strategy: opts.strategy ?? 'markdown',
|
||||
includeGitignored: opts.includeGitignored,
|
||||
});
|
||||
if (opts.exclude && opts.exclude.length > 0) {
|
||||
allFiles = allFiles.filter(abs => !matchesAnyGlob(relative(syncScopeRoot, abs), opts.exclude));
|
||||
}
|
||||
@@ -3591,6 +3632,7 @@ async function performFullSync(
|
||||
const { runImport } = await import('./import.ts');
|
||||
const importArgs = [syncScopeRoot];
|
||||
if (opts.noEmbed) importArgs.push('--no-embed');
|
||||
if (opts.includeGitignored) importArgs.push('--include-gitignored');
|
||||
if (fullConcurrency > 1) importArgs.push('--workers', String(fullConcurrency));
|
||||
// v0.31.2: thread strategy through so code-strategy first sync
|
||||
// actually enumerates code files (closes bug 1).
|
||||
@@ -3604,6 +3646,7 @@ async function performFullSync(
|
||||
strategy: opts.strategy,
|
||||
sourceId: opts.sourceId,
|
||||
exclude: opts.exclude,
|
||||
includeGitignored: opts.includeGitignored,
|
||||
slugRoot,
|
||||
// issue #1939: performFullSync owns the failure ledger + bookmark via the
|
||||
// shared gate below; don't let runImport double-record or write its own.
|
||||
@@ -3716,7 +3759,10 @@ async function performFullSync(
|
||||
// #774: scoped syncs store git-root-relative source_paths (slugRoot), so
|
||||
// relativize the walk to the same base — otherwise every page mismatches
|
||||
// and the mass-delete valve trips on a perfectly healthy scoped source.
|
||||
const currentFiles = collectSyncableFiles(syncScopeRoot, { strategy: opts.strategy ?? 'markdown' })
|
||||
const currentFiles = collectSyncableFiles(syncScopeRoot, {
|
||||
strategy: opts.strategy ?? 'markdown',
|
||||
includeGitignored: opts.includeGitignored,
|
||||
})
|
||||
.map(abs => relative(slugRoot ?? syncScopeRoot, abs));
|
||||
const rows = await engine.executeRaw<{ slug: string; source_path: string | null }>(
|
||||
`SELECT slug, source_path FROM pages WHERE source_id = $1 AND source_path IS NOT NULL AND deleted_at IS NULL`,
|
||||
@@ -4097,6 +4143,9 @@ Options:
|
||||
subdirectory directly as --repo also works.
|
||||
--exclude <glob> Exclude files matching the glob from sync (repeatable;
|
||||
matched against the scope-relative path).
|
||||
--include-gitignored Include otherwise-syncable files matched by .gitignore.
|
||||
Forces a full filesystem walk so periodic syncs see
|
||||
ignored untracked content.
|
||||
--dry-run Show what would be synced without writing.
|
||||
--skip-failed Acknowledge previously-recorded sync failures so
|
||||
the bookmark can advance past unparseable files.
|
||||
@@ -4120,12 +4169,22 @@ Options:
|
||||
connections per wave ≈ parallel × workers × 2
|
||||
(per-file pool) + parent pool. Pass --parallel 1
|
||||
to force serial.
|
||||
--missing-path M (with --all) What to do when a source's local_path
|
||||
does not exist on this machine: 'fail' (default —
|
||||
loud, current behavior) or 'skip' (classify as
|
||||
skipped_missing_path: ⊘ in the aggregate, excluded
|
||||
from error_count and the rc=1 gate). Use skip on
|
||||
brains whose sources were registered from more
|
||||
than one machine.
|
||||
--json Emit a structured JSON envelope on stdout
|
||||
({schema_version: 1, sources, parallel,
|
||||
ok_count, error_count}). Human banners route to
|
||||
stderr so '--json | jq' parses cleanly.
|
||||
Exit codes: 0 = all sources ok, 1 = any error,
|
||||
2 = cost-prompt-not-confirmed.
|
||||
ok_count, error_count, skipped_count}). Sources
|
||||
skipped by --missing-path skip appear with
|
||||
status 'skipped_missing_path' and their
|
||||
local_path. Human banners route to stderr so
|
||||
'--json | jq' parses cleanly.
|
||||
Exit codes: 0 = all sources ok or skipped,
|
||||
1 = any error, 2 = cost-prompt-not-confirmed.
|
||||
--yes Accept any interactive prompts (CI / non-TTY).
|
||||
|
||||
See also:
|
||||
@@ -4147,7 +4206,21 @@ See also:
|
||||
const skipFailed = args.includes('--skip-failed');
|
||||
const retryFailed = args.includes('--retry-failed');
|
||||
const noSchemaPack = args.includes('--no-schema-pack'); // v0.41.37.0 #1569
|
||||
const includeGitignored = args.includes('--include-gitignored');
|
||||
const syncAll = args.includes('--all');
|
||||
let missingPathMode: MissingPathMode = 'fail';
|
||||
try {
|
||||
missingPathMode = parseMissingPathMode(args);
|
||||
} catch (e) {
|
||||
console.error(e instanceof Error ? e.message : String(e));
|
||||
process.exit(2);
|
||||
}
|
||||
if (missingPathMode !== 'fail' && !syncAll) {
|
||||
// Single-source sync on a missing path should stay loud — an explicit
|
||||
// `--source X` naming an absent checkout is an operator error, not a
|
||||
// multi-machine artifact. Warn instead of silently ignoring the flag.
|
||||
console.error('[gbrain] WARN: --missing-path only applies to `sync --all`; ignored here.');
|
||||
}
|
||||
const jsonOut = args.includes('--json');
|
||||
const yesFlag = args.includes('--yes');
|
||||
// v0.41.6.0 D3: lock-recovery flags. --break-lock (safe) verifies the
|
||||
@@ -4403,7 +4476,7 @@ See also:
|
||||
if (!noEmbed) {
|
||||
const mode = willEmbedSynchronously({ v2Enabled, serialFlag, noEmbed });
|
||||
const gate = await runInlineCostGate(engine, {
|
||||
sources, mode, dryRun, jsonOut, yesFlag, full, label: 'sync --all',
|
||||
sources, mode, dryRun, jsonOut, yesFlag, full, includeGitignored, label: 'sync --all',
|
||||
});
|
||||
if (gate.action === 'stop') return;
|
||||
autoDeferEmbeds = gate.autoDeferEmbeds;
|
||||
@@ -4434,14 +4507,40 @@ See also:
|
||||
writeHuman(`Skipping ${disabledCount} disabled source(s).`);
|
||||
}
|
||||
|
||||
if (activeSources.length === 0) {
|
||||
// --missing-path skip: classify sources whose checkout is not on this
|
||||
// machine instead of failing them (see parseMissingPathMode's rationale).
|
||||
// Under the default 'fail' this is a no-op and behavior is unchanged.
|
||||
let skippedMissingPath: typeof activeSources = [];
|
||||
let runnableSources = activeSources;
|
||||
if (missingPathMode === 'skip') {
|
||||
const parts = partitionMissingPathSources(activeSources, existsSync);
|
||||
runnableSources = parts.runnable;
|
||||
skippedMissingPath = parts.missing;
|
||||
for (const src of skippedMissingPath) {
|
||||
writeHuman(` ⊘ ${src.name}: skipped — local_path not present on this host (${src.local_path})`);
|
||||
}
|
||||
if (skippedMissingPath.length > 0) {
|
||||
writeHuman(`Skipped ${skippedMissingPath.length} source(s) whose local_path is not present on this host (--missing-path skip).`);
|
||||
}
|
||||
}
|
||||
|
||||
if (runnableSources.length === 0) {
|
||||
if (jsonOut) {
|
||||
console.log(JSON.stringify({
|
||||
schema_version: 1,
|
||||
sources: [],
|
||||
sources: skippedMissingPath
|
||||
.slice()
|
||||
.sort((a, b) => a.id.localeCompare(b.id))
|
||||
.map((s) => ({
|
||||
source_id: s.id,
|
||||
name: s.name,
|
||||
status: 'skipped_missing_path',
|
||||
local_path: s.local_path,
|
||||
})),
|
||||
parallel: 0,
|
||||
ok_count: 0,
|
||||
error_count: 0,
|
||||
skipped_count: skippedMissingPath.length,
|
||||
}));
|
||||
}
|
||||
return;
|
||||
@@ -4451,11 +4550,20 @@ See also:
|
||||
type PerSourceResult = {
|
||||
sourceId: string;
|
||||
sourceName: string;
|
||||
status: 'ok' | 'error';
|
||||
status: 'ok' | 'error' | 'skipped_missing_path';
|
||||
result?: SyncResult;
|
||||
error?: string;
|
||||
localPath?: string;
|
||||
};
|
||||
const perSourceResults: PerSourceResult[] = [];
|
||||
for (const src of skippedMissingPath) {
|
||||
perSourceResults.push({
|
||||
sourceId: src.id,
|
||||
sourceName: src.name,
|
||||
status: 'skipped_missing_path',
|
||||
localPath: src.local_path ?? undefined,
|
||||
});
|
||||
}
|
||||
|
||||
// #1633 (Part B): one shared SIGINT controller for the whole --all fan-out.
|
||||
// process-cleanup.ts doesn't own SIGINT, so without this Ctrl-C hard-cuts the
|
||||
@@ -4501,6 +4609,7 @@ See also:
|
||||
noEmbed: effectiveNoEmbed,
|
||||
noExtract,
|
||||
skipFailed, retryFailed, noSchemaPack,
|
||||
includeGitignored,
|
||||
sourceId: src.id,
|
||||
strategy: cfg.strategy,
|
||||
concurrency,
|
||||
@@ -4564,7 +4673,7 @@ See also:
|
||||
};
|
||||
|
||||
const parallelEligible =
|
||||
v2Enabled && !serialFlag && engine.kind !== 'pglite' && activeSources.length > 1;
|
||||
v2Enabled && !serialFlag && engine.kind !== 'pglite' && runnableSources.length > 1;
|
||||
|
||||
// v0.42.42.0 (#2139, D13C): the v0.40.6.0 (D15) refusal of --skip-failed /
|
||||
// --retry-failed under parallel sync is LIFTED. It existed because the
|
||||
@@ -4578,7 +4687,7 @@ See also:
|
||||
// know how the run was actually dispatched. 1 in the serial fallback,
|
||||
// capped at min(sourceCount, --max-sources, 8) in the parallel path.
|
||||
const effectiveParallel = parallelEligible
|
||||
? Math.min(activeSources.length, maxSources ?? 8)
|
||||
? Math.min(runnableSources.length, maxSources ?? 8)
|
||||
: 1;
|
||||
|
||||
process.on('SIGINT', onAllSigint);
|
||||
@@ -4602,8 +4711,8 @@ See also:
|
||||
);
|
||||
}
|
||||
|
||||
writeHuman(`\nParallel sync: ${activeSources.length} sources, ${cap} concurrent workers.\n`);
|
||||
const results = await pMapAllSettled(activeSources, cap, async (src) => {
|
||||
writeHuman(`\nParallel sync: ${runnableSources.length} sources, ${cap} concurrent workers.\n`);
|
||||
const results = await pMapAllSettled(runnableSources, cap, async (src) => {
|
||||
const r = await runOne(src);
|
||||
return { name: src.name, result: r };
|
||||
});
|
||||
@@ -4611,7 +4720,7 @@ See also:
|
||||
writeHuman('\n--- sync --all aggregate ---');
|
||||
for (let i = 0; i < results.length; i++) {
|
||||
const r = results[i];
|
||||
const src = activeSources[i];
|
||||
const src = runnableSources[i];
|
||||
if (r.status === 'fulfilled') {
|
||||
writeHuman(` ✓ ${src.name}: ${r.value.result.status} (added=${r.value.result.added}, modified=${r.value.result.modified}, deleted=${r.value.result.deleted})`);
|
||||
perSourceResults.push({
|
||||
@@ -4632,7 +4741,7 @@ See also:
|
||||
}
|
||||
}
|
||||
} else {
|
||||
for (const src of activeSources) {
|
||||
for (const src of runnableSources) {
|
||||
writeHuman(`\n--- Syncing source: ${src.name} ---`);
|
||||
try {
|
||||
const result = await runOne(src);
|
||||
@@ -4672,6 +4781,7 @@ See also:
|
||||
source_id: r.sourceId,
|
||||
name: r.sourceName,
|
||||
status: r.status,
|
||||
...(r.localPath ? { local_path: r.localPath } : {}),
|
||||
...(r.result ? {
|
||||
sync_status: r.result.status,
|
||||
// #3068: surface the partial reason (e.g. pull_failed) so JSON
|
||||
@@ -4691,6 +4801,7 @@ See also:
|
||||
parallel: effectiveParallel,
|
||||
ok_count: okCount,
|
||||
error_count: errCount,
|
||||
skipped_count: perSourceResults.filter((r) => r.status === 'skipped_missing_path').length,
|
||||
}));
|
||||
}
|
||||
|
||||
@@ -4725,7 +4836,7 @@ See also:
|
||||
const singleSourceInterrupt = new AbortController();
|
||||
const onSingleSourceSigint = () => { try { singleSourceInterrupt.abort(new Error('SIGINT')); } catch { /* */ } };
|
||||
const opts: SyncOpts = {
|
||||
repoPath, dryRun, full, noPull, noEmbed, noExtract, skipFailed, retryFailed, noSchemaPack, sourceId,
|
||||
repoPath, dryRun, full, noPull, noEmbed, noExtract, skipFailed, retryFailed, noSchemaPack, includeGitignored, sourceId,
|
||||
strategy: strategyArg, concurrency,
|
||||
srcSubpath,
|
||||
exclude: excludePatterns.length > 0 ? excludePatterns : undefined,
|
||||
@@ -4754,7 +4865,7 @@ See also:
|
||||
chunker_version: gateRows[0].chunker_version,
|
||||
}];
|
||||
const gate = await runInlineCostGate(engine, {
|
||||
sources: gateSources, mode: 'inline', dryRun: false, jsonOut, yesFlag, full, label: 'sync',
|
||||
sources: gateSources, mode: 'inline', dryRun: false, jsonOut, yesFlag, full, includeGitignored, label: 'sync',
|
||||
});
|
||||
if (gate.action === 'stop') return;
|
||||
if (gate.autoDeferEmbeds) {
|
||||
@@ -4887,6 +4998,63 @@ See also:
|
||||
}
|
||||
}
|
||||
|
||||
/** Mode for `sync --all --missing-path`: what to do when a source's
|
||||
* local_path does not exist on this machine. */
|
||||
export type MissingPathMode = 'fail' | 'skip';
|
||||
|
||||
/**
|
||||
* Parse `--missing-path <fail|skip>` (default: fail).
|
||||
*
|
||||
* Why the flag exists: `sources.local_path` is machine-specific state in a
|
||||
* brain-wide table. Any brain whose sources were registered from more than
|
||||
* one machine — or a sanctioned setup mid-migration (topologies.md Topology 2,
|
||||
* or the system-of-record git flow before every repo is cloned here) — has
|
||||
* sources whose checkout simply is not present on the machine running
|
||||
* `sync --all`. Each used to surface as a hard failure ("Not a git
|
||||
* repository: <path>") and force rc=1 on every run; on one observed fleet
|
||||
* that was 12 phantom failures per hour, which trains operators to ignore
|
||||
* the exit code.
|
||||
*
|
||||
* The DEFAULT stays `fail`: on a single-machine brain a missing local_path
|
||||
* usually means an unmounted volume or a deleted checkout, and silently
|
||||
* skipping it would hide real data loss. Skip is an explicit opt-in.
|
||||
*
|
||||
* Throws on a bad/absent value with a paste-ready hint (caller converts to
|
||||
* stderr + exit 2, same as other flag-misuse exits).
|
||||
*/
|
||||
export function parseMissingPathMode(args: string[]): MissingPathMode {
|
||||
const idx = args.indexOf('--missing-path');
|
||||
if (idx === -1) return 'fail';
|
||||
const val = args[idx + 1];
|
||||
if (val === 'fail' || val === 'skip') return val;
|
||||
throw new Error(
|
||||
`--missing-path expects 'fail' or 'skip', got: ${val ?? '(nothing)'}. ` +
|
||||
`Use \`--missing-path skip\` to classify sources whose local_path is not ` +
|
||||
`present on this machine as skipped instead of failed, or \`--missing-path ` +
|
||||
`fail\` (the default) to keep them loud.`,
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Partition `--all` sources by whether their local_path exists on THIS
|
||||
* machine. Classification is driven only by the injected predicate so tests
|
||||
* never touch the filesystem. A null local_path passes through as runnable —
|
||||
* pure-DB sources are already excluded from `--all` by the
|
||||
* `local_path IS NOT NULL` SELECT; this is defensive, not load-bearing.
|
||||
*/
|
||||
export function partitionMissingPathSources<T extends { local_path: string | null }>(
|
||||
sources: T[],
|
||||
pathExists: (p: string) => boolean,
|
||||
): { runnable: T[]; missing: T[] } {
|
||||
const runnable: T[] = [];
|
||||
const missing: T[] = [];
|
||||
for (const s of sources) {
|
||||
if (s.local_path != null && !pathExists(s.local_path)) missing.push(s);
|
||||
else runnable.push(s);
|
||||
}
|
||||
return { runnable, missing };
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.40.3.0 — resolve effective per-source concurrency for `sync --all`.
|
||||
*
|
||||
@@ -4964,6 +5132,7 @@ export async function syncOneSource(
|
||||
noSchemaPack?: boolean;
|
||||
/** v0.42.7 #1696: propagate --no-extract into every per-source sync. */
|
||||
noExtract?: boolean;
|
||||
includeGitignored?: boolean;
|
||||
},
|
||||
): Promise<{ result: SyncResult; log: string }> {
|
||||
const cfg = (src.config || {}) as { strategy?: 'markdown' | 'code' | 'auto' };
|
||||
@@ -4978,6 +5147,7 @@ export async function syncOneSource(
|
||||
skipFailed: shared.skipFailed,
|
||||
retryFailed: shared.retryFailed,
|
||||
noSchemaPack: shared.noSchemaPack,
|
||||
includeGitignored: shared.includeGitignored,
|
||||
sourceId: src.id,
|
||||
strategy: cfg.strategy,
|
||||
concurrency: shared.concurrency,
|
||||
|
||||
+15
-8
@@ -3,6 +3,7 @@
|
||||
*
|
||||
* Subcommands:
|
||||
* takes <slug> — list takes for a page
|
||||
* takes list — list all active takes (#2079)
|
||||
* takes search "<query>" [--who h] — keyword search across all takes
|
||||
* takes add <slug> ...flags — append a take (markdown + DB)
|
||||
* takes update <slug> --row N ...flags — update mutable fields
|
||||
@@ -129,11 +130,10 @@ function writeBody(path: string, body: string): void {
|
||||
// --- Subcommands ---
|
||||
|
||||
async function cmdList(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const slug = args[0];
|
||||
if (!slug) {
|
||||
console.error('Usage: gbrain takes <slug> [--json]');
|
||||
process.exit(1);
|
||||
}
|
||||
// #2079: slug is optional. `gbrain takes list` (no slug) lists ALL active
|
||||
// takes — CLI parity with the takes_list operation. A leading flag is not
|
||||
// a slug.
|
||||
const slug = args[0] && !args[0].startsWith('-') ? args[0] : undefined;
|
||||
const json = flagPresent(args, '--json');
|
||||
const holder = flagValue(args, '--who');
|
||||
const kind = flagValue(args, '--kind') as string | undefined;
|
||||
@@ -153,17 +153,19 @@ async function cmdList(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
return;
|
||||
}
|
||||
|
||||
const scope = slug ?? 'this brain';
|
||||
if (takes.length === 0) {
|
||||
console.log(`No takes on ${slug}.`);
|
||||
console.log(`No takes on ${scope}.`);
|
||||
return;
|
||||
}
|
||||
console.log(`# Takes on ${slug}\n`);
|
||||
console.log(`# Takes on ${scope}\n`);
|
||||
for (const t of takes) {
|
||||
const tag = t.active ? '' : ' [superseded]';
|
||||
const w = Number(t.weight).toFixed(2);
|
||||
const since = t.since_date ?? '';
|
||||
const src = t.source ? ` — ${t.source}` : '';
|
||||
console.log(`#${t.row_num} [${t.kind} • ${t.holder} • w=${w}${since ? ` • ${since}` : ''}]${tag}\n ${t.claim}${src}\n`);
|
||||
const where = slug ? '' : `${t.page_slug} `;
|
||||
console.log(`${where}#${t.row_num} [${t.kind} • ${t.holder} • w=${w}${since ? ` • ${since}` : ''}]${tag}\n ${t.claim}${src}\n`);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -555,6 +557,8 @@ export async function runTakes(engine: BrainEngine, args: string[]): Promise<voi
|
||||
Subcommands:
|
||||
takes <slug> [--json] [--who h] [--kind k] [--sort weight|since_date|created_at] [--expired]
|
||||
List takes for a page
|
||||
takes list [--json] [--who h] [--kind k] [--sort ...] [--expired]
|
||||
List all active takes across the brain (#2079)
|
||||
takes search "<query>" [--limit N] [--json]
|
||||
Keyword search across all takes
|
||||
takes add <slug> --claim "..." --kind <fact|take|bet|hunch> --who <holder>
|
||||
@@ -584,6 +588,9 @@ Common flags:
|
||||
const rest = args.slice(1);
|
||||
|
||||
switch (sub) {
|
||||
// #2079: `takes list` used to be parsed as page slug "list" and printed
|
||||
// "No takes on list." — reading exactly like an empty takes table.
|
||||
case 'list': return cmdList(engine, rest);
|
||||
case 'search': return cmdSearch(engine, rest);
|
||||
case 'add': return cmdAdd(engine, rest, await resolveTakesSourceId(engine));
|
||||
case 'update': return cmdUpdate(engine, rest, await resolveTakesSourceId(engine));
|
||||
|
||||
+30
-1
@@ -9,6 +9,7 @@ import type { BrainEngine } from '../core/engine.ts';
|
||||
import { runThink, persistSynthesis, stripGapsSection } from '../core/think/index.ts';
|
||||
import { loadConfig, isThinClient } from '../core/config.ts';
|
||||
import { callRemoteTool, unpackToolResult } from '../core/mcp-client.ts';
|
||||
import { canonicalLookup } from '../core/model-pricing.ts';
|
||||
|
||||
function flagValue(args: string[], name: string): string | undefined {
|
||||
const i = args.indexOf(name);
|
||||
@@ -20,6 +21,27 @@ function flagPresent(args: string[], name: string): boolean {
|
||||
return args.includes(name);
|
||||
}
|
||||
|
||||
/**
|
||||
* think's own cost was previously unsurfaced anywhere: not in this CLI's own
|
||||
* `--json` output, not in `budget_ledger`, and invisible to a wrapping
|
||||
* caller's own token accounting (the LLM call `think` makes is its own,
|
||||
* separate API call). Returns undefined when `usage` is absent (no-client/
|
||||
* stub paths, or a remote-MCP call that didn't forward it) or when the
|
||||
* resolved model has no entry in the canonical pricing table.
|
||||
*/
|
||||
export function computeThinkCostUsd(
|
||||
usage: { input_tokens: number; output_tokens: number } | undefined,
|
||||
modelUsed: string,
|
||||
): number | undefined {
|
||||
if (!usage) return undefined;
|
||||
const pricing = canonicalLookup(modelUsed);
|
||||
if (!pricing) return undefined;
|
||||
return Number(
|
||||
((usage.input_tokens / 1_000_000) * pricing.input
|
||||
+ (usage.output_tokens / 1_000_000) * pricing.output).toFixed(4),
|
||||
);
|
||||
}
|
||||
|
||||
export async function runThinkCli(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
if (args.length === 0 || args.includes('--help') || args.includes('-h')) {
|
||||
console.log(`Usage: gbrain think "<question>" [options]
|
||||
@@ -146,9 +168,15 @@ prints what would have been the input (exit 0).
|
||||
}
|
||||
}
|
||||
|
||||
const costUsd = computeThinkCostUsd(
|
||||
(result as { usage?: { input_tokens: number; output_tokens: number } }).usage,
|
||||
result.modelUsed,
|
||||
);
|
||||
|
||||
if (json) {
|
||||
console.log(JSON.stringify({
|
||||
...result,
|
||||
cost_usd: costUsd ?? null,
|
||||
saved_slug: savedSlug ?? null,
|
||||
evidence_inserted: evidenceInserted,
|
||||
}, null, 2));
|
||||
@@ -165,7 +193,8 @@ prints what would have been the input (exit 0).
|
||||
console.log('');
|
||||
}
|
||||
console.log('---');
|
||||
console.log(`Model: ${result.modelUsed} | Pages: ${result.pagesGathered} | Takes: ${result.takesGathered} | Graph: ${result.graphHits} | Citations: ${result.citations.length}`);
|
||||
const costSuffix = costUsd !== undefined ? ` | Cost: $${costUsd.toFixed(4)}` : '';
|
||||
console.log(`Model: ${result.modelUsed} | Pages: ${result.pagesGathered} | Takes: ${result.takesGathered} | Graph: ${result.graphHits} | Citations: ${result.citations.length}${costSuffix}`);
|
||||
if (savedSlug) {
|
||||
console.log(`Saved: ${savedSlug} (${evidenceInserted} evidence rows)`);
|
||||
}
|
||||
|
||||
@@ -462,6 +462,53 @@ export async function runPostUpgrade(args: string[] = []): Promise<void> {
|
||||
// Banner is cosmetic; never block the upgrade.
|
||||
}
|
||||
|
||||
// #3390: ZeroEntropy sunset notice. ZE announced (2026-07-24) that
|
||||
// its hosted endpoints — including /models/embed and /models/rerank —
|
||||
// shut down on 2026-09-04. Any brain resolving to a zeroentropyai:*
|
||||
// embedding model (including default-config brains that never set
|
||||
// one) loses SEMANTIC RETRIEVAL ENTIRELY on that date: the query
|
||||
// embedding uses the same endpoint, so existing vectors become
|
||||
// unqueryable. One-shot per install, gated by
|
||||
// `ze_sunset_notice_shown` (same pattern as the search-mode banner).
|
||||
try {
|
||||
const shown = await engine.getConfig('ze_sunset_notice_shown');
|
||||
const { DEFAULT_EMBEDDING_MODEL } = await import('../core/ai/defaults.ts');
|
||||
const effectiveModel = cfgSchema.embedding_model ?? DEFAULT_EMBEDDING_MODEL;
|
||||
const rerankerModel = await engine.getConfig('search.reranker.model');
|
||||
const onZeEmbedding = effectiveModel.startsWith('zeroentropyai:');
|
||||
const onZeReranker = !!rerankerModel?.startsWith('zeroentropyai:');
|
||||
if (shown !== 'true' && (onZeEmbedding || onZeReranker)) {
|
||||
console.log('');
|
||||
console.log('═══════════════════════════════════════════════════════════════');
|
||||
console.log('[gbrain] ACTION REQUIRED: ZeroEntropy hosted API sunsets 2026-09-04.');
|
||||
if (onZeEmbedding) {
|
||||
console.log(`[gbrain] This brain embeds with ${effectiveModel}. After the sunset,`);
|
||||
console.log('[gbrain] semantic retrieval STOPS WORKING (queries can no longer be');
|
||||
console.log('[gbrain] embedded against your existing vectors).');
|
||||
}
|
||||
if (onZeReranker) {
|
||||
console.log(`[gbrain] The reranker (${rerankerModel}) also sunsets; search falls`);
|
||||
console.log('[gbrain] back to unreranked ordering.');
|
||||
}
|
||||
console.log('═══════════════════════════════════════════════════════════════');
|
||||
console.log('');
|
||||
console.log('Migrate before the sunset (resumable; preview cost first):');
|
||||
console.log(' gbrain migrate embeddings --to <provider:model> --dry-run');
|
||||
console.log(' gbrain migrate embeddings --to <provider:model>');
|
||||
console.log('');
|
||||
console.log('Self-hosting zembed-1 (weights are Apache-2.0) via llama-server /');
|
||||
console.log('ollama also works and preserves your existing vectors — point');
|
||||
console.log('embedding at the local endpoint instead of migrating.');
|
||||
if (onZeReranker) {
|
||||
console.log('Reranker: gbrain config set search.reranker.enabled false (or pick another).');
|
||||
}
|
||||
console.log('');
|
||||
await engine.setConfig('ze_sunset_notice_shown', 'true');
|
||||
}
|
||||
} catch {
|
||||
// Banner is cosmetic; never block the upgrade.
|
||||
}
|
||||
|
||||
// PR1: skill-catalog publish consent. New installs default ON at
|
||||
// `gbrain init`; EXISTING installs stay OFF (default-OFF runtime = no
|
||||
// silent capability grant on upgrade) until the owner opts in HERE.
|
||||
|
||||
@@ -44,6 +44,13 @@ export function buildGatewayConfig(c: GBrainConfig): AIGatewayConfig {
|
||||
// multimodal/image embeds despite config.json looking complete. process.env
|
||||
// still wins via the later spread.
|
||||
if (c.voyage_api_key) envFromConfig.VOYAGE_API_KEY = c.voyage_api_key;
|
||||
// Azure OpenAI (keyless/Entra): fold the non-secret endpoint/deployment + the
|
||||
// Entra opt-in into the gateway env so the azure-openai recipe works in any
|
||||
// shell (incl. non-interactive agent shells). The bearer token is minted at
|
||||
// request time via `az`; no secret is stored in config.json.
|
||||
if (c.azure_openai_endpoint) envFromConfig.AZURE_OPENAI_ENDPOINT = c.azure_openai_endpoint;
|
||||
if (c.azure_openai_deployment) envFromConfig.AZURE_OPENAI_DEPLOYMENT = c.azure_openai_deployment;
|
||||
if (c.azure_openai_use_entra) envFromConfig.AZURE_OPENAI_USE_ENTRA = c.azure_openai_use_entra;
|
||||
|
||||
// v0.32 codex finding #4+#5 fix: thread local-server _BASE_URL env vars
|
||||
// into base_urls so the gateway hits the user's configured port. Without
|
||||
|
||||
+148
-17
@@ -52,6 +52,7 @@ import {
|
||||
openrouterRequiresExplicitPromptCache,
|
||||
} from './recipes/openrouter.ts';
|
||||
import { resolveModel, TIER_DEFAULTS } from '../model-config.ts';
|
||||
import { parseLlmJson } from '../llm-json.ts';
|
||||
import type { BrainEngine } from '../engine.ts';
|
||||
import { dimsProviderOptions } from './dims.ts';
|
||||
import { hasAnthropicKey } from './anthropic-key.ts';
|
||||
@@ -451,6 +452,20 @@ export function resolveNativeBaseUrl(
|
||||
return /\/v1$/.test(trimmed) ? trimmed : `${trimmed}/v1`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether an openai-compatible recipe's backend honors OpenAI structured
|
||||
* outputs. Threaded into `createOpenAICompatible`'s `supportsStructuredOutputs`
|
||||
* at the chat + expansion build sites, and consulted by `expand()` to pick the
|
||||
* strict `generateObject` path over the schemaless text path. Single source of
|
||||
* truth read from the chat touchpoint: the backend serves both chat and
|
||||
* expansion, so the capability is declared once.
|
||||
*
|
||||
* @internal exported for tests.
|
||||
*/
|
||||
export function recipeSupportsStructuredOutputs(recipe: Recipe): boolean {
|
||||
return recipe.touchpoints.chat?.supports_structured_outputs === true;
|
||||
}
|
||||
|
||||
/** Configure the gateway. Called by cli.ts#connectEngine. Clears cached models. */
|
||||
export function configureGateway(config: AIGatewayConfig): void {
|
||||
_config = {
|
||||
@@ -627,8 +642,42 @@ function warnRecipesMissingBatchTokens(): void {
|
||||
}
|
||||
}
|
||||
|
||||
/** Reset (for tests). */
|
||||
export function resetGateway(): void {
|
||||
/**
|
||||
* Test-only reset baseline (#3554). The bunfig preload
|
||||
* (`test/helpers/legacy-embedding-preload.ts`) pins the gateway to the legacy
|
||||
* OpenAI/1536 config at process start, but `resetGateway()` used to wipe that
|
||||
* pin to `_config = null`. The next test file's engine connect then
|
||||
* reconfigured from the SHIPPED default (zembed-1 @ 1280) and every 1536-d
|
||||
* fixture in that file exploded with `expected 1280 dimensions, not 1536` —
|
||||
* a cross-file mine whose placement depended on shard bin-packing.
|
||||
*
|
||||
* When a baseline factory is registered, `resetGateway()` means "back to the
|
||||
* test baseline" instead of "unconfigured": it clears everything as before,
|
||||
* then re-applies the factory's config via `configureGateway()`. A factory
|
||||
* (not a frozen config) so each re-application captures fresh
|
||||
* `process.env`, matching the preload's original `applyLegacy()` semantics.
|
||||
*
|
||||
* Production is untouched: nothing in `src/` calls `resetGateway()` or this
|
||||
* setter, so in production the baseline is never registered and
|
||||
* `resetGateway()` still fully unconfigures. Same `__*ForTests` seam
|
||||
* convention as `__setEmbedTransportForTests` above.
|
||||
*/
|
||||
let _resetBaseline: (() => AIGatewayConfig) | null = null;
|
||||
|
||||
/**
|
||||
* Register (or clear, with `null`) the config factory that `resetGateway()`
|
||||
* re-applies. Called once by the bunfig test preload.
|
||||
*
|
||||
* @internal exported for tests; not part of the public gateway API.
|
||||
*/
|
||||
export function __setGatewayResetBaselineForTests(
|
||||
factory: (() => AIGatewayConfig) | null,
|
||||
): void {
|
||||
_resetBaseline = factory;
|
||||
}
|
||||
|
||||
/** Clear every piece of module state. Shared by both reset flavors. */
|
||||
function clearGatewayState(): void {
|
||||
_config = null;
|
||||
_modelCache.clear();
|
||||
_shrinkState.clear();
|
||||
@@ -640,6 +689,33 @@ export function resetGateway(): void {
|
||||
_extendedModels.clear();
|
||||
}
|
||||
|
||||
/**
|
||||
* Reset (for tests). Clears all module state (config, model cache, shrink
|
||||
* state, transports, warned recipes, extended models), then — if a test
|
||||
* baseline is registered — re-applies it so the gateway returns to the
|
||||
* process-wide test default instead of an unconfigured limbo (#3554).
|
||||
*/
|
||||
export function resetGateway(): void {
|
||||
clearGatewayState();
|
||||
// configureGateway re-clears _modelCache/_shrinkState/_extendedModels and
|
||||
// registers the baseline's models; transports are NOT touched by it, so a
|
||||
// stale test transport can never leak back in through this path.
|
||||
if (_resetBaseline) configureGateway(_resetBaseline());
|
||||
}
|
||||
|
||||
/**
|
||||
* Reset AND stay unconfigured, ignoring any registered baseline. For the
|
||||
* handful of tests that assert genuine no-gateway behavior
|
||||
* (`no_gateway_config` diagnosis, `isAvailable() === false`, graceful
|
||||
* degradation paths). The preload's per-test beforeEach restores the
|
||||
* baseline before the next test, so this cannot leak across tests.
|
||||
*
|
||||
* @internal exported for tests; not part of the public gateway API.
|
||||
*/
|
||||
export function __unconfigureGatewayForTests(): void {
|
||||
clearGatewayState();
|
||||
}
|
||||
|
||||
/**
|
||||
* Test-only seam. Replaces the function the gateway calls to embed a
|
||||
* sub-batch. Pass `null` to restore the real `embedMany` from the AI SDK.
|
||||
@@ -2418,6 +2494,7 @@ function instantiateExpansion(recipe: Recipe, modelId: string, cfg: AIGatewayCon
|
||||
baseURL: compat.baseURL,
|
||||
...(compat.fetch ? { fetch: compat.fetch } : {}),
|
||||
...auth,
|
||||
supportsStructuredOutputs: recipeSupportsStructuredOutputs(recipe),
|
||||
}).languageModel(modelId);
|
||||
}
|
||||
}
|
||||
@@ -2427,6 +2504,20 @@ const ExpansionSchema = z.object({
|
||||
queries: z.array(z.string()).min(1).max(5),
|
||||
});
|
||||
|
||||
/**
|
||||
* Recover expansion queries from a schemaless model response. Used by the
|
||||
* openai-compatible expansion paths: a tolerant JSON decode plus schema
|
||||
* validation pulls the `queries` array out of the model's text (the prompt
|
||||
* pins it to a bare JSON object). Returns null when the text carries no valid
|
||||
* `{ queries: string[] }` object.
|
||||
*
|
||||
* @internal exported for tests.
|
||||
*/
|
||||
export function parseExpansionResponse(text: string): string[] | null {
|
||||
const parsed = ExpansionSchema.safeParse(parseLlmJson<unknown>(text));
|
||||
return parsed.success ? parsed.data.queries : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Expand a search query into up to 4 related queries.
|
||||
* Returns the original query PLUS expansions. On failure, returns just the original.
|
||||
@@ -2443,24 +2534,63 @@ export async function expand(query: string): Promise<string[]> {
|
||||
metadata: { query_chars: query.length },
|
||||
});
|
||||
|
||||
const expansionPrompt = [
|
||||
'Rewrite the search query below into 3-4 different, related queries that would help find relevant documents. Respond with a JSON object in exactly this shape: {"queries": ["rewrite1", "rewrite2", "rewrite3"]}. The JSON key MUST be exactly "queries" (not "rewrites" or any other variation).',
|
||||
'Return ONLY the JSON object. Do NOT include the original query in the result.',
|
||||
'Each rewrite should emphasize different aspects, synonyms, or framings.',
|
||||
'',
|
||||
`Query: ${query}`,
|
||||
].join('\n');
|
||||
|
||||
try {
|
||||
const { model, recipe, modelId } = await resolveExpansionProvider(getExpansionModel());
|
||||
const result = await generateObject({
|
||||
model,
|
||||
schema: ExpansionSchema,
|
||||
// v0.42.20.0 (codex P0) — expansion had NO abortSignal; same stalled-socket
|
||||
// class as chat. Default the chat timeout.
|
||||
abortSignal: withDefaultTimeout(undefined, AI_CHAT_TIMEOUT_MS),
|
||||
prompt: [
|
||||
'Rewrite the search query below into 3-4 different, related queries that would help find relevant documents.',
|
||||
'Return ONLY the JSON object. Do NOT include the original query in the result.',
|
||||
'Each rewrite should emphasize different aspects, synonyms, or framings.',
|
||||
'',
|
||||
`Query: ${query}`,
|
||||
].join('\n'),
|
||||
});
|
||||
|
||||
const expansions = result.object?.queries ?? [];
|
||||
let expansions: string[];
|
||||
|
||||
// Schemaless text path for openai-compatible backends whose structured-output
|
||||
// support is unknown: the AI SDK can't send a json_schema response_format
|
||||
// there, so generateObject would warn and silently degrade. generateText + a
|
||||
// tolerant parse recovers the queries instead. Fresh abortSignal per call.
|
||||
const viaText = async (): Promise<string[]> => {
|
||||
const { text } = await generateText({
|
||||
model,
|
||||
abortSignal: withDefaultTimeout(undefined, AI_CHAT_TIMEOUT_MS),
|
||||
prompt: expansionPrompt,
|
||||
});
|
||||
return parseExpansionResponse(text) ?? [];
|
||||
};
|
||||
|
||||
if (recipe.implementation !== 'openai-compatible') {
|
||||
// Native providers (Anthropic, OpenAI, Google) support generateObject's
|
||||
// structured output natively — unchanged path.
|
||||
const result = await generateObject({
|
||||
model,
|
||||
schema: ExpansionSchema,
|
||||
abortSignal: withDefaultTimeout(undefined, AI_CHAT_TIMEOUT_MS),
|
||||
prompt: expansionPrompt,
|
||||
});
|
||||
expansions = result.object?.queries ?? [];
|
||||
} else if (recipeSupportsStructuredOutputs(recipe)) {
|
||||
// openai-compatible backend that honors strict json_schema: request the
|
||||
// schema (strict validation), and fall back to the text path if it is
|
||||
// rejected at call time so a mis-declared capability never drops expansion.
|
||||
try {
|
||||
const result = await generateObject({
|
||||
model,
|
||||
schema: ExpansionSchema,
|
||||
abortSignal: withDefaultTimeout(undefined, AI_CHAT_TIMEOUT_MS),
|
||||
prompt: expansionPrompt,
|
||||
});
|
||||
expansions = result.object?.queries ?? [];
|
||||
} catch {
|
||||
expansions = await viaText();
|
||||
}
|
||||
} else {
|
||||
// openai-compatible backend, structured-output support unknown: skip the
|
||||
// json_schema attempt entirely (no SDK warning, no silent degradation).
|
||||
expansions = await viaText();
|
||||
}
|
||||
|
||||
// Deduplicate + include the original query
|
||||
const seen = new Set<string>();
|
||||
const all = [query, ...expansions].filter(q => {
|
||||
@@ -2926,6 +3056,7 @@ function instantiateChat(recipe: Recipe, modelId: string, cfg: AIGatewayConfig):
|
||||
baseURL: compat.baseURL,
|
||||
...(compat.fetch ? { fetch: compat.fetch } : {}),
|
||||
...auth,
|
||||
supportsStructuredOutputs: recipeSupportsStructuredOutputs(recipe),
|
||||
}).languageModel(modelId);
|
||||
}
|
||||
default:
|
||||
|
||||
@@ -144,3 +144,35 @@ export function assertTouchpoint(
|
||||
export function knownProviderIds(): string[] {
|
||||
return [...RECIPES.keys()];
|
||||
}
|
||||
|
||||
/**
|
||||
* Native embedding width for `modelId` under `recipe`.
|
||||
*
|
||||
* Resolution: the recipe's `model_dims` entry for this model, else the
|
||||
* recipe-wide `default_dims`. Returns 0 when neither is known (the
|
||||
* user-provided-model recipes declare `default_dims: 0` to force an explicit
|
||||
* `--embedding-dimensions`), so callers keep their existing falsy checks.
|
||||
*
|
||||
* Accepts a bare model id (`bge-m3`) or a qualified one (`ollama:bge-m3`);
|
||||
* the provider prefix is stripped before lookup so call sites can pass
|
||||
* whichever they hold.
|
||||
*
|
||||
* Fixes #2051: a recipe-wide default silently picked 768 for every Ollama
|
||||
* model, so `init --embedding-model ollama:bge-m3` built a 768-wide column
|
||||
* for a model that emits 1024 and only failed at first insert.
|
||||
*/
|
||||
export function embeddingDimsForModel(
|
||||
recipe: Recipe,
|
||||
modelId: string | undefined,
|
||||
): number {
|
||||
const tp = recipe.touchpoints.embedding;
|
||||
if (!tp) return 0;
|
||||
if (!modelId) return tp.default_dims ?? 0;
|
||||
// Strip a leading `provider:` so both forms resolve. Slash-form ids
|
||||
// (openrouter nested) are left intact — they're the model id.
|
||||
const colon = modelId.indexOf(':');
|
||||
const bare = colon === -1 ? modelId : modelId.slice(colon + 1);
|
||||
const declared = tp.model_dims?.[bare];
|
||||
if (typeof declared === 'number' && declared > 0) return declared;
|
||||
return tp.default_dims ?? 0;
|
||||
}
|
||||
|
||||
@@ -24,6 +24,7 @@ export const anthropic: Recipe = {
|
||||
chat: {
|
||||
models: [
|
||||
'claude-fable-5',
|
||||
'claude-opus-5',
|
||||
'claude-opus-4-8',
|
||||
'claude-opus-4-7',
|
||||
'claude-sonnet-5',
|
||||
|
||||
@@ -1,8 +1,60 @@
|
||||
import type { Recipe } from '../types.ts';
|
||||
import { AIConfigError } from '../errors.ts';
|
||||
import { execSync } from 'node:child_process';
|
||||
|
||||
const DEFAULT_API_VERSION = '2024-10-21'; // stable Azure OpenAI version as of 2026-05
|
||||
|
||||
// Entra (keyless) auth support. Subscriptions that enforce disableLocalAuth via
|
||||
// Azure Policy reject api-key auth, so when no AZURE_OPENAI_API_KEY is present
|
||||
// (or AZURE_OPENAI_USE_ENTRA=1) we mint a short-lived AAD bearer token via the
|
||||
// Azure CLI and cache it. resolveAuth is synchronous, so execSync is the seam.
|
||||
// The caller needs `az login` + the "Cognitive Services OpenAI User" role.
|
||||
let _entraToken: { token: string; fetchedAt: number } | null = null;
|
||||
const ENTRA_TOKEN_TTL_MS = 45 * 60 * 1000; // refresh well before the ~60-90min expiry
|
||||
|
||||
function fetchEntraToken(): string {
|
||||
const now = Date.now();
|
||||
if (_entraToken && now - _entraToken.fetchedAt < ENTRA_TOKEN_TTL_MS) {
|
||||
return _entraToken.token;
|
||||
}
|
||||
let token = '';
|
||||
try {
|
||||
token = execSync(
|
||||
'az account get-access-token --resource https://cognitiveservices.azure.com --query accessToken -o tsv',
|
||||
{ encoding: 'utf-8', stdio: ['ignore', 'pipe', 'ignore'], timeout: 30_000 },
|
||||
).trim();
|
||||
} catch {
|
||||
throw new AIConfigError(
|
||||
'Azure OpenAI (Entra/keyless): could not get an access token via `az account get-access-token`.',
|
||||
'Run `az login` and ensure your identity has the "Cognitive Services OpenAI User" role on the resource.',
|
||||
);
|
||||
}
|
||||
if (!token) {
|
||||
throw new AIConfigError(
|
||||
'Azure OpenAI (Entra/keyless): `az account get-access-token` returned an empty token.',
|
||||
'Run `az login` and verify the active subscription owns the Azure OpenAI resource.',
|
||||
);
|
||||
}
|
||||
_entraToken = { token, fetchedAt: now };
|
||||
return token;
|
||||
}
|
||||
|
||||
/** @internal test seam: pre-populate (or clear) the Entra token cache so unit
|
||||
* tests never shell out to `az`. */
|
||||
export function __setEntraTokenForTests(token: string | null): void {
|
||||
_entraToken = token === null ? null : { token, fetchedAt: Date.now() };
|
||||
}
|
||||
|
||||
/** Entra/keyless mode: EXPLICIT opt-in only (AZURE_OPENAI_USE_ENTRA=1 /
|
||||
* config azure_openai_use_entra). A missing api-key must NOT silently shell
|
||||
* out to `az` — that surprises CI boxes and every non-Azure-CLI environment,
|
||||
* and it broke the cross-recipe auth iron-rule test. Keyless subscriptions
|
||||
* (disableLocalAuth) set the flag; missing key without the flag keeps the
|
||||
* original loud AIConfigError. */
|
||||
function isEntraMode(env: Record<string, string | undefined>): boolean {
|
||||
return env.AZURE_OPENAI_USE_ENTRA === '1';
|
||||
}
|
||||
|
||||
/**
|
||||
* Azure OpenAI. The first recipe in v0.32 to exercise both seams:
|
||||
* - resolveAuth returns `{headerName: 'api-key', token: <key>}` instead of
|
||||
@@ -30,11 +82,13 @@ export const azureOpenAI: Recipe = {
|
||||
// base_url_default omitted: Azure URLs are env-templated only.
|
||||
auth_env: {
|
||||
required: [
|
||||
'AZURE_OPENAI_API_KEY',
|
||||
'AZURE_OPENAI_ENDPOINT',
|
||||
'AZURE_OPENAI_DEPLOYMENT',
|
||||
],
|
||||
optional: ['AZURE_OPENAI_API_VERSION'],
|
||||
// AZURE_OPENAI_API_KEY optional: when absent (or AZURE_OPENAI_USE_ENTRA=1)
|
||||
// the recipe uses a refreshing Entra/AAD bearer token via the Azure CLI,
|
||||
// required on subscriptions that enforce disableLocalAuth (keyless).
|
||||
optional: ['AZURE_OPENAI_API_KEY', 'AZURE_OPENAI_USE_ENTRA', 'AZURE_OPENAI_API_VERSION'],
|
||||
setup_url:
|
||||
'https://learn.microsoft.com/en-us/azure/ai-services/openai/quickstart',
|
||||
},
|
||||
@@ -54,17 +108,18 @@ export const azureOpenAI: Recipe = {
|
||||
},
|
||||
},
|
||||
resolveAuth(env) {
|
||||
const key = env.AZURE_OPENAI_API_KEY;
|
||||
if (!key) {
|
||||
throw new AIConfigError(
|
||||
`Azure OpenAI requires AZURE_OPENAI_API_KEY.`,
|
||||
'Get a key from your Azure portal: https://learn.microsoft.com/en-us/azure/ai-services/openai/quickstart',
|
||||
);
|
||||
// Entra/keyless mode: no api-key (disableLocalAuth) or opt-in via
|
||||
// AZURE_OPENAI_USE_ENTRA=1. Mint a refreshing AAD bearer token. Returning
|
||||
// an `Authorization: Bearer …` pair makes the gateway use the SDK's native
|
||||
// bearer path (it strips the prefix and re-adds it), so no double-auth.
|
||||
if (isEntraMode(env)) {
|
||||
return { headerName: 'Authorization', token: `Bearer ${fetchEntraToken()}` };
|
||||
}
|
||||
// Azure uses `api-key:` (no Bearer); the unified seam routes this
|
||||
// Key mode: Azure uses `api-key:` (no Bearer); the unified seam routes this
|
||||
// through `headers` instead of the SDK's apiKey field to avoid any
|
||||
// double-auth Authorization header sneaking in.
|
||||
return { headerName: 'api-key', token: key };
|
||||
// double-auth Authorization header sneaking in. The key is present here:
|
||||
// !isEntraMode(env) implies AZURE_OPENAI_API_KEY is set.
|
||||
return { headerName: 'api-key', token: env.AZURE_OPENAI_API_KEY! };
|
||||
},
|
||||
resolveOpenAICompatConfig(env) {
|
||||
const endpoint = env.AZURE_OPENAI_ENDPOINT?.replace(/\/+$/, '');
|
||||
@@ -82,6 +137,7 @@ export const azureOpenAI: Recipe = {
|
||||
);
|
||||
}
|
||||
const apiVersion = env.AZURE_OPENAI_API_VERSION ?? DEFAULT_API_VERSION;
|
||||
const entra = isEntraMode(env);
|
||||
const baseURL = `${endpoint}/openai/deployments/${deployment}`;
|
||||
// Custom fetch wrapper splices ?api-version=... onto every request.
|
||||
// Azure rejects requests without it.
|
||||
@@ -102,10 +158,23 @@ export const azureOpenAI: Recipe = {
|
||||
typeof input === 'string' || input instanceof URL
|
||||
? finalUrl
|
||||
: new Request(finalUrl, input as Request);
|
||||
if (entra) {
|
||||
// Entra mode: refresh the AAD bearer on every request. The gateway
|
||||
// caches model instances (auth is baked in at instantiation), so a
|
||||
// long-running process would otherwise send an expired token after
|
||||
// ~1h. fetchEntraToken()'s 45-min TTL cache keeps `az` invocations
|
||||
// rare; the override here keeps the header fresh.
|
||||
const headers = new Headers(
|
||||
init?.headers ??
|
||||
(typeof finalInput !== 'string' ? (finalInput as Request).headers : undefined),
|
||||
);
|
||||
headers.set('Authorization', `Bearer ${fetchEntraToken()}`);
|
||||
init = { ...init, headers };
|
||||
}
|
||||
return fetch(finalInput, init);
|
||||
}) as unknown as typeof fetch;
|
||||
return { baseURL, fetch: wrappedFetch };
|
||||
},
|
||||
setup_hint:
|
||||
'Azure portal → Azure OpenAI resource. Set AZURE_OPENAI_API_KEY, AZURE_OPENAI_ENDPOINT, AZURE_OPENAI_DEPLOYMENT. Optionally AZURE_OPENAI_API_VERSION (default 2024-10-21).',
|
||||
'Azure portal → Azure OpenAI resource. Set AZURE_OPENAI_ENDPOINT + AZURE_OPENAI_DEPLOYMENT, and either AZURE_OPENAI_API_KEY or keyless Entra auth (`az login` + "Cognitive Services OpenAI User" role; force with AZURE_OPENAI_USE_ENTRA=1). Optionally AZURE_OPENAI_API_VERSION (default 2024-10-21).',
|
||||
};
|
||||
|
||||
@@ -31,6 +31,11 @@ export const dashscope: Recipe = {
|
||||
// path. Conservative declaration so the gateway pre-splits before
|
||||
// hitting whatever undocumented server-side limit exists.
|
||||
max_batch_tokens: 8192,
|
||||
// DashScope's OpenAI-compat /embeddings endpoint rejects requests with
|
||||
// more than 10 input items (documented Model Studio cap). The gateway's
|
||||
// capBatchItems pre-split enforces this; max_batch_tokens above keeps
|
||||
// guarding aggregate token size. Concept from community PRs #2643/#2405.
|
||||
max_batch_items: 10,
|
||||
// text-embedding-v3 mixes English + CJK heavily; the tokenizer is
|
||||
// closer to Voyage density than OpenAI tiktoken for CJK-dominant
|
||||
// content. Conservative chars_per_token=2 leaves headroom.
|
||||
|
||||
@@ -1,9 +1,10 @@
|
||||
import type { Recipe } from '../types.ts';
|
||||
|
||||
/**
|
||||
* `deepseek-reasoner` returns its answer in a separate `reasoning_content`
|
||||
* field and leaves `content` empty/whitespace when the whole response was
|
||||
* reasoning. The AI SDK's openai-compatible adapter reads only `content`, so
|
||||
* DeepSeek's thinking mode (default on `deepseek-v4-flash`/`deepseek-v4-pro`;
|
||||
* formerly the `deepseek-reasoner` model, retired 2026-07-24) returns its
|
||||
* answer in a separate `reasoning_content` field and leaves `content`
|
||||
* empty/whitespace when the whole response was reasoning. The AI SDK's openai-compatible adapter reads only `content`, so
|
||||
* the model appears to answer with nothing. This transport shim promotes
|
||||
* `reasoning_content` into `content` when `content` is empty, before the
|
||||
* adapter parses the body. Fail-open: any error returns the original response.
|
||||
@@ -80,20 +81,25 @@ export const deepseek: Recipe = {
|
||||
// gateway's expansion path is a plain languageModel call). Without this
|
||||
// declaration an explicit `expansion_model: deepseek:...` silently
|
||||
// yields no expansion (#1135).
|
||||
// `deepseek-chat` / `deepseek-reasoner` were retired by DeepSeek on
|
||||
// 2026-07-24 (#1255); both map to `deepseek-v4-flash` (non-thinking /
|
||||
// thinking mode). Do not re-add the old names — the API 404s them.
|
||||
// openai-compat tier means user-configured legacy names still pass
|
||||
// validation locally; the provider rejects them at call time.
|
||||
expansion: {
|
||||
models: ['deepseek-chat'],
|
||||
models: ['deepseek-v4-flash'],
|
||||
cost_per_1m_tokens_usd: 0.14,
|
||||
price_last_verified: '2026-04-20',
|
||||
price_last_verified: '2026-07-27',
|
||||
},
|
||||
chat: {
|
||||
models: ['deepseek-chat', 'deepseek-reasoner'],
|
||||
models: ['deepseek-v4-flash', 'deepseek-v4-pro'],
|
||||
supports_tools: true,
|
||||
supports_subagent_loop: true,
|
||||
supports_prompt_cache: false,
|
||||
max_context_tokens: 128000,
|
||||
cost_per_1m_input_usd: 0.14, // deepseek-chat off-peak baseline
|
||||
max_context_tokens: 1_000_000,
|
||||
cost_per_1m_input_usd: 0.14, // deepseek-v4-flash cache-miss baseline
|
||||
cost_per_1m_output_usd: 0.28,
|
||||
price_last_verified: '2026-04-20',
|
||||
price_last_verified: '2026-07-27',
|
||||
},
|
||||
},
|
||||
setup_hint: 'Get an API key at https://platform.deepseek.com/api_keys, then `export DEEPSEEK_API_KEY=...`',
|
||||
|
||||
@@ -14,17 +14,29 @@ export const ollama: Recipe = {
|
||||
touchpoints: {
|
||||
embedding: {
|
||||
// #2271: modern local embed models added so assertTouchpoint accepts them.
|
||||
// Each carries its own native dim (qwen3-embed-8b=4096, arctic-l-v2=1024);
|
||||
// the recipe-wide default_dims below is only the nomic fallback, so users
|
||||
// of the larger models pass --embedding-dimensions (allowed via
|
||||
// trust_custom_dims). Per-model dims metadata is a tracked follow-up.
|
||||
models: [
|
||||
'nomic-embed-text',
|
||||
'mxbai-embed-large',
|
||||
'all-minilm',
|
||||
'qwen3-embed-8b',
|
||||
'snowflake-arctic-embed-l-v2',
|
||||
'bge-m3',
|
||||
],
|
||||
// #2051: per-model native dims. Ollama serves models spanning 384..4096,
|
||||
// so the recipe-wide default_dims below is only correct for nomic. Without
|
||||
// this map `init --embedding-model ollama:bge-m3` built a 768-wide column
|
||||
// for a model that emits 1024, and the mismatch only surfaced at first
|
||||
// insert. Resolved via `embeddingDimsForModel()`; unlisted models still
|
||||
// fall back to default_dims, and trust_custom_dims keeps an explicit
|
||||
// --embedding-dimensions override working for models not named here.
|
||||
model_dims: {
|
||||
'nomic-embed-text': 768,
|
||||
'mxbai-embed-large': 1024,
|
||||
'all-minilm': 384,
|
||||
'qwen3-embed-8b': 4096,
|
||||
'snowflake-arctic-embed-l-v2': 1024,
|
||||
'bge-m3': 1024,
|
||||
},
|
||||
default_dims: 768, // nomic-embed-text native dim
|
||||
trust_custom_dims: true, // #2271: local models carry varied native dims
|
||||
cost_per_1m_tokens_usd: 0,
|
||||
|
||||
@@ -180,6 +180,17 @@ export const openrouter: Recipe = {
|
||||
// to pre-split batches, NOT per-input. Per-input is enforced upstream.
|
||||
max_batch_tokens: 300_000,
|
||||
},
|
||||
// Expansion uses the same routed OpenAI-compatible language-model endpoint
|
||||
// as chat. Keep a small cheap/fast advisory set; the openai-compat tier
|
||||
// still accepts any user-configured OpenRouter provider/model ID.
|
||||
expansion: {
|
||||
models: [
|
||||
'anthropic/claude-haiku-4.5',
|
||||
'google/gemini-3-flash-preview',
|
||||
'deepseek/deepseek-chat',
|
||||
],
|
||||
price_last_verified: '2026-05-20',
|
||||
},
|
||||
chat: {
|
||||
// Curated entry points (verified against OR's catalog 2026-05-20). The
|
||||
// openai-compat tier does NOT enforce this list at runtime — users can
|
||||
@@ -231,6 +242,6 @@ export const openrouter: Recipe = {
|
||||
},
|
||||
},
|
||||
setup_hint:
|
||||
'Get an API key at https://openrouter.ai/settings/keys, then `export OPENROUTER_API_KEY=...` and use `openrouter:<provider>/<model>`. Optional overrides: OPENROUTER_BASE_URL (proxy), OPENROUTER_REFERER (attribution URL), OPENROUTER_TITLE (attribution name).',
|
||||
'Get an API key at https://openrouter.ai/settings/keys, then `export OPENROUTER_API_KEY=...` or set `openrouter_api_key` in ~/.gbrain/config.json and use `openrouter:<provider>/<model>`. Optional overrides: OPENROUTER_BASE_URL (proxy), OPENROUTER_REFERER (attribution URL), OPENROUTER_TITLE (attribution name).',
|
||||
compat: { fetch: openrouterCompatFetch },
|
||||
};
|
||||
|
||||
@@ -28,6 +28,21 @@ export type Implementation =
|
||||
export interface EmbeddingTouchpoint {
|
||||
models: string[];
|
||||
default_dims: number;
|
||||
/**
|
||||
* Per-model native dimensions, keyed by bare model id (no `provider:`
|
||||
* prefix). Consulted before `default_dims` when resolving schema width
|
||||
* for a specific model.
|
||||
*
|
||||
* Local recipes (ollama, llama-server) serve models with very different
|
||||
* native widths — nomic-embed-text is 768, bge-m3 and mxbai-embed-large
|
||||
* are 1024, qwen3-embed-8b is 4096. A single recipe-wide `default_dims`
|
||||
* silently picks the wrong width for every model except the one it was
|
||||
* chosen for, producing a schema that only fails at first insert (#2051).
|
||||
*
|
||||
* Partial by design: a model absent from this map falls back to
|
||||
* `default_dims`, so a recipe can declare only the models it knows.
|
||||
*/
|
||||
model_dims?: Readonly<Record<string, number>>;
|
||||
dims_options?: number[]; // for Matryoshka-aware providers
|
||||
cost_per_1m_tokens_usd?: number;
|
||||
price_last_verified?: string; // ISO date
|
||||
@@ -240,6 +255,17 @@ export interface ChatTouchpoint {
|
||||
* model family).
|
||||
*/
|
||||
supports_prompt_cache?: boolean | ((modelId: string) => boolean);
|
||||
/**
|
||||
* Backend honors OpenAI structured outputs (a strict `json_schema`
|
||||
* response_format). Threaded into `createOpenAICompatible`'s
|
||||
* `supportsStructuredOutputs` so query expansion's `generateObject` sends a
|
||||
* real schema (strict validation) instead of degrading to schemaless JSON.
|
||||
* Default false: an openai-compatible recipe may front arbitrary backends,
|
||||
* most of which lack strict json_schema support, so `expand()` routes them
|
||||
* through the schemaless text path. Opt in per recipe when the backend is
|
||||
* known to honor it.
|
||||
*/
|
||||
supports_structured_outputs?: boolean;
|
||||
max_context_tokens?: number;
|
||||
cost_per_1m_input_usd?: number;
|
||||
cost_per_1m_output_usd?: number;
|
||||
|
||||
+125
-11
@@ -40,6 +40,7 @@ export const LINKABLE_ENTITY_TYPES = ['person', 'company', 'organization', 'enti
|
||||
* types in.
|
||||
*/
|
||||
const MIN_NAME_LENGTH = 4;
|
||||
const MIN_CJK_NAME_LENGTH = 2;
|
||||
|
||||
/**
|
||||
* Built-in ignore list — common ambiguous tokens whose body-text mentions
|
||||
@@ -104,12 +105,12 @@ export interface FindMentionsOpts {
|
||||
// ============================================================
|
||||
|
||||
/**
|
||||
* Token-only tokenizer. Returns `[token, offset]` pairs for every
|
||||
* `[a-zA-Z0-9]+` run, lowercased. Non-ASCII (CJK, accented) is
|
||||
* deliberately not tokenized in v1 — entity gazetteer is English-dominant
|
||||
* in production today. Widening to `\p{L}+` is a future option once a
|
||||
* real CJK entity catalog appears (filed under TODO-1 + a TODO for
|
||||
* Unicode-aware tokenization).
|
||||
* Token-only tokenizer. Returns `[token, offset]` pairs.
|
||||
*
|
||||
* ASCII: each `[a-zA-Z0-9]+` run is a single token, lowercased.
|
||||
* CJK: each CJK character (Chinese/Japanese/Korean) is an individual
|
||||
* token, lowercased. This allows the normal maximal-munch scan path
|
||||
* to reach CJK gazetteer entries without a separate substring pass.
|
||||
*
|
||||
* Possessive "Acme's" tokenizes as ['acme', 's'] (single-quote breaks the
|
||||
* run) — single-word "Acme" lookup succeeds at offset 0; the trailing 's'
|
||||
@@ -127,18 +128,129 @@ function tokenizeForScan(text: string): ScannedToken[] {
|
||||
const out: ScannedToken[] = [];
|
||||
TOKEN_RE.lastIndex = 0;
|
||||
let m: RegExpExecArray | null;
|
||||
|
||||
// Collect ASCII token spans first.
|
||||
const asciiSpans: Array<{ start: number; end: number }> = [];
|
||||
while ((m = TOKEN_RE.exec(text)) !== null) {
|
||||
out.push({ text: m[0].toLowerCase(), offset: m.index, length: m[0].length });
|
||||
asciiSpans.push({ start: m.index, end: m.index + m[0].length });
|
||||
}
|
||||
|
||||
// Walk character-by-character: emit ASCII tokens at their start positions,
|
||||
// then emit individual CJK characters for non-ASCII positions that fall
|
||||
// outside ASCII token spans.
|
||||
let asciiIdx = 0;
|
||||
for (let i = 0; i < text.length;) {
|
||||
const cp = text.codePointAt(i) ?? 0;
|
||||
const isCJK = (cp >= 0x4e00 && cp <= 0x9fff) || (cp >= 0x3400 && cp <= 0x4dbf) ||
|
||||
(cp >= 0x3040 && cp <= 0x309f) || (cp >= 0x30a0 && cp <= 0x30ff) ||
|
||||
(cp >= 0xac00 && cp <= 0xd7af);
|
||||
|
||||
// Advance asciiIdx past any spans that end before or at i.
|
||||
while (asciiIdx < asciiSpans.length && asciiSpans[asciiIdx]!.end <= i) {
|
||||
asciiIdx++;
|
||||
}
|
||||
|
||||
// If position i is inside an ASCII token span, emit the full ASCII token
|
||||
// and jump past it.
|
||||
if (asciiIdx < asciiSpans.length && i >= asciiSpans[asciiIdx]!.start && i < asciiSpans[asciiIdx]!.end) {
|
||||
const span = asciiSpans[asciiIdx]!;
|
||||
const token = text.slice(span.start, span.end);
|
||||
out.push({ text: token.toLowerCase(), offset: span.start, length: token.length });
|
||||
i = span.end;
|
||||
asciiIdx++;
|
||||
continue;
|
||||
}
|
||||
|
||||
// CJK: emit as individual character token.
|
||||
if (isCJK) {
|
||||
const charLen = cp > 0xffff ? 2 : 1; // surrogate pair
|
||||
const charStr = text.slice(i, i + charLen);
|
||||
out.push({ text: charStr.toLowerCase(), offset: i, length: charLen });
|
||||
i += charLen;
|
||||
} else {
|
||||
i++;
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
function hasCJK(s: string): boolean {
|
||||
for (const ch of s) {
|
||||
const cp = ch.codePointAt(0) ?? 0;
|
||||
if ((cp >= 0x4e00 && cp <= 0x9fff) || (cp >= 0x3400 && cp <= 0x4dbf) ||
|
||||
(cp >= 0x3040 && cp <= 0x309f) || (cp >= 0x30a0 && cp <= 0x30ff) ||
|
||||
(cp >= 0xac00 && cp <= 0xd7af)) return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
function cjkCharCount(s: string): number {
|
||||
let count = 0;
|
||||
for (const ch of s) {
|
||||
const cp = ch.codePointAt(0) ?? 0;
|
||||
if ((cp >= 0x4e00 && cp <= 0x9fff) || (cp >= 0x3400 && cp <= 0x4dbf) ||
|
||||
(cp >= 0x3040 && cp <= 0x309f) || (cp >= 0x30a0 && cp <= 0x30ff) ||
|
||||
(cp >= 0xac00 && cp <= 0xd7af)) count++;
|
||||
}
|
||||
return count;
|
||||
}
|
||||
|
||||
/**
|
||||
* Tokenize a page title for gazetteer insertion.
|
||||
*
|
||||
* ASCII titles: standard `[a-zA-Z0-9]+` tokenization, lowercased.
|
||||
* CJK titles (no ASCII content): split into individual characters —
|
||||
* e.g. "纳瓦尔" → ["纳","瓦","尔"]. This allows normal multi-token
|
||||
* maximal-munch matching to work with character-level CJK tokens
|
||||
* produced by `tokenizeForScan`.
|
||||
* Mixed CJK+ASCII titles: ASCII parts tokenized normally, CJK parts
|
||||
* split into individual characters.
|
||||
*/
|
||||
function tokenizeTitle(title: string): string[] {
|
||||
const tokens: string[] = [];
|
||||
TOKEN_RE.lastIndex = 0;
|
||||
let m: RegExpExecArray | null;
|
||||
while ((m = TOKEN_RE.exec(title)) !== null) tokens.push(m[0].toLowerCase());
|
||||
return tokens;
|
||||
const hasAscii = TOKEN_RE.test(title);
|
||||
if (hasAscii) {
|
||||
// Mixed ASCII+CJK or pure ASCII: tokenize ASCII normally, then
|
||||
// append individual CJK characters in order.
|
||||
TOKEN_RE.lastIndex = 0;
|
||||
let m: RegExpExecArray | null;
|
||||
const asciiSpans: Array<{ start: number; end: number; text: string }> = [];
|
||||
while ((m = TOKEN_RE.exec(title)) !== null) {
|
||||
asciiSpans.push({ start: m.index, end: m.index + m[0].length, text: m[0].toLowerCase() });
|
||||
}
|
||||
let asciiIdx = 0;
|
||||
for (let i = 0; i < title.length;) {
|
||||
while (asciiIdx < asciiSpans.length && asciiSpans[asciiIdx]!.end <= i) asciiIdx++;
|
||||
if (asciiIdx < asciiSpans.length && i >= asciiSpans[asciiIdx]!.start && i < asciiSpans[asciiIdx]!.end) {
|
||||
tokens.push(asciiSpans[asciiIdx]!.text);
|
||||
i = asciiSpans[asciiIdx]!.end;
|
||||
asciiIdx++;
|
||||
continue;
|
||||
}
|
||||
const cp = title.codePointAt(i) ?? 0;
|
||||
if (hasCJK(title[i]!)) {
|
||||
const charLen = cp > 0xffff ? 2 : 1;
|
||||
tokens.push(title.slice(i, i + charLen).toLowerCase());
|
||||
i += charLen;
|
||||
} else {
|
||||
i++;
|
||||
}
|
||||
}
|
||||
return tokens;
|
||||
}
|
||||
// Pure CJK (no ASCII content): split into individual characters.
|
||||
if (hasCJK(title)) {
|
||||
for (let i = 0; i < title.length;) {
|
||||
const cp = title.codePointAt(i) ?? 0;
|
||||
const charLen = cp > 0xffff ? 2 : 1;
|
||||
tokens.push(title.slice(i, i + charLen).toLowerCase());
|
||||
i += charLen;
|
||||
}
|
||||
return tokens;
|
||||
}
|
||||
// Non-ASCII, non-CJK title (emoji, symbols, etc.) — empty set.
|
||||
return [];
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -175,7 +287,9 @@ export async function buildGazetteer(
|
||||
|
||||
const gazetteer: Gazetteer = new Map();
|
||||
for (const row of rows) {
|
||||
if (!row.title || row.title.length < MIN_NAME_LENGTH) continue;
|
||||
if (!row.title) continue;
|
||||
if (!hasCJK(row.title) && row.title.length < MIN_NAME_LENGTH) continue;
|
||||
if (hasCJK(row.title) && cjkCharCount(row.title) < MIN_CJK_NAME_LENGTH) continue;
|
||||
if (ignoreSet.has(row.title) && !existingTitles.has(row.title)) continue;
|
||||
|
||||
const tokens = tokenizeTitle(row.title);
|
||||
|
||||
@@ -20,6 +20,7 @@
|
||||
|
||||
import { chunkText as recursiveChunk } from './recursive.ts';
|
||||
import { buildQualifiedName } from './qualified-names.ts';
|
||||
import { CJK_SLUG_CHARS, CJK_RANGES_REGEX } from '../cjk.ts';
|
||||
|
||||
// Embed the tree-sitter runtime + per-language grammars as files.
|
||||
// `with { type: 'file' }` returns a path (string) at runtime. Bun bundles
|
||||
@@ -716,7 +717,7 @@ export async function chunkCodeTextFull(
|
||||
}
|
||||
|
||||
if (chunks.length === 0) {
|
||||
return { chunks: capOversizedChunks(fallbackChunks(source, filePath, language, opts), filePath, language, opts), edges: rawEdges };
|
||||
return { chunks: fallbackChunks(source, filePath, language, opts), edges: rawEdges };
|
||||
}
|
||||
return { chunks: capOversizedChunks(mergeSmallSiblings(chunks, chunkTarget), filePath, language, opts), edges: rawEdges };
|
||||
} catch {
|
||||
@@ -842,10 +843,10 @@ function capOversizedChunks(
|
||||
opts: CodeChunkOptions,
|
||||
): CodeChunk[] {
|
||||
const cap = opts.maxChunkTokens ?? DEFAULT_MAX_CHUNK_TOKENS;
|
||||
if (!chunks.some((c) => estimateTokens(c.text) > cap)) return chunks;
|
||||
if (!chunks.some((c) => estimateEmbedTokens(c.text) > cap)) return chunks;
|
||||
const out: CodeChunk[] = [];
|
||||
for (const c of chunks) {
|
||||
if (estimateTokens(c.text) <= cap) {
|
||||
if (estimateEmbedTokens(c.text) <= cap) {
|
||||
out.push({ ...c, index: out.length });
|
||||
continue;
|
||||
}
|
||||
@@ -880,17 +881,43 @@ function splitToTokenBudget(text: string, cap: number, opts: CodeChunkOptions):
|
||||
chunkOverlap: opts.fallbackOverlapWords ?? 50,
|
||||
}).map((p) => p.text);
|
||||
for (const piece of pieces) {
|
||||
if (estimateTokens(piece) <= cap) {
|
||||
if (estimateEmbedTokens(piece) <= cap) {
|
||||
out.push(piece);
|
||||
continue;
|
||||
}
|
||||
// ~3.5 chars/token is a conservative cl100k estimate for source text.
|
||||
const charBudget = Math.max(1, Math.floor(cap * 3.5));
|
||||
// Hard-split slice size. Pure-ASCII pieces: ~3.5 chars/token is a
|
||||
// conservative cl100k estimate for source text. CJK-containing pieces:
|
||||
// the weighted estimate can reach 1 token/char, so budget 1 char/token
|
||||
// to keep every slice under cap by construction.
|
||||
const charBudget = Math.max(1, Math.floor(cap * (CJK_RANGES_REGEX.test(piece) ? 1 : 3.5)));
|
||||
for (let i = 0; i < piece.length; i += charBudget) out.push(piece.slice(i, i + charBudget));
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
const CJK_CHARS_G = new RegExp(`[${CJK_SLUG_CHARS}]`, 'g');
|
||||
|
||||
/**
|
||||
* Embedding-safe token estimate for the oversize cap. estimateTokens
|
||||
* (cl100k) matches embedding-family tokenizers closely on pure-ASCII source
|
||||
* (measured identical on English prose and JSON vs Qwen3-Embedding), but
|
||||
* UNDERCOUNTS mixed CJK+ASCII chunks — measured −31% on URL-dense Korean
|
||||
* text vs the Qwen3 embedding tokenizer, which is exactly the shape that
|
||||
* overflows strict embedding backends (#2826). For chunks containing CJK,
|
||||
* take the max of cl100k and a per-char-class overestimate (CJK 1.0/char,
|
||||
* other non-whitespace 0.75/char, whitespace 0.1/char). CJK-DOMINANT text
|
||||
* is unaffected too: cl100k already counts it above the weighted form, so
|
||||
* max() returns the same value as today. Only mixed-script chunks — the
|
||||
* measured divergence class — estimate higher.
|
||||
*/
|
||||
export function estimateEmbedTokens(text: string): number {
|
||||
const cjk = (text.match(CJK_CHARS_G) || []).length;
|
||||
if (cjk === 0) return estimateTokens(text);
|
||||
const ws = (text.match(/\s/g) || []).length;
|
||||
const weighted = Math.ceil(cjk + (text.length - cjk - ws) * 0.75 + ws * 0.1);
|
||||
return Math.max(estimateTokens(text), weighted);
|
||||
}
|
||||
|
||||
// ---------- Internals ----------
|
||||
|
||||
function fallbackChunks(
|
||||
@@ -901,7 +928,7 @@ function fallbackChunks(
|
||||
): CodeChunk[] {
|
||||
const size = opts.fallbackChunkSizeWords ?? 300;
|
||||
const overlap = opts.fallbackOverlapWords ?? 50;
|
||||
return recursiveChunk(source, { chunkSize: size, chunkOverlap: overlap }).map((chunk, index) =>
|
||||
const chunks = recursiveChunk(source, { chunkSize: size, chunkOverlap: overlap }).map((chunk, index) =>
|
||||
buildChunk({
|
||||
body: chunk.text, filePath, language,
|
||||
symbolName: null, symbolType: 'module',
|
||||
@@ -909,6 +936,14 @@ function fallbackChunks(
|
||||
index,
|
||||
}),
|
||||
);
|
||||
// Route every fallback emission through the oversize net. Previously only
|
||||
// the empty-AST branch wrapped its fallback in capOversizedChunks — the
|
||||
// no-language, parse-timeout, no-semantic-nodes (every JSON/YAML fence:
|
||||
// their node types aren't in TOP_LEVEL_TYPES) and parse-throw branches
|
||||
// shipped word-counted chunks unchecked, and the word pipeline undercounts
|
||||
// exactly the dense content (JSON, minified, CJK-mixed) that overflows
|
||||
// embedders. Hoisting the cap here covers all five paths at once.
|
||||
return capOversizedChunks(chunks, filePath, language, opts);
|
||||
}
|
||||
|
||||
function buildChunk(input: {
|
||||
|
||||
@@ -78,10 +78,10 @@ export const WALK_DEPTH_CAP = 32;
|
||||
|
||||
/**
|
||||
* Which languages get receiver-type resolution at extraction time. Per D18
|
||||
* from eng review — JS/TS/TSX + Python at full depth; Ruby/Go/Rust/Java
|
||||
* keep TODAY's bare-token call edges. Honest scope: tree-sitter shapes are
|
||||
* very different across these languages and writing+testing per-language
|
||||
* scope walkers for all of them is a v0.35 expansion.
|
||||
* from eng review — JS/TS/TSX + Python at full depth; Ruby/Go/Rust/Java/
|
||||
* Kotlin keep TODAY's bare-token call edges. Honest scope: tree-sitter
|
||||
* shapes are very different across these languages and writing+testing
|
||||
* per-language scope walkers for all of them is a v0.35 expansion.
|
||||
*/
|
||||
const RECEIVER_RESOLUTION_LANGS: ReadonlySet<SupportedCodeLanguage> = new Set([
|
||||
'typescript',
|
||||
@@ -93,12 +93,16 @@ const RECEIVER_RESOLUTION_LANGS: ReadonlySet<SupportedCodeLanguage> = new Set([
|
||||
/**
|
||||
* Per-language call-expression configuration. `callNodeTypes` lists the
|
||||
* AST node types that are call sites in that language. `calleeFieldName`
|
||||
* optionally names the child field that holds the callee expression;
|
||||
* when absent, the call-site text itself is scanned for the identifier.
|
||||
* names the child field that holds the callee expression. Grammars that
|
||||
* define no fields on their call node (Kotlin: `call_expression =
|
||||
* expression call_suffix`) set `calleeFirstNamedChild` instead — the
|
||||
* callee is positional, so namedChild(0) IS the callee.
|
||||
*/
|
||||
interface CallConfig {
|
||||
callNodeTypes: Set<string>;
|
||||
calleeFieldName?: string;
|
||||
/** Callee is namedChild(0) — for grammars whose call node has no fields. */
|
||||
calleeFirstNamedChild?: boolean;
|
||||
}
|
||||
|
||||
const CALL_CONFIG: Partial<Record<SupportedCodeLanguage, CallConfig>> = {
|
||||
@@ -110,6 +114,11 @@ const CALL_CONFIG: Partial<Record<SupportedCodeLanguage, CallConfig>> = {
|
||||
go: { callNodeTypes: new Set(['call_expression']), calleeFieldName: 'function' },
|
||||
rust: { callNodeTypes: new Set(['call_expression', 'method_call_expression']), calleeFieldName: 'function' },
|
||||
java: { callNodeTypes: new Set(['method_invocation']), calleeFieldName: 'name' },
|
||||
// tree-sitter-kotlin defines no fields on call_expression; the callee is
|
||||
// the first named child (simple_identifier for bare calls,
|
||||
// navigation_expression for `receiver.method(...)` — resolved to the
|
||||
// method name by the navigation_expression case in extractCalleeName).
|
||||
kotlin: { callNodeTypes: new Set(['call_expression']), calleeFirstNamedChild: true },
|
||||
};
|
||||
|
||||
/**
|
||||
@@ -120,7 +129,11 @@ const CALL_CONFIG: Partial<Record<SupportedCodeLanguage, CallConfig>> = {
|
||||
* null to skip the edge.
|
||||
*/
|
||||
function extractCalleeName(node: any, cfg: CallConfig): string | null {
|
||||
const callee = cfg.calleeFieldName ? node.childForFieldName(cfg.calleeFieldName) : null;
|
||||
const callee = cfg.calleeFieldName
|
||||
? node.childForFieldName(cfg.calleeFieldName)
|
||||
: cfg.calleeFirstNamedChild
|
||||
? (node.namedChild?.(0) ?? null)
|
||||
: null;
|
||||
if (!callee) return null;
|
||||
|
||||
// Unwrap common wrappers until we hit an identifier-shaped node.
|
||||
@@ -155,6 +168,15 @@ function extractCalleeName(node: any, cfg: CallConfig): string | null {
|
||||
if (name) { cur = name; continue; }
|
||||
return null;
|
||||
}
|
||||
// navigation_expression (Kotlin): `receiver.method` — the callee is the
|
||||
// simple_identifier inside the trailing navigation_suffix. No fields on
|
||||
// this node either, so walk to the last named child's identifier.
|
||||
if (cur.type === 'navigation_expression') {
|
||||
const suffix = cur.namedChild?.(cur.namedChildCount - 1);
|
||||
const ident = suffix?.namedChild?.(0);
|
||||
if (ident) { cur = ident; continue; }
|
||||
return null;
|
||||
}
|
||||
// Fallback: read the node text and take the last identifier-looking token.
|
||||
const m = (cur.text as string).match(/([A-Za-z_][A-Za-z0-9_]*)\s*$/);
|
||||
return m ? sanitizeIdent(m[1]!) : null;
|
||||
|
||||
+28
-5
@@ -21,12 +21,35 @@ export const CJK_SLUG_CHARS = '一-鿿-ゟ゠-ヿ가-';
|
||||
export const CJK_RANGES_REGEX = new RegExp(`[${CJK_SLUG_CHARS}]`);
|
||||
|
||||
/**
|
||||
* Page-slug segment grammar (no anchors): alnum-or-CJK lead char, then
|
||||
* alnum/CJK/hyphen continuation. Single source for validatePageSlug
|
||||
* (operations.ts), SlugRegistry's SLUG_RE, and the dream-cycle
|
||||
* SUMMARY_SLUG_RE so every slug validator shares one grammar (#738).
|
||||
* Slug "word" character class (#3417): every script's letters, not just
|
||||
* Latin + CJK. Unicode property escapes — REQUIRES the `u` flag on any
|
||||
* regex composed from this string (without `u`, `\p{Ll}` silently matches
|
||||
* the literal chars `p`, `L`, `l`, `{`, `}`).
|
||||
*
|
||||
* \p{Ll} lowercase letters (a-z, Cyrillic/Greek lowercase, đ, …)
|
||||
* \p{Lm} modifier letters
|
||||
* \p{Lo} caseless-script letters (Hebrew, Arabic, Thai, CJK, Devanagari, …)
|
||||
* \p{M} combining marks that survive the Latin accent-strip pass
|
||||
* (Hebrew niqqud, Arabic harakat, Thai/Devanagari vowel signs)
|
||||
* \p{N} numbers (0-9, Arabic-Indic digits, …)
|
||||
*
|
||||
* Uppercase (\p{Lu}/\p{Lt}) is deliberately excluded: slugifySegment()
|
||||
* lowercases before filtering, so validators stay lowercase-canonical.
|
||||
*
|
||||
* Distinct from CJK_SLUG_CHARS above, which also drives the
|
||||
* countCJKAwareWords density heuristic — do NOT merge the two, or slug
|
||||
* grammar changes silently change chunking behavior.
|
||||
*/
|
||||
export const PAGE_SLUG_SEG = `[a-z0-9${CJK_SLUG_CHARS}][a-z0-9${CJK_SLUG_CHARS}\\-]*`;
|
||||
export const SLUG_WORD_CHARS = '\\p{Ll}\\p{Lm}\\p{Lo}\\p{M}\\p{N}';
|
||||
|
||||
/**
|
||||
* Page-slug segment grammar (no anchors): word-char lead, then word-char or
|
||||
* hyphen continuation. Single source for validatePageSlug (operations.ts),
|
||||
* SlugRegistry's SLUG_RE, and the dream-cycle SUMMARY_SLUG_RE so every slug
|
||||
* validator shares one grammar (#738). Compose with the `u` flag — see
|
||||
* SLUG_WORD_CHARS.
|
||||
*/
|
||||
export const PAGE_SLUG_SEG = `[${SLUG_WORD_CHARS}][${SLUG_WORD_CHARS}\\-]*`;
|
||||
|
||||
export const CJK_SENTENCE_DELIMITERS = ['。', '!', '?']; // 。!?
|
||||
export const CJK_CLAUSE_DELIMITERS = [';', ':', ',', '、']; // ;:,、
|
||||
|
||||
@@ -63,6 +63,12 @@ export interface GBrainConfig {
|
||||
* config.json file-plane route is wired through today.
|
||||
*/
|
||||
voyage_api_key?: string;
|
||||
/** Azure OpenAI (keyless/Entra). Non-secret endpoint + deployment + Entra opt-in,
|
||||
* folded into the gateway env so the azure-openai recipe works in any shell.
|
||||
* The bearer token is minted at request time via `az` — no secret stored here. */
|
||||
azure_openai_endpoint?: string;
|
||||
azure_openai_deployment?: string;
|
||||
azure_openai_use_entra?: string;
|
||||
/** AI gateway config (v0.14+). v0.36+ default: "zeroentropyai:zembed-1" / 1280 / "anthropic:claude-haiku-4-5-20251001". */
|
||||
embedding_model?: string;
|
||||
embedding_dimensions?: number;
|
||||
@@ -913,6 +919,9 @@ export const KNOWN_CONFIG_KEYS: readonly string[] = [
|
||||
'zeroentropy_api_key',
|
||||
'openrouter_api_key',
|
||||
'voyage_api_key',
|
||||
'azure_openai_endpoint',
|
||||
'azure_openai_deployment',
|
||||
'azure_openai_use_entra',
|
||||
'embedding_model',
|
||||
'embedding_dimensions',
|
||||
'embedding_disabled',
|
||||
@@ -1085,6 +1094,27 @@ export const KNOWN_CONFIG_KEY_PREFIXES: readonly string[] = [
|
||||
'self_upgrade.', // v0.42 self-upgrade (mode, quiet_hours, state)
|
||||
];
|
||||
|
||||
/**
|
||||
* Canonical truthiness for DB-plane boolean config values (#2753).
|
||||
*
|
||||
* Config values arrive as opaque strings from `gbrain config set`, so every
|
||||
* reader has to decide what counts as "on". Left to each call site those sets
|
||||
* drift, and the drift is silent in the worst possible way: the doctor accepted
|
||||
* `yes`/`on` while the subagent worker accepted only `true`/`1`, so
|
||||
* `gbrain config set agent.use_gateway_loop yes` produced a healthy doctor
|
||||
* report AND a runtime refusal of the very job the setting was supposed to
|
||||
* enable. One parser, used by every reader, is what keeps a green health check
|
||||
* honest.
|
||||
*
|
||||
* Accepts `true` / `1` / `yes` / `on` (case-insensitive, surrounding whitespace
|
||||
* trimmed). Everything else — including `null`, non-strings, and the empty
|
||||
* string — is false, so an unset or garbled value fails closed.
|
||||
*/
|
||||
export function isConfigTruthy(raw: unknown): boolean {
|
||||
return typeof raw === 'string'
|
||||
&& ['true', '1', 'yes', 'on'].includes(raw.trim().toLowerCase());
|
||||
}
|
||||
|
||||
export function saveConfig(config: GBrainConfig): void {
|
||||
mkdirSync(getConfigDir(), { recursive: true });
|
||||
writeFileSync(getConfigPath(), JSON.stringify(config, null, 2) + '\n', { mode: 0o600 });
|
||||
|
||||
@@ -145,6 +145,19 @@ export function computeCorpusGeneration(args: {
|
||||
return h.digest('hex').slice(0, 16);
|
||||
}
|
||||
|
||||
/**
|
||||
* #3507 — the corpus_generation a page lands on when a plain re-embed path
|
||||
* (`embed --stale` and friends) re-embeds a `per_chunk_synopsis` page at the
|
||||
* title-only tier (the D14 fallback tier; synopsis re-generation is a paid
|
||||
* backfill concern). Callers restamp
|
||||
* `updatePageContextualRetrievalState(slug, sourceId, 'title', titleTierCorpusGeneration())`
|
||||
* so the stamped mode keeps describing the vectors actually in the column.
|
||||
* Matches what the inline import path writes for its title-tier pages.
|
||||
*/
|
||||
export function titleTierCorpusGeneration(): string {
|
||||
return computeCorpusGeneration({ crMode: 'title', haikuModel: DEFAULT_HAIKU_MODEL });
|
||||
}
|
||||
|
||||
/**
|
||||
* Compute source_text_hash for D27 P1-4 cache key composition. The
|
||||
* synopsis cache invalidates correctly when adjacent text changes (page
|
||||
|
||||
@@ -306,41 +306,7 @@ function splitCacheKey(key: string): [string?, string?, string?] {
|
||||
return [shape, model, sha];
|
||||
}
|
||||
|
||||
/**
|
||||
* 4-strategy JSON repair (lifted from `eval/longmemeval/extract.ts:50`
|
||||
* for object-shaped output; the original was array-shaped). Caller's
|
||||
* `parse` function uses this for tolerant LLM-output decoding.
|
||||
*
|
||||
* Strategies:
|
||||
* 1. Strip ```json...``` fences if present, then JSON.parse.
|
||||
* 2. Direct JSON.parse.
|
||||
* 3. Find first {...} substring (or [...] if array=true) and parse.
|
||||
* 4. Return null.
|
||||
*
|
||||
* Adversarial input throws caught by caller's try/catch (parse returns
|
||||
* null upstream).
|
||||
*/
|
||||
export function parseLlmJson<T>(raw: string, opts: { array?: boolean } = {}): T | null {
|
||||
if (typeof raw !== 'string' || !raw.trim()) return null;
|
||||
const fenceMatch = raw.match(/```(?:json)?\s*\n?([\s\S]*?)```/i);
|
||||
const cleaned = (fenceMatch ? fenceMatch[1] : raw).trim();
|
||||
try {
|
||||
const direct = JSON.parse(cleaned);
|
||||
if (opts.array && Array.isArray(direct)) return direct as T;
|
||||
if (!opts.array && direct !== null && typeof direct === 'object') return direct as T;
|
||||
} catch {
|
||||
// fall through
|
||||
}
|
||||
const pattern = opts.array ? /\[[\s\S]*\]/ : /\{[\s\S]*\}/;
|
||||
const match = cleaned.match(pattern);
|
||||
if (match) {
|
||||
try {
|
||||
const second = JSON.parse(match[0]);
|
||||
if (opts.array && Array.isArray(second)) return second as T;
|
||||
if (!opts.array && second !== null && typeof second === 'object') return second as T;
|
||||
} catch {
|
||||
// fall through
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
// Tolerant LLM-output JSON decoder. Re-exported from the leaf util so existing
|
||||
// importers (llm-fallback, llm-polish) keep their import path while the gateway
|
||||
// can reuse it without a dependency cycle.
|
||||
export { parseLlmJson } from '../llm-json.ts';
|
||||
|
||||
@@ -0,0 +1,94 @@
|
||||
/**
|
||||
* Block-format conversation normalizer.
|
||||
*
|
||||
* Some chat exports — notably the Slack collector gbrain's own ingestion
|
||||
* uses — emit a HEADER + indented-body BLOCK per message instead of the
|
||||
* single-line `**Name** (time): body` shape the built-in patterns
|
||||
* (`builtins.ts`) recognize:
|
||||
*
|
||||
* - **Theo** (Mon 11:18)
|
||||
* Hey everyone — quick update on the renewal.
|
||||
*
|
||||
* Second paragraph of the same message.
|
||||
* - **Juan** (Mon 11:20)
|
||||
* Reply body...
|
||||
*
|
||||
* None of the 14 line-oriented built-ins match this: a leading `- ` list
|
||||
* marker, a day-of-week + time with no trailing colon, and the message body on
|
||||
* the following indented lines. Result: `phase: 'no_match'`, zero messages,
|
||||
* and the whole comms corpus is silently un-extractable (facts stay empty →
|
||||
* `find_trajectory` returns nothing).
|
||||
*
|
||||
* This collapses each block into the canonical `**Name** (HH:MM): <body joined
|
||||
* to one line>` shape so the existing `bold-paren-time` pattern matches; the
|
||||
* per-message date fills in downstream via `fallbackDate` (the page date).
|
||||
*
|
||||
* STRICT no-op unless the block signature is present: the header regex requires
|
||||
* the paren-group to END the line (no inline `: body`), which is exactly what
|
||||
* the single-line patterns always produce — so feeding already-canonical
|
||||
* content through this function returns it unchanged.
|
||||
*/
|
||||
|
||||
// `- **Name** (Mon 11:18)` / `- **Name** (11:18 AM)` / `- **Name** (16:36)`.
|
||||
// Day-of-week optional; 12h/24h time; optional am/pm; the line ENDS at the
|
||||
// close paren (no inline `: body` — that is what distinguishes a block header
|
||||
// from the single-line `**Name** (time): body` patterns).
|
||||
const BLOCK_HEADER =
|
||||
/^\s*-\s+\*\*(.+?)\*\*\s+\((?:[A-Za-z]{2,9}\.?\s+)?(\d{1,2}):(\d{2})(?::\d{2})?\s*([AaPp][Mm])?\)\s*$/;
|
||||
|
||||
/** True when at least one line is a block-format message header. */
|
||||
export function looksLikeBlockConversation(body: string): boolean {
|
||||
for (const line of body.split('\n')) {
|
||||
if (BLOCK_HEADER.test(line)) return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
function to24h(hour: number, ampm?: string): number {
|
||||
if (!ampm) return hour;
|
||||
const lower = ampm.toLowerCase();
|
||||
if (lower === 'pm' && hour < 12) return hour + 12;
|
||||
if (lower === 'am' && hour === 12) return 0;
|
||||
return hour;
|
||||
}
|
||||
|
||||
/**
|
||||
* Collapse block-format messages into canonical single-line `**Name** (HH:MM):
|
||||
* body` lines. Returns `body` unchanged when no block header is present.
|
||||
*/
|
||||
export function normalizeBlockConversation(body: string): string {
|
||||
if (!looksLikeBlockConversation(body)) return body;
|
||||
|
||||
const lines = body.split('\n');
|
||||
const out: string[] = [];
|
||||
let current: { name: string; time: string } | null = null;
|
||||
let bodyParts: string[] = [];
|
||||
|
||||
const flush = () => {
|
||||
if (current) {
|
||||
const text = bodyParts.join(' ').replace(/\s+/g, ' ').trim();
|
||||
out.push(`**${current.name}** (${current.time}): ${text}`);
|
||||
}
|
||||
current = null;
|
||||
bodyParts = [];
|
||||
};
|
||||
|
||||
for (const line of lines) {
|
||||
const m = BLOCK_HEADER.exec(line);
|
||||
if (m) {
|
||||
flush();
|
||||
const hour = to24h(parseInt(m[2], 10), m[4]);
|
||||
const time = `${String(hour).padStart(2, '0')}:${m[3]}`;
|
||||
current = { name: m[1].trim(), time };
|
||||
} else if (current) {
|
||||
// Body line of the current message. Drop blank lines; keep the rest.
|
||||
const trimmed = line.trim();
|
||||
if (trimmed) bodyParts.push(trimmed);
|
||||
}
|
||||
// Lines before the first header (page title, leading blanks) are dropped —
|
||||
// they never matched a pattern anyway.
|
||||
}
|
||||
flush();
|
||||
|
||||
return out.length > 0 ? out.join('\n') : body;
|
||||
}
|
||||
@@ -29,6 +29,7 @@ import {
|
||||
BUILTIN_PATTERNS,
|
||||
cleanSpeaker,
|
||||
} from './builtins.ts';
|
||||
import { normalizeBlockConversation } from './normalize-block.ts';
|
||||
import type {
|
||||
DateContext,
|
||||
MatchedMessage,
|
||||
@@ -473,6 +474,12 @@ export function parseConversation(
|
||||
return { messages: [], phase: 'no_match' };
|
||||
}
|
||||
|
||||
// Pre-pass: collapse block-format chat exports (header + indented body, e.g.
|
||||
// the Slack collector's `- **Name** (Mon 11:18)\n body…`) into the canonical
|
||||
// single-line shape the built-in patterns recognize. Strict no-op when no
|
||||
// block header is present, so already-canonical content is untouched.
|
||||
body = normalizeBlockConversation(body);
|
||||
|
||||
const dateCtx = deriveDateContext(opts);
|
||||
|
||||
// Assemble candidate pool: built-ins (minus disabled) + user patterns.
|
||||
|
||||
+53
-3
@@ -43,7 +43,7 @@
|
||||
* trigger lock acquisition.
|
||||
*/
|
||||
|
||||
import { existsSync, readFileSync, writeFileSync, unlinkSync, mkdirSync, statSync } from 'fs';
|
||||
import { existsSync, readFileSync, writeFileSync, unlinkSync, mkdirSync, statSync, realpathSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { gbrainPath } from './config.ts';
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
@@ -899,7 +899,55 @@ export async function resolveSourceForDir(
|
||||
`SELECT id FROM sources WHERE local_path = $1 LIMIT 1`,
|
||||
[brainDir],
|
||||
);
|
||||
return rows[0]?.id;
|
||||
if (rows[0]) return rows[0].id;
|
||||
|
||||
// #2540: the exact match above compares two path SPELLINGS. `--dir` is
|
||||
// resolved via `resolve()` and `sources.local_path` stores the spelling
|
||||
// the source was registered with (`--path` as typed, or defaultCloneDir);
|
||||
// neither side is canonicalized. So a source registered through a symlink
|
||||
// and dreamt via the real path (or vice versa) never string-matches — the
|
||||
// `--dir` run derives no source, and the #1869 freshness stamp silently
|
||||
// does not land, leaving doctor's cycle_freshness permanently stale on a
|
||||
// healthy install. Symlinked vault locations are ordinary (a brain inside
|
||||
// a synced cloud-storage folder, a /home -> /mnt relocation).
|
||||
//
|
||||
// Retry canonicalized on BOTH sides, so the match is symmetric regardless
|
||||
// of which side holds the link. Kept strictly as a miss-path fallback: the
|
||||
// exact match stays a single indexed lookup, and the scan only pays for
|
||||
// itself when it would otherwise return nothing. Registered paths are
|
||||
// canonicalized here rather than at registration because storing a
|
||||
// canonical column would be a schema + backfill change; that is the
|
||||
// durable fix and is left as a follow-up.
|
||||
let realDir: string;
|
||||
try {
|
||||
realDir = realpathSync(brainDir);
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
// Archived sources are excluded deliberately: dream's --source guard
|
||||
// already refuses to stamp them (writing last_full_cycle_at to an
|
||||
// archived source masks staleness when it is later restored), so an
|
||||
// archived alias must not win a path match either.
|
||||
const candidates = await engine.executeRaw<{ id: string; local_path: string }>(
|
||||
`SELECT id, local_path FROM sources
|
||||
WHERE local_path IS NOT NULL AND archived = false
|
||||
ORDER BY (id = 'default') DESC, id`,
|
||||
);
|
||||
const matched: string[] = [];
|
||||
for (const row of candidates) {
|
||||
try {
|
||||
if (realpathSync(row.local_path) === realDir) matched.push(row.id);
|
||||
} catch {
|
||||
// Stale/unreadable registered path — not a match, keep scanning.
|
||||
}
|
||||
}
|
||||
// Fail closed when several registered paths canonicalize to the same
|
||||
// directory: a canonical alias is weaker evidence than an exact spelling
|
||||
// match, and picking one arbitrarily would scope the cycle — and its
|
||||
// freshness stamp — to whichever id happened to sort first. Returning
|
||||
// undefined leaves the caller on the pre-existing opts.sourceId/'default'
|
||||
// precedence, i.e. exactly the behaviour before this fallback existed.
|
||||
return matched.length === 1 ? matched[0] : undefined;
|
||||
} catch {
|
||||
// sources table might not exist on very old brains — fall through.
|
||||
return undefined;
|
||||
@@ -1131,7 +1179,9 @@ async function runPhaseExtractFacts(
|
||||
summary: `extract_facts skipped: ${result.legacyRowsPending} legacy v0.31 facts pending fence backfill`,
|
||||
details: {
|
||||
legacyRowsPending: result.legacyRowsPending,
|
||||
hint: 'gbrain apply-migrations --yes',
|
||||
// A bare `apply-migrations --yes` no-ops once the v0.32.2 ledger
|
||||
// entry is complete; the retry marker is what re-runs Phase B.
|
||||
hint: 'gbrain apply-migrations --force-retry 0.32.2 && gbrain apply-migrations --yes',
|
||||
warnings: result.warnings,
|
||||
},
|
||||
};
|
||||
|
||||
@@ -17,20 +17,26 @@
|
||||
* DB rows need cleanup (#1781 — the unconditional wipe-and-reinsert
|
||||
* made every cycle non-idempotent, re-appending duplicate rows).
|
||||
*
|
||||
* After the phase, the DB index for every affected page matches the
|
||||
* fence's canonical (claim, source) row set (modulo embeddings +
|
||||
* runtime-derived fields). Pages with no fence wipe DB rows for that
|
||||
* page coordinate only; legacy NULL-source_markdown_slug rows survive
|
||||
* because deleteFactsForPage targets source_markdown_slug = slug only.
|
||||
* After the phase, the DB index for every cleanly parsed affected page
|
||||
* matches the fence's canonical (claim, source) row set (modulo embeddings
|
||||
* + runtime-derived fields). Warning-bearing parses are non-authoritative
|
||||
* and preserve that page's existing index. Pages with no fence wipe DB rows
|
||||
* for that page coordinate only; legacy NULL-source_markdown_slug rows
|
||||
* survive because deleteFactsForPage targets source_markdown_slug = slug only.
|
||||
*
|
||||
* Empty-fence guard (Codex R2-#7; #2484): the phase refuses to do its
|
||||
* destructive reconciliation pass when genuinely-backfillable legacy
|
||||
* rows still exist — `row_num IS NULL` (never fenced) AND `entity_slug`
|
||||
* resolves to a live page in this source (so the v0_32_2 migration's
|
||||
* Phase B could fence them). Status returns `warn` with a hint to run
|
||||
* `gbrain apply-migrations --yes`. Without the guard, an interrupted
|
||||
* upgrade where v0_32_2 hasn't run could leave the cycle silently
|
||||
* misreporting "0 facts on people/alice" while legacy rows linger.
|
||||
* Empty-fence guard (Codex R2-#7; #2484; #2646): the phase refuses to do
|
||||
* its destructive reconciliation pass when genuinely-backfillable legacy
|
||||
* rows still exist — in THIS run's source only (`source_id = sourceId`;
|
||||
* a pending row in source A must not jam extraction for source B — the
|
||||
* source-isolation invariant) — `row_num IS NULL` (never fenced) AND
|
||||
* `entity_slug` resolves to a live page in this source (so the v0_32_2
|
||||
* migration's Phase B could fence them) AND the row is not soft-expired
|
||||
* (`expired_at IS NULL`). Status returns `warn` with a hint to re-run
|
||||
* the v0.32.2 fence backfill (`apply-migrations --force-retry 0.32.2`
|
||||
* then `--yes` — a bare `--yes` is a no-op once the ledger says
|
||||
* complete). Without the guard, an interrupted upgrade where v0_32_2
|
||||
* hasn't run could leave the cycle silently misreporting "0 facts on
|
||||
* people/alice" while legacy rows linger.
|
||||
*
|
||||
* The live-page requirement (#2484) is load-bearing: the inline facts
|
||||
* writer keeps producing `row_num IS NULL, entity_slug IS NOT NULL`
|
||||
@@ -41,6 +47,10 @@
|
||||
* the phase jams forever (~16/day observed). Requiring a backing page
|
||||
* keeps genuine pre-v0.32.2 rows (whose entity page exists) gating
|
||||
* while excluding the inline-writer's permanent-unfenceable rows.
|
||||
*
|
||||
* Soft-expired rows don't count either (#2646): they're what
|
||||
* `forget_fact` produces, so excluding them lets operators drain the
|
||||
* backlog through the sanctioned removal path instead of raw SQL.
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from '../engine.ts';
|
||||
@@ -88,6 +98,24 @@ function dedupeFactsByContentKey(facts: FenceExtractedFact[]): FenceExtractedFac
|
||||
* neither count as "stale" (which would force a wipe every cycle) nor
|
||||
* be compared against the fence's row set. Mirrors the
|
||||
* excludeSourcePrefixes filter deleteFactsForPage applies on the wipe.
|
||||
*
|
||||
* Also excludes soft-expired legacy rows (#2646: `row_num IS NULL AND
|
||||
* expired_at IS NOT NULL`) — rows that `forget_fact` expired via its
|
||||
* legacy DB-only path. They are not fence-owned (fence rows always
|
||||
* carry a row_num), so they must neither count as "stale" (forcing a
|
||||
* wipe every cycle) nor mask a fence row from insertion. Mirrors the
|
||||
* preserveExpiredLegacy filter deleteFactsForPage applies on the wipe.
|
||||
*
|
||||
* Deliberate consequence: if the fence still carries the same
|
||||
* (claim, source) as an expired legacy row, the reconcile inserts it
|
||||
* as a fresh ACTIVE fence-owned row. That is the fence-is-canonical
|
||||
* contract working as documented — legacy DB-only forgets "DO NOT
|
||||
* survive rebuild" (see forget.ts header); suppressing the insert
|
||||
* would instead create silent fence↔DB divergence, the exact failure
|
||||
* mode the empty-fence guard exists to prevent. To durably forget
|
||||
* such a claim, forget the fence-owned row (forget_fact now takes the
|
||||
* fence path, which strikes the row through in markdown). The expired
|
||||
* legacy row survives alongside as the record of the earlier forget.
|
||||
*/
|
||||
async function listExistingFactsForPage(
|
||||
engine: BrainEngine,
|
||||
@@ -100,6 +128,7 @@ async function listExistingFactsForPage(
|
||||
WHERE source_id = $1
|
||||
AND source_markdown_slug = $2
|
||||
AND COALESCE(source, '') NOT LIKE 'cli:%'
|
||||
AND NOT (row_num IS NULL AND expired_at IS NOT NULL)
|
||||
ORDER BY row_num ASC, id ASC`,
|
||||
[sourceId, slug],
|
||||
);
|
||||
@@ -173,7 +202,7 @@ export async function runExtractFacts(
|
||||
phantomsMorePending: false,
|
||||
};
|
||||
|
||||
// ── Empty-fence guard (Codex R2-#7; #2484) ─────────────────────
|
||||
// ── Empty-fence guard (Codex R2-#7; #2484; #2646) ──────────────
|
||||
// Pre-check: if any genuinely-backfillable legacy fact rows exist,
|
||||
// refuse to run the destructive reconciliation pass — the v0_32_2
|
||||
// orchestrator must fence them first.
|
||||
@@ -181,12 +210,13 @@ export async function runExtractFacts(
|
||||
// A row is a real backfill candidate only when `row_num IS NULL`
|
||||
// (never fenced) AND its `entity_slug` resolves to a LIVE page in
|
||||
// this source (the migration's Phase B only fences rows whose
|
||||
// entity_slug maps to a writable page). #2484: the original
|
||||
// predicate was just `row_num IS NULL AND entity_slug IS NOT NULL`,
|
||||
// which ALSO matched structurally-unfenceable hot-memory rows the
|
||||
// inline writer keeps producing post-migration: the legacy DB-only
|
||||
// fallback (backstop.ts) writes `entity_slug` (a resolved slug, e.g.
|
||||
// a slugify-floor or stub-guard-blocked unprefixed slug like
|
||||
// entity_slug maps to a writable page) AND it is not soft-expired.
|
||||
// #2484: the original predicate was just `row_num IS NULL AND
|
||||
// entity_slug IS NOT NULL`, which ALSO matched
|
||||
// structurally-unfenceable hot-memory rows the inline writer keeps
|
||||
// producing post-migration: the legacy DB-only fallback
|
||||
// (backstop.ts) writes `entity_slug` (a resolved slug, e.g. a
|
||||
// slugify-floor or stub-guard-blocked unprefixed slug like
|
||||
// `people-jane-doe`) with `row_num` NULL whenever the slug has no
|
||||
// fenceable page. Those rows can never satisfy the migration's exit
|
||||
// condition (no page to fence onto, and `apply-migrations` is a
|
||||
@@ -194,27 +224,49 @@ export async function runExtractFacts(
|
||||
// — ~16/day, mislabeled "v0.31 pending backfill." We now require a
|
||||
// live backing page, which both genuine pre-v0.32.2 rows (their
|
||||
// entity page exists) satisfy and inline-writer unfenceable rows do
|
||||
// not.
|
||||
// not. #2646: soft-expired rows (`expired_at IS NOT NULL`) are also
|
||||
// excluded — `forget_fact`, the officially sanctioned removal path,
|
||||
// soft-expires legacy rows rather than deleting them, so counting
|
||||
// expired rows would leave the guard permanently stuck with no
|
||||
// supported way to drain the backlog.
|
||||
//
|
||||
// Source isolation (#3526): the count is scoped to THIS run's
|
||||
// sourceId. The pre-fix query counted brain-wide, so a single pending
|
||||
// legacy row in any mounted source jammed extract_facts for every
|
||||
// source — a cross-source leak of one source's migration state into
|
||||
// another's cycle (CLAUDE.md source-isolation invariant).
|
||||
const legacy = await engine.executeRaw<{ n: string }>(
|
||||
`SELECT COUNT(*) AS n
|
||||
FROM facts f
|
||||
WHERE f.row_num IS NULL
|
||||
WHERE f.source_id = $1
|
||||
AND f.row_num IS NULL
|
||||
AND f.entity_slug IS NOT NULL
|
||||
AND f.expired_at IS NULL
|
||||
AND EXISTS (
|
||||
SELECT 1 FROM pages p
|
||||
WHERE p.source_id = f.source_id
|
||||
AND p.slug = f.entity_slug
|
||||
AND p.deleted_at IS NULL
|
||||
)`,
|
||||
[sourceId],
|
||||
);
|
||||
const legacyCount = parseInt(legacy[0]?.n ?? '0', 10);
|
||||
result.legacyRowsPending = legacyCount;
|
||||
if (legacyCount > 0) {
|
||||
result.guardTriggered = true;
|
||||
// Drain advice must actually work: a bare `apply-migrations --yes`
|
||||
// is a no-op once the v0.32.2 ledger entry says complete (the
|
||||
// runner classifies it as already-applied), so the sanctioned
|
||||
// re-run path is the explicit retry marker first. Phase B is
|
||||
// idempotent — it only touches `row_num IS NULL` rows and de-dupes
|
||||
// against the existing fence — so the re-run is safe. Individual
|
||||
// rows can instead be drained through `forget_fact` (soft-expired
|
||||
// rows stop counting).
|
||||
result.warnings.push(
|
||||
`extract_facts: ${legacyCount} legacy v0.31 fact rows (entity page present, not yet ` +
|
||||
`fenced) pending fence backfill. Run \`gbrain apply-migrations --yes\` to complete ` +
|
||||
`v0_32_2 before this phase can safely reconcile fence → DB.`,
|
||||
`extract_facts: ${legacyCount} legacy v0.31 fact rows in source "${sourceId}" ` +
|
||||
`(entity page present, not yet fenced) pending fence backfill. Re-run the v0.32.2 ` +
|
||||
`fence backfill: \`gbrain apply-migrations --force-retry 0.32.2\` then ` +
|
||||
`\`gbrain apply-migrations --yes\`. Or drain individual rows via \`forget_fact\`.`,
|
||||
);
|
||||
return result;
|
||||
}
|
||||
@@ -303,6 +355,11 @@ export async function runExtractFacts(
|
||||
result.warnings.push(
|
||||
...parsed.warnings.map(w => `${slug}: ${w}`),
|
||||
);
|
||||
// The parser deliberately skips malformed rows and returns any rows it
|
||||
// could still recover. That partial result is not authoritative: using
|
||||
// it for reconciliation would interpret skipped rows as deletions.
|
||||
// Preserve this page's existing index and continue with other pages.
|
||||
continue;
|
||||
}
|
||||
|
||||
if (parsed.facts.length > 0) result.pagesWithFacts += 1;
|
||||
@@ -334,9 +391,12 @@ export async function runExtractFacts(
|
||||
// partial-UNIQUE-index keyspace). #1928: `cli:`-origin facts
|
||||
// (conversation facts from extract-conversation-facts) are NOT
|
||||
// fence-owned — the page carries no `## Facts` fence to recreate
|
||||
// them — so they MUST survive this reconcile.
|
||||
// them — so they MUST survive this reconcile. #2646: soft-expired
|
||||
// legacy rows (forget_fact's record of the forget) likewise
|
||||
// survive via preserveExpiredLegacy.
|
||||
const deleted = await engine.deleteFactsForPage(slug, sourceId, {
|
||||
excludeSourcePrefixes: ['cli:'],
|
||||
preserveExpiredLegacy: true,
|
||||
});
|
||||
result.factsDeleted += deleted.deleted;
|
||||
}
|
||||
@@ -363,10 +423,11 @@ export async function runExtractFacts(
|
||||
if (hasStaleExisting || hasDuplicateExisting || hasRowNumDrift) {
|
||||
// Fall back to the legacy page-level reconcile when old DB rows must
|
||||
// be removed. Same delete scoping as above: legacy
|
||||
// NULL-source_markdown_slug rows and `cli:`-origin conversation
|
||||
// facts (#1928) survive.
|
||||
// NULL-source_markdown_slug rows, `cli:`-origin conversation
|
||||
// facts (#1928), and soft-expired legacy rows (#2646) survive.
|
||||
const deleted = await engine.deleteFactsForPage(slug, sourceId, {
|
||||
excludeSourcePrefixes: ['cli:'],
|
||||
preserveExpiredLegacy: true,
|
||||
});
|
||||
result.factsDeleted += deleted.deleted;
|
||||
toInsert = extracted;
|
||||
|
||||
+80
-12
@@ -20,6 +20,7 @@
|
||||
|
||||
import { join, dirname } from 'node:path';
|
||||
import { mkdirSync, writeFileSync } from 'node:fs';
|
||||
import { randomUUID } from 'node:crypto';
|
||||
import type { BrainEngine } from '../engine.ts';
|
||||
import type { PhaseResult, PhaseError } from '../cycle.ts';
|
||||
import { MinionQueue } from '../minions/queue.ts';
|
||||
@@ -29,7 +30,14 @@ import { serializeMarkdown } from '../markdown.ts';
|
||||
import type { Page, PageType } from '../types.ts';
|
||||
// #2415: allow-list + output-root resolution shared with the synthesize
|
||||
// phase — both phases must agree on the configured namespace.
|
||||
import { loadAllowedSlugPrefixes, loadOutputRoot } from './synthesize.ts';
|
||||
// runPgliteSubagentsInline is shared too: PGLite has no separate Minions
|
||||
// worker process (the embedded data-dir holds an exclusive file lock), so a
|
||||
// job submitted via queue.add() sits in 'waiting' forever unless something
|
||||
// drives the claim -> run -> complete loop inline. synthesize.ts already
|
||||
// does this for its own children; patterns.ts previously submitted and
|
||||
// waited without ever draining, so every real (non-dry-run) invocation on a
|
||||
// PGLite brain hung until subagentWaitTimeoutMs (default 35 min).
|
||||
import { loadAllowedSlugPrefixes, loadOutputRoot, runPgliteSubagentsInline } from './synthesize.ts';
|
||||
import { probeChatModel } from '../ai/gateway.ts';
|
||||
import { normalizeModelId } from '../model-id.ts';
|
||||
|
||||
@@ -115,7 +123,7 @@ export async function runPhasePatterns(
|
||||
}
|
||||
|
||||
// Gather reflections within lookback window.
|
||||
const reflections = await gatherReflections(engine, config.lookbackDays, config.outputRoot);
|
||||
const reflections = await gatherReflections(engine, config.lookbackDays, config.sourceSlugPrefix);
|
||||
if (reflections.length < config.minEvidence) {
|
||||
return skipped(
|
||||
'insufficient_evidence',
|
||||
@@ -152,6 +160,16 @@ export async function runPhasePatterns(
|
||||
return failed(makeError('InternalError', 'NO_ALLOWLIST',
|
||||
'skills/_brain-filing-rules.json missing dream_synthesize_paths.globs'));
|
||||
}
|
||||
// A configured dream.patterns.output_slug_prefix diverging from the
|
||||
// default `${outputRoot}/personal/patterns` composition (e.g. a flat
|
||||
// schema with no personal/ nesting) is not covered by the filing-rules
|
||||
// globs above, which only remap the `wiki/personal/patterns/*` literal
|
||||
// by outputRoot. Add it explicitly so the subagent's put_page allow-list
|
||||
// actually grants write access to wherever it's configured to write.
|
||||
const outputGlob = `${config.outputSlugPrefix}/*`;
|
||||
if (!allowedSlugPrefixes.includes(outputGlob)) {
|
||||
allowedSlugPrefixes.push(outputGlob);
|
||||
}
|
||||
|
||||
// #2781: budget the subagent from the REMAINING parent-job time, not
|
||||
// the fixed config default. Checked after the cheap gates (disabled /
|
||||
@@ -167,8 +185,15 @@ export async function runPhasePatterns(
|
||||
}
|
||||
|
||||
const queue = new MinionQueue(engine);
|
||||
// PGLite children drain inline (no separate worker can open the embedded
|
||||
// data-dir), so give this job a private per-run queue: the inline drain
|
||||
// must never claim unrelated 'default'-queue jobs a Postgres worker owns.
|
||||
// Mirrors synthesize.ts's childQueueName derivation exactly.
|
||||
const childQueueName = engine.kind === 'pglite'
|
||||
? `dream-inline-${Date.now()}-${randomUUID().slice(0, 8)}`
|
||||
: 'default';
|
||||
const data: SubagentHandlerData = {
|
||||
prompt: buildPatternsPrompt(reflections, config.minEvidence, config.outputRoot),
|
||||
prompt: buildPatternsPrompt(reflections, config.minEvidence, config.sourceSlugPrefix, config.outputSlugPrefix),
|
||||
model: config.model,
|
||||
max_turns: 30,
|
||||
allowed_slug_prefixes: allowedSlugPrefixes,
|
||||
@@ -176,11 +201,19 @@ export async function runPhasePatterns(
|
||||
const submitOpts: Partial<MinionJobInput> = {
|
||||
max_stalled: 3,
|
||||
timeout_ms: budgets.timeoutMs,
|
||||
queue: childQueueName,
|
||||
};
|
||||
const job = await queue.add('subagent', data as unknown as Record<string, unknown>, submitOpts, {
|
||||
allowProtectedSubmit: true,
|
||||
});
|
||||
|
||||
// PGLite cannot run a separate Minions worker because the embedded DB
|
||||
// holds an exclusive file lock. Drain this phase's private child queue
|
||||
// inline so the parent observes the terminal state instead of polling
|
||||
// waitForCompletion until subagentWaitTimeoutMs expires. No-op on
|
||||
// Postgres (a real worker process claims the job there).
|
||||
await runPgliteSubagentsInline(engine, queue, childQueueName, opts.yieldDuringPhase);
|
||||
|
||||
let outcome: string;
|
||||
try {
|
||||
const final = await waitForCompletion(queue, job.id, {
|
||||
@@ -273,6 +306,21 @@ interface PatternsConfig {
|
||||
model: string;
|
||||
/** #2415: shared output namespace (dream.synthesize.output_root, default 'wiki'). */
|
||||
outputRoot: string;
|
||||
/**
|
||||
* Slug prefix `gatherReflections` reads from (SQL `LIKE` scope). Defaults
|
||||
* to `${outputRoot}/personal/reflections`, matching pre-existing behavior.
|
||||
* Config `dream.patterns.source_slug_prefix` overrides it for brains whose
|
||||
* schema has no `personal/reflections/` convention (e.g. a flat
|
||||
* `meetings/` tree) so the phase can read from wherever compiled_truth
|
||||
* excerpts actually live.
|
||||
*/
|
||||
sourceSlugPrefix: string;
|
||||
/**
|
||||
* Slug prefix new pattern pages are written under. Defaults to
|
||||
* `${outputRoot}/personal/patterns`, matching pre-existing behavior.
|
||||
* Config `dream.patterns.output_slug_prefix` overrides it.
|
||||
*/
|
||||
outputSlugPrefix: string;
|
||||
/** #1594-family: subagent job timeout, config `dream.patterns.subagent_timeout_ms`. */
|
||||
subagentTimeoutMs: number;
|
||||
/** #1594-family: waitForCompletion timeout, config `dream.patterns.subagent_wait_timeout_ms`. */
|
||||
@@ -289,6 +337,14 @@ async function getNumberConfig(engine: BrainEngine, key: string, fallback: numbe
|
||||
return Number.isNaN(value) ? fallback : value;
|
||||
}
|
||||
|
||||
/** Trims leading/trailing slashes from a config-supplied slug prefix; falls back to `fallback` when unset or empty after trimming. */
|
||||
async function getSlugPrefixConfig(engine: BrainEngine, key: string, fallback: string): Promise<string> {
|
||||
const raw = await engine.getConfig(key);
|
||||
if (!raw) return fallback;
|
||||
const trimmed = raw.trim().replace(/^\/+|\/+$/g, '');
|
||||
return trimmed || fallback;
|
||||
}
|
||||
|
||||
async function loadPatternsConfig(engine: BrainEngine): Promise<PatternsConfig> {
|
||||
const enabledStr = await engine.getConfig('dream.patterns.enabled');
|
||||
const enabled = enabledStr === null ? true : enabledStr === 'true';
|
||||
@@ -302,12 +358,19 @@ async function loadPatternsConfig(engine: BrainEngine): Promise<PatternsConfig>
|
||||
tier: 'reasoning',
|
||||
fallback: 'sonnet',
|
||||
});
|
||||
const outputRoot = await loadOutputRoot(engine);
|
||||
return {
|
||||
enabled,
|
||||
lookbackDays: lookbackStr ? Math.max(1, parseInt(lookbackStr, 10) || 30) : 30,
|
||||
minEvidence: minEvidenceStr ? Math.max(1, parseInt(minEvidenceStr, 10) || 3) : 3,
|
||||
model,
|
||||
outputRoot: await loadOutputRoot(engine),
|
||||
outputRoot,
|
||||
sourceSlugPrefix: await getSlugPrefixConfig(
|
||||
engine, 'dream.patterns.source_slug_prefix', `${outputRoot}/personal/reflections`,
|
||||
),
|
||||
outputSlugPrefix: await getSlugPrefixConfig(
|
||||
engine, 'dream.patterns.output_slug_prefix', `${outputRoot}/personal/patterns`,
|
||||
),
|
||||
subagentTimeoutMs: await getNumberConfig(
|
||||
engine, 'dream.patterns.subagent_timeout_ms', DEFAULT_PATTERNS_SUBAGENT_TIMEOUT_MS,
|
||||
),
|
||||
@@ -328,11 +391,11 @@ interface ReflectionRef {
|
||||
async function gatherReflections(
|
||||
engine: BrainEngine,
|
||||
lookbackDays: number,
|
||||
outputRoot = 'wiki',
|
||||
sourceSlugPrefix = 'wiki/personal/reflections',
|
||||
): Promise<ReflectionRef[]> {
|
||||
const since = new Date(Date.now() - lookbackDays * 24 * 60 * 60 * 1000).toISOString();
|
||||
// #2415: reflections live under the configured output root (bound as a
|
||||
// parameter; outputRoot is slug-grammar-validated by loadOutputRoot).
|
||||
// Reflections live under the configured source slug prefix (bound as a
|
||||
// parameter; see PatternsConfig.sourceSlugPrefix / dream.patterns.source_slug_prefix).
|
||||
const rows = await engine.executeRaw<{ slug: string; title: string | null; compiled_truth: string | null }>(
|
||||
`SELECT slug, title, compiled_truth
|
||||
FROM pages
|
||||
@@ -340,7 +403,7 @@ async function gatherReflections(
|
||||
AND updated_at >= $1::timestamptz
|
||||
ORDER BY updated_at DESC
|
||||
LIMIT 100`,
|
||||
[since, `${outputRoot}/personal/reflections/%`],
|
||||
[since, `${sourceSlugPrefix}/%`],
|
||||
);
|
||||
return rows.map(r => ({
|
||||
slug: r.slug,
|
||||
@@ -351,7 +414,12 @@ async function gatherReflections(
|
||||
|
||||
// ── Prompt ────────────────────────────────────────────────────────────
|
||||
|
||||
function buildPatternsPrompt(reflections: ReflectionRef[], minEvidence: number, outputRoot = 'wiki'): string {
|
||||
function buildPatternsPrompt(
|
||||
reflections: ReflectionRef[],
|
||||
minEvidence: number,
|
||||
sourceSlugPrefix = 'wiki/personal/reflections',
|
||||
outputSlugPrefix = 'wiki/personal/patterns',
|
||||
): string {
|
||||
const today = new Date().toISOString().slice(0, 10);
|
||||
const corpus = reflections
|
||||
.map((r, i) => `### ${i + 1}. [[${r.slug}]] — ${r.title}\n${r.excerpt}`)
|
||||
@@ -361,15 +429,15 @@ function buildPatternsPrompt(reflections: ReflectionRef[], minEvidence: number,
|
||||
|
||||
OUTPUT POLICY
|
||||
- Only name a pattern if it appears in at least ${minEvidence} DISTINCT reflections.
|
||||
- Each pattern page MUST cite the reflections that constitute its evidence (use [[${outputRoot}/personal/reflections/...]] wikilinks).
|
||||
- Each pattern page MUST cite the reflections that constitute its evidence (use [[${sourceSlugPrefix}/...]] wikilinks).
|
||||
- Use \`search\` to check whether a similar pattern page already exists; if yes, update it (use the same slug). If no, create a new one.
|
||||
- Pattern slug format: \`${outputRoot}/personal/patterns/<topic-slug>\` (lowercase alphanumeric + hyphens; no underscores, no extension, no date).
|
||||
- Pattern slug format: \`${outputSlugPrefix}/<topic-slug>\` (lowercase alphanumeric + hyphens; no underscores, no extension, no date).
|
||||
- A "pattern" is a recurring theme, anxiety, decision pattern, relationship dynamic, or self-knowledge motif. NOT a single insight. NOT a list of unrelated topics.
|
||||
|
||||
DO NOT WRITE
|
||||
- A "patterns from today" digest (that's the dream-cycle-summaries page; not your job).
|
||||
- Patterns with <${minEvidence} reflections cited.
|
||||
- Anything outside ${outputRoot}/personal/patterns/.
|
||||
- Anything outside ${outputSlugPrefix}/.
|
||||
|
||||
CONTEXT
|
||||
- Today: ${today}
|
||||
|
||||
@@ -33,7 +33,7 @@ import { chat as gatewayChat, validateModelId, type ChatResult } from '../ai/gat
|
||||
import { AIConfigError } from '../ai/errors.ts';
|
||||
import { normalizeModelId } from '../model-id.ts';
|
||||
import { hasAnthropicKey } from '../ai/anthropic-key.ts';
|
||||
import { join, dirname, isAbsolute, resolve } from 'node:path';
|
||||
import { basename, join, dirname, isAbsolute, resolve } from 'node:path';
|
||||
import type { BrainEngine } from '../engine.ts';
|
||||
import type { PhaseResult, PhaseError } from '../cycle.ts';
|
||||
import { MinionQueue } from '../minions/queue.ts';
|
||||
@@ -48,8 +48,9 @@ import { safeSplitIndex } from '../text-safe.ts';
|
||||
import { PAGE_SLUG_SEG } from '../cjk.ts';
|
||||
|
||||
// Slug grammar from validatePageSlug — shared via PAGE_SLUG_SEG (#738).
|
||||
// Used for the orchestrator-written summary index slug.
|
||||
const SUMMARY_SLUG_RE = new RegExp(`^${PAGE_SLUG_SEG}(\\/${PAGE_SLUG_SEG})*$`);
|
||||
// Used for the orchestrator-written summary index slug. `u` flag required
|
||||
// by PAGE_SLUG_SEG's \p{...} classes (#3417).
|
||||
const SUMMARY_SLUG_RE = new RegExp(`^${PAGE_SLUG_SEG}(\\/${PAGE_SLUG_SEG})*$`, 'u');
|
||||
|
||||
// ── Model context budget (D1, D5, D7, D9) ─────────────────────────────
|
||||
|
||||
@@ -276,7 +277,7 @@ const INLINE_PGLITE_LOCK_MS = 30_000;
|
||||
* `yieldDuringPhase` is ticked on a 60s interval while a child runs so the
|
||||
* 5-min cycle lock TTL keeps refreshing during long (up to 30-min) children.
|
||||
*/
|
||||
async function runPgliteSubagentsInline(
|
||||
export async function runPgliteSubagentsInline(
|
||||
engine: BrainEngine,
|
||||
queue: MinionQueue,
|
||||
queueName: string,
|
||||
@@ -560,20 +561,31 @@ export async function runPhaseSynthesize(
|
||||
const skipReports: Array<{ filePath: string; reason: string }> = [];
|
||||
|
||||
const maxCharsPerChunk = computeChunkCharBudget(config.model, config.maxPromptTokens);
|
||||
const successfulLegacyKeys = await loadSuccessfulLegacySynthesisKeys(
|
||||
engine,
|
||||
opts.sourceId ?? 'default',
|
||||
);
|
||||
|
||||
for (const t of worthProcessing) {
|
||||
const hash16 = t.contentHash.slice(0, 16);
|
||||
const hash6 = t.contentHash.slice(0, 6);
|
||||
|
||||
// D8: single→multi-chunk migration safety. If a completed legacy
|
||||
// single-chunk job exists for this content_hash, treat as already-
|
||||
// synthesized and skip. Prevents duplicate writes when a transcript
|
||||
// that was previously single-chunk now multi-chunks (because budget
|
||||
// shrank or model changed).
|
||||
if (await hasLegacySingleChunkCompletion(engine, t.filePath, hash16)) {
|
||||
// D8: legacy-key migration safety. If this content hash already
|
||||
// completed under the pre-v2 path-based key family — single-chunk OR
|
||||
// a full chunked set — treat as already-synthesized and skip.
|
||||
// Prevents a full paid re-synthesis when the corpus root moves or
|
||||
// the chunking outcome changes across versions.
|
||||
const legacyCompletion = findLegacyCompletion(
|
||||
successfulLegacyKeys,
|
||||
t.filePath,
|
||||
hash16,
|
||||
);
|
||||
if (legacyCompletion) {
|
||||
skipReports.push({
|
||||
filePath: t.filePath,
|
||||
reason: 'already_synthesized_legacy_single_chunk',
|
||||
reason: legacyCompletion === 'chunked'
|
||||
? 'already_synthesized_legacy_chunked'
|
||||
: 'already_synthesized_legacy_single_chunk',
|
||||
});
|
||||
continue;
|
||||
}
|
||||
@@ -617,15 +629,15 @@ export async function runPhaseSynthesize(
|
||||
// so put_page writes land there instead of the hardcoded 'default'.
|
||||
...(opts.sourceId ? { source_id: opts.sourceId } : {}),
|
||||
};
|
||||
// Idempotency key parity:
|
||||
// - single-chunk → legacy `dream:synth:<filePath>:<hash16>` (byte-
|
||||
// equivalent across versions; preserves dedup for unchanged
|
||||
// transcripts on upgrade).
|
||||
// - multi-chunk → `<legacy>:c<i>of<n>` per chunk; durable across
|
||||
// runs because D9 splitTranscriptByBudget is hash-deterministic.
|
||||
// Keep producer identity stable when the corpus root moves. Source and
|
||||
// complete filename remain explicit so equal bytes in different source
|
||||
// or filename namespaces do not collide.
|
||||
const synthesisKey =
|
||||
`dream:synth-v2:${encodeURIComponent(opts.sourceId ?? 'default')}` +
|
||||
`:filename:${encodeURIComponent(basename(t.filePath))}:${hash16}`;
|
||||
const idempotency_key = isChunked
|
||||
? `dream:synth:${t.filePath}:${hash16}:c${i}of${chunks.length}`
|
||||
: `dream:synth:${t.filePath}:${hash16}`;
|
||||
? `${synthesisKey}:c${i}of${chunks.length}`
|
||||
: synthesisKey;
|
||||
const submitOpts: Partial<MinionJobInput> = {
|
||||
max_stalled: 3,
|
||||
on_child_fail: 'continue',
|
||||
@@ -1251,7 +1263,7 @@ async function collectChildPutPageSlugs(
|
||||
// cycle's resolved source via SubagentHandlerData.source_id, and stamps
|
||||
// the SAME source here so reverseWriteRefs / provenance reads target the
|
||||
// correct (source_id, slug) row. Unset → legacy 'default'.
|
||||
const rows = await engine.executeRaw<{ job_id: number; slug: string }>(
|
||||
const rows = await engine.executeRaw<{ job_id: number | bigint; slug: string }>(
|
||||
`SELECT job_id,
|
||||
COALESCE(input->>'slug', (input #>> '{}')::jsonb->>'slug') AS slug
|
||||
FROM subagent_tool_executions
|
||||
@@ -1265,10 +1277,13 @@ async function collectChildPutPageSlugs(
|
||||
const rewritten = new Map<string, string | undefined>();
|
||||
for (const r of rows) {
|
||||
if (typeof r.slug !== 'string' || r.slug.length === 0) continue;
|
||||
const ci = chunkInfo.get(r.job_id);
|
||||
// Postgres decodes the BIGINT FK as bigint; both metadata maps are keyed
|
||||
// by the INTEGER minion job id represented as a JavaScript number.
|
||||
const jobId = Number(r.job_id);
|
||||
const ci = chunkInfo.get(jobId);
|
||||
const slug = ci ? rewriteChunkedSlug(r.slug, ci.hash6, ci.idx) : r.slug;
|
||||
if (!rewritten.has(slug) || rewritten.get(slug) === undefined) {
|
||||
rewritten.set(slug, jobRawSource?.get(r.job_id));
|
||||
rewritten.set(slug, jobRawSource?.get(jobId));
|
||||
}
|
||||
}
|
||||
return Array.from(rewritten.keys()).sort().map(slug => {
|
||||
@@ -1278,29 +1293,73 @@ async function collectChildPutPageSlugs(
|
||||
}
|
||||
|
||||
/**
|
||||
* D8: query for any `completed` legacy single-chunk job at the canonical
|
||||
* idempotency key shape `dream:synth:<filePath>:<hash16>`. Used at fan-out
|
||||
* time to detect transcripts that were synthesized under the pre-chunking
|
||||
* code path; those should NOT be re-submitted under chunked keys.
|
||||
* D8: load every `completed` legacy job key in the pre-v2 path-based
|
||||
* family `dream:synth:<filePath>:<hash16>[:c<i>of<n>]`. Used at fan-out
|
||||
* time to detect transcripts already synthesized under an old key shape;
|
||||
* those should NOT be re-submitted under v2 keys. (v2 keys start with
|
||||
* `dream:synth-v2:` and don't match the LIKE prefix — the queue's own
|
||||
* idempotency dedupe already covers them.)
|
||||
*
|
||||
* Reuses the existing `minion_jobs.idempotency_key` index — no schema
|
||||
* additions. One indexed lookup per worth-processing transcript.
|
||||
* Plain `status = 'completed'` deliberately mirrors the queue-level
|
||||
* idempotency semantics the legacy keys relied on: a completed job blocks
|
||||
* re-submission regardless of `result.stop_reason` (pinned in
|
||||
* test/minions.test.ts). Filtering on stop_reason here would re-pay for
|
||||
* transcripts the old code path never re-ran, and reading `result` at all
|
||||
* would need the `(result #>> '{}')` double-encoded-jsonb defense.
|
||||
*
|
||||
* Loads source-scoped completions once per phase; no schema additions
|
||||
* and no repeated history scan for each transcript.
|
||||
*/
|
||||
async function hasLegacySingleChunkCompletion(
|
||||
async function loadSuccessfulLegacySynthesisKeys(
|
||||
engine: BrainEngine,
|
||||
sourceId: string,
|
||||
): Promise<string[]> {
|
||||
const rows = await engine.executeRaw<{ idempotency_key: string }>(
|
||||
`SELECT idempotency_key
|
||||
FROM minion_jobs
|
||||
WHERE name = 'subagent'
|
||||
AND status = 'completed'
|
||||
AND COALESCE(NULLIF(data->>'source_id', ''), 'default') = $1
|
||||
AND idempotency_key LIKE 'dream:synth:%'`,
|
||||
[sourceId],
|
||||
);
|
||||
return rows.map(row => row.idempotency_key);
|
||||
}
|
||||
|
||||
/**
|
||||
* Match a transcript (by filename + content hash) against completed legacy
|
||||
* keys. `'single'` when a `dream:synth:<path>:<hash16>` completion exists;
|
||||
* `'chunked'` when a FULL chunk set `:c0of<n>`..`:c<n-1>of<n>` completed
|
||||
* (chunk indices are 0-based). Partial chunk sets return null so the
|
||||
* transcript gets a fresh v2 synthesis instead of shipping with holes.
|
||||
*/
|
||||
function findLegacyCompletion(
|
||||
successfulKeys: string[],
|
||||
filePath: string,
|
||||
hash16: string,
|
||||
): Promise<boolean> {
|
||||
const legacyKey = `dream:synth:${filePath}:${hash16}`;
|
||||
const rows = await engine.executeRaw<{ status: string }>(
|
||||
`SELECT status
|
||||
FROM minion_jobs
|
||||
WHERE idempotency_key = $1
|
||||
AND status = 'completed'
|
||||
LIMIT 1`,
|
||||
[legacyKey],
|
||||
);
|
||||
return rows.length > 0;
|
||||
): 'single' | 'chunked' | null {
|
||||
const filename = basename(filePath);
|
||||
const hashSuffix = `:${hash16}`;
|
||||
/** total chunk count n → completed 0-based chunk indices */
|
||||
const chunkSets = new Map<number, Set<number>>();
|
||||
for (const key of successfulKeys) {
|
||||
const chunk = /:c(\d+)of(\d+)$/.exec(key);
|
||||
const base = chunk ? key.slice(0, -chunk[0].length) : key;
|
||||
if (!base.endsWith(hashSuffix)) continue;
|
||||
const historicalPath = base.slice('dream:synth:'.length, -hashSuffix.length);
|
||||
if (basename(historicalPath) !== filename) continue;
|
||||
if (!chunk) return 'single';
|
||||
const i = Number(chunk[1]);
|
||||
const n = Number(chunk[2]);
|
||||
if (n < 1 || i < 0 || i >= n) continue;
|
||||
let seen = chunkSets.get(n);
|
||||
if (!seen) chunkSets.set(n, seen = new Set());
|
||||
seen.add(i);
|
||||
}
|
||||
for (const [n, seen] of chunkSets) {
|
||||
if (seen.size === n) return 'chunked';
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
// ── Dream-provenance DB stamp (#2569) ────────────────────────────────
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user